AI-powered visual verification for GUI testing. More...

#include <ai_vision_verifier.h>

Collaboration diagram for yaze::test::AIVisionVerifier:

[legend]

Public Member Functions
	AIVisionVerifier (const VisionVerifierConfig &config={})

	~AIVisionVerifier ()

void	SetConfig (const VisionVerifierConfig &config)

const VisionVerifierConfig &	GetConfig () const

void	SetScreenshotCallback (ScreenshotCaptureCallback callback)

void	SetAIService (cli::AIService *service)
	Set the AI service to use for vision verification.

absl::StatusOr< VisionVerificationResult >	Verify (const std::string &condition)
	Verify a single condition using AI vision.

absl::StatusOr< VisionVerificationResult >	VerifyConditions (const std::vector< std::string > &conditions)
	Verify multiple conditions in a single screenshot.

absl::StatusOr< VisionVerificationResult >	CompareToReference (const std::string &reference_path, float tolerance=0.1f)
	Compare current state against a reference screenshot.

absl::StatusOr< std::string >	AskAboutState (const std::string &question)
	Ask the AI an open-ended question about the current state.

absl::StatusOr< VisionVerificationResult >	VerifyTileAt (int x, int y, int expected_tile_id)
	Verify tile at canvas position matches expected tile ID.

absl::StatusOr< VisionVerificationResult >	VerifyPanelVisible (const std::string &panel_name)
	Verify that a specific editor panel is visible.

absl::StatusOr< VisionVerificationResult >	VerifyEmulatorState (const std::string &state_description)
	Verify game state in emulator matches expected values.

absl::StatusOr< VisionVerificationResult >	VerifySpriteAt (int x, int y, const std::string &sprite_description)
	Verify sprite rendering at specific location.

absl::StatusOr< std::string >	CaptureScreenshot (const std::string &name)
	Capture and save a screenshot.

const std::vector< uint8_t > &	GetLastScreenshotData () const
	Get the last captured screenshot data.

void	ClearScreenshotCache ()
	Clear cached screenshots to free memory.

void	BeginIterativeSession (int max_iterations=5)
	Begin an iterative verification session.

absl::Status	AddIterativeCheck (const std::string &condition)
	Add a verification to the iterative session.

absl::StatusOr< VisionVerificationResult >	CompleteIterativeSession ()
	Complete the iterative session and get results.

Private Member Functions
absl::StatusOr< std::string >	CaptureAndEncodeScreenshot ()

absl::StatusOr< std::string >	CallVisionModel (const std::string &prompt, const std::string &image_base64)

VisionVerificationResult	ParseAIResponse (const std::string &response, const std::string &screenshot_path)

Private Attributes
VisionVerifierConfig	config_

ScreenshotCaptureCallback	screenshot_callback_

cli::AIService *	ai_service_ = nullptr

std::vector< uint8_t >	last_screenshot_data_

int	last_width_ = 0

int	last_height_ = 0

bool	in_iterative_session_ = false

int	iterative_max_iterations_ = 5

int	iterative_current_iteration_ = 0

std::vector< std::string >	iterative_conditions_

std::vector< VisionVerificationResult >	iterative_results_

Detailed Description

AI-powered visual verification for GUI testing.

This class provides multimodal AI capabilities for verifying GUI state through screenshot analysis. It integrates with Gemini, Ollama, or other vision-capable models to enable intelligent test assertions.

Usage example:

  AIVisionVerifier verifier(config);
  verifier.SetScreenshotCallback(my_capture_func);
 
  // Simple verification
  auto result = verifier.Verify("The overworld map should show Link at
position (100, 200)");
 
  // Structured verification
  auto result = verifier.VerifyConditions({
    "The tile selector shows 16x16 tiles",
    "The palette panel is visible on the right",
    "No error dialogs are displayed"
  });

Definition at line 101 of file ai_vision_verifier.h.

Constructor & Destructor Documentation

◆ AIVisionVerifier()

yaze::test::AIVisionVerifier::AIVisionVerifier ( const VisionVerifierConfig & config = {} )

explicit

Definition at line 20 of file ai_vision_verifier.cc.

◆ ~AIVisionVerifier()

yaze::test::AIVisionVerifier::~AIVisionVerifier ( )

default

Member Function Documentation

◆ SetConfig()

void yaze::test::AIVisionVerifier::SetConfig ( const VisionVerifierConfig & config )

inline

Definition at line 107 of file ai_vision_verifier.h.

◆ GetConfig()

const VisionVerifierConfig & yaze::test::AIVisionVerifier::GetConfig ( ) const

inline

Definition at line 108 of file ai_vision_verifier.h.

References config_.

◆ SetScreenshotCallback()

void yaze::test::AIVisionVerifier::SetScreenshotCallback ( ScreenshotCaptureCallback callback )

inline

Definition at line 111 of file ai_vision_verifier.h.

References screenshot_callback_.

◆ SetAIService()

void yaze::test::AIVisionVerifier::SetAIService ( cli::AIService * service )

inline

Set the AI service to use for vision verification.

When set, uses the provided AIService (e.g., GeminiAIService) for multimodal requests. When not set, uses placeholder responses.

Parameters

service Pointer to an AIService instance (caller owns memory)

Definition at line 123 of file ai_vision_verifier.h.

◆ Verify()

absl::StatusOr< VisionVerificationResult > yaze::test::AIVisionVerifier::Verify ( const std::string & condition )

Verify a single condition using AI vision.

Parameters

condition Natural language description of expected state.

Returns: Verification result with AI assessment.

Definition at line 25 of file ai_vision_verifier.cc.

References CallVisionModel(), CaptureAndEncodeScreenshot(), and ParseAIResponse().

Referenced by AddIterativeCheck(), VerifyEmulatorState(), VerifyPanelVisible(), VerifySpriteAt(), and VerifyTileAt().

Here is the call graph for this function:

◆ VerifyConditions()

absl::StatusOr< VisionVerificationResult > yaze::test::AIVisionVerifier::VerifyConditions ( const std::vector< std::string > & conditions )

Verify multiple conditions in a single screenshot.

Parameters

conditions List of expected conditions.

Returns: Combined verification result.

Definition at line 67 of file ai_vision_verifier.cc.

References CallVisionModel(), CaptureAndEncodeScreenshot(), and ParseAIResponse().

Here is the call graph for this function:

◆ CompareToReference()

absl::StatusOr< VisionVerificationResult > yaze::test::AIVisionVerifier::CompareToReference	(	const std::string &	reference_path,
		float	tolerance = 0.1f )

Compare current state against a reference screenshot.

Parameters

reference_path	Path to reference screenshot.
tolerance	Visual difference tolerance (0.0 = exact, 1.0 = any).

Returns: Verification result with comparison details.

Definition at line 110 of file ai_vision_verifier.cc.

References CallVisionModel(), CaptureAndEncodeScreenshot(), and ParseAIResponse().

Here is the call graph for this function:

◆ AskAboutState()

absl::StatusOr< std::string > yaze::test::AIVisionVerifier::AskAboutState ( const std::string & question )

Ask the AI an open-ended question about the current state.

Parameters

question Question to ask about the screenshot.

Returns: AI's response.

Definition at line 145 of file ai_vision_verifier.cc.

References CallVisionModel(), and CaptureAndEncodeScreenshot().

Here is the call graph for this function:

◆ VerifyTileAt()

absl::StatusOr< VisionVerificationResult > yaze::test::AIVisionVerifier::VerifyTileAt	(	int	x,
		int	y,
		int	expected_tile_id )

Verify tile at canvas position matches expected tile ID.

Definition at line 159 of file ai_vision_verifier.cc.

References Verify().

Here is the call graph for this function:

◆ VerifyPanelVisible()

absl::StatusOr< VisionVerificationResult > yaze::test::AIVisionVerifier::VerifyPanelVisible ( const std::string & panel_name )

Verify that a specific editor panel is visible.

Definition at line 167 of file ai_vision_verifier.cc.

References Verify().

Here is the call graph for this function:

◆ VerifyEmulatorState()

absl::StatusOr< VisionVerificationResult > yaze::test::AIVisionVerifier::VerifyEmulatorState ( const std::string & state_description )

Verify game state in emulator matches expected values.

Definition at line 175 of file ai_vision_verifier.cc.

References Verify().

Here is the call graph for this function:

◆ VerifySpriteAt()

absl::StatusOr< VisionVerificationResult > yaze::test::AIVisionVerifier::VerifySpriteAt	(	int	x,
		int	y,
		const std::string &	sprite_description )

Verify sprite rendering at specific location.

Definition at line 182 of file ai_vision_verifier.cc.

References Verify().

Here is the call graph for this function:

◆ CaptureScreenshot()

absl::StatusOr< std::string > yaze::test::AIVisionVerifier::CaptureScreenshot ( const std::string & name )

Capture and save a screenshot.

Parameters

name	Name for the screenshot file.

Returns: Path to saved screenshot.

Todo: Implement PNG saving

Definition at line 190 of file ai_vision_verifier.cc.

References config_, last_height_, last_screenshot_data_, last_width_, LOG_DEBUG, screenshot_callback_, and yaze::test::VisionVerifierConfig::screenshot_dir.

◆ GetLastScreenshotData()

const std::vector< uint8_t > & yaze::test::AIVisionVerifier::GetLastScreenshotData ( ) const

inline

Get the last captured screenshot data.

Definition at line 196 of file ai_vision_verifier.h.

◆ ClearScreenshotCache()

void yaze::test::AIVisionVerifier::ClearScreenshotCache ( )

Clear cached screenshots to free memory.

Definition at line 212 of file ai_vision_verifier.cc.

References last_height_, last_screenshot_data_, and last_width_.

◆ BeginIterativeSession()

void yaze::test::AIVisionVerifier::BeginIterativeSession ( int max_iterations = 5 )

Begin an iterative verification session.

Useful for complex verifications where the AI may need multiple screenshots to confirm a condition (e.g., animation completed).

Definition at line 218 of file ai_vision_verifier.cc.

References in_iterative_session_, iterative_conditions_, iterative_current_iteration_, iterative_max_iterations_, and iterative_results_.

Referenced by yaze::test::ScopedIterativeVerification::ScopedIterativeVerification().

◆ AddIterativeCheck()

absl::Status yaze::test::AIVisionVerifier::AddIterativeCheck ( const std::string & condition )

Add a verification to the iterative session.

Definition at line 226 of file ai_vision_verifier.cc.

References in_iterative_session_, iterative_conditions_, iterative_current_iteration_, iterative_max_iterations_, iterative_results_, and Verify().

Here is the call graph for this function:

◆ CompleteIterativeSession()

absl::StatusOr< VisionVerificationResult > yaze::test::AIVisionVerifier::CompleteIterativeSession ( )

Complete the iterative session and get results.

Definition at line 246 of file ai_vision_verifier.cc.

References yaze::test::VisionVerificationResult::confidence, yaze::test::VisionVerificationResult::discrepancies, in_iterative_session_, iterative_results_, yaze::test::VisionVerificationResult::observations, and yaze::test::VisionVerificationResult::passed.

◆ CaptureAndEncodeScreenshot()

absl::StatusOr< std::string > yaze::test::AIVisionVerifier::CaptureAndEncodeScreenshot ( )

private

Todo: Encode to base64 for API calls

Definition at line 276 of file ai_vision_verifier.cc.

References last_height_, last_screenshot_data_, last_width_, and screenshot_callback_.

Referenced by AskAboutState(), CompareToReference(), Verify(), and VerifyConditions().

◆ CallVisionModel()

absl::StatusOr< std::string > yaze::test::AIVisionVerifier::CallVisionModel	(	const std::string &	prompt,
		const std::string &	image_base64 )

private

Definition at line 292 of file ai_vision_verifier.cc.

References ai_service_, config_, yaze::cli::GeminiAIService::GenerateMultimodalResponse(), yaze::cli::AIService::GenerateResponse(), last_height_, last_screenshot_data_, last_width_, LOG_DEBUG, yaze::test::VisionVerifierConfig::model_name, and yaze::test::VisionVerifierConfig::screenshot_dir.

Referenced by AskAboutState(), CompareToReference(), Verify(), and VerifyConditions().

Here is the call graph for this function:

◆ ParseAIResponse()

VisionVerificationResult yaze::test::AIVisionVerifier::ParseAIResponse	(	const std::string &	response,
		const std::string &	screenshot_path )

private

Definition at line 355 of file ai_vision_verifier.cc.

References yaze::test::VisionVerificationResult::ai_response, yaze::test::VisionVerificationResult::confidence, yaze::test::VisionVerificationResult::discrepancies, yaze::test::VisionVerificationResult::observations, yaze::test::VisionVerificationResult::passed, and yaze::test::VisionVerificationResult::screenshot_path.

Referenced by CompareToReference(), Verify(), and VerifyConditions().

Member Data Documentation

◆ config_

VisionVerifierConfig yaze::test::AIVisionVerifier::config_

private

Definition at line 233 of file ai_vision_verifier.h.

Referenced by CallVisionModel(), CaptureScreenshot(), and GetConfig().

◆ screenshot_callback_

ScreenshotCaptureCallback yaze::test::AIVisionVerifier::screenshot_callback_

private

Definition at line 234 of file ai_vision_verifier.h.

Referenced by CaptureAndEncodeScreenshot(), CaptureScreenshot(), and SetScreenshotCallback().

◆ ai_service_

cli::AIService* yaze::test::AIVisionVerifier::ai_service_ = nullptr

private

Definition at line 235 of file ai_vision_verifier.h.

Referenced by CallVisionModel().

◆ last_screenshot_data_

std::vector<uint8_t> yaze::test::AIVisionVerifier::last_screenshot_data_

private

Definition at line 236 of file ai_vision_verifier.h.

Referenced by CallVisionModel(), CaptureAndEncodeScreenshot(), CaptureScreenshot(), and ClearScreenshotCache().

◆ last_width_

int yaze::test::AIVisionVerifier::last_width_ = 0

private

Definition at line 237 of file ai_vision_verifier.h.

Referenced by CallVisionModel(), CaptureAndEncodeScreenshot(), CaptureScreenshot(), and ClearScreenshotCache().

◆ last_height_

int yaze::test::AIVisionVerifier::last_height_ = 0

private

Definition at line 238 of file ai_vision_verifier.h.

Referenced by CallVisionModel(), CaptureAndEncodeScreenshot(), CaptureScreenshot(), and ClearScreenshotCache().

◆ in_iterative_session_

bool yaze::test::AIVisionVerifier::in_iterative_session_ = false

private

Definition at line 241 of file ai_vision_verifier.h.

Referenced by AddIterativeCheck(), BeginIterativeSession(), and CompleteIterativeSession().

◆ iterative_max_iterations_

int yaze::test::AIVisionVerifier::iterative_max_iterations_ = 5

private

Definition at line 242 of file ai_vision_verifier.h.

Referenced by AddIterativeCheck(), and BeginIterativeSession().

◆ iterative_current_iteration_

int yaze::test::AIVisionVerifier::iterative_current_iteration_ = 0

private

Definition at line 243 of file ai_vision_verifier.h.

Referenced by AddIterativeCheck(), and BeginIterativeSession().

◆ iterative_conditions_

std::vector<std::string> yaze::test::AIVisionVerifier::iterative_conditions_

private

Definition at line 244 of file ai_vision_verifier.h.

Referenced by AddIterativeCheck(), and BeginIterativeSession().

◆ iterative_results_

std::vector<VisionVerificationResult> yaze::test::AIVisionVerifier::iterative_results_

private

Definition at line 245 of file ai_vision_verifier.h.

Referenced by AddIterativeCheck(), BeginIterativeSession(), and CompleteIterativeSession().

The documentation for this class was generated from the following files:

src/app/test/ai_vision_verifier.h
src/app/test/ai_vision_verifier.cc

	yaze 0.3.2 Link to the Past ROM Editor

Public Member Functions

Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ AIVisionVerifier()

◆ ~AIVisionVerifier()

Member Function Documentation

◆ SetConfig()

◆ GetConfig()

◆ SetScreenshotCallback()

◆ SetAIService()

◆ Verify()

◆ VerifyConditions()

◆ CompareToReference()

◆ AskAboutState()

◆ VerifyTileAt()

◆ VerifyPanelVisible()

◆ VerifyEmulatorState()

◆ VerifySpriteAt()

◆ CaptureScreenshot()

◆ GetLastScreenshotData()

◆ ClearScreenshotCache()

◆ BeginIterativeSession()

◆ AddIterativeCheck()

◆ CompleteIterativeSession()

◆ CaptureAndEncodeScreenshot()

◆ CallVisionModel()

◆ ParseAIResponse()

Member Data Documentation

◆ config_

◆ screenshot_callback_

◆ ai_service_

◆ last_screenshot_data_

◆ last_width_

◆ last_height_

◆ in_iterative_session_

◆ iterative_max_iterations_

◆ iterative_current_iteration_

◆ iterative_conditions_

◆ iterative_results_