Overworld Agent Guide - AI-Powered Overworld Editing

Version: 1.0
Last Updated: October 6, 2025
Audience: AI Agents, z3ed users, automation developers

---

Overview

This guide explains how AI agents can interact with YAZE's overworld editor through the z3ed CLI and automation APIs. It covers:

• Available tools and commands
• Multimodal vision workflows
• Proposal-based editing
• Best practices for AI-generated edits

---

Quick Start

Prerequisites

# Build YAZE with AI and gRPC support
cmake -B build -DZ3ED_AI=ON -DYAZE_WITH_GRPC=ON
cmake --build build --target z3ed
 
# Set up AI provider (Gemini recommended for vision)
export GEMINI_API_KEY="your-key-here"

First Agent Interaction

# Ask AI about a map
z3ed agent simple-chat "What tiles are at position 10,10 on map 0?" --rom zelda3.sfc
 
# AI agent generates edits
z3ed agent run --prompt "Place trees in a 3x3 grid at position 10,10 on map 0" \
  --rom zelda3.sfc --sandbox
 
# Review and accept
z3ed agent diff --proposal-id <id>
z3ed agent accept --proposal-id <id>

---

Available Tools

Read-Only Tools (Safe for AI)

overworld-get-tile

Query tile ID at coordinates.

Purpose: Analyze existing tile placement
Safety: Read-only, no ROM modification
Rate Limit: None

{
  "tool": "overworld-get-tile",
  "parameters": {
    "map": 0,
    "x": 10,
    "y": 10
  }
}

Response:

{
  "tile_id": 66,
  "tile_id_hex": "0x0042",
  "position": {"x": 10, "y": 10}
}

Use Cases:

• Check what tile currently exists before painting
• Analyze patterns in tile placement
• Verify expected tiles after edits

---

overworld-get-visible-region

Get tiles currently visible on canvas.

Purpose: Understand what the user is looking at
Safety: Read-only
Rate Limit: None

{
  "tool": "overworld-get-visible-region",
  "parameters": {
    "map": 0
  }
}

Response:

{
  "region": {
    "x_start": 0,
    "y_start": 0,
    "x_end": 31,
    "y_end": 31
  },
  "tiles": [
    {"x": 0, "y": 0, "tile_id": 40},
    {"x": 1, "y": 0, "tile_id": 40},
    ...
  ]
}

Use Cases:

• Analyze visible area before suggesting edits
• Generate context-aware suggestions
• Understand user's current focus

---

overworld-analyze-region

Get tile composition and patterns in a region.

Purpose: Deep analysis of tile distribution
Safety: Read-only
Rate Limit: Large regions (>1000 tiles) may be slow

{
  "tool": "overworld-analyze-region",
  "parameters": {
    "map": 0,
    "x1": 0,
    "y1": 0,
    "x2": 31,
    "y2": 31
  }
}

Response:

{
  "tile_counts": {
    "40": 512,   // Grass
    "66": 128,   // Tree
    "80": 64     // Water
  },
  "patterns": [
    {
      "type": "forest",
      "center": {"x": 15, "y": 15},
      "size": {"width": 10, "height": 10}
    }
  ],
  "statistics": {
    "total_tiles": 1024,
    "unique_tiles": 15,
    "most_common_tile": 40
  }
}

Use Cases:

• Understand map composition before edits
• Detect patterns (forests, water bodies, paths)
• Generate statistics for reports

---

Write Tools (Sandboxed - Creates Proposals)

overworld-set-tile

Paint a single tile (creates proposal).

Purpose: Modify single tile
Safety: Sandboxed, creates proposal
Rate Limit: Reasonable (don't spam)

{
  "tool": "overworld-set-tile",
  "parameters": {
    "map": 0,
    "x": 10,
    "y": 10,
    "tile_id": 66
  }
}

Response:

{
  "proposal_id": "abc123",
  "success": true,
  "message": "Proposal created: Set tile at (10,10) to 0x0042"
}

Use Cases:

• Fix individual tiles
• Place objects at specific coordinates
• Correct tile placement errors

---

overworld-set-tiles-batch

Paint multiple tiles in one operation (creates proposal).

Purpose: Efficient multi-tile editing
Safety: Sandboxed, creates proposal
Rate Limit: Max 1000 tiles per batch

{
  "tool": "overworld-set-tiles-batch",
  "parameters": {
    "map": 0,
    "tiles": [
      {"x": 10, "y": 10, "tile_id": 66},
      {"x": 11, "y": 10, "tile_id": 66},
      {"x": 12, "y": 10, "tile_id": 66}
    ]
  }
}

Response:

{
  "proposal_id": "abc123",
  "tiles_painted": 3,
  "success": true
}

Use Cases:

• Create patterns (forests, paths, water bodies)
• Fill regions with specific tiles
• Generate complex map structures

---

Multimodal Vision Workflow

Step 1: Capture Canvas Screenshot

# From CLI
z3ed agent vision --capture-canvas "Overworld Canvas" \
  --prompt "Analyze this overworld map" \
  --rom zelda3.sfc
 
# From agent workflow
z3ed agent run --prompt "Analyze map 0 and suggest improvements" \
  --rom zelda3.sfc --sandbox

Step 2: AI Analyzes Screenshot

Gemini Vision API receives:

• Screenshot of canvas (PNG/JPEG)
• User prompt
• Context (map index, visible region)

AI returns:

{
  "analysis": {
    "observations": [
      "Grass tiles dominate the visible area",
      "Tree tiles are sparse and unnatural",
      "Water tiles at (15,15) have incorrect palette colors",
      "Path from (5,5) to (25,5) is broken"
    ],
    "composition_score": 6.5,
    "issues": [
      {
        "type": "sparse_trees",
        "severity": "medium",
        "location": {"x": 10, "y": 10},
        "suggestion": "Add more tree tiles for forest theme"
      }
    ]
  }
}

Step 3: Generate Edit Plan

AI creates actionable plan:

{
  "plan": [
    {
      "tool": "overworld-set-tiles-batch",
      "parameters": {
        "map": 0,
        "tiles": [
          {"x": 10, "y": 10, "tile_id": 66},
          {"x": 11, "y": 10, "tile_id": 66},
          {"x": 12, "y": 10, "tile_id": 66}
        ]
      },
      "reason": "Create denser forest area"
    }
  ]
}

Step 4: Execute Plan (Sandbox)

# z3ed executes plan in sandbox
z3ed agent run --plan plan.json --rom zelda3.sfc --sandbox

Step 5: Human Review

# View proposed changes
z3ed agent diff --proposal-id abc123
 
# Accept or reject
z3ed agent accept --proposal-id abc123
# or
z3ed agent reject --proposal-id abc123

---

Example Workflows

Workflow 1: Create Forest Area

User Prompt: "Create a forest clearing at position 15,15 with grass in the center"

AI Plan:

{
  "steps": [
    {
      "step": 1,
      "description": "Check current tiles in region",
      "tool": "overworld-analyze-region",
      "parameters": {
        "map": 0,
        "x1": 10,
        "y1": 10,
        "x2": 20,
        "y2": 20
      }
    },
    {
      "step": 2,
      "description": "Paint grass clearing (center 5x5)",
      "tool": "overworld-set-tiles-batch",
      "parameters": {
        "map": 0,
        "tiles": [
          {"x": 13, "y": 13, "tile_id": 40},
          {"x": 14, "y": 13, "tile_id": 40},
          ...
        ]
      }
    },
    {
      "step": 3,
      "description": "Plant trees around edges",
      "tool": "overworld-set-tiles-batch",
      "parameters": {
        "map": 0,
        "tiles": [
          {"x": 10, "y": 10, "tile_id": 66},
          {"x": 11, "y": 10, "tile_id": 66},
          ...
        ]
      }
    }
  ]
}

CLI Execution:

z3ed agent run --prompt "Create a forest clearing at position 15,15 with grass in the center" \
  --rom zelda3.sfc --sandbox

Result: Proposal created with 50+ tile changes

---

Workflow 2: Fix Tile Placement Errors

User Prompt: "Fix any water tiles that should be grass in the visible region"

AI Plan:

{
  "steps": [
    {
      "step": 1,
      "tool": "overworld-get-visible-region",
      "parameters": {"map": 0}
    },
    {
      "step": 2,
      "tool": "overworld-analyze-region",
      "parameters": {
        "map": 0,
        "x1": 0,
        "y1": 0,
        "x2": 31,
        "y2": 31
      }
    },
    {
      "step": 3,
      "description": "Identify misplaced water tiles",
      "logic": "Find water tiles (80) surrounded by grass (40)"
    },
    {
      "step": 4,
      "tool": "overworld-set-tiles-batch",
      "parameters": {
        "map": 0,
        "tiles": [
          {"x": 5, "y": 5, "tile_id": 40},
          {"x": 12, "y": 8, "tile_id": 40}
        ]
      }
    }
  ]
}

---

Workflow 3: Generate Path

User Prompt: "Create a path from (5,5) to (25,25) using path tiles"

AI Plan:

{
  "steps": [
    {
      "step": 1,
      "description": "Calculate path coordinates",
      "logic": "Line from (5,5) to (25,25)"
    },
    {
      "step": 2,
      "tool": "overworld-set-tiles-batch",
      "parameters": {
        "map": 0,
        "tiles": [
          {"x": 5, "y": 5, "tile_id": 50},
          {"x": 6, "y": 6, "tile_id": 50},
          {"x": 7, "y": 7, "tile_id": 50},
          ...
        ]
      }
    }
  ]
}

---

Common Tile IDs Reference

Grass & Ground

• 0x0028 (40) - Grass
• 0x0029 (41) - Dark grass
• 0x002A (42) - Dirt
• 0x002B (43) - Sand

Trees & Plants

• 0x0042 (66) - Tree
• 0x0043 (67) - Bush
• 0x0044 (68) - Flower

Water

• 0x0050 (80) - Water
• 0x0051 (81) - Deep water
• 0x0052 (82) - Shore

Paths & Roads

• 0x0032 (50) - Path
• 0x0033 (51) - Road
• 0x0034 (52) - Bridge

Structures

• 0x0060 (96) - Wall
• 0x0061 (97) - Door
• 0x0062 (98) - Window

---

Best Practices for AI Agents

1. Always Analyze Before Editing

# GOOD: Check current state first
z3ed agent run --prompt "Analyze map 0 then suggest improvements" --rom zelda3.sfc --sandbox
 
# BAD: Blindly paint without context
z3ed agent run --prompt "Paint trees everywhere" --rom zelda3.sfc --sandbox

2. Use Batch Operations

# GOOD: Single batch operation
overworld-set-tiles-batch (50 tiles)
 
# BAD: 50 individual operations
overworld-set-tile (×50)

3. Provide Clear Reasoning

{
  "tool": "overworld-set-tile",
  "parameters": {"x": 10, "y": 10, "tile_id": 66},
  "reason": "Creating forest theme - tree tile at center"
}

4. Respect Tile Boundaries

Large maps (0x00-0x09, 0x80-0x89) are 512×512 pixels = 32×32 tiles.
Don't paint beyond (31, 31) for these maps.

5. Check Visibility

{
  "step": 1,
  "tool": "overworld-get-visible-region",
  "reason": "Ensure tiles are visible before analysis"
}

6. Create Reversible Edits

Always generate proposals that can be rejected:

z3ed agent run --prompt "..." --rom zelda3.sfc --sandbox  # Creates proposal
z3ed agent reject --proposal-id abc123                     # Can undo

---

Error Handling

"Tile ID out of range"

• Cause: Invalid tile ID (>4095 for Tile16)
• Fix: Validate tile IDs before set-tile

"Coordinates out of bounds"

• Cause: Painting beyond map boundaries
• Fix: Check map dimensions (typically 32×32 tiles)

"Proposal rejected"

• Cause: Human reviewer rejected changes
• Fix: Analyze feedback, adjust plan, try again

"ROM file locked"

• Cause: ROM file open in another process
• Fix: Close other instances of YAZE

---

Testing AI-Generated Edits

Manual Testing

# Generate proposal
z3ed agent run --prompt "..." --rom zelda3.sfc --sandbox
 
# Review in YAZE GUI
yaze zelda3.sfc
# Open Debug → Agent Chat → Proposals
# Review proposal, accept/reject

Automated Testing

# GUI automation test
z3ed agent test replay overworld_ai_edit.jsonl --rom zelda3.sfc --grpc localhost:50051
 
# Validate tile placement
z3ed agent test assert --tile-at 10,10 --expected-tile 66 --rom zelda3.sfc

---

Advanced Techniques

Technique 1: Pattern Recognition

Use multimodal vision to detect patterns:

z3ed agent vision --capture-canvas "Overworld Canvas" \
  --prompt "Identify repeated tile patterns in this map" \
  --rom zelda3.sfc

AI detects:

• Forest clusters
• Water bodies
• Paths and roads
• Building layouts

Technique 2: Style Transfer

z3ed agent run --prompt "Make this map look like Kakariko Village from the dark world" \
  --rom zelda3.sfc --sandbox

AI:

1. Analyzes Kakariko Village (map 0x18)
2. Extracts tile palette and patterns
3. Applies similar patterns to target map

Technique 3: Procedural Generation

z3ed agent run --prompt "Generate a random forest area at 10,10 with natural-looking tree placement" \
  --rom zelda3.sfc --sandbox

AI uses procedural algorithms:

• Perlin noise for natural randomness
• Clustering for realistic tree placement
• Edge smoothing for organic boundaries

---

Integration with GUI Automation

Record Human Edits

# Record editing session
z3ed agent test record --suite overworld_forest.jsonl --rom zelda3.sfc

Replay for AI Training

# Replay recorded session
z3ed agent test replay overworld_forest.jsonl --rom zelda3.sfc
 
# AI learns from human edits
z3ed agent learn --from-recording overworld_forest.jsonl

Validate AI Edits

# AI generates edits
z3ed agent run --prompt "..." --rom zelda3.sfc --sandbox
 
# GUI automation validates
z3ed agent test verify --proposal-id abc123 --suite validation_tests.jsonl

---

Collaboration Features

Network Collaboration

# Connect to yaze-server
z3ed net connect ws://localhost:8765
 
# Join session
z3ed net join ABC123 --username "ai-agent"
 
# AI agent edits, humans review in real-time
z3ed agent run --prompt "..." --rom zelda3.sfc --sandbox
 
# Proposal synced to all participants

Proposal Voting

# Submit proposal to session
z3ed proposal submit --proposal-id abc123 --session ABC123
 
# Wait for votes
z3ed proposal wait --proposal-id abc123
 
# Check result
z3ed proposal status --proposal-id abc123
# Output: approved (3/3 votes)

---

Troubleshooting

Agent Not Responding

# Check AI provider
z3ed agent ping
 
# Test simple query
z3ed agent simple-chat "Hello" --rom zelda3.sfc

Tools Not Available

# Verify z3ed build
z3ed agent describe --resource overworld
 
# Should show:
# - overworld-get-tile
# - overworld-set-tile
# - overworld-analyze-region

gRPC Connection Failed

# Check YAZE is running with gRPC
z3ed agent test ping --grpc localhost:50051
 
# Start YAZE with gRPC enabled
yaze --enable-grpc zelda3.sfc

---

See Also

• Canvas Automation API - C++ API reference
• GUI Automation Scenarios - Test examples
• z3ed README - CLI documentation
• Multimodal Vision - Screenshot analysis