Evaluate Computer-Use Agents on Windows Sandboxes

Windows GUI tasks are a useful stress test for computer-use agents. They include the dialogs, menus, and desktop apps that terminal-only benchmarks miss.

This guide builds a reusable Windows snapshot, defines an 8-task eval suite, and runs Claude Code or Codex CLI through Daytona’s MCP tools.

What you’ll build

• A Windows snapshot based on windows-medium, with Python 3.13 and the eval files already installed.
• An 8-task GUI suite that uses stock Windows apps such as Notepad, Explorer, Calculator, Paint, and PowerShell.
• A harness that runs Claude Code or Codex CLI through Daytona’s computer_use_* MCP tools.
• Verifiers that run outside the agent loop and check the real filesystem, registry, or app output after each task.

---

1. Workflow Overview

1. Snapshot: prep_snapshot.py builds windows-cu-evals-v1 once.
2. Suite: tasks.py defines the 8 tasks, their setup, verifiers, and oracles.
3. Harness: harness.py runs an agent over the suite, one fresh sandbox per episode.
4. Artifacts: each run saves screenshots and a run log for every episode.

2. Setup

Requirements

• Python: Version 3.10 or higher (on your machine; the sandbox gets its own)
• Daytona: An account and API key
• Agent CLIs: Claude Code and/or Codex CLI, only for --agent claude / --agent codex runs. The oracle and nop agents need neither.

Clone the Repository

git clone https://github.com/daytona/guides.git
cd guides/python/computer-use/windows-evals

Create and Activate Virtual Environment

Create and activate a virtual environment:

python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Install Dependencies

Install dependencies:

pip install -e .

Configure Environment

Set your Daytona API key in .env (copy from .env.example):

cp .env.example .env
# edit .env with your API key

• DAYTONA_API_KEY: Required for all scripts. Get it from the Daytona Dashboard
• ANTHROPIC_API_KEY: Only for harness.py --agent claude
• OPENAI_API_KEY: Only for harness.py --agent codex

3. Build the eval snapshot

Daytona allows you to create sandboxes from a snapshot, and create a new snapshot from a sandbox you have modified. prep_snapshot.py creates a sandbox from windows-medium, installs what the suite needs, stops the sandbox, snapshots its filesystem as windows-cu-evals-v1, and then deletes it.

python prep_snapshot.py

[1/6] Creating preparation sandbox from 'windows-medium'...
      Sandbox ready: windows-cu-evals-v1-prep-20260706133816 (12a1ea6e-...)
[2/6] Downloading Python 3.13.9 installer...
      28761776
      Launching unattended Python installer...
      LaunchedPid=5848
      Waiting for the installer to finish...
      Python 3.13.9
[3/6] Removing installer at C:\python-installer.exe...
      installer removed
[4/6] Seeding C:\evals...
      C:\evals ready
[5/6] Stopping sandbox before creating a filesystem snapshot...
      Creating snapshot 'windows-cu-evals-v1' from stopped sandbox...
      Waiting for snapshot to become active...
[6/6] Snapshot ready: windows-cu-evals-v1
Usage hint: python harness.py --snapshot windows-cu-evals-v1
Deleting preparation sandbox...

From here on, every episode creates a fresh sandbox from the derived snapshot:

Python

from daytona import CreateSandboxFromSnapshotParams, Daytona

daytona = Daytona()  # reads DAYTONA_API_KEY from the environment
sandbox = daytona.create(CreateSandboxFromSnapshotParams(snapshot="windows-cu-evals-v1"))
sandbox.computer_use.start()
# ... run one episode ...
sandbox.delete()

4. The task suite

Each task is a plain dataclass:

Python

@dataclass
class Task:
    id: str
    instruction: str
    setup: list[str]
    verify: str
    oracle: list[str]
    timeout_s: int = 300

instruction is the only thing the agent sees. setup, verify, and oracle are PowerShell scripts the harness runs from outside the agent loop. The script prints PASS as its last non-empty line, or FAIL followed by a reason.

Task id	App(s)	Difficulty	Verifier checks
notepad-write-save	Notepad	easy	C:\evals\report.txt exists with the exact sentence, case-sensitive
explorer-folder-tree	Explorer	easy	both projects\alpha\docs and projects\beta\docs exist under C:\evals
env-var-gui	System Properties dialog	medium	HKCU:\Environment has EVALS_PROBE equal to 42 (User variable, not System)
zip-roundtrip	Explorer	medium	C:\evals\archive.zip expands to exactly alpha.txt, beta.txt, gamma.txt
calc-to-notepad	Calculator, Notepad	medium	C:\evals\result.txt contains 53361
mspaint-save	Paint	medium	C:\evals\drawing.png is over 1 KiB and starts with PNG magic bytes
powershell-interactive	PowerShell window	medium	C:\evals\when.txt exists and is non-empty
find-and-fix	Explorer, Notepad	hard	the one real settings.ini says colour=blue, no file under C:\evals\cfg still says colour=blu, decoys untouched

Every task has a headless oracle. The oracle scripts solve each task through PowerShell alone, so you can test the verifiers without LLM costs: --agent oracle must pass all 8 tasks and --agent nop (which does nothing) must fail all 8.

5. Run agents over MCP

Check the verifiers first, then run a real agent:

# sanity: oracle must go 8/8, nop must go 0/8
python harness.py --agent oracle
python harness.py --agent nop

# score Claude Code on a subset
python harness.py --agent claude --tasks notepad-write-save,calc-to-notepad

# full suite, four episodes at a time
python harness.py --agent codex --parallel 4

For each episode the harness creates a sandbox, writes the agent CLI an MCP server config that runs daytona mcp start, and hands the CLI a prompt. The MCP server exposes the computer_use_* tools (screenshot, mouse, keyboard, accessibility, windows) plus a small set of file tools such as list_files and file_download.

Each run saves an initial and final screenshot per episode under runs/<ts>/<task-id>/, which is usually the fastest way to debug a failed task.

6. Add custom tasks

To add a task, create another Task in tasks.py with an id, an agent-facing instruction, setup scripts, a verifier, and an oracle solution.

Keep the verifier outside the agent loop. It should check the final state of the sandbox and print PASS or FAIL as its last non-empty line. Keep the oracle too: python harness.py --agent oracle --tasks your-task-id should pass, and python harness.py --agent nop --tasks your-task-id should fail.

If the task needs extra software, files, fonts, browser state, registry settings, or first-run app configuration, add that setup to prep_snapshot.py and rebuild the snapshot. Avoid installing dependencies at episode start. Every task should begin from the same ready-to-use desktop, so failures come from the agent behavior, not from machine setup.

Tips for reliable evals

• Keep setup in the snapshot. If a task depends on an app, file, registry value, font, or first-run setting, prepare it in prep_snapshot.py so that sandboxes begin pre-built with everything they need for a task.
• Verify final state as deterministically as possible. The verifier should inspect files, registry keys, screenshots, or app output after the agent is done.
• Run oracle and nop before scoring agents. This catches broken tasks before you spend time on real runs.