# Evaluate Computer-Use Agents on Windows Sandboxes

Windows GUI tasks are a useful stress test for computer-use agents. They include the dialogs, menus, and desktop apps that terminal-only benchmarks miss.

This guide builds a reusable Windows snapshot, defines an 8-task eval suite, and runs Claude Code or Codex CLI through Daytona's MCP tools.

#### What you'll build

- A Windows snapshot based on `windows-medium`, with Python 3.13 and the eval files already installed.
- An 8-task GUI suite that uses stock Windows apps such as Notepad, Explorer, Calculator, Paint, and PowerShell.
- A harness that runs Claude Code or Codex CLI through Daytona's `computer_use_*` MCP tools.
- Verifiers that run outside the agent loop and check the real filesystem, registry, or app output after each task.

---

### 1. Workflow Overview

1. **Snapshot:** `prep_snapshot.py` builds `windows-cu-evals-v1` once.
2. **Suite:** `tasks.py` defines the 8 tasks, their setup, verifiers, and oracles.
3. **Harness:** `harness.py` runs an agent over the suite, one fresh sandbox per episode.
4. **Artifacts:** each run saves screenshots and a run log for every episode.

### 2. Setup

#### Requirements

- **Python:** Version 3.10 or higher (on your machine; the sandbox gets its own)
- **Daytona:** An account and API key
- **Agent CLIs:** Claude Code and/or Codex CLI, only for `--agent claude` / `--agent codex` runs. The `oracle` and `nop` agents need neither.

#### Clone the Repository


```bash
git clone https://github.com/daytona/guides.git
cd guides/python/computer-use/windows-evals
```

#### Create and Activate Virtual Environment

Create and activate a virtual environment:

```bash
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
```

#### Install Dependencies

Install dependencies:

```bash
pip install -e .
```

#### Configure Environment

Set your Daytona API key in `.env` (copy from `.env.example`):

```bash
cp .env.example .env
# edit .env with your API key
```

- `DAYTONA_API_KEY`: Required for all scripts. Get it from the [Daytona Dashboard](https://app.daytona.io/dashboard/keys)
- `ANTHROPIC_API_KEY`: Only for `harness.py --agent claude`
- `OPENAI_API_KEY`: Only for `harness.py --agent codex`

### 3. Build the eval snapshot

Daytona allows you to create sandboxes from a snapshot, and create a new snapshot from a sandbox you have modified. `prep_snapshot.py` creates a sandbox from `windows-medium`, installs what the suite needs, stops the sandbox, snapshots its filesystem as `windows-cu-evals-v1`, and then deletes it. 

```bash
python prep_snapshot.py
```

{/* <!-- SYNC: prep_snapshot.py --> */}
```
[1/6] Creating preparation sandbox from 'windows-medium'...
      Sandbox ready: windows-cu-evals-v1-prep-20260706133816 (12a1ea6e-...)
[2/6] Downloading Python 3.13.9 installer...
      28761776
      Launching unattended Python installer...
      LaunchedPid=5848
      Waiting for the installer to finish...
      Python 3.13.9
[3/6] Removing installer at C:\python-installer.exe...
      installer removed
[4/6] Seeding C:\evals...
      C:\evals ready
[5/6] Stopping sandbox before creating a filesystem snapshot...
      Creating snapshot 'windows-cu-evals-v1' from stopped sandbox...
      Waiting for snapshot to become active...
[6/6] Snapshot ready: windows-cu-evals-v1
Usage hint: python harness.py --snapshot windows-cu-evals-v1
Deleting preparation sandbox...
```

From here on, every episode creates a fresh sandbox from the derived snapshot:

{/* <!-- SYNC: common.py --> */}
    ```python
    from daytona import CreateSandboxFromSnapshotParams, Daytona

    daytona = Daytona()  # reads DAYTONA_API_KEY from the environment
    sandbox = daytona.create(CreateSandboxFromSnapshotParams(snapshot="windows-cu-evals-v1"))
    sandbox.computer_use.start()
    # ... run one episode ...
    sandbox.delete()
    ```

### 4. The task suite

Each task is a plain dataclass:

{/* <!-- SYNC: tasks.py --> */}
    ```python
    @dataclass
    class Task:
        id: str
        instruction: str
        setup: list[str]
        verify: str
        oracle: list[str]
        timeout_s: int = 300
    ```

`instruction` is the only thing the agent sees. `setup`, `verify`, and `oracle` are PowerShell scripts the harness runs from outside the agent loop. The script prints `PASS` as its last non-empty line, or `FAIL` followed by a reason.

| Task id | App(s) | Difficulty | Verifier checks |
| --- | --- | --- | --- |
| `notepad-write-save` | Notepad | easy | `C:\evals\report.txt` exists with the exact sentence, case-sensitive |
| `explorer-folder-tree` | Explorer | easy | both `projects\alpha\docs` and `projects\beta\docs` exist under `C:\evals` |
| `env-var-gui` | System Properties dialog | medium | `HKCU:\Environment` has `EVALS_PROBE` equal to `42` (User variable, not System) |
| `zip-roundtrip` | Explorer | medium | `C:\evals\archive.zip` expands to exactly `alpha.txt`, `beta.txt`, `gamma.txt` |
| `calc-to-notepad` | Calculator, Notepad | medium | `C:\evals\result.txt` contains `53361` |
| `mspaint-save` | Paint | medium | `C:\evals\drawing.png` is over 1 KiB and starts with PNG magic bytes |
| `powershell-interactive` | PowerShell window | medium | `C:\evals\when.txt` exists and is non-empty |
| `find-and-fix` | Explorer, Notepad | hard | the one real `settings.ini` says `colour=blue`, no file under `C:\evals\cfg` still says `colour=blu`, decoys untouched |

**Every task has a headless oracle.** The `oracle` scripts solve each task through PowerShell alone, so you can test the verifiers without LLM costs: `--agent oracle` must pass all 8 tasks and `--agent nop` (which does nothing) must fail all 8.

### 5. Run agents over MCP

Check the verifiers first, then run a real agent:

{/* <!-- SYNC: harness.py --> */}
```bash
# sanity: oracle must go 8/8, nop must go 0/8
python harness.py --agent oracle
python harness.py --agent nop

# score Claude Code on a subset
python harness.py --agent claude --tasks notepad-write-save,calc-to-notepad

# full suite, four episodes at a time
python harness.py --agent codex --parallel 4
```

For each episode the harness creates a sandbox, writes the agent CLI an MCP server config that runs `daytona mcp start`, and hands the CLI a prompt. The MCP server exposes the `computer_use_*` tools (screenshot, mouse, keyboard, accessibility, windows) plus a small set of file tools such as `list_files` and `file_download`.

Each run saves an initial and final screenshot per episode under `runs/<ts>/<task-id>/`, which is usually the fastest way to debug a failed task.

### 6. Add custom tasks

To add a task, create another `Task` in `tasks.py` with an `id`, an agent-facing `instruction`, setup scripts, a verifier, and an oracle solution.

Keep the verifier outside the agent loop. It should check the final state of the sandbox and print `PASS` or `FAIL` as its last non-empty line. Keep the oracle too: `python harness.py --agent oracle --tasks your-task-id` should pass, and `python harness.py --agent nop --tasks your-task-id` should fail.

If the task needs extra software, files, fonts, browser state, registry settings, or first-run app configuration, add that setup to `prep_snapshot.py` and rebuild the snapshot. Avoid installing dependencies at episode start. Every task should begin from the same ready-to-use desktop, so failures come from the agent behavior, not from machine setup.

:::tip[Tips for reliable evals]
- Keep setup in the snapshot. If a task depends on an app, file, registry value, font, or first-run setting, prepare it in `prep_snapshot.py` so that sandboxes begin pre-built with everything they need for a task.
- Verify final state as deterministically as possible. The verifier should inspect files, registry keys, screenshots, or app output after the agent is done.
- Run `oracle` and `nop` before scoring agents. This catches broken tasks before you spend time on real runs.
:::