Use OpenAI Computer-Use Helpers with Daytona Desktops

View as Markdown

This guide shows how to let an OpenAI model operate a browser running on a Daytona desktop. Daytona provides the desktop, and OpenAI’s ComputerToolset helper runs the model’s computer-use loop: it takes screenshots, sends them to the model, and executes the actions the model requests. The helper ships in the OpenAI Python SDK 3.31.0 and the OpenAI Node SDK 7.33.0. The toolset never creates or manages a sandbox, so your application stays in control of which actions the model may actually take.

Both SDKs document the driver contract in their “Connect a browser or desktop” provider pages: Python and Node. This guide uses Python.

1. Workflow Overview

The example is a single script. It creates a Daytona sandbox, starts its desktop, opens the Python documentation home page in Chromium, and connects a deliberately restrictive driver to OpenAI’s native computer tool. The model may take screenshots and exactly one plain left click, and only after you answer a y/N consent prompt. The script then prints the model’s answer, which should be the heading of the page that opened, and deletes the sandbox:

$ python main.py
Allow screenshots and one click? [y/N] y
The Python Tutorial
Deleted sandbox 05f343d2-e007-469e-8887-d221bf2f2737

The pieces that make this work:

Desktop (Daytona): a sandbox with Computer Use started, serving screenshots and accepting mouse input.
Driver (your application): a small class that implements the OpenAI driver methods and decides what each one is allowed to do.
Toolset (OpenAI): ComputerToolset, which runs the screenshot-and-act loop against the model.

2. Project Setup

Clone the Repository

git clone https://github.com/daytona/guides.git
cd guides/python/computer-use/openai-computer-use

Configure Environment

The example reads three environment variables:

OPENAI_API_KEY: your OpenAI key, from the OpenAI Developer Platform.
OPENAI_MODEL: the model that drives the desktop. It must support the native computer tool.
DAYTONA_API_KEY: your Daytona key, from the Daytona Dashboard.

Copy .env.example to .env and add your keys:

OPENAI_API_KEY=your_openai_api_key
OPENAI_MODEL=gpt-6-astra
DAYTONA_API_KEY=your_daytona_key

Each run bills OpenAI model usage and Daytona sandbox runtime.

Install and Run

pip install "openai==3.31.0" "daytona==0.224.1" python-dotenv
python main.py

3. Understanding the Integration

Create the Desktop

The script creates a sandbox from the default snapshot, which already contains the Computer Use packages, then starts the desktop processes. start() returns before the desktop is usable, so the script polls get_status() until the status reports active and only then opens the browser. Chromium is launched on DISPLAY=:0 in the background with its output redirected, so the call returns immediately, and the script waits for the page to load:

daytona = Daytona()
sandbox = None
toolset = None
try:
    sandbox = daytona.create(
        CreateSandboxFromSnapshotParams(
            auto_stop_interval=20,
            auto_delete_interval=60,
        ),
        timeout=REQUEST_TIMEOUT,
    )
    sandbox.computer_use.start(request_timeout=REQUEST_TIMEOUT)

    deadline = time.monotonic() + 60
    while True:
        status = sandbox.computer_use.get_status(
            request_timeout=REQUEST_TIMEOUT
        ).status
        if status == "active":
            break
        if time.monotonic() > deadline:
            raise RuntimeError(
                "Computer use did not become active in time "
                f"(last status: {status})."
            )
        time.sleep(2)

    sandbox.process.exec(
        f"DISPLAY=:0 nohup chromium --no-sandbox --disable-dev-shm-usage "
        f"{START_URL} >/tmp/cu-browser.log 2>&1 </dev/null &",
        timeout=REQUEST_TIMEOUT,
    )
    time.sleep(10)

auto_stop_interval=20 and auto_delete_interval=60 bound the sandbox’s lifetime, in minutes, if the script is interrupted before its cleanup runs.

Connect the Desktop

DaytonaDriver implements the nine methods ComputerToolset requires. Only two of them do anything: screenshot returns a base64 PNG in the computer_screenshot shape the toolset expects, and click allows a single plain left click inside the display bounds. Every other action name is aliased to unsupported, which raises:

class DaytonaDriver:
    def __init__(self, computer, display):
        self.computer = computer
        self.display = display
        self.clicked = False

    def screenshot(self):
        image = self.computer.screenshot.take_full_screen(
            show_cursor=False,
            request_timeout=REQUEST_TIMEOUT,
        )
        if not image.screenshot:
            raise RuntimeError("Daytona returned no screenshot.")
        return {
            "type": "computer_screenshot",
            "image_url": f"data:image/png;base64,{image.screenshot}",
        }

    def click(self, action):
        # Limit the model to the single click required by this example.
        if self.clicked or action.button != "left" or action.keys:
            raise RuntimeError("Only one plain left click is allowed.")
        inside_x = 0 <= action.x < self.display.width
        inside_y = 0 <= action.y < self.display.height
        if not inside_x or not inside_y:
            raise RuntimeError("Click is outside the display.")
        self.clicked = True
        self.computer.mouse.click(
            action.x,
            action.y,
            button="left",
            request_timeout=REQUEST_TIMEOUT,
        )

    def unsupported(self, action):
        raise RuntimeError(f"Unsupported action: {action.type}")

    double_click = drag = keypress = move = unsupported
    scroll = type = wait = unsupported

Run the Model

The approval callbacks belong to run(), not to the ComputerToolset constructor. before_action is called before every action and returns True only for the two allowed types, and approve_safety_checks returns False so a pending safety check stops the run instead of being waved through:

# Gate actions before dispatch even though unsupported driver methods also raise.
def before_action(action):
    return action.type in ("screenshot", "click")


# This example stops rather than approving pending model safety checks.
def approve_safety_checks(call):
    return False

The driver is bound to the one active display at (0, 0), so its bounds check matches the screen the model sees. ComputerToolset then takes the driver and nothing else, and run() drives the loop:

    displays = sandbox.computer_use.display.get_info(
        request_timeout=REQUEST_TIMEOUT
    ).displays or []
    active = [display for display in displays if display.is_active]
    if len(active) != 1 or active[0].x != 0 or active[0].y != 0:
        raise RuntimeError("Expected one active display at (0, 0).")
    if not active[0].width or not active[0].height:
        raise RuntimeError("Display dimensions are unavailable.")

    driver = DaytonaDriver(sandbox.computer_use, active[0])
    toolset = ComputerToolset(driver)
    request = {
        "model": os.environ["OPENAI_MODEL"],
        "store": False,
        "tools": [{"type": "computer"}],
        "tool_choice": "required",
        "input": TASK,
    }
    with OpenAI(timeout=30, max_retries=0) as client:
        response = toolset.run(
            client,
            request,
            max_steps=MAX_STEPS,
            before_action=before_action,
            approve_safety_checks=approve_safety_checks,
        )
    if not driver.clicked:
        raise RuntimeError("The model returned without clicking.")
    print(response.output_text)

store: False keeps the conversation out of OpenAI’s storage, and tool_choice: "required" forces the model to use the computer tool rather than answering from memory. The script raises if driver.clicked is still False, which catches a model that answers without clicking.

Clean Up

A finally block closes the toolset and deletes the sandbox, so an error mid-run still leaves nothing behind. The deletion sits in its own nested finally, so it also runs if toolset.close() raises, and wait=True returns only once Daytona has removed the sandbox:

finally:
    try:
        if toolset is not None:
            toolset.close()
    finally:
        if sandbox is not None:
            sandbox.delete(wait=True)
            print(f"Deleted sandbox {sandbox.id}")

4. Approvals and Safety

The consent prompt runs before anything else, including the first screenshot and the sandbox creation. Answer anything other than y and the script exits without contacting Daytona or OpenAI.

The driver, not the model, decides what happens on the desktop. It permits exactly one plain left click inside the display and raises on every other action, so a model that tries to type, scroll, or drag ends the run with an error. Pending model safety checks are never approved automatically; the callback returns False and the run stops. Review each check yourself before deciding to allow it.

Screenshots are sent to OpenAI as part of the model request, so treat everything visible on the desktop as shared with the model. The model’s answer is not proof that it clicked the right thing. The example only checks, through driver.clicked, that the driver accepted one click; it does not check where the click landed, so verify the result independently before relying on it. Use a dedicated, disposable desktop with no real accounts, sessions, or credentials on it.

5. Adapt the Example

To let the model do more, extend the driver one method at a time with the same bounds checks, rather than loosening the safety callbacks. The callbacks are the last line of defence, and a permissive driver cannot be undone by a strict prompt.

For faster repeated runs, create a Daytona snapshot with the browser and any page state already in place, so each run skips the install and first-load wait. Restrict the sandbox’s network to only the domains the task needs with the allow-list parameters on sandbox creation.

If you need more than screenshots and clicks, see Daytona’s Computer Use page for the full mouse, keyboard, display, and screen-recording API.

Key advantages:

Self-contained and disposable: the script creates and deletes its own desktop, with no manual setup
Restrictive by default: the model can only take the actions your driver explicitly allows
Decoupled: OpenAI owns the model loop, Daytona owns the desktop, and your application owns the policy