Computer Use

Daytona v2

Documentation for Daytona v2, currently in alpha. For the latest release, see Daytona v0.

Computer Use enables programmatic control of desktop environments within sandboxes. It provides mouse, keyboard, screenshot, screen recording, accessibility, and display operations for automating GUI interactions and testing desktop applications.

Computer Use and VNC work together to enable both manual and automated desktop interactions. VNC provides the visual interface for users to manually interact with the desktop, while Computer Use provides the programmatic API for AI agents to automate operations.

The desktop stack (D-Bus, Xvfb, the window manager, and the x11vnc server) ships with the sandbox runtime as a self-contained add-on, so it works on any Linux sandbox image, including images with no desktop packages. The virtual display runs at 1280x720. All Computer Use calls go through the toolbox API and require an API key with the sandbox write permission.

Use cases

Computer Use is used when a task needs a graphical desktop instead of an API or a shell.

• Computer-use agents: an agent reads screenshots and the accessibility tree, then moves the mouse and types to operate desktop applications.
• Desktop application tests: GUI tests click, type, and take screenshots to check the state of an application.
• Browser automation: a browser on the sandbox desktop is driven through mouse and keyboard input, for sites that have no API.
• Element-level automation: the accessibility tree finds a button or a text field by role, name, and state. An agent invokes the element or sets its text without screen coordinates.
• Screenshots for models: region and compressed screenshots capture part of the screen or a smaller image. An agent sends less image data to a model on each step.
• Window-aware automation: the window list reports the title, position, size, and active flag of each window. An agent checks which application is in front before it acts.
• Agents on macOS: macOS sandboxes provide screenshots, mouse and keyboard input, the accessibility tree, and screen recordings through the same API.
• Recorded sessions: screen recordings capture what an agent or a test did, for review and debugging. Recordings are MP4 files stored in the sandbox and downloaded through the API.
• Human in the loop: a person watches or takes over the same desktop through VNC while an agent works.

Start Computer Use

Start all computer use processes (D-Bus, Xvfb, the window manager, and the x11vnc server) in the sandbox. The call blocks until the X server and the VNC server accept connections and returns the status of each process. If either does not come up within 5 s, the desktop is marked failed and the call returns an error. Calling start on a desktop that is already ready returns the current status; calling it while the desktop is starting or failed returns an error.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()

result = sandbox.computer_use.start()
print("Computer use processes started:", result.message)
for name, process in result.status.items():
    print(f"{name}: running={process.running} pid={process.pid}")

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();

const result = await sandbox.computerUse.start();
console.log('Computer use processes started:', result.message);
for (const [name, process] of Object.entries(result.status)) {
  console.log(`${name}: running=${process.running} pid=${process.pid}`);
}

• restart() leaves the failed state.

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/start' \
  --request POST \
  --header 'Authorization: Bearer YOUR_API_KEY'

Stop Computer Use

Stop all computer use processes in the sandbox. Stopping a ready desktop returns it to not started. Stopping a failed desktop kills any remaining processes but keeps the failed state and its cause; only restart leaves the failed state.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

result = sandbox.computer_use.stop()
print("Computer use processes stopped:", result.message)

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const result = await sandbox.computerUse.stop();
console.log('Computer use processes stopped:', result.message);

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/stop' \
  --request POST \
  --header 'Authorization: Bearer YOUR_API_KEY'

Restart Computer Use

Tear the desktop down and build a new one. Restart is valid from every state and is the only way out of the failed state. Concurrent restart calls join the same in-flight restart and receive the same result. Application windows do not survive a restart.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

result = sandbox.computer_use.restart()
print(result.message, result.desktop)  # computer use restarted ready

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const result = await sandbox.computerUse.restart();
console.log(result.message, result.desktop); // computer use restarted ready

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/restart' \
  --request POST \
  --header 'Authorization: Bearer YOUR_API_KEY'

Get status

Get the desktop status. The status field is active only while the desktop field is ready.

desktop	status	Meaning
not_started	inactive	No desktop exists. Call start.
starting	inactive	Start is in progress. Poll status.
ready	active	Actions are accepted.
failed	inactive	The desktop is gone. cause.code and cause.message explain why. Call restart.

The cause object is present only when the desktop is failed. Possible cause codes:

Code	Meaning
desktop_exited	The X server (xvfb) exited. Windows in it are lost.
addon_crashed	The add-on that supervises the desktop died while a desktop existed.
start_failed	Creating the desktop or probing its readiness failed, for example a missing binary.
start_interrupted	The add-on died while the desktop was being created.
state_corrupt	Retained desktop state could not be read or validated.
state_persistence_failed	A state transition could not be persisted.
session_ended	Windows only. The interactive desktop session ended.
session_replaced	Windows only. Another interactive desktop session replaced the one the desktop was bound to.

Failure evidence is boot-scoped: a sandbox reboot resets the desktop to not started.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()

status = sandbox.computer_use.status()
print("Computer use status:", status.status, status.desktop)
if status.desktop == "failed":
    print(f"Failure: {status.cause.code}: {status.cause.message}")

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();

const status = await sandbox.computerUse.status();
console.log('Computer use status:', status.status, status.desktop);
if (status.desktop === 'failed') {
  console.log(`Failure: ${status.cause.code}: ${status.cause.message}`);
}

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/status' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Get process status

Get the status of one desktop process: D-Bus, Xvfb, the window manager, or the x11vnc server. Without a name, it returns the status of every process.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

xvfb_status = sandbox.computer_use.service.status("xvfb")
print(xvfb_status.process_name, xvfb_status.running)

• The process names are dbus, xvfb, wm, and x11vnc.

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const xvfbStatus = await sandbox.computerUse.service.status('xvfb');
console.log(xvfbStatus.processName, xvfbStatus.running);

• The process names are dbus, xvfb, wm, and x11vnc.

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/process/{processName}/status' \
  --header 'Authorization: Bearer YOUR_API_KEY'

• The process names are dbus, xvfb, wm, and x11vnc.

Get all process statuses

Get the status of every desktop process. The response is a map keyed by process name, with one entry each for D-Bus, Xvfb, the window manager, and the x11vnc server. Each entry reports whether the process is alive, its supervisor restart state, and its startup priority.

API field	Description
Running	Whether the process is alive.
Pid	Process ID while the process is running.
Status	running, stopped, or failed.
Failed	Whether the process entered the terminal failed state.
AutoRestart	Whether the supervisor restarts the process automatically.
RestartAttempts	Restart attempts consumed by the current generation of the process.
RestartBackoffMs	Current restart backoff in milliseconds.
Priority	Relative startup priority within the desktop stack.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

for name, process in sandbox.computer_use.service.status().items():
    print(f"{name}: {process.status} pid={process.pid} restarts={process.restart_attempts}")

• The call returns a dictionary keyed by process name: dbus, xvfb, wm, and x11vnc.
• The API fields map to snake_case attributes, for example RestartAttempts to restart_attempts and AutoRestart to auto_restart.

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const statuses = await sandbox.computerUse.service.status();
for (const [name, process] of Object.entries(statuses)) {
  console.log(`${name}: ${process.status} pid=${process.pid} restarts=${process.restartAttempts}`);
}

• The call returns a record keyed by process name: dbus, xvfb, wm, and x11vnc.
• The API fields map to camelCase properties, for example RestartAttempts to restartAttempts and AutoRestart to autoRestart.

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/process-status' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Restart process

Restart a specific desktop process. Per-process restart works only while the desktop is ready. The Xvfb process cannot be restarted on its own because application windows live inside it; use restart instead.

The D-Bus, window manager, and x11vnc processes are restarted automatically if they exit: up to 6 attempts with exponential backoff from 1 s to 30 s, reset after 30 s of uptime. When attempts run out, the process is marked failed in the status map while the desktop stays ready.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

result = sandbox.computer_use.service.restart("wm")
print(result.message, result.process_name)

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const result = await sandbox.computerUse.service.restart('wm');
console.log(result.message, result.processName);

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/process/{processName}/restart' \
  --request POST \
  --header 'Authorization: Bearer YOUR_API_KEY'

Get process logs

Get logs for a specific desktop process. Logs are retained up to 256 KiB per process and trimmed to the most recent 128 KiB while the process runs.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

logs = sandbox.computer_use.service.logs("x11vnc")
print("x11vnc logs:", logs.logs)

The response has the following fields:

Field	Description
logs	Holds the retained standard output of the process.

• Get process errors returns standard error.

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const logs = await sandbox.computerUse.service.logs('x11vnc');
console.log('x11vnc logs:', logs.logs);

The response has the following fields:

Field	Description
logs	Holds the retained standard output of the process.

• Get process errors returns standard error.

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/process/{processName}/logs' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Get process errors

Get error logs for a specific desktop process.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

errors = sandbox.computer_use.service.errors("x11vnc")
print("x11vnc errors:", errors.errors)

The response has the following fields:

Field	Description
errors	Holds the retained standard error output of the process. The output is capped at 256 KiB, like the logs. The value is an empty string when the process has written nothing to standard error.

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const errors = await sandbox.computerUse.service.errors('x11vnc');
console.log('x11vnc errors:', errors.errors);

The response has the following fields:

Field	Description
errors	Holds the retained standard error output of the process. The output is capped at 256 KiB, like the logs. The value is an empty string when the process has written nothing to standard error.

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/process/{processName}/errors' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Mouse operations

Every mouse, keyboard, screenshot, display, accessibility, and recording-start call requires the desktop to be ready; otherwise it returns 409 with a desktop not ready error. Coordinates are screen pixels in the range -32768 to 32767; values outside it return 400.

Click

Click the mouse at the specified coordinates with the left, right, or middle button, as a single or a double click.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

sandbox.computer_use.mouse.click(x=100, y=200)

Parameter	Type	Description
button	ComputerUseMouseButton	Sets the mouse button. The default is "left".
double	bool	Makes the click a double click. The default is False.

• The call returns the pointer position after the click.

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

await sandbox.computerUse.mouse.click({ x: 100, y: 200 });

Parameter	Type	Description
button	ComputerUseMouseButton	Sets the mouse button. The default is 'left'.
double	boolean	Makes the click a double click. The default is false.

• The call returns the pointer position after the click.

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/mouse/click' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --data '{
  "x": 100,
  "y": 200
}'

Field	Type	Description
button	string	Sets the mouse button. The default is left.
double	boolean	Makes the click a double click. The default is false.

The response has the following fields:

Field	Description
x	Holds the x coordinate of the pointer after the click.
y	Holds the y coordinate of the pointer after the click.

Move

Move the mouse cursor to the given coordinates. The response holds the final pointer position.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

result = sandbox.computer_use.mouse.move(x=100, y=200)
print(f"Mouse moved to: {result.x}, {result.y}")

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const result = await sandbox.computerUse.mouse.move({ x: 100, y: 200 });
console.log(`Mouse moved to: ${result.x}, ${result.y}`);

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/mouse/move' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --data '{
  "x": 100,
  "y": 200
}'

Drag

Drag the mouse from start to end coordinates while holding the selected button. The pointer moves in 10 interpolated steps so applications that ignore a teleporting drag still register it.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

result = sandbox.computer_use.mouse.drag(
    start_x=50, start_y=50, end_x=150, end_y=150, button="left"
)
print(f"Drag ended at {result.x}, {result.y}")

Parameter	Type	Description
button	ComputerUseMouseButton	Sets the mouse button to hold. The default is "left".

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const result = await sandbox.computerUse.mouse.drag({
  startX: 50,
  startY: 50,
  endX: 150,
  endY: 150,
  button: 'left',
});
console.log(`Drag ended at ${result.x}, ${result.y}`);

Parameter	Type	Description
button	ComputerUseMouseButton	Sets the mouse button to hold. This parameter is required.

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/mouse/drag' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --data '{
  "button": "left",
  "endX": 150,
  "endY": 150,
  "startX": 50,
  "startY": 50
}'

Field	Type	Description
button	string	Sets the mouse button to hold. The default is left.

Scroll

Scroll the mouse wheel up or down at the specified coordinates by a number of wheel ticks. One tick is one notch of a physical mouse wheel.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

result = sandbox.computer_use.mouse.scroll(x=100, y=200, direction="down")
print(result.success)

• The default scroll amount is 1.
• The scroll amount must not be negative.
• The maximum scroll amount is 100.

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const result = await sandbox.computerUse.mouse.scroll({ x: 100, y: 200, direction: 'down' });
console.log(result.success);

• The default scroll amount is 1.
• The scroll amount must not be negative.
• The maximum scroll amount is 100.

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/mouse/scroll' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --data '{
  "direction": "down",
  "x": 100,
  "y": 200
}'

• The default scroll amount is 1.
• The scroll amount must not be negative.
• The maximum scroll amount is 100.

Get position

Get the current mouse cursor position.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

position = sandbox.computer_use.mouse.position()
print(f"Mouse is at: {position.x}, {position.y}")

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const position = await sandbox.computerUse.mouse.position();
console.log(`Mouse is at: ${position.x}, ${position.y}`);

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/mouse/position' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Keyboard operations

Send keyboard input to the desktop. Type a text string, press one key with optional modifiers, or press a hotkey combination. Key names resolve against the keyboard map of the virtual display, and, like mouse calls, keyboard calls require the desktop to be ready.

Type

Type arbitrary text, including uppercase letters, symbols, and non-ASCII characters. Newlines are sent as Enter and tabs as Tab. Characters with no key in the X11 keymap are typed through a temporary key mapping.

Set a delay between characters to slow typing down for applications that drop fast input.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

sandbox.computer_use.keyboard.type(text="Hello, World!")

• The call sends \n and \r as Enter.
• The delay between characters is in milliseconds.
• The default delay is 0.
• The maximum delay is 1000 per character.
• The text is limited to 4096 characters.
• The number of characters multiplied by the delay must not exceed 25 s.

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

await sandbox.computerUse.keyboard.type({ text: 'Hello, World!' });

• The call sends \n and \r as Enter.
• The delay between characters is in milliseconds.
• The default delay is 0.
• The maximum delay is 1000 per character.
• The text is limited to 4096 characters.
• The number of characters multiplied by the delay must not exceed 25 s.

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/keyboard/type' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --data '{
  "text": "Hello, World!"
}'

• The call sends \n and \r as Enter.
• The delay between characters is in milliseconds.
• The default delay is 0.
• The maximum delay is 1000 per character.
• The text is limited to 4096 characters.
• The number of characters multiplied by the delay must not exceed 25 s.

Press

Press a key with optional modifiers. Modifiers are held in order, the key is pressed and released, and the modifiers are released in reverse, even if the press fails. See supported keys.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

sandbox.computer_use.keyboard.key(key="enter")

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

await sandbox.computerUse.keyboard.key({ key: 'enter' });

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/keyboard/key' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --data '{
  "key": "enter"
}'

Hotkey

Press a hotkey combination. The combination is a plus-separated chord string: the last entry is the key, the others are modifiers.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

sandbox.computer_use.keyboard.hotkey(keys="ctrl+c")

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

await sandbox.computerUse.keyboard.hotkey({ keys: 'ctrl+c' });

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/keyboard/hotkey' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --data '{
  "keys": "ctrl+c"
}'

Supported keys

Key names are case-insensitive. The following are supported for the key and modifiers parameters and for the parts of the keys parameter:

Category	Keys
Modifiers	ctrl (alias control), alt, shift, cmd (aliases super, meta, win)
Editing	enter (alias return), tab, escape (alias esc), backspace, delete (alias del), space
Navigation	home, end, pageup (alias page_up), pagedown (alias page_down), insert, up, down, left, right
Function keys	f1 through f12
Symbols	minus, plus, equal (alias equals)
Characters	Any single character, for example a, A, 5, /, -

The SDK modifiers type accepts ctrl, alt, shift, and cmd; the API also accepts the aliases.

An unknown key or modifier returns 400.

Accessibility operations

Use accessibility operations to inspect the AT-SPI tree and interact with UI elements by node ID. Start Computer Use before calling accessibility methods.

Each node carries an ID, role, name, description, bounds (x, y, width, height), states, actions, and children. Node IDs have the form <bus-name>:<object-path> and are valid only while the owning application keeps the object alive; an unknown or malformed ID returns 404 or 400.

The scope parameter selects the root of the search:

Scope	Root
focused	The application whose state includes active, else the first registered application. Default.
pid	The application owned by process pid. Returns 404 if no accessible application has that PID.
all	The AT-SPI registry root, covering every application.

Traversal is bounded: 2000 nodes and 2 s for the tree, 5000 nodes and 2 s for find. When a bound is reached, the response is marked as truncated.

App accessibility support

Accessibility operations read the semantic UI information that applications expose over AT-SPI. Apps or custom widgets that do not expose accessibility objects may return sparse nodes, generic roles, or no actionable nodes; mouse, keyboard, and screenshot operations remain available for those cases.

Get tree

Read an accessibility tree for the focused app, a specific process, or all apps. The maximum depth parameter limits the depth below the root: 0 returns only the root, and omitting it or passing -1 returns the whole subtree within the traversal budget.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

tree = sandbox.computer_use.a11y.tree(scope="focused")
print(tree.root.role, tree.root.name, tree.truncated)

Parameter	Type	Description
scope	ComputerUseA11yScope	Sets the root of the search: "focused", "pid", or "all". With scope="pid", also pass pid. This parameter is required.
max_depth	int	Limits the depth below the root. Omit the parameter to get the whole subtree.

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const tree = await sandbox.computerUse.a11y.tree({ scope: 'focused' });
console.log(tree.root.role, tree.root.name, tree.truncated);

Parameter	Type	Description
scope	ComputerUseA11yScope	Sets the root of the search: 'focused', 'pid', or 'all'. With scope: 'pid', also pass pid. This parameter is required.
maxDepth	number	Limits the depth below the root. Omit the parameter to get the whole subtree.

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/a11y/tree?scope=focused' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Field	Type	Description
scope	string	Sets the root of the search. An unknown scope returns 400. The default is focused.
pid	integer	Selects the application when scope=pid.
maxDepth	integer	Limits the depth below the root. The query parameter name is camelCase.

Find nodes

Search the accessibility tree by role, accessible name, state, and scope. All filters must match.

Node states: active, armed, busy, checked, collapsed, defunct, editable, enabled, expandable, expanded, focusable, focused, has-tooltip, horizontal, iconified, modal, multi-line, multiselectable, opaque, pressed, resizable, selectable, selected, sensitive, showing, single-line, stale, transient, vertical, visible, manages-descendants, indeterminate, required, truncated, and animated.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

buttons = sandbox.computer_use.a11y.find(scope="focused", role="button", name="Submit")
for node in buttons.matches:
    print(node.id, node.name, node.bounds.x, node.bounds.y)

Parameter	Type	Description
scope	ComputerUseA11yScope	Sets the root of the search. This parameter is required.
role	str	Matches the node role exactly, for example button or entry.
name	str	Matches the accessible name according to name_match.
name_match	ComputerUseA11yNameMatch	Sets how name is compared: exact, substring, or regex. The default is exact.
states	Sequence[str]	Requires the node to have every listed state, for example ["enabled", "focusable"].
limit	int	Caps the number of matches. With more matches, truncated is True. The default is 50.

• Filters other than scope are optional.
• An omitted filter matches every node.

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const buttons = await sandbox.computerUse.a11y.find({ scope: 'focused', role: 'button', name: 'Submit' });
for (const node of buttons.matches) {
  console.log(node.id, node.name, node.bounds.x, node.bounds.y);
}

Parameter	Type	Description
scope	ComputerUseA11yScope	Sets the root of the search. This parameter is required.
role	string	Matches the node role exactly, for example 'button' or 'entry'.
name	string	Matches the accessible name according to nameMatch.
nameMatch	ComputerUseA11yNameMatch	Sets how name is compared: 'exact', 'substring', or 'regex'. The default is 'exact'.
states	string[]	Requires the node to have every listed state, for example ['enabled', 'focusable'].
limit	number	Caps the number of matches. With more matches, truncated is true. The default is 50.

• Filters other than scope are optional.
• An omitted filter matches every node.

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/a11y/find' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --data '{
  "scope": "focused",
  "role": "button",
  "name": "Submit"
}'

Field	Type	Description
scope	string	Sets the root of the search. The default is focused.
role	string	Matches the node role exactly, for example button or entry.
name	string	Matches the accessible name according to nameMatch.
nameMatch	string	Sets how name is compared: exact, substring, or regex. The default is exact.
states	string[]	Requires the node to have every listed state, for example ["enabled", "focusable"].
limit	integer	Caps the number of matches. A value of 0 or less uses the default. With more matches, truncated is true. The default is 50.

Focus node

Focus a node from get tree or find nodes.

Python

sandbox.computer_use.a11y.focus(id=node.id)

TypeScript

await sandbox.computerUse.a11y.focus({ id: node.id });

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/a11y/node/focus' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --data '{"id":"node-id"}'

Invoke node

Run a node action, such as pressing a button. Without an action name, the node’s click action is invoked, falling back to its first action. An action name must appear in the node’s actions list.

Python

sandbox.computer_use.a11y.invoke(id=node.id)

TypeScript

await sandbox.computerUse.a11y.invoke({ id: node.id });

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/a11y/node/invoke' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --data '{"id":"node-id"}'

Set node value

Write text content to nodes that implement editable text, such as entries and text areas. The value replaces the current content.

Python

sandbox.computer_use.a11y.set_value(id=node.id, value="hello")

TypeScript

await sandbox.computerUse.a11y.setValue({ id: node.id, value: 'hello' });

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/a11y/node/value' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --data '{"id":"node-id","value":"hello"}'

Screenshot operations

Screenshots return the image as a base64 string, its encoded size in bytes, and, when the cursor is requested, the pointer coordinates. The cursor is never drawn. Full-screen and region screenshots are PNG; compressed variants accept format, quality, and scale parameters.

Take full screen

Take a screenshot of the entire screen.

Python

import base64

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

screenshot = sandbox.computer_use.screen.capture()
print(f"Screenshot size: {screenshot.size_bytes} bytes")
with open("screen.png", "wb") as f:
    f.write(base64.b64decode(screenshot.screenshot))

TypeScript

import { writeFile } from 'node:fs/promises';
import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const screenshot = await sandbox.computerUse.screen.capture();
console.log(`Screenshot size: ${screenshot.sizeBytes} bytes`);
await writeFile('screen.png', Buffer.from(screenshot.screenshot, 'base64'));

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/screenshot' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Take region

Take a screenshot of a specific region. The width and height must be positive, and the region must lie within the screen.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

screenshot = sandbox.computer_use.screen.region(x=100, y=100, width=300, height=200)
print(f"Captured region: {screenshot.size_bytes} bytes")

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const screenshot = await sandbox.computerUse.screen.region({ x: 100, y: 100, width: 300, height: 200 });
console.log(`Captured region: ${screenshot.sizeBytes} bytes`);

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/screenshot/region?x=100&y=100&width=300&height=200' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Take compressed

Take a compressed screenshot of the entire screen.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

screenshot = sandbox.computer_use.screen.capture_compressed(format="jpeg", quality=80)
print(f"Compressed size: {screenshot.size_bytes} bytes")

Parameter	Type	Description
format	str	Sets the image format, png or jpeg. The default is png.
quality	int	Sets the image quality, from 1 to 100. The quality applies to jpeg. The default is 85.
scale	float	Resizes the captured image by a factor from 0.1 to 1.0. The default is 1.0.
show_cursor	bool	Returns the pointer coordinates with the screenshot. The cursor is never drawn. The default is False.

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const screenshot = await sandbox.computerUse.screen.captureCompressed({ format: 'jpeg', quality: 80 });
console.log(`Compressed size: ${screenshot.sizeBytes} bytes`);

Parameter	Type	Description
format	string	Sets the image format, 'png' or 'jpeg'. The default is 'png'.
quality	number	Sets the image quality, from 1 to 100. The quality applies to 'jpeg'. The default is 85.
scale	number	Resizes the captured image by a factor from 0.1 to 1.0. The default is 1.0.
showCursor	boolean	Returns the pointer coordinates with the screenshot. The cursor is never drawn. The default is false.

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/screenshot/compressed?format=jpeg&quality=80' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Field	Type	Description
format	string	Sets the image format, png or jpeg. The default is png.
quality	integer	Sets the image quality, from 1 to 100. The quality applies to jpeg. The default is 85.
scale	number	Resizes the captured image by a factor from 0.1 to 1.0. The default is 1.0.
show_cursor	boolean	Returns the pointer coordinates with the screenshot. The cursor is never drawn. The default is false.

Take compressed region

Take a compressed screenshot of a specific region. Accepts the region parameters of take region and the compression parameters of take compressed.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

screenshot = sandbox.computer_use.screen.region_compressed(
    x=0, y=0, width=800, height=600, format="jpeg", quality=80
)
print(f"Compressed size: {screenshot.size_bytes} bytes")

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const screenshot = await sandbox.computerUse.screen.regionCompressed({
  x: 0,
  y: 0,
  width: 800,
  height: 600,
  format: 'jpeg',
  quality: 80,
});
console.log(`Compressed size: ${screenshot.sizeBytes} bytes`);

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/screenshot/region/compressed?x=0&y=0&width=800&height=600&format=jpeg&quality=80' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Screen Recording

Computer Use records desktop sessions for debugging, documentation, or automation workflows. Recordings are MP4 files captured at 10 frames per second and stored inside the sandbox; the file path field in every recording response reports the location.

Each recording has the following fields:

Field	Description
id	Recording ID, prefixed rec_. Used for stop, get, download, and delete.
file_name	<label>-<id>.mp4. The label is reduced to letters, digits, and -, and defaults to recording.
file_path	Path of the file inside the sandbox.
start_time	RFC 3339 timestamp.
end_time	RFC 3339 timestamp, set once the recording is finalized.
status	See the status table below.
duration_seconds	Elapsed seconds while active, or the final duration.
size_bytes	Current file size while active, or the final size.

Status	Meaning
recording	Capturing.
stopping	Stop was requested and the encoder is finalizing the file.
duration_limit_reached	The 15 minute cap was hit and the recording is being finalized.
size_limit_reached	The 512 MiB cap was hit and the recording is being finalized.
completed	Terminal. The file is ready to download.
failed	Terminal. The encoder exited abnormally or produced no usable file.

Limits:

• At most 2 recordings can be active at once; a third start returns 400.
• A recording stops on its own after 15 minutes or 512 MiB, whichever comes first.
• The 20 most recent finished recordings are kept; older ones are deleted as new ones finish.
• Labels are limited to 120 characters.

Start Recording

Start a screen recording with an optional label. The desktop must be ready.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

recording = sandbox.computer_use.recording.start()
print(f"Recording started: {recording.id}")

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const recording = await sandbox.computerUse.recording.start();
console.log(`Recording started: ${recording.id}`);

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/recordings/start' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --data '{}'

Stop Recording

Stop an active recording session by providing the recording ID. The call returns once the file is finalized, with the terminal recording.

Python

# Stop the recording
stopped_recording = sandbox.computer_use.recording.stop(id=recording.id)
print(f"Recording stopped: {stopped_recording.duration_seconds} seconds")
print(f"Saved to: {stopped_recording.file_path}")

TypeScript

// Stop the recording
const stoppedRecording = await sandbox.computerUse.recording.stop({ id: recording.id });
console.log(`Recording stopped: ${stoppedRecording.durationSeconds} seconds`);
console.log(`Saved to: ${stoppedRecording.filePath}`);

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/recordings/stop' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --data '{
  "id": "recording-id"
}'

List Recordings

Get a list of all recordings in the sandbox, active and finished:

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

recordings_list = sandbox.computer_use.recording.list()
print(f"Total recordings: {len(recordings_list.recordings)}")
for rec in recordings_list.recordings:
    print(f"- {rec.file_name}: {rec.status} {rec.duration_seconds}s ({rec.size_bytes} bytes)")

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const recordingsList = await sandbox.computerUse.recording.list();
console.log(`Total recordings: ${recordingsList.recordings.length}`);
for (const rec of recordingsList.recordings) {
  console.log(`- ${rec.fileName}: ${rec.status} ${rec.durationSeconds}s (${rec.sizeBytes} bytes)`);
}

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/recordings' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Get Recording

Get details about a specific recording.

Python

recording_detail = sandbox.computer_use.recording.get("recording-id")
print(f"Recording: {recording_detail.file_name}")
print(f"Status: {recording_detail.status}")
print(f"Duration: {recording_detail.duration_seconds}s")

TypeScript

const recordingDetail = await sandbox.computerUse.recording.get('recording-id');
console.log(`Recording: ${recordingDetail.fileName}`);
console.log(`Status: ${recordingDetail.status}`);
console.log(`Duration: ${recordingDetail.durationSeconds}s`);

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/recordings/{id}' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Delete Recording

Delete a recording by ID. Deleting an active recording stops it first and then removes the file.

Python

sandbox.computer_use.recording.delete("recording-id")
print("Recording deleted successfully")

TypeScript

await sandbox.computerUse.recording.delete('recording-id');
console.log('Recording deleted successfully');

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/recordings/{id}' \
  --request DELETE \
  --header 'Authorization: Bearer YOUR_API_KEY'

Download Recording

Download a recording file from the sandbox. The SDKs return the file content as bytes; write it to disk yourself. The daemon streams the file, so downloads are not limited by a buffer size.

Python

from pathlib import Path

data = sandbox.computer_use.recording.download(recording.id)
Path("local_recording.mp4").write_bytes(data)
print(f"Downloaded {len(data)} bytes")

TypeScript

import { writeFile } from 'node:fs/promises';

const data = await sandbox.computerUse.recording.download(recording.id);
await writeFile('local_recording.mp4', data);
console.log(`Downloaded ${data.byteLength} bytes`);

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/recordings/{id}/download' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --output local_recording.mp4

The response is an MP4 video with the following headers:

Header	Description
Content-Disposition	Marks the response as an attachment with the file name.
x-daytona-recording-status	Reports the recording status.
x-daytona-recording-incomplete	Reports true when the recording is not completed.

Display operations

The virtual display is created when Computer Use starts and runs at 1280x720 with 24-bit color. The resolution is fixed, with no per-sandbox setting. Verify the live geometry with get info.

Get info

Get information about the displays. The response lists every X screen with its ID, position, size, and active flag. The virtual desktop has one screen.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

info = sandbox.computer_use.display.info()
print(f"Total displays: {len(info.displays)}")
for display in info.displays:
    print(f"Display {display.id}: {display.width}x{display.height} at {display.x},{display.y}")

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const info = await sandbox.computerUse.display.info();
console.log(`Total displays: ${info.displays.length}`);
for (const display of info.displays) {
  console.log(`Display ${display.id}: ${display.width}x${display.height} at ${display.x},${display.y}`);
}

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/display/info' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Get windows

Get the list of open windows. Each window carries its ID, title, position, size, and active flag. The list is read from the client list property that the window manager maintains, which is why start refuses to run without a window manager.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.create()
sandbox.computer_use.start()

windows = sandbox.computer_use.display.windows()
print(f"Found {len(windows.windows)} open windows:")
for window in windows.windows:
    print(f"- {window.title} (ID: {window.id}, active: {window.is_active})")

TypeScript

import { Daytona } from '@daytona/sdk';

const daytona = new Daytona();
const sandbox = await daytona.v2.create();
await sandbox.computerUse.start();

const windows = await sandbox.computerUse.display.windows();
console.log(`Found ${windows.windows.length} open windows:`);
for (const window of windows.windows) {
  console.log(`- ${window.title} (ID: ${window.id}, active: ${window.isActive})`);
}

API

curl 'https://toolbox.daytona.com/toolbox/{sandboxId}/computeruse/display/windows' \
  --header 'Authorization: Bearer YOUR_API_KEY'