Accessibility

Accessibility operations for computer use functionality.

Constructors

new Accessibility()

new Accessibility(apiClient: ComputerUseApi): Accessibility

Parameters:

• apiClient ComputerUseApi

Returns:

• Accessibility

Methods

findNodes()

findNodes(options?: FindAccessibilityNodesRequest): Promise<AccessibilityNodesResponse>

Finds AT-SPI accessibility nodes matching the provided filters.

Parameters:

• options? FindAccessibilityNodesRequest = - Search scope, node filters, and result limit

Returns:

• Promise<AccessibilityNodesResponse> - Matching accessibility nodes

Example:

const buttons = await sandbox.computerUse.accessibility.findNodes({
  scope: 'all',
  role: 'button',
  name: 'Submit',
  nameMatch: 'substring',
});
console.log(buttons.matches?.length);

---

focusNode()

focusNode(id: string): Promise<void>

Focuses an AT-SPI accessibility node.

Parameters:

• id string - Accessibility node ID returned by getTree or findNodes

Returns:

• Promise<void>

Example:

const node = (await sandbox.computerUse.accessibility.findNodes({ scope: 'all', limit: 1 })).matches?.[0];
if (node?.id) {
  await sandbox.computerUse.accessibility.focusNode(node.id);
}

---

getTree()

getTree(options?: AccessibilityTreeOptions): Promise<AccessibilityTreeResponse>

Fetches the AT-SPI accessibility tree.

Parameters:

• options? AccessibilityTreeOptions = - Scope and depth options

Returns:

• Promise<AccessibilityTreeResponse> - Accessibility tree response

Example:

const tree = await sandbox.computerUse.accessibility.getTree({ scope: 'all', maxDepth: 3 });
console.log(tree.root?.name);

---

invokeNode()

invokeNode(id: string, action?: string): Promise<void>

Invokes an AT-SPI accessibility node action.

Parameters:

• id string - Accessibility node ID returned by getTree or findNodes
• action? string - Action name to invoke. If omitted, the API invokes the primary action

Returns:

• Promise<void>

Example:

const button = (await sandbox.computerUse.accessibility.findNodes({ scope: 'all', role: 'button', limit: 1 }))
  .matches?.[0];
if (button?.id) {
  await sandbox.computerUse.accessibility.invokeNode(button.id, 'click');
}

---

setNodeValue()

setNodeValue(id: string, value: string): Promise<void>

Sets an AT-SPI accessibility node value.

Parameters:

• id string - Accessibility node ID returned by getTree or findNodes
• value string - Value to write to the node

Returns:

• Promise<void>

Example:

const field = (await sandbox.computerUse.accessibility.findNodes({ scope: 'all', role: 'entry', limit: 1 }))
  .matches?.[0];
if (field?.id) {
  await sandbox.computerUse.accessibility.setNodeValue(field.id, 'hello');
}

---

ComputerUse

Computer Use functionality for interacting with the desktop environment.

Properties:

• accessibility Accessibility - Accessibility operations interface
• display Display - Display operations interface
• keyboard Keyboard - Keyboard operations interface
• mouse Mouse - Mouse operations interface
• recording RecordingService - Screen recording operations interface
• screenshot Screenshot - Screenshot operations interface

Provides access to mouse, keyboard, screenshot, display, recording, and accessibility operations for automating desktop interactions within a sandbox.

Constructors

new ComputerUse()

new ComputerUse(apiClient: ComputerUseApi): ComputerUse

Parameters:

• apiClient ComputerUseApi

Returns:

• ComputerUse

Methods

getProcessErrors()

getProcessErrors(processName: string): Promise<ProcessErrorsResponse>

Gets error logs for a specific VNC process

Parameters:

• processName string - Name of the process to get error logs for

Returns:

• Promise<ProcessErrorsResponse> - Process error logs

Example:

const errorsResp = await sandbox.computerUse.getProcessErrors('x11vnc');
console.log('X11VNC errors:', errorsResp.errors);

---

getProcessLogs()

getProcessLogs(processName: string): Promise<ProcessLogsResponse>

Gets logs for a specific VNC process

Parameters:

• processName string - Name of the process to get logs for

Returns:

• Promise<ProcessLogsResponse> - Process logs

Example:

const logsResp = await sandbox.computerUse.getProcessLogs('novnc');
console.log('NoVNC logs:', logsResp.logs);

---

getProcessStatus()

getProcessStatus(processName: string): Promise<ProcessStatusResponse>

Gets the status of a specific VNC process

Parameters:

• processName string - Name of the process to check

Returns:

• Promise<ProcessStatusResponse> - Status information about the specific process

Example:

const xvfbStatus = await sandbox.computerUse.getProcessStatus('xvfb');
const noVncStatus = await sandbox.computerUse.getProcessStatus('novnc');

---

getStatus()

getStatus(): Promise<ComputerUseStatusResponse>

Gets the status of all computer use processes

Returns:

• Promise<ComputerUseStatusResponse> - Status information about all VNC desktop processes

Example:

const status = await sandbox.computerUse.getStatus();
console.log('Computer use status:', status.status);

---

restartProcess()

restartProcess(processName: string): Promise<ProcessRestartResponse>

Restarts a specific VNC process

Parameters:

• processName string - Name of the process to restart

Returns:

• Promise<ProcessRestartResponse> - Process restart response

Example:

const result = await sandbox.computerUse.restartProcess('xfce4');
console.log('XFCE4 process restarted:', result.message);

---

start()

start(): Promise<ComputerUseStartResponse>

Starts all computer use processes (Xvfb, xfce4, x11vnc, novnc)

Returns:

• Promise<ComputerUseStartResponse> - Computer use start response

Example:

const result = await sandbox.computerUse.start();
console.log('Computer use processes started:', result.message);

---

stop()

stop(): Promise<ComputerUseStopResponse>

Stops all computer use processes

Returns:

• Promise<ComputerUseStopResponse> - Computer use stop response

Example:

const result = await sandbox.computerUse.stop();
console.log('Computer use processes stopped:', result.message);

Display

Display operations for computer use functionality

Constructors

new Display()

new Display(apiClient: ComputerUseApi): Display

Parameters:

• apiClient ComputerUseApi

Returns:

• Display

Methods

getInfo()

getInfo(): Promise<DisplayInfoResponse>

Gets information about the displays

Returns:

• Promise<DisplayInfoResponse> - Display information including primary display and all available displays

Example:

const info = await sandbox.computerUse.display.getInfo();
console.log(`Primary display: ${info.primary_display.width}x${info.primary_display.height}`);
console.log(`Total displays: ${info.total_displays}`);
info.displays.forEach((display, index) => {
  console.log(`Display ${index}: ${display.width}x${display.height} at ${display.x},${display.y}`);
});

---

getWindows()

getWindows(): Promise<WindowsResponse>

Gets the list of open windows

Returns:

• Promise<WindowsResponse> - List of open windows with their IDs and titles

Example:

const windows = await sandbox.computerUse.display.getWindows();
console.log(`Found ${windows.count} open windows:`);
windows.windows.forEach(window => {
  console.log(`- ${window.title} (ID: ${window.id})`);
});

---

Keyboard

Keyboard operations for computer use functionality

Constructors

new Keyboard()

new Keyboard(apiClient: ComputerUseApi): Keyboard

Parameters:

• apiClient ComputerUseApi

Returns:

• Keyboard

Methods

hotkey()

hotkey(keys: string): Promise<void>

Presses a hotkey combination

Parameters:

• keys string - A single atomic hotkey chord (e.g., ‘ctrl+c’, ‘alt+tab’, ‘cmd+shift+t’, ‘ctrl + c’, ‘shift’). Uses the same normalized key contract as press().

Returns:

• Promise<void>

Throws:

If the hotkey operation fails

Example:

// Copy
try {
  await sandbox.computerUse.keyboard.hotkey('ctrl+c');
  console.log('Operation success');
} catch (e) {
  console.log('Operation failed:', e);
}

// Paste
try {
  await sandbox.computerUse.keyboard.hotkey('ctrl+v');
  console.log('Operation success');
} catch (e) {
  console.log('Operation failed:', e);
}

// Alt+Tab
try {
  await sandbox.computerUse.keyboard.hotkey('alt+tab');
  console.log('Operation success');
} catch (e) {
  console.log('Operation failed:', e);
}

---

press()

press(key: string, modifiers?: string[]): Promise<void>

Presses a key with optional modifiers

Parameters:

• key string - The key to press. Canonical names include ‘enter’, ‘escape’, ‘tab’, letters, digits, unshifted punctuation, function keys, and grammar-safe numpad names such as ‘num_plus’. Named keys are case-insensitive, and common aliases such as ‘Return’ and ‘Escape’ are normalized.
• modifiers? string[] = [] - Canonical modifier names are ‘ctrl’, ‘alt’, ‘shift’, and ‘cmd’. Common aliases such as ‘control’, ‘option’, ‘meta’, and ‘win’ are normalized.

Returns:

• Promise<void>

Throws:

If the press operation fails

Example:

// Press Enter
try {
  await sandbox.computerUse.keyboard.press('enter');
  console.log('Operation success');
} catch (e) {
  console.log('Operation failed:', e);
}

// Press Ctrl+C
try {
  await sandbox.computerUse.keyboard.press('c', ['ctrl']);
  console.log('Operation success');
} catch (e) {
  console.log('Operation failed:', e);
}

// Press Ctrl+Shift+T
try {
  await sandbox.computerUse.keyboard.press('t', ['ctrl', 'shift']);
  console.log('Operation success');
} catch (e) {
  console.log('Operation failed:', e);
}

---

type()

type(text: string, delay?: number): Promise<void>

Types the specified text

Parameters:

• text string - The text to type
• delay? number - Delay between characters in milliseconds

Returns:

• Promise<void>

Throws:

If the type operation fails

Example:

try {
  await sandbox.computerUse.keyboard.type('Hello, World!');
  console.log('Operation success');
} catch (e) {
  console.log('Operation failed:', e);
}

// With delay between characters
try {
  await sandbox.computerUse.keyboard.type('Slow typing', 100);
  console.log('Operation success');
} catch (e) {
  console.log('Operation failed:', e);
}

---

Mouse

Mouse operations for computer use functionality

Constructors

new Mouse()

new Mouse(apiClient: ComputerUseApi): Mouse

Parameters:

• apiClient ComputerUseApi

Returns:

• Mouse

Methods

click()

click(
   x: number,
   y: number,
   button?: string,
double?: boolean): Promise<MouseClickResponse>

Clicks the mouse at the specified coordinates

Parameters:

• x number - The x coordinate to click at
• y number - The y coordinate to click at
• button? string = ‘left’ - The mouse button to click (‘left’, ‘right’, ‘middle’)
• double? boolean = false - Whether to perform a double-click

Returns:

• Promise<MouseClickResponse> - Click operation result

Example:

// Single left click
const result = await sandbox.computerUse.mouse.click(100, 200);

// Double click
const doubleClick = await sandbox.computerUse.mouse.click(100, 200, 'left', true);

// Right click
const rightClick = await sandbox.computerUse.mouse.click(100, 200, 'right');

---

drag()

drag(
   startX: number,
   startY: number,
   endX: number,
   endY: number,
button?: string): Promise<MouseDragResponse>

Drags the mouse from start coordinates to end coordinates

Parameters:

• startX number - The starting x coordinate
• startY number - The starting y coordinate
• endX number - The ending x coordinate
• endY number - The ending y coordinate
• button? string = ‘left’ - The mouse button to use for dragging

Returns:

• Promise<MouseDragResponse> - Drag operation result

Example:

const result = await sandbox.computerUse.mouse.drag(50, 50, 150, 150);
console.log(`Drag ended at: ${result.x}, ${result.y}`);

---

getPosition()

getPosition(): Promise<MousePositionResponse>

Gets the current mouse cursor position

Returns:

• Promise<MousePositionResponse> - Current mouse position with x and y coordinates

Example:

const position = await sandbox.computerUse.mouse.getPosition();
console.log(`Mouse is at: ${position.x}, ${position.y}`);

---

move()

move(x: number, y: number): Promise<MousePositionResponse>

Moves the mouse cursor to the specified coordinates

Parameters:

• x number - The x coordinate to move to
• y number - The y coordinate to move to

Returns:

• Promise<MousePositionResponse> - Position after move

Example:

const result = await sandbox.computerUse.mouse.move(100, 200);
console.log(`Mouse moved to: ${result.x}, ${result.y}`);

---

scroll()

scroll(
   x: number,
   y: number,
   direction: "up" | "down",
amount?: number): Promise<boolean>

Scrolls the mouse wheel at the specified coordinates

Parameters:

• x number - The x coordinate to scroll at
• y number - The y coordinate to scroll at
• direction The direction to scroll - "up" | "down"
• amount? number = 1 - The amount to scroll

Returns:

• Promise<boolean> - Whether the scroll operation was successful

Example:

// Scroll up
const scrollUp = await sandbox.computerUse.mouse.scroll(100, 200, 'up', 3);

// Scroll down
const scrollDown = await sandbox.computerUse.mouse.scroll(100, 200, 'down', 5);

---

RecordingService

Recording operations for computer use functionality.

Constructors

new RecordingService()

new RecordingService(apiClient: ComputerUseApi): RecordingService

Parameters:

• apiClient ComputerUseApi

Returns:

• RecordingService

Methods

delete()

delete(id: string): Promise<void>

Deletes a recording by ID

Parameters:

• id string - The ID of the recording to delete

Returns:

• Promise<void>

Example:

await sandbox.computerUse.recording.delete(recordingId);
console.log('Recording deleted');

---

download()

download(id: string, localPath: string): Promise<void>

Downloads a recording file and saves it to a local path

The file is streamed directly to disk without loading the entire content into memory.

Parameters:

• id string - The ID of the recording to download
• localPath string - Path to save the recording file locally

Returns:

• Promise<void>

Example:

// Download recording to file
await sandbox.computerUse.recording.download(recordingId, 'local_recording.mp4');
console.log('Recording downloaded');

---

get()

get(id: string): Promise<Recording>

Gets details of a specific recording by ID

Parameters:

• id string - The ID of the recording to retrieve

Returns:

• Promise<Recording> - Recording details

Example:

const recording = await sandbox.computerUse.recording.get(recordingId);
console.log(`Recording: ${recording.fileName}`);
console.log(`Status: ${recording.status}`);
console.log(`Duration: ${recording.durationSeconds} seconds`);

---

list()

list(): Promise<ListRecordingsResponse>

Lists all recordings (active and completed)

Returns:

• Promise<ListRecordingsResponse> - List of all recordings

Example:

const recordings = await sandbox.computerUse.recording.list();
console.log(`Found ${recordings.recordings.length} recordings`);
recordings.recordings.forEach(rec => {
  console.log(`- ${rec.fileName}: ${rec.status}`);
});

---

start()

start(label?: string): Promise<Recording>

Starts a new screen recording session

Parameters:

• label? string - Optional custom label for the recording

Returns:

• Promise<Recording> - Started recording details

Example:

// Start a recording with a label
const recording = await sandbox.computerUse.recording.start('my-test-recording');
console.log(`Recording started: ${recording.id}`);
console.log(`File: ${recording.filePath}`);

---

stop()

stop(id: string): Promise<Recording>

Stops an active screen recording session

Parameters:

• id string - The ID of the recording to stop

Returns:

• Promise<Recording> - Stopped recording details

Example:

const result = await sandbox.computerUse.recording.stop(recording.id);
console.log(`Recording stopped: ${result.durationSeconds} seconds`);
console.log(`Saved to: ${result.filePath}`);

---

Screenshot

Screenshot operations for computer use functionality

Constructors

new Screenshot()

new Screenshot(apiClient: ComputerUseApi): Screenshot

Parameters:

• apiClient ComputerUseApi

Returns:

• Screenshot

Methods

takeCompressed()

takeCompressed(options?: ScreenshotOptions): Promise<ScreenshotResponse>

Takes a compressed screenshot of the entire screen

Parameters:

• options? ScreenshotOptions = - Compression and display options

Returns:

• Promise<ScreenshotResponse> - Compressed screenshot data

Example:

// Default compression
const screenshot = await sandbox.computerUse.screenshot.takeCompressed();

// High quality JPEG
const jpeg = await sandbox.computerUse.screenshot.takeCompressed({
  format: 'jpeg',
  quality: 95,
  showCursor: true
});

// Scaled down PNG
const scaled = await sandbox.computerUse.screenshot.takeCompressed({
  format: 'png',
  scale: 0.5
});

---

takeCompressedRegion()

takeCompressedRegion(region: ScreenshotRegion, options?: ScreenshotOptions): Promise<ScreenshotResponse>

Takes a compressed screenshot of a specific region

Parameters:

• region ScreenshotRegion - The region to capture
• options? ScreenshotOptions = - Compression and display options

Returns:

• Promise<ScreenshotResponse> - Compressed screenshot data

Example:

const region = { x: 0, y: 0, width: 800, height: 600 };
const screenshot = await sandbox.computerUse.screenshot.takeCompressedRegion(region, {
  format: 'webp',
  quality: 80,
  showCursor: true
});
console.log(`Compressed size: ${screenshot.size_bytes} bytes`);

---

takeFullScreen()

takeFullScreen(showCursor?: boolean): Promise<ScreenshotResponse>

Takes a screenshot of the entire screen

Parameters:

• showCursor? boolean = false - Whether to show the cursor in the screenshot

Returns:

• Promise<ScreenshotResponse> - Screenshot data with base64 encoded image

Example:

const screenshot = await sandbox.computerUse.screenshot.takeFullScreen();
console.log(`Screenshot size: ${screenshot.width}x${screenshot.height}`);

// With cursor visible
const withCursor = await sandbox.computerUse.screenshot.takeFullScreen(true);

---

takeRegion()

takeRegion(region: ScreenshotRegion, showCursor?: boolean): Promise<ScreenshotResponse>

Takes a screenshot of a specific region

Parameters:

• region ScreenshotRegion - The region to capture
• showCursor? boolean = false - Whether to show the cursor in the screenshot

Returns:

• Promise<ScreenshotResponse> - Screenshot data with base64 encoded image

Example:

const region = { x: 100, y: 100, width: 300, height: 200 };
const screenshot = await sandbox.computerUse.screenshot.takeRegion(region);
console.log(`Captured region: ${screenshot.region.width}x${screenshot.region.height}`);

---

AccessibilityTreeOptions

Options for fetching the AT-SPI accessibility tree.

Properties:

• maxDepth? number - Maximum depth to descend. Use 0 for the root only.
• pid? number - Process ID when scope is ‘pid’.
• scope? string - Tree scope to inspect: ‘focused’, ‘pid’, or ‘all’.

ScreenshotOptions

Interface for screenshot compression options

Properties:

• format? string
• quality? number
• scale? number
• showCursor? boolean

ScreenshotRegion

Interface for region coordinates used in screenshot operations

Properties:

• height number
• width number
• x number
• y number

AccessibilityFindOptions

type AccessibilityFindOptions = FindAccessibilityNodesRequest;

Options for finding AT-SPI accessibility nodes.