Preemption

Daytona v2

Documentation for Daytona v2, currently in alpha. For the latest release, see Daytona v0.

Daytona sandboxes run on reserved capacity by default: once placed, a sandbox keeps its resources until you delete it. A spot sandbox is preemptible: it is billed at a discount, and Daytona can reclaim it at any time to free capacity for full-price sandboxes. The tier is set when the sandbox is created and cannot be changed afterwards.

A reclaimed spot sandbox is deleted, with its filesystem. To continue the work, create a new sandbox and keep results in storage outside the sandbox, such as your own object storage.

Use cases

Spot sandboxes are used for work that tolerates an interruption in exchange for a lower price.

• Batch jobs: data processing, builds, and test runs start again from the beginning when a sandbox is reclaimed. Spot multiplies the CPU, memory, and disk price by 0.7.
• Evaluation and reinforcement learning (RL): many short test cases or episodes run on spot sandboxes. A reclaimed test case or episode runs again in a new sandbox.
• CI runs: build and test jobs run on spot sandboxes. A reclaim costs one rerun of the job.
• Stateless workers: workers that keep no state in the sandbox lose nothing on a reclaim. A new spot sandbox replaces the reclaimed one.
• Results outside the sandbox: a spot sandbox writes its output to storage outside the sandbox, so a reclaim loses only the work in progress.
• Development and experiments: sandboxes whose loss costs only a restart run at the spot price.

Spot sandboxes

A spot sandbox differs from an on-demand sandbox only in price and reclaim. In every other respect it is an ordinary sandbox: it supports the same images, resources, and lifecycle operations as an on-demand sandbox of its type. Spot multiplies the CPU, memory, and disk price by 0.7; see pricing dimensions.

Linux sandboxes can be spot.

Python

from daytona import Daytona

daytona = Daytona()

sandbox = daytona.v2.create(spot=True)

Parameter	Type	Description
spot	bool	Places the sandbox on discounted, reclaimable capacity. The default is False. The value is fixed at create.

• A sandbox created from a snapshot with create_from_snapshot() is never spot.

TypeScript

import { Daytona } from '@daytona/sdk'

const daytona = new Daytona()

const sandbox = await daytona.v2.create({ spot: true })

Parameter	Type	Description
spot	boolean	Places the sandbox on discounted, reclaimable capacity. The default is false. The value is fixed at create.

• A sandbox created from a snapshot with createFromSnapshot() is never spot.

CLI

daytona sandbox create --spot

Flag	Type	Description
--spot	bool	Places the sandbox on discounted, reclaimable capacity. The default is false. The value is fixed at create. sandbox create-from-snapshot has no --spot flag.

• A sandbox created with sandbox create-from-snapshot is never spot.

API

curl 'https://api.daytona.com/sandboxes' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --data '{"spot": true}'

Field	Type	Description
spot	boolean	Places the sandbox on discounted, reclaimable capacity. The default is false. The value is fixed at create. POST /sandboxes/from-snapshot does not accept spot.

• A sandbox created with POST /sandboxes/from-snapshot is never spot.

Reclaim policy

Daytona tracks the CPU, memory, and disk reserved on each host. When the reservation on a host crosses a threshold on any of the three, Daytona reclaims running spot sandboxes on that host until the projected reservation drops below it. Newer spot sandboxes are reclaimed before older ones, interrupting the least accumulated work first. On-demand sandboxes are never reclaimed.

A reclaim is immediate. Daytona does not notify the sandbox or your application in advance, and there is no grace period. Daytona does not replace a reclaimed sandbox on its own when capacity frees up: creating a new one is up to you.

A reclaimed sandbox is billed like any deleted sandbox.

Reclaim outcomes

Daytona records the reclaim time, then tears the sandbox down like a user request: the sandbox moves to deleting, then deleted, and only data stored outside the sandbox remains.

For the full state model, see lifecycle.

Reclaim detection

Reclaim detection tells a reclaim apart from a delete you requested. Every reclaimed sandbox carries the time of the reclaim. A sandbox that was never reclaimed carries no reclaim time.

Python

from daytona import Daytona

daytona = Daytona()
sandbox = daytona.v2.get("{sandboxId}")

# None unless the platform reclaimed this spot sandbox
if sandbox.spot_evicted_at:
    print(f"reclaimed at {sandbox.spot_evicted_at}, state: {sandbox.state}")

# Find spot sandboxes
page = daytona.v2.search(spot=True)
for sandbox in page.items:
    print(sandbox.id, sandbox.state, sandbox.spot_evicted_at)

Parameter	Type	Description
spot	bool	Filters by capacity type. True returns only spot sandboxes, and False returns only on-demand sandboxes. The default is None, which returns both.

• spot_evicted_at is an ISO 8601 timestamp string, or None.
• A reclaimed sandbox leaves search results after about 2 to 3 seconds.

TypeScript

import { Daytona } from '@daytona/sdk'

const daytona = new Daytona()
const sandbox = await daytona.v2.get('{sandboxId}')

// undefined unless the platform reclaimed this spot sandbox
if (sandbox.spotEvictedAt) {
  console.log(`reclaimed at ${sandbox.spotEvictedAt}, state: ${sandbox.state}`)
}

// Find spot sandboxes
const page = await daytona.v2.search({ spot: true })
for (const item of page.items) {
  console.log(item.id, item.state, item.spotEvictedAt)
}

Parameter	Type	Description
spot	boolean	Filters by capacity type. true returns only spot sandboxes, and false returns only on-demand sandboxes. By default, the search returns both.

• spotEvictedAt is an ISO 8601 timestamp string, or undefined.
• A reclaimed sandbox leaves search results after about 2 to 3 seconds.

CLI

# spot_evicted_at is present only if the platform reclaimed the sandbox
daytona sandbox get {sandboxId} --output json

# Find spot sandboxes
daytona sandbox search --spot

Flag	Type	Description
--spot	bool	Filters by capacity type. --spot returns only spot sandboxes, and --spot=false returns only on-demand sandboxes. By default, the search returns both.

• The default table output of daytona sandbox get omits spot_evicted_at.
• Use --output json or --output yaml to see spot_evicted_at.

API

# spot_evicted_at is an RFC 3339 timestamp, or null if the sandbox was never reclaimed
curl 'https://api.daytona.com/sandboxes/{sandboxId}' \
  --header 'Authorization: Bearer YOUR_API_KEY'

# Find spot sandboxes
curl 'https://api.daytona.com/sandboxes/search?spot=true' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Field	Type	Description
spot	boolean	Filters by capacity type. true returns only spot sandboxes, and false returns only on-demand sandboxes. By default, the search returns both.

Reclaim recovery

A reclaimed sandbox is gone; continue in a new sandbox from data stored outside the sandbox.

Python

from daytona import Daytona

daytona = Daytona()
previous = daytona.v2.get("{sandboxId}")

# A reclaimed sandbox is deleted; only data outside it remains
if previous.spot_evicted_at:
    sandbox = daytona.v2.create(spot=True)

    # Continue from the last checkpoint the previous sandbox wrote to your own storage
    sandbox.process.run("python3 train.py --resume-from s3://my-bucket/checkpoint")

TypeScript

import { Daytona } from '@daytona/sdk'

const daytona = new Daytona()
const previous = await daytona.v2.get('{sandboxId}')

// A reclaimed sandbox is deleted; only data outside it remains
if (previous.spotEvictedAt) {
  const sandbox = await daytona.v2.create({ spot: true })

  // Continue from the last checkpoint the previous sandbox wrote to your own storage
  await sandbox.process.run('python3 train.py --resume-from s3://my-bucket/checkpoint')
}

Opting in to preemption

Spot suits work that tolerates interruption and can be restarted or continued: batch jobs, evaluation and reinforcement learning rollouts, CI runs, and stateless workers. Keep spot off sandboxes that serve live traffic or hold state that exists nowhere else.

• Checkpoint to storage outside the sandbox from spot sandboxes, so a replacement sandbox can continue where the reclaimed one stopped.
• Check after interruptions: when a sandbox stops responding or changes state unexpectedly, read its reclaim time to tell a reclaim apart from a failure.