OpenTelemetry Collection

View as Markdown

OpenTelemetry (OTEL) tracing allows you to monitor and debug your Daytona SDK operations by collecting distributed traces. This is particularly useful for understanding performance bottlenecks, debugging issues, and gaining visibility into your sandbox operations.

Sandbox Telemetry Collection

Daytona can collect traces, logs, and metrics directly from your sandboxes. This provides complete observability across your entire Daytona environment, from SDK calls to sandbox runtime behavior.

Configure Sandbox Collection

To enable telemetry collection from sandboxes:

Navigate to the Daytona Dashboard
Go to Settings → Experimental
Configure the following fields:
- OTLP Endpoint: Your OpenTelemetry collector endpoint (e.g., https://otlp.nr-data.net)
- OTLP Headers: Authentication headers in key=value format (e.g., api-key=YOUR_API_KEY)

Once configured, all sandboxes will automatically export their telemetry data to your specified OTLP endpoint.

What Gets Collected from Sandboxes

When sandbox telemetry is enabled, the following data is collected:

Metrics:

daytona.sandbox.cpu.utilization - CPU usage percentage (0-100%)
daytona.sandbox.cpu.limit - CPU cores limit
daytona.sandbox.memory.utilization - Memory usage percentage (0-100%)
daytona.sandbox.memory.usage - Memory used in bytes
daytona.sandbox.memory.limit - Memory limit in bytes
daytona.sandbox.filesystem.utilization - Disk usage percentage (0-100%)
daytona.sandbox.filesystem.usage - Disk space used in bytes
daytona.sandbox.filesystem.available - Disk space available in bytes
daytona.sandbox.filesystem.total - Total disk space in bytes

Traces:

HTTP requests and responses
Custom spans from your application code

Logs:

Application logs (stdout/stderr)
System logs
Runtime errors and warnings

Viewing Telemetry in the Dashboard

Logs, traces, and metrics collected from sandboxes can be viewed directly in the Daytona Dashboard. Open the Sandbox Details sheet for any sandbox and use the Logs, Traces, and Metrics tabs to inspect the collected telemetry data.

SDK Tracing Configuration

When enabled, the Daytona SDK automatically instruments all SDK operations including:

Sandbox creation, starting, stopping, and deletion
File system operations
Code execution
Process management
HTTP requests to the Daytona API

Traces are exported using the OTLP (OpenTelemetry Protocol) format and can be sent to any OTLP-compatible backend such as New Relic, Jaeger, or Zipkin.

1. Enable OTEL in SDK

To enable OpenTelemetry tracing, pass the otelEnabled experimental flag when initializing the Daytona client:

Alternatively, you can set the DAYTONA_EXPERIMENTAL_OTEL_ENABLED environment variable to true instead of passing the configuration option:

export DAYTONA_EXPERIMENTAL_OTEL_ENABLED=true

from daytona import Daytona, DaytonaConfig

# Using async context manager (recommended)
async with Daytona(DaytonaConfig(
    _experimental={"otelEnabled": True}
)) as daytona:
    sandbox = await daytona.create()
    # All operations will be traced
# OpenTelemetry traces are flushed on close

Or without context manager:

daytona = Daytona(DaytonaConfig(
    _experimental={"otelEnabled": True}
))
try:
    sandbox = await daytona.create()
    # All operations will be traced
finally:
    await daytona.close()  # Flushes traces

import { Daytona } from '@daytonaio/sdk'

// Using async dispose (recommended)
await using daytona = new Daytona({
  _experimental: { otelEnabled: true }
})

const sandbox = await daytona.create()
// All operations will be traced
// Traces are automatically flushed on dispose

Or with explicit disposal:

const daytona = new Daytona({
  _experimental: { otelEnabled: true }
})

try {
  const sandbox = await daytona.create()
  // All operations will be traced
} finally {
  await daytona[Symbol.asyncDispose]()  // Flushes traces
}

import (
    "context"
    "log"

    "github.com/daytonaio/daytona/libs/sdk-go/pkg/daytona"
    "github.com/daytonaio/daytona/libs/sdk-go/pkg/types"
)

client, err := daytona.NewClientWithConfig(&types.DaytonaConfig{
    Experimental: &types.ExperimentalConfig{
        OtelEnabled: true,
    },
})
if err != nil {
    log.Fatal(err)
}
defer client.Close(context.Background()) // Flushes traces

sandbox, err := client.Create(context.Background(), nil)
// All operations will be traced

require 'daytona'

config = Daytona::Config.new(
  _experimental: { 'otel_enabled' => true }
)
daytona = Daytona::Daytona.new(config)

sandbox = daytona.create
# All operations will be traced

daytona.close # Flushes traces

Or with ensure block:

daytona = Daytona::Daytona.new(
  Daytona::Config.new(_experimental: { 'otel_enabled' => true })
)
begin
  sandbox = daytona.create
  # All operations will be traced
ensure
  daytona.close # Flushes traces
end

2. Configure OTLP Exporter

The SDK uses standard OpenTelemetry environment variables for configuration. Set these before running your application:

Required Environment Variables

# OTLP endpoint (without the /v1/traces path)
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.nr-data.net:4317

# Authentication headers (format: key1=value1,key2=value2)
OTEL_EXPORTER_OTLP_HEADERS="api-key=your-api-key-here"

Provider-Specific Examples

New Relic

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.nr-data.net:4317
OTEL_EXPORTER_OTLP_HEADERS="api-key=YOUR_NEW_RELIC_LICENSE_KEY"

Jaeger (Local)

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

Grafana Cloud

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-<region>.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <BASE64_ENCODED_CREDENTIALS>"

Setup: Go to Grafana Cloud Portal → Connections → Add new connection → Search for OpenTelemetry (OTLP) → Follow the wizard to create an access token. The endpoint and headers will be provided in the instrumentation instructions. See the Grafana dashboard example for detailed setup steps.

Complete Example

Here’s a complete example showing how to use OpenTelemetry tracing with the Daytona SDK:

import asyncio
import os
from daytona import Daytona, DaytonaConfig

# Set OTEL configuration
os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = "https://otlp.nr-data.net:4317"
os.environ["OTEL_EXPORTER_OTLP_HEADERS"] = "api-key=YOUR_API_KEY"

async def main():
    # Initialize Daytona with OTEL enabled
    async with Daytona(DaytonaConfig(
        _experimental={"otelEnabled": True}
    )) as daytona:

        # Create a sandbox - this operation will be traced
        sandbox = await daytona.create()
        print(f"Created sandbox: {sandbox.id}")

        # Execute code - this operation will be traced
        result = await sandbox.process.code_run(""

import numpy as np
print(f"NumPy version: {np.__version__}")
        "")
        print(f"Execution result: {result.result}")

        # Upload a file - this operation will be traced
        await sandbox.fs.upload_file("local.txt", "/home/daytona/remote.txt")

        # Delete sandbox - this operation will be traced
        await daytona.delete(sandbox)

    # Traces are automatically flushed when exiting the context manager

if __name__ == "__main__":
    asyncio.run(main())

// Set OTEL configuration
process.env.OTEL_EXPORTER_OTLP_ENDPOINT = "https://otlp.nr-data.net:4317"
process.env.OTEL_EXPORTER_OTLP_HEADERS = "api-key=YOUR_API_KEY"

import { Daytona } from '@daytonaio/sdk'

async function main() {
  // Initialize Daytona with OTEL enabled
  await using daytona = new Daytona({
    _experimental: { otelEnabled: true }
  })

  // Create a sandbox - this operation will be traced
  const sandbox = await daytona.create()
  console.log(`Created sandbox: ${sandbox.id}`)

  // Execute code - this operation will be traced
  const result = await sandbox.process.codeRun(`

import numpy as np
print(f"NumPy version: {np.__version__}")
  `)
  console.log(`Execution result: ${result.result}`)

  // Upload a file - this operation will be traced
  await sandbox.fs.uploadFile('local.txt', '/home/daytona/remote.txt')

  // Delete sandbox - this operation will be traced
  await daytona.delete(sandbox)

  // Traces are automatically flushed when the daytona instance is disposed
}

main().catch(console.error)

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/daytonaio/daytona/libs/sdk-go/pkg/daytona"
    "github.com/daytonaio/daytona/libs/sdk-go/pkg/types"
)

func main() {
    // Set OTEL configuration
    os.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", "https://otlp.nr-data.net:4317")
    os.Setenv("OTEL_EXPORTER_OTLP_HEADERS", "api-key=YOUR_API_KEY")

    ctx := context.Background()

    // Initialize Daytona with OTEL enabled
    client, err := daytona.NewClientWithConfig(&types.DaytonaConfig{
      Experimental: &types.ExperimentalConfig{
        OtelEnabled: true,
      },
    })
    if err != nil {
      log.Fatal(err)
    }
    defer client.Close(ctx) // Flushes traces on exit

    // Create a sandbox - this operation will be traced
    sandbox, err := client.Create(ctx, nil)
    if err != nil {
      log.Fatal(err)
    }
    fmt.Printf("Created sandbox: %s\n", sandbox.ID)

    // Execute code - this operation will be traced
    result, err := sandbox.Process.CodeRun(ctx, &types.CodeRunParams{
        Code: `

import numpy as np
print(f"NumPy version: {np.__version__}")
        `,
    })
    if err != nil {
      log.Fatal(err)
    }
    fmt.Printf("Execution result: %s\n", result.Result)

    // Upload a file - this operation will be traced
    err = sandbox.Fs.UploadFile(ctx, "local.txt", "/home/daytona/remote.txt")
    if err != nil {
      log.Fatal(err)
    }

    // Delete sandbox - this operation will be traced
    err = client.Delete(ctx, sandbox, nil)
    if err != nil {
      log.Fatal(err)
    }

    // Traces are flushed when client.Close is called via defer
}

require 'daytona'

# Set OTEL configuration
ENV["OTEL_EXPORTER_OTLP_ENDPOINT"] = "https://otlp.nr-data.net:4317"
ENV["OTEL_EXPORTER_OTLP_HEADERS"] = "api-key=YOUR_API_KEY"

# Initialize Daytona with OTEL enabled
config = Daytona::Config.new(
  _experimental: { 'otel_enabled' => true }
)
daytona = Daytona::Daytona.new(config)

begin
  # Create a sandbox - this operation will be traced
  sandbox = daytona.create
  puts "Created sandbox: #{sandbox.id}"

  # Execute code - this operation will be traced
  result = sandbox.process.code_run("

import numpy as np
print(f'NumPy version: {np.__version__}')
  ")
  puts "Execution result: #{result.result}"

  # Upload a file - this operation will be traced
  sandbox.fs.upload_file("local.txt", "/home/daytona/remote.txt")

  # Delete sandbox - this operation will be traced
  daytona.delete(sandbox)
ensure
  daytona.close # Flushes traces
end

What Gets Traced

The Daytona SDK automatically instruments the following operations:

SDK-Level Operations

create() - Sandbox creation and initialization
get() - Retrieving sandbox instances
findOne() - Finding sandboxes by filters
list() - Listing sandboxes
start() - Starting sandboxes
stop() - Stopping sandboxes
delete() - Deleting sandboxes
All sandbox, snapshot and volume operations (file system, code execution, process management, etc.)

HTTP Requests

All API calls to the Daytona backend
Request duration and response status codes
Error information for failed requests

Trace Attributes

Each trace includes valuable metadata such as:

Service name and version
HTTP method, URL, and status code
Request and response duration
Error details (if applicable)
Custom SDK operation metadata

Dashboard Examples

Troubleshooting

Verify Traces Are Being Sent

Check that environment variables are set correctly
Verify your OTLP endpoint is reachable
Confirm API keys/headers are valid
Check your observability platform for incoming traces
Look for connection errors in application logs

Common Issues

Traces not appearing:

Ensure otelEnabled: true is set in the configuration
Verify OTLP endpoint and headers are correct
Check that you’re properly closing/disposing the Daytona instance to flush traces

Connection refused:

Verify the OTLP endpoint URL is correct
Ensure the endpoint is accessible from your application
Check firewall rules if running in a restricted environment

Authentication errors:

Verify API key format matches your provider’s requirements
Check that the OTEL_EXPORTER_OTLP_HEADERS format is correct (key=value pairs)

Best Practices

Always close the client: Use async with (Python), await using (TypeScript), defer client.Close() (Go), or ensure daytona.close (Ruby) to ensure traces are properly flushed
Monitor trace volume: Be aware that enabling tracing will increase network traffic and storage in your observability platform
Use in development first: Test OTEL configuration in development before enabling in production
Configure sampling: For high-volume applications, consider configuring trace sampling to reduce costs