Inference

Inference is the compute a model consumes, which runs on GPUs and is billed in tokens.

Inference is the compute that a model consumes. Inference runs on GPUs and is billed in tokens. A GPU executes instructions with thousands of simpler cores that are suited to parallel math, and that design is the reason GPUs dominate machine learning workloads. The GPUs that serve inference are elsewhere, separate from the sandboxes in which the code of an agent runs.