Inference

A language-model request turns supplied context into generated output. An agent task may make many such requests as it uses tools and reacts to results. Retrieval, application logic, and external actions add work around those model calls.

For autoregressive transformers, input processing is commonly called prefill and incremental output generation is called decode. Long input and long output can create different serving demands. The model’s architecture and runtime determine the details.

Example: a task that costs little on its first call can become expensive after repeated searches, large tool responses, and retries. Measure the complete task as well as individual calls. Model size, output length, hardware, batching, and reused input computation all affect serving behavior.

Training is not necessarily a one-time event: teams may periodically retrain or fine-tune. The practical distinction is that inference uses an existing model to do work. See how to cut agent costs for the choices available around it.