Latency is response time, and for language models it has structure worth separating. Time-to-first-token is how long until the model starts streaming output — what users feel as “is it working?” Total latency is how long until it finishes. The two trade off against model size, prompt length, and how much the model reasons before answering.
It belongs next to accuracy in any real evaluation, because it’s a product decision as much as a technical one. A 100-millisecond interaction and a ten-second one are different products even at identical quality, which is why teams routinely trade a little accuracy for a lot of speed — a smaller model, trimmed context, or caching — when the use case is interactive. Latency is also an attack surface: model denial of service uses expensive requests to slow or block a system.
Sources
- Anthropic: Reducing latency — Separates baseline latency from time to first token, the time from sending the prompt to the first generated token, which matters most when streaming; checked October 26, 2026.
Go deeper
From the conversation
-
Architecting Reliable Agentic AI | Cisco’s Giovanna Carofiglio on the AGNTCY Collective -
How Intercom Cut $250K/Month by Ditching GPT for Open Models | Fergal Reid -
Explaining Eval Engineering | Galileo's Vikram Chatterji -
Agents Are Overtaking Humans in Search: Why Search Must Be Rebuilt -
The Enterprise AI Deployment Playbook | ServiceTitan, Indeed & Twilio -
The AI Agent Trust Gap: Bridging Risk to Reliability | Elastic’s Philipp Krenn -
The Critical Infrastructure Behind the AI Boom | Cisco CPO Jeetu Patel -
Beyond Transformers: How Liquid AI Is Rethinking LLM Architecture | Maxime Labonne