Name the start and the first output
An application can start timing at the user’s action, at request send or inside the model server. Those intervals include different work. AIPerf’s reference defines client TTFT from request start to the first non-empty streamed content response, including network, queuing and prompt processing. Retrieval done before that request is outside that timer, but still part of the user’s wait.
The first content response can contain several tokens. It can also contain reasoning content rather than text the user sees. vLLM’s benchmark guide warns that metric names are not standardized: record the actual start/end events and formula. Its gap between streamed outputs and its per-request average time per output token need not give the same statistics.
Compare the first token with the last token
Invented streams: each request produces ten output tokens, one token per streamed response, with no extra buffering or separate reasoning output. Both clocks start at request send. The gap after the first token is constant within each stream.
| Measure | Stream A | Stream B |
|---|---|---|
| First token arrives | 0.6 seconds | 0.2 seconds |
| Each later gap | 0.1 seconds | 0.2 seconds |
| Tenth token arrives | 1.5 seconds | 2.0 seconds |
There are nine gaps after the first token: A ends at 0.6 + 9 × 0.1 = 1.5 seconds; B at 0.2 + 9 × 0.2 = 2.0 seconds. B starts earlier but delivers the last token later. Real streams with multi-token chunks or uneven gaps need the measured timestamps, and a protocol’s final completion event or the application’s checks may finish later still.
Pair initial responsiveness with a useful result
A chatbot that immediately says “Let me check” but needs another 20 seconds to answer has a quick first visible response, not quick useful completion. Latency follows several milestones on one timeline. Keep first-output, request-completion and task-completion measurements separate; do not add overlapping intervals.
In episode 59 at 2:20, Superhuman’s Loïc Houssier explains why adding model inference challenges a product built around very fast interactions. That discussion concerns the whole experience; its 100-millisecond product goal is not a claim that every model request finishes in that time.
Report typical and slow-tail timings with input/output lengths, request rate or concurrency, cache state and failures or timeouts. Inference throughput adds a different question: how much work the whole serving system completes. The response-latency guide connects these measurements to preparation, caching and product tradeoffs.
Sources
- vLLM: Metrics — Lists server-side histograms for time to first token, inter-token latency, time per output token, queue time and end-to-end latency; checked November 2, 2026.
- NVIDIA: AIPerf metrics reference — Defines client TTFT up to the first non-empty streamed response, which can be a chunk of tokens and can include reasoning tokens, and includes network, queuing and prompt processing; checked November 2, 2026.
- vLLM: Benchmark CLI — Says metric terminology is not standardized across tools and distinguishes inter-token latency from per-request time per output token; checked November 2, 2026.