Fix the numerator and the observation window
vLLM’s metrics documentation separates prompt-token counters, generated-token counters and finished-request counts. They measure different work. Name the counted unit and the observation window before comparing rates, including whether reasoning tokens, failed requests and partially completed work are counted.
AIPerf’s reference distinguishes aggregate token throughput over a benchmark’s duration from per-request streaming rates that exclude the initial wait. A rate over the post-first-token interval is not interchangeable with a rate over the full request or benchmark window. Check formulas and token accounting rather than matching only “tokens per second” labels.
Compare aggregate work with one user’s wait
Invented windows: every request starts at its window’s beginning, produces ten output tokens and finishes at its end. There are no failures or partial requests in these examples.
| Measure | One request | Four concurrent |
|---|---|---|
| Completed requests | 1 | 4 |
| Total output tokens | 10 | 40 |
| Window duration | 1 second | 2 seconds |
| Aggregate output rate | 10 tokens/s | 20 tokens/s |
| Each request’s latency | 1 second | 2 seconds |
The aggregate rates are 10 / 1 = 10 and 40 / 2 = 20 output tokens per second. In the second window the system completes more work per second, while each user waits longer. It does not mean one user receives 20 tokens each second.
Check capacity under the workload you need
vLLM’s benchmark guide distinguishes arrival rate, burstiness and outstanding-request limits; a client-side concurrency limit can add waiting before a request reaches the server. Record those conditions alongside input/output lengths, cache warmth, hardware, model and serving-software versions. Repeated prompts can benefit from prefix reuse, so a warm run need not describe the first request.
Pair aggregate rates with time to first token, latency distributions and failure or timeout counts. Count work that meets the application’s timing and correctness requirements as well as raw completions. A batch summarizer may prioritize documents per hour; interactive chat also needs a usable first response and a readable stream. A capacity number alone does not establish answer quality or a safe operating limit.
The response-latency guide follows Loïc Houssier’s product-level decisions in episode 59.
Sources
- vLLM: Metrics — Documents separate prompt-token and generation-token counters, finished-request counts and per-request timing histograms; checked November 2, 2026.
- NVIDIA: AIPerf metrics reference — Separates aggregate output token throughput over the benchmark duration from a per-user rate that excludes time to first token; the aggregate figure counts reasoning tokens; checked November 2, 2026.
- vLLM: Benchmark CLI — Explains request rate, burstiness, maximum concurrency with client-side queueing, and prefix-cache reuse across repeated runs; checked November 2, 2026.