Reuse the input computation, not a stored answer
A support agent may send the same instructions and reference manual with each new question. Claude’s prompt-caching documentation describes reusing a matching prefix up to a cache breakpoint. vLLM’s overview describes reusing the prefix’s key/value state. Both leave a new response to generate; this is different from returning a stored answer to a similar question.
Prefix identity is more specific than similar meaning. In vLLM’s design, a block’s identity includes its tokens and preceding-prefix hash, with additional key inputs. Editing the beginning can invalidate later reuse even if the rest looks unchanged. The exact matching rules, block boundaries and supported inputs depend on the serving system.
Account for reads, writes and full context
Invented token sequences: prefix P contains 4,096 tokens and each new question adds 128. Assume this prefix is eligible and stored exactly, the first request’s cache is available before the second begins, and no expiry or eviction occurs between them. There are no other cached prefixes. P′ replaces P’s first token while keeping the same token count.
| Request | Prefix tokens read | Prefix tokens written |
|---|---|---|
| First P | 0 | 4,096 |
| Repeated P | 4,096 | 0 |
| Changed P′ | 0 | 4,096 |
Each request also has 128 ordinary question tokens and a full logical input of 4,096 + 128 = 4,224 tokens. A cache hit does not turn that into a 128-token context. Across the three requests, reads are 4,096 tokens, writes 8,192 and ordinary input 384: 4,096 + 8,192 + 384 = 12,672 total input tokens.
Claude’s documentation distinguishes cache_read_input_tokens, cache_creation_input_tokens and input_tokens; in that API the last field alone is not the entire supplied input. Other systems can expose different counters. Keep the actual provider’s accounting with a result rather than assuming every field called “input tokens” has the same scope.
Test the miss path as well as the warm path
vLLM’s benchmark guide warns that repeated runs can reuse cached prompts. Compare an initial request, a matching repeat and a request after the relevant source changes. Record cache-hit usage with time to first token, full-request latency, throughput and answer checks. A warm benchmark alone does not describe a cold request or establish useful completion time.
In episode 59 at 2:20, Loïc Houssier describes preparing voice and tone context before dictation is ready. That is the same idea applied to a product: move repeatable work off the waiting path. He does not say which caching mechanism Superhuman uses. The response-latency guide develops that broader product decision.
Keep stable material before changing questions when that layout fits the task, then check current minimum lengths, lifetime, write/read charges and isolation rules for the chosen API and model. Caching does not refresh outdated source content or remove irrelevant material from the logical context. Use cost per completed task alongside timing and quality.
Sources
- Claude Platform: Prompt caching — Documents matching prefixes, cache breakpoints and the separate cache_read_input_tokens, cache_creation_input_tokens and input_tokens usage fields; checked November 2, 2026.
- vLLM: Automatic prefix caching design — Explains block identities built from exact tokens, the preceding-prefix hash and extra cache-key inputs; checked November 2, 2026.
- vLLM: Automatic prefix caching overview — Describes reusing a shared prefix's KV state to skip repeated input processing; checked November 2, 2026.
- vLLM: Benchmark CLI — Warns that repeating a benchmark against the same server can reuse prompts left in the prefix cache and inflate throughput; checked November 2, 2026.