A help-center assistant gets “How do I reset my password?” thousands of times a day, phrased many ways: “forgot my password”, “can’t log in, need a new password”. A semantic cache turns each incoming question into an embedding, searches earlier questions in a vector database, and if one is close enough, returns its saved answer without calling the model. The reply arrives in milliseconds and costs almost nothing.
The key setting is the similarity threshold. Set it loose and “How do I cancel my order?” may get the answer to “How do I cancel my subscription?”. Set it tight and the cache rarely hits. Cached answers also go stale when the underlying policy changes, and a shared cache must never return one user’s personal answer to another. So teams cache only generic, low-risk questions, partition the cache by user or tenant where needed, expire entries, and measure wrong-hit rates, not just hit rates.
Semantic caching is different from prompt caching. Prompt caching reuses the model’s computation for an identical prefix and still generates a fresh answer; semantic caching skips generation and returns an old answer. Many AI gateways offer one or both.
In episode 32 at 24:31, Elastic’s Philipp Krenn describes the pattern without naming it: a search engine can cache results so that when somebody asks a very similar question, the app reuses the earlier answer for a faster and cheaper response. Elastic sells that search engine. The episode was sponsored by Galileo; Conor introduces Galileo CEO Vikram Chatterji as a guest host at 0:04. See semantic search for the retrieval technique underneath.
Hear it from the guest
“If somebody has a very similar question, you might be able to skip like regenerating with the LLM”
Quotes lightly edited to remove filler words.
Sources
- Microsoft Learn: Enable semantic caching for LLM APIs in Azure API Management — Semantic caching returns cached responses for identical prompts and for prompts similar in meaning, reducing backend load and latency; configured with an embeddings backend and a similarity score threshold; checked October 24, 2026.
- GitHub: zilliztech/GPTCache — Open-source semantic cache for LLM applications built on embeddings and vector search; checked October 24, 2026.
