Match meaning in the task you actually have
Someone asking “How do I stop my subscription?” may need a document titled “Cancellation policy.” Embeddings can connect those phrases even though the words differ. Test it with the paraphrases your users actually type.
Sentence Transformers distinguishes symmetric search, such as finding similar questions, from asymmetric search, such as finding a long answer passage for a short question. The query and document encodings must suit that task; some models use different prompts or trained paths for each. A setup tuned for finding duplicate questions can still retrieve policy passages badly.
Test near matches against the right scope
Invented test cases: keep the intended product and policy version with each question.
| Query case | Evidence to require |
|---|---|
| “Stop renewal” phrased differently from the policy | The applicable cancellation passage for the requested product |
| An exact error code such as ERR-4106 | The matching code and version, rather than a nearby code’s fix |
| A question the collection cannot answer | A recorded coverage gap, rather than treating a similar passage as support |
Add a cancellation passage for the wrong product as a deliberately tempting near match. A high similarity score does not make it applicable. Judge entity, date and access scope separately from wording. Exact identifiers may need lexical retrieval or structured filters; Weaviate’s hybrid-search documentation illustrates combining keyword and vector results.
Separate retrieval from generation
Semantic search can return ranked passages directly, without a generative model. RAG adds generation using selected evidence. Inspect the actual passages sent onward: useful material in a search result may still be omitted when context is assembled.
In episode 5 at 26:00, Vivienne Zhang describes early copilot failures where the needed answer was absent from the database, as well as chunks cut at unhelpful places. Both are reasons to check coverage and passage boundaries before blaming the retriever.
Keep the corpus snapshot, query set, relevance judgments, encoding setup and filters with a comparison. Test hybrid search and reranking as separate changes, then check the generated answer. The RAG evaluation guide connects those component checks to the show.
Sources
- Sentence Transformers: Semantic search — Explains query/corpus embeddings, symmetric versus asymmetric retrieval and model-dependent query/document encoding; checked November 2, 2026.
- Weaviate: Hybrid search — Distinguishes vector similarity from BM25 keyword retrieval and explains combining their candidate rankings; checked November 2, 2026.
Go deeper
- Vector database or knowledge graph — which should you use for AI retrieval? AI, decoded · Vector Database vs. Knowledge Graph for AI Retrieval
- What is RAG, and why do AI systems use it? AI, decoded · What Is RAG (Retrieval-Augmented Generation)
- How do you evaluate a RAG system? AI, decoded · How to Evaluate a RAG System