Skip to content

AI, decoded

Vector database or knowledge graph — which should you use for AI retrieval?

Use a vector database when relevance is about meaning — finding passages similar to a question across unstructured text. Use a knowledge graph when the answer depends on explicit relationships and facts — who connects to what, and how. They complement each other: vectors find the right neighborhood, a graph enforces the right facts, and pairing them is increasingly how teams cut hallucination. On Chain of Thought, Neo4j's Sudhir Hasbe said customers starting with pure vector search saw accuracy around 60 to 65%, while some combining graphs and vectors are well above 90% accurate.

· Updated · Chain of Thought

Level 4: How AI systems are built · 4.1 Retrieval and RAG

RAG & Retrieval

  • 2 episodes
  • 2 guest clips
  • 8 glossary terms
A loose cloud of vector points beside a small graph of linked nodes, under the words: close is not correct. Vectors find it; a graph constrains it.

In detail

Both power retrieval-augmented generation — giving a model the right context at answer time — but they retrieve on different principles. The choice follows the shape of your data and your tolerance for a confidently wrong answer.

1. Vector databases retrieve by meaning

A vector database stores text as embeddings and finds passages whose meaning is closest to the query, even with no shared keywords. Weaviate’s Bob van Luijt says large-scale search, mostly hybrid search, was the first thing his customers took to production: it’s a fast way to make a large pile of unstructured documents searchable by intent. Where it’s weak is precision on facts — “close in meaning” can surface a passage that’s relevant but not correct.

2. Knowledge graphs retrieve by relationship

A knowledge graph stores entities and the explicit links between them — facts, hierarchies, who-relates-to-what. On the show, Neo4j’s Sudhir Hasbe argues that hallucination is a data-architecture problem: when answers must respect real relationships and constraints, a graph gives the model grounded, structured facts instead of fuzzy nearest neighbors. The cost is that you have to model and maintain that structure.

3. The pattern is “and,” not “or”

The strongest setups use both — often called GraphRAG. Vectors cast a wide net to find the relevant neighborhood; the graph then constrains and enriches it with the exact entities and relationships that make the answer factual. Vectors handle recall over messy text; the graph handles precision over facts.

How to choose

Mostly unstructured text, relevance by topic, speed to stand up → vector database. Answers that hinge on explicit relationships, compliance, or traceable facts → knowledge graph. Hallucination is biting and your domain has real structure → combine them.

Why it matters

Picking one because it’s the default — usually the vector database — leaves precision on the table for problems that are really about relationships. Let the data decide: meaning-similarity and fact-structure are different jobs, and the best retrieval stacks do both.

Vector database vs. knowledge graph: close in meaning is not the same as correct Left, a vector database retrieves by meaning: points in an embedding space, and the query lands near its nearest neighbors, even with no shared keywords. Its weakness is precision on facts. Neo4j's Sudhir Hasbe, in episode 55, retells a Spotify example: listening to a pop song, the next vector-similar result could be country music, because in vector space it is just mathematical similarity. Right, a knowledge graph retrieves by relationship: entities joined by explicit links, such as a song belonging to a genre and recorded by an artist (an illustration). The cost is modeling and maintaining that structure. Bottom, the pattern is "and," not "or": GraphRAG, which Hasbe describes as a vector search with a filtering layer of metadata applied as a graph layer on top. Vectors find the neighborhood; the graph constrains it by entities, relationships and metadata. Choose by the data: mostly unstructured text goes to vectors, answers that hinge on explicit relationships go to a graph, and when hallucination bites in a structured domain, combine them. Close in meaning isn’t the same as correct Two ways to retrieve context. Different jobs, and the strongest setups do both. VECTOR DATABASE · BY MEANING query and its nearest neighbors Listening to a pop song… pop song next could be: country “just mathematicalsimilarities” · Sudhir Hasbe Wins: recall over messy text. Weak: precision on facts. KNOWLEDGE GRAPH · BY RELATIONSHIP song genre: pop artist other pop songs label belongs to by Entities and explicit links: facts the model must respect. Wins: precision over facts. Cost: modeling the structure. “And,” not “or”: GraphRAG vector search finds the neighborhood graph layer on top filters by relationships answer Unstructured text → vectors Relationships, compliance → graph Hallucination + real structure → combine them GraphRAG as Neo4j’s Sudhir Hasbe describes it in episode 55: a vector search, then a metadata filter as a graph layer on top.
Meaning-similarity and fact-structure are different jobs; the best retrieval stacks do both. The Spotify example is Neo4j’s Sudhir Hasbe retelling another interview, and the GraphRAG description is his, episode 55; the song graph is an illustration. Download the image

Hear it from the guest

“You may be listening to a pop song and suddenly the next vector similarity could show you in country music or some other music genre. Because in vector space, it's just mathematical similarities that you're looking at.”
Sudhir Hasbe, Neo4j EP 55 · around 12:54 · read the transcript
Watch on YouTube
“When I say graph RAG, it is a combination of, I can do a vector search, And then I'm applying a filtering layer with all the other metadata as a graph layer on top of it.”
Sudhir Hasbe, Neo4j EP 55 · around 14:04 · read the transcript
Watch on YouTube

Quotes lightly edited to remove filler words.

Go deeper

  • Getting started with Faiss Faiss maintainers · Technical tutorial Trace nearest-neighbor IDs and squared distances in a flat L2 index. The example uses synthetic vectors, not text embeddings; running it needs Faiss and Python/NumPy or C++.
  • MATCH patterns in Cypher Neo4j · Technical documentation Compare the distance-based shortlist with explicit relationship patterns and returned rows. Running the examples needs a sample Neo4j database; stored relationships still need accurate source data.

From the conversation

This explainer is drawn from these episodes — each carries its full transcript.

Concepts in this explainer

Knowledge GraphVector DatabaseGraphRAGSemantic SearchEmbeddingsRetrieval-Augmented Generation (RAG)AI HallucinationAccuracy

Keep exploring