Skip to content

AI Glossary · Concepts

What is Embeddings?

Embeddings

Embeddings are numerical representations of text, images, or other data as vectors, where things with similar meaning land close together. They're what lets a system search by meaning instead of by exact keyword.

Also known as: vector embeddings, text embeddings

· Chain of Thought

RAG & Retrieval

  • 2 explainers
  • 7 episodes
  • 1 source

An embedding turns a piece of data into a list of numbers, a vector, that captures its meaning. Two sentences that mean similar things produce vectors that sit near each other in that space, even if they share no words. That’s the trick behind semantic search: instead of matching keywords, you compare vectors and find the closest ones. A recommendation system can use the same comparison to find items close to what a user already watched or bought.

Embeddings are the foundation of retrieval-augmented generation. When you store documents in a vector database, you’re storing their embeddings; when a question comes in, you embed it too and pull back the chunks whose vectors are nearest. The quality of the embedding model decides how well “near in vector space” matches “actually relevant” — which is why embedding choice is one of the levers that makes or breaks a retrieval system.

Word embeddings such as word2vec were an early, word-level version of the idea. Many embedding models are trained with contrastive learning, which teaches a model which pairs of examples belong together.

Compare direction and length

These are invented two-dimensional vectors, not actual word embeddings. A stays at (1, 0). Rotate or resize B to compare a dot product with cosine similarity.

The worked example uses B at 60 degrees with length 1. With JavaScript enabled, you can change it below.

Two vectors from the same origin A = (1, 0), shown as a solid arrow. B is dashed. B = (0.500, 0.866). Dot product: 0.500. Cosine similarity: 0.500. (0, 0)
Solid arrow: A. Dashed arrow: B. The outer arc marks length 2; both axes use the same scale.

B = (0.500, 0.866). Dot product: 0.500. Cosine similarity: 0.500.

Dot product A · B
0.500
Length product ‖A‖ × ‖B‖
1.000
Cosine: dot ÷ length product
0.500

Check the worked cases
  • At 60 degrees, doubling B's length from 1 to 2 changes the dot product from 0.500 to 1.000. Cosine stays 0.500.
  • At 90 degrees and length 1, both scores are 0.000. At 180 degrees and length 1, both are −1.000.
  • At length 0, B has no direction. The mathematical cosine divides by zero and is undefined. Libraries can use a convention; scikit-learn returns zero in its zero-vector example.

Cosine divides out positive vector length. A dot product retains it. When both vectors have length 1, their dot product equals their cosine similarity.

Real embeddings can have many dimensions, with similarity depending on the model and comparison method. A high score alone establishes neither relevant evidence nor a correct answer.

Check the formula and library convention in scikit-learn's cosine similarity documentation. For length normalization in retrieval, read Introduction to Information Retrieval: dot products.

Sources

  • Reimers and Gurevych: Sentence-BERT — A concrete text-embedding method that compares learned sentence vectors with cosine similarity; it does not guarantee retrieval relevance.

Go deeper

  • Sentence Transformers: Semantic Search docs

    Encode queries and documents, compare search results, and distinguish similar-question search from question-to-passage retrieval. Python examples download a model.

From the conversation

Agentic RAGRetrieval-Augmented Generation (RAG)Vector DatabaseAnswer Engine