Skip to content

AI Glossary · Concepts

What is Chunking?

Chunking

In retrieval systems, chunking divides source material into smaller pieces that can be indexed and returned as evidence. Chunk boundaries determine how much context survives when a passage is retrieved on its own.

· Updated · Chain of Thought

RAG & Retrieval

  • 2 explainers
  • 2 sources

Consider a returns policy with a general 30-day rule and an exception for custom orders. If the exception lands in a separate chunk, a search can find the general rule and miss the qualification. Keep the rule and its relevant exception together, or give retrieval a way to return both.

A chunk saying “the limit is 30 days” also needs the product or customer tier named by its heading. Preserve useful headings and source identifiers with the text. Anthropic’s Contextual Retrieval approach addresses this loss of context by adding a short explanation of each chunk’s place in the document before indexing it. Check any generated explanation against the source and keep it distinguishable from the verbatim text; added context can be wrong too.

There is no universal best size. Small chunks can retrieve a precise passage but omit context; large ones can include the context but also unrelated text. Overlap can protect boundaries at the cost of duplicates. Test representative questions and inspect whether their supporting evidence survives the split, using retrieval evaluation.

Count overlap in the units your splitter uses

A splitter’s size and overlap settings are counted in whatever unit its length function measures. LangChain’s recursive splitter example passes length_function=len, which counts Python string characters, not model tokens, and its documentation calls chunk_overlap a target overlap. Inspect the actual splits rather than assuming every neighboring pair shares exactly that amount.

Hand-worked example, not splitter output: split the ten characters ABCDEFGHIJ into two slices of six. Positions are one-based and inclusive.

SliceTextSource positions
1ABCDEF1–6
2EFGHIJ5–10

The two slices store 12 characters for 10 source characters. EF appears in both: the overlap repeats positions 5–6, it does not add new evidence. If both slices come back for one question, the repeated EF is not independent corroboration; the non-overlapping text can contain different information. Keep each chunk’s source ID and position, so repeated hits from overlapping chunks can be recognized as duplicates. These counts say nothing about model tokens or the right chunk size.

Keep the exception with the ruleA returns policy has a general 30-day rule and an exception that custom orders cannot be returned. Splitting them into separate chunks can leave an answer with only the general rule. A context-preserving chunk includes the heading, rule and exception, with its source. Keep the exception with the rule Chunk boundaries change the evidence an answer seesSOURCE: RETURNS POLICYReturns accepted within 30 days.Exception: custom orderscannot be returned.CHUNK AReturns accepted within 30 days.CHUNK BCustom orders cannot be returned.Retrieving only A misses the exception.CONTEXT-PRESERVING CHUNKReturns policy: within 30 days, except custom orders.Retain the heading and source; retrieve the qualification too.
An illustrative returns policy. Anthropic’s Contextual Retrieval article explains why isolated chunks need their document context. Download the image

Sources

Go deeper