Chunking is one of the most critical but often underestimated components in Retrieval-Augmented Generation (RAG) systems. It determines how documents are split into smaller pieces before being embedded and stored in a vector database.
Good chunking improves retrieval accuracy, reduces hallucinations, and ensures that language models receive the right amount of context within their limited context window.
Why Chunking Matters
Large language models have a fixed context window, meaning they can only process a limited number of tokens at once. If retrieved documents are too large, important information gets diluted or truncated.
If chunks are too small, they lose semantic meaning and context. The goal of chunking is to find the optimal balance between completeness and focus.
# Too large chunk (bad for retrieval)
chunk = "This entire book chapter with multiple topics mixed together..."
# Too small chunk (loses meaning)
chunk = "AI"
print(chunk)What is a Context Window?
A context window is the maximum number of tokens a language model can process at once. This includes both the input prompt and the generated output.
If the retrieved content exceeds this limit, older tokens are truncated, potentially removing important context.
Basic Chunking Strategy
The simplest chunking approach splits documents into fixed-size blocks based on character count, word count, or token count.
While easy to implement, this approach often breaks semantic meaning mid-sentence or mid-paragraph.
def simple_chunk(text, chunk_size=200):
return [text[i:i+chunk_size] for i in range(0, len(text), chunk_size)]
text = "Artificial Intelligence is transforming industries globally..."
chunks = simple_chunk(text)
print(chunks)Token-Based Chunking
More advanced systems split text based on token count instead of characters. This aligns better with how language models actually process input.
Token-based chunking ensures that chunks stay within model limits and improves consistency across different languages and formats.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('gpt2')
text = "Machine learning enables systems to learn patterns from data."
tokens = tokenizer.encode(text)
print(len(tokens))Chunk Overlap Strategy
Chunk overlap is a technique where adjacent chunks share some portion of text. This helps preserve context across boundaries and reduces information loss.
Without overlap, important relationships between sentences may be lost during retrieval.
def overlapping_chunks(text, size=100, overlap=20):
chunks = []
i = 0
while i < len(text):
chunks.append(text[i:i+size])
i += size - overlap
return chunks
print(overlapping_chunks('Deep learning is powerful and widely used in AI systems.', 30, 10))Semantic Chunking
Semantic chunking splits documents based on meaning rather than fixed size. It uses NLP techniques or embedding similarity to identify natural boundaries like paragraphs or topic shifts.
This approach produces higher-quality retrieval results because each chunk represents a coherent idea.
sentences = [
'Transformers revolutionized NLP.',
'They use self-attention mechanisms.',
'Pizza is a popular food.'
]
# Group semantically related sentences
semantic_chunk = sentences[:2]
print(semantic_chunk)Hierarchical Chunking
Hierarchical chunking organizes documents into multiple levels such as sections, paragraphs, and sentences. This allows retrieval systems to first find high-level context and then drill down into finer details.
It is especially useful in long documents like legal contracts, research papers, and technical manuals.
Parent-Child Chunking
In parent-child chunking, small chunks are embedded for retrieval, but larger parent chunks are returned to the LLM for context. This improves both precision and completeness.
This strategy is widely used in production RAG systems where small embeddings improve search accuracy while larger context improves reasoning.
child_chunk = 'Self-attention computes relationships between tokens.'
parent_chunk = 'Transformer architecture includes self-attention, feed-forward layers, and normalization.'
print(child_chunk)
print(parent_chunk)Chunk Size Trade-offs
Choosing the right chunk size is a balancing act. Small chunks improve precision but reduce context. Large chunks preserve context but reduce retrieval accuracy.
Most production systems use chunk sizes between 200 and 1000 tokens depending on the use case.
chunk_sizes = {
'small': 200,
'medium': 500,
'large': 1000
}
print(chunk_sizes)Context Window Optimization
Even after retrieval, the total context passed to the model must fit within its context window. Optimization involves selecting only the most relevant chunks and removing redundancy.
Techniques like ranking, deduplication, and summarization are often applied before feeding context into the LLM.
Retrieval + Compression
Advanced systems compress retrieved content before sending it to the model. This may include summarization, key sentence extraction, or embedding-based filtering.
Compression allows more relevant information to fit inside limited context windows.
retrieved_chunks = ['AI is powerful', 'AI is transforming industries', 'AI is widely used']
compressed = 'AI is transforming industries and widely used'
print(compressed)Chunking for RAG Systems
In Retrieval-Augmented Generation systems, chunking directly impacts retrieval quality. Poor chunking leads to irrelevant results, while good chunking ensures semantically coherent retrieval units.
Chunking is often more important than the choice of vector database or embedding model.
Chunking for Multimodal Data
Chunking is not limited to text. In multimodal systems, audio, images, and video are also segmented into meaningful units before embedding.
For example, videos are chunked into scenes, and audio into speech segments.
Common Mistakes in Chunking
Common mistakes include using arbitrary fixed sizes, ignoring sentence boundaries, skipping overlap, and failing to align chunks with semantic structure.
These mistakes often lead to poor retrieval performance even if embeddings and vector databases are high quality.
Modern AI Engineering Reality
Chunking and context window optimization are foundational to building scalable RAG systems. They directly influence retrieval accuracy, response quality, and system efficiency.
For AI Engineers, RAG Architects, and LLMOps teams, mastering chunking strategies is essential for building production-grade AI applications that reliably handle large-scale knowledge.