Chunking decides what your retriever can find. If the answer to a question is split across two chunks, or buried in a chunk about something else, no reranker or prompt will fully rescue it.
Most guides list strategies in the abstract. This one takes a single sample document, chunks it three ways and shows what each chunk actually contains. Then it explains how to pick a strategy and size for your own corpus.
What Chunking Does in a RAG Pipeline
A RAG system splits source documents into chunks, embeds each chunk and stores it in an index. At query time it retrieves the chunks closest to the question and passes them to the model as context.
Two forces pull against each other. Small chunks are precise but may lose context. Large chunks keep context but blur the embedding and add noise. Chunking is how you balance the two.
For where chunking sits among the other stages, see what a RAG pipeline is, stage by stage.
The Main Strategies in One Paragraph Each
• Fixed-size: split every N tokens or characters, often with overlap. Simple and predictable, but blind to sentences, tables and headings.
• Recursive: split on the largest natural separator first (paragraphs), then lines, sentences and words only when a piece is still too large. LangChain's text splitter docs describe this approach.
• Structure-aware (heading-based): split along headings, sections or pages, and attach the heading path as metadata.
• Semantic: embed sentences and start a new chunk where similarity between neighbors drops. More compute at ingest time.
• Parent-child (small-to-big): retrieve on small chunks, then pass the larger parent section to the model.
Weaviate's chunking guide suggests overlap of roughly 10% to 20% of chunk size as a starting point for fixed-size chunking.
Worked Example: One Document, Three Strategies
The sample below is an original, invented IT policy written for this example. It is short on purpose so you can see every chunk.
Sample document: "Laptop and Remote Access Policy"
• Title (H1): Laptop and Remote Access Policy
• Section heading (H2): 1. Eligibility — Full-time employees receive a company laptop on day one. Contractors receive a laptop only if their contract runs longer than 90 days and their manager approves.
• Section heading (H2): 2. Replacement — Laptops are replaced every four years. Earlier replacement requires a ticket from IT confirming a hardware fault. Accidental damage is covered once per employee; a second incident is charged to the department.
• Section heading (H2): 3. Remote Access — VPN is required for all internal systems. Multi-factor authentication must use the company authenticator app. SMS codes are not accepted. Access from personal devices is not allowed, except for webmail through the browser.
• Section heading (H2): 4. Lost or Stolen Devices — Report a lost laptop to IT within 24 hours. IT will lock the device remotely. If the loss involves customer data, IT also informs the security team, who decide on further steps.
We will test three realistic questions against the chunks:
➤ Q-A: "Can a contractor on a 6-month contract get a laptop?"
➤ Q-B: "Can I use SMS codes for VPN login?"
➤ Q-C: "What happens if I break my laptop a second time?"
Strategy 1: Fixed-Size, About 40 Words, No Overlap
Fixed-size splitting ignores the headings. With roughly 40-word windows, the document breaks like this (shortened):
• Chunk F1: "Laptop and Remote Access Policy. 1. Eligibility. Full-time employees receive a company laptop on day one. Contractors receive a laptop only if their contract runs longer than 90 days and"
• Chunk F2: "their manager approves. 2. Replacement. Laptops are replaced every four years. Earlier replacement requires a ticket from IT confirming a hardware fault. Accidental damage is covered once per"
• Chunk F3: "employee; a second incident is charged to the department. 3. Remote Access. VPN is required for all internal systems. Multi-factor authentication must use the company authenticator app."
• Chunk F4: "SMS codes are not accepted. Access from personal devices is not allowed, except for webmail through the browser. 4. Lost or Stolen Devices. Report a lost laptop to IT within 24 hours..."
What goes wrong:
○ Q-A: the contractor rule is split. F1 says "longer than 90 days and" while the manager-approval condition sits in F2. An answer from F1 alone may omit the approval step.
○ Q-B: "SMS codes are not accepted" lands at the start of F4, separated from "Multi-factor authentication" in F3. The F4 chunk mixes remote access with lost devices, which muddies its embedding.
○ Q-C: "covered once per" is in F2, "a second incident is charged" is in F3. The key fact now lives in a chunk that is mostly about VPN.
Adding about 20% overlap would repair some of these boundary cuts, at the cost of duplicate text in the index.
Strategy 2: Recursive, Split on Paragraphs First
A recursive splitter with a maximum of about 60 words tries paragraph breaks first. Each numbered section fits under the limit, so each becomes one chunk.
• Chunk R1: "1. Eligibility. Full-time employees receive a company laptop on day one. Contractors receive a laptop only if their contract runs longer than 90 days and their manager approves."
• Chunk R2: "2. Replacement. Laptops are replaced every four years... Accidental damage is covered once per employee; a second incident is charged to the department."
• Chunk R3: "3. Remote Access. VPN is required... SMS codes are not accepted. Access from personal devices is not allowed, except for webmail through the browser."
• Chunk R4: "4. Lost or Stolen Devices. Report a lost laptop to IT within 24 hours..."
What improves: all three answers now sit inside a single, on-topic chunk. What is still missing: the chunks do not carry the document title, so R3 does not say which policy it belongs to. In a corpus with ten policies that mention VPN, that matters.
Strategy 3: Heading-Aware With Metadata
A structure-aware splitter cuts on the markdown headings and prepends or stores the heading path.
• Chunk H3 text: "SMS codes are not accepted..." with the full section body
• Chunk H3 metadata: title = "Laptop and Remote Access Policy"; section = "3. Remote Access"; doc_id = "it-pol-007"; effective_date = (from source)
Now the retriever can filter by document, the answer can cite "Laptop and Remote Access Policy, section 3", and a reviewer can trace the claim back to its source.
Side-by-Side Trade-offs for This Document
| Question | Fixed-size (no overlap) | Recursive | Heading-aware + metadata |
|---|---|---|---|
| Q-A contractor laptop | Condition split across F1/F2 | Whole rule in R1 | Whole rule in H1, citable |
| Q-B SMS codes | Fact in mixed-topic chunk F4 | Whole rule in R3 | Whole rule in H3, filterable by policy |
| Q-C second damage | Fact split across F2/F3 | Whole rule in R2 | Whole rule in H2, citable |
| Ingest effort | Lowest | Low | Needs clean headings |
This comparison is a reasoning exercise on one small document, not a benchmark. Real corpora behave differently, which is why the next section matters.
When Each Strategy Fits
• Fixed-size with overlap: messy text with no structure, such as transcripts or OCR output, and quick baselines
• Recursive: general prose such as articles, emails and wiki pages; a sensible default
• Heading-aware: policies, manuals, contracts and docs with reliable headings
• Semantic: long documents whose topics shift without headings, when you can afford extra ingest compute
• Parent-child: when questions need precise matching but answers need surrounding context
Tables, code and lists deserve special handling. Keep a table with its header row, and never split a code block mid-function.
How to Choose Chunk Size Without Guessing
There is no universal best size. NVIDIA's chunking experiments found the best strategy varied by dataset, even among similar document types. Use your own data.
- Collect 30 to 100 real questions with the source passage that answers each
- Chunk the corpus with two or three candidate settings
- For each setting, check whether the answering passage appears in the top k results
- Read the failures; note whether the cause is a split, a diluted chunk or missing metadata
- Pick the simplest setting that fixes the failures you care about
Pinecone's chunking guide also recommends iterating over sizes against your own queries. Our separate guide on combining keyword and vector search covers retrieval changes that complement chunking.
Metadata to Store With Every Chunk
• Document ID and title
• Section or heading path
• Page number for PDFs
• Source URL or file path
• Effective or last-modified date
• Access control labels, so retrieval respects permissions
• Chunk position (previous and next chunk IDs) for context expansion
Access labels are easy to forget and costly to add later. Attach them at ingest time.
Common Chunking Mistakes
○ Splitting tables row by row so headers are lost
○ Dropping titles and headings, then wondering why answers cannot cite sources
○ Using one chunk size for every document type in the corpus
○ Re-chunking without re-running the same question set
○ Overlapping so heavily that the top results are near-duplicates
○ Forgetting to re-embed when the chunking code changes
Chunking Checklist
• Documents grouped by type (policy, manual, transcript, table-heavy)
• Strategy chosen per group, with a written reason
• Tables and code blocks kept whole
• Heading path and document title attached to every chunk
• Access labels attached at ingest
• Question set with known answer passages ready
• Two or three settings compared on that set before launch
• Chunking config versioned alongside the index
Our Own Product Note
Disclosure: this is our own service. LuMay's AI engineering services include RAG development. The worked method above is vendor-neutral and works with any framework or vector database; see our vector database comparison for storage options.
Next Step
Take your ten most-asked questions, find the passages that answer them and run the three-strategy comparison above on those documents. You will learn more from those ten failures than from any default setting.
References
• LangChain, Text splitters (concepts): https://python.langchain.com/docs/concepts/text_splitters/
• Weaviate, Chunking strategies to improve LLM RAG pipeline performance: https://weaviate.io/blog/chunking-strategies-for-rag
• Pinecone, Chunking strategies for LLM applications: https://www.pinecone.io/learn/chunking-strategies/
• NVIDIA Technical Blog, Finding the best chunking strategy for accurate AI responses: https://developer.nvidia.com/blog/finding-the-best-chunking-strategy-for-accurate-ai-responses/
