Split RAG content without losing its meaning

Three settings describe how much content a RAG system retrieves; they interact but are not interchangeable.

Smaller chunks with an unchanged result count may provide less total evidence. Increasing result count may retrieve many similar passages from one file. A larger model context window does not correct poor selection.

Record these settings separately in experiments. Otherwise an apparent improvement cannot be attributed to better boundaries, more candidates or simply more text supplied to the model. The token count alone does not explain the result.

Chunking works on the representation produced by ingestion. If PDF extraction already lost table headings, a splitter cannot reliably recreate them. If a wiki macro is missing, larger chunks will not recover its exception.

Inspect representative types: a short policy, long manual, table, FAQ and scanned appendix. Compare extracted content with the original. Mark places where headings, lists or page boundaries affect meaning.

Microsoft treats chunking as a design phase following document analysis and distinguishes methods by source structure. Its guidance recommends experimentation for the actual collection. Microsoft: RAG chunking phase. Product examples are not universal optimal settings.

The following table compares four chunking approaches by useful starting point and common risk.

Recursive splitting typically tries larger natural boundaries before falling back to smaller units. The implementation matters: the name does not establish how tables, headings or oversized paragraphs are handled.

Semantic splitting is not automatically better. It can detect topic changes but must be evaluated with your language, terminology and questions. Extra processing is justified when it improves the relevant failure cases.

A fictional service policy says standard parts can be returned within thirty days. The next paragraph requires separate approval for custom-made parts. The user asks specifically about a custom-made item.

If retrieval returns only the first paragraph, the model may issue an incorrect general permission. A larger chunk might preserve both statements, but could also absorb unrelated rules and make the relevant passage less discoverable.

One design uses a compact object containing both statements and the heading “Returns by part type.” Another retrieves a smaller passage and then loads its complete parent rule block. Evaluate both against the same questions; the example does not establish a measured winner.

A sentence such as “This applies only to contract customers” is unusable without the rule it qualifies. Add the necessary section path or title so the passage remains interpretable. Repeating the entire table of contents in every chunk is unnecessary.

Keep document identity, version, section, page reference and audience. These fields support filtering, citations and deletion, and must remain consistent after rechunking.

Generated summaries may enrich retrieval objects, but they introduce additional model-authored claims. Mark them as derivatives and retain the original text. A summary that removes an exception must not silently replace authoritative evidence.

Overlap repeats content near adjacent chunk boundaries. It can preserve meaning across a cut, but increases index size and can return near-duplicate candidates.

Do not treat overlap as insurance against every structural problem. A small overlap cannot connect an exception several pages away. Repeating whole paragraphs consumes context without adding evidence.

Measure whether overlap improves relevant boundary cases and count duplicate or highly similar results. A variant can retrieve better while creating repetitive, unnecessarily expensive answers if context assembly is left unchanged.

A table row often depends on column headings, units and scope. Stored as an isolated sequence of numbers, it loses meaning. Large tables may need row groups with the relevant headings repeated.

An aggregate question such as which three products have the highest value differs from finding one named entry. Reliable aggregation may require a structured query rather than retrieval of a few text chunks. Choose the appropriate information tool before tuning passage length.

Inspect footnotes and tables continued across pages. A footnote can qualify an entire column. The PDF guide covers preparation before chunking decides which units should travel together.

An application can retrieve small chunks and then load their parent sections. This separates precise discovery from sufficient answer context. It requires stable relationships between children and source sections.

Expansion must not bypass authorization. A larger section containing differently restricted material needs appropriate filtering too. Version and effective period must also match.

Bound the expansion. One relevant hit should not automatically load a hundred-page manual. Define which neighboring sections are needed and evaluate their contribution on questions involving cross-section rules.

Sample and questions. Create a representative source sample and questions with known required passages. Include exact product codes, paraphrases, rules with exceptions, tables and questions crossing section boundaries.

Three variants. Compare a fixed-length baseline, a structure-aware option and an option using controlled parent expansion. Initially hold the model, retrieval method, access rules and scoring constant. Actual sizes are experiment settings chosen for the documents, not industry guarantees.

Record the following for each question in a chunking experiment:

• whether the necessary evidence was found
• whether conditions reached context
• whether the answer was correct

Add index size, processing effort, tokens supplied and response time. This exposes the cost of any quality improvement.

Unanswerable questions. Include questions that require no answer because the source lacks the information. A chunking change that makes unsupported answers sound more confident is not a retrieval improvement. Track abstention and clarification behavior separately from successful evidence retrieval.

Record the exact document revisions used for each variant. If an owner corrects a policy during the experiment, an apparent chunking improvement may actually come from better source content. Either rerun the affected cases across all variants or explicitly separate the changed-source evaluation.

The same applies to permissions. Different role filters can change which passages are eligible before chunking quality is assessed. Keep identities and access state consistent when comparing methods, then run a separate permission regression check after selecting the preferred configuration. This preserves a clear explanation of both quality gains and the access boundary.

An overall average can hide a problematic subgroup. Separate short FAQs, long procedures, tables and multilingual sources. A method that handles prose well may lose important headings on short lists.

Test German compound words, English terminology and product-code spelling where they occur in real work. Translation or normalization must not alter an exact identifier. A semantically similar model name can refer to the wrong machine.

Have specialists review cases where retrieval looks correct but the answer suggests an incorrect action. These often involve a missing condition that a main-sentence comparison overlooks.

A new split changes identities, results and potentially citations. Plan removal of old chunks and invalidation of stored answers. Adding a new import without cleanup can leave competing representations of the same source.

Record the processing version so a failure can be attributed to source changes or a different split. A separate evaluation index can help with larger changes when its cost is justified.

After switching, rerun deletion and permission cases. New boundaries must not produce passages without required metadata. Better retrieval metrics do not compensate for losing access control.

Explain which questions the selected method improves and which limitations remain. “Structural boundaries preserve exceptions in our service manuals; large tables use a separate path” is a meaningful decision. A token count alone does not explain the benefit.

Keep a simple comparison variant and the evaluation set. New document types can then trigger targeted reassessment. The RAG acceptance plan links retrieval to business correctness, and the hybrid-search guide covers the next selection stage.

Before finalizing the choice, inspect a handful of actual chunks without their original documents. Can a reviewer identify what each statement refers to, when it applies and where it came from? If not, the unit is probably incomplete for answering even if its vector retrieves successfully.

Repeat that inspection for the largest and smallest objects rather than only typical examples. Outliers often expose hidden failure modes: a single oversized table, a one-line exception or an appendix without its parent title. Resolve those cases explicitly instead of expecting the average chunk size to describe the entire collection. Document how such outliers are handled so that a later change to the splitter does not recreate them.

These modernagents guides cover related decisions about AI knowledge assistants:

• AI knowledge assistants: sources, access and answers
• Prepare PDFs, scans and tables for RAG
• Improve RAG retrieval with hybrid search and reranking
• Test a RAG knowledge assistant: sources, access and answers

For a workflow review, start with the task, systems involved and typical exceptions. Discuss it with modernagents.