Chunking Is the Whole Game: A Retrieval-Quality Chunking Architecture

When a RAG system gives a vague or wrong answer, the instinct is to blame the model or the prompt. Most of the time the real culprit is upstream and invisible: chunking. Retrieval can only return what your chunking carved out, and if the right answer was split across two chunks — or buried in a chunk full of unrelated text — no amount of prompt tuning will recover it. Chunking is not preprocessing. It is the retrieval architecture.

The chunk is the unit of truth

Your vector index does not store documents; it stores chunks. A chunk is the smallest thing retrieval can return and the largest thing it can return as a single unit. Get it wrong and you fail in one of two directions. Too small, and a self-contained answer gets sliced across chunk boundaries, so retrieval surfaces a fragment that is true but incomplete. Too large, and each chunk dilutes its own embedding with unrelated content, so the vector means everything and matches nothing sharply.

Three strategies, escalating in cost and quality

Fixed-size

Split every N characters with some overlap. Trivial to implement and almost always wrong at the edges, because it cheerfully cuts through the middle of a sentence, a code block, or a numbered list. It is the baseline, and the source of most "my RAG is bad" complaints.

Recursive / structural

Split along the document's own structure first — headings, paragraphs, list items — and only fall back to size limits within those boundaries. For Markdown documentation this is the high-leverage default: a heading and its body stay together because the document already told you they belong together.

Semantic

Split where the meaning shifts. Embed sentences, measure the distance between neighbours, and cut where the topic changes. It produces the most coherent chunks and costs the most to build. For a stable knowledge base embedded once, that one-time cost is usually worth it.

// structure first, size only as a fallback within a section
function chunkMarkdown(md: string, max = 1200) {
  const sections = splitByHeadings(md);        // keep heading + body together
  return sections.flatMap((s) =>
    s.length <= max ? [s] : splitByParagraph(s, max)
  );
}

// every chunk carries its heading as metadata — for attribution AND recall

Carry the heading into the chunk

One technique pays for itself everywhere: prepend each chunk with the heading path it came from. "Setup > OBS > Connecting" prefixed onto the chunk text does two things at once — it sharpens the embedding with topical context, lifting recall, and it gives you a ready- made citation for the grounding contract. The same metadata serves retrieval quality and answer attribution.

Measure it, do not guess

Chunking strategy is the highest-leverage knob in a RAG system, so treat it like one: build a small evaluation set of real questions with their correct source passages, then measure recall — did the right chunk make the top-k? — for each strategy. The heatmap in the diagram is exactly that experiment. You will usually find that fixing chunking moves the quality needle further than any model upgrade.

"My retrieval is bad" is almost never a model problem. It is a chunking problem wearing a model costume.

Good chunks make the grounding contract enforceable and single-model embeddings worth their precision. The chunk is where retrieval quality is won or lost.