Frontend Beauty
ESC

Type to search the knowledge base.

Chunking Strategies for RAG

Chunking decides what the retriever can return. Size, overlap, headings, and why splitting on a 500-character timer wrecks tables and code.

intermediate7 min read
  • rag
  • chunking
  • indexing

A 200-token chunk that cuts a table in half will retrieve half a truth. The generator never sees the missing rows. It will still answer.

RAG chunking is the indexing decision that sets the unit of retrieval. You are not splitting for pretty pagination. You are choosing what a search hit is allowed to be.

Key takeaways

  • A chunk is the smallest thing retrieve can return. Split on meaning, not a character timer.
  • Headings, tables, and code fences are boundaries. Mid-row and mid-function splits are bugs.
  • Keep heading markers in the chunk text, and store the heading path in metadata.
  • Overlap is insurance against a cut sentence. Skip it on tables and whole functions.
  • Re-chunk when the corpus shape changes. Size is a hyperparameter you eval, not a religion.

The retriever only returns chunks

The RAG mental model is search, pack, generate. Chunking sits in index. If the refund policy’s exception lives in the next paragraph, and you split between them, cosine search can return the general rule and hide the exception.

People copy chunkSize: 500 from a tutorial because it “feels” like a paragraph. Your handbook is tables. Your API docs are code. Your blog is MDX with callouts. One width does not fit those.

RAG chunking as an engineering choice

Strategy How it splits Pays off when Breaks when
Fixed size Every N characters or tokens, optional overlap Homogeneous prose Tables, lists, code, multi-column PDFs
Recursive Try big separators first (\n\n, then \n, then . , then space), keep the separator on the next piece Mixed docs, you want cheap structure A 4k-character cell still exceeds the cap
Heading-aware Split on Markdown/HTML headings, keep the path Docs, policies, this site’s MDX Pages with no headings, or heading spam
Semantic Split where embedding similarity drops Messy prose with topic shifts Short text, tables, high embed cost
Element One table, one list, one code block per chunk (or a caption plus the object) Specs, invoices, API reference Naive flatteners that already destroyed the element

Recursive plus heading path is the default that earns its keep on software docs. Semantic splitting is a later experiment, measured on your queries, not a blog’s anecdote.

Recursive split that keeps headings

Count with a tokenizer in production. The sketch below uses characters so the control flow is visible. Do not ship text.length as your only meter; see tokenizer gotchas.

A recursive splitter that calls text.split(sep) and throws the separator away is a heading stripper. After the first \n## split, later sections start with Refunds instead of ## Refunds. LangChain-style splitters keep the separator on the piece that followed it.

type Chunk = {
  text: string;
  headingPath: string[];
  start: number;
  end: number;
};

const SEPARATORS = ['\n## ', '\n### ', '\n\n', '\n', '. ', ' '];

function splitKeepSep(text: string, sep: string): string[] {
  const parts = text.split(sep);
  if (parts.length === 1) return parts;
  const out: string[] = [];
  for (let i = 0; i < parts.length; i++) {
    const piece = i === 0 ? parts[i] : sep + parts[i];
    if (piece.trim()) out.push(piece);
  }
  return out;
}

function splitRecursive(
  text: string,
  maxChars: number,
  seps = SEPARATORS,
): string[] {
  if (text.length <= maxChars) return [text];
  const sep = seps.find((s) => text.includes(s));
  if (!sep) {
    // Failure: hard-cut. Last resort, not the plan.
    const parts: string[] = [];
    for (let i = 0; i < text.length; i += maxChars) {
      parts.push(text.slice(i, i + maxChars));
    }
    return parts;
  }

  const out: string[] = [];
  for (const piece of splitKeepSep(text, sep)) {
    if (piece.length <= maxChars) out.push(piece);
    else out.push(...splitRecursive(piece, maxChars, seps.slice(1)));
  }
  return out;
}

function overlapChars(chunks: string[], n: number): string[] {
  if (n <= 0) return chunks;
  return chunks.map((c, i) => (i === 0 ? c : chunks[i - 1].slice(-n) + c));
}

Walk ATX headings first so every chunk inherits a path. Push on a deeper heading, pop (by truncating the stack) when a heading at the same or higher level appears. Recursive split runs inside a section, not across heading boundaries.

function splitMarkdown(markdown: string, maxChars: number, overlap = 0): Chunk[] {
  const lines = markdown.split('\n');
  const path: string[] = [];
  const sections: { headingPath: string[]; text: string }[] = [];
  let buf: string[] = [];

  const flush = () => {
    const text = buf.join('\n');
    if (text.trim()) sections.push({ headingPath: path.slice(), text });
    buf = [];
  };

  for (const line of lines) {
    const m = /^(#{1,3})\s+(.+?)\s*#*$/.exec(line);
    if (m) {
      flush();
      const level = m[1].length;
      path[level - 1] = m[2].trim();
      path.length = level;
      buf.push(line); // keep the heading line in the section text
      continue;
    }
    buf.push(line);
  }
  flush();

  const out: Chunk[] = [];
  let cursor = 0;
  for (const section of sections) {
    const looksAtomic = /^\s*\|.+\|\s*$/m.test(section.text) || /```/.test(section.text);
    let pieces = splitRecursive(section.text, maxChars);
    if (overlap > 0 && !looksAtomic) pieces = overlapChars(pieces, overlap);
    for (const text of pieces) {
      out.push({
        text,
        headingPath: section.headingPath,
        start: cursor,
        end: cursor + text.length,
      });
      cursor += text.length;
    }
  }
  return out;
}

headingPath: ['Refunds', 'Exceptions'] is gold at pack time. The model can see where the paragraph lived. LangChain documents the same two ideas as library code: a recursive character splitter and a markdown header splitter that copies headers into metadata.

Overlap is not free

Overlap copies the tail of chunk N onto the head of chunk N+1 so a sentence that straddles the cut still appears whole in at least one hit. overlapChars above is that suffix copy.

Costs:

  • Index size. 10-20% overlap is 10-20% more embeddings and storage.
  • Near-duplicates in top-k. You retrieve the same policy twice and crowd out a different source.
  • Packing waste. Your budget fills with repeats.

Use overlap on prose. Skip it on atomic elements (a whole table, a whole function) that you already kept intact. After retrieve, drop chunks with high Jaccard overlap before packing.

Metadata you must keep

A chunk without provenance is a quote you cannot defend.

type IndexedChunk = {
  id: string;
  text: string;
  metadata: {
    url: string;
    headingPath: string[];
    page?: number;
    tenantId: string;
    updatedAt: string; // ISO-8601
    contentType: 'prose' | 'table' | 'code' | 'list';
  };
};

url and headingPath feed citations. tenantId feeds metadata filters. updatedAt lets you prefer the 2026 policy over the 2021 PDF someone forgot to delete. contentType lets you pack tables with extra care (or refuse to split them).

PDFs and HTML need extra extraction work before this object exists. That is RAG on PDFs and HTML, not this page.

What still breaks

  • Tiny chunks. 50 tokens of “see above” retrieve as junk. Merge short leftovers into the previous sibling.
  • Giant chunks. A 4k-token “chunk” is a document. Search cannot point. Generation drowns.
  • Boilerplate. Nav, cookie banners, “share this article.” They embed as if they were content. Strip before split.
  • Code. Split on function or heading in the file, not on the 500th character in a minified blob.
  • Reindex drift. You changed the splitter and forgot to delete old vectors. The index now contains two realities.

Eval chunking like any other hyperparameter: labeled queries, retrieval hit rate, then answer faithfulness. A prettier splitter that drops recall is a loss.

Interview answer (out loud)

“Chunking sets the retrieval unit. I split on structure first: headings, then paragraphs, then sentences, and I keep tables and code blocks intact. The recursive splitter has to keep heading markers, not strip them, and every chunk stores url, heading path, tenant, and updated-at. Overlap is a small insurance policy on prose, not a default on atomic objects. I would eval size on our queries rather than copying 500 characters from a tutorial, because a cut table retrieves half a truth.”

Further reading (primary sources)

Related guides