Frontend Beauty
ESC

Type to search the knowledge base.

Building a Docs RAG Chatbot

Build a docs chatbot on MDX: ingest pages, embed, retrieve, then stream a cited answer. The shape you can ship on a content site.

intermediate6 min read
  • rag
  • tutorial
  • chatbot

This site is MDX under src/content. A docs chatbot that cannot cite /learn/javascript/event-loop is a novelty widget: fluent, ungrounded, and one hallucination away from teaching the wrong event-loop order. The product is not “ChatGPT over our blog.” It is search, a packed prompt, a streamed answer, and links the reader can open.

You already have the corpus. frontend.beauty pages carry title, description, category, tags, and a slug that maps to /learn/<category>/<slug> (or a handful of other prefixes like /system-design/). That metadata is the citation UI.

Key takeaways

  • Ingest real files you can rebuild. For this site, that is the MDX tree, not a scraped HTML dump.
  • Retrieve first. The model only sees the chunks you stuff into the prompt.
  • Cite with site paths the reader already trusts. Drop answers when retrieval is empty.
  • Stream tokens to a React chat UI. Do not wait for the full essay.
  • Skip is honest: no auth, no evals, no hybrid search in this walkthrough. Those are why production still hurts.

What you are actually shipping

Five jobs, in order. If one is fake, the chatbot is fake.

  1. Load each MDX file, strip frontmatter into metadata, keep the body.
  2. Chunk on headings so a section about microtasks does not glue to a later figure caption.
  3. Embed the chunks with one model. Store { id, path, heading, text, embedding }.
  4. Retrieve top-k for the user question. Pack them into a budgeted prompt.
  5. Generate a streamed answer that may only use those chunks, with citations.

The RAG mental model is this list. The rest of the page is the frontend.beauty-shaped version.

Ingest the MDX corpus

Walk src/content/**/*.mdx. Parse frontmatter. Drop draft: true pages if you do not want them searchable. Keep category and slug so you can rebuild the public URL.

import { readdir, readFile } from 'node:fs/promises';
import path from 'node:path';
import matter from 'gray-matter';

async function walkMdx(dir: string): Promise<string[]> {
  const entries = await readdir(dir, { withFileTypes: true });
  const files = await Promise.all(
    entries.map(async (entry) => {
      const full = path.join(dir, entry.name);
      if (entry.isDirectory()) return walkMdx(full);
      return entry.name.endsWith('.mdx') ? [full] : [];
    }),
  );
  return files.flat();
}

type DocChunk = {
  id: string;
  url: string;
  title: string;
  heading: string;
  text: string;
};

function urlFor(category: string, slug: string): string {
  const roots = new Set([
    'dsa',
    'machine-coding',
    'system-design',
    'interview',
    'roadmaps',
    'experiences',
  ]);
  if (roots.has(category)) return `/${category}/${slug}`;
  return `/learn/${category}/${slug}`;
}

function splitOnHeadings(body: string): { heading: string; text: string }[] {
  const parts = body.split(/\n(?=##\s)/);
  return parts
    .map((part) => {
      const match = part.match(/^##\s+(.+)\n([\s\S]*)$/);
      if (!match) return { heading: '', text: part.trim() };
      return { heading: match[1].trim(), text: match[2].trim() };
    })
    .filter((p) => p.text.length > 0);
}

export async function loadDocs(root: string): Promise<DocChunk[]> {
  const files = await walkMdx(root);
  const chunks: DocChunk[] = [];

  for (const file of files) {
    const raw = await readFile(file, 'utf8');
    const { data, content } = matter(raw);
    if (data.draft) continue;

    const slug = path.basename(file, '.mdx');
    const category = String(data.category ?? path.basename(path.dirname(file)));
    const url = urlFor(category, slug);
    const title = String(data.title ?? slug);

    for (const [i, section] of splitOnHeadings(content).entries()) {
      chunks.push({
        id: `${url}#${i}`,
        url,
        title,
        heading: section.heading,
        text: section.text.slice(0, 4000),
      });
    }
  }
  return chunks;
}

Then embed offline. Chat request time is the wrong place to re-embed the whole tree.

import { embedMany } from 'ai';

export async function indexChunks(chunks: DocChunk[]) {
  const { embeddings } = await embedMany({
    model: 'openai/text-embedding-3-small',
    values: chunks.map((c) => `${c.title} ${c.heading}\n${c.text}`),
  });
  return chunks.map((c, i) => ({ ...c, embedding: embeddings[i] }));
}

A few hundred pages fit in memory with cosine search. That is enough to ship a site assistant. When it is not, you graduate to a vector database. Do not start there.

Failure mode: splitting on a 500-character timer. You will cut a table in the event loop page in half and retrieve “microtasks cut the line” without the rule that explains it. Heading-aware chunks are the default for MDX.

Retrieve, pack, refuse

At request time, embed the question with the same model, take top-k, and pack until a token budget. If the best score is noise, refuse. The model should not “helpfully” answer from pretraining about this site’s API.

import { embed, streamText } from 'ai';

function cosine(a: number[], b: number[]) {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    na += a[i] * a[i];
    nb += b[i] * b[i];
  }
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
}

export async function answerDocsQuestion(
  question: string,
  index: { chunk: DocChunk; embedding: number[] }[],
) {
  const { embedding } = await embed({
    model: 'openai/text-embedding-3-small',
    value: question,
  });

  const ranked = index
    .map((row) => ({ row, score: cosine(embedding, row.embedding) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, 8);

  // Threshold is corpus-specific. Tune it. Do not ship 0.
  const hits = ranked.filter((r) => r.score > 0.3).slice(0, 5);
  if (hits.length === 0) {
    return { empty: true as const };
  }

  const context = hits
    .map(
      (h, i) =>
        `[${i + 1}] ${h.row.url} ${h.row.title} / ${h.row.heading}\n${h.row.text}`,
    )
    .join('\n\n');

  const result = streamText({
    model: 'openai/gpt-4.1-mini',
    system: `Answer only from the numbered sources. Cite like [1]. If the sources do not contain the answer, say you do not know. URLs are frontend.beauty paths.`,
    prompt: `Sources:\n${context}\n\nQuestion: ${question}`,
  });

  return { empty: false as const, result, citations: hits.map((h) => h.row) };
}

The Route Handler streams result.toUIMessageStreamResponse() and can send citation metadata as a side channel. The UI renders chips that link to /learn/javascript/event-loop, not to a vector id.

A footnote is not proof. Citations and grounding is the next layer: require quotes, verify the citation index exists, strip invented URLs.

Wire the docs chatbot to the site

Surface Implementation
Index job node scripts/index-docs.ts in CI when MDX changes
Chat route POST /api/docs-chat with the question plus optional conversation id
UI Composer + streaming bubble on a /chat page or a docs sidebar
Citations Links under the answer, using url from retrieved rows

Keep the API key on the server. This is still client vs server. The browser never talks to the embedding provider.

Rebuild the index when a page ships. Stale chunks are how the bot quotes a heading you renamed last week. That ops problem is vector index updates.

What this tutorial skips (on purpose)

  • Auth. A public docs bot can be anonymous with a tight rate limit. An internal bot cannot.
  • Evals. You need a golden set of questions like “what runs first, promise or timeout?” and a check that the citation includes event-loop.
  • Hybrid search. "ORD-18422" is a keyword problem. MDX slugs and error strings will miss in pure cosine. Add BM25 when you feel that.
  • HTML chrome. You are indexing source MDX, which is the right move. If you ever index the built site, strip nav or you will retrieve the footer.

Those skips are the difference between a weekend and a product. Ship the cited stream first. Then measure retrieval misses.

Interview answer (out loud)

“I would treat the MDX tree as the corpus. Chunk on headings, embed offline, retrieve top-k at request time, and stream an answer that may only use those chunks. Citations are site paths like /learn/javascript/event-loop, not invented URLs. If retrieval is empty I refuse. The browser hits my route. Keys stay on the server. Evals and hybrid search come after the first cited answer works.”

Further reading (primary sources)

Related guides