AI Engineering

AI engineering is shipping LLM products: prompts, retrieval, tools, evals, and cost control. How it differs from ML research and from wrapping a chat API.

beginner

Treat a large language model like a weird HTTP API: tokens in, tokens out, no memory, no guarantees. What that means for UI, retries, and types.

beginner

Tokens are the billing unit and the memory limit. How context windows fill up, why counts surprise you, and how to budget a request.

beginner

Temperature and top-p change how a model samples the next token. When to pin them down for tools and JSON, and when variation is the feature.

beginner

System, user, assistant, and tool roles are the real API. How message lists grow, how to keep a system prompt short, and where policy belongs.

beginner

An embedding is a vector that places text near similar text. How cosine similarity works in practice, and why it is not understanding.

beginner

Tokenizers split numbers, spaces, and code in ways that break length checks and cheap string hacks. Count with a tokenizer, not with words.

intermediate

A prompt is a spec: role, task, constraints, examples, output shape. How to write one you can version, test, and stop rewriting by feel.

beginner

Force the model into a schema instead of parsing its vibes. When constrained decoding helps, and when the model still fills fields with junk.

intermediate

Prompt caching reuses a stable prefix so you stop paying full price for the same system prompt and docs on every turn you send.

intermediate

Streaming is tokens over the wire so the UI can paint before the full answer exists. Backpressure, abort, and what to do with a partial JSON object.

intermediate

You can ship retrieval and agents without deriving backprop. What ML literacy actually buys you, and which math can wait.

beginner

Retrieval-augmented generation is search, then generation, with a budget. The model only sees what you stuffed into the prompt.

beginner

Chunking decides what the retriever can return. Size, overlap, headings, and why splitting on a 500-character timer wrecks tables and code.

intermediate

Nearest-neighbor search over embeddings is fast guesswork. Distance metrics, top-k, and why the nearest chunk can still be the wrong chunk.

intermediate

Keyword search still wins on SKUs, error codes, and rare names. Hybrid retrieval blends BM25 with vectors so neither failure mode owns you.

intermediate

A vector database stores embeddings plus metadata and runs ANN search. What you actually buy vs pgvector in Postgres, and the ops you will own.

intermediate

Retrieve broadly, then rerank with a cross-encoder or a small model so the prompt gets the 5 chunks that actually answer the question.

intermediate

PDFs are layout, not text. HTML is a tree. If your parser dumps reading-order garbage, retrieval will confidently fetch the footer.

intermediate

Filter by tenant, product, language, and date before you rank. Vector search without metadata is how one customer sees another customer's contract.

intermediate

Users type pronouns and slang. Query rewriting expands, decomposes, or HyDE-drafts a better search string before you hit the index.

intermediate

Grounding means the answer is entailed by retrieved text. Citations are the UI. How to require quotes, and how models still hallucinate a source.

intermediate

Split retrieval metrics from answer metrics. Hit rate, MRR, faithfulness, and why a single 1–5 "quality" score hides the broken half.

intermediate

RAG cannot fix stale facts you should query, math you should compute, or permissions you should enforce in code. Know when to skip the index.

intermediate

An agent is a model in a loop with tools and a stopping condition. Agency is a spectrum, not a personality. Most products need less of it than the demo.

beginner

The agent loop is: reason, pick a tool, run it, feed the result back, repeat until stop. Timeouts, max steps, and what "done" means.

beginner

Tool calling is the model emitting a structured call you execute. You run the function. The model never gets a shell unless you hand it one.

intermediate

A tool schema is an API the model will misuse. Names, enums, tight types, and descriptions that prevent "search" from becoming "drop table".

intermediate

MCP is a standard for exposing tools and resources to any compatible client. Why a protocol beats one-off function lists glued to one vendor SDK.

intermediate

Agents forget when the context window ends. Working memory, session summaries, and long-term stores are three different systems. Do not mix them.

intermediate

Multiple agents means multiple prompts, tools, and traces. Use them when roles truly diverge. A second agent is not a substitute for a better tool.

advanced

Side-effect tools pause for a human. How to design approval, show diffs, expire tokens, and resume the loop without replaying the refund.

intermediate

A handoff passes the conversation to another agent with a slimmer prompt and different tools. What to copy, what to drop, and how to avoid ping-pong.

advanced

ReAct interleaves thought and tools. Plan-and-execute writes the plan first. Pick based on tool cost, reversibility, and how often the world changes mid-run.

intermediate

A harness is the loop, tools, tests, and permissions around a coding model. The model is the easy part. The harness is why some repos get useful diffs.

advanced

Context engineering is selecting, compressing, and ordering what the model sees. Prompts are one input. History, tools, and retrieval are the rest.

intermediate

Million-token windows still miss the middle and still cost money. When to stuff the repo in, and when to retrieve instead of dumping.

intermediate

A subagent gets a fresh context, a narrow job, and a report back. Use it to keep the parent window small. Isolate blast radius.

advanced

A coding agent reads, edits, and runs tests in a repo. Permissions, diffs, and why "it compiles" is not the acceptance test.

intermediate

Agents loop, lie about tool results, skip confirmation, and get stuck apologizing. Catalog the failures and the circuit breakers.

intermediate

An eval is a scored dataset you run on every prompt change. Offline, sampled online, and why "we tried it and it looked good" is not a release bar.

intermediate

A judge model scores outputs against a rubric. It is biased, cheap, and useful if you calibrate it against humans and keep the rubric tight.

intermediate

A golden set is labeled inputs and acceptable outputs you own. How to collect it from production, keep it from rotting, and stop cheating on it.

intermediate

Prompt injection is untrusted text that tries to become instructions. Treat retrieved pages and user files like you treat innerHTML.

intermediate

Guardrails are checks around the model: input filters, output filters, tool allowlists, and policy engines. Put safety in code, not only in the system prompt.

intermediate

A trace is every prompt, tool call, token count, and latency span for one user request. Without it you cannot debug, eval, or explain a bill.

intermediate

Cost is tokens × price × retries, plus the tools you invoked. Cache, route, shrink context, and stop looping before you negotiate a contract.

intermediate

Users feel TTFT. Total time is TTFT plus generation. Streaming, smaller models, and prompt cache hit rate are the knobs that actually move the UI.

intermediate

Speculative decoding drafts tokens with a cheap model and verifies them with the big one. Why checking a draft can beat writing every token the hard way.

advanced

A gateway sits in front of many providers: auth, routing, fallbacks, and logging. Route simple tasks to small models and keep one SDK in the app.

intermediate

LLMOps is how an LLM feature is versioned, evaluated, deployed, and watched. Prompts are artifacts. Traces are tests waiting to happen.

intermediate

LLM APIs flake. Retry with jitter on 429/5xx, bound the wait, fall back to a smaller model, and never retry a successful side-effecting tool.

intermediate

Fine-tuning teaches format and style, not new facts you could retrieve. When the dataset is worth it, and when a better prompt plus tools wins.

advanced

RAG supplies facts at request time. Fine-tuning changes weights. Most product problems are RAG or tools. Fine-tune for stubborn format and tone.

intermediate

Images, audio, and PDFs are tokens too. How to send them, what the model actually sees, and the UI problems of slow vision calls.

intermediate

A hallucination is a fluent claim without support. Debug it as retrieval miss, prompt pressure, sampler noise, or missing refusal path.

intermediate

Fail the build when a golden set regresses. Thresholds, flake budgets, and why an LLM judge in CI needs pinning and retries of its own.

advanced

Autoregressive decoding reuses key/value tensors from earlier tokens. That is why a long prompt has a prefill cost and why cache hits feel instant.

advanced

A semantic cache returns a previous answer when a new query is close enough. Great for FAQs. Dangerous for personalized or time-sensitive asks.

intermediate

Retrieval must respect ACLs. Filter by what the user can read before ranking, and never let the model "see" a forbidden chunk to decide.

advanced

The AI SDK gives TypeScript apps streaming, tools, and UI hooks without hand-rolling SSE. Where it helps, and where you still own the server.

beginner

A React chat UI is a list, a composer, in-flight state, and abort. Render tokens without jank, and keep the input usable.

intermediate

Run Next.js tool calling in a Route Handler: stream the UI, execute tools on the server, and keep secrets off the client.

intermediate

Build a docs chatbot on MDX: ingest pages, embed, retrieve, then stream a cited answer. The shape you can ship on a content site.

intermediate

An LLM API key in the browser is public. Stream from your origin, quota per user, and treat client SDKs as an explicit product choice.

beginner

AI SDK generateObject turns a Zod schema into a typed object. Map it to provider structured outputs, and fail the request when parse fails.

intermediate

An LLM rate limit is a UX problem: gate routes per session, disable Send while streaming, and surface 429s instead of silent retries.

intermediate

Chat conversation state is messages, in-flight status, and a conversation id. Persist on the server, trim for the model, cache in the UI.

intermediate

A document ingestion pipeline extracts, cleans, chunks, embeds, upserts, and deletes. Version it so a parser bug can be rebuilt.

intermediate

Vector index updates keep retrieval honest: upserts, tombstones, and rebuilds so yesterday's policy does not outrank today's.

advanced

An agent graph workflow makes loops, branches, and human pauses explicit. Use it when a hidden while-loop is how you lose the plot.

advanced

Logs, traces, and prompts store whatever users type. How to redact LLM PII, set retention, and know when a provider trains on traffic.

intermediate

Questions interviewers actually use for AI engineering: RAG failure modes, tool auth, eval design, and cost. Short answers you can say out loud.

intermediate

RAG interview questions on chunking, hybrid search, ACLs, and evals. Practice answers that name the layer that broke, not a vendor.

intermediate

Design a support or coding agent in a loop: tools, approvals, traces, evals, and cost. The frontend surfaces matter as much as the model.

advanced

An AI engineering roadmap for frontend engineers: LLM APIs, RAG, agents, evals, then production. Checkpoints you can build, not course trophies.

beginner

Chat is realtime. Embeddings, evals, and night-time summaries can wait. Use batch inference when the user is not in the loop.

intermediate

LLM output parsing when structured outputs are unavailable: strip fences, repair JSON, re-ask with validator errors, then give up.

intermediate