Frontend Beauty
ESC

Type to search the knowledge base.

AI Engineering Interview Questions

Questions interviewers actually use for AI engineering: RAG failure modes, tool auth, eval design, and cost. Short answers you can say out loud.

intermediate6 min read
  • interview
  • ai-engineering

“Explain RAG” is a filter. It tells them you watched a talk. AI engineering interview questions that decide the offer sound like: walk me through a bad citation in production, tell me who is allowed to call refund_order, show me the eval that would have caught last week’s prompt tweak.

You already interview well on HTTP, types, and “where does the key live.” This round adds a sampler that lies fluently. The job is still software: auth, budgets, tests, and a UI that can abort.

Key takeaways

  • Name the layer that broke (retrieval, packing, sampling, tools, evals). Vendor names are not a design.
  • Tool JSON is a proposal. Authorization and idempotency stay in your code.
  • Evals start as assertions on structured fields, not as a 1-5 vibe score.
  • Cost and latency are product requirements. Max tokens, cache, and max steps belong in the answer.
  • Say what you would refuse to build: unbounded loops, client-side keys, retrieval without ACLs.

How to talk, not recite

Out loud, use this shape: definition, failure, control. “RAG is search then generate. If search misses, the model invents. I refuse when the index returns nothing.” That is an engineer. A glossary dump is not.

Keep a tiny typed sketch in your head. Interviewers calm down when they see you could ship the route.

type ToolCall = { name: 'refund_order'; args: { orderId: string; amountCents: number } };

async function executeTool(userId: string, call: ToolCall) {
  const order = await db.orders.findOwned(userId, call.args.orderId);
  if (!order) throw new Error('not_found');
  if (call.args.amountCents > order.paidCents) throw new Error('amount');
  return refundProvider.credit({ orderId: order.id, amountCents: call.args.amountCents, idempotencyKey: order.id });
}

Deeper RAG and agent loops have their own drills: RAG interview questions and agent system design.

AI engineering interview questions with spoken answers

1. What is AI engineering, if you are not training models?

Say this: “Shipping software that calls LLMs: prompts as versioned specs, retrieval, tools with auth, evals, traces, and a cost ceiling. The model is a sampler with a context window. It is not a database and not an authorization layer. I would start with a typed server route, not a fine-tune.”

Common miss: Equating the job with “prompting” or with ML research. See what is AI engineering?.

2. Walk me through a bad citation in production.

Say this: “I split the failure. Did we retrieve the wrong chunk, pack the right chunk too far down, or generate a sentence the chunk does not entail? I check chunk ids in the trace, reject citation ids that were not in the prompt, and refuse when retrieval is empty. A footnote is UI. Entailment is the claim.”

Common miss: “Hallucination, so we need a smarter model.” That skips the index.

3. How do you authorize a tool call?

Say this: “The model emits structured args. My process loads the user from the session, not from the JSON, then checks ownership and limits before any side effect. Irreversible money movement waits on a human receipt bound to that exact payload. I never give the model a raw shell.”

Common miss: Validating Zod types and calling that security. Types are not ACLs. Function calling.

4. Design an eval for a refund assistant.

Say this: “A golden set of tickets with expected orderId, amountCents, and allow/deny. Deterministic checks first: schema, id match, ‘did we call the tool.’ A judge only for tone. Gate the set in CI on prompt changes. I would rather have 40 ugly real tickets than 12 imagined ones.”

Common miss: A single 1-5 “quality” score that hides a broken tool policy. LLM evals.

5. The bill jumped overnight. What do you inspect?

Say this: “Trace: token counts per span, retry loops, cache hit rate, retrieval dump size, agent step count. I look for a missing max-step cap, a system prompt that doubled, or a client that retries complete in a tight loop. I would not start by renegotiating the contract.”

Common miss: Jumping to “use a smaller model” before counting retries and context. Cost.

6. Prompt injection versus XSS. Same or not?

Say this: “Same family: untrusted text tries to become instructions. XSS executes in the browser origin. Injection executes in the model’s instruction channel and gets dangerous when tools have blast radius. Delimiters are not a fix. Authorization in code is. I treat retrieved HTML like innerHTML.”

Common miss: “We told the model to ignore untrusted text.” Prompt injection, XSS.

7. When do you not use RAG?

Say this: “When the fact is in a database I can query, when I need an aggregate, when the workflow is deterministic, or when freshness is ‘this morning.’ I use a tool or I write code. RAG is for unstructured docs I cannot afford to stuff in the prompt every time.”

Common miss: Indexing the SQL catalog instead of running SQL.

8. Temperature 0 means deterministic, right?

Say this: “No. It concentrates the sampler. Providers, batching, and ties still move. For extraction I pin temperature low and use structured outputs. For brainstorming I allow variation on purpose. I do not A/B a prompt at two temperatures and call that science.”

Common miss: Treating temperature as a creativity slider with a guarantee.

9. How does the chat UI fail closed?

Say this: “Keys on the server, AbortController on Stop, disable Send while streaming, per-user quota on the route. Partial JSON is not parsed until the object closes. I would rather show a stop than a hung composer.”

Common miss: Putting the provider SDK in a client component with a public key. Client vs server.

10. What belongs in a trace?

Say this: “Prompt version, model id, retrieved chunk ids, tool names and redacted args, token counts, TTFT, step index, user id hash. Enough to replay the decision without dumping raw PII into Slack. If I cannot answer ‘what did we send,’ I cannot debug.”

Common miss: Logging the full messages array forever. PII.

11. Fine-tune or not?

Say this: “After a prompt, tools, and retrieval cannot move a measured eval. Fine-tuning is for stubborn format and tone, not for facts that change this morning. I would not fine-tune on 40 Slack screenshots.”

Common miss: Fine-tuning to “teach the handbook.” That is RAG’s job.

Interview answer (out loud)

“I treat the model as a typed HTTP dependency with bad guarantees. I can walk RAG failures by layer, put auth on tools, gate a small golden set, and explain a bill from traces. I keep keys on the server and I cap loops. If we have time I will whiteboard the refund bot, including the approval UI.”

Further reading (primary sources)

Related guides