Frontend Beauty
ESC

Type to search the knowledge base.

LLMOps Lifecycle

LLMOps is how an LLM feature is versioned, evaluated, deployed, and watched. Prompts are artifacts. Traces are tests waiting to happen.

intermediate5 min read
  • llmops
  • production
  • process

Shipping a prompt from a Slack screenshot is not a lifecycle. It is folklore. LLMOps is how an LLM feature is versioned, evaluated, deployed, and watched: the same release discipline you already use for a payment API, applied to prompts, retrieval, tools, and models. Prompts are artifacts. Traces are tests that have not been labeled yet.

If nobody can answer “what prompt is in prod?” you do not have operations. You have a group chat.

Key takeaways

  • Version prompts, tool schemas, retriever config, and model ids together. A “prompt change” that silently moved models is two changes.
  • The loop is develop → eval on a golden set → canary → observe traces → fold failures back into the set.
  • CI fails the build on regressions you defined. Dashboards do not ship software.
  • Own the feature like a service: on-call, rollback, budget. A model vendor’s status page is not your runbook.
  • Refresh gold from production or the set rots. Last quarter’s labels bless last quarter’s product.

Prompts are versioned artifacts

Treat the prompt like code, because it is a function spec with worse diffs.

type ToolSchema = { name: string; description: string; parameters: Record<string, unknown> };

type PromptArtifact = {
  id: 'support-rag-v1';
  version: string; // git sha or semver
  model: string; // pinned snapshot, not an alias
  system: string;
  tools: ToolSchema[];
  retriever: { index: string; k: number; filters: string[] };
};

const SUPPORT_RAG_SYSTEM = `Answer only from retrieved chunks. Cite chunk ids. Refuse when retrieval is empty.`;

const supportTools: ToolSchema[] = [
  {
    name: 'search_docs',
    description: 'Search the tenant-scoped docs index',
    parameters: { type: 'object', properties: { query: { type: 'string' } } },
  },
];

const registry: Record<string, PromptArtifact> = {
  'support-rag-v1.4.2': {
    id: 'support-rag-v1',
    version: '1.4.2',
    model: 'provider-snapshot-id',
    system: SUPPORT_RAG_SYSTEM, // keep the long template in git; this is the pin
    tools: supportTools,
    retriever: { index: 'docs-2026-11-01', k: 8, filters: ['tenant'] },
  },
};

function loadProd(id: PromptArtifact['id']): PromptArtifact {
  const version = process.env.PROMPT_VERSION ?? '1.4.2';
  const art = registry[`${id}.${version}`];
  if (!art) throw new Error(`unknown artifact ${id}@${version}`);
  return art;
}

Store the template in git. Render it in the server. Log prompt.version on every trace. If a teammate edits a string in a vendor playground and pastes it into prod, you skipped the lifecycle on purpose.

Model ids are part of the artifact. Aliases (“latest”, family names that float) belong in experiments, not in the production pin, until you have re-run evals.

The loop: develop, eval, canary, observe

A change (prompt wording, chunk size, tool description, model) goes through the same stations every time.

  1. Develop on a playground that is wired to the same artifact format as prod. No one-off Python in a notebook that cannot run in CI.
  2. Eval on the held-out golden dataset. Deterministic checks first, judge second. Compare to the current prod artifact, not to vibes.
  3. Gate. Eval gates in CI fail the PR when refund accuracy or groundedness drops past the budget. Flaky judge rows are quarantined, not ignored forever.
  4. Canary. Ship to a small slice of traffic or an internal tenant. Watch TTFT, tool error rate, and thumbs-down, not only the offline score.
  5. Observe. Sample traces. Label misses. Add rows. The dataset is a living fixture, not a launch artifact.
  6. Rollback. Prompt version is a deploy variable. Reverting code without reverting the prompt is a partial rollback.
Station Artifact you touch Fail closed means
Develop Branch of prompt + schema Unit tests on packers and tools
Eval Golden set + pin Score vs prod baseline
Canary Prod config flag Error / cost / TTFT budgets
Observe Traces PII-redacted payloads
Refresh New gold rows Human label, then CI

Skip canary on a model swap and you will learn about tool-calling differences from paying users.

Who owns what

LLMOps dies when “the model” is everybody’s job and nobody’s.

  • Product owns the spec: intents, refusal policy, what “good” means on the golden set.
  • Eng owns the artifact, the gateway, traces, and the CI gate.
  • On-call owns budget alarms (tokens per turn), provider 5xx, and a rollback command that they have actually run.
  • Legal / security own data flows: what may leave the VPC, retention on traces, tool allowlists.

A vendor Model Garden (catalog + deploy buttons) can host the weights. It does not label your traces or write your evals. Google’s Vertex Model Garden, OpenAI’s production guides, and similar catalogs are infrastructure. The lifecycle above is yours.

What is not LLMOps

  • A shared Google Doc named “final prompt FINAL v7.”
  • Fine-tuning as the first response to a bad demo. Get the loop working on prompts and tools first.
  • Online-only “we look at chats on Fridays.” That is support, and it does not block regressions.
  • Dashboards without owners. A Grafana panel of token spend with no page-out is decoration.

If you are a frontend team shipping a docs chatbot, you can still do a thin version of this: prompt file in the repo, a few dozen gold rows, a CI script, traces with versions, a flag to revert. That is already more lifecycle than most wrappers.

Rollback is a one-line env change only if the previous artifact is still in the registry. Delete nothing. Keep support-rag-v1.4.1 loadable while 1.4.2 is canary. If 1.4.2 burns tokens or fails tools, flip PROMPT_VERSION and you are done. If you hot-edited the file in place, you have no previous version, and you will reconstruct it from Slack. That is the folklore this page is against.

Interview answer (out loud)

“LLMOps is versioning the prompt, retriever, tools, and model as one artifact, running a golden set in CI, canarying the change, and feeding production traces back into the set. I pin model snapshots, log prompt versions on traces, and keep a rollback that does not depend on Slack. A playground paste is not a release. The vendor catalog is not the lifecycle.”

Further reading (primary sources)

Related guides