Frontend Beauty
ESC

Type to search the knowledge base.

Chat Messages and System Prompts

System, user, assistant, and tool roles are the real API. How message lists grow, how to keep a system prompt short, and where policy belongs.

beginner6 min read
  • prompts
  • chat
  • system-prompt

People dump a novel into the system prompt, then wonder why the model ignores the last sentence. Attention is finite. A 2,400-word policy dump is not a rules engine. It is a pile of instructions competing with the user turn, the tool schemas, and three pages of retrieved HTML.

The real API is a list of messages with roles. System, user, assistant, and tool (or vendor-specific cousins) are how you tell the model who said what. If you mash everything into one string, you lose that structure, and you lose the ability to trim, cache, or test any piece on its own.

Keep the system prompt short. Put the task in the user turn. Put retrieved context in a labeled block you can drop. Put policy that must not fail in code.

Key takeaways

  • Roles are the API: system for durable instructions, user for the task, assistant for prior model text, tool for observations you actually ran.
  • A system prompt is not a junk drawer. If a sentence is per-request, it does not belong there.
  • Message lists grow until you trim them. The model does not remember a chat id unless you resend the turns.
  • Authorization, PII rules, and “never call this tool” belong in server code, not in a paragraph the model can deprioritize.

The API is a list, not a blob

Vendors disagree on field names. The idea does not. You send an ordered transcript. The model conditions on that transcript and samples the next assistant (or tool-call) message.

type ChatMessage =
  | { role: 'system'; content: string }
  | { role: 'user'; content: string }
  | { role: 'assistant'; content: string; toolCalls?: ToolCall[] }
  | { role: 'tool'; toolCallId: string; content: string };

type ToolCall = {
  id: string;
  name: string;
  arguments: unknown;
};

type ChatRequest = {
  model: string;
  messages: ChatMessage[];
  tools?: { name: string; description: string; parameters: object }[];
};

Anthropic often takes system as a top-level field instead of a message. OpenAI Chat Completions uses a system role in messages. Newer APIs add a developer role. Adapter on the server. React should see your union, not the vendor’s.

Never invent a role: 'policy' the vendor does not accept. Extra roles get dropped or 400.

What belongs in the system prompt

The system prompt is the durable contract for this product surface: who the assistant is for this route, output rules that never change, and what it must refuse. Anthropic’s own guidance puts a role in system and the task in the user turn. That split is the right default even on other vendors.

Good system prompt:

const DOCS_SYSTEM = [
  'You answer questions about Frontend Beauty docs.',
  'Use only the passages in <sources>. If they are missing, say you do not know.',
  'Cite as [n] matching the passage id. Do not invent URLs.',
  'Do not run tools other than search_docs.',
].join(' ');

Bad system prompt: the above, plus today’s date pasted by a client, plus the user’s plan tier, plus a 12-page brand book, plus “also be concise, but also be thorough, but also use humor.” Those are either per-request data or contradictions.

Content Put it in
Product role, refusal rules, citation format System (versioned)
The user question, UI locale, “explain like I know React” User turn
Retrieved chunks, file excerpts User (or a dedicated context message), labeled and droppable
Prior Q&A Assistant / user history
JSON the tool returned tool role with the matching id
“User is admin so refunds are allowed” Not a prompt. Check auth in the tool handler

If a line must be true for safety, it is not a system prompt line. The model can be talked around. Your route handler cannot, if you write the if.

Policy, task, retrieved context

Separate them in code so you can cache the prefix and test the pieces.

function buildMessages(input: {
  system: string;
  history: ChatMessage[];
  sources: { id: string; url: string; body: string }[];
  question: string;
}): ChatMessage[] {
  const sourceBlock = input.sources
    .map((s) => `<passage id="${s.id}" url="${s.url}">\n${s.body}\n</passage>`)
    .join('\n');

  return [
    { role: 'system', content: input.system },
    ...input.history,
    {
      role: 'user',
      content: [
        '<sources>',
        sourceBlock || '(none)',
        '</sources>',
        `<question>${input.question}</question>`,
      ].join('\n'),
    },
  ];
}

Labels (<sources>, <question>) are delimiters for you and for the model. They are not a security boundary. Retrieved HTML can still say “ignore the question.” That is prompt injection, which rhymes with treating innerHTML as data.

Keep input.system stable so prompt caching can hit. Do not splice the username into the first line of the system prompt on every request unless you enjoy cache misses.

How lists grow, and how they rot

Every turn you append: user, assistant, maybe tool calls and tool results. You resend the whole list next time. That is conversation state, not magic memory.

Rot looks like:

  • Tool results from ten turns ago still sitting there, eating the window.
  • A failed call’s traceback becoming “facts” the model quotes.
  • Duplicate user messages from a retry that already painted a bubble.
  • System prompt appended again as a user message “just to be sure,” so the real system text is now buried.

Trim from the head. Summarize only if you must, and store the summary as a message you control, not as something the model whispered to itself. Context engineering is this job with a budget.

On the client, the transcript is UI state. Do not keep a second, longer copy “for the model” that you never prune. One list, a serializer, a token budget.

Footguns

System prompt as a feature flag dump. enable_beta_markdown: true belongs in your code paths. The model will not reliably honor a boolean in a paragraph.

Putting secrets in messages. Users paste keys. Retrieved docs contain keys. Log redaction and a server-side deny list matter more than “please do not repeat secrets.”

Role confusion. Sending tool JSON as role: 'assistant' teaches the model that it already ran the tool. Sending your own commentary as role: 'tool' forges an observation.

Multi-system messages. Some APIs allow one. Some concatenate. Do not assume the last system message wins like CSS.

Client-authored system prompts. If the browser can set role: 'system', a user can. Fix the system string on the server.

This is the same discipline as not letting the client pick isAdmin. The composer sends content. Your server wraps it.

Out loud, in an interview

“The API is a message list with roles, not a single blob. I keep the system prompt short: role, refusals, output rules. The task and retrieved context go in the user turn, labeled so I can drop them. History is resent every call, so I trim it. Anything that must not fail, auth and tool allowlists, lives in code. A long system prompt is how you hide the sentence that mattered.”

Further reading (primary sources)

Related guides