Frontend Beauty
ESC

Type to search the knowledge base.

Agent Memory

Agents forget when the context window ends. Working memory, session summaries, and long-term stores are three different systems. Do not mix them.

intermediate5 min read
  • agents
  • memory
  • context

“It remembered my dog’s name” is usually the last 20 messages, not a soul. Agent memory is a set of stores you built: the live context window, a rolling summary of this thread, and a long-term blob you retrieve on purpose. When the window ends, the model knows nothing you did not send back.

Mixing those three into one “memory layer” is how a profile note about a dog collides with a secret from another tenant and a summary that drifted two weeks ago.

Key takeaways

  • Working memory is the context window. It is finite and you pack it.
  • Session summaries compress a thread. They drift. Version them.
  • Long-term memory is a store you retrieve, with a write policy and a forget path.
  • Do not dump secrets, tokens, or other people’s data into memory blobs.
  • “It remembered” almost always means “you resent the tokens.”

Agent memory is three systems, not one

Store Lifetime What goes in Failure mode
Working (scratchpad) This request / this window Live messages, tool results, the current plan Overflow, lost-in-the-middle
Session summary This conversation id Compressed decisions and open questions Drift, invented facts
Long-term User or org, across sessions Explicit preferences, durable facts you chose to save Stale profile, cross-tenant leak

Working memory is context engineering plus the window budget in tokens and context windows. You already do a version of this in a chat UI: send N turns, drop the rest. See conversation state in chat UIs.

The model has no other place to put a thought. A “scratchpad” tool that writes to a string you reattach is still working memory. It is a named buffer, not magic.

type ChatTurn = {
  role: 'user' | 'assistant' | 'tool';
  content: string;
  tokens: number;
};

/** Budgeting only. Count with a real tokenizer before you bill or truncate. */
function estimateTokens(s: string): number {
  return Math.ceil(s.length / 4);
}

function packWorkingMemory(opts: {
  system: string;
  summary?: string;
  profile?: string;
  turns: ChatTurn[];
  budgetTokens: number;
}): ChatTurn[] {
  const reserved =
    estimateTokens(opts.system) +
    estimateTokens(opts.summary ?? '') +
    estimateTokens(opts.profile ?? '');
  let used = reserved;
  const kept: ChatTurn[] = [];
  for (let i = opts.turns.length - 1; i >= 0; i--) {
    const turn = opts.turns[i];
    if (used + turn.tokens > opts.budgetTokens) break;
    kept.push(turn);
    used += turn.tokens;
  }
  return kept.reverse();
}

When kept starts dropping user turns, you need a summary or you are pretending the early constraint still exists.

Scratchpad vs rolling summary vs retrieved profile

Scratchpad. Ephemeral notes for this run: “issue 412 looks like a bug, waiting on get_order.” Store it in the message list or a tool result. Delete it when the loop stops. Do not write it to the user profile.

Rolling summary. After N turns, ask a cheap model to compress this conversation id into a short brief: goal, decisions, open items, ids that matter. Keep the last raw turns plus the brief. Rules:

  • Pin ids (ord_08k2) verbatim. Summaries love to drop them.
  • Mark the summary as untrusted relative to the raw turns. If they disagree, the raw turn wins.
  • Version it (summaryRev: 4). A bad compression should be roll-backable.
  • Do not summarize PII you would not show in the UI.

Long-term profile. A row you retrieve at the start of a new session: “prefers compact answers,” “dog is named Mango,” “billing timezone is America/Chicago.” This is a database with a write policy, not a second chat log.

type MemoryWrite = {
  userId: string;
  kind: 'preference' | 'durable_fact';
  text: string;
  source: 'user_explicit' | 'user_confirmed' | 'inferred';
};

const FORBIDDEN = /api[_-]?key|password|secret|bearer\s+[a-z0-9._-]+/i;

export async function maybeWriteMemory(
  userId: string,
  candidate: Omit<MemoryWrite, 'userId'>,
  opts: { userConfirmed: boolean },
): Promise<'saved' | 'skipped'> {
  if (FORBIDDEN.test(candidate.text)) return 'skipped';
  if (candidate.source === 'inferred' && !opts.userConfirmed) return 'skipped';
  if (candidate.text.length > 280) return 'skipped';
  await db.memories.insert({ userId, ...candidate, at: new Date().toISOString() });
  return 'saved';
}

Inferred memories (“the user seems angry”) are how you store a bad day forever. Prefer explicit: a UI that says “remember this?” the way a password manager asks to save. Forget is a DELETE by id, exposed to the user.

Long-term lookup is retrieval, often the same embedding path as RAG, with a tenant filter. It is not a semantic cache. A cache returns a previous answer. Memory returns facts to pack. Mixing them serves yesterday’s weather as a personality trait.

Write policies, and what never belongs in a blob

Decide what is allowed to be saved before you train the model to “remember everything useful.”

  1. User-explicit always wins: “remember that I use pnpm.”
  2. User-confirmed for inferences you surfaced in the UI.
  3. Never inferred-only for health, finance, or legal facts.
  4. Never secrets. Tokens, session cookies, dumped .env, tool results that included a key. Your working window may have seen them; the profile must not.
  5. Never another tenant’s data. userId on the row, filter on read, tests that a forbidden note cannot appear in someone else’s pack. Same instinct as permission-aware retrieval.

Provider “memory” products (where they exist) are still stores with policies. OpenAI’s conversation-state and compaction APIs persist items you send; they do not invent a soul. Anthropic’s long-context tips are about packing documents in the window, which is working memory. Read those as packing advice, not as a third-party hippocampus.

TTL is allowed. “Prefers dark mode” can live a year. “On hold with support about incident 8841” should die when the ticket closes. A memory without a TTL is a compliance review.

Frontend caching rules still apply: do not cache personalized memory in a shared CDN. See frontend caching strategies for the “what is private” half of that instinct.

Interview answer (out loud)

“Agent memory is three systems. Working memory is the context window I pack. A session summary compresses this thread and can drift, so I keep recent raw turns. Long-term memory is a per-user store I retrieve with a write policy: explicit or confirmed facts, no secrets, a forget path, tenant filters. If the bot knew the dog’s name, I resent those tokens. I would not collapse the three into one blob.”

Further reading (primary sources)

Related guides