Frontend Beauty
ESC

Type to search the knowledge base.

Harness Engineering for Coding Agents

A harness is the loop, tools, tests, and permissions around a coding model. The model is the easy part. The harness is why some repos get useful diffs.

advanced5 min read
  • agents
  • coding-agents
  • harness

Drop the same model into a chat box and you get an essay about your bug. Drop it into a loop that can list files, patch, and run tests, and you get a diff. That surrounding loop is the agent harness: tools, permissions, repo maps, a test runner, a sandbox, and a stop rule. The model is a sampler you can swap. The harness is the product.

Claude Code and Codex-shaped tools are harnesses with opinions (repo-aware file tools, permission prompts, CI). Shipping your own is the same job at a smaller scale: make the next action cheap, observable, and hard to do as root.

Key takeaways

  • An agent harness is the loop plus tools, tests, and permissions around a coding model.
  • Give the model a repo map and file tools. Do not paste the tree into the prompt.
  • Sandbox the shell. Allowlist commands and paths. Prod credentials do not belong in that process.
  • Retry on red tests with a budget. Stop if the same failure repeats.
  • The diff and the test log are the artifacts. The model’s “it works” sentence is not.

What an agent harness contains

Think of a CI job that can think. The model proposes the next tool call. Your process runs it under policy. Observations go back. You cap steps. That is the whole architecture. If you skip the test runner, you built autocomplete with extra tokens. If you skip the allowlist, you built a junior with root.

Piece Job
Repo map Paths, exports, test command. Cheap orientation.
File tools Read, search, patch. Not “write entire file from memory.”
Shell Sandboxed, allowlisted, with timeouts.
Test runner The oracle. Red/green is data, not a vibe.
Permissions Path allowlist, no .env, no git push --force.
Stop rules Max steps, repeated test failure, human reject.

Tool schema design matters here more than prompt poetry. A tool named run with a free-string command is how you get incident review. A tool named run_tests with no args is how you get a patch.

The loop you ship

type AgentEvent =
  | { type: 'read'; path: string }
  | { type: 'search'; query: string }
  | { type: 'apply_patch'; path: string; diff: string }
  | { type: 'run_tests' }
  | { type: 'stop'; reason: string };

const ALLOWED_ROOTS = ['src/', 'tests/', 'package.json'];
const MAX_STEPS = 20;
const MAX_TEST_ROUNDS = 3;

export async function runHarness(goal: string, repo: RepoFs): Promise<string> {
  const map = await repo.summarize({ maxFiles: 80 });
  let trace = `Goal: ${goal}\nRepo map:\n${map}\n`;
  let testFails = 0;
  let lastFail = '';

  for (let step = 0; step < MAX_STEPS; step++) {
    const event = await model.nextEvent(trace);
    if (event.type === 'stop') return event.reason;

    if (event.type === 'read' || event.type === 'apply_patch') {
      assertAllowed(event.path, ALLOWED_ROOTS);
    }

    if (event.type === 'apply_patch') {
      await repo.applyPatch(event.path, event.diff);
      trace += `patched ${event.path}\n`;
      continue;
    }

    if (event.type === 'run_tests') {
      const result = await repo.exec('npm test', { timeoutMs: 60_000, cwd: repo.root });
      trace += `tests exit=${result.code}\n${tail(result.stdout, 4_000)}\n`;
      if (result.code === 0) return 'tests green';
      testFails += 1;
      if (result.stdout === lastFail || testFails >= MAX_TEST_ROUNDS) {
        return 'stopped: tests still red';
      }
      lastFail = result.stdout;
      continue;
    }

    const observation = await repo.dispatch(event);
    trace += `obs ${observation}\n`;
  }
  return 'stopped: max steps';
}

function assertAllowed(path: string, roots: string[]) {
  const norm = path.replace(/^\.\//, '');
  if (norm.includes('..') || norm.startsWith('/')) throw new Error('path');
  if (!roots.some((r) => norm === r || norm.startsWith(r))) {
    throw new Error(`blocked path ${path}`);
  }
}

Failure modes the comments are for: path escape, infinite red-retry, stuffing 80k of test output into the next prompt. Tail the log. If you need the full log, write it to a file the model can read by offset.

Sandbox and permissions

The shell is the blast radius. Run it as a user who cannot reach production, with a network allowlist (package registry, nothing else) or no network. Do not mount cloud keys. Do not let apply_patch touch .env, id_rsa, or auth.json.

Frontend-specific: npm test is not enough if the agent can delete the snapshot folder and call that green. Pin the test command in the harness. The model should not choose npm test -- -u unless a human said so. Put that in code, not in a system-prompt plea.

CI is part of the same product. A patch that only “looks good” in the agent’s sandbox still goes through eval gates. The harness is the inner loop. CI is the outer loop.

Retry on red, then stop

A useful harness treats a failing test like a compiler: feed the error, try again, bound it.

  1. Run the targeted test, not the universe, after a small patch.
  2. If the error text is identical twice, stop. The model is looping.
  3. If the agent deletes tests, fail the run in the harness, not in code review two hours later. Diff the tests/ tree before you trust green.
  4. Show the user the patch and the last test log. That pair is the PR.

Products like Claude Code ask before destructive edits. Copy that instinct even if you never use their CLI: file writes are a permissioned tool, not a default. Coding agents are what the user sees. The harness is why the diff is small.

Interview answer (out loud)

“An agent harness is the product around a coding model: repo map, file and test tools, a sandbox, path allowlists, and a max-step loop that retries on red tests then stops. I would not paste the repo into the prompt. I would not give a free-form shell. Green tests after deleting tests is a harness bug. The model is swappable. The loop is not.”

Further reading (primary sources)

Related guides