Evaluating AI Agents in Production

01 · The eval gap

Why agents fail in production (and pass every demo)

A demo is a single golden path, run once, watched by its creator. Production is thousands of unscripted tasks, ambiguous inputs, flaky tools, and users who phrase things no test ever covered. The gap between those two is not model quality — it's measurement.

Every agent failure we've seen in production traces back to an eval that didn't exist: no golden task set, no regression suite, no definition of "good enough to ship." Teams tune prompts against vibes, ship on a good demo day, and discover the failure modes from users. Evaluation isn't a phase of agent development. It is the development loop.

02 · Dimensions

What to measure: five dimensions

An agent can be fluent, fast, cheap — and wrong. Measure all five, because optimizing one in isolation creates the others' failures:

• Task success. Did the workflow complete correctly end-to-end? Binary or graded rubrics per task. This is the headline metric; everything else is diagnostic.
• Groundedness. For agents that answer from retrieved knowledge: is every claim traceable to a source? Citation fidelity is measurable — check it like the Enterprise RAG pattern demands.
• Tool-use correctness. Right tool, right arguments, right order — and no destructive calls without approval. Log every call; grade the sequences, not just outcomes.
• Latency & cost. Per-task tokens, tool calls, and wall-clock time. A 95%-accurate agent that costs 40× the alternative is a prototype, not a product.
• Safety & policy. Prompt-injection resistance, PII handling, and policy-gate compliance — measured with adversarial tasks, not assumed from good intentions.

03 · Golden tasks

Build the golden set before you tune anything

A golden set is 100–300 realistic tasks with graded outcomes, sampled from real usage — including the ambiguous, the adversarial, and the boring. Rules we've learned:

• Mine production. The best tasks come from real user requests and real failures, anonymized. Synthetic tasks test your imagination, not your users.
• Grade outcomes, not vibes. Each task needs a rubric: what counts as success, partial success, failure. If two graders disagree, the rubric is the bug.
• Version it. Golden sets rot as products change. Someone owns maintenance, and stale tasks get retired — a passing suite on dead tasks is a lullaby.
• Keep a holdout. A slice never used during development, for honest release decisions. Tuning against your only test set is self-deception with extra steps.

04 · LLM judges

LLM-as-judge, done right

Human grading doesn't scale to every CI run, so teams use model judges. They work — with discipline:

• Calibrate against humans. Measure judge-vs-human agreement on a sample before trusting the judge. Below ~85% agreement, fix the rubric or the judge prompt, not the agent.
• Know the biases. Judges favor longer answers, confident tone, and their own model family. Counter with rubrics that reward correctness over style, and spot-check with humans continuously.
• Judge with the rubric, not the answer. Give the judge the task, the rubric, and the trace — never a reference answer to pattern-match against. Reference-matching grades plagiarism, not problem-solving.
• Separate the judge from the worker. The model grading the work shouldn't be the model that did it — or at minimum, not the same run.

05 · Red-teaming

Adversarial evaluation is not optional

Agents with tool access are attack surface. Red-team before production expansion: prompt injection via tool outputs and documents, privilege-escalation attempts through tool chaining, data-exfiltration paths (can the agent be tricked into pasting sensitive data into an external call?), and resource-exhaustion loops. Treat findings like security findings — tracked, prioritized, re-tested — because that's what they are.

06 · Eval-gated delivery

CI for agents: no green suite, no ship

The operational rule: no prompt change, tool change, or model change ships without the eval suite passing. This is the same backtest-harness discipline we describe in the retail demand forecasting blueprint — the harness decides what ships, not the developer's confidence. In practice this means evals run in CI on every change, regressions block merges, and the golden set grows with every production failure (each incident becomes a regression task).

07 · Production monitoring

Watch the traces, not just the metrics

Production eval never stops. Capture full traces — plans, tool calls, intermediate states — with sampling that keeps the interesting failures. Monitor task-success proxies (user corrections, escalations, approval rejections), cost per task, and drift in input distributions. When the world changes, your golden set is suddenly testing last quarter's reality: the feedback loop from production back into evals is what keeps the system honest.

08 · Where to start

A starting checklist

• Define task-level success criteria for your top 3 agent workflows.
• Collect 100+ real tasks with graded outcomes — your golden set v1.
• Instrument full traces and per-task cost before the next tuning session.
• Set up a calibrated judge and measure its agreement with human graders.
• Make evals a CI gate: no green suite, no deploy.
• Schedule the first red-team pass before production expansion, not after.

Agents you can measure are agents you can improve. Everything else is a demo with uptime.