Evals: how to actually know whether your AI system works

Ask a team how they know their AI feature works and you'll usually hear a version of: "we tried a bunch of examples and it seemed good."

That answer is fine for a prototype and fatal for anything in production, because it means you cannot tell the difference between a prompt change that helped and one that helped on the four examples you happened to check while quietly breaking a fifth case you never thought about.

Evals are the fix. They're also the single most reliable predictor we've seen of whether an AI project ships.

Start with 20 examples, not 2,000

The most common reason teams don't have evals is that building them sounds like a quarter of work. It isn't.

Start with 20 examples covering your real distribution, including the ugly parts:

• The five most common cases (the boring majority of your traffic)
• Three genuine edge cases you know about
• Two cases where the correct answer is "I don't know" or "escalate"
• Two adversarial or malformed inputs
• Anything a human has already complained about

Twenty examples will catch the majority of prompt regressions. Getting from twenty to two hundred is a background task you feed from production failures forever.

Source them from reality. Sampled production traffic beats anything you invent, because your imagination is biased toward the cases you already designed for. Every failure that reaches a human becomes a permanent eval case — that single habit compounds harder than anything else on this list.

Pick the cheapest metric that can fail

Not everything needs a language model to grade it. Work down this list and stop at the first level that can express what you care about:

Level 1 — deterministic checks. Free, instant, and unambiguous. Is the JSON valid? Does it match the schema? Is the enum value in the allowed set? Is the cited order ID one that actually exists? Did it call the tool you expected with the arguments you expected?

A surprising share of real failures are caught here. Run these on every case.

Level 2 — reference-based scoring. When there's a known-correct answer: exact match, F1 over extracted fields, or semantic similarity for freer text. Perfect for classification, extraction, and routing.

Level 3 — LLM-as-judge. For anything genuinely subjective — tone, helpfulness, faithfulness to a source. Powerful and easy to get wrong; see below.

Level 4 — human review. The ground truth, and expensive. Reserve it for calibrating your judge and for periodic spot audits, not for the inner loop.

Making LLM-as-judge trustworthy

A model grading another model's output works well, if you treat the judge as a system that itself needs evaluating.

Score one dimension at a time. "Rate this response 1–10" produces noise. "Is every factual claim in this response supported by the provided context? Yes/No" produces signal. Run several narrow judges instead of one broad one.

Give the judge a rubric with concrete criteria, not adjectives. Not "is this a good answer" but "does the answer state a specific delivery date, cite an order ID, and avoid promising a refund." Independently gradeable criteria are the whole game.

Prefer binary or three-point scales. Judges are unreliable at fine-grained numeric distinctions and reliable at pass/fail. A 1–10 scale mostly measures the judge's mood.

Watch for position bias. In pairwise comparisons, judges favour whichever response came first. Run both orderings and average, or you'll ship an improvement that is really just an ordering artefact.

Calibrate against humans. Have a person label 50 cases the judge has scored. If agreement is below ~80%, fix the rubric before trusting a single number it produces. Re-check after any judge model change.

Don't let the judge see who wrote what. Strip identifying formatting. Models show measurable preference for text in their own style.

What to measure, by system type

System	Primary metrics
Classification / routing	Accuracy, per-class recall, confusion matrix
Extraction	Field-level F1, schema validity rate
RAG	Recall@k, faithfulness, answer relevance, citation validity
Agents	Task completion rate, tool-call correctness, steps to completion, escalation precision
Generation	Faithfulness, instruction adherence, format compliance, safety

For agents specifically, task completion is not enough on its own. An agent that completes the task in 40 tool calls costs several times what one that does it in 6 costs, and is far more likely to go wrong somewhere in the middle. Track efficiency alongside success.

And track the escalation rate in both directions. An agent that escalates 40% of tickets isn't saving anything; one that escalates 0% is quietly making decisions it shouldn't. Precision and recall on the decision to escalate is a first-class metric, not an afterthought.

Run them where they change decisions

Evals that live in a notebook someone runs occasionally are decoration. Wire them into the places where behaviour changes:

• On every prompt change. Prompts are code. They belong in version control and they get a test run.
• On every model change. Including minor version bumps. Model behaviour shifts between versions in ways that don't announce themselves — instruction-following gets more literal, verbosity recalibrates, tool-calling eagerness moves. A prompt tuned around one model's reluctance becomes an over-trigger on the next.
• On a schedule against production samples. Your input distribution drifts even when your code doesn't. Sample real traffic weekly and score it.
• Before and after every retrieval change. Chunking, index rebuilds, embedding upgrades — all of them move recall.

Set a threshold and treat crossing it as a failed build, not a note in a dashboard.

The regression trap

The pattern that burns teams: a change improves the average and breaks a specific case that mattered enormously.

Average scores hide this by design. Two defences:

Keep a golden set that must never regress. A small number of cases — the ones where being wrong is genuinely expensive — where any drop blocks the release regardless of what the aggregate did.

Diff per case, not just in aggregate. After every run, list the cases whose result changed. Improvements and regressions both. A change with net-zero score movement that flipped eleven cases in each direction is not a neutral change; it's a completely different system with the same average.

What good looks like

A team with functioning evals can answer these in under a minute:

• What's our task success rate this week, and last week?
• Which failure mode is most common right now?
• If we switched models tomorrow, would quality go up or down?
• Which of the last ten prompt changes actually helped?

A team without them answers all four with "it feels about the same." That's the difference between a system you're improving and a system you're hoping about.

Where to start on Monday

1. Pull 20 real inputs from your logs, including three that went wrong.
2. Write the correct output for each. Yes, by hand.
3. Add the deterministic checks — schema, format, required fields. An hour's work.
4. Run it. Note the score. That number is now your baseline.
5. Add every new production failure to the set, forever.

Step five is the one that matters. Everything else is setup.

---

Every agent we ship comes with its eval set, its threshold, and a 30-day review against it — and if it misses, we fix it at our cost. Book an audit call.