Evals Are the New Tests

May 24, 2026

On 25 April 2025, OpenAI shipped an update to GPT-4o. The offline evaluations looked good. The A/B tests looked good - the users who saw it liked it. Within four days it was rolled back.

The model had become a flatterer. It validated doubts, amplified anger, and agreed with almost anything, and at the scale it was serving that stopped being a quirk and became a safety problem.

The postmortem contains one sentence that should stop any engineering leader shipping AI features: “We also didn’t have specific deployment evaluations tracking sycophancy.” Some expert testers had reported that the model “felt” slightly off. It shipped anyway, because the metrics looked fine¹.

This is not a story about a careless team. It is a story about the best-instrumented AI organization in the world discovering that its instruments did not measure the dimension that broke.

If it can happen there, the interesting question is not how do I avoid shipping a bad model - it is what would have had to exist for anyone to notice in time?

Why Nothing Caught It#

The reflex is to say testing no longer works. That is too broad, and it will cost you the argument with your own senior engineers.

Testing works fine. It is a stack, and one layer of it is missing:

• Unit tests verify deterministic components. Given X, return Y. These still work perfectly and you still need all of them.
• Contract and integration tests verify interfaces - shapes, schemas, status codes, the boundary between your system and someone else’s. Also still work.
• Evals measure behavioral quality across a distribution of probabilistic outputs. How often is the answer good, on which kinds of input, judged by whom.

Every deterministic layer of OpenAI’s stack presumably passed. Nothing was broken in the sense that the word broken traditionally means. The output was well-formed, on-topic, fast, and correctly typed. It was just worse.

That is the specific gap. Tests assert what the code does. Evals measure what the model says, in aggregate, against a standard you had to write down first.

So the question changes shape. “Does it pass?” stops being answerable, because the same input can produce a brilliant answer, a mediocre one, or a confidently wrong one. Every meaningful question becomes a rate: how often is it right, on which inputs, says who, and what does a miss cost?

An eval is the tool that answers those, and it is less exotic than it sounds - a fixed set of real examples, a written definition of a good answer, and a score. Quality stops being pass-or-fail and becomes “94% good enough, on the inputs that matter.”

Try finishing that sentence for a feature you are about to ship: a number, on defined inputs, with a failure rate the business has explicitly accepted. Most teams cannot, and it shows up later in the aggregate. The share of companies abandoning most of their AI initiatives reached 42% in 2025, up from 17% the year before².

Start With Failures, Not Frameworks#

The good news is that the first version costs less than you think, and it starts somewhere unglamorous: reading.

Pull a hundred real interactions with your feature. Read them yourself - not a summary, not a dashboard. Mark every bad one. Sort the bad ones into piles.

The piles are the point, because they are never evenly distributed. In one documented case, three categories accounted for over 60% of all problems, and the largest was mundane date handling - the assistant failed 66% of the time when a user said something like “schedule a tour two weeks from now.” Fixing that one category took the success rate on it from 33% to 95%³. Anthropic’s guidance says the same thing from the other direction: twenty to fifty cases drawn from real failures is enough to start⁴.

Why start from real interactions rather than writing a spec first? Because with these systems you discover your standards by grading. Researchers studying evaluation tools found people cannot fully articulate their criteria until they have judged real outputs - the judging is what teaches them⁵.

In deterministic software you write the test, then the code. Here you ship narrow, collect failures, and let the failures write the tests.

What an Eval Actually Looks Like#

Here is the artifact. This is the shape of a release gate for a customer-facing answering feature - adapt the dimensions, keep the structure:

feature: support-answer-generation

dataset:
  normal:        50   # representative of real traffic
  known_failures: 20   # every bug that reached a customer, forever
  adversarial:   10   # prompt injection, hostile framing, refusal probing
  multilingual:  10   # if you serve them, you evaluate them
  ambiguous:     10   # underspecified requests where the right move is to ask

release_thresholds:
  overall_quality:        ">= 92%"
  unsupported_claims:     "<= 1%"
  wrong_entity:           "0%"
  policy_violations:      "0"     # blocking, no exceptions

required_comparison:
  - current_production_version    # never evaluate a candidate alone
  - candidate_version
  - human_reviewed_disagreements  # sample where grader and human differ

on_regression: block_release
owner: <a named person, not a team>

Three things about this are load-bearing and easy to skip.

The known-failures set only grows. Every incident that reached a customer becomes a permanent row. This is the regression suite, and it is the reason the second year is easier than the first.

The comparison is against current production, not against an absolute bar. You are not asking “is this good?” - you are asking “is this better than what users have right now?” An absolute threshold tells you nothing about a candidate that is 91% when production is 88%.

The thresholds are split by consequence. Overall quality is a percentage you can trade against. Policy violations are a count, and the count is zero. Never average those two together, because a single number lets a catastrophic failure be offset by a hundred pleasant ones.

Then the rubric your graders actually apply:

Three levels, not five. Five-point scales feel more precise and produce markedly worse agreement between graders, which means your scores move when your graders change rather than when your product does. If two experienced people cannot independently land on the same cell, the rubric is the thing that is broken, not the model.

Calibrate the Grader#

At some scale you will want a model to grade the outputs, because humans do not scale. That pattern is called LLM-as-a-judge, and it is fine. Just be clear about what you have built: the grader is a measurement instrument, and instruments need calibrating.

Models grading models have known biases. They favor the first answer they see, the longer answer, and their own writing⁶.

The correction is mechanical. Have humans grade a sample. Compare. Adjust the grader until it agrees with your experts nearly all the time, which takes a few honest rounds³. Until then, treat its scores as decoration.

Anthropic’s internal rule is the right one: nobody trusts an eval score until someone has read the actual transcripts⁴.

The Ground Moves Under You#

One more thing makes this permanent rather than a launch checklist. Your feature sits on a model you do not control, and it changes without asking you.

Researchers tracking GPT-4 through 2023 watched a coding benchmark collapse - not because the model got worse at coding, but because it started wrapping its output in markdown, silently breaking every pipeline that executed the result directly⁷. A cosmetic change. A contract test on the output format would have caught it in seconds, and nobody writes that test for a dependency they believe is frozen.

So pin your model versions - name an exact version rather than whatever “latest” points at this week. Then do not rely on the pin. Re-run the eval set on every model, prompt, or data change, and score a sample of live traffic continuously.

The gap between your offline score and your live score is your early warning. OpenAI’s expert testers were that signal, reporting that something felt off while the dashboards stayed green. The lesson is not that dashboards are useless. It is that a qualitative signal contradicting a quantitative one is information, not noise.

A Definition of Done for AI Features#

A deterministic feature is done when the tests pass. An AI feature is done when five things are true.

1. A versioned test set of real examples exists, weighted toward known failure modes.
2. “Good enough” has a number - a pass rate on that set, plus a failure rate the business has explicitly accepted in writing.
3. The grader is calibrated - if a model grades, its agreement with human judgment has been measured rather than assumed.
4. Never-events have deterministic guardrails - ordinary code between the model and the customer, enforcing the handful of things that must never happen. Not model judgment. The dealership chatbot that agreed to sell a new SUV for one dollar was not a bad model; it was a missing guardrail⁸.
5. There is a rollback path and a named owner - because the answer to a bad behavioral release is the same as for any bad release: reverse it, fast⁹.

Anything less is a demo with production traffic.

The cost is real and worth stating plainly. Building the first eval set takes a senior engineer a week or two, and human grading stays expensive because it never fully goes away - calibration is recurring, not one-off. What you buy is the ability to change your prompts and models on purpose instead of by feel.

Closing#

Air Canada found this out in a tribunal, which ordered it to pay damages over a bereavement policy its chatbot invented. The airline argued the bot was “a separate legal entity responsible for its own actions.” The tribunal disagreed¹⁰.

That is the part worth remembering. The organization is accountable for the output, whether or not anyone specified it, tested it, or intended it.

You cannot test a probabilistic feature into correctness. You can only measure it into acceptability - and courts, customers, and screenshots are how you find out you did not.

References#