An eval asks if it’s good. An experiment asks if it helped.
What AI evals are, how they differ from A/B tests, and why product designers should write the rubric before the model ships.
An AI eval is a test that scores a model’s output against a standard before people see it. An A/B test measures what happens to people after they see it. This year evals became the discipline AI product teams keep saying they need, and anyone who has run an experimentation platform will recognize most of it.
What is an AI eval?
An eval is a set of inputs, the outputs your system gives for them, and a judgment of each output: right or wrong, pass or fail, or a score against a rubric. The judge can be code, a person or another model. You run it every time you change a prompt, a model or a tool, and you watch whether the score moves.
The hard part is the rubric: deciding, in writing, what good means for your product.
Why evals became the job this year
AI output isn’t deterministic. A change that fixes one case can quietly break ten others, and ordinary software tests won’t catch it.
In April, OpenAI’s chief product officer Kevin Weil spent part of his Lenny’s Podcast episode on why evals matter, and the show’s takeaway was that writing them is becoming a core skill for product managers. In March, Hamel Husain’s field guide to improving AI products called error analysis the single most valuable activity in AI development. His first principle was simply to look at your data. By June, experimentation vendors like Statsig were describing the sequence most teams now follow: offline evals first, then an A/B test on real users.
How is an eval different from an A/B test?
At Uber I led design for the experimentation platform. Uber’s engineering blog said in 2018 that it had over 1,000 experiments running at any given time. An experiment there answered one question: did this change help or hurt the people who got it, compared with people who didn’t?
An eval answers an earlier question: is this output good?
- An eval runs before launch, on a fixed set of cases, every time something changes. It measures the quality of an output against a rubric, judged by code, a reviewer or a model. It can’t tell you whether anyone is better off.
- An A/B test runs after launch, on live traffic. It measures the change in people’s behavior against a control. It can’t tell you why an output was good or bad.
You need both. An AI feature can pass every eval and still not help anyone, because the rubric measured the wrong thing. And a feature can win an A/B test while quietly giving a few people terrible answers that an average hides.
The rubric is a design artifact
The most useful lesson from experimentation carries straight over: the metric is the product decision. At Uber, choosing which metric a launch had to protect was choosing what the company cared about. A guardrail on sign-in errors said that a driver who can’t sign in matters more than whatever the launch was meant to improve.
An eval rubric does the same. “Answers the question” and “answers the question in under three sentences, names its source and asks before guessing” are two different products. Whoever writes that sentence is designing the product, whatever their title.
So designers should write rubrics: what a great answer contains, what a bad one looks like, which mistakes are tolerable and which ones end the conversation. It’s the same skill as a design critique, written down so a machine can apply it a thousand times.
Where to start
Read real outputs, dozens of them, before writing any criteria. Sort the failures into kinds. Write one rule for each kind that matters, with a good and a bad example. Then run it every time anything changes, and keep the A/B test for the question the eval can’t answer: did it help?