“Evals”: putting AI back in school

What testing AI taught me about exams, graders, and the mistakes a passing grade can hide.

Lately, I’ve been testing AI on real knowledge-work tasks to understand how far models have come and what I can trust them with next. I wanted tests I could rerun as the models improved, so those decisions wouldn’t depend on gut feeling.

The hardest part turned out to be deciding what deserved a passing grade. I went in comparing models and came out questioning whether I could trust the grading. Somewhere in the middle, evals started looking a lot like a school (with a few very confident teaching assistants).

In this article

1. Start with the school
2. Then grade the graders
3. Open the mistakes that look reasonable
4. Ask how the winner was chosen
5. Test everything around the model
6. Bring it back to the product

1. Start with the school

An “eval” is part of the work of making unpredictable AI more reliable. You test what it gets right, where it breaks, and what needs to change before you trust it with more.

The students are the models (Fable, Astra, Sol, Opus, etc.). The exam is a set of tasks, with an answer key or a rubric explaining what good work looks like. You ask everyone to do the same work, then see who can do it reliably.

Some parts are easy to check with code. Does the math add up? Is the response in the right format? Does the cited page exist? These checks are like the machine that scans an answer sheet: useful for the things it can actually verify.

Then come the answers that need judgment. Is this explanation complete? Does the evidence support the conclusion? Would someone reading it walk away with the wrong impression? Another AI can help grade those answers. This is what people call “LLM-as-a-judge”, and in this school, it plays the role of a teaching assistant (TA), or a GSI/Reader at Berkeley (my Haasie crew will know this 🐻).

2. Then grade the graders

The awkward part is that the TA can be wrong too. Research has found that AI judges can favor their own answers, and a panel of different models can reduce some of that bias. But adding more graders doesn’t automatically make the grading right. Several models can agree because they missed the same thing (a very efficient way to be confidently wrong together).

That’s where the professor comes in: someone who understands the work well enough to check the grading. They review a sample, look at disagreements, and decide whether the rubric needs to change. The professor can make mistakes too, so their reasoning needs to be inspectable.

That also made companies like micro1 click for me. They recruit domain experts to create and review material used to train and evaluate AI. Its legal expert roles include reviewing legal reasoning, checking citations, and giving feedback on training data. In the school analogy, part of what they’re selling is access to the professors.

The value is someone who can explain why an answer is good, spot what’s missing, and challenge a grading rule that rewards the wrong thing.

Improving the students and improving the graders are two separate jobs. A low score might mean the model struggled. It might also mean the question was ambiguous, the answer key was incomplete, or the grader punished a perfectly acceptable answer. Changing models before figuring that out can leave you solving the wrong problem.

3. Open the mistakes that look reasonable

During my MBA, I used AI to help me work through business cases. One answer sounded convincing until I went back to the case and checked what it was based on. The model had flagged a problem, but its explanation didn’t match the facts. There was a real issue elsewhere. It felt like a smoke alarm ringing in the wrong room: you check, find nothing, and become less likely to trust the warning.

That made me want a better way to check these answers. How could such a capable model get this wrong? And what would actually fix it: a better model, clearer instructions, or a better way to check its work?

• The AI’s warning: Something is wrong in this section.
• What the document showed: This section was fine. The actual problem was elsewhere.

The warning pointed to the wrong place.

That is a harder failure to notice than a completely invented answer. Most of the response can look reasonable, including the citation. You have to open the source and follow the reasoning to discover where it went wrong.

For my financial-document tests, my past life in banking came in handy. I kept asking myself whether I’d actually trust the answer enough to use it. When I checked ten responses myself, I agreed with the automated decision on nine. In the remaining case, the grader demanded something I didn’t think was necessary. Ten checks don’t establish that a grader is reliable, but they gave me something specific to improve.

4. Ask how the winner was chosen

The definition of passing turned out to matter just as much. I gave each model the same 12 questions twice and had three AI judges grade each answer. Then I counted the same grades in two ways: an answer passed if at least two judges approved it, or only if all three did.

For each model, I counted how many questions it passed on both attempts. The answers stayed the same; the definition of passing changed.

That made me more careful about the question “Which model is best?” Best under whose rubric? How strict is the grading? Does a missing caveat count the same as a false number? A single accuracy score can hide decisions that matter much more to the person using the product.

5. Test everything around the model

I also learned to be more explicit about what I wanted the model to notice. If the information is missing, say it’s missing. If two numbers disagree, flag the conflict. Don’t quietly turn an incomplete document into a confident answer.

It’s like telling students before an exam that noticing missing information and showing their work are part of the grade. They know what to pay attention to before they start. With an agent, those expectations belong in the instructions.

In my comparison, two models became less likely to flag contradictions when I removed the guidance and the fields asking them to report gaps. I changed both together, so I can’t pin the difference on one sentence. The practical lesson is still useful: tell the model what to pay attention to before you grade it.

The answer that isn’t there

Students perform differently depending on the question, the materials they’re given, and the time available. An agent has similar constraints: what it can search, what it can read, and when it has to stop.

That last part became especially interesting when the answer wasn’t in the document. Finding a number gives the agent somewhere to stop. Deciding that the number isn’t there requires enough searching to justify that conclusion. “I couldn’t find it” and “the document doesn’t contain it” make different claims.

Some failures I initially attributed to a model were runs that my own tool limit had cut short. Removing the limit didn’t guarantee success, but opening the traces changed my understanding of what had failed. The final answer alone wasn’t enough to explain the result.

The cost of getting to a useful answer

Cost deserves the same care. A cheap answer that sends someone back to check everything may be expensive in practice. An expensive model used for every small step may be unnecessary. Box’s published routing experiment is a useful example: assigning different models to different reasoning steps reported similar task success at lower inference cost.

Your subscription might fund your first experiment

This experiment used $176.06 in API calls. I paid $0 extra. Here’s how.

At the time of writing, Anthropic and OpenAI offer API credits with eligible paid subscriptions. I used those alongside promotional credits from Meta. If you’re already paying for a subscription, check what’s included. You may already have the budget for your first eval.

And if you’ve just built an MCP and want to test whether agents can actually use it, I’d recommend Oqoqo (I’m using it right now to continuously improve trackly's MCP). You can run the same tasks through agents like Claude Code and Codex, compare results, and inspect where they get stuck. It also lets you connect your existing subscriptions instead of using API keys (a very welcome detail).

6. Bring it back to the product

For me, this brings evals back to product management. We spend so much time discussing the model that it’s easy to overlook the decisions around it. Someone still has to decide what work matters, what evidence is sufficient, and which mistakes a user cannot afford.

An eval makes those decisions visible. If the score improves but the experience doesn’t, I want to understand what the exam rewarded. And if a real user finds a failure the exam never covered, that failure belongs in the next version.

Before I ask which model won, I want to see the exam.

Closing note

This picks up where Designing for users you’ll never see left off. That one was about designing the paths agents follow through a product. This one asks how we know whether the work they produce is any good.

You can start by trying a few prompts with your agent and checking the answers yourself. That’s already a small evaluation. A repeatable process becomes more useful as you start relying on agents, changing their instructions, or choosing between models.

Whether those agents write code for you or complete tasks for your customers, the question eventually reaches your desk: how will you know the next version is actually better?

Until next time,

Kevin