An AI agent that works in the demo can still fail in production. Here is how to build an evaluation suite that catches it first, with a practical template.
If you only test an AI agent by chatting with it a few times, you have tested nothing. You picked the inputs, you ran it until it looked good, and you learned that the happy path exists. Production does not use the inputs you picked. It uses the five thousand you did not think of.
An evaluation suite is how you find the failures before your users do. It is the single thing that separates an agent you can trust from a demo with a chat box. Here is how to build one, using the same approach we use on every agent we ship.
An evaluation suite is a set of real test cases, each with a known-correct outcome, that runs automatically every time you change the agent. Not a vibe check. A fixed set of inputs, an expected result, and a pass/fail.
The reason it matters: language models are probabilistic. Give one the same messy input twice and you can get two different answers. Without a suite, every prompt tweak is a guess, and you cannot tell whether today's change fixed one thing and broke three. With a suite, you know.
Pull your test cases from reality. If the agent handles support tickets, take a hundred actual tickets. If it processes invoices, take real invoices, including the ugly ones. Cover:
Fifty to a few hundred cases is usually enough to start. Quality beats quantity. One real edge case is worth twenty invented happy-path ones.
This is the part teams skip, and it is the part that matters. For each case, write down the expected outcome in a way a script can check. Depending on the agent, "correct" might be:
If you cannot state what correct looks like for a case, you do not understand the case well enough to automate it yet.
Accuracy alone lies. Track a few things together:
A support agent at 90% task success sounds close to done. Run a thousand times a day, and 90% means a hundred failures a day. For anything that touches money or customer trust, you are chasing the gap from 90 to 99, and that gap is where the engineering lives.
The suite has to be a command, not a ritual. Wire it so it runs on every prompt change, model swap, and tool update, and blocks a deploy if the success rate drops. The value is not the first run. It is the tenth, when you change one line to fix an edge case and the suite tells you it broke two others.
From the field. A team asked us to "just improve the accuracy" of an agent that was already live and misrouting a slice of requests. There was no eval suite, so every change they made was a coin flip. We spent the first few days doing nothing but building the suite from their real logs. Once we could see task success and tool-call accuracy on every change, fixing the routing took a fraction of the time it had been costing them. The suite was not overhead. It was the thing that made the fix possible.
Each row of your eval set needs, at minimum:
id and a short descriptioninput (the real message, ticket, or document)expected (the correct outcome, in checkable form)check_type (exact, contains, tool-call, action)category (happy-path, edge, adversarial) so you can see success rate per bucketRun the agent over every row, compare to expected, and output a pass/fail plus the per-metric scores. Store the results so you can see the trend across changes. That is the whole shape of it. It does not need to be fancy. It needs to be real and to run every time.
How many test cases do I need? Enough to cover your real happy path plus your real edge cases, usually 50 to a few hundred. Add every new failure you find in production as a permanent case so it can never regress silently.
What is a good task success rate? Depends on the stakes. For low-risk information tasks, high 80s to low 90s may be fine. For anything touching money or customer records, you want 99%+ on the cases that matter, plus correct escalation for the rest.
Can I evaluate an agent without writing code? For a small set, you can grade by hand once. But the value comes from running automatically on every change, and that needs a script. Hand-grading does not scale past the first week.
When should I build the eval suite? Before you build the agent, ideally. The suite is how you define "done." Building the agent first and the evaluation later is how demos that impressed a room end up misrouting refunds in production.
Every agent we ship comes with its evaluation suite, guardrails, and logging by default, because that is the part production requires and demos skip. If you have an agent that works in testing and stumbles on real data, that is a normal place to be, and the fix is a known quantity.
Tell us what you are building on the start a project page, or see how we approach AI agent development.