Every operator we talk to has an AI agent doing something in their business right now: routing leads, drafting replies, summarizing calls. Almost none of them can tell you how often it gets things wrong. That gap is why AI agent evals are becoming the most important thing we build before we ever hand a client the keys.
Why agent pilots stall quietly
Most AI pilots do not fail with a bang. They fail by drifting. A lead-routing agent starts misclassifying a new inbound source. A support agent starts calling the wrong tool three times out of ten. Nobody notices for weeks because nobody set up a way to notice. The team just starts trusting the agent a little less, using it a little less, and eventually it quietly gets shelved.
We have said before that chatting with a model is not the same as running a system. The same logic applies to testing one. An agent that has never been evaluated against real failure cases is not production software. It is a demo that has not failed in front of you yet.
What an eval actually is
Anthropic's engineering team published a detailed breakdown of how they test agents internally. The definition they use is the one we have adopted for every build: an eval gives an AI system an input, then applies grading logic to its output to measure success. That sounds simple for a single question and answer. It gets complicated fast once the system is calling tools, changing state in a CRM, or handing off to a human across multiple turns, which is exactly what an operations agent does all day.
The full breakdown from Anthropic is worth reading if you want the technical detail. The short version for operators: a chatbot eval checks one answer. An agent eval has to check a whole trajectory, meaning every tool call, every decision to escalate or not, and whether the final state of the world (the CRM record, the calendar entry, the sent email) is actually correct.
Why agents are harder to test than chatbots
This is the part that trips up most 10 to 50 person companies rolling out their first agent. A chatbot only produces text, so a human can eyeball whether the reply sounds right. An agent takes actions. It can call the wrong API, update the wrong field, or escalate a deal that should have closed automatically, and none of that shows up if you are only reading the chat transcript.
Anthropic's earlier guidance on building effective agents makes the same point from the design side: the more autonomy you hand an agent, the more you need a harness that can catch it before a client does. Autonomy without a test suite is not a feature. It is a liability with a nice interface.
The eval categories we run before any agent goes live
Every build we ship gets tested across the same five categories before a client ever sees it. That applies to a voice agent answering inbound calls just as much as an operations agent triaging inbox and CRM work. This is not theoretical. It is the checklist that decides whether a build goes live this week or gets another pass.
| Eval category | What it catches | Why it matters for a 10 to 50 person company |
|---|---|---|
| Task completion rate | Did the agent actually finish the job, not just respond | A lead that gets a friendly reply but no calendar link is a lost deal |
| Tool call accuracy | Wrong tool, wrong parameters, wrong order of operations | One bad API call can corrupt a CRM record or double-book a client |
| Escalation correctness | Does it hand off to a human when it should, and only when it should | Over-escalating kills the ROI, under-escalating loses trust |
| Latency under load | Response time when three requests hit at once, not one in a demo | A speed-to-lead agent that misses a 5-minute response window has failed its one job |
| Regression across model swaps | Whether last week's passing test still passes on this week's model version | Model providers ship updates constantly; your test suite is what protects you from surprises |
None of these require a research team. They require twenty to fifty real test cases pulled from actual failures, run every time something changes, whether that is a prompt edit, a new tool, or a model version bump. That number comes straight from Anthropic's own field experience running evals across coding agents, research agents, and conversational systems. It matches what we see in practice: you do not need thousands of test cases, you need the right twenty or fifty.
Build your own harness: the open-source precedent
You do not have to build an eval framework from nothing. OpenAI open-sourced its own testing framework, Evals, along with a registry of benchmarks, and it now ships a hosted version documented in the Evals API guide. The structure is the same whether you use their tooling, Anthropic's approach, or a custom harness: define the input, define what a correct outcome looks like, run it automatically, and track the pass rate over time. What matters is that the test suite exists and runs on a schedule, not which vendor's framework you borrow the pattern from.
For an operator, the practical version of this is a spreadsheet or a lightweight script that replays twenty to fifty real scenarios against the agent every time it changes, and flags anything that used to pass and now fails. That is the entire discipline. It is not complicated. It is just a step most teams skip because nobody assigned it to anyone.
More models means more evals, not fewer
Part of why this matters more this quarter than it did a year ago is that swapping the model behind an agent has never been cheaper or easier. NVIDIA now offers free API access to a wide catalog of models through its NIM developer program, on top of the frontier releases already competing on price and context window. That is good news for cost. It is also exactly why a fixed eval suite matters: every time you point an agent at a new or updated model, you need the same twenty to fifty test cases to confirm it still does the job, not just that it sounds more fluent in a demo.
We have written before about how model overload wastes operator time when every swap is a guess. Evals are what turn a swap from a guess into a decision you can defend.
What to do this quarter
- Pull twenty to fifty real cases from your agent's actual history, including the ones where it got something wrong or a human had to step in.
- Write down what a correct outcome looks like for each one: the right tool called, the right record updated, the right handoff triggered.
- Re-run that same test set every time you change a prompt, add a tool, or swap the underlying model.
- Track the pass rate over time instead of trusting a single demo run.
- Set a hard latency threshold for anything client-facing. If a lead-routing agent cannot respond inside five minutes under real load, it fails, no matter how good the reply reads.
How this maps to what we build
This is not a side process for us. It is the difference between a system a client can trust and one they quietly stop using. Every build under our product line ships with its own eval set before it goes live, whether it is the AI Chief of Staff or the AI Operations Agent. We re-run that eval set every time the client's process or the underlying model changes. Clients own 100% of the code and files we build, which means they can see the test suite too, not just take our word for it.
We start most engagements with our AI Concierge assessment, a €999 engagement credited in full to the build. We need to see the real workflow and the real failure cases before we can write a single eval. You cannot test an agent against scenarios you have not seen happen. If you want to see what this looks like on a live agency, the case studies page has the builds and the specific workflows they replaced. For teams whose main leak is response time on inbound leads, our revenue leak heatmap tool is a fast way to see where a missed 5-minute window is already costing pipeline before an agent even gets built.
The operators who get the most out of agents this year will not be the ones with the fanciest model. They will be the ones who tested it properly before it touched a client, and who keep testing it every time something underneath it changes. Our automation team builds that discipline into every system from day one, because a first system that is live in days-to-weeks only stays valuable if it keeps passing the same test six months later.
The first system we deploy for a new client typically frees up more than ten hours a week that used to go into manual admin work. That number only holds if the agent keeps doing the job correctly after the demo ends. Evals are how you make sure it does.