# How do you know it works? AI for Software Engineers, part 11 (19 of 22) > Because the same input can give different outputs and a good answer can hide a bad path, agents need evals: real cases with assertions on the result and the path, run more than once. In production, traces of each model and tool call show what actually happened, and eval results should block a release. You've probably had this happen. You show your agent to the team, it answers the question perfectly, everyone nods. The next morning someone asks almost the same thing and it calls the wrong tool, or confidently answers a different question. It worked in the demo. So does it work? You already have unit tests and dashboards. Why don't they answer this? Two reasons. The first is that the same input doesn't give the same output. A unit test assumes that if it passed once, it passes every time. With a model behind it, one green run tells you it can work. It doesn't tell you it will. The second is that an agent's answer is the end of a path. It read some things, called some tools, maybe asked for a permission. A good final message can sit on top of a path you would never have allowed. So you need two things you already know from running services. Tests before you ship, which people here call evals, and observability once it's live. **Start with the cases you already have** Remember the five prompts from 04c, each with a good answer written down? That's your first eval set. Grow it from real life. Every time the agent gets something wrong, in testing or in use, write it down as a case. Anthropic's guidance on agent evals says to start with 20 to 50 simple tasks drawn from real failures rather than waiting for hundreds. Keep two kinds. Regression cases work today and should keep passing. Capability cases are things it can't do yet, so you can see whether a change helps. **What do you assert?** Most people stop at the final text. For an agent, that's the least interesting part. Say your team assistant is asked to summarise open bugs. Did it call the issue tracker or make the list up? Did it pass the right team as the filter? Did it stay read-only, or did it try to close a ticket on the way? If you have a hook that blocks edits to `.env`, does the agent stop when it's denied, or does it look for another way round? I wrote about this in [Output Scoring Passes The Agent That Leaked Your Credentials](https://www.allsrc.dev/posts/pattern-agent-evaluations/), and the short version is that a correct answer reached by a forbidden path is a failure. Don't over-do it the other way, though. If you assert the exact sequence of steps, your tests break every time the agent finds a different valid route, and Anthropic notes that agents often find approaches the eval writer didn't think of. Assert the outcome plus the rules. The right result, the tools that must or must not be called, the permissions that must hold, a cost ceiling. Most of that is plain code. Was the JSON valid, was the tool called, was the delete denied. Do those checks first. They're cheap and deterministic. **When code can't judge it** Some things code can't check. Is this summary actually correct? Is the tone right for a customer? For those, people use another model as the judge. You give it the case, the answer and a rubric, and it returns a grade. It works surprisingly well, and it has known habits. A well-known paper on this approach found judges biased by the position of an answer, by length, and towards answers like their own. So treat the judge like a new reviewer. Grade a few dozen cases yourself, compare, and adjust the rubric until you mostly agree. And read the transcripts now and then. Sometimes the agent was right and the judge was wrong. **Run it more than once** Your agent passes a case. How sure are you it'll pass the next time a customer tries it? Run it several times and count in one of two ways. pass@k asks whether it succeeded at least once in k tries. pass^k asks whether it succeeded every time in k tries. The second idea comes from the tau-bench paper in 2024, which tested agents talking to simulated customers. Even the strong models of the time succeeded on under half the tasks, and fewer than a quarter of tasks in the retail domain passed eight runs in a row. The maths is unkind. An agent that succeeds 75% of the time passes three runs in a row only about 42% of the time. Think of it like availability. Nobody cares that your service answered once. They care that it answers on every request. A customer meets your agent many times, so pass^k is closer to what they experience. For a coding assistant, where you review every change anyway, succeeding at least once is often good enough. **Seeing what happened** When a case fails, you need to know why. Which step went wrong? If you've used distributed tracing, you know this already. One request becomes a tree of spans, and an agent run is no different. A root span for the run, and a child span for each model call and each tool call, with timing, errors, and the token counts from the `usage` field we saw in 02a, which turn into cost. Now you can see that one tool was slow, or that the agent looped and cost five times the usual. OpenTelemetry has semantic conventions for this, with names for model-call spans, agent spans, tool-execution spans and token attributes. As of October 2026 they're still marked Development and have moved to their own repository with no tagged release yet. Use them, pin the version, and expect renames. One thing they get right is that recording the actual prompts and responses is opt-in. Those often contain customer data, so decide on purpose what your traces keep and who can read them. **Once it's live** Your test set will never cover everything real users ask. So sample a slice of live runs, score them in the background with your code checks and your judge, and watch the trend. It's real-user monitoring for quality, and a drop after a deploy or a model update shows up within days. Then turn each bad production run into a new case in your eval set. Of everything here, that habit pays back the most. **Make it block** Finally, run the eval set on every change to a prompt, a tool, a model or a framework version. In Part 10 I said a model retirement should be a measured change. This is the measuring. And make it able to fail the build. Policy and security cases block every time. Quality cases can have a small budget against a baseline. More on that in [An Eval Suite That Never Blocks Is A Dashboard](https://www.allsrc.dev/posts/pattern-cicd-evaluation-gates/). ![Feedback loop separating observability from evaluation and feeding production failures into a CI release gate.](assets/15-observability-evaluation-loop.png) *Traces explain what happened; evals decide whether the behavior was acceptable.* **When not to** Don't build an eval platform before you have twenty cases. For a script only you use, five prompts you rerun by hand is fine. Don't run every case ten times on every commit either. Repeated runs cost real tokens, so keep them for the cases that matter and for nightly runs. And don't take a public benchmark score in place of your own cases. Next: governance and security, which have been coming up all along. ## References - Demystifying evals for AI agents (https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) - Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (https://arxiv.org/abs/2306.05685) - τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (https://arxiv.org/abs/2406.12045) - OpenTelemetry semantic conventions for generative AI (https://opentelemetry.io/docs/specs/semconv/gen-ai/)