An agent works perfectly during development. It answers the sample questions, calls the expected tools, completes the demo workflow, and produces results convincing enough to ship.
A week later, production tells a different story.
Some users receive correct answers through inefficient twenty-step trajectories. Others get confident success messages even though the requested operation never happened. A model upgrade improves reasoning on difficult tasks but quietly makes a common workflow less reliable. A new tool description fixes one class of failures while introducing another somewhere else. Nothing is obviously broken in the traditional software sense. The API is online, the database is healthy, and no exception appears in the logs.
The system simply behaves worse.
This is one of the central engineering problems of agentic software. Traditional applications can rely heavily on deterministic assertions because identical inputs usually produce predictable computational paths. AI agents operate differently. They reason probabilistically, choose tools dynamically, interact with mutable environments, and may take different routes toward the same objective across repeated runs.
Testing only whether the application is technically running is no longer enough.
Teams need a way to measure whether the agent is still behaving correctly.
That is the role increasingly being filled by evaluations, or evals.
A Demo Can Tell You That an Agent Works. It Cannot Tell You How Reliably.
The earliest stage of an AI product often relies on informal evaluation. Developers repeatedly try prompts, inspect outputs, adjust instructions, change models, and decide whether the result feels better. This works surprisingly well when the system is small because large behavioral changes are easy to notice manually.
The approach stops scaling as the product becomes more capable.
An agent that completes a task once has demonstrated possibility, not reliability. Because model outputs vary between runs, the same task may succeed several times and then fail for reasons that are difficult to reproduce. Once tools and environments enter the loop, additional variables appear. The model might choose a different tool, retrieve different context, receive a slightly different external response, or explore an alternative sequence of actions.
Anthropic formalizes this distinction in Demystifying Evals for AI Agents. A single evaluation task can be executed through multiple trials, precisely because one attempt cannot fully characterize probabilistic behavior. Anthropic also distinguishes between the agent's transcript, which records what happened during execution, and the outcome, which represents the final state produced by that execution.
That distinction sounds minor until an agent starts taking real actions.
Consider a flight-booking agent. Its final message may confidently state that a reservation has been completed. Evaluating only the response would mark the interaction as successful. Evaluating the environment might reveal that no reservation exists in the database.
The language looks correct. The task failed.
For agentic systems, correctness increasingly belongs to the state of the world rather than the fluency of the final response.
The Object Being Tested Has Changed
A traditional LLM evaluation can often be represented as a relatively simple relationship between an input, a model response, and a grader. Agent evaluations have a larger surface area because the model is embedded inside an execution system.
A coding agent might receive a repository and a task, inspect files, modify source code, run commands, install dependencies, execute tests, interpret failures, edit the implementation again, and finally report completion. The final answer contains only a tiny fraction of the behavior that produced the result.
This is why evaluating agents only at the output layer can hide important failure modes.
Anthropic uses the term transcript, also called a trace or trajectory, for the complete record of a trial, including model outputs, tool calls, intermediate results, and other interactions. The final environment state is treated separately as the outcome. A useful evaluation can inspect either or both depending on what the product actually needs to guarantee.
OpenAI has moved in the same direction. Its evaluation tooling now includes trace grading, which is intended to evaluate an agentic workflow end to end rather than grading only its final text response. OpenAI's platform also supports datasets and several forms of automated graders so repeated evaluations can be run across model and configuration changes.
The important change is conceptual.
You are no longer testing a prompt.
You are testing a behavior-producing system.
Success Has More Than One Dimension
A useful agent evaluation rarely reduces to a single question such as "Was the answer correct?"
Imagine an internal support agent that can investigate customer issues and modify account state. A successful interaction might require several conditions to hold at the same time. The correct account must be identified, the right policy must be followed, the appropriate tool must be called, no unauthorized field should be modified, the user should receive an understandable explanation, and the resulting account state must match what the agent claims happened.
These dimensions require different ways of grading.
Some properties are deterministic. If the agent was supposed to create a database record, code can check whether the row exists. Coding agents can be tested with unit tests, static analysis, type checking, or security scanners. Structured outputs can be validated against schemas.
Other properties are qualitative. A research agent may need to produce a comprehensive synthesis, use authoritative sources, distinguish evidence from speculation, and avoid omitting material facts. There may be no single string that represents the correct answer.
This is why modern evaluation systems combine multiple grader types. Anthropic groups them broadly into code-based, model-based, and human graders, while OpenAI's grader APIs support mechanisms including string checks, text similarity, model-assigned scores, model-assigned labels, and combinations of multiple graders.
The strongest evaluation strategy depends on what is being measured. Deterministic assertions are preferable when an objective property can be checked directly. Model-based graders become useful when the criterion involves semantic quality that is difficult to express as code. Human review remains important for calibration, edge cases, and dimensions where automated judgment has not been shown to be sufficiently reliable.
An eval suite therefore behaves less like a single test and more like a measurement system.
The Path Can Matter Even When the Destination Is Correct
Outcome-based grading is powerful because it focuses on whether the agent actually accomplished the task. But there are situations where two agents can reach the same correct outcome while one behaves substantially better than the other.
Suppose two research agents produce equally accurate reports. One completes the task with a handful of targeted searches and authoritative sources. The other performs dozens of unnecessary searches, repeatedly revisits the same pages, consumes several times more tokens, and still happens to arrive at the correct result.
If the only metric is final correctness, the two systems appear equivalent.
They are not equivalent in production.
Latency, token consumption, API usage, reliability, and exposure to external content all depend on the trajectory. A path containing unnecessary actions also creates more opportunities for failures and side effects.
The reverse problem can occur as well. An evaluator that requires a specific trajectory may accidentally punish a model for discovering a better solution than the evaluation designer expected. Anthropic describes an example where Claude Opus 4.5 found a policy loophole while solving a flight-booking task in the τ2-bench environment. According to the evaluation as written, the behavior counted as failure even though the model had identified a solution that was arguably better for the user.
This is one reason good agent evaluation requires judgment about what must be constrained and what should remain flexible.
If only the outcome matters, grading the exact path may overfit the evaluation to one implementation strategy. If the path itself contains safety, cost, compliance, or efficiency requirements, ignoring it may hide serious problems.
Agent eval design is partly the art of deciding where that boundary belongs.
Capability Tests and Regression Tests Solve Different Problems
Once an evaluation suite exists, not every test should be interpreted the same way.
Anthropic separates agent evals into two especially useful categories: capability evals and regression evals. Capability evaluations ask what the system can do today. They are intentionally difficult and may begin with relatively low pass rates because they identify a frontier that the team wants to improve.
Regression evaluations serve a different purpose. They represent behaviors the agent already performs reliably and should continue performing after changes to prompts, tools, models, retrieval systems, or orchestration logic. These tests should remain close to consistently passing because a decline signals that the system has lost an existing capability.
The relationship between the two is useful over time. A difficult task may begin inside a capability suite while engineers improve the system. Once the agent handles it reliably, that task can graduate into the regression suite.
This creates a moving quality boundary.
Yesterday's research problem becomes tomorrow's regression test.
The same pattern is familiar in conventional software engineering. A bug is discovered, fixed, and then converted into a test so the same failure does not quietly return. Agent development can adopt the same discipline, but the test case may involve an entire interaction trajectory rather than a single function call.
Anthropic specifically recommends building early eval datasets from the failures and manual checks teams already encounter. It argues that teams do not need hundreds of examples to start and suggests that approximately 20 to 50 well-chosen tasks can already provide useful signal during early development, when product changes tend to produce relatively large effects.
That makes eval development less mysterious. The initial dataset is often already hiding inside bug reports, support tickets, QA notes, and the examples engineers repeatedly test before shipping.
Model Upgrades Turn Evaluation Into Release Engineering
Frontier models improve quickly enough that AI products face an unusual deployment problem. A new model may offer better reasoning, lower latency, lower cost, improved tool use, or a larger context window. The temptation is to switch immediately.
But "better model" is not equivalent to "better product."
An agent is the interaction between a model and its surrounding harness, tools, prompts, retrieval systems, environment, and product constraints. A model that performs better on public benchmarks can still behave differently enough to introduce regressions inside a specific application.
Without an evaluation suite, teams often fall back to manual migration testing. Engineers rerun representative tasks, inspect outputs, collect subjective impressions, and hope the sample is broad enough to expose important changes.
With a maintained eval suite, the migration can be treated as an experiment.
Run the existing agent configuration against the current model. Run the same suite against the candidate model. Compare outcome quality, failure categories, tool behavior, latency, token usage, and cost. Inspect representative trajectories where the scores diverge. Only then decide whether the upgrade is actually beneficial.
Anthropic argues that this is one of the compounding advantages of maintaining evals. Teams without them may spend weeks determining whether a model change improves their product, while teams with established suites can evaluate the change substantially faster. Its July 2026 session on evaluating AI agents frames model adoption as one of the practical reasons product teams need end-to-end evaluation rather than single-turn testing.
This is where evals start to look less like AI research infrastructure and more like release infrastructure.
A Green Score Can Still Be Wrong
Metrics create confidence only when the evaluation itself is trustworthy.
A poorly specified task may mark a correct agent as incorrect. A grader may encode assumptions that were never stated in the task. An LLM judge may reward style instead of substance. A regression dataset can become too easy and stop detecting meaningful differences between systems.
More surprisingly, even infrastructure can distort an agent evaluation.
Anthropic quantified this problem in Quantifying Infrastructure Noise in Agentic Coding Evals. While examining Terminal-Bench 2.0, researchers found that infrastructure configuration alone could change success rates by several percentage points. Across the most and least resourced configurations they tested, the difference reached six percentage points, even though the model, harness, and task set remained constant.
The reason is specific to agentic evaluation. A static benchmark may only require a model to return an answer. A coding agent operates inside a real runtime where it installs packages, launches subprocesses, compiles software, and consumes memory. Resource limits therefore change what actions are practically available to the agent.
Anthropic observed infrastructure-related task failure rates as high as 5.8 percent under strict resource enforcement in one configuration. Increasing resource headroom reduced those failures, but sufficiently generous resources eventually began helping the agent solve tasks it could not solve under tighter constraints. At that point, infrastructure was not merely reducing noise. It was changing the problem itself.
This complicates any interpretation of benchmark scores.
If two coding agents run with different memory limits, execution timeouts, network policies, or sandbox behavior, they may not actually be taking the same test.
Reliable evaluation therefore requires reproducibility not just at the model layer but across the entire execution environment.
Evals Should Be Connected to Production, Not Frozen Before It
An evaluation suite built once and never updated gradually becomes a historical record of what developers used to worry about.
Production is where new failure modes appear.
Users combine capabilities in unexpected ways, external APIs behave differently than mocked environments, tool descriptions encounter ambiguous requests, edge cases expose product assumptions, and new model releases create behavioral changes that were difficult to predict beforehand.
A useful evaluation process feeds those discoveries back into the test suite.
A production incident should not end only with a prompt fix or code patch. When possible, the failure should become a reproducible eval case. The next release then has to demonstrate that it solves the new problem without damaging existing behavior.
This creates a feedback loop:
- Production Failure
- Reproducible Task
- Evaluation Case
- System Change
- Regression Check
Over time, the suite becomes a compressed record of what the product team has learned about its own agent.
Anthropic describes eval suites as living artifacts that require ongoing ownership and maintenance. Its guidance recommends allowing domain experts and product teams to contribute tasks while dedicated evaluation infrastructure supports execution and aggregation.
This is important because correctness in an AI product is ultimately a product question before it becomes an engineering metric. Engineers can build the measurement machinery, but the definition of success often belongs to the people who understand what users actually expect.
Why the CI/CD Analogy Works, and Where It Does Not
Calling evals the CI/CD layer for AI agents is an analogy rather than an industry-standard definition.
It is useful because evals increasingly occupy a similar position in the release process.
Continuous integration protects conventional software from regressions by automatically exercising expected behavior whenever code changes. Agent evals can perform a comparable role when prompts, tools, retrieval systems, model versions, orchestration code, or policies change. A candidate configuration can be executed against a known task suite before it reaches users.
The difference is that agent behavior is probabilistic.
A unit test usually produces the same result every time when the code and environment are unchanged. An agent eval may need repeated trials, statistical interpretation, and several graders to determine whether a change represents a real improvement. A release gate might therefore operate on pass-rate thresholds or confidence intervals rather than requiring every trial to produce an identical execution trace.
OpenAI's Evals platform reflects this increasingly operational role. Evaluations can be defined once and rerun across different models and model parameters, while grader types can be composed depending on the property being measured. OpenAI has also incorporated datasets and trace grading into its broader agent development platform, positioning evaluation alongside the tools used to build and optimize agent workflows.
So the analogy should not be taken literally.
Agent evals are not unit tests with an LLM attached.
They are closer to a continuously maintained behavioral quality system.
The Real Asset Is Not the Score
It is easy to focus on the final number produced by an evaluation suite: 72 percent success, 91 percent instruction adherence, $0.18 average cost per task.
Those metrics are useful, but the deeper value of an eval system is the structure behind them.
A mature evaluation suite contains an explicit definition of expected product behavior, examples of historically difficult tasks, documented failure cases, grading criteria, representative environments, and enough instrumentation to understand why behavior changed.
That becomes increasingly valuable as the underlying models evolve.
Prompts can be rewritten. Tools can be redesigned. Models can be replaced. Retrieval systems can be migrated. The evaluation suite remains the contract describing what the product is supposed to accomplish.
This changes how agent development can be practiced.
Instead of modifying an agent and asking whether it feels better, teams can form hypotheses and measure them. Instead of assuming that a new frontier model must improve the product, they can test the claim against their own workloads. Instead of discovering regressions through users, they can preserve previous failures as release gates.
Software engineering became dramatically more reliable when testing moved from an activity performed before release to infrastructure continuously attached to the development process.
Agent engineering is beginning to make the same transition.
The systems are different, the measurements are noisier, and correctness is often harder to define. But the direction is increasingly clear: as AI agents take on more consequential work, evaluation cannot remain an occasional benchmark exercise.
It has to become part of how the software ships.