Evaluating an AI agent before production: tasks, graders and a regression gate

An AI agent is evaluated on its outcome, over several trials, with combined graders. A regression gate then blocks setbacks before production.

Transparency note. This article follows IgnitionAI's editorial policy. It draws on an Anthropic engineering post, the LangSmith documentation, OWASP and a research paper on LLM judges, all consulted on 29 October 2026. The regression gate example comes from Ignition Agent Trainer, an open-source project published by IgnitionAI, in alpha, with no published quantified results. The article contains no market statistics and no quantified estimates. See the Methodology and sources section at the end. Salim Laimeche is co-author, with Glaucia Lemos, of the Packt book Building Agentic AI Apps with Next.js and TypeScript, to be published on 1 December 2026 (Packt page).

Evaluating an AI agent means checking what it produced, not rereading what it said. An agent that announces "your booking is confirmed" has succeeded only if the booking exists. Because its outputs vary from one run to the next, each task is replayed several times. And because every change to the prompt, the model or the tools can break what used to work, a regression suite must block release when the score drops, latency rises or cost drifts.

In short

  • Define unambiguous tasks, each with a success criterion that can be checked on the final outcome.
  • Replay each task several times: an agent does not have one success rate, it has one per task.
  • Combine code-based, model-based and human graders, and calibrate LLM judges against human scores.
  • Also test what the agent must not do, especially sensitive actions.
  • Keep a capability suite, which measures progress, separate from a regression suite, which blocks setbacks in the integration pipeline.

This article is for teams with an agent at prototype stage that must decide whether to move it to production, or whether to change models. For evaluating a document RAG, see our guide on evaluating a RAG in production.

Why an agent is harder to evaluate than a chatbot

A chatbot receives a question and returns an answer. An agent chains tool calls, changes the state of a system and adapts to intermediate results. Anthropic sums it up in its January 2026 post on agent evaluation: agents operate over many turns, and their mistakes can propagate and compound.

Two practical consequences follow.

The final text is not enough. Anthropic distinguishes the transcript (the complete record of a trial: outputs, tool calls, reasoning, intermediate results) from the outcome (the final state of the environment). For a booking agent, the outcome is whether the booking exists in the database, not the sentence announcing it.

A single run proves nothing. Outputs vary between runs. Anthropic offers two measures. pass@k is the probability of at least one success in k trials. pass^k is the probability that all k trials succeed. With a 75% per-trial success rate and 3 trials, pass^3 falls to about 42%. For a customer-facing agent, pass^k is often what matters: users expect the same good behaviour every time.

The minimum vocabulary

TermDefinitionSource
TaskA test with defined inputs and success criteriaAnthropic
TrialOne attempt at a task; several are runAnthropic
GraderLogic that scores one aspect of performanceAnthropic
Transcript, traceThe complete record of a trialAnthropic
OutcomeThe final state of the environment at the end of the trialAnthropic
Dataset, exampleA collection of input / reference output pairsLangSmith
ExperimentThe results of one application version on a datasetLangSmith

Sources: Anthropic, "Demystifying evals for AI agents", 9 January 2026; LangSmith, "Evaluation concepts". Consulted on 29 October 2026.

LangSmith adds a useful distinction. Offline evaluation runs before deployment on examples that have a reference output: it is used for benchmarking and regression testing. Online evaluation runs on real traffic, without references: it is used for monitoring. Each feeds the other: a problem seen in production becomes a new offline example.

Three families of graders, to combine

GraderExamplesStrengthsWeaknesses
Code-basedString matching, unit tests, state checks, tool call verificationFast, cheap, reproducibleBrittle to valid variations, lacks nuance
Model-based (LLM judge)Rubric scoring, pairwise comparison, reference-based evaluationFlexible, handles open-ended answersNon-deterministic, more expensive, needs calibration against human scores
HumanSubject-matter expert review, spot-check samplingGold-standard qualitySlow and expensive

Source: Anthropic, "Demystifying evals for AI agents", section "Types of graders for agents".

LLM judges have documented biases. The paper by Zheng et al. (2023), which introduced MT-Bench and Chatbot Arena, examines position, verbosity and self-enhancement biases, as well as limited reasoning ability. It also reports over 80% agreement between GPT-4 and human preferences on its own benchmarks, the same level as agreement between humans. Anthropic recommends calibrating judges against human experts, allowing them to answer "unknown" when information is missing, and grading each dimension with a separate judge.

The trap to avoid: grading the path instead of the outcome

The instinct is to check that the agent called the right tools in the right order. Anthropic considers this too rigid: agents regularly find valid paths that test designers did not anticipate. It is better to grade what the agent produced than the path it took, and to give partial credit on multi-step tasks.

There is one exception: actions the agent must never take. There, checking tool calls remains essential. OWASP lists this risk as Excessive Agency (LLM06:2025): damaging actions triggered by unexpected, ambiguous or manipulated model output, whatever the cause. Its root causes are excessive functionality, permissions or autonomy.

An agent test set should therefore be balanced, as Anthropic recommends: tasks where the action should happen, and tasks where it should not. An agent tested only on "should act" learns to act too often. To decide where to place a human approval before a sensitive action, see our article AI agents connected to business tools: where to place human approvals.

Capability and regression: two suites, two roles

Anthropic distinguishes two kinds of suites.

  • The capability suite asks what the agent does well. It starts with a low pass rate and gives the team a hill to climb.
  • The regression suite asks whether the agent still handles everything it used to. Its pass rate should be close to 100%. A drop signals that something is broken.

A capability task that passes reliably can then graduate to the regression suite. LangSmith states the same principle from the tooling side: an evaluation metric can become a test, for example by requiring a new version to do at least as well as the baseline.

To start, Anthropic considers 20 to 50 simple tasks drawn from real failures enough. Begin with the checks you already run by hand before each release, and with support tickets if the agent is already live.

The regression gate in practice: an open-source example

A regression gate compares the candidate version with a baseline, on the same dataset, and fails the integration pipeline when the gap exceeds a threshold. Ignition Agent Trainer, IgnitionAI's open-source project, provides an example.

What the project is, according to its README: an evaluation, experimentation and optimisation layer for TypeScript AI agents. It does not replace LangChain, LangGraph, Mastra or the Vercel AI SDK. It wraps an existing agent in a standard loop: dataset, adapter, trace collection, rewards and scorers, leaderboard, optimisation.

The repository's ci-regression-gate example replays an experiment, compares it with a recorded baseline and fails the process if the score drops too much, if latency or cost rises too much, or if a baseline variant disappears. The example's thresholds:

ThresholdValue in the example
Maximum score drop0.03
Maximum latency increase150 ms
Maximum cost increase$0.001

Source: IgnitionAI/ignition-agent-trainer repository, examples/ci-regression-gate/README.md, consulted on 29 October 2026.

The limits, as the repository states them. The README still describes the project as an "initial scaffold". Release notes for v0.1.0-alpha.0 exist in the repository, but no GitHub release is published. The examples use synthetic or mocked data, and no quantified results are published. These thresholds belong to an example: yours depend on your task, your budget and the variance you observe between trials.

The same principle applies with other tools. Anthropic lists several evaluation frameworks, open source and commercial, and concludes that they are only as good as the tasks you run through them.

Decision grid

Your situationApproach to consider
Prototype with no tests20 to 50 tasks drawn from real failures, code-based graders first
Agent that changes data or triggers actionsFinal-outcome checks, "must not act" tasks, checks on forbidden tool calls
Open-ended answers (writing, synthesis)LLM judge with a rubric, calibrated on a human-scored sample
Planned model or prompt changeRegression suite replayed over several trials, gate on score, latency and cost
Agent already in productionOnline evaluation on real traffic, with every observed failure added to the offline dataset

Basis of the grid: Anthropic, LangSmith and OWASP, cited below. Matching situations to approaches is our own reading.

To trace what the agent does at each step, see also our article on AI agent traceability.

FAQ

  • How many tasks are needed to evaluate an AI agent?

    Anthropic considers 20 to 50 simple tasks drawn from real failures enough to start. A more mature agent needs larger and harder suites, because the differences to detect become smaller.

  • Should the order of tool calls be checked?

    Usually not: Anthropic recommends grading what the agent produced rather than the path it took. The exception is forbidden or sensitive actions, which must be checked explicitly.

  • Can an LLM judge replace human review?

    No. It is useful for open-ended answers, but its position, verbosity and self-enhancement biases are documented. Calibrate it against human scores and regularly review a sample.

  • What is the difference between pass@k and pass^k?

    pass@k is the probability of at least one success in k trials. pass^k is the probability that all k trials succeed. For a customer-facing agent, pass^k is often the more demanding and more useful measure.

  • Is Ignition Agent Trainer production-ready?

    No. The project is in alpha, its README describes it as an "initial scaffold", no GitHub release is published and no quantified results are published. It serves here as an example of a regression gate.

Methodology and sources

Technical sources

Open-source project cited

  • IgnitionAI/ignition-agent-trainer, MIT licence, consulted on 29 October 2026: README and examples/ci-regression-gate/README.md. Project published by IgnitionAI, in alpha, with no GitHub release and no published quantified results.

Interpretation

The advice, the order of steps and the decision grid are our own reading of these sources. The article contains no market statistics and no quantified IgnitionAI estimates. The figures quoted (75% and 42%, 20 to 50 tasks, over 80% agreement) are those of the named sources.

Corrections policy

If you spot a factual error or an outdated source, report it to contact@ignitionai.fr. IgnitionAI's editorial policy provides for reviewing every report and, for a substantial error, a visible correction note at the top of the article.

Last source review

29 October 2026.


Preparing to move an AI agent into production? Describe your context to IgnitionAI. A first conversation can help turn your manual checks into a first task suite and a regression gate.

Contact

AI agents that answer from your documents.

Sourced answers, a written scope before we start, and a fixed price written before the first piece of work.

Free 30-minute call, no commitment.