How to Measure AI Agent Quality in Production

An agent can produce fluent messages while quietly failing the job. Production quality therefore starts with a unit that matters to the user: a completed task, not a model response.

Suppose an agent researches a lead, updates a CRM, and drafts an email. “The run finished” is not success. The correct company must be identified, claims need evidence, the right record must be updated once, and the draft must respect policy.

Build an outcome record

For every run, record a stable workflow version, input class, permitted tools, final status, elapsed time, model and tool cost, human interventions, and a link to privacy-safe trace data. Do not copy every sensitive prompt into analytics by default.

Use statuses that describe reality: completed, rejected_by_policy, needs_review, failed_transient, failed_permanent, and unknown_external_outcome are more useful than a single error flag.

Five measures that belong together

Accepted-task rate

Count results that a user or objective checker accepts without substantive correction. Define “substantive” for the workflow. Fixing punctuation is different from replacing a fabricated claim.

Critical-failure rate

Track severe errors separately: unauthorized access, an unsupported high-impact claim, disclosure of protected data, a duplicated payment, or an external message sent to the wrong person. A high average score must never hide one of these.

Human rescue and correction

Measure how often a reviewer changes the proposed action, takes over the workflow, or repeats work manually. Automation that moves effort into a hidden review queue has not necessarily saved time.

End-to-end latency

Measure from the user's trigger to the usable business result, including queues, tool calls, retries, and approval. Model response time alone misses most of a long workflow.

Cost per accepted result

Add inference, paid tools, compute, and estimated review effort, then divide by accepted outcomes. Cost per token cannot compare a cheap run that often needs repair with a more expensive run that finishes correctly.

Create an evaluation set from real work

Start with 30–100 reviewed examples rather than thousands of synthetic prompts. Include common tasks, costly edge cases, past incidents, ambiguous requests, missing data, tool failures, and attempts to exceed authorization.

Attach observable grading criteria to each example. If two reviewers routinely disagree, the rubric needs work. Preserve important slices—language, customer type, workflow variant, risk level—because an improved overall average can conceal a regression for one group.

Grade tools as first-class behavior

For each tool call, check:

• Was the tool allowed for this user and task?
• Were the arguments valid and grounded in available information?
• Was a write previewed or approved when required?
• Did the agent interpret the result correctly?
• Could a timeout cause the action to be repeated?
• Did the workflow stop after achieving the outcome?

Deterministic checks should grade identifiers, schemas, totals, and state changes. Calibrated model graders can help with nuanced text at scale, while humans periodically audit both passes and failures.

Compare versions with paired runs

When changing a prompt, model, tool schema, or retrieval pipeline, run old and new versions on the same frozen examples. Change one major variable at a time. Report wins, losses, critical regressions, latency, and cost by slice.

An aggregate improvement is not an automatic release. Set hard gates for critical failures and minimum performance in important slices. Document any accepted regression with an owner and expiration date.

Monitor without turning customers into a test dataset

Sample production outcomes according to a declared privacy policy. Minimize retained content, restrict access, redact secrets, and honor deletion requirements. Feed novel, reviewed failures back into the evaluation set; do not blindly train on every interaction.

Use an alert for symptoms that require action: a sudden fall in completion, a rise in unknown outcomes, repeated policy rejections, unusual tool volume, or review queues older than their service target.

A weekly scorecard

One page is enough:

Add three reviewed failures with links to the responsible workflow version. Numbers show where quality moved; examples show why.

The purpose of evaluation is not to produce a flattering benchmark. It is to decide whether a specific version can be trusted with a specific amount of responsibility—and to detect quickly when that trust is no longer deserved.

Frequently asked questions

What is the best metric for AI agent quality?

Accepted-task rate is a strong primary metric, but it must be paired with critical failures, human rescue, latency, and cost per accepted result.

How many examples should an agent evaluation start with?

A carefully reviewed set of 30–100 representative and high-risk examples is often more useful than a large weak synthetic set.

Can a model grade another model's output?

Yes for calibrated, scalable judgment, but deterministic checks and periodic human audits are still needed, especially for high-impact failures.