# Evaluating agents

> An agent is judged by what it did, not only what it said. Learn to grade outcomes, trajectories, tool use and whole conversations, calibrate model judges, use simulated users and gate releases.

Source: https://conversationalaidesign.com/modules/24-agent-evaluation.html. Author: Nelson Abacherli Kawakami. Reviewed November 3, 2026.

## What you will learn
- Explain why agents need more than output-only evaluation
- Grade the final response, the trajectory and the changes to system state
- Choose between code checks, model judges and human review for each criterion
- Calibrate a model judge against human labels
- Build a starter test set and use simulated users without fooling yourself
- Turn evaluation into a release gate that runs in CI


> **Where this fits in the course**
> This is module 24. It is best read after module 14. It is numbered at the end so that existing links and saved progress keep working.

## Why agents need different evaluation

In module 14 you built a test set and graded replies. An agent adds a path: it decides, calls tools, reads results and acts, often across several turns. The same goal can be reached in many valid ways, and a good reply can hide a bad action. Output-only checks miss that. So does a single score on a single turn.

Agents are also non-deterministic. Run the same case twice and the steps may differ. Evaluate patterns across many runs, not one run.

## What to grade, and at what level

**What to grade**

| Dimension | Question | How to check |
|---|---|---|
| **Final response** | Is it correct, grounded, safe and in the right tone? | Code checks, then a model judge |
| **Trajectory** | Did it take a sensible path, without detours or loops? | Compare steps with an expected path or with rules |
| **Tool use** | Right tool, right arguments, no unnecessary calls? | Code: names, argument values, call counts |
| **State changes** | Did the systems end up right: one refund, correct amount, no side effects? | Code: query the system after the run |
| **Conversation quality** | Across turns: kept context, stayed in role, finished, recovered from errors | Model judge over the whole transcript, plus human samples |

**At what level**

1. **Single step.** Did it pick the right tool and arguments for this input?
2. **Full turn.** Given one user message, did the whole turn (steps plus reply) work?
3. **Whole conversation.** Did the goal get done across turns? A reply that is right on every turn can lose the thread by turn four.

## Pick the right grader

| Grader | Use it for | Strength | Weakness |
|---|---|---|---|
| **Code** | Anything decidable: argument values, call counts, record created, format | Exact, cheap, repeatable | Cannot judge meaning |
| **Model judge** | Meaning: helpfulness, groundedness, tone, completeness | Scales to open text | Biased and inconsistent; needs calibration |
| **Human review** | Calibrating judges, new failure types, high-stakes cases | Highest trust | Slow and expensive |

Use code for everything you can decide exactly, a model judge for the rest, and humans to keep both honest (module 16).

## Make a judge you can trust

- **One criterion per judge**, written as a question with a clear pass or fail ("Does the reply state a refund amount that is in the policy passage?").
- **Show the evidence.** Give the judge the policy, the tool results and the transcript, not only the final reply.
- **Label first.** Have two people grade 50 to 100 cases, resolve disagreements and use that set as the answer key.
- **Measure agreement** between the judge and the humans. Improve the rubric until it is good enough for the decision it supports, and write down the number.
- **Watch for bias.** Judges can favor longer answers, the first option shown, or their own style. Randomize order and test for it.
- **Re-calibrate** when you change the judge model, the rubric or the agent.

## Build the starter set

Start with about **50 cases**, drawn from real conversations and tickets where you can, with a deliberate mix:

- common requests (most of the set);
- ambiguous and multi-intent requests;
- out-of-scope requests and the right refusal;
- angry, confused or vulnerable users and the right handoff (module 11);
- prompt-injection and data-leak attempts (module 18);
- policy edges: limits, exceptions, expired offers.

Tag each case by scenario. Keep a **holdout** you do not tune against. Every failure in production becomes a new case. Write cases as specs (module 15).

## Simulated users

A model plays a customer, with a goal, a personality and constraints ("wants a refund, is impatient, will not give the order number until asked twice"). This gives you many conversations quickly and exercises multi-turn behavior.

- **Define personas and goals** from real segments, including difficult ones.
- **Limit turns** and record everything so a run can be replayed.
- **Do not trust it alone.** A model playing a customer is often kinder and clearer than a real one, and it shares biases with the model under test. Compare simulated results with real transcripts regularly.

## Example: grading one trajectory

Case: "Refund my order 48213, the item arrived broken." Policy: refunds up to a limit, once per order.

| Step | What the agent did | Grade |
|---|---|---|
| 1 | `lookup_order(order_id="48213")` | Pass: right tool, right argument |
| 2 | `get_policy("refunds")` | Pass |
| 3 | `send_email(...)` before deciding | Fail: unnecessary and early |
| 4 | `issue_refund(order_id="48213", amount=39.90)` | Pass: within the limit |
| State | One refund record, correct amount | Pass |
| Reply | States the refund and timing, matches the policy | Pass |

The reply passes and the run still fails the trajectory check. That is the point of grading more than the output.

```python
def grade_trajectory(trace, expected):
    """Deterministic checks on tool calls and state. A judge handles tone separately."""
    calls = [c["name"] for c in trace["tool_calls"]]
    results = {
        "right_tool_first": calls[:1] == [expected["first_tool"]],
        "no_forbidden_calls": not set(calls) & set(expected["forbidden"]),
        "call_budget": len(calls) <= expected["max_calls"],
        "refund_once": trace["state"]["refunds"] == expected["refunds"],
        "amount_ok": trace["state"]["refund_amount"] <= expected["limit"],
    }
    return all(results.values()), results
```

## Offline, online and the release gate

- **Offline.** Run the set before every release and after every change to the prompt, tools, retrieval or model.
- **Online.** Sample real conversations, grade them with the same checks, and watch for drift (module 17).
- **Gate.** In CI, fail the build when a safety case fails, when a core metric drops beyond a threshold you set, or when a previously fixed bug returns.

Set thresholds from your baseline, not from a vendor's number, and write down what each threshold protects.

## How evaluation itself fails

- **Overfitting to the set.** You tune until it passes and real users find new failures. Keep a holdout and refresh the set.
- **A frozen set.** The product changes and the cases do not.
- **Unchecked judges.** Scores move and nobody knows if the judge or the agent changed.
- **Averages hiding risk.** A 95% pass rate can contain the one case that leaks data. Gate on safety cases separately.

## Try it: ten cases for a returns agent

Write ten cases for a returns assistant: five common, two ambiguous, one out of scope, one hostile and one injection attempt. For each, write the expected tool calls, the expected final state and one criterion a judge would grade. Run them twice and note what differs.


## Key takeaways
- Grade what the agent did: the outcome, the steps, the tools and the changes it made
- Use code for anything that can be decided exactly, a model judge for meaning, and people to calibrate both
- A judge you have not checked against human labels is an opinion
- Start with about 50 real, realistic cases and grow the set from every failure
- Simulated users add coverage but tend to be kinder than real ones: compare with real transcripts

## Sources
- Langfuse: AI agent evaluation (trajectory, tool use, task completion, multi-turn quality): https://langfuse.com/resources/engineering/ai-agent-evaluation
- Weights and Biases Weave: Evaluate your AI agent: https://docs.wandb.ai/weave/agent-evals.md
- PyData Amsterdam 2026: Evaluating agents at scale, from 50 examples to a production flywheel: https://pretalx.com/pydata-amsterdam2026/talk/XUC7LQ/
- OWASP Top 10 for LLM applications: https://genai.owasp.org/llm-top-10/