Evaluation and observability

Why evaluation is the job

Conventional software behaves the same way every time. An assistant doesn't: wording varies, models are updated, and a prompt tweak that fixes one conversation can break ten others. Evaluation is how you know. In practice it is the difference between a demo and a product.

Build a test set

Start small and grow it. Fifty to a hundred well-chosen conversations beat thousands of random ones. Draw from:

• Your transcripts from module 1 and, once live, anonymized real conversations.
• Messy inputs: typos, two questions at once, changed minds, other languages.
• Out-of-scope requests: things the assistant must decline or hand off.
• Adversarial prompts: attempts to extract the system prompt, make it swear, or promise refunds (see module 18).
• Real incidents from the AI Incident Database and the case studies in this course.

Write each case as a proper spec (module 15), and label it with the expected behavior, not an exact reply: "asks for booking code," "cites the baggage policy," "declines and offers a person."

Measure what matters

• Task success: did the user's goal get done correctly?
• Groundedness: are claims supported by sources or tool results?
• Policy compliance: scope, tone and safety rules respected?
• Tool correctness: right tool, valid arguments, no unneeded calls?
• Handoff quality: triggered when it should be, with context?
• Latency and cost: per turn and per resolved conversation.

Automated checks, in two layers

Deterministic assertions are cheap and exact: the reply contains a citation, includes no forbidden promises, the tool was called with valid arguments, the output parses.

LLM judges grade what code can't: helpfulness, tone, faithfulness. Make them reliable by writing a rubric with concrete criteria, comparing two answers (pairwise) rather than scoring in a vacuum, calibrating against human labels, and watching for known biases such as favoring longer answers or the judge's own style.

import json

CASES = [
    {"id": "policy-bereavement", "user": "Can I get a bereavement refund after I've travelled?",
     "must_include": ["[1]"], "must_not_include": ["I've refunded", "guaranteed"], "expect_tool": None},
    {"id": "change-date", "user": "Move my trip K7M2QD to Friday",
     "must_include": [], "must_not_include": [], "expect_tool": "get_booking"},
]

def run_case(case, assistant, judge):
    reply, tools_called = assistant(case["user"])                 # your assistant under test
    problems = []
    problems += [f"missing {s!r}" for s in case["must_include"] if s not in reply]
    problems += [f"contains {s!r}" for s in case["must_not_include"] if s.lower() in reply.lower()]
    if case["expect_tool"] and case["expect_tool"] not in tools_called:
        problems.append(f"expected tool {case['expect_tool']}")
    verdict = judge(case["user"], reply)                          # e.g. {"helpful": 4, "grounded": True}
    if not verdict.get("grounded", True):
        problems.append("judge: not grounded")
    return {"id": case["id"], "pass": not problems, "problems": problems, "reply": reply}

def run_suite(assistant, judge, cases=CASES):
    results = [run_case(c, assistant, judge) for c in cases]
    passed = sum(r["pass"] for r in results)
    print(f"{passed}/{len(results)} passed")
    for r in results:
        if not r["pass"]: print(" -", r["id"], r["problems"])
    return results

Run this suite on every change to a prompt, model, tool or knowledge base, in your continuous-integration pipeline. A failing suite blocks the release.

Observe in production

For the data model, metrics and dashboards behind this, see module 17. For how testers work through all of it, see module 16.

Offline tests can't cover everything real users do. Instrument the live system:

• Trace every conversation: prompts, retrieved passages, tool calls and results, model and prompt versions, latency and cost.
• Sample and review: read a slice of real conversations weekly, especially low-rated and escalated ones.
• Collect feedback: a thumbs rating plus an optional reason, and "was this resolved?"
• Watch for drift: shifts in topics, rising escalations, latency or refusal rates after a provider change.
• Roll out gradually: canary releases and A/B tests, with your success metrics as the judge.
• Keep a kill switch: a way to disable the model path and fall back to a safe flow or a human within minutes.

Make it routine

1. Every bug becomes a test case.
2. Every prompt, model or knowledge change runs the suite first.
3. Someone owns the dashboard and the weekly review.
4. Share results with designers and product, not just engineers.

Tools such as Promptfoo, Langfuse and Ragas can run suites, traces and retrieval metrics so you don't build everything yourself.