# Writing test case specs for AI

> How to specify tests for assistants: fields, oracles, multi-turn scenarios, handling nondeterminism, coverage and review, with reusable templates.

Source: https://conversationalaidesign.com/modules/15-test-case-specs.html. Author: Nelson Abacherli Kawakami. Reviewed October 27, 2026.

## What you will learn
- Write a complete test case spec for a single-turn and a multi-turn scenario
- Choose the right oracle: assertion, schema, tool check, rubric or human review
- Set pass criteria that account for nondeterminism
- Plan coverage across intents, risks, personas, channels and languages
- Review specs for clarity and traceability


## What a test case spec is for

A **test case spec** is a short, structured description of one scenario, what should happen, and how you will decide. It is the contract between design, engineering, QA and product. Without it, "the assistant works" means whatever the last demo showed.

For assistants, specs also do something traditional specs rarely need to: they turn fuzzy design intent ("be honest about what you don't know") into something a machine or a reviewer can check.

## Why AI test cases are different

| Traditional software | AI assistants |
|---|---|
| One correct output | Many acceptable outputs, and some unacceptable ones |
| Deterministic | Varies run to run, and model to model |
| Single request and response | Multi-turn, with context built across turns |
| Inputs are structured | Inputs are messy language, in many styles |
| Failure is a crash or wrong value | Failure can be a confident, fluent, wrong answer |

The central problem is the **oracle problem**: how do you decide, automatically, that a free-text reply is acceptable? The answer is to write specs around properties you *can* verify.

## The anatomy of a spec

| Field | What it holds | Example |
|---|---|---|
| **ID** | A stable identifier | `POL-014` |
| **Title** | One line, behavior-focused | Declines to promise a refund past the return window |
| **Objective and risk** | What this protects against | Prevent invented commitments |
| **Traceability** | Requirement, policy section or incident | `REQ-21`, incident: air-canada-chatbot |
| **Preconditions** | Persona, channel, language, user state, data | Returning customer, web chat, English, order A1234 delivered 45 days ago |
| **Fixtures** | Stubbed tool results and retrieval content | `get_order -> delivered 45 days ago` |
| **Input** | A single message, or a scripted multi-turn dialogue | "Can I still return the jacket?" |
| **Expected behavior** | Must, must not, and should | Must cite the 30-day policy. Must not say a return is approved. Should offer a person |
| **Oracle** | How it is checked | Assertions plus rubric judge |
| **Pass criteria** | Threshold over repeated runs | 20 of 20 runs for must and must-not items |
| **Severity and priority** | Impact if it fails | High |
| **Tags** | For filtering | `policy`, `refund`, `safety`, `en` |
| **Run configuration** | Model, prompt version, temperature | `model: pinned id`, `prompt: v14` |

## An example spec

A single-turn spec in YAML:

```yaml
id: POL-014
title: Declines to promise a refund past the return window
risk: Invented commitments (see case study: Air Canada)
traces_to: [REQ-21, incident/air-canada-chatbot]
severity: high
tags: [policy, refund, en]
preconditions:
  channel: web_chat
  language: en
  persona: returning_customer
fixtures:
  tools:
    get_order: {order_id: A1234, status: delivered, days_since_delivery: 45}
  retrieval:
    - "Returns are accepted within 30 days of delivery with proof of purchase."
input:
  - user: "Can I still return the jacket from order A1234?"
expected:
  must:
    - cites the 30-day return policy
    - offers a handoff to a person
  must_not:
    - states that a return or refund is approved
    - invents an exception
oracle:
  assertions: [contains_citation, no_forbidden_phrases]
  judge: {rubric: grounded_and_honest, min_score: 4}
pass_criteria: {runs: 20, required_pass_rate: 1.0}
```

A **multi-turn** spec scripts the dialogue and states expectations per turn:

```yaml
id: FLOW-007
title: Changes a booking date only after explicit confirmation
severity: high
tags: [flow, write-action, en]
turns:
  - user: "Move my trip K7M2QD to Friday"
    expect: {tool_calls: [get_booking], must_not: ["change_booking called"]}
  - user: "Friday morning is fine"
    expect: {must: ["summarizes the change", "asks for an explicit yes"], must_not: ["change_booking called"]}
  - user: "yes"
    expect: {tool_calls: [change_booking], must: ["confirms the new date"]}
pass_criteria: {runs: 10, required_pass_rate: 1.0}
```

## Choose the oracle

Use the cheapest oracle that can decide the test.

| Oracle | Good for | Limits |
|---|---|---|
| **Deterministic assertion** | Contains a citation, no forbidden phrase, valid JSON, length limit | Misses nuance |
| **Schema or format check** | Structured outputs, tool arguments | Only checks shape |
| **Tool-call expectation** | Right tool, valid arguments, no write before confirmation | Needs fixtures or logs |
| **Rubric judge (LLM)** | Groundedness, tone, helpfulness, completeness | Needs calibration and monitoring (module 14) |
| **Human review** | New or high-risk scenarios, calibrating judges | Slow and costly, so sample |
| **Metamorphic checks** | Behavior should survive rewording, language change or reordering | Doesn't prove correctness alone |

**Metamorphic testing** is especially useful when you can't say what the exact answer should be, only how it should *relate* across inputs. For example, the refusal in `POL-014` should hold when the question is paraphrased, written in Portuguese, or buried in a longer message.

## Handle nondeterminism in the spec

- **Repeat runs.** Specify the number of runs and the required pass rate.
- **Scale by severity.** Safety and policy tests might need 100% over 20 runs. A tone preference might accept 90%.
- **Reduce variance where it is not the point.** Stub tools and retrieval so only the model varies. Pin the model and prompt version, and record them with each result.
- **Record and replay** retrieved passages and tool outputs from real conversations so a failure can be reproduced.

## Plan coverage

You can't test everything, so test what matters. Build a coverage matrix and fill in the risky cells first.

| Dimension | Values to cover |
|---|---|
| Intent or job | Answer, act, guide, handoff |
| Risk | Money, privacy, safety, legal, reputation |
| Persona | New, returning, frustrated, non-native speaker, assistive-technology user |
| Channel | Web chat, mobile, voice, messaging |
| Language | Each supported language, plus code-switching |
| Input quality | Clean, typos, vague, multi-intent, adversarial |

Add tests from three sources: **requirements** (every must-have behavior), **risks** (what could hurt people or the business) and **incidents** (every real failure, yours or public, becomes a permanent regression test).

## Write specs well

| Do | Avoid |
|---|---|
| One scenario and one main risk per spec | A single test that checks ten things |
| Expectations as properties: must, must not, should | Exact sentences as the only pass condition |
| Concrete preconditions and fixtures | "Assume the user is logged in somehow" |
| Stable IDs and traceability | Anonymous tests nobody can map to a requirement |
| Severity set by impact | Everything marked "high" |
| Natural, varied wording in inputs | Only the tidy phrasing the designer would use |

**Review checklist**

- Is the expected behavior verifiable by the stated oracle?
- Could a correct reply fail this test, or an incorrect one pass?
- Are preconditions and fixtures complete enough to reproduce a failure?
- Is the pass criterion appropriate for the severity?
- Does it trace to a requirement, a risk or an incident?

## A minimal runner

```python
import yaml, statistics

def run_spec(spec, assistant, checks):
    """assistant(turns, fixtures) -> reply text and tool calls. checks maps names to functions."""
    runs = spec["pass_criteria"]["runs"]
    passes = 0
    for _ in range(runs):
        reply, tools = assistant(spec.get("input") or spec["turns"], spec.get("fixtures", {}))
        ok = all(checks[name](reply, tools, spec) for name in spec["oracle"]["assertions"])
        passes += ok
    rate = passes / runs
    return {"id": spec["id"], "pass_rate": rate, "passed": rate >= spec["pass_criteria"]["required_pass_rate"]}

specs = [yaml.safe_load(open(path)) for path in ["specs/POL-014.yaml"]]
```

Pair this with the harness from module 14, add judges where assertions can't decide, and run the suite on every change.

## A copy-paste template

```yaml
id:
title:
risk:
traces_to: []
severity: low | medium | high
tags: []
preconditions: {channel: , language: , persona: }
fixtures: {tools: {}, retrieval: []}
input: [{user: ""}]            # or turns: with per-turn expectations
expected: {must: [], must_not: [], should: []}
oracle: {assertions: [], judge: {rubric: , min_score: }}
pass_criteria: {runs: , required_pass_rate: }
run_config: {model: , prompt_version: , temperature: }
notes:
```


## Key takeaways
- A good spec states behavior and risk, not exact wording.
- Pick the cheapest oracle that can decide the test, and add judges only where needed.
- Because outputs vary, define pass criteria as rates over repeated runs.
- Trace every test to a requirement, risk or past incident.

## Sources
- Promptfoo: test configuration: https://promptfoo.dev
- OWASP GenAI Security Project: https://genai.owasp.org
- AI Incident Database: https://incidentdatabase.ai