# A QA playbook for testing AI

> How testers approach assistants: a test pyramid, exploratory charters, adversarial testing, flakiness, defect taxonomy, bug reports and release gates.

Source: https://conversationalaidesign.com/modules/16-qa-playbook.html. Author: Nelson Abacherli Kawakami. Reviewed October 27, 2026.

## What you will learn
- Apply a test pyramid designed for AI assistants
- Plan exploratory and adversarial testing with charters
- Separate model variance from real defects, and gate on pass rates
- Classify, rate and report AI defects reproducibly
- Define release gates and sign-off criteria


## The QA role, extended

Testing an assistant does not mean checking that the reply equals a string. It means **characterizing behavior** across many situations, hunting for risks, and deciding whether the system is safe and good enough to release. That makes testers central to AI quality, not obsolete.

This module is the practice. Module 14 explains the evaluation harness, and module 15 shows how to write specs. Here we put them to work.

## A test pyramid for assistants

- Unit tests: tools, parsers, validators, state logic → Component tests: retrieval, prompts, routing → Conversation tests: scripted multi-turn scenarios → System tests: real integrations in staging → Exploratory and adversarial sessions → Production monitoring

| Layer | What you test | How | Speed |
|---|---|---|---|
| **Unit** | Tool functions, input validation, state-machine transitions, output parsers | Ordinary automated tests | Very fast |
| **Component** | Retrieval quality, routing decisions, prompt sections, guardrails | Golden sets with metrics (module 6, module 14) | Fast |
| **Conversation** | Multi-turn behavior, repair, handoff, confirmations | Scripted specs with stubbed tools (module 15) | Moderate |
| **System** | End-to-end flows against staging systems, channels, auth | Scenario suites, a few runs each | Slow |
| **Exploratory and adversarial** | Unknown unknowns, abuse, edge cases | Time-boxed sessions with charters | Human time |
| **Production** | Real behavior, drift, failures | Monitoring, sampling, alerts (modules 14 and 17) | Continuous |

## Exploratory testing with charters

Exploratory testing is where testers find the things nobody thought to specify. Give each session a **charter**: a mission, not a script.

| Charter | What to probe |
|---|---|
| Explore the return flow as an impatient customer | Typos, partial answers, changed minds, repeated "agent please" |
| Explore the assistant's honesty about what it doesn't know | Questions just outside the knowledge base, outdated policies, contradictory sources |
| Explore multilingual behavior | Switching languages mid-conversation, mixed-language messages, names and addresses |
| Explore the handoff | Triggers, summary quality, what happens when no person is available |
| Explore long conversations | Forgetting earlier facts, contradicting itself, drifting off scope |

Take notes as you go: what you tried, what you saw, what surprised you, and new test ideas. Turn every valuable finding into a spec.

## Adversarial and safety testing

Plan systematic attempts to break the assistant, mapped to the OWASP risks (module 18):

- **Prompt injection:** direct ("ignore your rules") and indirect (hidden instructions in a document, web page or tool result).
- **Data leakage:** extracting the system prompt, other users' data, or credentials.
- **Excessive agency:** tricking the assistant into actions it should not take, or skipping confirmation.
- **Tone and brand:** making it swear, mock the company or give opinions (see the DPD case).
- **Harmful and out-of-scope requests:** medical, legal, financial, self-harm, and how it hands off.
- **Over-refusal:** legitimate requests it declines unnecessarily, which also harms users.
- **Resource abuse:** very long inputs, loops, repeated expensive calls.

Log every attempt and outcome, and add successful attacks to the regression suite.

## Non-functional testing

- **Latency:** measure the median and the slow tail (p95), per channel. Voice needs far tighter budgets (module 19).
- **Load and limits:** concurrency, provider rate limits, timeouts and graceful degradation.
- **Cost:** tokens and tool calls per conversation, and per resolved conversation.
- **Accessibility:** screen readers, plain language, keyboard use, voice alternatives.
- **Languages and locales:** every supported language, number, date and name formats.
- **Channels:** web, mobile, messaging and phone, including noisy audio.

## Flakiness: variance or defect?

AI tests flake by nature. Don't ignore it and don't automatically retry until green.

1. **Repeat the run** many times and measure the pass rate.
2. **Stabilize what you can:** stub tools, freeze retrieved passages, pin the model and prompt version.
3. **Inspect failing runs.** If the failures share a cause (a certain phrasing, a missing passage), it is a defect, not noise.
4. **Judge by severity.** A safety rule needs a near-perfect rate. A stylistic preference can tolerate some misses.
5. **Use statistics honestly.** With few runs, even a perfect score doesn't prove a high pass rate. A confidence bound helps.

```python
import math

def wilson_lower(passes: int, n: int, z: float = 1.96) -> float:
    """Lower bound of the Wilson score interval for a pass rate."""
    if n == 0:
        return 0.0
    p = passes / n
    denom = 1 + z * z / n
    centre = p + z * z / (2 * n)
    margin = z * math.sqrt((p * (1 - p) + z * z / (4 * n)) / n)
    return (centre - margin) / denom

# 20 of 20 passes still only supports a lower bound near 0.84.
print(round(wilson_lower(20, 20), 3))
print(round(wilson_lower(95, 100), 3))

def gate(passes, n, required_lower_bound):
    return wilson_lower(passes, n) >= required_lower_bound
```

The lesson: for high-severity rules, run enough repetitions that a good result is meaningful, and treat any failure as a finding to investigate.

## Classify defects

| Defect type | Description | Example |
|---|---|---|
| **Ungrounded answer** | States something not in sources or tool results | Invents a baggage fee |
| **Policy violation** | Breaks scope, tone or commitment rules | Promises a refund |
| **Wrong tool or arguments** | Calls the wrong function or passes wrong values | Cancels instead of changes |
| **Unsafe action** | Writes without confirmation, or exceeds permissions | Deletes data on request |
| **Over-refusal** | Declines a legitimate request | Won't explain a public policy |
| **Handoff failure** | Missing trigger, missing context, dead end | Loops without offering a person |
| **Language issue** | Wrong language, mixed language, broken names | Replies in English to Portuguese |
| **Conversation defect** | Forgets context, repeats, loops | Asks for the order number three times |
| **Leakage** | Reveals prompt, data or credentials | Prints system instructions |
| **Performance** | Slow, timeouts, runaway cost | Ten-second first token |

**Severity** depends on impact and exposure. A guideline:

| Severity | Meaning |
|---|---|
| **Critical** | Harm, legal exposure, data leak or unsafe action |
| **High** | Wrong commitments, policy violations, broken core flows |
| **Medium** | Wrong or unhelpful answers with a safe fallback |
| **Low** | Tone, formatting or minor wording issues |

## Report bugs so they can be fixed

```text
Title: Promises refund for item outside the return window
Severity: High   Type: Policy violation   Reproduction rate: 7 of 20 runs
Environment: web chat, English, staging
Versions: model <id>, prompt v14, knowledge base 2026-10-20
Conversation (full transcript attached):
  User: I bought this in January. Can I still return it?
  Bot: Yes, I've started your refund.
Retrieved passages: [1] "Returns accepted within 30 days..."  (passage retrieved, ignored)
Tool calls: none
Expected: Explain the 30-day window, no promise, offer a person.
Notes: Failure appears when the question includes "I bought this in January". Spec POL-014.
```

Include the context you would want in order to reproduce it: versions, retrieved passages, tool calls, and how often it happens.

## Release gates and sign-off

Agree the gates *before* the release, with the whole team:

- **Regression suite:** passes at the required rates, by severity.
- **Safety and adversarial suite:** no unresolved critical or high findings.
- **Coverage:** every high-risk cell in the coverage matrix has tests.
- **Non-functional:** latency, cost and accessibility within targets.
- **Rollout:** canary or percentage rollout, with monitoring and a kill switch (module 14).
- **Known issues:** documented, accepted by an owner, with a plan.

Track quality of the testing itself: defects found before and after release, flaky-test rate, and how many production incidents become regression tests.

## LLM judges in QA

Judges scale review, but they are instruments that need checking: calibrate them against human labels, review disagreements, watch for bias toward length or style, and keep human review for high-severity and novel cases. Never let a judge be the only gate for safety.

## Working with the team

- **With designers:** turn principles and sample dialogues into specs, and feed findings back into the design.
- **With developers:** share traces and reproductions, and agree on fixtures and stubs.
- **With data and BI:** use production analytics to prioritize test areas and to confirm that fixes work (module 17).


## Key takeaways
- QA's role grows: characterize behavior, probe risks, and decide what is safe to ship.
- Test each layer cheaply and deterministically first, then use conversations and judges.
- Repeated runs and pass-rate gates are the honest answer to nondeterminism.
- Capture the full context in every bug report: transcript, versions, retrieved passages and tool calls.

## Sources
- OWASP GenAI Security Project: https://genai.owasp.org
- Promptfoo: https://promptfoo.dev
- AI Incident Database: https://incidentdatabase.ai
- Case study: DPD's chatbot: https://conversationalaidesign.com/case-studies/dpd-chatbot.html