Orchestration: state machines, agents and multi-agent systems

Three architectures

Everyone wants "an agent." The better question is how much freedom the model should have. There are three common answers.

More freedom means more flexibility, and less predictability.

1. State machine (flow)

You define explicit states (collect order number, confirm item, offer options, finish) and transitions between them. The LLM may understand messy input and write natural replies, but your code decides what happens next.

Strengths: predictable, testable, auditable. Limits: rigid, you must anticipate paths.

2. Single agent with tools

One model runs in a loop: think, call a tool, read the result, repeat until done. It chooses the steps.

Strengths: flexible, handles unforeseen requests. Limits: less predictable, harder to guarantee behavior, can loop or wander.

3. Multi-agent

Several agents, each with its own instructions, tools and context, coordinated by a supervisor or by handoffs. Examples: a triage agent that routes to billing, technical and sales agents.

Strengths: separation of concerns, different permissions per agent, parallel work, smaller contexts. Limits: more cost and latency, coordination bugs, harder debugging.

A decision framework

Ask in order and stop at the first yes:

1. Is the process well defined and the stakes high? (payments, identity, legal commitments) Use a state machine.
2. Are the steps unknowable in advance, but the tools few? Use a single agent with tight tools and limits.
3. Do you need different permissions, knowledge domains or parallel work that one context can't hold? Consider multi-agent.
4. Is it just questions about content? Plain RAG (module 6) may be enough, no agent needed.

The hybrid pattern: a deterministic skeleton with the LLM at the joints

The most robust designs let code own the what and the model own the language.

from enum import Enum, auto

class S(Enum):
    ASK_BOOKING = auto(); ASK_DATE = auto(); CONFIRM = auto(); DONE = auto(); HANDOFF = auto()

def step(state, user_text, ctx, llm_extract, tools):
    """One turn. Code decides the transition; the LLM only extracts and phrases."""
    found = llm_extract(user_text)                       # {"booking_code": ..., "new_date": ...}
    ctx.update({k: v for k, v in found.items() if v})

    if state == S.ASK_BOOKING:
        if not ctx.get("booking_code"):
            ctx["misses"] = ctx.get("misses", 0) + 1
            return (S.HANDOFF, "I'm having trouble. Let me connect you to a person.") if ctx["misses"] >= 2 \
                   else (S.ASK_BOOKING, "What's your six-character booking code?")
        ctx["fare"] = tools["get_booking"](ctx["booking_code"])      # grounded in your system
        return S.ASK_DATE, f"Found it. Which date would you like instead?"

    if state == S.ASK_DATE:
        if not ctx.get("new_date"):
            return S.ASK_DATE, "Which date would you like?"
        return S.CONFIRM, f"Move your trip to {ctx['new_date']}? Please reply yes to confirm."

    if state == S.CONFIRM:
        if user_text.strip().lower() in {"yes", "y", "confirm"}:      # explicit confirmation gate
            tools["change_booking"](ctx["booking_code"], ctx["new_date"])
            return S.DONE, "Done. Your trip has been moved."
        return S.ASK_DATE, "No problem. Which date would you like?"
    return state, "How else can I help?"

Notice the guard rails: a miss counter that triggers handoff, a confirmation gate before the write action, and tool calls that come from code, not from the model's imagination. You could replace llm_extract with a small model, and swap the reply strings for model-generated text constrained by the state.

Routing with confidence

A router classifies each request and sends it down the right path. This replaces the intent-platform-plus-LLM pattern from the first edition of this course with a version that works with any stack.

def route(message, history):
    result = classify(message, history)        # returns {"route": "...", "confidence": 0.0-1.0}
    if result["confidence"] < 0.5:
        return "clarify"                        # ask a question rather than guess
    if result["route"] == "faq":
        return "rag"                            # grounded answer (module 6)
    if result["route"] in {"change_booking", "cancel_booking"}:
        return "flow"                           # state machine for transactions
    if result["route"] == "open_request":
        return "agent"                          # tools with limits
    return "human"

Tune the thresholds with your evaluation set, and log every routing decision so you can see where it goes wrong.

When multi-agent is worth it

Good reasons: different permissions (the billing agent can issue refunds, the FAQ agent cannot); separate knowledge domains that overflow one context; parallel subtasks; independent teams owning separate agents (where A2A helps).

Poor reasons: it looks sophisticated; the prompt got long (try splitting the prompt first).

If you do build one, keep the number of agents small, make handoffs explicit and logged, share state through your own store rather than agents chatting freely, set budgets for steps and cost, and test each agent alone before testing them together.

Memory and state

• Conversation memory: what you send each turn. Summarize long histories instead of replaying everything.
• Session state: structured facts (booking code, chosen date) in your own store, not only in the transcript.
• Long-term memory: preferences across sessions. Collect with consent, let users view and delete it, and keep it minimal.

The framework landscape (October 2026)

Every major lab and cloud now ships an agent toolkit, and several independent ones are popular. As of this writing, commonly compared options include LangGraph, the OpenAI Agents SDK, the Claude Agent SDK, Google's Agent Development Kit, Microsoft Agent Framework (which merged AutoGen and Semantic Kernel), CrewAI, Pydantic AI, Mastra and LlamaIndex. They fall into a few groups: graph engines for stateful workflows, vendor SDKs tuned for one model family, and managed runtimes.

Don't pick one because it is trending. Pick by language and team, by how much durability and tracing you need, and by how committed you are to one vendor. And remember the point of this module: the patterns (states, tools, routing, handoffs, budgets, tests) outlive any framework.