Context engineering and memory

From prompts to context

A prompt is the text you write. Context is everything the model can see when it produces its next output: your instructions, the conversation so far, documents you retrieved, the results of tool calls, and anything you remembered from earlier sessions. Prompt engineering tunes the first part. Context engineering designs all of it, on every step, as a system.

The shift matters because assistants and agents run for many turns and call tools. A perfect prompt inside a cluttered context still produces poor answers, and a plain prompt inside a clean, relevant context often works well. The model is the runtime. The context is what you give it to run on.

What goes into the context

Ask of each part: is it needed for this step, is it current, is it trusted, and what does it cost in tokens and latency?

How long conversations go wrong

• Burying. An important rule sits among thousands of tokens of history and is followed less reliably.
• Distraction. Irrelevant passages or many similar tools make the model choose badly.
• Staleness. An old answer, price or policy stays in the context after it changed.
• Clash. Two sources disagree and the model picks one (module 7).
• Poisoning. Untrusted text, such as a retrieved page, a ticket or a saved memory, carries instructions (module 18).
• Cost and latency. Every token is paid for and processed again on each step.

Models can lose details in the middle of long contexts and give extra weight to the most recent text. Degradation as irrelevant text accumulates is often called context rot. Do not assume a larger window removes the problem. Test it.

Four moves: write, select, compress, isolate

Most problems need one or two moves. Start with select (retrieve less, trim tool results), then compress, and use isolation when one context is serving jobs that do not belong together.

def build_context(system, state, memory, history, passages, tool_results, budget=6000):
    """Assemble context in priority order. The fixed parts stop at the budget; recent turns are always kept and older ones are summarized."""
    parts = [("system", system), ("state", state), ("memory", memory)]
    parts += [("passage", p) for p in passages[:4]]            # select: a few, not all
    parts += [("tool", trim(r, fields=("status", "eta"))) for r in tool_results]
    used, out = 0, []
    for kind, text in parts:
        cost = count_tokens(text)
        if used + cost > budget: break
        out.append((kind, text)); used += cost
    recent = summarize(history[:-6]) + history[-6:]            # compress old turns, keep recent ones verbatim
    return out + [("history", h) for h in recent]

Sessions and memory

A session is the raw record of one conversation. It is useful for coherence inside that conversation and unsuited to long-term use: it is noisy, temporary and large. Memory is what you extract and keep from sessions on purpose.

Retrieval and memory are different. Retrieval (RAG) brings in your content. Memory is about this user or this task. Treat them as separate sources with separate rules.

A memory policy for support

Decide before you build. A table like this goes into your design document, and its consent and deletion rows become test cases (module 15).

Test your context

• Late rule test. Put a rule in the system prompt, run 30 turns of unrelated chat, then ask something the rule governs.
• Needle test. Place one fact in a long history and ask for it.
• Staleness test. Change a policy and check the old answer is gone from every source.
• Conflict test. Give two sources that disagree and see what it does.
• Deletion test. Ask to forget something, then try to get it back.
• Injection test. Store a hostile instruction in memory and see if it is obeyed.

Run them as part of the evaluation set (module 24) and re-run them when you change the model, because context behavior differs between models (module 5).

Try it: write a context budget

Pick an assistant you know. For one step, list each part of its context, how many tokens it uses, whether it is needed, and whether it is trusted. Remove or shorten two parts. Predict what improves, then test it.