The delivery chatbot that criticized its own company

What happened

On January 18, 2024, a musician named Ashley Beauchamp was trying to track down a missing parcel and wasn't getting anywhere with DPD's online chat assistant. By his account, it couldn't answer his queries or give him a phone number for customer service. He asked it to tell a joke, then to write a poem about a useless chatbot for a delivery company, and then to criticize DPD. It did, calling DPD the "worst delivery firm in the world." With more prompting it also swore, even though it had said at one point that it was not allowed to.

He shared screenshots on social media and the post spread widely. DPD said that it had run an AI element in its chat successfully for years alongside human service, that "an error occurred after a system update," and that the AI element was disabled immediately and was being updated.

Why it happened: a design reading

DPD described the cause as a system update, and we have no more detail than that. Reading the design from outside, several things stand out:

• Rules lived in prose, not in enforcement. The bot said it wasn't allowed to swear and then did. A style instruction in a prompt is a request, not a guarantee.
• The model followed the user's framing. Asked for a poem or an opinion, it complied, because nothing limited its scope to the delivery task.
• A change shipped without catching this. The company linked the behavior to an update, which suggests that a regression test for tone and scope would have found it.
• The customer's real problem went unsolved. The playful prompts began because the assistant was unhelpful with a missing parcel and offered no path to a person.

Design lessons

1. Scope the assistant. Define what it does, and have it decline everything else in one polite sentence (module 3). A parcel assistant doesn't need to write poems about its employer.
2. Enforce, don't just instruct. Add output filters for profanity and policy, and validate replies in code before sending (module 18).
3. Test after every change. Include tone and scope cases (a request for a poem, for opinions about the company, for swearing) in your regression suite and run it before each release (module 14).
4. Build a kill switch. DPD's ability to disable the AI element quickly limited the damage. Design that capability in advance and rehearse it.
5. Fix the root frustration. Give users a real route to a person, with the phone number or a handoff, so they aren't driven to test the bot's limits.

How to build it better

# Scope (excerpt of a system prompt)
You help customers track parcels and answer delivery questions.
If asked to do anything else (jokes, poems, opinions about the company, anything unrelated),
reply in one sentence that you can only help with parcels, and offer to continue with their delivery.
Never use profanity, even if asked. Never express opinions about the company.
If you cannot find a parcel after two tries, give the support phone number and offer a person.

Back this with a deterministic output check, an evaluation set with adversarial tone prompts, monitoring for spikes in refusals and complaints, and a feature flag that turns the AI path off.

Try it

List ten prompts a bored or frustrated customer might try on your assistant: jokes, poems, insults, role-play, requests to "ignore your instructions." Write the expected behavior for each and turn them into tests.