2026-10-23 · AI in production
A golden set for the AI assistant: tests that check actions, not sentences
One day of manual testing, nine different model failures. Every fix could break the neighbouring case, and only manual testing could tell. So I built regression tests.
A marketplace’s AI assistant searches offers and prepares requests through the platform’s MCP tools. Manual testing turned up: “prepared” although the tool had refused; a card number instead of an ID; “Shall I send it?” instead of a call; a German answer to an English question; invented prices and venues. Half of the fixes were in the tools’ response texts, half in the prompt.
Where the test lives
The mechanism belongs to the chat service, because it serves several customers. The cases belong to the platform: it is their search and their requests, and they change in the same commits as the tools.
A review of the plan by two models found a blocker: a conversation in which the platform’s tools answer can only be opened by the platform itself. So it opens test conversations in a sandbox — outside statistics, moderation and notifications. A draft from a test conversation cannot be sent by any path on the server, and the runner calls writing tools only on the staging environment. The first run stopped exactly at that guard: the staging address was a different one for the chat service.
Two tiers instead of “2 out of 3”
I rejected recorded responses (record/replay): half of the failures sat in the tool texts, and a recording would have frozen them.
Hard rules check facts: which tool, which arguments, no success words on top of a refusal, the language of the answer. They must pass 3 out of 3 and gate the release. Wording is scored by a model acting as judge — for the report only. A “2 out of 3” gate is almost always red, and then people ignore it.
Dates in the conversations are templates (“+6 weeks”), and expectations check the shape, not specific offers: staging regularly gets a copy of production data.
Result: 12 cases, a run takes a few minutes. Four runs in one evening — the first at 10 of 12, with two consistently red cases the manual test had missed; the fourth with all 8 hard rules green. Along the way the set caught a bug in the judge itself: it did not know today’s date and treated a past day as future.