2026-10-25 · AI in production
The model reads, the code decides
Three bugs in three days across two AI systems — and the same answer each time: the language model is for language and for extracting facts. Whatever can be decided deterministically is decided by code.
Labels that jump
A report shows which sources AI assistants cite. The model labelled the same online shop sometimes “marketplace · other brand”, sometimes “unknown” — at temperature 0, with a versioned prompt and a logged decision. The log showed the model even called listings of the customer’s own brand “other”.
I rejected a stricter prompt. Instead the model now only reads facts from the page: page type and the single brand, if the URL or title names one. Who owns the page is computed by code, with the same rule that already worked for other data. Since then the same page always gets the same label, and classification is testable without a model. Temperature 0 does not guarantee consistency between runs.
A button the model had to guess
In a marketplace assistant’s chat, the guest picks a package with a button. The click sent words into the chat, and the model was supposed to call the tool from them. In one case out of three it answered “I prepared your request” — without a call. The hint telling it when to call sat in the previous turn’s tool result, and the model no longer sees that in the next turn.
Now the card itself carries the call the click triggers. The server runs it before the model — only for the card of the latest answer and only for non-destructive tools — and the model answers on the finished result. The call log now shows the call coming from the card, not from the model.
Cologne, New Jersey
A Spanish guest wrote “en Colonia”. The model correctly passed the word to search; the place lookup found Colonia in New Jersey, because the visitor’s country only boosts ranking there, it does not filter. The bug was in the deterministic layer. Now the tool tells the model that the place is abroad and asks for the German name. Same conversation: two calls, “Colonia”, then “Köln”, an answer in Spanish with offers from Cologne. The case is a hard gate in the regression set.
That is why I check AI agents by their calls and results, not just by the text of the answer.