Your prompt is not a security boundary

If your AI agent owns tools with side effects, one question decides whether it is safe to ship: what happens when the model confidently calls a money tool on invented grounds. This is a writeup of one mechanism that closes that hole, and of where the mechanism stops working. The context is assistants that talk to real customers in messengers and can do irreversible things: confirm a payment, issue an invoice, book a slot, notify the business owner. Why an instruction in the prompt is not a control A line like "only confirm payment after you received the receipt" executes with a probability, not with a guarantee. That is not a quality problem with the model. It follows from the training objective: be helpful, agree with the person in front of you. The conversation goes like this. The customer writes "I already paid, I will send the receipt later, please confirm". There is no receipt. The model sees a polite persistent human, sees an instruction that contradicts him, and over a long context it picks cooperation. It answers "payment confirmed" and calls the tool. For tone of voice, probabilistic execution is fine. For money it is not. One more hope worth killing early: the tool config field that looks like a predicate. Most function schemas carry something like trigger_type , and ai_decides literally means "the model decides". That field controls when the tool is offered, never under which facts the tool is allowed to fire. A precondition is a declarative fact about the conversation The idea is small. A function carries a list of facts that the executor checks against the database before dispatch. Not "the model believes a receipt exists", but "there is an inbound attachment in this conversation". Stored as JSONB next to the function: [ { "type": "client_sent_media", "within_messages": 10, "media_kinds": ["image", "document"] }, { "type": "lead_field_filled", "field": "phone" } ] The type list is deliberately short and covers nearly every real requirement: type FunctionPrecondition = | { type: "client_sent_media"; within_messages?: number; media_kinds?: string[] } | { type: "lead_field_filled"; field: string } | { type: "function_called_before"; name: string } | { type: "min_client_messages"; count: number }; client_sent_media scans the last N messages written by the customer rather than the last N rows of the thread. An owner who configures "the last 10 messages" means ten customer replies, not ten rows half of which the bot wrote itself. The window is capped by a constant so that within_messages: 100000 in a config cannot turn the check into a table scan. function_called_before reads the event log and requires an earlier successful call in the same conversation. That is how you build chains like "verify identity first, then modify the booking". Three implementation properties that carry the whole design The check lives in exactly one place. It sits in the tool executor, after argument validation and strictly before dispatch to any handler. Put it inside the handlers instead and you fix the class one handler at a time, which means the next money-touching tool ships without a guard. A block is returned to the model as a tool error with a reason. Not a silent refusal: Blocked: the customer must have sent a image/document attachment in their last 10 messages. This did NOT happen. Do not tell the customer it did. Ask the customer for what is missing, then call this function again. The difference matters more than it looks. After a silent refusal the model assumes the call went through and keeps lying to the customer. An error with a cause produces self correction inside the same round: the bot goes and asks for the receipt. The requirement is appended to the tool description, so the model sees it before spending a call: HARD REQUIREMENT: this function is blocked and will refuse to run unless the customer must have sent a image/document attachment in their last 10 messages. Do not claim the action happened until the call actually succeeds. Fail open, on purpose When the check itself throws, the call goes through. It is not blocked. } catch (err) { logger.error("Precondition check failed, letting the call through", { ... }); } Here is the reasoning. A precondition defends against model hallucination, not against an attacker. An attacker has no reach into this layer at all: he speaks to the bot in words, while the facts come from our own database. So the failure mode should be chosen by cost. Blocking every function for every customer because Postgres blinked means breaking live conversations (no invoice, no booking, no answer) over a hypothesis. The failure goes loudly into the log, and the decision falls back to the prompt, exactly as it was before the guard existed. If this were access control the choice would be the opposite, fail closed. It is not access control, and pretending otherwise would be worse than having no guard. What it does not solve It does not replace authorization, idempotency or rate limits. It answers one question: is there a fact in this conversation without which the action makes no sense. It does not rescue a badly specified function. If your only guard is min_client_messages: 2 , you moved the problem one message down the road. It costs nothing where it is not used. A function with an empty precondition list issues zero queries, the branch returns on an empty array. That property is what keeps the guard alive past the second release: a check that slows down every conversation for the sake of one money flow gets removed by whoever is on call. The takeaway An LLM in production behaves like a capable intern. Most of its calls are good, and nobody lets an intern sign the cheques. Boundaries belong in code, get verified against data, and get logged in a way that survives a restart. The prompt owns the quality of the conversation, and nothing beyond it. All of the above runs in the platform I build, DOS AI: AI assistants for WhatsApp and Telegram with a built in CRM, configured in plain text. If you are building your own, our REST API, webhooks and MCP server are public, so you can plug your agent in and look at the contract from the outside. The machine readable spec sits at https://dosai.pro/llms.txt and the code samples are on GitHub: https://github.com/adsytd1/dosai-api Happy to go deeper on failure modes in the comments. Top comments (0)