Where Should AI Stop and Code Start?

Part 13 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The companion repo contains the full code.

"Is this order eligible for a refund?" is four rules: delivered, paid, inside the return window, belongs to the customer. An LLM can answer that. It would probably answer correctly almost every time. The interesting question isn't whether it can - it's what it costs to ask, multiplied by how often you ask.

Refund eligibility gets checked on every refund request, every status enquiry that mentions a return, and every retry. Say 50,000 checks a day for a mid-sized shop.

$ ./gradlew checkCost

One refund-eligibility check, 50,000 times a day

Measured: ~70 ns per deterministic check

The model rows are published per-token prices times an estimated prompt: the policy as a system prompt, the order as JSON, the request, a structured verdict back. Call it 500 tokens in, 60 out. The Java row is RefundEligibility.evaluate measured in a warmed-up loop and costed as rented CPU time. Seventy nanoseconds at $0.036 per vCPU-hour.

The gap is about a billion to one. Not a percentage - a factor with nine zeros. The deterministic check's entire annual compute bill is roughly one thousandth of a cent.

Where the Number Comes From

Nothing clever, which is the point:

public static double perCall(TokenPrice price, PromptSize prompt) {
    return prompt.inputTokens() / PER_MILLION * price.inputPerMillion()
         + prompt.outputTokens() / PER_MILLION * price.outputPerMillion();
}

The prompt estimate lives in a value called