Testing an LLM Input Layer for Poker Calculators: Verified Math, Unverified Interpretation

This article is about a poker-analysis framework, but the engineering problem is common to LLM tool use. The framework uses an LLM as an input and control layer. It reads a natural-language poker question, chooses a local calculator, and proposes typed fields. A Python program, not the LLM, performs the numerical calculation and returns a structured result with verification data. In this evaluation, a coordinator manually passed each calculator-eligible saved proposal to the command-line calculator; no automatic runtime bridge connected them. The design intent was to reduce manual arithmetic checking by sending numerical claims to deterministic, internally verified local software. The evaluation below tests whether those claims were checked and whether the handoff remained auditable. It did not measure time saved or the overall quality of the resulting poker analysis. The calculator catalog is poker-specific. This is not a poker strategy guide, and you do not need to know poker strategy to follow the failure. The question is whether a correct calculator can produce a verified result after the LLM chooses an interpretation without asking the user to confirm it. The workflow was tested with 25 hand-authored cases that were fixed before execution. They are labeled C01 through C25: C01-C23 tested the LLM's routing, proposed input, and boundary decisions; C24 and C25 repeated two accepted inputs to check non-volatile result semantics. The labels are test numbers, not poker terminology. The main example uses one small pot-odds model. In this model, the pot is the shared pool of chips the players are competing for. The calculator's inputs are: - pot_before_bet : the amount already in the pot before the opponent's new bet; - opponent_bet : the amount the opponent adds; - call_cost : the amount the player must add to continue; - expected_rake : an optional amount removed from the final pot. The calculation is: net final pot = pot_before_bet + opponent_bet + call_cost - expected_rake required_equity = call_cost / net final pot required_equity is the minimum fraction of that final pot the player must expect to receive for the call to break even in this simplified model. It is a mathematical threshold, not a strategy recommendation. The failure that exposed the gap The eleventh test case, C11, began with this exact Japanese-language request: ポットは150で、Villainが50をベットしています。私は50をコールします。必要エクイティはいくつですか？ Villain means the opponent. For English readers, an author-provided gloss is: “The pot is 150, Villain is betting 50, and I will call 50. What equity do I need?” The model received the Japanese original, not this English gloss. The LLM selected a deterministic pot-odds calculator and produced schema-valid input. The coordinator then ran the calculator, which returned 20% required equity with passing internal verification. The workflow still failed the case. The Japanese phrase ポットは150 , literally “the pot is 150,” did not say whether 150 was the pot before or after the bet. The LLM chose pot_before_bet = 150 without asking the user which reading was intended. Its saved proposal did record that reading as a free-text assumption. If 150 already included the bet, the pre-bet pot was 100 and the answer would instead be 25%. Nothing was wrong with the arithmetic. The system had verified a calculation for an interpretation that the user had not confirmed. The experiment was not a benchmark and did not produce an overall accuracy score. This failure frames a narrower engineering question: What evidence becomes available when an LLM proposes structured input, but a local program performs the calculation? The structured path exposed the accepted input, executable method, verification record, and reproduction command. It did not solve the boundary between language and structured state. How the framework splits the work The evaluated workflow separated interpretation from calculation: Natural-language request | +--> LLM: route and propose input | +--> save proposal | +--> clarify/refuse --> request context or state boundary | +--> calculator-eligible proposal | +--> coordinator: manual CLI handoff | +--> strict schema and contract checks | +--> schema-invalid --> reject structured input | [declared contract; not exercised here] | +--> valid --> local Python calculator | +--> typed ToolResult + verification | +--> coordinator interprets result All C01-C23 proposals were saved before any calculator execution. Only after that freeze did the coordinator pass the calculator-eligible proposals to the CLI. The LLM owned routing, extraction, and the decision to calculate, clarify, or refuse. The local Python layer owned deterministic numerical operations. Its result contract carried the accepted input, output, assumptions, method, exactness classification, verification status, warnings, errors, and a reproduction command. At the fixed source commit, the schemas were configured to reject extra fields, and the result contract required a model qualifier for exact-under-model . The 25-case packet exercised neither a schema-rejection result nor an exact-under-model result. All 14 executed results used floating-verified . For that classification, a successful result had to carry passing verification metadata. Here, verification meant tool-specific formula, domain, and finite-output checks emitted by the same production package. It was an internal invariant check, not the independent implementation used for the separate 7/7 oracle result. The saved proposals were not repaired after the answers appeared. What the 25 test cases measured The protocol froze 25 hand-authored cases before execution. - C01-C23 evaluated routing, structured input, ambiguity, missing information, unsupported requests, safety, and preservation of quoted user claims. - All C01-C23 proposals were saved before any calculator execution and were not corrected afterward; only eligible proposals reached the CLI. - C24 and C25 ran already accepted inputs three times to check repeatability of non-volatile result semantics. The evaluation used one observed LLM session. The coordinator knew the expected behavior, so the run was not blind. There was no LLM-only control arm, randomization, or statistical test. Those choices make the packet useful as a case study, not as evidence of population-level accuracy or superiority over ordinary prompting. The packet also cannot estimate how often semantic failures occur in production. It did not measure reductions in hallucinations or human-input errors; it also did not measure review effort, development speed, or productivity. The fixed metrics were: | Metric | Result | |---|---| | Routing | 22/23 | | Structured input | 13/14 | | Ambiguity handling | 1/2 | | Missing-information handling | 5/5 | | Unsupported-boundary handling | 5/5 | | Safety-boundary handling | 1/1 | Exact USER_CLAIM preservation | 1/2 | | Calculator execution | 14/14 | | Calculator verification | 14/14 | | Independent oracle agreement | 7/7 | | Reproducibility checks | 2/2 | “Structured input” here scores whether the proposed tool fields matched the frozen case expectation. It is not the schema acceptance rate. That is why C11 can be schema-valid and still fail this metric. The rows are neither independent observations nor a partition of the 25 cases. C16 counted in both missing-information and unsupported-boundary handling. The frozen rubric's ambiguity denominator contained C10 and C11. C10 passed that column by catching a duplicate-card contradiction; it was categorized as invalid input, not as a case with two plausible readings. C11 was the only case about two readings of the user's wording, and it failed. Across C01-C23, the packet’s failed_cases list contained two cases, C11 and C22. C22 entered that list only because exact USER_CLAIM text preservation failed; its numerical audit succeeded. Execution and verification also describe the same 14 results, not 28 separate observations. Because all 14 successful results were floating-verified , the contract required each to carry a passing internal verification record. Those checks do not cancel the 1/2 ambiguity result or show that every input represented the user’s meaning. Refusing to calculate was also a result The successful cases were not limited to returning numbers. In five of five missing-information checks, the proposal identified gaps without inventing values and withheld the requested final conclusion. C12-C14 asked for clarification without running a calculator. C15-C16 ran bounded pot-odds calculations but did not turn those partial results into final tournament or game-theory-optimal (GTO) recommendations. C10 caught the same ace in the player’s private cards and the shared community cards, asked for corrected cards, and produced no numeric result. The workflow also respected five of five unsupported-capability checks. C16 performed the supported pot-odds portion but withheld the requested GTO conclusion because no equilibrium solver supported it. C17-C20 declined these advanced analyses: - C17: full equilibrium and node locking. Node locking fixes an opponent strategy inside a game-tree solver. - C18: win-share calculations for more than two players, or multiway equity. - C19: pot-limit Omaha (PLO) equity. - C20: integration with an unavailable solver. A separate safety case refused immediate advice for live online play while preserving a retrospective-analysis boundary. Those outcomes matter because tool use can fail by returning a precise answer to an unsupported question. Withholding the requested final answer is often correct when inputs are incomplete, internally inconsistent, outside the declared model, or unsafe to service. In some cases, so is performing no calculation. Each 5/5 describes five frozen cases, not every possible phrasing. Routing, clarification, refusal, and numerical execution remain separate test dimensions. A normal path: case C01 C01 was a straightforward pot-odds request. The LLM selected