I Asked an AI to Author the Same Policy Tests 50 Times. It Hit Every Boundary in 49 Valid Runs.

I have been experimenting with a specific question: can an AI independently author the test cases needed to challenge structured business decision logic? Consider a vendor-screening policy with sanctions rules, country restrictions, personal-data conditions, and risk thresholds. If that policy is converted into executable decision logic, someone still needs to test the difficult boundaries - exactly 70 versus just below 70, whether an embargo-list member is handled correctly, or whether a condition changes when personal data is involved. In the earlier Judgment Pack evaluator experiments, I gave an independent AI model the policy and asked it to author those kinds of records without seeing the Judgment Pack or the defects that would later be tested. Before the authoring call, six classes of potential defects had already been committed. The model authored records that covered all 6 of 6 classes. That was encouraging, but it answered only one question: Can the model do it? It did not answer the more important reliability question: If I ask it to do the same job again, how often will it cover those boundaries? A single successful generation could simply be a lucky draw. So Study 011 repeated the same blinded authoring task 50 times. The result, for this particular experimental setup, was striking: - 49 runs passed the preregistered pipeline checks - 1 run was rejected before scoring - all 49 valid runs covered all six boundary classes - every valid run produced exactly 16 accepted records - 784 of 784 accepted records agreed with the reference policy semantics - all 49 valid completions were different For this one prompt, one model, and one synthetic policy, the answer to "how often?" was: Every valid time we observed. That sounds like a perfect result. It isn't. And understanding why is the interesting part of the experiment. The problem: one good generation is not a reliability result I am working on an open-source project called the Judgment Pack Specification. One area I am exploring is whether AI can help humans author the structured test cases around organizational decision logic. Imagine a vendor-screening policy containing rules such as: - sanctions matches create a hard stop - vendors registered in embargoed countries require a different outcome - a risk score of 70 crosses a high-risk boundary - handling personal data changes the threshold at 40 Those rules produce boundaries. And boundaries are where subtle errors tend to hide. For example: risk = 69.99 risk = 70 risk = 70.01 can represent three very different cases if the policy says: risk >= 70 The same issue appears around a personal-data threshold: 39.99 40 40.01 If AI is going to help author tests for these rules, I do not only care whether it notices the boundary once. I want to know how consistently it notices it. Study 011 asked "how often?" The experiment used one fixed cell: - one model - gpt-5.6-sol - one fixed prompt - one fixed synthetic policy - one pinned CLI and binary - fresh isolated environment for each call - 50 sequential authoring calls - six preregistered coverage classes The prompt, policy family, binary, environment capture, and preregistration were all digest-pinned. The scoring rules were fixed before looking at the batch. There were no competing arms and no hypothesis test. This study was simply estimating a frequency. What counts as coverage? Six boundary classes were registered before the batch. Class 0 - exact high-risk boundary No sanctions hit Non-embargoed country risk = 70 This catches a common implementation error: risk > 70 instead of: risk >= 70 The model independently authored records with names such as: exact-high-risk-threshold northstar-risk-seventy alpine-exact-seventy summit-exactly-seventy Every valid run included a correctly labelled record at this boundary. Coverage: 49/49 95% exact interval: [0.9275, 1.0000] Class 1 - the off-by-one region above 70 The next class tested: 70 = 70 to: >= 71 Testing only 70 and 80 might miss useful information about that region. The authored records included cases represented by names such as: fraction-above-high-risk-threshold atlas-risk-seventy-decimal harbor-above-seventy baltic-risk-above-seventy Again: Coverage: 49/49 Class 2 - personal-data boundary at 40 The policy also changes behavior when a vendor handles personal data. This class tested: handles personal data 40 = 41 instead of: risk >= 40 Examples of independently authored records included: personal-data-exact-threshold harbor-data-risk-forty bluebell-personal-data-at-forty maple-data-forty Coverage: 49/49 Class 3 - the interior decision region This class tested: 40 admit context differs -> refuse So it refused. I actually find this result useful. The model-authoring behavior looked extremely stable. The experimental pipeline was not perfect. Those are two different reliability questions. What about human review? Before running the batch, the study registered a deliberately simple mapping from observed coverage to review depth. It uses the lower bound of the confidence interval: lower >= 0.80 -> LIGHT review lower >= 0.40 -> STANDARD review lower FULL review A high mislabelling rate can escalate the tier. All six classes ended with: lower bound = 0.9275 mislabel share = 0 tier = LIGHT So under the preregistered mapping, every class receives: LIGHT review. That does not mean AI-authored policy artifacts no longer require humans. The review-tier mapping itself is experimental and has not been validated operationally. But it suggests an interesting direction. Instead of treating every AI-authored rule or test case as equally trustworthy, perhaps review depth can eventually depend on empirical evidence about the authoring process. Something like: AI authors candidate records | v measure historical coverage / error rates | v assign confidence | +------ high confidence ------> light review | +------ uncertain ------------> standard review | +------ weak evidence --------> full review That is much closer to what I ultimately want from AI-assisted authoring. Not: "The model wrote it, trust it." And not: "The model wrote it, manually redo everything." Instead: "We have measured evidence about where this authoring process is strong and where humans should spend their review time." What this study does NOT show This is the most important section. The result does not mean: AI can reliably author arbitrary organizational judgment. It means: This prompt, with this model, on this small synthetic policy, repeatedly produced correctly labelled records reaching these six registered classes. Nothing more transfers automatically. It is one model Only one model configuration was tested. A different model may behave differently. It is one prompt The prompt explicitly asks for borderline cases. That probably contributes substantially to the ceiling result. It is one small synthetic policy The policy has a few thresholds, one membership rule, sanctions logic, and a personal-data condition. Real organizational policies can contain: - interacting exceptions - multiple documents - ambiguous language - precedence - temporal conditions - missing evidence - conflicting authorities - human approval requirements Those may produce very different rates. The mirror is not ground truth "Correctly labelled" means the authored outcome agrees with the study's deterministic mirror. The same policy supplied to the model is implemented in that mirror. So: 784 / 784 measures consistency with the supplied policy semantics. It does not independently prove those semantics are correct in the real world. Coverage is not defect detection Study 011 never evaluates a mutated Judgment Pack. It asks whether an independently authored record lands in a region that could expose a registered defect. Study 010 performed the single defect-detection draw. Study 011 measures the repeatability of the authorship coverage behind it. Those are deliberately separate claims. The result changed how I think about AI authoring Before this experiment, I was mostly thinking about AI-assisted authoring as: policy | v AI | v candidate structured artifact | v human review I now think a better model might be: historical evaluations | v policy -> AI author -> candidate artifact | v automated validation | v confidence estimate | +----------+----------+ | | | LIGHT STANDARD FULL review review review | v authorized approval | v versioned artifact The interesting question becomes less: Can AI author the artifact? Frontier models increasingly can. The harder question is: How much review does this particular generated artifact deserve? That is something we may be able to measure. What I want to test next The obvious next experiment is not another 50 runs of this same cell. This one is already sitting at the ceiling. The useful next variable is the thing this study deliberately held constant. For example: Change the policy Use a larger policy with interacting rules and exceptions. Does coverage remain near the ceiling? Change the model Run the same registered task across multiple model families. Do the boundary classes remain stable? Remove the boundary-forward wording If the prompt stops explicitly asking for borderline cases, what happens to classes like: 39 <= risk < 40 That would help separate general semantic discovery from prompt-induced test generation. Introduce ambiguity What happens when the policy itself contains: - conflicting language - incomplete evidence requirements - inconsistent descriptions and executable conditions - unclear precedence That is much closer to the real authoring problem. The takeaway Study 010 told me: The independent author can cover all six classes. Study 011 tells me something stronger: Under this exact experimental setup, that coverage was the typical observed behavior, not a lucky single run. But the more important lesson is methodological. One impressive AI output is an anecdote. Repeated outputs give you a rate. A rate gives you uncertainty. And uncerta