Why API Test Generation Is a Judgment Problem, Not a Code Generation Problem

TL;DR - Key Takeaways - AI test generators often produce too many tests while still missing important boundary, business-logic and cross-field scenarios. - Better prompting improves results, but tends to plateau because the hardest testing problems require judgment rather than more instructions. - The stronger architecture separates judgment from mechanics: Fine-tuned models decide what deserves testing, while frontier models generate executable payloads and code. - RAG supplies missing information, while fine-tuning calibrates QA behavior such as prioritization, redundancy reduction and assertion quality. - Execution feedback turns test generation into an improving system by feeding accepted, rejected, edited and flaky tests back into the judgment layer. When we started using large language models for API test generation at KushoAI, the results were impressive on the surface. Tests appeared in seconds. Coverage breadth went up. The team was excited. Then we looked more carefully at what was being produced. On one endpoint, the model generated 26 tests where a senior QA engineer would have written 9. On another, it completely missed the boundary cases that mattered most. On a third, it generated tests that checked whether a response arrived but said nothing meaningful about what the response contained. The tests were valid. The test suite was not useful. What we had was a fast generator with no taste, and in testing, taste is almost everything. This article is about what we learned while building toward something better: How we diagnosed what was actually wrong, why common fixes such as better prompting fall short, and what the architecture looks like when you treat test generation as a judgment problem rather than a code generation problem. The Six Ways Fast Test Generation Fails After running our pipeline across thousands of real APIs, six failure modes consistently emerged. Understanding them precisely is what pointed us toward the right solution. Over-generation is the most visible problem. The model produces 20 to 30 test cases where 8 would be sufficient. The excess cases are often plausible but redundant, and they make suites harder to review and trust. A QA engineer looking at a 30-test suite for a single endpoint will struggle to identify what is actually being covered versus what is noise. Under-generation is the opposite and arguably more dangerous. The model skips obvious scenarios - required fields left out, invalid enum values, boundary conditions, malformed formats, missing authentication cases and expected error responses. These omissions are hard to see precisely because they are absent. Redundancy is subtle. Three generated tests labeled ‘invalid value’, ‘incorrect value’ and ‘unsupported value’ often map to a single check. The model treats them as distinct because the phrasing differs. A human reviewer sees immediately that they are the same assertion stated in three ways. Inconsistency is what makes a test suite hard to maintain. Two structurally similar endpoints receive vastly different treatment on each run. One gets thorough positive, negative and boundary coverage. The other gets five tests, mostly a happy path. There is no predictable standard. Weak assertions are perhaps the subtlest failure. A generated test checks if a response arrives and matches a broad schema shape. What it does not check is whether the transaction ID is absent when a payment is rejected, whether the error code is machine-readable rather than a human string or whether the rejected state is reflected consistently across nested fields. The test passes. The bug ships. Cross-field relationship failures are the hardest to generate and the most expensive to miss in production. Individual fields are tested in isolation: Is the amount missing, is the currency wrong, is the payment method invalid. What gets skipped is the interaction: What happens when the amount is valid, the currency is valid, the payment method is valid and the combination creates an invalid business state. These failures do not trigger a 500 error. They return a 200 with corrupt data somewhere three levels deep in a nested response. All six of these are failures of judgment, not knowledge. The model knows what an HTTP endpoint is. It knows what schema validation means. What it lacks is the calibration to decide what deserves to be tested, at what depth and with what assertions. Why Prompting Helps but Plateaus The natural first response to these problems is better prompting. We did a lot of it: Structured prompts, multistep chains, context injection, explicit output schemas, few-shot examples and detailed instructions about redundancy, coverage breadth and assertion specificity. Prompting helped. It improved structure and reduced the most obvious redundancy. Few-shot examples produced the largest single improvement, pulling outputs closer to what a reviewer would keep. That result was informative. If a few examples in the prompt improve behavior, the missing ingredient is broader exposure to examples of preferred QA decisions. The model already has the general capability. It needs calibration toward specific judgment patterns. But prompting plateaus. It mostly improves field-level exhaustiveness: More missing-field tests, more wrong-type tests and more boundary tests. These are useful, but they remain independent mutations. Prompting does not reliably produce reasoning about field relationships, business states and workflow behavior. The hardest failure mode - cross-field relationship testing - is the one that prompting improves least. This is exactly what we see in our public benchmark, APIEval-20. Schema-level bugs are easier to generate tests for. Complex bugs that only surface when multiple valid fields combine into an invalid state are systematically harder. Prompting narrows the gap but does not close it. Separating Judgment From Mechanics The architectural shift that actually helped was treating test generation as two distinct problems requiring different models. The first problem is judgment: Which scenarios should exist, which edge cases matter, what should be asserted, how much is appropriate and what is redundant. This is a calibration problem. It requires exposure to many examples of reviewed QA decisions across many API types. The second problem is mechanics: Constructing valid payloads, generating executable tests, writing framework-specific output and handling authentication context. This is a code generation problem. General-purpose frontier models already handle it well. We fine-tune narrow models for the judgment layer. Frontier models handle the mechanics layer. The boundary between them is the most consequential design decision in the architecture. The test intent model takes API metadata and a target field, operation or group of fields and then outputs a structured list of scenarios in plain language. It is responsible for coverage breadth, relevance, redundancy reduction and cross-field scenario selection. For a payments endpoint, it should not only produce tests for a missing amount or a wrong currency type but also identify cases where amount, currency, payment method, refund status and idempotency key interact to create invalid business states. The assertion intent model takes an expected response schema and an observed response and outputs validation statements in plain language. The difference between a weak assertion and a useful one is specificity. A weak assertion says the response should be valid. A useful set says the status should be 400, the error response should include a machine-readable error code, no transaction ID should be issued and the rejected state should be reflected consistently in the response body. Both models output intent, not executable code. The frontier model receives that intent and turns it into payloads and tests. This keeps the fine-tuned surface small and easy to evaluate. When the frontier model changes, the judgment layer remains undisturbed. What Gets Fine-Tuned and Why A reasonable question is whether retrieval-augmented generation (RAG) solves this instead of fine-tuning. The distinction matters. RAG supplies context the model does not have - internal documentation, product behavior, customer-specific policies, historical incidents and organization-specific testing standards. It is the right tool when the system needs information. Fine-tuning calibrates behavior the model already has but applies inconsistently: Coverage calibration, scenario prioritization, assertion quality, redundancy reduction and consistency across similar APIs. These are behavior-shaping problems, not retrieval problems. No amount of supplied context teaches a model which tests a QA engineer would keep. A mature system uses both as they are solving different problems. What we fine-tune on is reviewed QA decisions, not raw API examples. For each example, the pipeline generates tests, a reviewer then removes redundant cases, adds missing scenarios, rewrites vague assertions, normalizes phrasing and stores the corrected output as the training target paired with its input context. The correction signals include accepted tests, rejected tests, redundant cases removed, missing edge cases added, weak assertions rewritten, flaky tests flagged after execution and cross-field cases added by a reviewer. Each of these is an explicit judgment about coverage, prioritization or assertion quality. The dataset is not a collection of API examples. It is a collection of QA decisions. Reviewer consistency is architecture, not preprocessing. Fine-tuning amplifies whatever patterns the data contains. If reviewers handle similar APIs differently, the model learns the inconsistency. Correction guidelines, accepted and rejected examples and review standards have to be documented and applied uniformly. The goal is a clean distribution of preferred decisions, not the largest possible pile of examples. The Full Pipeline The judgment and mechan