Stop Letting AI Write Tests You Won’t Run: Enforce SDLC Gates That Catch LLM Hallucinations, Drift, and Silent Regressions

Originally published on tamiz.pro. The Problem: AI Tests That Never Run LLMs generate test code faster than developers can review it. But unchecked, these tests introduce hallucinated assertions, behavioral drift, and silent regressions that only surface in production. The fix isn’t banning AI-it’s enforcing SDLC gates that verify generated tests actually pass, cover real code paths, and match expected behavior before they merge. The Gate Strategy Instead of trusting generated tests, treat them as untrusted input. Every AI-generated test must pass through four gates: - Execution Gate - Tests must compile and run successfully. - Coverage Gate - New tests must increase meaningful coverage. - Behavior Gate - Tests must fail when target behavior changes (mutation testing). - Drift Gate - Tests must not assert on hallucinated or irrelevant logic. Prerequisites - A CI pipeline (GitHub Actions, GitLab CI, etc.) - A test runner (Jest, Pytest, Go test, etc.) - A coverage tool (nyc, coverage.py, go test -cover) - Optionally: a mutation testing tool (Stryker, Cosmic-Ray, go-mutesting) Step 1: Enforce the Execution Gate AI-generated tests often contain syntax errors or reference non-existent APIs. Block merges where tests fail to run. GitHub Actions Example name: Test Execution Gate on: [push, pull_request] jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Install dependencies run: npm ci - name: Run tests run: npm test -- --ci --maxWorkers=2 Why This Matters If a test doesn’t run, it provides zero protection. This gate ensures every generated test is at least syntactically valid and executable. Step 2: Enforce the Coverage Gate AI can generate tests that execute code without asserting meaningful behavior. Require new tests to increase line and branch coverage. Jest Coverage Threshold // package.json { "jest": { "collectCoverageFrom": ["src//.{js,ts}"], "coverageThreshold": { "global": { "branches": 80, "functions": 80, "lines": 80, "statements": 80 } } } } GitLab CI Coverage Check test: script: - npm test -- --coverage coverage: '/Statements\s:\s*(\d+.?\d*)%'/{% endraw %} ### Why This Matters Coverage thresholds prevent low-value tests from sneaking in. If an AI test doesn’t meaningfully increase coverage, it’s likely asserting on trivial or hallucinated paths. ## Step 3: Enforce the Behavior Gate (Mutation Testing) The strongest guard against hallucinated assertions is mutation testing. If a test doesn’t detect a mutated version of the code, it’s not actually validating behavior. ### StrykerJS Example {% raw %}bash # package.json turbo run mutate --filter=src//.test.js # .strykerrc.json { "mutate": ["src/**/.ts"], "testRunner": "jest", "thresholdHigh": 80, "thresholdLow": 60, "thresholdBreak": 0 } GitHub Actions mutation: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 18 - run: npm ci - run: npx stryker run --configuration .strykerrc.json Why This Matters Mutation testing proves tests fail when behavior changes. An AI test that survives mutations is asserting on irrelevant details, not real invariants. Step 4: Enforce the Drift Gate AI tests can drift from intended behavior by asserting on implementation details or hallucinated logic. Use snapshot testing or golden-master techniques to lock in expected outputs. Jest Snapshot Testing // user.service.test.js test('returns active users only', () => { const result = userService.getActiveUsers(); expect(result).toMatchSnapshot(); }); If the snapshot changes, the CI gate fails unless the developer explicitly approves the diff. This prevents silent behavioral drift. Approval Tests Pattern For non-JS ecosystems, use approval tests: # Python example from approvaltests import verify def test_process_order(): result = order_service.process_order(order_data) verify(result) Why This Matters Drift gates catch unintended behavioral changes. AI-generated tests can accidentally encode wrong assumptions-this gate forces explicit approval when assumptions change. Step 5: Combine Gates in a Single Pipeline Run all gates in parallel to catch failures early without slowing feedback. name: AI Test Validation Pipeline on: [pull_request] jobs: execution: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: npm ci - run: npm test -- --ci coverage: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: npm ci - run: npm test -- --coverage mutation: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: npm ci - run: npx stryker run A PR merges only when all gates pass. AI-generated tests that don’t survive execution, coverage, mutation, or drift checks are rejected automatically. Best Practices - Review generated code, but don’t trust it. Gates replace manual review for correctness. - Fail fast. Run execution and coverage gates before mutation testing to save time. - Use baselines. Track coverage and mutation scores over time to detect degradation. - Automate approvals. For snapshot/diff-based gates, require explicit developer approval for changes. - Log failures. Capture which gate failed and why, so teams can improve AI prompts or training data. Frequently Asked Questions **Q: Won’t mutation testing slow down CI? A: Yes, significantly. Run it on a scheduled basis or for high-risk changes only. Use execution, coverage, and drift gates as the primary PR gates. **Q: How do I handle AI-generated tests that are intentionally exploratory? A: Separate exploratory tests from committed tests. Only enforce gates on committed test files. **Q: What if my coverage threshold is too strict? A: Start with a low threshold (e.g., 50%) and increase it gradually as tests improve. The goal is to catch zero-value tests, not enforce arbitrary numbers. The Bottom Line AI-generated tests are a productivity lever, not a quality guarantee. By enforcing execution, coverage, behavior, and drift gates in your SDLC pipeline, you turn untrusted AI output into verified, reliable test coverage. The result: faster development without silent regressions slipping into production. For deeper insights on test reliability and pipeline design, see Tamiz's Insights. Top comments (0)