Canary Agentic Autofix With Failure Classes and Reliability Gates

GitHub announced agentic autofix for code scanning alerts in public preview on July 10, 2026. Primary source: GitHub Changelog, July 10, 2026.

The wrong metric is “percentage of alerts with a generated patch.” Generation is only the first transition:

alert -> candidate -> build -> tests -> security oracle -> human review -> merge -> post-merge observation

This is an evaluation proposal, not a benchmark or assessment of GitHub's preview.

Choose a Bounded Canary

Start with repositories that have active owners, deterministic builds, relevant isolated tests, reversible releases, and no automatic production deployment from candidate patches. Exclude abandoned code, safety-critical paths, and repositories with unreliable tests.

Assign the canary deterministically-for example, hash a stable alert ID into a fixed percentage. Do not move difficult results out of the cohort after seeing them.

Record every attempt, including abstentions and failures:

attempt_id: "<id>"
alert_class: "<normalized class>"
base_revision: "<commit>"
outcome:
  generated: true
  applied_cleanly: true
  build_passed: true
  tests_passed: false
  security_oracle_passed: false
  human_decision: "rejected"
  failure_class: "semantic_incomplete"
  escaped_to_default_branch: false

The schema is local evaluation metadata; it does not imply that GitHub exposes these fields.

Classify the Earliest Broken Invariant

Use one primary class and optional secondary classes. Otherwise one attempt contaminates several denominators.

Add Security Oracles

Existing unit tests may not encode the alert's security property. For authorization, require unauthorized denial, authorized success, alternate-entry coverage, and preserved audit behavior. Run the oracle on the unpatched baseline first; if it cannot expose the problem, a patched pass proves little.

Predeclare gates such as zero critical scope violations, zero test-manipulation events, zero automatic merges, bounded infrastructure failure, and a minimum completed sample.

Segment results by alert class, language, repository tier, and patch size. Report confidence intervals rather than promoting after a lucky week.

Stop immediately for privilege expansion, weakened policy, revision mismatch, secret exposure, or unexplained production impact. Rollback includes disabling new attempts, freezing open candidates, reviewing already merged canary patches, and preserving evidence.

Offline fixtures are imperfect, alerts are correlated, and escapes may surface late. That is why wider deployment should be earned through bounded evidence, explicit denominators, and safety events that outweigh an attractive average.