17 PRs a Day, One QA: How We Automated E2E Failure Triage

How We Got Here

Our team is AI-first by design: developers ship with AI assistants, and instead of scaling manual testing along with headcount, we bet on automation. Autotests aren't a safety net "just in case" - they're the primary trigger that we're doing something wrong.

We release once a day, and originally the release was where everything got verified: fifteen-plus PRs roll up, tests run, and if something's red, the one QA engineer investigates. A single release could contain several broken PRs at once, each to be identified among 17+ suspects while the release waits. Mornings turned into detective work. So we shifted the loop left.

Quick infra note: every PR gets a Vercel preview (a dev environment - running the full e2e scope there makes no sense: it's slow, and half the features sit behind flags that don't match production), while staging is effectively pre-prod: every merge lands there, and code ships to production from there. E2e now runs on every merge to staging - the suspect range becomes one fresh merge instead of a day's batch, and the release turns into a final "works assembled" check plus manual testing that automation can't cover.

The price: red runs now happen potentially after every one of those 17 merges, each needs a quick investigation, and until it's done the release is in question. Hiring a person for "look at a failed run and decide who to hand it to" is overkill - that's not a role, it's a distraction. The reasoning wrote itself: if our tests are automation, triaging their failures should be automation too.

Handily, the failure sources are a closed list - a PR regression in the main repo, the neighboring backend repo (its deploys trigger our e2e too), external services, CI infrastructure, or a flake. Manual triage was always the same mechanical chain over structured inputs: failed tests, diffs, logs → a three-way call. Exactly the kind of task LLMs are good at.

Architecture: Verdict → Routing

The triage is a reusable composite GitHub Action, invoked after every e2e run - red and green. Around it: Allure TestOps (test results), Linear (tickets), Slack (notifications), Claude API (the brain).

e2e run on staging
    │
    ├── red ──────────┴────────── green
    │                               │
    │                          AI verdict
    │                    Recovery: close tickets,
    │                    rebase-nudge PR authors
    │
    ┌──────────┴───────────────┐
    │                          │
likely-pr-related    likely-not-pr-related
                    / insufficient-data
    │                          │
Linear ticket       Slack heads-up
+ Slack thread      + ONE auto-rerun
    │                          │
    ┌──────────┴──────────┐
    │                     │
rerun green          rerun red
    │                     │
it was a flake -    escalation: ticket
note it, move on    + @oncall mention

On a red run the bot collects failed tests with steps from Allure, job logs, the deploy commit range, and the diffs of the PRs that went in - and asks Claude (Opus 4.8, Sonnet 5 fallback) for a verdict plus a fix hypothesis:

const verdict = await askClaude({
  failedTests,
  jobLogs,
  deployRange,
  prDiffs
})
// => 'likely-pr-related' | 'likely-not-pr-related' | 'insufficient-data'

if (attempt === 1 && verdict !== 'likely-pr-related') {
  await slack.post(headsUp(verdict))
  // "doesn't look PR-related - rerunning"
  return requestRerun()
  // exactly one auto-rerun
}

// PR-related, or a red rerun → Linear ticket + Slack post,
// on a red rerun - with an on-call mention
await createLinearIssue({ verdict, hypothesis, failedTests })

The key decision: the verdict picks a route, it doesn't just write text. PR-related → ticket with a hypothesis and a Slack thread. Anything else → no ticket yet: a heads-up and one auto-rerun. Green rerun - it was a flake, no noise. Red rerun - ticket plus an on-call mention.

🔴 e2e failed on staging - 3 tests
Verdict: likely-pr-related
Hypothesis: regression in PR #1234 - the file upload handler changed,
tests fail on the "attach document" step
Ticket: QA-482 · Staging: 🔴 red - release on hold

Tickets land in one Linear queue, with labels telling frontend deploys from backend ones. One manual habit pays for it all: a confirmed real bug gets a bug label - so "how many bugs did autotests catch, and how much was noise" is a plain queue filter.

Noise control, four mechanisms:

• Dedup: repeat failures match open auto-tickets instead of spawning new ones
• Auto-close: green run closes the ticket, posts recovery with outage duration, and nudges open PRs to rebase
• Two channels: ticket-backed posts go to the important channel, rerun chatter stays in the working one
• Manual muting: tickets closed without a fix go to Canceled, not Done - recovery leaves them alone

Neat on paper. Now the four rakes.

Rake #1: The Ticket Stuck Open Forever

Recovery writes to three places: a Slack post, the Done transition, and a recovered-at: <sha> marker comment so the next green run skips the ticket. The first version wrote the marker first. Slack post succeeds, Done transition fails on a network hiccup - the marker is already there, and every following run says "already processed". Forever.

The fix is a classic idempotent sequence - each step guarded by its own marker, "fully processed" written last:

if (!hasSlackPostedComment(issue.comments)) {
  const resp = await slack.postMessage(buildRecovery({ issue, attribution, cause }))
  await linear.addComment(issue.id, `slack-recovery-posted: ${resp.ts}`)
}
if (issue.stateType !== 'completed') {
  await linear.transitionToDone(issue.id)
}
if (!hasRecoveredAtComment(issue.comments)) {
  await linear.addComment(issue.id, `recovered-at: ${sha7}`)
  // strictly last
}

Fail in the middle - the next run completes the rest without duplicating anything. Bonus: state lives in the ticket's own comments - no external database, human-readable, garbage-collected with the ticket.

Rake #2: The Bot That Blamed the Innocent

Recovery posts ended with "thanks for the fix, @author" - author computed from the attached fix-PR, or, as a fallback, from the commits in the green run's deploy range. But when a ticket closed because an external service came back to life, the fallback still found an "author": some random person whose innocent PR rode the same deploy. Publicly thanked for a fix that never existed. A couple of those and trust in the bot is gone.

The fix is a recovery-cause classifier - a pure function, no AI:

type RecoveryCause = 'code-fix' | 'external' | 'ambiguous'

const classifyFixCause = ({ deployRange, originalVerdict }): RecoveryCause => {
  const codeFiles = deployRange.files.filter(f => isCodeFile(f.filename))
  if (codeFiles.length === 0) return 'external'
  // only docs/lockfiles deployed
  if (originalVerdict === 'likely-not-pr-related') return 'ambiguous'
  return 'code-fix'
}

Attribution now fires only for code-fix; external and ambiguous use a no-names template. ambiguous doubles as telemetry: how often the model's verdict diverged from the actual cause.

Rake #3: Mixed Runs and the False "Rerun Helped"

The first dispatcher was if (hasFailures) runAnalysis() else runRecovery(). Breaks on a mixed run: old ticket's tests are fixed, but something new failed - the run is red, recovery never fires, the fixed ticket hangs open.

The fix: both paths run on every invocation, recovery first, each no-oping when idle:

await runRecovery(ctx)  // close everything that recovered
await runAnalysis(ctx)  // triage everything that failed

Immediate blast radius: our "rerun helped" signal was just runAttempt >= 2, which was safe while recovery ran only on green runs. Once it ran always, the bot cheerfully posted "✅ recovered - auto-rerun helped" on a red rerun. Hence the explicit guard: runAttempt >= 2 && failureSet.size === 0. Every check that implicitly relied on the old call order has to become explicit.

Rake #4: The Run Triggered by Someone Else's Deploy

Our e2e lives in the frontend repo and walks the full user scenario through the UI - so it catches backend breakage too, and the backend's staging deploy triggers the same workflow via workflow_dispatch. The GitHub Actions quirk you learn the hard way: a dispatched run has no push context. GITHUB_EVENT_BEFORE is empty, there is no deploy range - and the bot died silently: no ticket, no post. Red staging, and silence. The worst failure mode for a triage system: everyone assumes quiet bot = all under control.

The fix - the deploy range became an explicit input:

- uses: our-org/staging-triage@v1.6
  with:
    deploy-repo: our-org/backend  # whose deploy triggered the e2e
    deploy-before-sha: ${{ inputs.before-sha }}
    deploy-head-sha: ${{ inputs.head-sha }}

Everything downstream - diffs for the verdict, authors for attribution and mentions - now comes from the repo that actually deployed. One triage, both repos.

How We Tested It

You can't unit-test a system whose behavior lives across CI, Linear, Slack, and Allure. Two things worked:

• A smoke PR walked commit by commit through the whole state series - first failure (ticket created?), repeat failure (dedup quiet?), mixed run (both paths fire?), green run (closed + recovery post?) - real runs, real APIs, no mocks.
• A kill switch: manual workflow_dispatch is suppressed unless STAGING_TRIAGE_SIMULATED=true, so you can poke the workflow without spamming Linear.

What the Bot Can't Do

The most dangerous failure mode of an LLM classifier isn't a wrong verdict - it's confident text about nothing: "possibly a flake", "check the environment", "probably a cold start". So the prompt is built on bans, not wishes:

• Verdict first, justified in one sentence with a concrete signal (a diff file, a trace quote, a locator).
• The hypothesis format is keyed to the verdict: likely-pr-related → file:line and what to change; likely-not-pr-related → facts with quotes, what's missing, where a human should look.
• Hard bans: no files or locators not present in the provided context; no generic phrasing without a quote; and insufficient-data is declared a correct, useful answer - otherwise the model invents hypotheses for the sake of pretty text.
• Budgeted context: traces, logs, commit range, full PR diffs and affected files under a 1.5 MB budget; on overflow files drop first, diffs trim from the tail, and a truncation note tells the model not to conclude from missing data.

What still happens: hypotheses right about the area, wrong in the details - the first critical bug the bot caught, it pointed at the right place, but a human wrote the actual fix, just faster. That's fine. The bot doesn't replace an engineer - it replaces the first 15–30 minutes of their work.

Numbers and Gut Feel

Honest part first: we haven't collected rigorous metrics, and two months is a small sample. But releases got noticeably faster - bugs are localized when the code lands on staging, not on release morning, and the 17-suspects detective game is gone. The bug label groundwork means in six months there'll be real numbers.

Cost: one model call per red run (recovery needs no AI), typically 30–80K input tokens - $0.3–0.5 per red run at Opus 4.8 prices, ~$2.5 worst case, $30–70 a month total.

The QA team has grown since - I went from the only QA to leading it - but triage stayed with the bot: people do automation and agent experiments, not red-run duty. The automation was born from a shortage of hands and stayed because it's simply better.

"But off-the-shelf tools exist!" They do. Cognition sells Devin Auto-Triage; GitHub shipped a "Fix with Copilot" button on failed workflow runs. We saw all that and deliberately built our own - because our task isn't "fix the red CI". It's a decision about the fate of a release: classify the cause, pick a route (ticket / rerun / escalation), dedup, auto-close, wake nobody without a reason - welded to our exact stack (Allure TestOps, Linear, Slack, two repos) and our processes.

And the price tag: for an AI-first team this is days, not weeks - the code is almost entirely written with Claude Code. The honest division of labor: the human part is the prototype and the architecture - verdict separated from routing, idempotent recovery steps, pure-function classifiers, state in ticket comments instead of a database. AI writes great code inside a given structure and speeds iteration up by an order of magnitude; the structure that survives six releases in two months is set by a human.

Why not open-source it, then? Similar "AI triages your failed CI" actions sit at 3–11 GitHub stars, and ours needs the exact Linear + Slack + Allure TestOps + Anthropic combination - that audience is vanishingly small, while baseline "AI explains the failure" is arriving in GitHub out of the box. What's worth open-sourcing here are the ideas, not the code - which is exactly what this article is. AI-first means AI-first.