I automated Dependabot PR cleanup - and drew a hard line on what NOT to automate

If you use GitHub, you know the rhythm: Dependabot opens a PR, CI runs green, and then a human has to answer the same questions again - is this a patch or a major? did anyone touch source code? does it conflict? I got tired of answering those questions by hand, so I built dep-triage: a CLI that sorts open Dependabot PRs into five buckets according to a policy file you commit to your repository. 🔗 https://github.com/sunnydachs/dep-triage - 🟢 auto-merge - patch/minor bumps, green CI, dependency-only changes, no conflicts - 🔔 escalate - majors, failed CI, conflicts. Reported, never changed - 🗑️ close - PRs outrun by a newer-version PR for the same package - 💬 rebase suggestion - stale PRs get a @dependabot rebase comment - ⏭️ skip - CI still running, or the diff touches non-dependency files Dry-run is the default. Decisions are fully deterministic - there is no LLM anywhere in the loop. This post is about the line I drew between what to automate and what never to automate, and the three traps real-world data exposed. 1. Only dependency-only diffs are candidates "CI is green" does not mean "the change is safe" - CI can only check what it knows about. So the mechanical gate is: every changed file must be a dependency manifest or lockfile. One stray source file and the PR is out of scope, no matter how nice its diff looks. def scope_check(paths: list) -> dict: if not paths: return {"dependency_only": False, "offending": []} offending = [p for p in paths if not is_dependency_file(p)] return {"dependency_only": not offending, "offending": offending} The check is dumb on purpose - a path list against a known set of manifests and lockfiles. No cleverness, no exceptions to reason about. 2. The world changes between judging and acting Triage says "auto-merge this one". But between that judgment and the API call that enables auto-merge, a new commit can land, or CI can flip red. So --apply re-fetches the head SHA, CI state, and changed files immediately before enabling auto-merge, and aborts to escalate if anything moved: if action == "auto_merge": fresh = api.get(f"/repos/{repo}/pulls/{num}") or {} if fresh.get("head", {}).get("sha") != r["facts"]["head_sha"]: record["action"] = "escalate" # head SHA changed since triage ... ci = api.ci_state(repo, fresh["head"]["sha"]) if not ci["ci_green"] or ci["ci_pending"]: record["action"] = "escalate" # CI state changed before merge ... Classic TOCTOU (time-of-check to time-of-use), but it's the difference between "safe to put in cron" and "safe only while you watch it". 3. Majors are never auto-merged. Ever. Even if your policy listed major as allowed, the never-list wins. And titles that don't parse are escalated, not guessed: bump = facts.get("bump") or "unknown" if bump in policy["never_auto_merge_bumps"]: reasons.append(f"{bump} bump is never auto-merged") return _out("escalate", reasons) The scariest failure mode of automation is behavior that is "usually smart, occasionally dangerous". The dangerous side is capped by construction, not by config discipline. Real data broke it three times I ran the tool (dry-run) against a real public repository with 11 open Dependabot PRs. Three problems surfaced that unit tests could never catch: 1. GitHub's "no CI" trap. A commit with no CI checks at all comes back from the combined-status endpoint as state=pending with statuses=0 . So "no CI" and "CI running" are indistinguishable in the raw response - my first run classified all 11 PRs as "CI still running". Fix: an empty pending status means "nothing to wait for", and that fact (ci_none ) is surfaced in the reasons. 2. Conventional-commit prefixes. Real Dependabot titles look like "chore(deps): bump @nestjs/core from 11.2.1 to 12.0.1" - prefix and lowercase. My ^Bump regex matched nothing. Every PR came back "unknown level". Fix: accept the prefix, case-insensitively. 3. A small propagation miss. The ci_none flag wasn't carried into the decision facts, so the reason line never showed it. Cosmetic, but transparency matters when a bot touches your repo. After the fixes, the same 11 PRs triaged like this: summary: {"auto_merge": 6, "comment_rebase": 1, "escalate": 4} The six auto-merge candidates were all patch/minor. Three of the four escalations were real major upgrades - @nestjs/core 11→12, @types/node 24→26 - and the never-merge rule caught every one of them. That was the moment I stopped worrying about running this unattended. (The last one was a 14-day-old requirements range bump, routed to a rebase suggestion.) Same lesson as always: tests were green; the real world was a different dataset. Using it pip install git+https://github.com/sunnydachs/dep-triage.git dep-triage --repo owner/name # dry-run: print the plan, change nothing dep-triage --repo owner/name --apply # perform actions (needs write token) Policy is one TOML file: auto_merge_bumps = ["patch", "minor"] # levels eligible for auto-merge never_auto_merge_bumps = ["major"] # always wins over the list above require_ci_green = true require_dependency_only = true close_superseded = true rebase_stale_days = 7 merge_method = "squash" Auth is a GITHUB_TOKEN env var; dry-run works with read access only. ※If Dependabot triage is eating your review time too, give it a spin - and tell me where the line should be drawn differently. Top comments (0)