I built a Forge major-version predictor on a guess, and it took a review to find the answer inside my own commit

I built a Forge major-version predictor on a guess, and it took a review to find the answer inside my own commit Key takeaways - END STATE: a predictor that PARSES two manifests and reports every documented major-version trigger - not a regex over the shapes your own manifest happens to use. - There are eleven documented triggers, on TWO pages: nine on the versions page, plus remotes and customer-managed egress - and the llm module documents its own, elsewhere. - Our 4.0.0 had TWO independent triggers: a new scope AND the llm module. Either alone would have forced it. - Regex extractors fail in the dangerous direction: re-indent a scopes list and every scope vanishes, so a real new scope reads as minor. - forge version bulk-upgrade can apply eligible majors WITHOUT site-admin approval - a major is not automatically a stalled rollout. We shipped a release of Sentinel Vault that went from 3.x to 4.0.0, and our deploy note records the reason as "major bump from the new llm module". I decided that note was wrong. The llm module was the headline feature, the version went up, and I read that as the classic post-hoc mistake - because the same commit also added one scope, read🏷️confluence , and a scope is the trigger everybody knows about. So I wrote a tool, and an article, explaining that the module was innocent and one line of YAML had done it. That was wrong, and the correction was sitting in the same commit the whole time. The manifest comment, written when the module was added, reads: # Atlassian-hosted LLM (Forge LLMs) - powers Semantic AI Validations with no # external egress, so the app keeps its "Runs on Atlassian" badge. Adding this # module triggers a major version bump + admin re-consent. Atlassian's llm module reference says the same thing outright: "Adding the *llm module to your manifest will trigger a major version upgrade."* Two independent triggers fired on that release, not one. Either would have forced 4.0.0 by itself. The note was right, my correction was the misattribution, and the tool I built to stop people guessing was itself built on a guess - it knew about six triggers when the documentation lists eleven, across two different pages. This is that tool, rebuilt properly, and the specific ways the first version was wrong. Prerequisites Node 18 or later. Verified on v24.15.0. Two versions of a manifest.yml - most easilygit show :manifest.yml for the release you are comparing against.No Forge app, no deploy, no tunnel. The analysis is a diff of two files, so it runs offline. Useful but optional: @forge/cli , if you want to confirm the prediction against a real deploy in step 5.About forty minutes. Why this matters more than a version number A major version is not cosmetic. Atlassian's versions documentation is explicit: By default, major version upgrades are not applied to an app installation immediately. This is because major versions involve significant changes that may require users and admins to re-consent or review the changes before continuing. So a major stops auto-upgrading. But it is not automatically a stalled rollout, and I had that wrong too. The same page: For eligible major version updates that don't require an escalation in privilege, you can use the forge version bulk-upgrade CLI to start, cancel, and track updates in large batches without site admin approval. So the cost depends on whether the major escalates privilege. A new scope does; a change that does not escalate can often be pushed with bulk-upgrade , and Rolling releases let you ship code while admins approve permissions separately. The thing to avoid is a surprise privilege escalation, not a major version as such. The rule that decides it is one sentence: Not all permission changes trigger a major version upgrade. Only changes that require user consent, such as OAuth scopes and Atlassian app permissions, result in a major version change. Consent, not size. That is why a workflow engine can be a minor and one line of YAML cannot. - Enumerate the triggers from the documentation, and note there are more than one page of them. - Parse the manifest instead of pattern-matching it this is the difference between a tool that works on your repo and one that works. - Compare the two parsed manifests trigger by trigger, respecting which removals count. - Prove each trigger fires, including the ones your own manifest has never used. - Confirm against a release that already shipped, and reconcile any disagreement. - Wire it into the path you deploy from. Step 1 - Enumerate the triggers, from more than one page The versions page lists nine bullets: adding, swapping or removing a scope; adding or swapping a content CSP option; adding or swapping an external CSP option or URL; adding a dynamic web trigger or making a static one dynamic; adding or modifying the category of an existing egress permission; flipping inScopeEUD from false to true for the first time; enabling licensing; adding or removing providers; and changing a provider client ID. Below that list, two more in prose: "In most cases, updating your app's remote backends will result in a new major version", and adding permissions.external.configurable.enabled for Customer-managed Egress. That is eleven. And there is a twelfth that is not on that page at all - the llm module documents its own trigger in its own reference. That is the one that caught me, and it is the reason to state the snapshot date on any list you encode: export const TRIGGERS_SNAPSHOT = "2026-08-29"; How you know it worked: count what you encoded and compare it against the page, not against memory. node -e "import('./predict.mjs').then(m => console.log(m.TRIGGERS.length, m.TRIGGERS.map(t=>t.key).join(', ')))" If your count is smaller than the page's, you have already built the bug I did. Mine said six. Step 2 - Parse the manifest, do not pattern-match it This is the step that decides whether the tool is real. My first version used regexes with hard-coded indentation - scopes at four spaces under permissions: . It worked perfectly on our manifest and failed silently on anything shaped differently, in the worst possible direction: the extractor returned an empty list, an empty list compared to an empty list is no change, and a genuine new scope reported minor . A manifest is YAML. Parse it: import { createRequire } from "node:module"; const require = createRequire(import.meta.url); const yaml = require("js-yaml"); const get = (o, path) => path.split(".").reduce((a, k) => (a == null ? a : a[k]), o); const asList = (v) => (Array.isArray(v) ? v : v == null ? [] : [v]); /** Deterministic deep signature, independent of key order. / function sig(v) { if (v === null || v === undefined) return "null"; if (Array.isArray(v)) return "[" + v.map(sig).sort().join(",") + "]"; if (typeof v === "object") { return "{" + Object.keys(v).sort().map((k) => ${k}:${sig(v[k])}).join(",") + "}"; } return String(v); } sig() is what makes reordering and reformatting safe properly. My regex version sorted raw lines to achieve the same thing, which threw away nesting - and that had a real consequence, covered in step 4. How you know it worked: re-indent your manifest and confirm nothing changes. python3 -c "import yaml,sys; print(yaml.safe_dump(yaml.safe_load(open('manifest.yml')), sort_keys=False))" > reindented.yml node predict.mjs manifest.yml reindented.yml You should see minor and triggers: none of the 9 fired . A parse-identical file that reports MAJOR means you are still comparing text somewhere. Step 3 - Compare trigger by trigger, and get removals right Each trigger becomes a set of comparable items, and the comparison is a set difference. The subtlety is that removal does not count everywhere. export const TRIGGERS = [ { key: "scopes", label: "OAuth scopes", removalIsMajor: true, items: (m) => asList(get(m, "permissions.scopes")).map(String) }, { key: "external", label: "External permissions", removalIsMajor: false, items: (m) => Object.entries(get(m, "permissions.external") || {}) .flatMap(([cat, v]) => typeof v === "object" && !Array.isArray(v) // keep the CATEGORY in the key, so moving a URL backend->client is a change ? Object.entries(v).flatMap(([sub, u]) => asList(u).map((x) => ${cat}.${sub}=${sig(x)})) : asList(v).map((x) => ${cat}=${sig(x)})) }, { key: "llm", label: "Forge LLM module", removalIsMajor: false, items: (m) => asList(get(m, "modules.llm")).map((x) => llm:${x && x.key}) }, { key: "providers", label: "Providers", removalIsMajor: true, // TOP-LEVEL key - not nested under anything. items: (m) => Object.entries(get(m, "providers") || {}) .flatMap(([kind, list]) => asList(list).map((p) => ${kind}:${sig(p)})) }, { key: "remotes", label: "Remote backends", removalIsMajor: false, items: (m) => asList(get(m, "remotes")).map((r) => sig(r)) }, // …scopes' siblings: content, webtrigger (dynamic only), licensing, configurable egress ]; Three details, each of which I got wrong first time. Removal is a major for scopes, and not for the others. The docs list "Removing a scope" explicitly, but for content and external they list only adding and swapping - and for remotes they say the opposite outright: removing an entry, or any update that decreases a remote's scope, is a minor. A blanket "removals count" rule produces false majors. providers ** is top level.* Mine looked for it nested and therefore never fired at all - adding a provider, changing a client ID, swapping one provider for another, all reported minor . A trigger that cannot match is worse than an absent one, because the output looks complete. Only dynamic web triggers count. The documented cases are adding a dynamic trigger and changing a static one to dynamic. Counting every web trigger produces false majors on ordinary module additions. How you know it worked: run it on the real commit and read the trigger list, not just the verdict. node predict.mjs before.yml after.yml Real output from the commit that took Sentinel Vault to 4.0.0: bump :