Automating Android Play Store Releases, Part 3: The Storage-Quota Wall
The Wall
We'd stopped thinking about this pipeline. That's usually the sign a system is working - until a release run failed with:
Run actions/upload-artifact@v4
Error: Failed to CreateArtifact: Artifact storage quota has been hit. Unable to upload any new artifacts. Usage is recalculated every 6-12 hours.
GitHub's free organization plan includes a fixed Actions storage allowance - not per-repo, org-wide, and small: 500MB, total, forever. That number is really the punchline of this entire post - everything below is different ways of finding out what actually filled it.
Checking Usage Properly
Checking usage properly meant going past the summary numbers and actually listing every artifact and every cache GitHub was holding for this one repository:
gh api repos/<org>/<repo>/actions/artifacts --paginate \
-q '.artifacts[].size_in_bytes' | \
awk '{sum+=$1; n++} END {printf "count=%d total_MB=%.1f\n", n, sum/1024/1024}'
gh api repos/<org>/<repo>/actions/caches --paginate \
-q '.actions_caches[] | [.key, .size_in_bytes] | @tsv'
Artifacts alone accounted for nearly two gigabytes - signed release AABs at roughly 70MB apiece, internal test APKs at 20-40MB apiece, accumulated over about two weeks. But a cache-by-cache audit turned up something bigger.
The Real Dominant Cause: Ten Copies of the Same Cache
The Gradle dependency cache - meant to be one cache, reused across runs - showed up as ten separate entries, all effectively identical, together accounting for the largest single share of the quota by far.
GitHub scopes Actions caches per-branch by default: a cache saved on one branch isn't shared with another, it's a full new copy. Ten active branches meant ten full copies of the same dependency set, sitting in storage simultaneously, none of them expiring on their own because each one kept getting refreshed by ongoing work on its branch.
This, not the artifact files, was the actual wall. It's also the least visible of everything in this post - nobody sees "10 duplicate caches" in an error message. You only find it by going and actually listing what's in storage instead of assuming you already know.
A Cache That Was Never Actually Caching
Digging into why ten copies existed at all - rather than one cache correctly reused across every branch - turned up a second, compounding bug: the cache key itself was broken.
key: ${{ runner.os }}-gradle-${{ hashFiles('android/gradle/wrapper/gradle-wrapper.properties', 'android/build.gradle') }}
Both paths live in an android/ directory that is .gitignored and only generated later, inside the build script's own prebuild step. Neither file existed yet at the point this cache step ran. hashFiles() against paths that don't exist returns an empty string every single time - so the key never changed based on real config, every run was silently a cache miss dressed up as a cache step, full re-downloads of every Gradle dependency every single build, and (worse, for the quota) every branch's "identical" cache got saved fresh under the same static key instead of ever being correctly reused.
Two bugs, same root: a cache system that looked like it was working, doing none of its actual job, on both counts.
Retention Set for "Might Need It Someday," Not "Might Actually Look at It"
The signed release AABs - the 70MB files driving most of the artifact total - had 30-day retention. That number had never been a deliberate choice; it was just Actions' own suggested default, left as-is.
The AAB itself gets published to the Play Store in the very same job, seconds after the GitHub copy is uploaded - so the GitHub copy exists purely as a debugging convenience, in case someone needs to grab the exact file that shipped without re-running a build. Nobody had ever actually needed to reach for a 3-week-old copy of one.
Thirty days of retention on a large binary that's a debugging convenience, not the deliverable, was pure waste that nobody had sized against what the free storage tier could actually hold.
One Leak That Was Already Closed
Worth naming, if only to be precise about timing: an earlier, unrelated cleanup had already caught and fixed a real duplicate-storage bug - the internal-testing flow used to upload every built APK to both cloud storage and GitHub Actions artifacts, with the GitHub copy never actually used by anything. That fix landed weeks before this specific wall.
What the quota investigation actually found wasn't that bug recurring - it was old, already-orphaned artifacts left over from before that earlier fix, still sitting in storage because nothing had gone back to clear the backlog once the leak itself was closed.
A useful reminder on its own: fixing where a duplicate gets created doesn't retroactively delete duplicates that already exist.
GitHub also documents a clean pattern for exactly the kind of cache pileup described above: delete a cache the moment the branch or PR it belongs to closes, rather than waiting on an age-based sweep to eventually catch it. Adding that alongside the key fix meant a closed PR's caches stop costing anything the same day, not whenever a weekly cleanup job gets around to them.
The Cleanup That Wasn't the Real Fix
Deleting the backlog and tightening retention buys time. It doesn't change the underlying fact: as long as this pipeline runs on GitHub-hosted infrastructure, it is subject to GitHub-hosted quotas - minutes and storage both, shared across the whole org, and rationed the same way EAS's free build queue was rationed back in Part 1. A quiet accumulation of artifacts had just demonstrated that the exact ceiling we moved off of EAS to escape was still there, one layer up.
The real fix was to stop depending on that shared allowance at all: move the pipeline's actual build job onto a self-hosted runner - infrastructure we control, with disk space and build minutes that aren't rationed by anyone else's billing plan.
That's a bigger decision than a cleanup script, and it came with its own first-run surprises, starting with a runner image that turned out to be missing a tool GitHub's hosted images ship by default (/usr/bin/time, used purely to capture build duration and peak memory for diagnostics) - a small, fast fix once the actual migration was live, but a reminder that "self-hosted" means you now own every assumption a hosted image used to make for you silently.
Where Part 3 Leaves Off
The build job no longer runs anywhere GitHub's shared minutes or storage quota can touch it. That closed off the cause of the wall we hit. It didn't yet close off the symptoms - the AAB still sitting in GitHub storage after every publish, the retention nobody had sized - and fixing those surfaced one more bug, this time inside the very notification system meant to tell us when something went wrong.
Next: Part 4 → - killing the artifact-relay design that was quietly reporting real successes as failures, and the first real production ship after all of it.
I write about the debugging journeys nobody puts in the docs - more at wamzai.com.
Comments
No comments yet. Start the discussion.