DEV Community

Stop Guessing: Measure Your Flaky Test's Reproduction Rate First

Ask a team about their worst flaky test and you will get an opinion: "it fails sometimes," "it's the checkout one, it's really bad," "just re-run it, it always passes the second time." None of those statements can support a decision.

You cannot tell from "sometimes" whether the test deserves a day of root-cause work, a quarantine slot, or nothing at all - and teams routinely burn days fixing flakes that fail once a month while ignoring the one quietly taxing every pull request.

The fix is not a better opinion. It is a number: how often does this test fail, under real CI conditions? Measure that first, and the decisions mostly make themselves.

Why the measurement has to happen on a real runner

Flakiness is usually a product of the environment - CPU contention, parallelism, network behaviour, timing. A test that never fails on your fast local machine may fail regularly on a loaded CI runner, and a loop run locally measures the wrong machine.

So measure where the flake lives: on the same kind of runner, with the same parallelism, as the job that keeps going red.

The method is deliberately simple:

  • Pick one test - rates are per-test, and a suite-level number hides the offender.
  • Run that single test N times in a loop on a real CI runner, in a job that does nothing else.
  • Count passes and failures, and keep the per-iteration logs so a measured failure is one you can actually inspect.

A minimal GitHub Actions workflow for this is a loop around your test command that writes the pass/fail count and the observed rate into the job summary. CodeOps keeps a free, public version of exactly that instrument - workflow plus a filled-in sample report - at github.com/codeopscontact/flake-scan-demo.

Copy the workflow, point its command input at your suspect test, and run it from the Actions tab.

How many iterations are enough?

Enough to see the flake you suspect. The arithmetic is unforgiving: if a test fails about 1 time in 10, a 20-run loop has roughly a 12% chance of passing all 20 and showing you a clean bill of health that is simply wrong (0.9²⁰ ≈ 0.12).

For a suspected ~1-in-10 flake, use at least 100 iterations before drawing conclusions. For a test you believe fails half the time, 20 runs will show it clearly.

Match the loop size to the rate you are trying to confirm or rule out - and when a fix is claimed, hold it to the same bar: zero failures in a loop at least as large as the one that measured the problem.

What the number changes

With a rate in hand, the fix / quarantine / ignore decision stops being an argument:

  • High rate on a merge-blocking job - fix it now. A test failing 1 run in 10 is not an annoyance; it is a tax on every pull request, paid in re-runs and delayed merges, and it trains the team to ignore red builds - which is how real failures slip through. This one earns root-cause time first.
  • Real but low rate, or a fix that will take a while - quarantine it, properly. Move it out of the blocking path, but with the guardrails that keep quarantine from becoming a graveyard: a named owner, an expiry date, and a cap on how many tests may sit there at once. The rate is what you re-measure to decide whether it comes back.
  • Very low rate, cheap re-run, non-blocking - ignore it, deliberately. A test that fails once in several hundred runs on a job nobody waits for may not be worth a day of anyone's time. "Ignore" is a legitimate decision when it is made from a measured rate and revisited on a schedule - a weekly scheduled scan keeps a baseline, so a rate creeping from near-zero upward shows up before it starts blocking merges.

What is not legitimate is ignoring a flake you have never measured.

Notice what the number does to the conversation. "The checkout test is really bad" becomes "the checkout test fails 8% of runs on the merge-blocking job; the totals test fails 0.5% on a nightly." One of those gets fixed this week. The other gets a quarantine slot or a scheduled watch. Nobody had to win an argument to get there.

Where to go next

Measuring is step one of a longer method: classify the flake into a root-cause class (timing and races, shared state, external dependencies, resource limits, ordering dependence), reproduce it on purpose, apply the fix pattern for its class, and put guardrails in place so the backlog does not regrow.

That full method - the five-class taxonomy with diagnostic signatures, per-class fixes, a quarantine policy template with owners and expiry dates, and a one-week sprint plan - is the Flaky CI Rescue Kit (USD 49).

If you would rather have it done for you, the Flaky Test Repair Sprint (fixed USD 500) root-causes your recurring flakes and delivers the fixes as PRs with before/after reproduction rates: cod************@gmail.com.

But start with the free part. Pick the test your team argues about most, run it 100 times on a real runner, and look at the number. Everything after that is a decision, not a debate.

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.