22 tests passing, and my solver still told you to click a mine

I wrote a minesweeper probability solver. You hand it a board, it hands back the probability that each unknown square is a mine, so that when logic runs out and you have to guess, you at least guess the cheapest square. It has 22 unit tests. They all pass. I also cross-checked it against an independent brute-force implementation - one that enumerates every legal mine placement and counts - over 8,000 random boards. Maximum deviation: 5.551e-16 . That is floating-point noise. There is no meaningful sense in which those two implementations disagree. Then I integrated it into a website, played about 300 games, and hit a board where the solver reported six squares as certain mines. Two of them actually were. This is the story of how a test suite that green can miss a bug that bad. First, the part that works The core is standard constraint counting. Every revealed number is a constraint: "exactly K mines among these N unknown neighbours." The solver splits constraints into connected components by shared unknowns, enumerates every legal mine configuration per component indexed by mine count (total[k] = number of configurations using k mines), then convolves components together, weighting each global total T by C(outsideCells, remainingMines - T) . That last weighting step is the part most naive solvers skip, and it matters enormously. Here is a 5x3 board with one mine left and two 1 clues: · 1 · · · · · · · · · · · 1 · The two clues' neighbourhoods intersect in exactly one square, (2,1) . Only one mine remains. If the mine were anywhere other than that intersection, you would need two mines to satisfy both clues. So (2,1) is a guaranteed mine, and all 12 other squares are guaranteed safe. Enumerate the frontier without the global mine budget and you get this instead: .24 1 .24 ? ? .24 .24 .06 .24 .24 ? ? .24 1 .24 (2,1) comes out at 0.0588 - the lowest number on the board. A solver using that would recommend clicking the one square that is certain to lose. Correct answer 1.0, naive answer 0.06. The gap doesn't just shift confidence, it inverts the decision. That is the part I tested heavily, and it works. The board that shouldn't exist The failing position reported six certain mines when only two were real. What made it obviously, mathematically broken was a different number: the probabilities across the whole board summed to 13.96, on a board with 10 mines. The sum of per-cell mine probabilities has to equal the number of mines remaining. Not approximately - exactly, by linearity of expectation. A sum of 13.96 on a 10-mine board isn't a rounding artifact. It means the output isn't a probability distribution over anything. The bug When a component's frontier grows past maxComponentCells , exhaustive enumeration explodes combinatorially, so the solver falls back to a mean-field approximation - iterative relaxation, 12 rounds, each cell estimating its share of each constraint's residual: // Subtract the current estimate of the other open cells to get this // cell's share, then clamp - this is the mean-field step. let sumOpen = 0; for (const i of open) sumOpen += p[i]; for (const i of open) { const others = sumOpen - p[i]; let est = residual - others; if (est 1) est = 1; acc[i] += est; cnt[i] += 1; } Those values are estimates. They are not enumerated, not exact, and carry no guarantee. Some of them land very close to 0 or 1. Now here is finalize() , as it was (src/solver.js , before the fix): const EPS = 1e-9; function finalize(probabilities, rows, cols, stats, board) { const safe = []; const mines = []; for (let y = 0; y = 1 - EPS) { probabilities[y][x] = 1; mines.push({ x, y }); } } } EPS exists for a good reason. Exact enumeration involves binomial weighting and division, so a genuinely safe cell can come back as 3e-17 rather than a clean 0 . Snapping that to 0 is correct and necessary - without it the library can't do its main job. The problem is that finalize() receives a grid of numbers with no record of where they came from. An estimated 0.9999999999 from the mean-field relaxation and a truly-enumerated 1 - 2e-17 are the same float by the time they arrive. So the threshold promotes both into mines[] . EPS is a tool for absorbing floating-point noise in exact arithmetic. Applying it to a heuristic's output launders an estimate into a proof. And a caller - my website - takes safe[] and paints those squares green, meaning "this one is guaranteed, click it freely." The user clicks. It's a mine. The fix that didn't work The obvious fix: skip cells that live inside approximated components, keep certainties from exact components. Estimates stay quarantined, proofs stay proofs. Reasonable. I measured it against known mine positions across 12 Expert boards that triggered approximation. 65 of 244 claimed certainties were still wrong. The reason took me a while, and it's the most interesting thing I learned here. Contamination is not local. Look at how the global budget is computed: let approxExpected = 0; for (const a of approxComponents) approxExpected += a.expectedMines; let effectiveRemaining = hasTotal ? Math.max(0, Math.round(remainingMines - approxExpected)) : null; An approximated component contributes an estimated mine count to approxExpected . That estimate is subtracted from the global budget. Then every exact component divides by that same effectiveRemaining when it does its C(outside, leftover) weighting. A wrong budget makes exhaustive enumeration produce confident, wrong numbers. The enumeration is still perfectly correct - it correctly answers a question about a board that doesn't exist. The Math.round() makes it worse by quietly converting a fractional estimate into a hard integer. So the guard has to be global: /* * If ANY component was approximated, no cell on the board earns a * certainty - not even one in a fully enumerated component. * ... * Probabilities are still reported for every cell - callers can render * the gradient and pick a best guess. What is withheld is the claim * "this is proven". */ if (approx.size > 0) continue; If any component anywhere was approximated, nothing on the board gets certified. Probabilities are still returned for every cell, so the colour gradient and the best-guess pick both still work. The only thing withheld is the claim of proof - which is the only part that can get someone killed. The second bug, found while measuring the first To measure any of this I needed to know how often the approximation path actually ran. The cutoff was: const DEFAULT_MAX_COMPONENT_CELLS = 24; 24 sounds like a safety valve for pathological boards. It wasn't. Measured over random Beginner (9x9, 10 mine) positions, the approximation was engaging on ordinary mid-game boards - on the smallest standard difficulty in the game. A cutoff the easiest board doesn't reliably clear isn't a fallback, it's the default path. Raising it to 32 took Beginner approximation to 0 out of 200 positions in my measurements. And the cap wasn't paying for itself anyway - timing the same 200 boards, forced enumeration ran at 0.10ms per board versus 0.24ms when pushed through the approximation. Constraint pruning kills most branches immediately, so enumerating was faster than approximating. The optimisation was a pessimisation that also happened to be unsound. Verification Same 400 generated Expert boards, old code versus new, certainties checked against actual mine positions: BEFORE: 400 boards, 162 approximated, 8292 claimed certainties, 1089 WRONG AFTER: 400 boards, 101 approximated, 3165 claimed certainties, 0 WRONG Roughly one in eight "guarantees" was a lie. The AFTER run still emits 3,165 certainties and every one checks out, so the guard didn't overcorrect into uselessness. The 22 original tests still pass; slowest of 120 Expert solves was 8.03ms. What I actually take from this The 22 passing tests were real. The 8,000-board cross-validation at 5.551e-16 was real. Neither was fake, sloppy, or badly written. They just all covered the same path. Every one of those tests exercised exact enumeration. The brute-force reference implementation can only handle small boards - that's what makes it a usable oracle - and small boards never trigger the approximation. My strongest piece of correctness evidence was structurally incapable of reaching the broken code. The more I leaned on it, the more confident I got about a region of the input space it could never visit. There were two tests touching the approximation path. Both asserted things like "returns promptly" and "probabilities stay in [0, 1] ". Both passed throughout. Neither asked the one question that mattered: are the certainties it claims actually true? The bug wasn't inside either path. It lived in the seam - a well-tested exact path, an untested approximate path, and an output layer that couldn't tell them apart because by then they were both just float s. Type systems don't catch this. Coverage tools report finalize() as fully covered; it was fully covered, by data from one path only. The part that still bothers me is the failure mode. This bug didn't throw. It didn't return NaN , didn't hang, didn't corrupt memory, didn't turn anything red. It calmly returned a confident, well-formed, wrong answer, and every automated check I had agreed it was fine. If it had crashed I'd have found it in the first hour. The dangerous failures aren't the ones that look broken. They're the ones that look right. If a function's output can come from two sources with different guarantees, the weaker guarantee has to travel with the data. Otherwise something downstream will eventually treat all of it as the stronger one. The solver is MIT-licensed and zero-dependency, single file, if you want to read the actual code or check my arithmetic: github.com/aijobclub/minesweeper-solver. The headline demo (node demo.mjs headline ) prints that 5x3 board comparison if you want to see the 0.06-versus-1.0 inversion yourself. There's a playable version if you'd rather poke at boards than read