My GPU Database Lost to a Single CPU Thread. The Bug Was One Constant, 128x Too Small

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry. I maintain a GPU SQL engine - a DuckDB community extension that runs aggregates and GROUP BY on Apple Silicon via Metal. Last night I ran its own benchmark and got a result that should not be possible for a GPU database: gpudb-groupby-bench rows=10000000 groups=1024 [CPU] single-threaded std::unordered_map median wall = 14.65 ms [Metal] Apple M4 Max median wall = 25.39 ms Ten million rows. The GPU took 1.7x longer than one CPU thread walking a hash map. Not a tuning opportunity - an embarrassment. The bug turned out to be a single integer. Fixing it was a 3.0x speedup. But the more interesting part is the second thing I found, which is the reason nobody had ever noticed. Step 1: don't profile, sweep The instinct here is to open a GPU profiler. Don't - not first. A single bad data point tells you nothing about shape, and shape is what identifies the bug class. So I swept the one parameter the workload had: group cardinality. rows = 10M, median of 3 CPU Metal groups = 8 14.62 7.37 GPU wins 2.0x groups = 64 14.52 7.07 GPU wins 2.1x groups = 256 15.84 6.50 GPU wins 2.4x groups = 1024 15.86 25.09 GPU LOSES groups = 8192 27.42 21.23 GPU wins 1.3x groups = 65536 30.37 13.30 GPU wins 2.3x That is not a curve. A curve means "the algorithm degrades." A cliff - 6.50 ms to 25.09 ms between 256 and 1024, then a slow recovery - means a branch. Something in the code chose differently. And it announced itself, once I actually read stderr instead of grepping past it: groups=256 [gpudb metal groupby] using radix-opt path groups=1024 [gpudb metal groupby] using slot-lock path Two kernels. A dispatcher. The cliff was the dispatcher changing its mind. Step 2: the constant // * slot-lock (4K hash partitions x 1K threadgroup slots): // beats CPU 2.5-4x at any size when expected_groups is in // [~1K, ~3M]. Becomes lock-contended at very low cardinality. constexpr std::size_t kSlotLockMinGroups = 1024; constexpr std::size_t kSlotLockSafeCap = 16'000'000; use_slotlock = (expected_groups >= kSlotLockMinGroups && expected_groups 1.81x groups = 2048 23.62 7.85 3.01x 0.64x -> 1.90x groups = 8192 21.13 8.17 2.59x 1.30x -> 3.44x groups = 65536 13.33 8.19 1.63x 2.32x -> 3.77x The GPU stops losing to the CPU. 96/96 unit checks and 72/72 SQL tests still pass, and the benchmark verifies every result against a CPU reference on every run - so those timings are all correctness-checked, not just fast. I checked the crossover holds at other scales before committing to the number: at 1M rows it sits at ~131K, at 50M rows it drifts up to ~196K. A single constant at 131,072 is slightly conservative at 50M (costing ~8% in one cell) and dramatically better everywhere in the 1K-64K band. That trade is not close. Step 4: the part that actually taught me something Before writing any of this up, I asked the question I should always ask about a benchmark finding: does a real query ever hit this? There's a second dispatcher above the Metal one - a hybrid planner that decides CPU vs GPU before the kernel choice is even reached. Its rules: if (n CPU; // hash map is cache-resident if (n > 2'000'000) -> CPU; // sort O(N log^2 N) loses to hash O(N) if (expected_groups CPU; if (expected_groups >= n / 2) -> GPU; // the only GPU door -> CPU; // "borderline", still CPU Work its arithmetic. The GPU door needs n >= 500,000 , so it needs expected_groups >= 250,000 . And 250,000 is already above the 131,072 crossover - so every workload the planner ever sends to the GPU is one where slot-lock is genuinely the right kernel. The bad constant was unreachable through the normal path. I confirmed it empirically too: the first configuration I could find that actually reaches Metal is 1M rows x 1M requested groups, which lands at expected_groups=632357 - comfortably in slot-lock's real territory. So the only thing in the entire project that ever exercised the broken band was the benchmark. Which is to say: the only thing exercising it was the thing that publishes my performance numbers. That's the lesson, and it's a nastier one than "check your constants": Two heuristics in series don't compose. They hide each other. The outer planner was so conservative that it never routed traffic into the region where the inner planner was wrong. From the outside everything looked fine - no slow queries, no complaints, no telemetry anomaly, because no query ever went there. The inner heuristic's error was perfectly masked by the outer heuristic's caution. It could have sat there through every future refactor, waiting for someone to widen the outer rule and quietly hand a 3x regression to real users. A comment claiming [~1K, ~3M] had been sitting above that constant the whole time. It was confidently written, plausible, and wrong by two orders of magnitude at one end. Nothing tested it, because the only code path that could test it was the one the layer above had decided never to take. What I'd take from this - Sweep before you profile. A cliff means a branch; a slope means an algorithm. Five minutes of sweeping told me more than a profiler would have. - Read stderr. The dispatcher was printing which kernel it chose, every single run. I'd been grepping for median wall and scrolling past the answer for weeks. - If a heuristic has an env override, race it. The override existed precisely so someone could ask "is this choice right?" - and nobody ever had. - Put the measurement in the comment, not the conclusion. // beats CPU 2.5-4x in [~1K, ~3M] is unfalsifiable prose. A table of numbers with the hardware named is something the next person can re-run and disagree with. - Benchmark the parameter your heuristic keys on. The suite measured throughput at one cardinality. The bug lived in the derivative - how behavior changed with cardinality - which no single-point benchmark can see. - Ask whether the bug is reachable, before you claim impact. I nearly wrote this up as "3x faster GROUP BY" and stopped. It's 3x faster on the operator path and the published benchmark; it is currently 0% faster for a SQL user. Saying so is the difference between a finding and a press release. The constant was wrong for months and cost nobody anything - yet. That's not a reason to relax. A latent 3x is just a regression with a delay on it. Top comments (0)