Launching vizcrush: Three Beliefs My Benchmarks Killed

It's the week before vizcrush goes public, and I have two files open side by side. On the left, the launch copy: the JS core beats the most popular npm downsampling package by 32×, "and WASM adds another 5-10x on top." On the right, the repo's own benchmark control run: wasm/js ≈ 1.00× . One million points, same algorithm, same machine. Parity. I go looking for the measurements behind the claim. Half of it holds up: the 32× JS comparison has a result file (1.72ms against 55.52ms, real). The claimed additional 5-10× from WASM has nothing behind it, and the repo's own control run contradicts it. That afternoon set the shape of the whole launch: before anything shipped, every performance claim would either get a measurement behind it or get deleted. Three beliefs didn't survive. Each one got a public retraction, written up as an ADR in the repo. vizcrush is a set of data primitives for browser visualization (downsampling, binning, spatial indexing, streaming sketches), written in Rust, compiled to WebAssembly, with a pure-JS core behind the same API as a fallback and explicitly selectable backend. It went open source this week: the repo and the book are public, and all 11 packages are live on npm. npm install @vizcrush/core @vizcrush/downsample This is a launch story about turning benchmark results into product policy: claims, documentation, and WebGPU policy follow the measurements, while WASM dispatch stays availability-based pending further investigation. One scope note before the data. Every result here is workload-specific: LTTB (Largest-Triangle-Three-Buckets, the downsampling algorithm that picks, per bucket, the point that best preserves the visual shape of the line) is downsampling, the stats kernel is a reduction, and bin2d is histogramming. Which backend wins is algorithm- and engine-dependent, so none of what follows is a library-wide WASM-versus-JS verdict. It is three specific workloads measured on specific engines, with the claims and documentation following each measurement. Belief one: SIMD is doing the work The WASM build had carried -C target-feature=+simd128 from early on, and the docs treated the flag as the fast path: a doc comment in stats.rs even described a "SIMD pre-scan" over the data. Then that control run showed the public WASM path at parity with the JS core, and the obvious question landed: is SIMD engaged at all? The test is simple: build the module twice, with and without the flag, and hash the binaries. The downsample module built with +simd128 is byte-identical to the scalar build: same SHA-256. The flag produced zero different code for LTTB. The aggregate module differs by a few hundred bytes, so the flag changed code generation somewhere. And then the timings say it doesn't matter: | Algorithm | Size | wasm-simd | wasm-scalar | js core | |---|---|---|---|---| | lttb | 100K | 157µs | 160µs | 250µs | | lttb | 1M | 1.85ms | 1.81ms | 1.82ms | | lttb | 10M | 16.79ms | 16.61ms | 16.65ms | | compute_stats | 1M | 4.06ms | 4.03ms | n/a | | compute_stats | 10M | 41.15ms | 40.72ms | n/a | Takeaway: for these measured kernels, enabling the flag produced no material runtime change. SIMD-on and SIMD-off are the same speed at every size, even for the stats loop I would have sworn was vectorizable. Method, since numbers without one are how this mess started: raw WASM exports called directly (so marshalling doesn't blur the comparison), Node v24.14.1 on darwin/arm64, release profile (opt-level=3 , LTO, a single codegen unit), one machine: treat the absolutes as approximate and read the ratios. And the gaps, stated instead of papered over: the Node result files retain median, p95, and minimum per case, but percentile bands like p10/p90, allocation counts, and machine power-state control are not captured yet, in this harness or the others; that's noted as future benchmark work. The likely reason is mundane. These hot loops branch on every element: LTTB's core is an argmax (if area > max_area on each candidate point), and the stats kernel skips non-finite values and tracks min/max, all branches. Branch-heavy loops like these did not autovectorize in these builds, and there are no hand-written SIMD intrinsics in the Rust to force the issue. The "SIMD pre-scan" comment described code that does not exist; it's gone. THE WASM-OPT SUBPLOT The same investigation found the shipped binaries had never been run through wasm-opt : the build script only invoked it if binaryen happened to be installed, and it wasn't, so the step silently skipped for the project's entire life. Fixing the script (it now fails loudly) bought a ~10-12% smaller binary and no runtime gain (LTTB measured 1.05-1.10×, marginally slower; stats 1.00×). The decision in ADR 0002 is deliberately boring: keep the flag, since it's harmless and occasionally shrinks a binary, but don't hand-write intrinsics, because at the input sizes that matter these algorithms are memory-bandwidth bound, and vectorizing the compute would optimize something that isn't the bottleneck. And delete every SIMD speedup claim from the docs. Belief two: WASM is fast everywhere Killing the SIMD claim left a follow-up question: why keep WASM at all? ADR 0002's answer was cross-engine consistency: JS engine performance varies wildly, WASM is predictable, so WASM is the safe default. It sounded right. It was also untested, because every measurement so far had run in Node. Which is to say: in V8, one engine. So the follow-up drove real browsers: playwright-core driving locally cached browser binaries, specifically Chromium (V8), Firefox Nightly rv:144 (SpiderMonkey), and WebKit 2227 (JavaScriptCore), running the raw wasm-bindgen LTTB export against the JS core, served over local HTTP and timed in-page. One measurement trap worth passing on: Firefox and WebKit coarsen performance.now() to about a millisecond as a Spectre mitigation, so per-call timing returns zeros and round milliseconds. The workaround is batch timing: run N calls as one block, take the minimum over several reps, divide by N. Anything not listed in ADR 0003's method section, headless versus headed mode per engine, machine power conditions, was not controlled for. | Engine | Size | wasm | js core | wasm/js | |---|---|---|---|---| | Chromium (V8) | 100K | 150µs | 643µs | 0.23× | | Chromium (V8) | 1M | 1.52ms | 6.03ms | 0.25× | | Firefox (SpiderMonkey) | 100K | 1.97ms | 268µs | 7.37× | | Firefox (SpiderMonkey) | 1M | 17.2ms | 2.13ms | 8.09× | | WebKit (JavaScriptCore) | 100K | 188µs | 142µs | 1.32× | | WebKit (JavaScriptCore) | 1M | 1.98ms | 1.38ms | 1.44× | Takeaway: WASM is a decisive win in Chromium, about 4× faster than the JS core; it's roughly 8× slower in Firefox at a million points and modestly slower in WebKit, cold start (first call, instantiation and JIT warmup included) is slower than JS in every engine, and the JS core is the more consistent of the two (1.4-6ms across engines at 1M points, against WASM's 1.5-17ms). Method: every cell is a batch-timed minimum, the minimum over several reps of an N-call block divided by N, per the ADR's method section; that minimum is the recorded data, so within-engine run-to-run spread was not kept. BEFORE QUOTING THESE NUMBERS Within-engine wasm/js ratios are the trustworthy output; the absolutes are runtime-dependent (headless Chromium's JS core is about 3× slower than Node's on the same V8, so never compare absolutes across runtimes). And Firefox's 17.2ms WASM is flagged, not root-caused: per-call marshalling in SpiderMonkey is a hypothesis, nothing more, and the repo tracks it as an open investigation (ADR 0003 lists root-causing it as a precondition for any per-engine dispatch). The qualitative conclusion doesn't depend on it: WebKit also shows WASM slower, cleanly. ADR 0003 keeps WASM anyway, for an honest reason instead of a wrong one. A ~4× Chromium/V8 win is worth preserving rather than regressing. But the consistency rationale is retracted in writing, and the README now carries the framing the data supports: substantially faster than the JS fallback in Chromium/V8, comparable-to-slower in Firefox and Safari. Belief three: WebGPU is the endgame This one is the most embarrassing, so it gets the full story. The repo had carried five WGSL compute-shader drafts since the original spec, never wired to a dispatch path. At one point the docs claimed the WebGPU path was "~10× faster on a million-point input over WASM" and auto-selected when available. There was no WebGPU path. Not a slow one: none. The claim went out in the honesty pass, but that left a fork: delete the shaders too, or wire one up and get a number. We wired the most GPU-favourable draft (bin2d , a 2D histogram, embarrassingly parallel, atomicAdd with workgroup-local accumulation) into a real compute path: lazy device acquisition with device-loss reset, cached pipeline, upload, dispatch, readback. Every failure mode resolves to a silent fallback onto the wasm/js kernel; the GPU path never throws into user code. The one genuinely interesting engineering problem on the way there: WGSL has no f64, and vizcrush's inputs are Float64Array , where a realistic x-axis is epoch-millisecond timestamps, thirteen-digit numbers. Narrow those to f32 directly and the 24-bit mantissa can't distinguish nearby milliseconds at that magnitude; neighbouring timestamps collapse onto the same float, and bin assignment quietly breaks. The fix is to rebase before narrowing: subtract the range minimum in f64, then convert, so the shader only ever sees small offsets, exact for up to ~2^24 distinguishable values per axis. Bin edges are computed and returned in f64; the GPU never touches them. Then we measured it the way a caller experiences it, end-to-end: rebase, upload, dispatch, readback. Chrome 150 on Apple Silicon (Metal 3), 256×256 grid. The harness is committed at benchmarks/webgpu-bin2d.html : serve the repo root with python3 -m http.server , open the page in a WebGPU-capable browser, and it reruns the whole table