WebGPU and SMs: A Post-Mortem of an OOM in a Hand-Written WGSL Kernel
Spent today fixing the cowshed. Busy as ever.
For singlefs's differential testing (or rather, value cross-checking), we use WGSL. WGSL is WebGPU's shading language, and on the Rust side we call into it through a library called wgpu. wgpu translates the WGSL and hands it to the GPU driver to run; on Linux we go through Vulkan. Put simply, it carves up the GPU's units so the GPU can crunch enough data in a single batch.
So what are we cross-checking? If a filesystem loses power halfway through a write, it leaves a "crash state" on disk. singlefs has to capture every possible crash point and check, one by one, whether what's on disk is still correct. These states easily run into the hundreds of millions, far more than a CPU can judge, so the whole checker was moved onto the GPU.
The way we use the GPU here is a bit different from gaming or AI: one GPU thread is responsible for judging an entire crash state. Each state needs its own "scratch area" laid out in VRAM, and in the worst case a single state takes about 7.67MB. So for us, how many states one card can judge at once comes down almost entirely to how much VRAM we can allocate. Compute isn't really the bottleneck.
In a recent round of optimization, the GPUs were organized into an independent card pool that starts cross-checking data batch by batch once a certain watermark is reached. And then the problem showed up...
The OOM Loop
While working, the 5060 Ti and 5080 sat at 14GB and 15GB of VRAM respectively, at 99% utilization. But the 5090's VRAM usage kept swinging between 3GB and 17GB and never settled. At one point nvidia-smi showed a 32GB card using only 3.5GB, at 0% utilization. The logs showed it was hitting OOM.
In the logs, the 5090 went around the loop about once a second: load a checker, open a 29.4GB scratch area, OOM, put the data back in the pool, load again, OOM again... In one hour of logs it loaded a checker 2,294 times, hit OOM 1,791 times, and judged exactly 0 states. In other words, the 5090 looked busy, and it really was busy. It just wasn't producing anything.
Faulty VRAM Accounting
After some digging, it turned out the cards have extra overhead while working: about 0.8GB on the 5060 Ti, about 2.0GB on the 5080, and about 3.4GB on the 5090. But the original code only left 2560MB free on each card.
Here's how the original accounting worked:
- usable VRAM on a card = min(configured quota, free VRAM when the card is opened)
- scratch area size = usable VRAM − 2560MB
The funny part is that this 2560 was originally measured on a 5090. Back then we measured 2326MB and rounded up to 2560. But in the card-pool service, the 5090's extra overhead had grown to 3.4GB. We haven't yet broken down where that extra 1,000-plus MB comes from. So the card really only had 28.6GB free, not 29.4GB.
On top of that, after an OOM the program just put the data back in the pool and changed nothing else. The next round ran the same accounting, came up with 29.4GB again, and OOMed again.
At first I also suspected there were too many checkers loaded on one card. But with 1 checker or 3 on the 5090, the extra overhead was 3.3-3.4GB either way. With 4 running at once on the 5060 Ti, it was still only 0.8GB. So it has nothing to do with the number of checkers.
SMs and Per-Thread Local Memory
So where do the 5060 Ti's ~0.8GB, the 5080's ~2.0GB and the 5090's ~3.4GB come from? The most basic compute unit in a GPU is called an SM, a Streaming Multiprocessor. CUDA cores, RT cores and AI cores aren't scattered around the chip; they're packed into each SM in fixed proportions.
Each SM typically contains:
- Compute units: 128 CUDA cores (general-purpose compute and rendering), 4 Tensor Cores (AI matrix math) and 1 RT Core (ray tracing). So the card's total CUDA core count = number of SMs × 128 (e.g. the RTX 5090 has 170 × 128 = 21,760 CUDA cores).
- Schedulers (Warp Schedulers): these feed work into the compute units in groups of 32 threads (called a warp). When one group is waiting on data from VRAM, the scheduler immediately switches to another group to hide the latency. To always have something to switch to, each SM has to keep a lot of threads resident at once. On the 50 series, each SM can hold up to 48 warps, i.e. 1,536 threads.
- On-chip fast memory: registers, plus L1 cache and shared memory (typically 128KB per SM on recent consumer generations, and 256KB on the data-center H100).
GPU computation is organized in this hierarchy: the whole GPU chip → GPC (Graphics Processing Cluster) → TPC (Texture Processing Cluster) → SM (Streaming Multiprocessor) → the individual cores. You can picture it as lots of small processors working together to form a cluster.
Different cards have different SM counts: 170 on the 5090, 84 on the 5080 and 36 on the 5060 Ti. Divide each card's extra overhead by its SM count, and every SM's share lands between 20MB and 24MB. In other words, while carrying the compute load, each of these SMs occupies a 20-something MB chunk of VRAM.
My guess is local memory. The on-chip registers and shared memory are small, while in our kernel every thread judges an entire crash state, so the WGSL is full of thread-private arrays (there are 214 declarations like var<private> B: array<u32, TABLE_COUNT>;). Whatever doesn't fit on chip, the driver puts in local memory. It's called "local", but it actually lives in VRAM.
The driver doesn't know ahead of time how many threads will run at once, so it reserves for the fullest case:
size needed per thread × up to 1,536 threads per SM × number of SMs
This number is naturally proportional to the SM count. Divide 20-24MB per SM by 1,536 and you get roughly 13-16KB per thread. For a checker that lays out an entire tree structure inside each thread, that order of magnitude makes sense. We haven't measured this yet, though: on this machine we can't read how much the driver reserved for the kernel, so it remains a guess.
Here's another fun bit: after the 5090 was fixed, one batch's scratch area holds 3,527 states. With 64 threads per workgroup that's only 56 workgroups in total, not enough to fill 170 SMs.
Here's the table:
| Card | SMs | Used (nvidia-smi) | Scratch area (logs) | Checker tables total (approx.) | Unaccounted overhead (approx.) | Per SM | vs. the 2560 MiB reserve |
|---|---|---|---|---|---|---|---|
| 5090 | 170 | 30587 MiB | 27124 MiB | 78 MiB | 3.4 GB | 19.9 MiB | ~0.8 GB over, hence the OOM |
| 5080 | 84 | 15105 MiB | 13018 MiB | 80 MiB | 2.0 GB | 23.9 MiB | only ~0.5 GB of headroom left |
| 5060 Ti | 36 | 14186 / 14190 MiB | 13278 MiB | 80 MiB | 0.8 GB | 23.0 MiB | plenty of headroom |
The Fix
So how did we finally fix it? The fix was fairly simple. Each card now measures for itself when it's opened. After loading the first checker and before opening the scratch area, it reads once via nvidia-smi how much it's actually using on this card. Subtracting what's recorded on the books gives this card's extra overhead. After judging a batch it reads again and keeps the larger value. The scratch area is sized as "what the card can give − measured extra overhead − a 512MB margin".
Looks busy, and really is busy, just like the 5090 today.
Comments
No comments yet. Start the discussion.