Linux 7.3 improves performance when running out of vRAM

Earlier this year, I blogged about work I did to improve VRAM management for games. Now, after many months of floating around in mailing lists, the kernel patches are finally merged upstream and queued for Linux 7.3! Hooray! To celebrate, let’s look a bit deeper at one sentence I wrote in my previous post: [Games] should perform much more stable - as long as the game itself doesn’t use more VRAM than you actually have. So, one may ask: What if they do, in fact, use more VRAM than you actually have? Typical expectations for this seem to be that once this happens you’re pretty much screwed. Games will start crashing left and right, performance plummets to unplayable levels, a good gaming experience becomes impossible. But is that really just an unavoidable fact of life? What really makes running out of VRAM suck so hard? And, most importantly: How can we make it suck as little as possible? Setting expectations In theory, running out of VRAM should exclusively be a performance issue, not a stability one. Support for overcommitting VRAM has existed for as long as GPU drivers have: If the driver overcommits VRAM, you are generally allowed to request as much VRAM as you’d like, and you’ll get as much as the kernel driver decides it can fit into the physical memory that exists on GPU. On the performance side, the big-picture reason for bad performance when you run out of VRAM is fairly simple. As soon as the game requests more VRAM than is physically present, some of the game’s memory will have to be moved/evicted to CPU RAM instead. For the GPU, accessing CPU RAM is much slower than VRAM: Not only is CPU RAM slower than a dedicated GPU’s VRAM in general, all memory accesses also have to go over the PCI bus. The PCI bus adds latency and is typically also the limiting factor in bandwidth when fetching from CPU memory. Due to PCI speed limitations, there are some truly unavoidable performance constraints when overcommitting VRAM. Assuming the GPU is hooked up via a PCIe 4.0x16 connection, you get a little less than 32GiB/s of bandwidth. Each millisecond, that PCIe bus can transfer ~32.2MiB of data. For a minimum framerate of 30 frames per second (33.3ms per frame), the absolute maximum amount of data the GPU is able to access is ~1,075.5MiB, a tiny bit over 1GiB of data. In other words, if so much memory gets evicted that the GPU needs to fetch more than 1GiB from evicted memory in one single frame, it is simply impossible to still hit 30 FPS. Not all memory is equal At the same time, just reading a little bit of CPU memory on the GPU is not immediately a death sentence for performance. In fact, GPU drivers sometimes decide to let things like command buffer data and related allocations live in CPU RAM even when there’s plenty of VRAM available! Whenever the GPU executes these commands, it has to access CPU memory, and yet in these cases everything runs completely fine. So what makes these accesses different - why are they fine and yet running out of VRAM seems catastrophic? One thing that influences the calculus significantly is caching. Since the access latency in case of a cache hit is the same regardless of whether the cached memory lives on CPU or GPU, the high initial cost of fetching over the PCI bus can be amortized by cache hits (to some extent). We can estimate latency differences between fetching CPU RAM and VRAM by writing microbenchmarks that measure access latency for different buffer sizes (using an adversarial access pattern to minimize cache hitrates as far as possible). The result you get may look something like this (captured on RDNA3): As expected, if the buffer fits into L2 (or any higher-level cache), access latencies are exactly the same for memory backed by CPU RAM and memory backed by VRAM, because the data gets fetched directly from cache in either case. At a size of 6MB (the L2 cache size on RDNA3), CPU memory latencies go up to about 2400 cycles per access, while device memory latencies stay within the same rough ballpark. Note that VRAM accesses also go through the Infinity Cache, but CPU memory accesses do not (they hit PCIe directly on an L2 miss). I suspect this is because the Infinity Cache sits directly on top of VRAM, so any access that doesn’t hit VRAM also doesn’t reach the Infinity Cache. Obviously, memory doesn’t start off with being cached anywhere, so the first access will still have considerably higher latency. Also, losing the Infinity Cache definitely hurts as well: PCIe fetches seem to have somewhere around 7.3x as much latency than an Infinity Cache hit, and around 4.6x as much latency as a fetch from VRAM. This increased latency needs really high cache hitrates to fully amortize the cost of going over PCIe. That means there is only a small set of use cases where using CPU memory has such minuscule slowdowns that you’d actively decide to use it in favor of VRAM when you have the choice. When you’re evicting memory from VRAM, there will almost unavoidably be at least some degree of slower performance. Still, even though slowdown is unavoidable, there is going to be memory where eviction matters more and memory where eviction has a lesser effect on overall perf. Memory that is accessed in very cache-friendly ways is not affected by the slowdown of CPU RAM as much. If the access patterns aren’t cache-friendly but the memory isn’t accessed very often, things may also still be fine since the GPU only rarely needs to actually fetch data from CPU RAM. There might be many memory allocations where the GPU will only access a small part of the total allocation size, and never even read the rest. If these allocations were to be evicted, you might evict multiple GiBs of data, but still remain well below the 1GiB hard limit of data that is actually accessed per frame. All of these variables make it surprisingly hard to predict how performance actually pans out in practice when memory is being evicted. But in short: Depending on how much the evicted memory gets accessed and how well these accesses cache, you might just be able to run out of VRAM without (completely) ruining performance! Confronting reality We’ve theorycrafted ourselves all the way towards having performant VRAM overcommitment now. Great! Let’s just boot up SteamOS, start some game and crank up the setti- radv/amdgpu: Not enough memory for command submission. oh. As it turns out, running out of VRAM in practice does carry plenty of stability issues with it. This error isn’t quite like a regular “couldn’t allocate, out of memory” error, though. Note that the message specifically complains about command submission: RADV prints this message when the kernel returns -ENOMEM when trying to submit commands, but merely submitting commands does not allocate any new resources! All the command buffers were allocated in advance, and clearly their allocation succeeded. Even though all memory was successfully allocated, using it in a GPU submission suddenly results in “out of memory” errors being thrown. It’s time for another kernel adventure! Surely getting the kernel to accept the submission can’t be that hard - after all, the kernel already accepted all the allocations! The horrors of kernel locking One thing the amdgpu driver has to do on every submission, before it can direct the GPU to start executing commands, is to make sure that all memory that may potentially be referenced by the GPU commands is accessible. With more modern bindless graphics APIs, you have to assume all allocated memory may at some point get referenced. Therefore, amdgpu will try to make sure all allocated memory is also accessible. Each memory allocation carries information about which type of memory (for our purposes here, system RAM or GPU VRAM) it can be properly accessed from. Most allocations can be accessed from either CPU RAM or VRAM, and amdgpu will be happy with the memory allocation being in either of these memory types. Some allocations, however, have to be placed in VRAM and VRAM only. If these memory allocations have been evicted to system RAM because some other application allocated VRAM in the meantime, amdgpu will have to move them back into VRAM. Because there is no free VRAM available at all, moving the allocation back requires evicting something else. For some reason, that failed and the kernel reported an out-of-memory condition. In order to explain why evicting something randomly fails, we’ll have to take a small detour to look at how the kernel handles (CPU-side) locking for GPU allocations. In order to evict a memory allocation, you have to acquire a lock associated with that allocation. However, during a submission, you also have to lock every allocation that’s referenced in a submission, to prevent some other application from moving the allocation somewhere else while you’re busy preparing GPU work. But if another GPU submission is doing the same thing concurrently, you can end up in a situation like this: If one submit wants to evict an allocation that another submit has already locked, but that other submit also needs to lock an allocation from the first one to make progress, we have a textbook ABBA deadlock condition. But fear not, the kernel knows how to detect and resolve deadlocks! The details about how deadlock detection works are explained in this kernel documentation page, but in very broad strokes, the kernel associates locking operations with a “transaction” (which basically just keeps track of which locks were acquired). If two transactions would deadlock, one of the transactions is marked as “wounded”, and the next time it tries to acquire a lock, the -EDEADLCK error is returned. This error requests the transaction to be aborted: All locks acquired during the transaction should be released, and the transaction is restarted from scratch. In the context of command submission, this just means the driver will restart the process of going over all memory allocations and making sure they’re accessible. So where’s the catch? The