The error pointed at the GPU. The culprit was the HBA. - a Proxmox passthrough recovery

The plan looked textbook. PCIe passthrough an NVIDIA RTX A4000 from a Proxmox 8 host into an Ubuntu 24.04 VM, giving a Filecoin sealing worker direct GPU access for the compute phases that need it. VT-d on the Xeon. IOMMU enabled at the kernel line. OVMF BIOS on the VM. vfio-pci bound to the device IDs. Machine type set to q35 for PCIe support. Standard sequence, followed by every guide, ordered exactly as those guides ordered it. Then qm start returned, and the VM did not come up. Not a kernel panic. Not a host lockup. Not a "device not found" error surfaced to the operator. QEMU exited from the host's perspective and left nothing behind - no running guest, no console output, no obvious pointer at what had failed. The Proxmox web UI showed the VM as stopped, exactly as it had before I hit start. The errors were there. They were just in a different terminal. dmesg -w on the host, running in a second window, produced a burst of IRQ allocation failures and vfio interrupt remapping errors scrolling past at VM startup, then silence. The host itself was completely fine throughout. Nothing was wrong with the box. Only the VM layer was broken, and it was broken in a way that read more like a misconfigured VM than a hardware conflict - which is exactly why the real cause took three sessions to identify. The segmentation piece covered the network work that came before this. This piece is one of the workload-layer stories that followed - the GPU passthrough that took two evenings longer than it should have, because the error pointed at the GPU and the culprit was somewhere else entirely. Two things this piece is not. It is not a step-by-step passthrough guide - there are plenty of those, and they are all fine until they aren't. It is the story of what happens when the checklist runs clean and the passthrough still refuses to work. // what the GPU was actually there to do Filecoin sealing - the process of turning raw storage into provable, verifiable data on the network - is not one workload. It's two compute phases with fundamentally different characteristics, and only one of them benefits from a GPU. The first phase is SDR: Sequential Data Replication. It's designed to be resistant to parallelisation - memory-latency bound, CPU-heavy, and immune to GPU acceleration by design. On this hardware, SDR takes three to four hours per sector regardless of what else is in the box. No GPU speeds it up. That is intentional at the protocol level, not a limitation of the implementation. The second phase is where the GPU matters. Tree building - specifically the TreeRC computations that produce the Merkle tree structure used in Proof-of-Replication - is GPU-eligible. Without acceleration, TreeRC runs on CPU and adds another two to three hours per sector. With the A4000 handling it, the same work completes in roughly fifteen to twenty minutes. The framing that gets used in Filecoin marketing - "GPU acceleration makes sealing faster" - is technically true and operationally misleading. The GPU doesn't make sealing fast. It makes the second half of sealing fast. A storage provider without GPU acceleration is bottlenecked on tree computation as badly as on SDR, and the observable sealing throughput is roughly the SDR time plus the CPU tree time, sector by sector. Where the GPU actually earns its place is in pipeline parallelism at scale. With the CPU pipeline running SDR on one sector while the GPU pipeline runs TreeRC on the previous one, sealing throughput becomes gated by SDR alone. That's the operator-relevant payoff - not making any single sector faster, but decoupling the two phases so they can run against different sectors concurrently. For a small storage provider, that is the difference between sealing two sectors a day and sealing five or six. That was what the A4000 was there to do. Which is why the VM not coming up was a problem. // the diagnostic that didn't converge The investigation stretched across three sessions over about a week, mostly limited by the time I had rather than by the problem itself. Each session ran ninety minutes or so, and each one eliminated one correct-looking explanation without producing the actual cause. Session one focused on the obvious: was the GPU actually bound to vfio-pci , and was the VM config correct? lspci -nnk -d 10de:24b0 That output showed Kernel driver in use: vfio-pci - the host had surrendered the card, exactly as intended. No nouveau , no nvidia . The VM's PCI passthrough config referenced the correct BDF for the A4000, machine type was q35, OVMF was enabled, memory ballooning was off. Nothing in the config was wrong. That was session one. Session two went down the driver blacklist and IOMMU group mapping path. /etc/modprobe.d/vfio.conf had the right device IDs listed. /etc/modprobe.d/blacklist.conf blocked nouveau , nvidia , nvidiafb , and snd_hda_intel . update-initramfs -u -k all had been run, and the host had been rebooted since. All correct. Then the IOMMU group listing: find /sys/kernel/iommu_groups/ -type l | sort -V The A4000 at 03:00.0 appeared in one group with its HDMI audio companion at 03:00.1 - expected. But the same group also contained an IBM SAS2008 HBA. The HBA was managing the host's scratch storage, which meant it couldn't be passed through to the VM. And in an IOMMU group, all devices are the atomic unit of passthrough. You pass through the entire group or none of it. That was where the pointer to the SAS2008 first appeared. But at that moment, I did not yet understand what shared-group membership specifically causes to fail. Group sharing is a widely documented passthrough concern, and the standard advice ("try the ACS override kernel patch") felt like a hack for a build that would need to run reliably long-term. I closed session two aware the SAS2008 was somehow involved but not yet certain how. Session three was the crystallisation. Not a eureka moment - a slow narrowing. Once every other explanation was eliminated, the IOMMU group problem was the only candidate left. But the specific mechanism - why exactly a non-passthrough device in the same group breaks the passthrough of the target device - needed one more step of understanding before the physical fix made sense. // FLR: what makes an IOMMU group problem actually a passthrough blocker IOMMU group membership alone doesn't break passthrough. The Linux community's shorthand - "everything in the group has to be passed through together" - is correct but incomplete. It describes the safety constraint, not the failure mode. Understanding what actually goes wrong requires knowing what QEMU does at VM startup. When the VM starts and takes ownership of a passed-through PCI device, QEMU has to reset it - bring it to a known state before the guest OS initialises. The reset mechanism it relies on is Function Level Reset (FLR), a PCIe capability that allows a device to be reset in isolation without affecting anything else on the bus. FLR is the clean, safe reset. It's what QEMU wants. The A4000 supports FLR. The SAS2008 does not. And here is the mechanism: when QEMU tried to perform FLR on the A4000, it could not do so cleanly because the SAS2008 was in the same IOMMU group and could not itself be reset. The interrupt remapping infrastructure that isolates devices during passthrough could not correctly separate the GPU's IRQ handling from the HBA's, because from the IOMMU's perspective the two devices were treated as a single reset domain. The IRQ allocation failed. The VM didn't start. The error pointed at the GPU because the GPU was the device QEMU was trying to bring up. The culprit was the HBA because the HBA was the device that couldn't cleanly get out of the way. FLR support is directly checkable from the host. For any PCI device, the reset_method sysfs file lists the reset mechanisms the kernel considers available: cat /sys/bus/pci/devices/0000:XX:XX.X/reset_method For the SAS2008, that file did not list flr among its supported methods. That was the confirmation the mechanism I was reasoning about actually applied to this hardware. Once confirmed, the fix stopped being a question of ACS overrides or kernel patches. It became a question of physical topology. This is the class of understanding most passthrough guides skip. They tell you what to configure. They don't tell you what to check when the configuration is right and the passthrough still fails. FLR-vs-non-FLR devices sharing an IOMMU group is the silent blocker at the heart of this failure mode. // moving the HBA to a slot that mattered The fix was physical. The SAS2008 had to move to a different PCIe slot - one that was not just physically distant from the A4000 but sat behind a genuinely separate PCIe root port. That distinction is critical and easy to miss. Not every PCIe slot on a server board sits behind its own root port. Slot layout on the motherboard reflects physical trace routing, not IOMMU topology. Two adjacent slots may share a root port; two slots on opposite ends of the board may or may not. Moving the HBA one slot over on the same root complex would have done nothing - the IOMMU group membership would have been identical after reboot, and the failure would have repeated. The target slot had to be chosen carefully. Two sources helped. The board's manual documents the PCIe topology - which slots connect to which CPU root complex, and how the lanes are allocated across the PCH. The existing IOMMU group listing also carries information: any two devices that had shown up in different groups before must, by definition, be behind different root ports. A slot known to be safe for the HBA was a slot whose currently-installed device (or its adjacent devices) had shown up in a group separate from the A4000's. The physical work itself was straightforward. Full shutdown of the host - no live PCIe hotswap for this class of card. Chassis open. HBA out of its original slot. SAS cables temporarily disconnected (they had to be re-plugged after the move - a