Triton: DirectX 11 Driver for QEMU

In the prequel, we introduced Neptune, a Direct3D protocol forwarding layer for VirtIO. Neptune allowed us to serialize Direct3D API calls across the hypervisor boundary and this allowed us to run Wine games on a Linux guest with a Linux host faster than with DXVK directly in the guest. Admittedly, the payoff there was not that exciting but it laid the groundwork for our real goal: modern graphics acceleration for Windows guests. We have now achieved this by building a brand new Windows driver called Triton which along with Neptune brings full DirectX 11 support to QEMU virtual machines.

What is Triton?

You might be wondering: if Neptune can serialize Direct3D API calls and Windows uses Direct3D, then aren’t we already done? If Direct3D works in Wine, it should also work in Windows, right? After all, what is Wine, if not a Windows emulator?

The short answer is: you sort of can. The Neptune Mesa drivers build a d3d11.dll and dxgi.dll which fully implements the Direct3D API set and so if you just put those files next to the game’s executable, it should load them instead of Windows’ own drivers and you can get some games to run that way. This is also the approach done by previous attempts which used DXVK→Vulkan→Venus to run Direct3D locally inside an application.

There are a few disadvantages to this approach.

• First and most importantly, you cannot get good performance because the window compositor (DWM) “sees” your frame as an image so it needs to use CPU blitting to copy the GPU image buffer to the correct window location. You might be able to do some tricks for full-screen applications to scanout natively but you will never get a smooth desktop experience.
• Second, because d3d11.dll and dxgi.dll are core components of Windows, you cannot replace the system files themselves and expect Windows to still work. Even if you do manage to get it to work, you will not be able to play many games with anti-cheats which specifically detect this kind of modification. That is why it is only possible to get the DLL to load on a per-application basis (and compatibility varies).
• Which brings us to the last point: needing to copy files to every application you want working graphics acceleration is not a user friendly experience.

The correct approach is not to implement the DirectX APIs but to implement the DirectX DDIs (Device Driver Interface).

DDI

In Windows, the application communicates with the system Direct3D and DXGI libraries. The d3d11.dll (as well as older versions) do the complicated work of state tracking and send a more sanitized stream of commands to the user-mode driver (UMD) which implements the DDI. The application also talks to dxgi.dll to initialize the graphics adapters, set up the swapchain, etc. The UMD also goes through the DXGI to talk with the kernel-mode driver (KMD). The KMD is implemented by the graphics vendor (us) to drive the actual hardware (or in our case the virtual hardware).

For Wine, we implemented a custom d3d11.dll and dxgi.dll to intercept the API calls and now for Windows, we need to instead implement the UMD and KMD. So that’s the challenge: implement the UMD with the DirectX DDI interface and also set up a private interface with the KMD which communicates with the VirtIO device.

Lucky for us, the second part has already been solved. Both anonymix007 and arehnman had independently been working on a KMD for Venus (Vulkan). Since Vulkan is a completely independent graphics API, it does not need to implement the DirectX DDI and its UMD is similar to the “replace d3d11.dll” approach in that it talks directly with the KMD to drive commands to QEMU. Since Neptune is modelled after Venus, the high level kernel interfaces (for DMA, command buffers, etc) is very similar and the interface between UMD and KMD is exactly the same. Ultimately, we chose to use anonymix007’s branch as the base because their implementation had more features working on the KMD side.

That leaves us with the hard part. We have to implement the DDI for DirectX 11.

When you are designing a new system, it is always wise to understand how others who have come before have solved similar problems. Unfortunately, there are not many open source DDI implementations to draw inspiration from. Windows graphics drivers is a very niche subject and most of the experts work in one of the handful of graphics hardware vendors. This is one of the reasons that QEMU has never got far with Windows GPU acceleration.

Fortunately for us, there are two working open source implementation that we can learn from.

Mesa

Mesa has a DirectX 10 UMD. If you did not read the last article, the short version is that Mesa implements OpenGL for Linux. Mesa performs state tracking for OpenGL and emits Gallium API calls. The Mesa DirectX 10 UMD is an alternative to OpenGL that emits the same Gallium API calls. Then the Gallium backend driver (AMD, Intel, VirGL, etc) converts them into native graphics driver APIs.

The upstream Mesa only supports the software rasterization backend for DirectX 10 but there was some recent work to get it working with VirGL. Unfortunately, the macOS virglrenderer lacks support for many of the features that this UMD requires so it is not a viable way to get graphics acceleration for macOS hosts. However, the integration with the Mesa codebase provides us with a clean example for integrating Triton.

VirtualBox

VirtualBox has the only working open source DirectX 11 UMD. However, this driver cannot really be “adopted” for our use. The way their driver works is that they translate the DDI calls into an intermediate bytecode and then on the host side, they interpret the bytecode into DirectX API calls.

While it would be easy (with AI help) to just port this bytecode emitter and interpreter into QEMU, we decided against it for a couple of reasons.

• First, we think this conversion of DDI into a bytecode and then lifting that bytecode back to DirectX API can result in bugs that limit compatibility with games. Indeed, reading forum threads online, it seems like many games do not run in VirtualBox for this reason. If there is a missing feature or bug in the translation engine, it would require a lot of active maintenance effort to fix and we do not want to depend on Oracle for that.
• Second, there is a licence incompatibility between VirtualBox’s GPLv3 and virglrenderer’s MIT License or QEMU’s LGPLv2.

VirtualBox’s code can’t be integrated but we did learn some valuable insight from it. Their list of which DDI prototypes were implemented and which ones returned error is used as the minimal requirements for a working implementation. This information isn’t readily available from MSDN documentation and trying to implement every single prototype would be massive in scope. Their DXBC signature algorithm is also helpful to understand because Microsoft does not publish it anywhere.

Since we didn’t want to take the VirtualBox approach of using an intermediate transport format for DDI calls, we can do something better. If you imagine d3d11.dll as a component that roughly transforms DirectX API calls into UMD DDI calls, then what we want our UMD to do is to transform the DDI call back to DirectX API calls.

Why is this useful? Because then we can use our tested and working Neptune protocol without having to invent a new transport for serializing DDI calls. On the host side, we do not have to do any extra work to execute those calls. VirtualBox needs an emitter and transport layer on the guest as well as an interpreter and dispatcher on the host. Each step adds latency and the chance to introduce errors and incompatibility.

We still need an emitter and transport on the guest but on the host side we do not need an interpreter because the deserialized Neptune commands ARE DirectX 11 API calls and can be dispatched without any additional parsing. One less transform step means less opportunity for mistakes.

Another advantage of a DDI→API transform is that most DDI calls in D3D11 have an API equivalent meaning that the transform is as simple as mapping some API handles to device handles and sometimes doing a lookup for API→DDI enum differences. The biggest win however, also turns out to be the most complicated part of the story, which is the DXBC shader code.

DXBC

DXBC (DirectX Byte Code) is the IR code that Microsoft’s shader compiler (FXC) emits. Specifically, it is the older (pre-DirectX 12) format and is compiled from HLSL, the shader language that DirectX uses.

Since Triton acts as a reverse transform from DDI to API, it does not need to disassemble and convert this shader bytecode. This is a huge win for us in terms of complexity and compatibility.

Unfortunately, it is not as simple as passing the bytecode unmodified to the host. The compiler (FXC) emits the DXBC bytecode along with other metadata. d3d11.dll expects to see this metadata and consumes it. When the DDI is called, only the bytecode is passed. That means for us to make a valid “inverse transform” back to the API call, we need to re-construct all that metadata by interpreting the bytecode.

In the end we still pass through the bytecode unmodified but since we don’t see the original DXContainer file, we have to synthesize fields that the host DirectX renderer expects. This was a lot of trial and error that the AI assistant handled but it is the weakest and most error prone part of our implementation.

Host Renderer

Here’s what we have so far:

• Application makes DirectX and DXGI API calls to the system libraries.
• System libraries invoke Triton through DDI calls.
• Triton DDI converts raw DXBC bytecode back into DXContainer and makes DirectX and DXGI API calls to Neptune.
• Neptune UMD serializes the API calls and passes them through a ring buffer managed by the KMD.
• KMD uses the VirtIO interface to send commands to the host.
• QEMU host handles the command and passes Neptune calls to virglrenderer.
• Neptune host module in virglrenderer deserializes the API calls and forwards them to the host side DirectX implementation.
• Host DirectX implementation renders the frame.

Let’s zoom in on the last point. Once the DirectX API calls make it to the host, we still need to render it.

When we brought up Neptune for Wine on Linux, we forked DXVK to support exporting the swapchain images as DMAbuf resources. At the time, we decided to implement the swapchain on the host side in order to sidestep the issue of shared textures.

Internally, swapchain images are represented as textures but these textures are special in that the host needs to be able to find them and use them to show the final frame on screen. Our Wine DXGI library forwards all the swapchain API calls directly to the host and therefore the host “knows” which textures will be used as a backbuffer. Then separate VirGL commands can be used to scan out the texture blob to the VM window. This trades simplicity in the guest driver and minimal changes in DXVK for the complexity of swapchain logic in the host virglrenderer process.

When bringing up Triton, we realized that host side swapchain handling was a mistake. On Windows, DXGI is a system component which talks to the UMD. DXGI handles the backbuffer creation, frame pacing, mode switching, etc. The UMD (mostly) doesn’t give special treatment to DXGI and so our method of doing an “inverse transform” of DDI calls back to API calls does not really work with DXGI. That means all the swapchain logic we added to the host side is largely bypassed.

The desktop compositor (DWM) operates on shared textures. One process’s DXGI renders content to its own backbuffer and that backbuffer is shared with the DWM process which draws the desktop, window chrome, etc. The final frame that DWM constructs is set for scan out. This means that in addition to DMAbuf exports, we also need to implement DMAbuf imports in DXVK as well (separate guest contexts map to separate host contexts).

Once we have both import and export implemented, there is no need for host side swapchain logic anymore and so to make the Wine driver more unified, we moved all the swapchain logic into the guest Neptune driver. An added benefit of this move is that it more closely tracks with how Venus is designed and so virglrenderer is kept clean.

macOS

There’s some challenge in getting virglrenderer working on macOS but since Venus now runs in macOS, most of the backend challenges have been fleshed out. The remaining task is to connect Neptune to a host side DirectX renderer.

There are three major projects that can handle the goal of DirectX on macOS. However, they all been designed with running Wine as the main target. They lack the shared textures and shared fences features that Neptune and Triton requires.

DXVK + MoltenVK

DXVK is the project we used on Linux hosts. It translates D3D11 API to Vulkan API and then uses the host Vulkan driver to do the rendering. On Linux, this works great because Vulkan is a first class citizen and all modern graphics hardware have a good Vulkan driver at this point. On macOS though, Vulkan is handled by another translation layer, MoltenVK, which translates Vulkan API to Metal API. In the last post, we talked about the unique challenge of getting DXVK + MoltenVK working and the short version is: it is unstable and requires a lot more work for compatibility.

DXMT

DXMT sidesteps the Vulkan issue by translating D3D11 directly to Metal (with D3D12 coming soon). Just like DXVK, the project is designed primarily with Wine in mind so the first step was to implement a native variant of the library. With our fork, DXMT can be built as a macOS shared library and exposes some additional exports for import/export of textures and fences.

Shared Textures

One major hurdle in the design of dxmt-native is in the implementation of shared textures that can cross the process boundary. We need to cross the process boundary because virglrenderer spawns helper processes for each renderer context so roughly every guest D3D