We Rebuilt the Linux MicroVM Stack on Apple Silicon

Encore builds and deploys backend applications, and since mid-2022 every one of those builds has run inside a Firecracker microVM. Firecracker strips the emulated hardware down to what a Linux kernel needs, giving each build the isolation of a virtual machine with startup close to the cost of a container. Firecracker drives KVM, so it needs a Linux host with /dev/kvm , which no Mac has, and most engineers at Encore develop on a Mac. The maintainers have no plans to close that, given they turned down a working proof of concept built on Apple's Virtualization.framework and said they do not plan to support macOS any time soon. So for four years, working on the build system meant working on it somewhere else. We wanted to run the same build system on our laptops while keeping Firecracker in production, so we built crackling, a single microVM API that drives Firecracker on Linux and Apple's hypervisor on macOS; booting the same images on both required rebuilding much of the Linux image toolchain to run on macOS. We onboarded each engineer with a script you ran once. It SSHed into the shared build machine as root, pulled your public key from https://github.com/ .keys and created you a user, then added you to the kvm and docker groups so you could reach the hypervisor and run containers. It copied the VM images into your ~/images and hard-linked the firecracker binary into your ~/binaries , since every user needed it under their own tree on the one box. You ended up with a personal environment in a datacentre, reachable over Tailscale, sitting next to everybody else's. Getting a change onto that environment took a second script, which read your username and your port out of the CUE config, from a gitignored per-engineer file, because we all shared that host and had to agree not to collide. Binaries were the easy half: we cross-compiled with GOOS=linux GOARCH=amd64 , rsynced the results across, and counted the transferred files to work out whether anything needed restarting. Images were the hard half, because Firecracker boots a block device and Docker produces layers. We could not find an existing tool that converted Docker layers into a block device Firecracker could boot, so we built the conversion ourselves, half on your laptop and half over SSH: # tools/dev-builder/deploy-dev-builder.sh docker save -o "$imagesdir/$name.tar" "$docker_image" tar -C "$layersdir" -xf "$imagesdir/$name.tar" # explode the layers tar -C "$dst/" -xf "$imagesdir/$name.tar" manifest.json rsync -azP $layersdir ${username}@builder:~/images/ rsync -azP "$dst" ${username}@builder:~/images/ ssh ${username}@builder -- \ "bash -l -s squash_layers "images/${outputdir}" "images/${name}"" { // SAFETY: we pass the reactor's own serial queue; the VM is // stored and only ever used from this queue henceforth. let vm = unsafe { VZVirtualMachine::initWithConfiguration_queue( VZVirtualMachine::alloc(), &vm_cfg, &reactor().queue) }; let id = shared.id; if let Ok(mut g) = reactor().state.vms.lock() { g.insert(id, ReactorVm { vm, shared }); } let _ = reply.send(Ok(())); } Err(e) => { /* mark Failed, then: */ let _ = reply.send(Err(e)); } } }); The dispatch API requires Send + 'static closures, so the compiler prevents a !Send VM object from being captured. The registry and the reactor that holds it still need hand-written Send and Sync impls; the queue invariant depends on those two implementations. VZVirtualMachineConfiguration is !Send too, so we split lowering into two phases: a MachineSpec becomes a structure containing only Send data on tokio, then that structure becomes a VZVirtualMachineConfiguration on the queue. A completion handler receives a raw NSError pointer that is valid only for the duration of the block, so we convert it into an owned error on the queue before replying. Dropping the last handle to a machine dispatches a dispose, because the framework requires that release on its own queue too. The VZ backend could now create and control a VM, but booting one still required replacing the Linux-only image pipeline. Turning an OCI image into a bootable root filesystem normally requires root and a loop mount, neither of which exists on macOS, while building an initramfs usually calls the cpio binary. The kernel tree's own extract-vmlinux is written for x86 bzImage and cannot unpack an arm64 kernel. Neither hypervisor boots anything until it has an uncompressed kernel image, and the vmlinuz an arm64 distribution ships is usually an EFI zboot file, which is a small EFI executable wrapping a compressed payload that the firmware would normally decompress at boot. There is no firmware here, so we unwrap it ourselves. The MZ and zimg signatures identify the format, and the header gives us the payload's offset, size, and compression: // crates/crackling-image/src/kernel.rs // EFI zboot: "MZ" at offset 0 and the "zimg" signature at offset 4. if bytes.len() > 64 && &bytes[0..2] == b"MZ" && &bytes[4..8] == b"zimg" { let payload_offset = u32::from_le_bytes(bytes[8..12].try_into().unwrap()) as usize; let payload_size = u32::from_le_bytes(bytes[12..16].try_into().unwrap()) as usize; let comp_end = bytes[24..32].iter().position(|&b| b == 0).unwrap_or(8); let compression = std::str::from_utf8(&bytes[24..24 + comp_end]).unwrap_or(""); let end = payload_offset .checked_add(payload_size) .filter(|&e| e gunzip(&bytes[payload_offset..end])?, other => return Err(ImageError::Kernel( format!("unsupported zboot compression: {other:?}"))), }; } Hand Virtualization.framework a compressed kernel and it fails at start with a generic internal error and no detail attached. The virtualization entitlement was our first suspect, and we lost an afternoon re-signing binaries before looking at the kernel. We now check for the ARMd signature at offset 0x38, which identifies a raw arm64 Image , and report a compressed kernel before trying to boot it. The kernel needs a root filesystem to boot into, so we apply OCI layers entirely in userspace and honor .wh. whiteout entries as squash_layers did with find , but in-process and without a cleanup pass afterwards. Pulling the image also required a custom platform resolver because the default keys off the host OS and never matches a linux/arm64 image for a request from a Mac. Booting the rootfs from RAM requires an initramfs: a newc-format cpio archive inside a gzip stream. We generate both layers in pure Rust, and by default the rootfs remains in memory until the VM stops. We unpack and normalize an image once, write a .built sentinel, and publish the completed output by atomic rename so a crash leaves the existing cache untouched. Each VM clones the cached rootfs using clonefile on APFS, a per-file FICLONE reflink on Linux filesystems that support it, or a plain copy elsewhere, and the RAM path repacks that clone into the VM's own initramfs. Once the kernel and rootfs booted, crackling needed a way to run commands and move data inside the guest. Both platforms provide vsock, and Alpine's virt kernel ships AF_VSOCK as loadable modules, so the guest /init loads vsock , vmw_vsock_virtio_transport_common and vmw_vsock_virtio_transport , among others, before anything can listen. Those modules carry a vermagic string that must match the running kernel exactly, and a mismatch fails at load with nothing useful downstream: the VM boots, the agent never comes up, and the host waits for a connection that never arrives. We fetch the kernel and its modules from one linux-virt package so they stay in step. Mounting ext4 pulls in a crc32c hash even with checksums disabled, so crc32c_generic and libcrc32c have to be loaded, and an interactive shell needs /dev/pts mounted before openpty will work. Every VM runs the same agent, a static musl binary built for aarch64-unknown-linux-musl on the Mac and x86_64-unknown-linux-musl for amd64 hosts. It listens on AF_VSOCK and uses a small framed protocol: an 8-byte header followed by either an encoded control frame or raw bytes for bulk data, with one connection per operation. The protocol supports exec with streamed stdout and stderr, an interactive shell on a PTY, cp in both directions, and forward , which turns a connection into a tunnel to a port inside the guest. The agent uses vsock for control, leaving the guest without manual network configuration or an SSH daemon installed by crackling. Outbound networking is a separate opt-in, while inbound access is available only through a control-plane forward authenticated with a per-VM token generated at boot. On macOS the host end of that transport is a VZVirtioSocketDevice connection whose file descriptor has to be dup(2) ed immediately, because the framework closes the original when its Objective-C object deallocates. On Linux it is a Unix socket with a text handshake, and the reply has to be read one byte at a time: // crates/crackling-firecracker/src/machine.rs // Read the reply one byte at a time so we never swallow payload // bytes past the newline (a real hazard with buffered reads). stream.write_all(format!("CONNECT {port}\n").as_bytes()).await?; let mut line = Vec::with_capacity(16); loop { let b = stream.read_u8().await.map_err(Error::Io)?; if b == b'\n' { break; } line.push(b); } The macOS and Linux implementations differ, but both return a byte stream connected to the agent for crackling shell , exec and cp . Firecracker captures memory and device state natively through PUT /snapshot/create , and a spec carrying restore_from spawns a fresh VMM, checks the snapshot's host fingerprint, loads the snapshot paused and resumes it without booting. Upstream only restores onto a matching architecture and Firecracker version, so the fingerprint check protects the operation when we suspend an idle sandbox and resume it on another host. Apple's framework appears to offer the same thing, since VZVirtualMachine exposes saveMachineStateToURL , and validateSaveRestoreSupport on the configuration asks whether it is eligible. We wrote the implementation and the va