Running Qwen Flash-Next NVFP4 in vLLM: PLE Loading, B12x Fixes, and Stable Inference

Companion code: PLE and B12x compatibility patches, tests, and deployment notes. The repository contains extracted fixes and validation notes, not a complete runtime image. The model loaded. The health endpoint returned a successful response. A few requests completed. Then concurrent traffic arrived, and the inference process ran out of GPU memory. That was the most instructive part of bringing Qwen Flash-Next NVFP4 online. There was no single switch that made the deployment work. We had to get a large checkpoint onto disk, adapt a loader to its tensor layout, and find a memory configuration that survived actual generation. Each milestone removed one obstacle while exposing the next. The eventual result was a server configured for 131,072 tokens of context, with a limit of 16 sequences, that completed three rounds of 16 concurrent requests without a recorded container restart. Getting there required giving up some of our initial context and concurrency targets-and being precise about what the successful test actually demonstrated.

Before getting into the details, a big shout-out to windowsxp811203. The quality of the work and the detailed reports deserve real credit. It is great to see someone put that much care into both the result and its documentation.

This account follows the initial bfloat16 KV-cache baseline and the additional B12x integration work needed by the developed serving stack. The measured 48-request test belongs to the initial baseline. The later B12x work is described separately, without transferring that test result to a different runtime. The loader and backend patches were required independently of KV-cache quantization.

Checkpoint Download and Storage

The first practical problem was simply getting the checkpoint onto the machine. The recorded download comprised 31 files, approximately 186.4 GB in total. A single PLE shard accounted for roughly 102.4 GB. With artifacts of that size, storage preparation and download recovery were part of the deployment work rather than incidental setup. We provisioned a dedicated volume with an ext4 filesystem and used the Hugging Face CLI with eight download workers. The transfer ran through a local proxy. At one point, progress stalled at around 14 GB. After restarting, the reported progress fell to approximately 4.7 GB before the transfer resumed at about 93 MiB/s. The practical consequence was extra transfer time and uncertainty about completion. A progress counter was useful for watching the download, but it was not sufficient evidence that the checkpoint was ready. We waited for the final shard to finish before moving on to loading.

It is also important to keep disk size separate from GPU residency. The total checkpoint size was not a prediction of the amount of VRAM the process would consume. This deployment used PLE CPU offload, so some of the storage and loading work belonged on the host side. Later failures still depended on the memory available on the GPU during generation.

Initial Runtime and Incompatibilities

Before replacing the existing model server, we saved its container configuration and retained a rollback target. That gave the experiments a defined starting point and preserved a route back if the new model failed. It also prevented the previous deployment from becoming another variable while we were changing the new one.

The initial runtime was a model-specific vLLM container image. It provided a starting point, but it did not accept this checkpoint without local changes. Two incompatibilities surfaced before inference tuning could begin.

The first was in the attention configuration. Twelve layer entries used qwen_sparse_attention, while the runtime path we were using expected full_attention. We supplied a local configuration override that changed those entries. That was an operational compatibility change, and it has an important limit: getting past configuration parsing does not establish that the two settings are computationally equivalent. This work did not include a controlled comparison of output quality or attention behavior before and after the override. The evidence supports the narrower claim that the override allowed this runtime to proceed with loading.

The second incompatibility involved the layout of the PLE embedding table. The loader expected checkpoint shards with numeric suffixes.