vLLM: Anatomy of a High-Throughput LLM Inference System

Inside vLLM: Anatomy of a High-Throughput LLM Inference System From paged attention, continuous batching, prefix caching, specdec, etc. to multi-GPU, multi-node dynamic serving at scale August 29, 2025 In this post, I'll gradually introduce all of the core system components and advanced features that make up a modern high-throughput LLM inference system. In particular I'll be doing a breakdown of how vLLM [1] works. This post is the first in a series. It starts broad and then layers in detail (following an inverse-pyramid approach) so you can form an accurate high-level mental model of the complete system without drowning in minutiae. Later posts will dive into specific subsystems. This post is structured into five parts: - LLM engine & engine core: fundamentals of vLLM (scheduling, paged attention, continuous batching, etc.) - Advanced features: chunked prefill, prefix caching, guided & speculative decoding, disaggregated P/D - Scaling up: from single-GPU to multi-GPU execution - Serving layer: distributed / concurrent web scaffolding - Benchmarks and auto-tuning: measuring latency and throughput - Analysis is based on commit 42172ad (August 9th, 2025). - Target audience: anyone curious about how state-of-the-art LLM engines work, as well as those interested in contributing to vLLM, SGLang, etc. - I'll focus on the V1 engine. I also explored V0 (now deprecated), which was valuable for understanding how the project evolved, and many concepts still carry over. - The first section on LLM Engine / Engine Core might be a bit overwhelming/dry - but the rest of the blog has plenty examples and visuals. :) LLM Engine & Engine Core The LLM engine is the fundamental building block of vLLM. On its own, it already enables high-throughput inference - but only in an offline setting. You can't serve it to customers over the web yet. We'll use the following offline inference snippet as our running example (adapted from basic.py). from vllm import LLM, SamplingParams prompts = [ "Hello, my name is", "The president of the United States is", ] sampling_params = SamplingParams(temperature=0.8, top_p=0.95) def main(): llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0") outputs = llm.generate(prompts, sampling_params) if name == "main": main() - VLLM_USE_V1="1" # we're using engine V1 - VLLM_ENABLE_V1_MULTIPROCESSING="0" # we're running in a single process This configuration is: - offline (no web/distributed system scaffolding) - synchronous (all execution happens in a single blocking process) - single-GPU (no data/model/pipeline/expert parallelism; DP/TP/PP/EP = 1) - using standard transformer [2] (supporting hybrid models like Jamba requires a more complex hybrid KV-cache memory allocator) From here, we'll gradually build up to an online, async, multi-GPU, multi-node inference system - but still serving a standard transformer. In this example we do two things, we: - Instantiate an engine - Call generate on it to sample from the given prompts Let's start analyzing the constructor. LLM Engine constructor The main components of the engine are: - vLLM config (contains all of the knobs for configuring model, cache, parallelism, etc.) - processor (turns raw inputs → EngineCoreRequests via validation, tokenization, and processing) - engine core client (in our running example we're using InprocClient which is basically ==EngineCore ; we'll gradually build up toDPLBAsyncMPClient which allows serving at scale) - output processor (converts raw EngineCoreOutputs →RequestOutput that the user sees) Engine core itself is made up of several sub components: - Model Executor (drives forward passes on the model, we're currently dealing with UniProcExecutor which has a singleWorker process on a single GPU). We'll gradually build up toMultiProcExecutor which supports multiple GPUs - Structured Output Manager (used for guided decoding - we'll cover this later) - Scheduler (decides which requests go into the next engine step) - it further contains: - policy setting - it can be either FCFS (first come first served) or priority (higher priority requests are served first) waiting andrunning queues- KV cache manager - the heart of paged attention [3] The KV-cache manager maintains a free_block_queue - a pool of available KV-cache blocks (often on the order of hundreds of thousands, depending on VRAM size and block size). During paged attention, the blocks serve as the indexing structure that map tokens to their computed KV cache blocks. 2 (key/value) * block_size (default=16) * num_kv_heads * head_size * dtype_num_bytes (e.g. 2 for bf16)During model executor construction, a Worker object is created, and three key procedures are executed. (Later, with MultiProcExecutor , these same procedures run independently on each worker process across different GPUs.) - Init device: - Assign a CUDA device (e.g. "cuda:0") to the worker and check that the model dtype is supported (e.g. bf16) - Verify enough VRAM is available, given the requested gpu_memory_utilization (e.g. 0.8 → 80% of total VRAM) - Set up distributed settings (DP / TP / PP / EP, etc.) - Instantiate a model_runner (holds the sampler, KV cache, and forward-pass buffers such asinput_ids ,positions , etc.) - Instantiate an InputBatch object (holds CPU-side forward-pass buffers, block tables for KV-cache indexing, sampling metadata, etc.) - Load model: - Instantiate the model architecture - Load the model weights - Call model.eval() (PyTorch's inference mode) - Optional: call torch.compile() on the model - Initialize KV cache - Get per-layer KV-cache spec. Historically this was always FullAttentionSpec (homogeneous transformer), but with hybrid models (sliding window, Transformer/SSM like Jamba) it became more complex (see Jenga [5]) - Run a dummy/profiling forward pass and take a GPU memory snapshot to compute how many KV cache blocks fit in available VRAM - Allocate, reshape and bind KV cache tensors to attention layers - Prepare attention metadata (e.g. set the backend to FlashAttention) later consumed by kernels during the fwd pass - Unless --enforce-eager is provided, for each of warmup batch sizes do a dummy run and capture CUDA graphs. CUDA graphs record the whole sequence of GPU work into a DAG. Later during fwd pass we launch/replay pre-baked graphs and cut on kernel launch overhead and thus improve latency. - Get per-layer KV-cache spec. Historically this was always I've abstracted away many low-level details here - but these are the core pieces I'll introduce now, since I'll reference them repeatedly in the following sections. Now that we have the engine initialized let's proceed to thegenerate function.Generate function The first step is to validate and feed requests into the engine. For each prompt we: - Create a unique request ID and capture its arrival time - Call an input preprocessor that tokenizes the prompt and returns a dictionary containing prompt ,prompt_token_ids , and atype (text, tokens, embeds, etc.) - Pack this info into an EngineCoreRequest , adding priority, sampling params, and other metadata - Pass the request into the engine core, which wraps it in a Request object and sets its status toWAITING . This request is then added to the scheduler'swaiting queue (append if FCFS, or heap-push if priority) At this point the engine has been fed and execution can begin. In the synchronous engine example, these initial prompts are the only ones we'll process - there's no mechanism to inject new requests mid-run. In contrast, the asynchronous engine supports this (aka continuous batching [6]): after each step, both new and old requests are considered. Next, as long as there are requests to process, the engine repeatedly calls its step() function. Each step has three stages: - Schedule: select which requests to run in this step (decode, and/or (chunked) prefill) - Forward pass: run the model and sample tokens - Postprocess: append sampled token IDs to each Request , detokenize, and check stop conditions. If a request is finished, clean up (e.g. return its KV-cache blocks tofree_block_queue ) and return the output early - The request exceeds its length limit ( max_model_length or its ownmax_tokens ) - The sampled token is the EOS ID (unless ignore_eos is enabled -> useful for benchmarking when we want to force a generation of a certain number of out tokens) - The sampled token matches any of the stop_token_ids specified in the sampling parameters - Stop strings are present in the output - we truncate the output until the first stop string appearance and abort the request in the engine (note that stop_token_ids will be present in the output but stop strings will not). Next, we'll examine scheduling in more detail. Scheduler There are two main types of workloads an inference engine handles: - Prefill requests - a forward pass over all prompt tokens. These are usually compute-bound (threshold depends on hardware and prompt length). At the end, we sample a single token from the probability distribution of the final token's position. - Decode requests - a forward pass over just the most recent token. All earlier KV vectors are already cached. These are memory-bandwidth-bound, since we still need to load all LLM weights (and KV caches) just to compute one token. The V1 scheduler can mix both types of requests in the same step, thanks to smarter design choices. In contrast, the V0 engine could only process either prefill or decode at once. The scheduler prioritizes decode requests - i.e. those already in therunning queue. For each such request it:- Computes the number of new tokens to generate (not always 1, due to speculative decoding and async scheduling - more on that later). - Calls the KV-cache manager's allocate_slots function (details below). - Updates the token budget by subtracting the number of tokens from step 1. waiting queue, it:- Retrieves the number of computed blocks (returns 0 if prefix caching is disabled - we'll cover that later). - Calls the KV-cache manager's alloca