"V cache quantization requires flash_attn" - the llama.cpp error that quietly halves your context window

I did not meet this error while debugging a crash. I met it while writing a calculator. llama_context: quantized V cache requires flash_attn to be enabled There is a second wording, thrown as an exception a little later in startup and surfacing as failed to initialize the context : quantized V cache was requested, but this requires Flash Attention and a third, older one - V cache quantization requires flash_attn - which is no longer in the tree but is what most of the search results still show you, because most of the world runs llama.cpp through something that vendors a build from six months ago. All three read like a configuration nag: you asked for one thing, turn on the other thing, move along. That framing is why almost nobody asks the interesting question, which is why those two settings are welded together. The answer is a memory-layout decision several levels below the flag you typed, and it is worth knowing, because it tells you precisely which half of the cache you can still quantize when flash attention isn't available to you. But first the calculator, because that is how I got here and it is the part that cost me real time. The number I actually needed I ship an offline desktop app that runs llama.cpp locally. Users have whatever machine they have. Before the app picks a context window it has to answer one question honestly: window = (RAM - model weights - reserve) / cost_per_token_of_KV Three of those four terms are easy. RAM you ask the OS. Weights you take from the file. The reserve is a policy number you choose - mine is deliberately fat, because on macOS unified memory, overshooting what the GPU can wire does not politely hand you an allocation failure. It panics the kernel. I have the scars and the commit history. The fourth term is where I went wrong. cost_per_token_of_KV looks like something you compute from model metadata: layers, KV heads, head dimension, two tensors, two bytes each. Multiply, done. Every context-size calculator on the internet does exactly this. On the model I care about it was wrong by a factor of four. That model is a Gemma-family 12B, and Gemma interleaves its attention: a minority of layers attend over the full context, the rest run a short sliding window that does not grow with n_ctx at all. Metadata math doesn't know that. It multiplies one per-layer cost by every layer and confidently describes a model that does not exist. On a 24 GB box a 4x overestimate is not a rounding error - it is the difference between offering the user 16k of context and telling them their machine can manage 4k. Metadata describes the model. I needed a number that describes the allocation. Those are different things, and only one of them gets printed at runtime. So I stopped computing and started booting llama.cpp already knows the answer. It says it at startup, in a line most people scroll past on the way to the prompt: llama_kv_cache: size = 160.00 MiB ( 4096 cells, 8 layers, 1 seqs), K (f16): ... Bytes, cells, layers. No metadata, no architecture assumptions - this is the allocator reporting what it actually took. Note the layer count: eight, on a model with far more layers than that. The interleaved model gets more than one of these lines, one per cache, and you want the sum. So the probe is dumb and reliable: boot the engine with a small context, parse its own log, divide, kill it. Two seconds, no inference, nothing downloaded. My app does this once per model on first run and caches the result. One trap, and it's why the probe takes a context argument instead of using the smallest number that loads. Until late 2025, llama.cpp padded the cache size itself, and the multiple depended on flash attention: // the FA kernels require padding to avoid extra runtime boundary checks return cparams.flash_attn ? 256u : 32u; That's gone - PR #16812 removed KV cache size padding in October 2025, and the only rounding left is on the per-graph n_kv view, a flat 256 whether or not flash attention is on. Good news you should not rely on, because the llama.cpp inside your LM Studio or your ollama is quite possibly older than that commit. Probe well above the padding floor regardless. It costs nothing, and it's the difference between measuring a model and measuring a rounding rule with beautiful precision. The probe was lying too First real run, the probe reported: 368,640 bytes per token. Meanwhile the production config, same machine, same model, was demonstrably holding a window that this number says is impossible. So I read the production allocation directly: 182,784 bytes per token. Ratio: 2.02x. My careful runtime measurement was off by more than the metadata error I had built it to fix - same direction, same machine, same model. The reason is embarrassing and took ten minutes to find. The probe booted the engine with default flags: f16 K, f16 V, flash attention off. The app boots it with q8_0 K, q8_0 V, flash attention on. I had measured, very rigorously, a configuration I do not ship. The fix is one line of "pass the same flags." The lesson outlived the fix, because the arithmetic doesn't land where you'd guess. f16 is 2 bytes per value; q8_0 is 34 bytes per 32 values, or 1.0625. That predicts a 1.88x gap. I measured 2.02x. The remainder comes from layout and padding differences that ride along with flash attention, which appear nowhere in the dtype arithmetic and which I would never have thought to include. Which is the argument for measuring, made better by how nearly I missed it: had the gap come out at exactly 1.88x, I'd have hardcoded the ratio and shipped a formula that drifts silently every time llama.cpp changes its padding. The confirmation was that with the honest number, the formula reproduces the 16,384-token ceiling my app had been running for months - a figure originally arrived at by hand, by trial, by someone getting tired of crashes. The measurement agreed with the scar tissue. That's when I believed it. | probe under defaults | probe under production flags | | |---|---|---| | K cache | f16 | q8_0 | | V cache | f16 | q8_0 | | flash attention | off | on | | measured cost | 368,640 B/token | 182,784 B/token | | window from the same ~2.8 GiB KV budget | 8,123 tokens | 16,384 tokens | Same 24 GB, same model, same afternoon. One of those rows is a product decision and the other is a support ticket with a head start. Why quantized V needs flash attention at all Now the part that sent me into the source, and the part I couldn't find written down anywhere. Last time I wrote about llama.cpp, the internet's confident answer to my problem was "it's the quantized KV cache," and it wasn't. So it seems only fair that I now explain what the quantized KV cache is legitimately guilty of. The classic, non-flash attention path ends like this: ggml_tensor * kqv = ggml_mul_mat(ctx0, v, kq); ggml_mul_mat reduces over ne[0] , the first dimension. So V has to arrive with the KV-position axis as its row axis - that is, V transposed. llama.cpp could transpose on the fly, and the code comments explain why it doesn't: that means a ggml_cont(ggml_transpose(...)) over the whole cache every single step. So V is stored pre-transposed instead, behind a flag declared exactly like this: bool v_trans = true; // the value tensor is transposed and set, at every single cache construction site, to literally !cparams.flash_attn . That flag is the entire story. Flash attention on, V is stored naturally, because ggml_flash_attn_ext wants it the other way round. Flash attention off, V is stored transposed. Now consider what transposed storage does to a write. Appending one token in the natural layout means writing one contiguous row of n_embd_v_gqa values. In the transposed layout those same values scatter: one element into each of n_embd_v_gqa different rows, striding by kv_size . llama.cpp expresses that scatter with ggml_set_rows , and the transposed branch does something that looks unhinged until you see why - it reshapes the destination so that every row is exactly one element long: // in this branch the v_idxs are constructed in such a way that each row is a single head element ggml_tensor * v_view = ggml_reshape_2d(ctx, v, 1, ggml_nelements(v)); v_cur = ggml_reshape_2d(ctx, v_cur, 1, ggml_nelements(v_cur)); return ggml_set_rows(ctx, v_view, v_cur, v_idxs); And ggml_set_rows quantizes one whole row at a time - it calls the type's from_float(src, dst, nc) with nc equal to the row length. With nc == 1 and q8_0's 32-element blocks there is simply nothing to quantize: quantize_row_q8_0 asserts that the count is a multiple of 32. ggml_set_rows also hard-asserts that its source is F32 or F16. The operation you'd need instead is read the 32-element block, dequantize it, replace one value, recompute the shared scale, requantize. ggml does not have that operation, and you would not want it in the hot path anyway - every new token would rewrite a block whose scale then shifts underneath values written several tokens ago. So it isn't a policy or an unfinished feature. There is no quantized write path in ggml with sub-block granularity, and the non-flash-attention V layout offers nothing but sub-block writes. K is a different tensor with a different fate. K is never transposed. Its update writes whole rows of n_embd_k_gqa values, one row per token, contiguous and block-aligned by construction - the same code with or without flash attention. And the non-FA path consumes it as ggml_mul_mat(ctx0, k, q) , where a quantized first operand is the ordinary, thoroughly supported case. There is no guard anywhere in llama.cpp rejecting a quantized type_k without flash attention. The only type_k check is gated on flash attention not being disabled, and all it verifies is that the head dimension divides evenly by the block size - a constraint that exists because the FA path views K per-head, while the non-FA path only ever needs whole rows. Which means the workaround people trade in the issue threads - drop -ctv q8_0 , keep -ctk q8_0 - isn't