Show HN: DeepSeek-V4 Latent Reasoning - moving "thinking" into latent space

DeepSeek-V4-Flash-0731-Latent-Reasoning. A self-contained model that does thinking in latent space, NVFP4-quantized, with a production vllm form for serving runtime. https://huggingface.co/nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning Published on blog.n.ichol.ai Where the last edition left off Last time I grafted a CoLaR head (Compressed Latent Reasoning) onto DeepSeek Flash v4. The work had all the makings of a demo. A head that lets the model think in latent space. A learned stop head that decides when it has thought enough. A riddle that showed why plain autoregressive generation regurgitates cached answers instead of reasoning. There was one honest problem though: the head was an adapter. Something bolted on the side. To serve it you had to assemble the base model, the head, the stop criterion, and a custom runtime by hand, then hope the pieces fit. The weights lived in one place. The inference machinery in another. "Here's how you run it" was a jump-through-hoops story. This edition closes that gap. The work is now a complete, self-contained model. Every weight needed to serve it ships in one repository. The backbone is quantized down to NVFP4 so it fits on real silicon. And the latent loop is driven by a proper, benchmarked serving runtime. Model: nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning The big change: this is not an adapter anymore The whole point of this edition is packaging. The old release was a head you had to attach. The new one is a model. Every weight needed to serve it now sits in one HuggingFace repo: The DeepSeek-V4-Flash-0731 backbone, quantized to NVFP4 (group size 16) on the routed MoE experts. Attention, shared experts, LM head and draft block stay at higher precision. Roughly 79 GiB of weights per GPU at TP=2 (158-164 GiB total). The DSpark draft block (3 layers), preserved from the source, so speculative decoding ships in the box. The trained latent reasoning head. 35.7M params, loaded from a single latent_reasoning_head.safetensors (~152 MB). Because the weights are complete, the model card can finally report a real benchmark instead of "more benchmarks soon." first evals BBH (BIG-Bench Hard), cot_zeroshot , 27 subtasks: aggregate 0.94 ± 0.008. Measured with lm-evaluation-harness 0.4.12 against an OpenAI-compatible endpoint. Deepseek latent reevaluated the results after giving separate 0.94 vs 0.88 exact match. Thinking enabled. 50 items per subtask, 1350 items total. | Subtask | Score | Subtask | Score | | tracking_shuffled_objects_three_objects | 1.00 | date_understanding | 0.92 | | tracking_shuffled_objects_five_objects | 1.00 | sports_understanding | 0.88 | | tracking_shuffled_objects_seven_objects | 1.00 | logical_deduction_five_objects | 0.88 | | penguins_in_a_table | 1.00 | web_of_lies | 0.86 | | formal_fallacies | 1.00 | snarks | 0.84 | | boolean_expressions | 1.00 | ruin_names | 0.84 | | word_sorting | 0.98 | movie_recommendation | 0.84 | | temporal_sequences | 0.98 | salient_translation_error_detection | 0.76 | | object_counting | 0.98 | geometric_shapes | 0.74 | | navigate | 0.98 | causal_judgement | 0.66 | | logical_deduction_three_objects | 0.98 | disambiguation_qa | 0.58 | | reasoning_about_colored_objects | 0.96 | dyck_languages | 0.26 | | hyperbaton | 0.96 | | | | multistep_arithmetic_two | 0.94 | | | | logical_deduction_seven_objects | 0.94 | | | The pattern is exactly what you would hope for from a latent reasoning model. It is strongest where reasoning means multi-step state tracking. tracking_shuffled_objects , boolean_expressions , formal_fallacies , penguins_in_a_table all hit 1.00. It is weakest on the mechanical, syntax-heavy jobs. dyck_languages (bracket matching) sits at 0.26, the clear outlier. That is a genuine weakness, not a measurement artifact. Two honest notes on reading the table: Read flexible-extract , not strict-match . BBH's strict-match regexes for the literal phrase The answer is X . This model does not emit that phrase, because it reasons in latent space. Its near-zero strict-match score is an answer-formatting artifact, not a reasoning failure. Per-subtask values carry about ±0.05-0.07 at 50 items each. The aggregate of 0.880 is the reliable number. Why the head looks different now The architecture picks up where the original CoLaR idea left off, but it has a proper home in the model now. A small head reads the backbone's layer-35 hidden state, projects it into a 1024-d latent, and decodes it back into the residual stream. One latent step stands in for several reasoning tokens (a recorded compression_factor of 6). A learned stop head self-terminates the loop at a variable, content-dependent depth. layer 35 hidden (4096-d) | v LayerNorm +--------- ReasoningCompressionHead ----------+ | Linear 4096 -> 2048 . SiLU | | Linear 2048 -> 2048 . SiLU | | Linear 2048 -> 2048 -> [mu, log_sigma] | | | | stop_head: | | Linear 4096 -> 1024 . SiLU | | Linear 1024 -> 1 | -> end-of-reasoning +---------------------------------------------+ | mu (1024-d latent) v LayerNorm +-------------- LatentDecoder ----------------+ | Linear 1024 -> 2048 . SiLU | | Linear 2048 -> 2048 . SiLU | | Linear 2048 -> 4096 | +---------------------------------------------+ | v written back into the residual stream DeepSeek-V4-Flash-0731 backbone (frozen, NVFP4) | Config | Value | | hidden_size | 4096 | | latent_dim | 1024 | | mlp_dim | 2048 | | source_layer / target_layer | 35 / 42 | | activation | SiLU | | learned stop head | yes | | head + decoder params | 35.7M (float32 ) | | backbone layers | 43 | One detail worth mentioning: the head is a variational compression. It predicts [mu, log_sigma] and clamps log_sigma . The decoder redistributes the latent back into the 4096-d stream. The whole thing is a single flat tensor dict distinguished by key prefix. The geometry lives in the checkpoint's own metadata, so the serving runtime needs zero configuration; it reads the head's shape straight from the file. target_proj , a frozen projection that defined the regression target during training, is included for completeness but is not used at inference. The serving runtime is now a real thing The old runtime was a set of env vars and headers bolted onto a fork, with the model weights kept separate. This time the runtime is released in its own right, split cleanly into two pieces you install once: | Repository | What it is | nickmitchko/ds4-reasoning-addon | The latent-reasoning addon: the closed-loop driver, serve script, pinned requirements and GPU-sizing guide. Start here. | nickmitchko/vllm-ds4-sm120 | The DS4 vLLM fork it runs on (branch ds4-sm120-preview-dev ). Required. Upstream vLLM cannot serve this model. | Installing the addon does nothing by itself. The plugin registers but stays dormant until VLLM_DS4_REASONING_CKPT is set. That is a deliberate safety choice; it is safe to leave installed. The supported entrypoint is a single script that sets all the measured-good defaults for you: # 1. the engine (a full build takes a while) git clone https://github.com/nickmitchko/vllm-ds4-sm120.git && cd vllm-ds4-sm120 git checkout ds4-sm120-preview-dev export CUDA_HOME=/usr/local/cuda-13.0 PATH=/usr/local/cuda-13.0/bin:$PATH export TORCH_CUDA_ARCH_LIST="12.0" pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu130 pip install -e . --no-build-isolation # 2. the addon + its pinned deps git clone https://github.com/nickmitchko/ds4-reasoning-addon.git && cd ds4-reasoning-addon pip install -e . --no-deps pip install -r release/requirements-serve.txt \ --extra-index-url https://flashinfer.ai/whl/cu130/torch2.11 # 3. serve: no arguments needed, the head is resolved from the model repo release/serve_ds4_reasoning.sh The serve script defaults to the bundled model head (latent_reasoning_head.safetensors , ~152 MiB, cached after the first run), so the recommended configuration is already applied. If you trained your own head, point HEAD_BUNDLE at it and you are done. Why a fork at all? This is not a cargo-cult fork. There is a real technical reason stock vLLM can't serve this model. DeepSeek-V4 routes MoE experts by a hash keyed on input_ids . vLLM's native prompt_embeds path nulls input_ids when you supply embeddings, which would crash the engine at startup with DeepSeek V4 hash MoE routing requires input_ids . So the addon cannot use the standard embeddings path. Instead, injection overwrites the embed_tokens output at the target positions with the decoded latent, while token ids keep flowing normally so hash-MoE routing still works. Latent steps carry a reserved pad token id purely for accounting. A latent step occupies a real KV position, so it must advance num_tokens in lockstep with num_computed_tokens . That id never reaches the client, and its embedding is never read (the row is marked is_token_ids=False , so the injected latent survives). Because this runs at the execute_model seam, it stays on the cudagraph fast path. No enforce_eager , and it batches across concurrent requests. Speculative drafting is suppressed during the latent phase (VLLM_DS4_SUPPRESS_LATENT_DRAFTS , on by default). Leave it on. The rider injects one embedding at the last query row, so a draft slot would steal it and the real position would get the pad token's embedding instead of the latent. The knobs, revised The env-var / header split survives, but it is cleaner now, with a couple of new dials the model-card work made necessary. Server-wide environment variables (defaults = measured-good) | Variable | Default | What it does | MAX_LATENT | 256 | Safety cap on latent steps (bounds a stop-head misfire). Matches the head's K=256 training. | MIN_LATENT | 4 | Floor on latent steps, so answers never no-think. | USE_STOP | 1 | Learned stop; 0 uses a fixed-N cap. | STOP_THRESHOLD | 0.5 | Stop-head threshold. 0.5 is correct. Don't "fix" warmup by lowering it. | MIN_OUTPUT_TOKENS | 4096 | Floor on the answer's token budget. | RIDER | 1 | 0 serves the bare backbone (A/B baseline against latent reas