DEV Community

llm-d v1.0 on Kubernetes: KV-Cache Routing, Disaggregation, and Production Readiness

Why We Are Evaluating llm-d

We want an inference gateway to answer a more useful question than "Which backend is next?" We want it to answer: "Which backend can finish this request fastest without repeating expensive work?"

That distinction matters when coding agents repeatedly submit repository context, conversations resend their history, and tool loops carry the same tool definitions across successive requests. A healthy Kubernetes Service can distribute connections while still distributing inference work badly.

Our interest in llm-d is therefore specific: preserve useful KV-cache locality across model replicas without turning a cache-warm replica into a queueing hotspot.

Evidence boundary: this review is a source-backed architecture and deployment assessment, not a completed Effloow GPU benchmark. We did not execute Kubernetes installation or reproduce the Google Cloud and Red Hat performance runs. We assess those benchmark configurations and provide a worked validation path; we do not claim to have executed the installation or performance tests.

Under the hood, llm-d adds inference-aware scheduling and coordination around serving engines such as vLLM and SGLang. Its Endpoint Picker participates in the gateway request path, using cache affinity and scheduling signals to choose a backend. In disaggregated serving, the router coordinates separate prefill and decode stages.

We see three distinct buying decisions here:

  • Cache-aware routing: improve placement across existing replicas.
  • Flow control: queue, prioritize, or shed requests before backend queues become overloaded.
  • Prefill/decode disaggregation: separate prompt processing from output generation and transfer KV state between them.

We would not take on the operational complexity of all three at once. Our starting point would be routing in front of an existing replicated vLLM deployment. Disaggregation needs its own justification.

In our llm-d v1.0 architecture review, we identified three important operational mechanisms: Endpoint Picker hot standby, priority-and-fairness flow control, and revision-aware coordination through DisaggregatedSet. We treat these as mechanisms to validate - not as proof that every rollout will preserve every stream.

Deployment Preparation and Validation Steps

Our first installation checkpoint is version provenance. "llm-d 1.0" is not enough information to reproduce a deployment spanning a gateway, extension CRDs, router charts, serving images, transfer connectors, and controllers.

In our review of the prefill/decode installation path, we traced version selection to the release-1.0 repository branch and guides/env.sh. We would source that file to resolve installation variables, including Gateway API Inference Extension and router chart versions. A release branch can move. We would record its commit and archive the manifests before treating a result as reproducible.

The following runnable preparation script requires Git, Bash, and network access. It does not install anything into a cluster:

#!/usr/bin/env bash
set -euo pipefail
WORKDIR="$(mktemp -d)"
git clone --branch release-1.0 --single-branch \
  https://github.com/llm-d/llm-d.git "${WORKDIR}/llm-d"
cd "${WORKDIR}/llm-d"
export REPO_ROOT="$PWD"
export GUIDE_NAME="pd-disaggregation"
export NAMESPACE="llm-d-pd-disaggregation"
export MONITORING="false"
export MONITORING_VALUES=""
export ACCELERATOR_VALUES=""
export ACCELERATOR_TYPE="gpu"
export MODEL_SERVER="vllm"
export INFRA_PROVIDER="base"
export MODEL="openai/gpt-oss-120b"
# Load the versions and URLs used by the official release guide.
source "${REPO_ROOT}/guides/env.sh"
printf 'Repository: %s\n' "$REPO_ROOT"
printf 'Commit: %s\n' "$(git rev-parse HEAD)"
for name in GAIE_VERSION GAIE_URL ROUTER_CHART_VERSION LWS_VERSION; do
  value="${!name:-}"
  if [[ -z "$value" ]]; then
    printf 'ERROR: required variable %s is unset\n' "$name" >& 2
    exit 1
  fi
  printf '%s=%s\n' "$name" "$value"
done
printf '\nRouter chart references:\n'
grep -nEi 'router.*chart|chart.*router' guides/env.sh || true
printf '\nPreparation complete; no cluster resources were changed.\n'

Illustrative stdout shape - not captured execution output:

Repository: /tmp/<temporary-directory>/llm-d
Commit: <resolved-commit-sha>
GAIE_VERSION=<resolved-release-version>
GAIE_URL=<resolved-manifest-path>
ROUTER_CHART_VERSION=<resolved-chart-version>
LWS_VERSION=<resolved-controller-version>
Router chart references: <matching variable definitions from guides/env.sh>
Preparation complete; no cluster resources were changed.

We deliberately leave these versions unresolved here. The available source extracts show the variable names and installation method, but not the actual router and extension pins. Filling them in from memory would defeat the purpose of pinning.

For a new cluster, we would use LWS 1.0.0 as the recommended version from our model-server orchestration review, with this Helm installation:

export LWS_VERSION="1.0.0"
helm install lws oci://registry.k8s.io/lws/charts/lws \
  --version "${LWS_VERSION}" \
  --namespace lws-system \
  --create-namespace \
  --set enableDisaggregatedSet=true

We would first reconcile that recommendation with the version resolved from the release checkout. For the P/D path, we would enforce the minimum of LWS 0.11.1 with DisaggregatedSet enabled; we distinguish that compatibility floor from the recommended version pin.

With the release environment loaded, the official Gateway API Inference Extension CRD step is:

kubectl apply -f \
  "https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml"
kubectl get crd | grep -Ei 'inference|leaderworkerset|disaggregatedset'
kubectl api-resources | grep -Ei 'inference|leaderworkerset|disaggregatedset'
kubectl get gatewayclass

Our acceptance condition would be more than successful CRD creation. We would check served API versions, controller readiness, gateway-provider compatibility, and route status. Installing extension CRDs does not install a working L7 gateway.

For the router, the P/D guide layers base values from guides/recipes/router/base.values.yaml with guide-specific values under guides/pd-disaggregation/router/. We would use the complete Helm command from the pinned checkout. The source extract available for this review stops before that command, so we cannot responsibly present a complete, verified router installation.

Cache-aware routing also needs explicit validation. We would inspect the rendered router configuration for the selected plugins and weights, and confirm that engine-side prefix caching is enabled. A scorer's name is not sufficient evidence that the engine is reusing KV blocks.

The P/D recipe provides a concrete scheduling configuration: Prefill uses prefix-cache-affinity-filter, gated by calibrated peakPrefillThroughput, alongside token-load-scorer. Decode uses active-request-scorer. The recipe selects always-disagg-pd-decider. Both roles belong to one InferencePool.

Prefill computes prompt KV state; decode pulls it through a transfer connector and generates output tokens. The default NVIDIA/vLLM topology is one TP=1 prefill pod plus one TP=4 decode pod: five GPUs total, serving openai/gpt-oss-120b. We cannot present that as a CPU-only kind tutorial or a single-GPU production benchmark.

Likewise, public weights do not automatically make the supplied manifests secret-free. The guide creates llm-d-hf-token; for a no-token deployment, we would explicitly review and adjust the manifest's secret references.

What We Could Not Verify and Deployment Risks

Our review hit several verification limits before runtime testing. We distinguish those from failures actually observed in a cluster.

Local validation cannot establish default P/D performance. A CPU-only cluster may help validate selected Kubernetes objects if supported components and images are available. It cannot establish accelerator KV-transfer performance. A single GPU also cannot reproduce the five-GPU default or the sixteen-GPU routing benchmark. Our workaround is to separate control-plane validation, routing validation, and accelerator performance testing into different gates. We would not let a green manifest check stand in for a working inference pipeline.

Helm upgrades do not update CRDs automatically. For an existing LWS installation, we would apply the new CRDs before upgrading the controller:

helm pull oci://registry.k8s.io/lws/charts/lws \
  --version "${LWS_VERSION}" --untar
kubectl apply --server-side --force-conflicts -f lws/crds
helm upgrade lws oci://registry.k8s.io/lws/charts/lws \
  --version "${LWS_VERSION}" \
  --namespace lws-system \
  --set enableDisaggregatedSet=true

We would review field ownership before using --force-conflicts in a shared cluster. More critically, we would block a direct upgrade from a Helm-installed LWS 0.7.0 or earlier to a later release: the upgrade can delete the LWS CRD and cascade-delete existing LeaderWorkerSet resources. That requires a migration plan, not a routine chart bump.

Cache affinity can become misplaced confidence. In our review of the June scheduling configuration, we distinguished the routing-history-based prefix tracker from the separate precise-cache mode. A router's estimate that a backend holds a prefix is not equivalent to observing the engine's current cache after eviction or restart. We would compare routing decisions against engine cache metrics, then test cache churn and replica replacement. We would also verify the selected v1.0 configuration instead of assuming the June defaults remain unchanged.

Disaggregation makes the network part of serving correctness and performance. For our P/D deployment plan, we would treat NIXL over TCP as supported but strongly prefer RDMA for production. We would verify the actual transport, device visibility, and topology from inside the pods. Ordinary HTTP connectivity does not prove a fast KV-transfer path.

Cancellation is engine-specific. In our v1.0 P/D cancellation review, we identified an SGLang-specific risk: cancellation before decode pulls KV state can strand that state on prefill until restart because prefill-side free notification and reclaim timeout are absent. We have not reproduced this condition, and would not generalize it to vLLM. Our acceptance test would repeatedly cancel requests between phases, watch prefill memory recovery, and confirm a bounded remediation procedure.

Finally, we would validate support scope separately from upstream maturity. In our review of Red Hat's June support scope, we found that its third-party managed-Kubernetes distributed-inference path was Technology Preview and excluded from Red Hat production SLAs. We would confirm the current supported product configuration before buying support.

Scale, Latency & Cost vs. Alternatives

The published results justify a serious evaluation. They do not establish Effloow's p99 latency or cost per token.

Deployment choice What we would evaluate Available performance evidence Primary tradeoff Single vLLM instance Engine-local prefix reuse and simplest operations No equivalent single-instance comparison in the supplied results No cross-replica placement decision Replicated vLLM behind a Kubernetes Service Baseline capacity and queue behavior Baseline for the cited routing comparisons Routing lacks inference-specific cache placement llm-d cache-aware routing Reuse across replicas while balancing load Red Hat's shared-prefix result; Google's routing results Router, gateway, and cache-state integration llm-d P/D disaggregation Independent phase scaling and interference reduction A concrete deployment recipe, but no isolated P/D cost result here KV transport, topology, coordinated upgrades Managed GKE Inference Gateway Managed gateway operation using llm-d routing components Google's production and benchmark figures Cloud dependency and separate pricing analysis

In our reading of the Red Hat scheduling benchmark, the test uses eight vLLM replicas on sixteen H100 GPUs, TP=2, and Qwen/Qwen3-32B. Its synthetic workload contains 150 shared-prefix groups with five prompts per group at 60 requests per second.

In our assessment of that shared-prefix benchmark, we examined TTFT falling from 35 seconds to 120 milliseconds, throughput increasing 151%, and request latency declining 35%. A 151% increase means 2.51 times baseline throughput. The excerpt does not establish the TTFT percentile, so we cannot turn that figure into a p99 result.

In our assessment of the TPU 7x agentic-coding benchmark, we examined prefix-aware routing combined with host-memory offload: more than 85% lower median TTFT and 2.4 times total throughput versus a plain Kubernetes Service. We cannot isolate routing's contribution from that combined result.

Separately, in our assessment of Gemini/Vertex AI production traffic through GKE Inference Gateway, we examined prefix-cache hit rates increasing from 35% to 70%, coding-agent TTFT declining more than 35%, and bursty-chat p95 TTFT declining more than 50%. These are different workload and measurement scopes - not interchangeable entries in one benchmark table.

Our reproduction status for both sets remains not reproduced. For an independent test, we would preserve model revision, engine image, GPU allocation, prompt trace, output lengths, and arrival process. We would compare:

  1. A single vLLM deployment as an operational reference.
  2. Equal-capacity replicated vLLM behind the baseline route.
  3. The same replicas behind llm-d.
  4. P/D with its total accelerator allocation disclosed.

We would measure cold and warm traffic separately, then add unique prompts, bursts, cancellation, and rolling replacement. Client-side TTFT, inter-token latency, end-to-end latency, failures, and SLO-compliant output throughput would all matter.

Our break-even calculation We use: Monthly net saving = avoided GPU cost βˆ’ added infrastructure cost βˆ’ added engineering cost. For a hypothetical fleet spending $24,000 per month on GPUs, assume l

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.