Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch
Google Developers Blog

Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train-Inference Mismatch

How we eliminated floating-point train-inference logprob drift for on-policy LLM reinforcement learning across 64 Cloud TPU v5p chips using a generalizable four-part JAX and Pallas recipe in Tunix.

Reinforcement learning (RL) for large language models is notoriously sensitive to numerical instability: even tiny discrepancies between rollout generation and gradient updates can destabilize training or stall convergence.

In LLM RL, the model serves as the policy, π(action | previous tokens), and true on-policy learning requires every action in an update to be generated and evaluated by that exact same policy. While a synchronous loop ensures each rollout batch is scored using the same weights that generated it, a critical numerical requirement remains: the sampler and trainer must evaluate that policy identically.

In practice, production RL stacks run sampling and training through separate, specialized execution paths. The sampler chooses each action token during incremental decoding and records its behavior logprob. The trainer evaluates the same token in batched forward-backward passes to compute its current logprob and gradients. Under the same weights, these differing execution paths can still assign different logprobs to the same token and context. The training schedule is on-policy, but the numerical execution has introduced a hidden off-policy gap that compounds over multi-turn trajectories.

Our zero train-inference mismatch (Zero-TIM) implementation, developed in Tunix and verified at multi-host scale across 64 TPU v5p chips (DP8×TP8), closes this numerical gap. At the same weights, the sampler and trainer assign bitwise-identical logprobs to the same action and token history - bringing state-of-the-art, true on-policy stability to TPU RL training.

Floating-Point Reductions and Drift

A reduction is a tree of floating-point additions. Floating-point addition is not associative, so changing the tree changes where rounding occurs. The effect is easier to expose in bfloat16, which retains less precision than float32.

The following bfloat16 example isolates the effect of reduction order:

The same issue appears in model reductions: matrix products accumulate partial sums, normalization computes statistics, and tensor-parallel shards combine partial outputs.

In our original stack, sampler decode and trainer forward used different token-array shapes and forward implementations. JAX specializes computations to their input shapes, and the training forward also runs inside a differentiated program. Across these paths, reduction order and intermediate precision were not consistently aligned. Floating-point rounding turned those arithmetic differences into different logprob bits. The weights and model equations were identical; the arithmetic was not.

The Three-Path Verification Contract

The contract is simple: across sampler decode, serving prefill, and trainer forward, all three paths must produce bitwise-identical logprobs for every sampled action - comparing raw floating-point bytes under the same rollout weights, token history, and sampling settings, with zero rounding or tolerance.

Why three paths instead of two? In a decoupled RL stack, a direct sampler-trainer disagreement shows that a numerical gap exists, but not where it enters. Introducing serving prefill as a middle checkpoint turns alignment into a systematic two-step diagnostic that any RL stack can adopt:

  • Sampler decode vs. prefill verifies that the serving engine agrees with itself between incremental token generation and full-sequence re-scoring, while prefill vs. trainer verifies that the serving and training engines agree across compilation and differentiation boundaries.

Achieving Zero-TIM does not require merging the sampler and trainer into a single monolithic engine. Instead, we use a generalizable four-part recipe that bridges separate serving and training pipelines across multi-host TPU meshes: locking within-shard kernel accumulation, ordering cross-shard reductions, standardizing the logprob scoring tail, and wrapping serving kernels with custom differentiation rules.

Locking Within-Shard Kernel Accumulation

Inside each tensor-parallel rank, batch-invariant Pallas kernels control accumulation order and intermediate precision. Projection kernels traverse fixed contraction blocks in order, keeping the reduction blocking independent of token-row count. Partial products accumulate in float32, with explicit output casts.

The production kernel initializes its scratch accumulator before the tile traversal, then applies this core to each tile:

input_tile = x_ref[...]
weight_tile = y_ref[...]
partial = jnp.dot(
  input_tile, weight_tile,
  preferred_element_type=jnp.float32,
)
acc_ref[...] = acc_ref[...] + partial
out_ref[...] = acc_ref[...].astype(out_ref.dtype)

The next tile continues from the float32 accumulator, not the cast output. This construction covers the attention and MLP projections and the LM head. Dedicated kernels fix RMSNorm reductions and SwiGLU's intermediate casts. The existing Pallas paged-attention forward is retained.

Ordering Cross-Shard Reductions

For row-parallel projections, each tensor-parallel rank produces a partial output. We separate communication from addition: ppermute transports the partials, then every rank adds them in the same source-rank order. The reduction order is explicit in the program rather than selected by a collective implementation.

Standardizing the Logprob Scoring Tail

Alignment must extend beyond model logits to the probability of the sampled action. Decode, prefill, and trainer apply matching sampling transforms, use the same fixed Pallas log-softmax construction, and select the recorded action token. Prefill and trainer score the action generated during rollout rather than choosing another action. We also preserve sampled token IDs across environment turns so that every path evaluates the same token history.

Wrapping Serving Kernels With Custom Differentiation

The trainer reuses the serving prefill computation. Custom vector-Jacobian products retain the serving kernels for forward evaluation and provide their backward rules. The matmul example shows this separation:

@jax.custom_vjp
def serving_matmul(inputs, weights):
  return pallas_matmul(inputs, weights)

def matmul_forward(inputs, weights):
  output = pallas_matmul(inputs, weights)
  return output, (inputs, weights)

def matmul_backward(saved, output_gradient):
  inputs, weights = saved
  _, pullback = jax.vjp(
    differentiable_matmul, inputs, weights,
  )
  return pullback(output_gradient)

serving_matmul.defvjp(matmul_forward, matmul_backward)

The differentiable replica mirrors the kernel's accumulation and casts, but is used only to construct the pullback. Forward values still come from the serving Pallas kernel. Training places that forward inside a differentiated JAX program. We pass --xla_allow_excess_precision=false through XLA_FLAGS to the compiler process, constraining excess intermediate precision in the operations surrounding the Pallas kernels.

Long trajectories are evaluated in fixed-size chunks, carrying the paged KV cache forward. All action logprobs feed the complete train-batch RL loss. The reverse pass propagates gradients through those cache connections and combines parameter-gradient contributions before the optimizer update.

State of the Art and Open-Source Context

Across the AI systems community, eliminating train-inference mismatch has rapidly become a state-of-the-art requirement for stabilizing large-scale RL convergence. Recent open-source efforts - including vLLM with TorchTitan and RadixArk/LMSys Miles - have demonstrated how batch-invariant kernels and bitwise numerical alignment prevent off-policy drift during policy optimization.

Our work delivers an open-source, modular JAX/XLA and Pallas implementation of this contract for TPU in Tunix. Verified across a multi-host 64 TPU v5p (DP8×TP8) mesh, these kernel and collective patterns provide a reusable blueprint that teams can adopt to achieve bit-exact on-policy RL across distributed JAX/TPU stacks.

Benchmark Verification

To benchmark our TPU kernels, we run a 200-step numerical verification test on the synthetic FrozenLake environment using a public Qwen3-8B checkpoint as a baseline test case using 64 TPU v5p chips with a DP8×TP8 mesh. The recipe uses a GSPO-token loss with leave-one-out advantages.

We compare Standard training with trainer-old logprobs and no additional sampler correction, token-level truncated importance sampling (TIS), and Zero-TIM. The Zero-TIM configuration also uses token-in/token-out (TiTO) handling to preserve sampled token IDs across environment turns. The comparison runs retain their existing tokenization paths.

Zero-TIM records zero mean absolute sampler-trainer logprob difference at every logged step in this window, while its training solve rate increases. Standard and TIS retain nonzero differences: TIS adjusts their contribution to the update without removing the underlying mismatch. Zero-TIM reaches higher observed solve rates in these runs.

We developed Zero-TIM prototype as part of Tunix, Google's open-source, JAX-native library for LLM RL and post-training on Cloud TPU. The kernel patterns and three-way verification contract are modular by design - making them straightforward to adopt across existing multi-host JAX/TPU serving and training pipelines - and we are actively extending support to linear-attention and Mixture-of-Experts (MoE) architectures.

Read on Google Developers Blog ↗ ← Back to News

Comments

No comments yet. Start the discussion.