Proximal Policy Optimization: Why Clipping the Update Beats Trusting the Gradient
The policy gradient starting point
Consider a policy πθ(a|s) parameterised by θ, which gives the probability of taking action a in state s. The goal is to maximise expected return J(θ) = Eτ∼πθ [∑t=0∞ γt rt] where γ is the discount factor and rt is the reward at step t.
The policy gradient theorem says the gradient of this objective can be written as an expectation over states and actions visited by the current policy:
∇θ J(θ) = E s,a∼πθ [∇θ log πθ(a|s) A(s,a)]
Here A(s,a) is the advantage: how much better action a is than the average action in state s. In practice the advantage is estimated from returns, often with a baseline or with generalised advantage estimation, which trades bias against variance.
The key point for PPO is what happens when you try to reuse data. If you collect data with an old policy πθold and then update to a new policy πθ, you can correct for the mismatch with importance sampling. Define the probability ratio
rt(θ) = πθ(at|st) / πθold(at|st)
Then a surrogate objective that approximates the true objective near the old policy is
LCPI(θ) = Et[rt(θ) At]
CPI stands for conservative policy iteration. Maximising this surrogate is equivalent to the policy gradient step when θ = θold, because then rt(θ) = 1.
The trouble starts when you optimise it for several steps on the same batch of data. The ratio can drift far from 1, the surrogate stops approximating the true objective, and the update can destroy the policy. A single large ratio on a high-advantage action can dominate the gradient, pushing the policy to take that action almost always, even if the advantage estimate was noisy.
Trust regions, without the second-order machinery
The principled fix is a trust region: only allow the new policy to move a limited distance from the old policy. Trust Region Policy Optimization, or TRPO, enforces this with a hard constraint on the KL divergence between the old and new policies, and solves the constrained problem with second-order methods involving conjugate gradients and line search.
TRPO works well, but it is complicated to implement, sensitive to implementation details, and awkward to combine with architectures that use dropout, parameter sharing, or recurrent layers.
PPO keeps the trust region idea and drops the machinery. Instead of a hard KL constraint, it uses a clipped surrogate objective that penalises moving the ratio too far, but only when that movement would improve the surrogate. The clipped objective is
LCLIP(θ) = Et[min(rt(θ) At, clip(rt(θ), 1−ϵ, 1+ϵ) At)]
where ϵ is a small constant, typically 0.2, and clip forces the ratio into the interval [1−ϵ, 1+ϵ].
The minimum is doing the real work, and it behaves differently depending on the sign of the advantage:
- If At > 0, the action was good, so increasing its probability is desirable. But once rt(θ) exceeds 1+ϵ, the clipped term stops growing, and the minimum picks the clipped term. The gradient with respect to that sample becomes zero. The policy is not pushed further in that direction.
- If At < 0, the action was bad, so decreasing its probability is desirable. But once rt(θ) falls below 1−ϵ, the clipped term again caps the gain, and the gradient becomes zero.
- If the ratio moves in the wrong direction, for example the ratio increases for a bad action, the unclipped term is smaller than the clipped term, so the minimum picks the unclipped term and the gradient still pushes the policy back.
Clipping never prevents correction, only excessive improvement. In short, clipping creates a flat region beyond the trust region boundary for moves that would otherwise be rewarded, while leaving the gradient intact for moves that undo a mistake. That asymmetry is the whole trick.
A useful diagnostic is the fraction of samples that are clipped in a given epoch. If almost nothing is clipped, the updates are small and you could afford a larger step or more epochs. If a large fraction is clipped, the policy is trying to move far, and the clip is doing heavy work. Monitoring that fraction, together with the KL divergence between the old and new policies, tells you whether the trust region is binding in practice.
Why multiple epochs on the same data become safe
Vanilla policy gradients are on-policy: each gradient step needs fresh data, because the gradient formula assumes data came from the current policy. That is sample-inefficient. PPO reuses each batch for several epochs of minibatch updates, which would be dangerous with the unclipped surrogate, because repeated steps on the same batch can compound into a large move.
With clipping, the first epoch may move the ratio to the boundary, but later epochs on the same samples contribute no gradient once they are clipped, so the total move stays bounded.
This is what makes PPO practical. You collect a batch of trajectories, compute advantages once, and then run several epochs of stochastic gradient ascent on the clipped objective, plus a value function loss and optionally an entropy bonus. The standard combined loss, written as a loss to minimise, is
L(θ) = Et[−LCLIP(θ) + c1 LVF(θ) − c2 Sπθ]
where LVF is the squared-error loss for the value function, S is the entropy of the policy, and c1 and c2 are small coefficients, commonly 0.5 and 0.01.
The entropy term encourages exploration by penalising a policy that collapses to a single action too early. The value loss is usually clipped as well in modern implementations, so that the value function does not move too far in one update either.
A minimal sketch of the PPO loop looks like this:
for iteration in range(num_iterations):
trajectories = collect(policy_old, num_steps)
advantages = compute_gae(trajectories) # generalised advantage estimation
returns = advantages + values
for epoch in range(num_epochs):
for minibatch in minibatches(trajectories):
ratio = policy.prob(minibatch) / policy_old.prob(minibatch)
clipped = clip(ratio, 1 - eps, 1 + eps)
policy_loss = - mean(min(ratio * advantages, clipped * advantages))
value_loss = mean((value(minibatch) - returns) ** 2)
entropy = policy.entropy(minibatch)
loss = policy_loss + 0.5 * value_loss - 0.01 * entropy
optimiser.step(loss)
policy_old = policy
The code is deliberately simple. There is no conjugate gradient solver, no line search, and no Fisher matrix. That simplicity is a feature: it means PPO can be implemented in a few dozen lines on top of a standard deep learning library, and it composes cleanly with shared actor-critic networks, recurrent policies, and distributed data collection.
Advantage estimation matters as much as clipping
Clipping bounds the policy move, but it cannot fix a bad advantage estimate. If advantages are noisy, PPO will confidently move in the wrong direction, just not very far in one step. Generalised advantage estimation, or GAE, is therefore part of the standard PPO recipe, not an optional extra.
GAE builds the advantage from a weighted sum of temporal difference errors. Let δt = rt + γV(st+1) − V(st) be the one-step temporal difference error, where V is the value function. Then the GAE advantage with parameter λ is
AtGAE = ∑l=0∞ (γλ)l δt+l
Setting λ = 0 gives a low-variance but biased one-step estimate. Setting λ = 1 gives an unbiased but high-variance Monte Carlo estimate. A value such as λ = 0.95 is a common compromise.
In practice, advantage normalisation within each minibatch, subtracting the mean and dividing by the standard deviation, also stabilises training, although it changes the objective slightly and should be applied consistently.
A related practical point is reward scaling and normalisation. In environments where rewards have very different scales, advantage normalisation helps, but reward clipping or normalisation must be done carefully, because it changes what the policy is optimising. For language model fine-tuning, a KL penalty against a reference model is often added to the reward, which serves a similar purpose to the trust region: it stops the policy from drifting too far from the supervised model while chasing the reward model.
When PPO helps, and when it does not
PPO helps most when the policy is a large neural network, data collection is expensive, and stability matters more than squeezing out the last bit of sample efficiency. That is why it is the default for reinforcement learning from human feedback: the reward model is imperfect, the policy is a large language model, and a destructive update would be costly. The clip, together with a KL penalty, keeps the fine-tuned model close to a model that still produces fluent text.
PPO is less attractive when sample efficiency is the dominant constraint and you can afford a more complex algorithm. Off-policy methods that reuse data in a replay buffer, such as soft actor-critic for continuous control, often learn faster per environment step, because they are not limited to data from recent policies. PPO also struggles in sparse-reward settings where most advantages are zero: clipping cannot create signal where there is none, and exploration bonuses or reward shaping are needed first.
There are also failure modes that clipping does not prevent. If the value function is poorly trained, advantages are systematically wrong, and PPO will move steadily in the wrong direction within the trust region at each step. If the entropy coefficient is too low, the policy can collapse prematurely, and the clip will then freeze it in place, because large moves are blocked. If the reward model is hacked, as in language model fine-tuning where the model learns to produce text that scores highly without being genuinely better, PPO will optimise that hacked reward efficiently. The trust region bounds how far the policy moves, but it does not judge whether the direction is correct.
Limitations and common pitfalls
- Clipping is not a KL constraint. The clip bounds the probability ratio per state-action pair, not the average KL divergence. The policy can still drift in aggregate if many small ratio changes accumulate in the same direction. Monitor the approximate KL divergence, and stop an update early if it exceeds a threshold.
- Too many epochs undo the protection. With many epochs, clipped samples contribute no gradient, but unclipped samples keep moving, and the value function keeps updating. Four epochs is a common starting point; ten epochs on the same batch often degrades performance.
- The clip range is a hyperparameter, not a law. A value of 0.2 works widely, but tasks with very noisy advantages may need a smaller value, and tasks with reliable advantages may benefit from a larger one. Tune it on a validation task, not on the test environment.
- Shared networks couple the losses. When the policy and value function share early layers, the value loss gradient can distort the policy representation. Separate networks, or a smaller value loss coefficient, reduce this interference at the cost of more parameters.
- Implementation details dominate. Advantage normalisation, value clipping, gradient clipping, learning rate annealing, and minibatch shuffling all affect results as much as the clipped objective itself. Two implementations that both claim to be PPO can perform very differently, which is why comparing against a known reference implementation matters.
The takeaway
PPO is best understood as a compromise made explicit. Trust region methods are principled but hard to implement. Vanilla policy gradients are simple but unstable. PPO takes the surrogate objective that policy gradients already use, clips it so that no single batch can push the policy too far, and then reuses data for a few epochs inside that guardrail. The result is not optimal in any formal sense, but it is stable, simple, and widely applicable.
When you use PPO, watch three numbers: the fraction of clipped samples, the KL divergence between successive policies, and the explained variance of the value function. If clipping is rare, KL is small, and the value function explains returns well, the algorithm is in its comfortable regime. If any of those signals deteriorates, fix the advantage estimation or the value function before blaming the clip. The clip stops a bad step from becoming a catastrophe. It does not turn a bad signal into a good policy.
Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial
Comments
No comments yet. Start the discussion.