PIRL: From Open-Loop Exploration to Closed-Loop Reinforcement Learning [R]

TL;DR

Most RL post-training algorithms optimize the current batch and move on. But after an update, did the new policy actually become better? We introduce Policy Improvement Reinforcement Learning (PIRL) and its practical implementation, Policy Improvement Policy Optimization (PIPO) - a plug-and-play framework that lets RL training look back, verify the previous update, and reinforce or correct it accordingly.

Why is current RL post-training often “open-loop”?

Methods such as PPO, GRPO, GSPO, DAPO, on-policy distillation, and self-distillation differ in how they construct learning signals. But they largely share the same pattern:

1. Sample a batch from the current policy.
2. Compute rewards, advantages, or distillation targets.
3. Update the policy.
4. Move to the next batch.

The update may improve the local objective, but that does not necessarily mean the resulting policy is actually better. Finite sampling, generation stochasticity, noisy feedback, and imperfect local credit assignment can all push the policy in an unhelpful direction. Without explicitly checking the result of an update, training may drift, become unstable, or even collapse. That is what we call open-loop RL optimization.

PIRL: Make policy improvement itself the objective

PIRL introduces a missing feedback signal: the measured performance gain between successive policies. Instead of asking only:

• Does this step improve the local objective?

PIRL also asks:

• Does this step actually improve the policy?

The cumulative policy-improvement objective remains aligned with final task performance, providing a closed-loop view of RL post-training.

PIPO: A two-phase closed-loop update

PIPO turns this idea into a practical, plug-and-play algorithm.

Phase 1 - Exploration

The base algorithm runs normally. Whether it is PPO, GRPO, DAPO, or a self-distillation objective, it uses its original local attribution signal to update the policy. The model takes an exploratory step in the direction that the base algorithm currently considers useful - but that direction is not guaranteed to be correct.

Phase 2 - Retrospective verification

In the next iteration, PIPO evaluates the updated policy and compares its performance with a sliding-window historical anchor. This produces a policy-improvement feedback signal:

• If performance improved, PIPO reinforces and consolidates the previous update.
• If performance declined, PIPO suppresses or rectifies the corresponding learning direction.

Importantly, PIPO does not replace the base algorithm's local credit assignment. It adds a second layer of feedback that verifies the empirical effect of the previous update. In one sentence: If it helped, reinforce it. If it hurt, correct it.

Results

Across mathematical reasoning, code generation, tool use, and self-distillation, PIPO delivers consistent gains when added to PPO, group-relative optimization methods, and self-distillation objectives. The experiments also show improved training stability across random seeds and better overall wall-clock efficiency, with PIPO reaching higher accuracy under comparable or moderately increased training time.

Final note

PIPO is designed as a general closed-loop layer rather than a replacement for existing RL post-training algorithms. We would love to hear what the RL community thinks - especially about retrospective verification and whether policy improvement should become a first-class training signal.

• 📄 Paper: Policy Improvement Reinforcement Learning
• 💻 Code: github.com/JacckMa/pipo_verl