Reproducing OpenAI’s “persistently beneficial models” - GRPO trait install barely moves. Ideas? [P] [R]

TL;DR: I’m reproducing the trait-persistence result from arXiv:2606.24014 on one RTX 3090. Before I can test persistence I need to install a trait via RL - and my GRPO run moves the trait only +2.4 points (95% CI [+0.2, +4.8]) when I need ~+15. Training is mechanically healthy and I’ve ruled out the obvious culprits. Looking for advice from people who’ve done small-scale RLHF/GRPO trait or persona installation.

What I’m reproducing

The paper trains beneficial traits via RL and shows they persist under adversarial prompting and harmful finetuning. My end goal is the persistence phenomenon; the install is the prerequisite I’m stuck on.

Setup

• Qwen2.5-7B-Instruct + LoRA (r=32), GRPO (unsloth + vLLM colocation), 200 steps, single 3090 (~10⁻⁵ of the paper’s compute).
• Trait: consistent (OCEAN low-Openness / “traditionalism”) - a stylistic trait, chosen because I need measurable headroom in a 7B base. Base scores 57/100 on the trait rubric, wide distribution (not saturated).
• Reward: model-graded (gpt-4.1-mini judge), R = 0.85·quality + 0.15·coherence, hard validity gate for degenerate/looping/refusal output. 25% trait prompts / 75% general (no_robots).

Result: install fails

On the frozen eval set, trait went 57.0 → 59.4 ( +2.4 ). I don’t think this is very appreciable.

What I’ve already ruled out

This is where I’d love a second opinion:

• Not degeneracy / reward hacking: post-train coherence 76, answer length ratio exactly 1.00 vs base, 0% repetition, 0% refusals.
• Not memorization: the 20 training prompts were seen 10× each; the model scores the same on them (58.9) as on held-out (59.4). It didn’t memorize-then-fail-to-generalize - it never learned them.
• Not a dead gradient: the judge separates the 6 sampled answers per prompt by ~18 points on average; only ~25% of GRPO groups have degenerate reward spread.
• Not a question artifact: independent upstream eval questions (+3.4) and my generated ones (+2.8) agree.
• I did find and fix a real confound first - a completion-length cap was truncating ~30-70% of samples → zeroing their reward → ~90% of early “learning” was just the model learning to be shorter. Fixed; trait still flat.

Author feedback

I reached out to one of the authors, who kindly confirmed my leading hypothesis: 20 distinct trait prompts is far too few, per-example prescriptive rubrics (vs my single global rubric) probably matter, and first-order install should work at small scale even if persistence is weaker there.

Where I need help

1. Anyone installed a persona/trait via GRPO at 7B-ish scale - how many distinct prompts did it actually take?
2. Is per-example rubric grading (3-4 specific imperatives per prompt) the real unlock, or is raw prompt count the dominant factor?
3. For a stylistic trait with no single “correct” behavior per situation, does model-graded RL install differently than for task-like traits?
4. Anyone reproduced (or failed to reproduce) this or similar trait-RL work?

Github Code <!-- placeholder - original had link text but no URL; keeping as is -->