Robust reinforcement learning research for small-scale language model agents: stable alignment of 70M–500M parameter models

Haque et al. systematically studied training instability in PPO alignment of small language models with 70M–500M parameters, identified three failure mechanisms: silent freezing of LoRA parameters, bfloat16 numerical overflow, and reward model errors, and proposed merging reinitialized adapters, using float32 precision for PPO updates, and a three-layer safety mechanism of reward whitening, importance ratio clipping, and weight rollback.