💡 The paper discusses the issue of instability in reinforcement learning for large language models. Current methods like PPO and GRPO use a ratio-clipping mechanism to control the trust region, but this can be ineffective in certain cases, particularly with long-tailed vocabularies. Recent work like DPPO has addressed this by using a divergence-based mask, but it still has a limitation where it discards the gradient of a token once it crosses the trust region boundary.

To address this, the authors propose a new method called Divergence Regularized Policy Optimization, or DRPO. This method replaces the hard mask with a smooth advantage-weighted quadratic regularizer on policy shift. This allows DRPO to preserve the same trust region geometry as DPPO while providing bounded and continuous gradient weights that correct diverging updates and provide corrective signals beyond the boundary.

The experiments show that DRPO improves the stability and efficiency of large language model reinforcement learning training across different model scales, architectures, and precision settings. This is a significant contribution because it allows for more stable and efficient training of large language models, which is an important area of research in natural language processing. Overall, the paper presents a new method that addresses the limitations of current methods and provides a more effective way to train large language models using reinforcement learning.