💡 The paper addresses the problem of mode collapse in on-policy reinforcement learning, where methods like GRPO concentrate probability mass on a single solution and cease exploring alternative strategies. This is due to the reverse KL minimization method used, which reinforces the first high-reward trajectory found rather than maintaining a distribution over multiple diverse solutions. To solve this problem, the authors propose DMPO, a distribution-matching policy optimization method that uses forward KL minimization to maintain solution diversity and improve performance in combinatorial optimization and reasoning tasks. DMPO constructs a target distribution over sampled trajectories proportional to their rewards and aligns the policy distribution to this target, providing mode-covering behavior without requiring sampling from the intractable global target distribution. The authors validate DMPO on NP-hard combinatorial optimization tasks and achieve significant improvements over GRPO, with a 43.9 percent quality ratio on text-based tasks and 43.1 percent on vision-based tasks. These gains generalize to mathematical reasoning and out-of-domain tasks, demonstrating that diversity-preserving training enhances general reasoning capabilities across modalities. The results show that DMPO achieves consistent quality improvements and sustained exploration across diverse reasoning tasks, establishing distribution matching as a practical approach to preventing mode collapse in on-policy reinforcement learning.