✨SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models

📝 Summary:
SRPO is a VLA-RL framework that eliminates the need for expert demonstrations. It assigns progress-wise rewards to failed trajectories using latent world representations and the models own successes. This achieved 99.2% success on LIBERO, a significant improvement.

🔹 Publication Date: Published on Nov 19

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2511.15605
• PDF: https://arxiv.org/pdf/2511.15605

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#ReinforcementLearning #VLAModels #PolicyOptimization #AIResearch #MachineLearning

✨Soft Adaptive Policy Optimization

📝 Summary:
SAPO improves RL training stability for LLMs. It uses a smooth adaptive gate to attenuate off-policy updates, unlike hard clipping. This selectively down-weights problematic tokens, leading to improved training stability and higher performance.

🔹 Publication Date: Published on Nov 25

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2511.20347
• PDF: https://arxiv.org/pdf/2511.20347

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#ReinforcementLearning #LLMs #PolicyOptimization #DeepLearning #AI

✨Agentic Policy Optimization via Instruction-Policy Co-Evolution

📝 Summary:
INSPO introduces a novel framework dynamically optimizing instructions within the reinforcement learning loop for autonomous agents. It substantially outperforms static instruction methods in multi-turn reasoning by discovering innovative, strategic reasoning paths.

🔹 Publication Date: Published on Dec 1

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.01945
• PDF: https://arxiv.org/pdf/2512.01945
• Github: https://github.com/cambridgeltl/inspo

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#ReinforcementLearning #AIAgents #PolicyOptimization #MachineLearning #AI

✨Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies

📝 Summary:
This paper decomposes LLM policies into internal layer and modular policies, revealing distinct reasoning patterns across layers. It finds early layers explore and top layers refine. Motivated by this, Bottom-up Policy Optimization BuPO is proposed to optimize internal layer policies for superior...

🔹 Publication Date: Published on Dec 22

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.19673
• PDF: https://arxiv.org/pdf/2512.19673

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LLM #PolicyOptimization #DeepLearning #AIResearch #NLP

✨GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization

📝 Summary:
GRPO in multi-reward RL suffers from reward normalization collapse, hindering training. GDPO resolves this by decoupling individual reward normalization, improving stability and accuracy. GDPO consistently outperforms GRPO across various reasoning tasks.

🔹 Publication Date: Published on Jan 8

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2601.05242
• PDF: https://arxiv.org/pdf/2601.05242
• Project Page: https://nvlabs.github.io/GDPO/
• Github: https://github.com/NVlabs/GDPO

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#ReinforcementLearning #MultiRewardRL #PolicyOptimization #MachineLearning #AI

✨AT^2PO: Agentic Turn-based Policy Optimization via Tree Search

📝 Summary:
AT^2PO is a framework for multi-turn agentic reinforcement learning. It uses a turn-level tree search with entropy-guided expansion and turn-wise credit assignment. This improves exploration, reward propagation, and policy optimization, achieving state-of-the-art results.

🔹 Publication Date: Published on Jan 8

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2601.04767
• PDF: https://arxiv.org/pdf/2601.04767
• Github: https://github.com/zzfoutofspace/ATPO

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#ReinforcementLearning #AgenticAI #TreeSearch #PolicyOptimization #ArtificialIntelligence

✨Reinforcement Learning via Self-Distillation

📝 Summary:
Self-Distillation Policy Optimization SDPO leverages rich textual feedback to address the credit-assignment bottleneck in reinforcement learning. SDPO treats the model as a self-teacher, distilling feedback-informed predictions to improve sample efficiency and accuracy. It significantly enhances ...

🔹 Publication Date: Published on Jan 28

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2601.20802
• PDF: https://arxiv.org/pdf/2601.20802

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#ReinforcementLearning #SelfDistillation #MachineLearning #AI #PolicyOptimization

✨A Unified Framework for Rethinking Policy Divergence Measures in GRPO

📝 Summary:
This paper presents a unified framework for policy divergence measures in reinforcement learning. It introduces the KL3 estimator as a key constraint, which improves GRPO training stability and performance by promoting stronger exploration.

🔹 Publication Date: Published on Feb 5

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.05494
• PDF: https://arxiv.org/pdf/2602.05494

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#ReinforcementLearning #MachineLearning #AI #GRPO #PolicyOptimization

✨Soft Adaptive Policy Optimization

📝 Summary:
Soft Adaptive Policy Optimization SAPO enhances reinforcement learning for LLMs. It uses a smooth, temperature-controlled gate to adaptively attenuate off-policy updates, improving training stability and performance on reasoning tasks.

🔹 Publication Date: Published on Nov 25, 2025

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2511.20347
• PDF: https://arxiv.org/pdf/2511.20347
• Project Page: https://swift.readthedocs.io/en/latest/Instruction/GRPO/AdvancedResearch/SAPO.html
• Github: https://github.com/NovaSky-AI/SkyRL/pull/762

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#ReinforcementLearning #LLMs #PolicyOptimization #AI #MachineLearning

✨Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing

📝 Summary:
SRPO unifies GRPO and SDPO in reinforcement learning by routing correct samples to GRPO's reward-aligned reinforcement and failed samples to SDPO's targeted logit-level correction. This novel approach achieves superior stability, rapid improvement, and better performance than either baseline.

🔹 Publication Date: Published on Apr 2

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2604.02288
• PDF: https://arxiv.org/pdf/2604.02288

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#ReinforcementLearning #PolicyOptimization #SampleRouting #MachineLearning #AIResearch

🔥 EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments

💡 The paper introduces EvoPolicyGym, a benchmark for evaluating autonomous policy evolution in interactive environments. The problem addressed is that existing evaluations of autonomous agents often oversimplify the process of policy improvement through feedback, either by reducing it to a final score or confounding it with software engineering progress. To address this, the authors propose a controlled evaluation setting called Autonomous Policy Evolution, where an agent repeatedly edits an executable policy system under a fixed interaction budget.

The method involves instantiating this setting in EvoPolicyGym, a suite of compact interactive reinforcement learning environments that assess how agents iteratively improve explored policies. The authors evaluate the performance of various agents, including GPT-5.5, on the EvoPolicyGym suite.

The results show that GPT-5.5 achieves the strongest aggregate rank score and top-two performance on all 16 environments, demonstrating its effectiveness in autonomous policy evolution. However, the paper goes beyond leaderboard results and provides trajectory-level diagnostics that reveal how agents allocate budget and convert feedback into parametric tuning. These analyses indicate that strong autonomous policy evolution depends on discovering task-appropriate mechanisms and refining policies under bounded feedback, rather than just achieving isolated task wins.

The paper's contributions include the introduction of EvoPolicyGym as a benchmark for evaluating autonomous policy evolution, and the demonstration that successful policy evolution requires both task-specific mechanisms and feedback-constrained refinement. The results provide insights into the importance of iterative improvement and bounded feedback in autonomous policy evolution, and highlight the need for more nuanced evaluations of autonomous agents beyond simple final scores.

📅 Published on Jul 2

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.02440
• PDF: https://arxiv.org/pdf/2607.02440
• Project Page: https://linzwcs.github.io/EvoPolicyGym

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#AutonomousPolicyEvolution #InteractiveReinforcementLearning #PolicyOptimization #EvoPolicyGym #AutonomousAgents

🔥 Soft Adaptive Policy Optimization

💡 The paper proposes Soft Adaptive Policy Optimization, a method to improve the stability and performance of reinforcement learning in large language models. The problem addressed is that token-level importance ratios in reinforcement learning can have high variance, leading to unstable updates, especially in Mixture-of-Experts models. Existing methods, such as GSPO and GRPO, use hard clipping to alleviate this issue, but this can make it difficult to maintain both stability and effective learning.

The proposed method, SAPO, replaces hard clipping with a smooth, temperature-controlled gate that adaptively attenuates off-policy updates while preserving useful learning signals. This approach allows SAPO to be both sequence-coherent and token-adaptive, maintaining sequence-level coherence while selectively down-weighting only the offending tokens and preserving the learning signal from the near-on-policy ones.

The results show that SAPO exhibits improved training stability and higher performance compared to GSPO and GRPO. Empirical results on mathematical reasoning benchmarks indicate that SAPO achieves higher Pass@1 performance under comparable training budgets. Additionally, SAPO is used to train the Qwen3-VL model series, demonstrating consistent performance gains across diverse tasks and different model sizes.

Overall, the paper contributes a more reliable, scalable, and effective optimization strategy for reinforcement learning training of large language models, addressing the challenges of stable and performant policy optimization in these models.

📅 Published on Nov 25, 2025

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2511.20347
• PDF: https://arxiv.org/pdf/2511.20347
• Project Page: https://swift.readthedocs.io/en/latest/Instruction/GRPO/AdvancedResearch/SAPO.html

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#ReinforcementLearning #LargeLanguageModels #MixtureOfExperts #PolicyOptimization #AdaptiveLearningTechniques