🔥 Self-Distilled Agentic Reinforcement Learning

💡 The paper introduces Self Distilled Agentic Reinforcement Learning, a method that improves reinforcement learning for multi turn agent training. The problem with traditional reinforcement learning is that it provides only coarse supervision for long horizon interaction, which can lead to instability in multi turn agents. On Policy Self Distillation is a technique that complements reinforcement learning by providing dense token level guidance from a teacher branch, but it has limitations when applied to multi turn agents, such as compounding instability and negative teacher rejections.

The proposed method, Self Distilled Agentic Reinforcement Learning, addresses these limitations by treating On Policy Self Distillation as a gated auxiliary objective, while keeping reinforcement learning as the primary optimization backbone. It uses a sigmoid gate to selectively strengthen positive token level guidance and mitigate negative teacher rejections. This allows the method to stabilize supervision and improve the performance of multi turn agents.

The results show that Self Distilled Agentic Reinforcement Learning substantially improves over existing methods, such as GRPO, and avoids the instability of naive combinations of GRPO and On Policy Self Distillation. The method consistently outperforms hybrid reinforcement learning and On Policy Self Distillation baselines across different model scales and datasets, including Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA. Overall, the paper contributes a new method that improves the performance and stability of multi turn agents in reinforcement learning.

📅 Published on May 14

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.15155
• PDF: https://arxiv.org/pdf/2605.15155

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#AgenticReinforcementLearning #MultiTurnAgentTraining #OnPolicySelfDistillation #ReinforcementLearningMethods #SelfDistilledLearning

🔥 EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL

💡 The paper introduces EnvFactory, a framework that automates the creation of executable tool environments and natural multi-turn trajectories for training large language models with agentic reinforcement learning. The problem addressed is that current approaches to equip large language models with tool-use capabilities are limited by the lack of scalable and robust execution environments and the scarcity of realistic training data. Existing methods rely on costly real-world APIs, simulators that are prone to hallucination, or synthetic environments that are often single-turn or based on pre-collected documents.

EnvFactory addresses these challenges by autonomously exploring and verifying stateful, executable tool environments from authentic resources, and synthesizing natural multi-turn trajectories through topology-aware sampling and calibrated refinement. This approach produces grounded queries with implicit intents, which are more effective for reinforcement learning training.

The method involves using a fully automated framework to generate environments and trajectories. The results show that using only 85 verified environments across 7 domains, EnvFactory generates a large number of trajectories, achieving superior training efficiency and downstream performance. The framework improves the performance of Qwen3-series models by up to 15 percent on certain benchmarks, and by up to 8.6 percent and 6 percent on other conversational benchmarks.

The contributions of the paper are that EnvFactory provides a scalable, extensible, and robust foundation for agentic reinforcement learning, and that it achieves superior performance with fewer resources compared to prior work. The framework has the potential to advance the field of large language models and their application to real-world problems. Overall, the paper presents a significant contribution to the field of artificial intelligence and natural language processing.

📅 Published on May 18

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.18703
• PDF: https://arxiv.org/pdf/2605.18703

🤖 Models citing this paper:
• https://huggingface.co/LARK-Lab/EnvFactory-1.7B
• https://huggingface.co/LARK-Lab/EnvFactory-4B
• https://huggingface.co/LARK-Lab/EnvFactory-8B

📊 Datasets citing this paper:
• https://huggingface.co/datasets/LARK-Lab/EnvFactory-SFT-ALL
• https://huggingface.co/datasets/LARK-Lab/EnvFactory-SFT-FILTERED
• https://huggingface.co/datasets/LARK-Lab/EnvFactory-RL

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#ExecutableEnvironments #ToolUseAgents #AgenticReinforcementLearning #RobustRL #LanguageModelTraining

🔥 OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning

💡 The paper proposes a framework called On-Policy Skill Distillation, or OPID, which aims to improve the efficiency and performance of language agent training in reinforcement learning. The problem addressed is that outcome-based reinforcement learning provides sparse rewards that do not offer sufficient guidance on intermediate decisions, while existing self-distillation methods often rely on external skill memories that can be costly to maintain and may not match the current policy.

OPID addresses this issue by extracting skill supervision directly from completed on-policy trajectories, representing trajectory hindsight as hierarchical skills that capture both global and local decision knowledge. The framework uses a critical-first routing mechanism to select the most relevant skill and inject it into the interaction history, allowing the old policy to re-score responses under both original and skill-augmented contexts. This yields a token-level self-distillation advantage that is combined with the outcome advantage for policy optimization.

The results of the experiments demonstrate that OPID generally improves agent performance, sample efficiency, and robustness over outcome-only reinforcement learning and existing skill-distillation baselines. The framework preserves reinforcement learning as the primary training objective while introducing dense, distribution-matched hindsight supervision. The experiments were conducted on several datasets, including ALFWorld, WebShop, and Search-based QA, and the code is available for further research. Overall, OPID offers a novel approach to skill distillation that can enhance the training of language agents in reinforcement learning.

📅 Published on Jun 25

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.26790
• PDF: https://arxiv.org/pdf/2606.26790

🤖 Models citing this paper:
• https://huggingface.co/Jinyang23/OPID-ALFWorld-1.7B

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#AgenticReinforcementLearning #OnPolicyLearning #SkillDistillation #ReinforcementLearningFrameworks #HierarchicalReinforcementLearning