💡 The paper introduces Self Distilled Agentic Reinforcement Learning, a method that improves reinforcement learning for multi turn agent training. The problem with traditional reinforcement learning is that it provides only coarse supervision for long horizon interaction, which can lead to instability in multi turn agents. On Policy Self Distillation is a technique that complements reinforcement learning by providing dense token level guidance from a teacher branch, but it has limitations when applied to multi turn agents, such as compounding instability and negative teacher rejections.

The proposed method, Self Distilled Agentic Reinforcement Learning, addresses these limitations by treating On Policy Self Distillation as a gated auxiliary objective, while keeping reinforcement learning as the primary optimization backbone. It uses a sigmoid gate to selectively strengthen positive token level guidance and mitigate negative teacher rejections. This allows the method to stabilize supervision and improve the performance of multi turn agents.

The results show that Self Distilled Agentic Reinforcement Learning substantially improves over existing methods, such as GRPO, and avoids the instability of naive combinations of GRPO and On Policy Self Distillation. The method consistently outperforms hybrid reinforcement learning and On Policy Self Distillation baselines across different model scales and datasets, including Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA. Overall, the paper contributes a new method that improves the performance and stability of multi turn agents in reinforcement learning.