💡 The paper addresses the challenge of generating high quality agent skills through reinforcement learning. Existing methods rely on heuristics or pipeline style consolidation, which must be specially designed for different evidence sources. In contrast, learning based approaches offer a more unified way to model skill generation across heterogeneous sources. However, learning based skill generation is challenging because skills lack a natural supervision signal based on relevance or correctness, and their value can only be determined by whether they improve the behavior of the agent on downstream tasks.

To address this challenge, the authors propose Skill-alpha, a reinforcement learning method for progressively generating high quality agent skills. The method formulates skill generation as a sequential editing process that decomposes skill construction into individually evaluable edits. A novel roll back reward is introduced to evaluate each edit by comparing downstream execution under the original and edited skills on an anchored query.

The authors conducted extensive experiments to evaluate the effectiveness of Skill-alpha. The results show that Skill-alpha generates more effective skills than methods based on heuristics or pipelines in both document to skill and experience to skill settings. Under the main GPT-4 worker, Skill-alpha improves average downstream success rates over the strongest skill generation baseline by 3.3 points on CL-Bench and 6.7 points on Tau2-Bench. Further ablations validate the importance of roll back reward and progressive generation.

Overall, the paper contributes a novel reinforcement learning method for progressive agent skill generation, which outperforms existing methods in various settings. The proposed method has the potential to improve the performance of agents in downstream tasks by generating high quality skills through a sequential editing process with a novel roll back reward.