💡 The paper introduces Harness-1, a 20 billion parameter search agent trained with reinforcement learning to improve retrieval performance across multiple domains. The problem with traditional search agents is that they are trained as policies that must decide how to search while also remembering what they have seen, which can lead to inefficient state management. To address this, the authors propose separating semantic decision-making from environmental bookkeeping by using a stateful search harness that maintains environment-side working memory. This harness includes features such as a candidate pool, importance-tagged curated set, compact evidence links, verification records, and budget-aware context rendering. The policy is then responsible for making semantic decisions, such as what to search, which documents to keep or discard, and when to stop. The results show that Harness-1 achieves an average curated recall of 0.730 across eight retrieval benchmarks, outperforming the next strongest open search subagent by 11.4 points. The gains are especially strong on held-out transfer benchmarks, suggesting that the approach can produce retrieval behaviors that generalize beyond the training domains. Overall, the paper demonstrates that using a stateful search harness to separate state management from semantic decision-making can lead to improved retrieval performance and more generalizable search behaviors.

💡 The paper proposes a reinforcement learning algorithm called QGF that improves policies at test time by using a value gradient to guide a pre-trained flow policy. The problem addressed is that incorporating flow models into reinforcement learning pipelines for policy improvement can be difficult due to stability and scalability issues. The method involves pre-training a reference flow policy and a value function critic, then using the value gradient to guide the reference policy to generate higher-value actions at test time, without any additional policy learning. This approach avoids the instability of actor-critic training and sidesteps the need for specialized training objectives or backpropagating through denoising processes. The results show that QGF outperforms prior test-time reinforcement learning methods on single-task and goal-conditioned offline benchmarks with high-dimensional action spaces, and is competitive with state-of-the-art training-time algorithms while being much cheaper to run. Additionally, QGF exhibits favorable scaling with model size, offering a practical and effective alternative reinforcement learning algorithm with expressive policies. Overall, the paper contributes a new approach to reinforcement learning that improves policies at test time, avoiding training-time instability while maintaining competitive performance.

💡 The paper introduces Molt, a scalable PyTorch-native training framework for agentic reinforcement learning. The problem addressed is the high cost of modifying algorithms, estimators, pipelines, and rollout schemes in existing frameworks, which can lead to significant overhead for researchers. Molt aims to keep this cost small by providing a compact and clean codebase that is easy for researchers to understand and modify.

The method involves designing Molt as a PyTorch-native framework, where the agent is an ordinary program that can train multi-modal and mixture-of-experts policies in a single asynchronous loop. This approach allows for consistent tokenization, policy versions, and model semantics, without requiring the agent to train on tokens it did not generate.

The results show that Molt is statistically comparable to a state-of-the-art Megatron-based stack, even when using an unmatched, fully asynchronous protocol. This demonstrates that the lean design of Molt does not come at the cost of performance. Additionally, Molt is open-source and provides recipes and containers, making it accessible to researchers and practitioners.

Overall, the paper contributes a novel framework for agentic reinforcement learning that prioritizes simplicity, flexibility, and performance. By providing a clean and compact codebase, Molt enables researchers to easily modify and extend the framework, reducing the overhead associated with algorithmic changes and promoting faster innovation in the field.