💡 The paper introduces Harness-1, a 20 billion parameter search agent trained with reinforcement learning to improve retrieval performance across multiple domains. The problem with traditional search agents is that they are trained as policies that must decide how to search while also remembering what they have seen, which can lead to inefficient state management. To address this, the authors propose separating semantic decision-making from environmental bookkeeping by using a stateful search harness that maintains environment-side working memory. This harness includes features such as a candidate pool, importance-tagged curated set, compact evidence links, verification records, and budget-aware context rendering. The policy is then responsible for making semantic decisions, such as what to search, which documents to keep or discard, and when to stop. The results show that Harness-1 achieves an average curated recall of 0.730 across eight retrieval benchmarks, outperforming the next strongest open search subagent by 11.4 points. The gains are especially strong on held-out transfer benchmarks, suggesting that the approach can produce retrieval behaviors that generalize beyond the training domains. Overall, the paper demonstrates that using a stateful search harness to separate state management from semantic decision-making can lead to improved retrieval performance and more generalizable search behaviors.

💡 The paper proposes a reinforcement learning algorithm called QGF that improves policies at test time by using a value gradient to guide a pre-trained flow policy. The problem addressed is that incorporating flow models into reinforcement learning pipelines for policy improvement can be difficult due to stability and scalability issues. The method involves pre-training a reference flow policy and a value function critic, then using the value gradient to guide the reference policy to generate higher-value actions at test time, without any additional policy learning. This approach avoids the instability of actor-critic training and sidesteps the need for specialized training objectives or backpropagating through denoising processes. The results show that QGF outperforms prior test-time reinforcement learning methods on single-task and goal-conditioned offline benchmarks with high-dimensional action spaces, and is competitive with state-of-the-art training-time algorithms while being much cheaper to run. Additionally, QGF exhibits favorable scaling with model size, offering a practical and effective alternative reinforcement learning algorithm with expressive policies. Overall, the paper contributes a new approach to reinforcement learning that improves policies at test time, avoiding training-time instability while maintaining competitive performance.