💡 The paper proposes a reinforcement learning algorithm called QGF that improves policies at test time by using a value gradient to guide a pre-trained flow policy. The problem addressed is that incorporating flow models into reinforcement learning pipelines for policy improvement can be difficult due to stability and scalability issues. The method involves pre-training a reference flow policy and a value function critic, then using the value gradient to guide the reference policy to generate higher-value actions at test time, without any additional policy learning. This approach avoids the instability of actor-critic training and sidesteps the need for specialized training objectives or backpropagating through denoising processes. The results show that QGF outperforms prior test-time reinforcement learning methods on single-task and goal-conditioned offline benchmarks with high-dimensional action spaces, and is competitive with state-of-the-art training-time algorithms while being much cheaper to run. Additionally, QGF exhibits favorable scaling with model size, offering a practical and effective alternative reinforcement learning algorithm with expressive policies. Overall, the paper contributes a new approach to reinforcement learning that improves policies at test time, avoiding training-time instability while maintaining competitive performance.