💡 The paper introduces AstraFlow, a dataflow-oriented reinforcement learning system designed to improve the efficiency and scalability of large language model agents. The problem addressed is that current reinforcement learning systems are prohibitively expensive and struggle to support complex workloads, such as multi-policy collaborative training, while efficiently using diverse compute resources.

The authors propose AstraFlow as a solution, which replaces conventional trainer-centered control with principled component abstractions. In AstraFlow, rollout services, dataflow management, and training are decoupled into autonomous components, allowing the system to natively support complex multi-policy agentic RL workloads and efficiently exploit diverse compute resources.

The results show that AstraFlow supports multi-policy training, elastic scaling, heterogeneous cross-region execution, and composable data algorithms without requiring system-level code changes. The system achieves comparable or better accuracy than existing RL systems while speeding up training time by 2.7 times in multi-policy collaborative training. The evaluation is done across various workloads, including math, code, search, and AgentBench, demonstrating the system's versatility and efficiency.

Overall, AstraFlow's contributions include its ability to efficiently support complex workloads, scale to large language model agents, and provide a principled abstraction for reinforcement learning system components, making it a significant advancement in the field of reinforcement learning for large language models.

💡 The paper introduces PEEK, a system designed to improve the performance of large language model agents operating over long and recurring external contexts, such as document corpora and code repositories. The problem with existing approaches is that they do not preserve reusable orientation knowledge about the recurring context itself, which includes information about what the context contains, how it is organized, and which entities, constants, and schemas have historically been useful.

To address this issue, PEEK uses a context map, a small and constant-sized artifact in the agent's prompt, to cache and maintain this orientation knowledge. The context map is maintained by a programmable cache policy consisting of three modules: a Distiller that extracts transferable knowledge from inference-time signals, a Cartographer that translates it into structured edits, and a priority-based Evictor that enforces a fixed token budget.

The results show that PEEK improves over strong baselines in long-context reasoning and information aggregation tasks by 6.3-34.0 percent, while using 93-145 fewer iterations and incurring 1.7-5.8 times lower cost than the state-of-the-art prompt-learning framework, ACE. Additionally, PEEK improves solving rate and rubric accuracy in context learning tasks by 6.0-14.0 percent and 7.8-12.1 percent, respectively, at 1.4 times lower cost than ACE. These gains generalize across different language models and agent architectures, including OpenAI Codex, a production-grade coding agent. Overall, the paper demonstrates that using a context map helps long-context language model agents interact with recurring external contexts more accurately and efficiently.

💡 The paper introduces GenericAgent, a self-evolving large language model agent system designed to overcome the limitations of long-horizon interactions. The main problem addressed is that as interactions become longer, the accumulation of tool descriptions, memories, and environmental feedback pushes out the information needed for decision-making, leading to poor performance. The authors argue that the key to improving long-horizon performance is not the length of the context, but rather how much decision-relevant information is maintained within a finite context budget.

To address this problem, the GenericAgent system is built around the principle of context information density maximization. The system consists of four main components: a minimal atomic tool set, a hierarchical on-demand memory, a self-evolution mechanism, and a context truncation and compression layer. The minimal atomic tool set keeps the interface simple, while the hierarchical on-demand memory only shows a small high-level view by default. The self-evolution mechanism turns verified past trajectories into reusable standard operating procedures and executable code, allowing the agent to learn from its experiences. The context truncation and compression layer maintains information density during long executions by removing unnecessary information.

The results show that GenericAgent consistently outperforms leading agent systems in terms of task completion, tool use efficiency, memory effectiveness, self-evolution, and web browsing. Moreover, GenericAgent achieves these results while using significantly fewer tokens and interactions, demonstrating its efficiency. The system also continues to evolve over time, allowing it to adapt to new situations and improve its performance. Overall, the paper presents a novel approach to building self-evolving large language model agents that can effectively handle long-horizon interactions and maximize context information density.

💡 The paper introduces a new approach to studying long-horizon large language model agents, called AgenticSTS. The problem addressed is that current methods for analyzing memory components in these agents are limited, as they append past observations and reflections to every prompt, making it hard to isolate the effect of a single memory component. To solve this, the authors propose a bounded contract approach, where every decision is made from a fresh user message assembled by typed retrieval, with no raw cross-decision transcript appended. This allows for isolated analysis of memory components and demonstrates improved performance in complex decision-making tasks.

The method involves instantiating this contract in a closed-rule stochastic deck-building game, where runs require hundreds of tactical and strategic decisions. The authors create a testbed, called AgenticSTS, which includes a reproducible environment, frozen memory and skill snapshots, prompt records, and analysis scripts. This testbed allows for the study of how explicit memory layers shape long-horizon LLM-agent decisions.

The results show that the proposed approach leads to improved performance in the game, with a fixed-A0 ablation showing the largest observed difference when triggered strategic skills are enabled. The no-store baseline wins 3 out of 10 games, while adding the skill layer wins 6 out of 10 games. Although the comparison is directional rather than statistically decisive, the results demonstrate the effectiveness of the proposed approach. The authors also release a public online benchmark of frontier LLMs on the same game, which reports zero wins at the lowest difficulty across five configurations, highlighting the challenge of the task. Overall, the paper contributes a new methodology for studying long-horizon LLM agents and demonstrates its effectiveness in a complex decision-making task.