💡 The paper introduces Recursive Language Models, a novel approach to enable large language models to process arbitrarily long prompts. The problem addressed is that current language models have limited context windows, which restricts their ability to handle long inputs. The proposed method treats long prompts as part of an external environment and allows the language model to programmatically examine, decompose, and recursively call itself over snippets of the prompt. This approach enables the model to handle inputs that are up to two orders of magnitude beyond the model context window. The results show that Recursive Language Models successfully handle long inputs and outperform base language models and common long-context scaffolds across four diverse long-context tasks, while having comparable or cheaper cost per query. Overall, the paper contributes a general inference strategy that improves the ability of large language models to process long prompts, making them more effective and efficient.

💡 The paper introduces UniPrefill, a universal prefill acceleration framework designed to improve the inference efficiency of long-context processing in large language models. The problem addressed is that existing prefill acceleration methods are limited to specific model architectures and suffer performance degradation when applied to emerging architectures. Additionally, these methods are often incompatible with continuous batching, making it difficult to integrate them into modern inference engines.

The proposed UniPrefill framework overcomes these limitations by directly accelerating the model's computation at the token level, making it applicable to virtually any model architecture. UniPrefill is implemented as a continuous batching operator and is integrated into the vLLM inference engine, enabling seamless support for prefill-decode co-processing and tensor parallelism.

The results show that UniPrefill achieves significant speedup, with up to 2.1x improvement in Time-To-First-Token, and the acceleration becomes more pronounced as the number of concurrent requests grows. This makes UniPrefill a valuable contribution to the field, enabling more efficient and scalable long-context processing in large language models.