💡 The paper introduces MiniMax Sparse Attention, a method for efficient processing of ultra-long contexts in large language models. The problem addressed is that the quadratic cost of softmax attention makes it difficult to jointly attend over hundreds of thousands to millions of tokens, which is necessary for applications such as agentic workflows, repository-scale code reasoning, and persistent memory.

To address this problem, the authors propose a blockwise sparse attention built upon Grouped Query Attention, called MiniMax Sparse Attention. This method consists of two branches: a lightweight Index Branch that scores key-value blocks and selects a Top-k subset for each group, and a Main Branch that performs exact block-sparse attention over only the selected blocks.

The method is designed to be simple and scalable, making it easy to deploy efficiently across a range of GPUs. The authors also co-design a GPU execution path that uses exp-free Top-k selection and KV-outer sparse attention to improve tensor-core utilization under block-granular access.

The results show that MiniMax Sparse Attention performs on par with Grouped Query Attention while reducing per-token attention compute by 28.4x at 1M context. When paired with the co-designed kernel, it achieves 14.2x prefill and 7.6x decoding wall-clock speedups on H800. The authors also release a production-grade natively multimodal model powered by MiniMax Sparse Attention, as well as the inference kernel, making it available for use by others.

Overall, the paper contributes a new method for efficient processing of ultra-long contexts in large language models, which has significant implications for applications that require joint attention over large numbers of tokens. The method is designed to be efficient, scalable, and easy to deploy, making it a valuable contribution to the field of natural language processing.