💡 DeepSeek-V3 is a language model that achieves high performance with efficient training and minimal computational cost. The model uses a Mixture-of-Experts architecture with 671 billion total parameters, but only 37 billion are activated for each token, making it parameter-efficient. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention and DeepSeekMoE architectures, which were validated in the previous version of the model.

The model is trained on 14.8 trillion diverse and high-quality tokens, followed by supervised fine-tuning and reinforcement learning stages to fully harness its capabilities. The training process is stable and requires only 2.788 million H800 GPU hours for full training, which is relatively low compared to other models.

The results show that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. The model also pioneers an auxiliary-loss-free strategy for load balancing and uses a multi-token prediction training objective for stronger performance. The model checkpoints are available for further research and development.

Overall, the DeepSeek-V3 model makes significant contributions to the field of natural language processing by providing a highly efficient and effective language model that can be trained with minimal computational resources. The model's stable training process and low computational cost make it an attractive option for researchers and developers who want to build high-performance language models without incurring high costs.

💡 The paper introduces MiniMax Sparse Attention, a method for efficient processing of ultra-long contexts in large language models. The problem addressed is that the quadratic cost of softmax attention makes it difficult to jointly attend over hundreds of thousands to millions of tokens, which is necessary for applications such as agentic workflows, repository-scale code reasoning, and persistent memory.

To address this problem, the authors propose a blockwise sparse attention built upon Grouped Query Attention, called MiniMax Sparse Attention. This method consists of two branches: a lightweight Index Branch that scores key-value blocks and selects a Top-k subset for each group, and a Main Branch that performs exact block-sparse attention over only the selected blocks.

The method is designed to be simple and scalable, making it easy to deploy efficiently across a range of GPUs. The authors also co-design a GPU execution path that uses exp-free Top-k selection and KV-outer sparse attention to improve tensor-core utilization under block-granular access.

The results show that MiniMax Sparse Attention performs on par with Grouped Query Attention while reducing per-token attention compute by 28.4x at 1M context. When paired with the co-designed kernel, it achieves 14.2x prefill and 7.6x decoding wall-clock speedups on H800. The authors also release a production-grade natively multimodal model powered by MiniMax Sparse Attention, as well as the inference kernel, making it available for use by others.

Overall, the paper contributes a new method for efficient processing of ultra-long contexts in large language models, which has significant implications for applications that require joint attention over large numbers of tokens. The method is designed to be efficient, scalable, and easy to deploy, making it a valuable contribution to the field of natural language processing.

💡 The paper introduces VibeThinker-3B, a compact language model with 3 billion parameters, that achieves state-of-the-art performance on verifiable reasoning tasks, challenging the conventional assumption that large models are necessary for such tasks. The model was developed using a specialized training pipeline that includes curriculum-based supervised fine-tuning, multi-domain reinforcement learning, and offline self-distillation. The model was evaluated on several highly demanding verifiable tasks and achieved impressive results, including a score of 94.3 on AIME26, 80.2 Pass@1 on LiveCodeBench v6, and a 96.1 percent acceptance rate on recent unseen LeetCode contests. These results place VibeThinker-3B in the performance band of first-tier reasoning systems, matching or exceeding the performance of much larger models. The paper also shows that the model's performance does not compromise its instruction controllability, with a score of 93.4 on IFEval. The results of this study support the Parametric Compression-Coverage Hypothesis, which suggests that verifiable reasoning can be compressed into compact reasoning cores, while open-domain knowledge and general-purpose competence require larger models with broader parameter coverage. Overall, the paper demonstrates that compact models can be a complementary path to achieving frontier-level performance on verifiable reasoning tasks, and that they are not just efficient substitutes for larger models.

💡 The paper proposes a new attention mechanism called Hierarchical Landmark Sparse Attention, which enables efficient long-context language modeling. The problem with current large language models is that they are limited by the quadratic computation cost and poor length extrapolation of dense attention, making it difficult to scale them to long contexts. Existing chunk-wise sparse attention methods have inaccurate chunk selection, which falls short of full attention performance.

The proposed method, Hierarchical Landmark Sparse Attention, learns chunk selection end-to-end under the language modeling loss. It factorizes attention hierarchically, where each query performs attention independently with each retrieved chunk to extract chunk-specific information, and the resulting outputs are fused according to chunk retrieval scores. This allows the model to optimize retrieval scores directly with the language modeling loss, enabling end-to-end retrieval learning and native sparse training.

The experimental results show that Hierarchical Landmark Sparse Attention achieves performance comparable to, and in some cases better than, full attention at in-domain context lengths. Moreover, it extrapolates more than 64 times the training context length with 90 percent retrieval accuracy, far beyond full attention. The method also allows existing full-attention models to be converted to Hierarchical Landmark Sparse Attention with lightweight continued pretraining, preserving in-domain performance while acquiring ultra-long-context extrapolation.

Overall, the proposed method breaks the usual efficiency-performance trade-off, enabling long-context language models that are both more efficient and more effective on general long-context tasks than their full-attention counterparts.