💡 DeepSeek-V3 is a language model that achieves high performance with efficient training and minimal computational cost. The model uses a Mixture-of-Experts architecture with 671 billion total parameters, but only 37 billion are activated for each token, making it parameter-efficient. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention and DeepSeekMoE architectures, which were validated in the previous version of the model.

The model is trained on 14.8 trillion diverse and high-quality tokens, followed by supervised fine-tuning and reinforcement learning stages to fully harness its capabilities. The training process is stable and requires only 2.788 million H800 GPU hours for full training, which is relatively low compared to other models.

The results show that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. The model also pioneers an auxiliary-loss-free strategy for load balancing and uses a multi-token prediction training objective for stronger performance. The model checkpoints are available for further research and development.

Overall, the DeepSeek-V3 model makes significant contributions to the field of natural language processing by providing a highly efficient and effective language model that can be trained with minimal computational resources. The model's stable training process and low computational cost make it an attractive option for researchers and developers who want to build high-performance language models without incurring high costs.

💡 The paper introduces MiniMax Sparse Attention, a method for efficient processing of ultra-long contexts in large language models. The problem addressed is that the quadratic cost of softmax attention makes it difficult to jointly attend over hundreds of thousands to millions of tokens, which is necessary for applications such as agentic workflows, repository-scale code reasoning, and persistent memory.

To address this problem, the authors propose a blockwise sparse attention built upon Grouped Query Attention, called MiniMax Sparse Attention. This method consists of two branches: a lightweight Index Branch that scores key-value blocks and selects a Top-k subset for each group, and a Main Branch that performs exact block-sparse attention over only the selected blocks.

The method is designed to be simple and scalable, making it easy to deploy efficiently across a range of GPUs. The authors also co-design a GPU execution path that uses exp-free Top-k selection and KV-outer sparse attention to improve tensor-core utilization under block-granular access.

The results show that MiniMax Sparse Attention performs on par with Grouped Query Attention while reducing per-token attention compute by 28.4x at 1M context. When paired with the co-designed kernel, it achieves 14.2x prefill and 7.6x decoding wall-clock speedups on H800. The authors also release a production-grade natively multimodal model powered by MiniMax Sparse Attention, as well as the inference kernel, making it available for use by others.

Overall, the paper contributes a new method for efficient processing of ultra-long contexts in large language models, which has significant implications for applications that require joint attention over large numbers of tokens. The method is designed to be efficient, scalable, and easy to deploy, making it a valuable contribution to the field of natural language processing.

💡 The paper introduces VibeThinker-3B, a compact language model with 3 billion parameters, that achieves state-of-the-art performance on verifiable reasoning tasks, challenging the conventional assumption that large models are necessary for such tasks. The model was developed using a specialized training pipeline that includes curriculum-based supervised fine-tuning, multi-domain reinforcement learning, and offline self-distillation. The model was evaluated on several highly demanding verifiable tasks and achieved impressive results, including a score of 94.3 on AIME26, 80.2 Pass@1 on LiveCodeBench v6, and a 96.1 percent acceptance rate on recent unseen LeetCode contests. These results place VibeThinker-3B in the performance band of first-tier reasoning systems, matching or exceeding the performance of much larger models. The paper also shows that the model's performance does not compromise its instruction controllability, with a score of 93.4 on IFEval. The results of this study support the Parametric Compression-Coverage Hypothesis, which suggests that verifiable reasoning can be compressed into compact reasoning cores, while open-domain knowledge and general-purpose competence require larger models with broader parameter coverage. Overall, the paper demonstrates that compact models can be a complementary path to achieving frontier-level performance on verifiable reasoning tasks, and that they are not just efficient substitutes for larger models.