💡 The paper introduces Kimi K3, a 2.8 trillion parameter mixture of experts model with native vision capabilities and a 1 million token context window. Kimi K3 is built on Kimi Delta Attention and Attention Residuals, which improve information flow across sequence length and model depth. The model also incorporates Stable Latent MoE, which effectively activates 16 of 896 routed experts per token, and refined training and data recipes. These advances yield an approximately 2.5 times improvement in overall scaling efficiency over Kimi K2.

The model achieves frontier level performance across long horizon coding, genetic, knowledge, reasoning, and vision tasks. Although its overall performance still trails the most powerful proprietary models, namely Claude Fable 5 and GPT 5.6 Sol, Kimi K3 consistently outperforms other open and proprietary models evaluated in the study.

The key contributions of the paper include the introduction of Kimi K3, which is supported by infrastructure advances in multiple areas, such as algorithm system co design for KD, perfectly balanced expert parallel training with efficient memory management, million token genetic RL with persistent rollout and sandbox states, and deployment innovations. The full Kimi K3 model weights are released to facilitate future research and accelerate the broader deployment and adoption of frontier intelligence.

The problem addressed in the paper is the development of a highly efficient and scalable model that can achieve state of the art performance across a wide range of tasks. The method used to address this problem is the introduction of Kimi K3, which incorporates several key innovations, including Kimi Delta Attention, Attention Residuals, and Stable Latent MoE. The results of the study demonstrate the effectiveness of Kimi K3, which achieves frontier level performance across a range of tasks and outperforms other open and proprietary models. Overall, the paper contributes to the development of highly efficient and scalable models that can achieve state of the art performance across a wide range of tasks.