🔥 MiniCPM4: Ultra-Efficient LLMs on End Devices

💡 The paper introduces MiniCPM4, a highly efficient large language model designed for end-side devices. The goal is to achieve superior performance while being efficient, which is a challenge for large language models due to their computational requirements. To address this, the authors propose innovations in four key areas: model architecture, training data, training algorithms, and inference systems.

In terms of model architecture, the authors propose InfLLM v2, a trainable sparse attention mechanism that accelerates both prefilling and decoding phases for long-context processing. For training data, they propose UltraClean, an efficient and accurate pre-training data filtering and generation strategy, and UltraChat v2, a comprehensive supervised fine-tuning dataset. These datasets enable satisfactory model performance to be achieved using just 8 trillion training tokens.

The authors also propose ModelTunnel v2 for efficient pre-training strategy search and improve existing post-training methods by introducing chunk-wise rollout for load-balanced reinforcement learning and data-efficient ternary LLM, BitCPM. For inference systems, they propose CPM.cu, which integrates sparse attention, model quantization, and speculative sampling to achieve efficient prefilling and decoding.

The MiniCPM4 model is available in two versions, with 0.5B and 8B parameters, respectively. The evaluation results show that MiniCPM4 outperforms open-source models of similar size across multiple benchmarks, highlighting both its efficiency and effectiveness. Notably, MiniCPM4-8B demonstrates significant speed improvements over Qwen3-8B when processing long sequences.

The results also show that MiniCPM4 can be adapted to power diverse applications, including trustworthy survey generation and tool use with model context protocol, clearly showcasing its broad usability. Overall, the paper presents a highly efficient large language model that achieves superior performance on end-side devices, making it a significant contribution to the field of natural language processing.

📅 Published on Jun 9, 2025

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2506.07900
• PDF: https://arxiv.org/pdf/2506.07900
• Project Page: https://huggingface.co/collections/openbmb/minicpm4-6841ab29d180257e940baa9b

🤖 Models citing this paper:
• https://huggingface.co/openbmb/MiniCPM4.1-8B
• https://huggingface.co/openbmb/MiniCPM5-1B
• https://huggingface.co/openbmb/MiniCPM4-8B

📊 Datasets citing this paper:
• https://huggingface.co/datasets/openbmb/Ultra-FineWeb

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/openbmb/MiniCPM5-1B-Demo
• https://huggingface.co/spaces/openbmb/Ultra-FineWeb-L2-Selector
• https://huggingface.co/spaces/openbmb/MiniCPM4.1-8B-Demo

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#EfficientLLMs #LargeLanguageModels #SparseAttentionMechanisms #EndDeviceComputing #LowResourceNLP