🔥 Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

💡 The paper presents Mage-VL, an efficient codec-native streaming multimodal foundation model for real-time multimodal understanding and interaction. Standard vision-language models suffer from Moravec's paradox, exceling at complex offline visual reasoning but struggling with simple streaming perception tasks and processing them inefficiently. To address this, the authors propose a custom tokenizer, Mage-ViT, which replaces uniform frame sampling with selective encoding of dynamic, entropy-rich regions using motion vectors and residual energy across sparse anchor and predicted frames. This approach reduces visual token consumption by over 75 percent while preserving spatiotemporal context.

Mage-ViT is trained from scratch on approximately 560 million unlabeled images and 100 million unlabeled video frames, and it matches or outperforms flagship encoders trained on billions of image-text pairs. The authors also establish AI4AI data pipelines encompassing prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes.

Furthermore, through a bio-inspired dual-system architecture, consisting of a lightweight System 1 event gate and a causal System 2 decoder, Mage-VL enables proactive streaming perception. Extensive evaluations show that Mage-VL-4B matches Qwen3-VL-4B on static tasks while achieving strong gains in video understanding and 2D/3D spatial reasoning, with up to a 3.5 times wall-clock inference speedup, and comprehensively surpasses the 15B Phi-4-reasoning-vision baseline.

The paper delivers seven key empirical findings covering pre-training data efficiency, variable-resolution scaling, codec system acceleration, Video QA SF redundancy, motion-spatial synergy, AI4AI data pipelines, and Zero-Vision SFT for multimodal RL. Overall, the paper presents a novel approach to multimodal understanding and interaction, with significant improvements in efficiency, accuracy, and speed.

📅 Published on Jul 27

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.24904
• PDF: https://arxiv.org/pdf/2607.24904
• Project Page: https://microsoft.github.io/Mage

🤖 Models citing this paper:
• https://huggingface.co/microsoft/Mage-VL

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/microsoft/mage-vl-demo

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalFoundationModels #CodecNativeStreaming #VisionLanguageModels #RealTimeMultimodalUnderstanding #EfficientMultimodalProcessing