✨Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss

📝 Summary:
Expert-Router Coupling ERC loss aligns MoE router decisions with expert capabilities. It uses proxy tokens and activation constraints to ensure experts specialize, improving performance and computational efficiency. ERC also allows tracking expert specialization during training.

🔹 Publication Date: Published on Dec 29

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.23447
• PDF: https://arxiv.org/pdf/2512.23447

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MixtureOfExperts #DeepLearning #MachineLearning #AI #NeuralNetworks

✨YOLO-Master: MOE-Accelerated with Specialized Transformers for Enhanced Real-time Detection

📝 Summary:
YOLO-Master proposes an Efficient Sparse Mixture-of-Experts ES-MoE block for real-time object detection. It adaptively allocates computational resources based on scene complexity using a dynamic routing network, overcoming static computation limits. This improves accuracy and speed, especially on...

🔹 Publication Date: Published on Dec 29

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.23273
• PDF: https://arxiv.org/pdf/2512.23273

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#ObjectDetection #YOLO #MixtureOfExperts #Transformers #RealTimeAI

✨The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models

📝 Summary:
Mixture of Experts models exhibit a Standing Committee of experts that consistently dominates routing across domains, challenging the assumption of widespread specialization. This reveals a strong structural bias toward centralized computation, limiting effective specialization.

🔹 Publication Date: Published on Jan 6

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2601.03425
• PDF: https://arxiv.org/pdf/2601.03425

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MixtureOfExperts #DeepLearning #MachineLearning #AISpecialization #NeuralNetworks

✨FlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Federatedly-trained Large Language Models

📝 Summary:
FlexMoRE proposes replacing full-sized experts with low-rank adapters in Mixture-of-Experts for federated LLMs. This flexible approach improves performance using significantly fewer parameters, with optimal expert rank depending on task complexity.

🔹 Publication Date: Published on Feb 9

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.08818
• PDF: https://arxiv.org/pdf/2602.08818

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LLM #FederatedLearning #MixtureOfExperts #AI #DeepLearning

✨Large Language Lobotomy: Jailbreaking Mixture-of-Experts via Expert Silencing

📝 Summary:
This paper introduces Large Language Lobotomy L3, an attack on Mixture-of-Experts LLMs. L3 exploits routing dynamics to identify and silence safety-critical experts, achieving high jailbreaking success while retaining language utility. This highlights a fundamental tension in MoE design.

🔹 Publication Date: Published on Feb 9

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.08741
• PDF: https://arxiv.org/pdf/2602.08741
• Github: https://github.com/jonatelintelo/LargeLanguageLobotomy

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LLM #MixtureOfExperts #Jailbreaking #AISafety #AIResearch

✨Arcee Trinity Large Technical Report

📝 Summary:
Arcee Trinity introduces sparse Mixture-of-Experts models Nano, Mini, Large with up to 400B total parameters. They feature advanced attention, novel normalization, and sigmoid MoE routing, trained on massive token datasets.

🔹 Publication Date: Published on Feb 19

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.17004
• PDF: https://arxiv.org/pdf/2602.17004

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MixtureOfExperts #LargeLanguageModels #SparseModels #DeepLearning #AI

✨Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

📝 Summary:
Nemotron 3 Nano is an efficient Mixture-of-Experts hybrid Mamba-Transformer model. It achieves better accuracy and up to 3.3x higher inference throughput than similar models, while using fewer active parameters and supporting 1M token contexts for enhanced agentic reasoning.

🔹 Publication Date: Published on Dec 23, 2025

🔹 Paper Links:
• arXiv Page: https://arxivlens.com/PaperView/Details/nemotron-3-nano-open-efficient-mixture-of-experts-hybrid-mamba-transformer-model-for-agentic-reasoning-1072-37bf9190
• PDF: https://arxiv.org/pdf/2512.20848
• Github: https://github.com/NVIDIA-NeMo/Nemotron

🔹 Models citing this paper:
• https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
• https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8
• https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16

✨ Spaces citing this paper:
• https://huggingface.co/spaces/FINAL-Bench/all-bench-leaderboard
• https://huggingface.co/spaces/hadadxyz/ai
• https://huggingface.co/spaces/hadadxyz/blog

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#Nemotron3Nano #MixtureOfExperts #MambaTransformer #AgenticAI #LLM

✨A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications

📝 Summary:
Mixture of Experts MoE models enhance large AI model efficiency and performance by dynamically selecting sub-models for diverse data. This survey details MoE design, algorithms, theory, and applications in various machine learning fields.

🔹 Publication Date: Published on Mar 10, 2025

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2503.07137
• PDF: https://arxiv.org/pdf/2503.07137
• Github: https://github.com/deepseek-ai/DeepEP

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MixtureOfExperts #MoE #AI #MachineLearning #DeepLearning

🔥 Lance: Unified Multimodal Modeling by Multi-Task Synergy

💡 The paper introduces Lance, a unified multimodal model that combines understanding, generation, and editing capabilities for images and videos. The goal is to develop a model that can handle multiple tasks without relying on large model capacity or focusing on specific modalities like text or images. Lance achieves this through a dual-stream architecture and collaborative multi-task training, which enables joint context learning while separating the pathways for understanding and generation.

The model uses a mixture-of-experts architecture on shared multimodal sequences, allowing it to learn from both images and videos simultaneously. To address interference among different visual tokens, the model employs modality-aware rotary positional encoding, which helps to align tasks across different modalities.

During training, Lance uses a staged multi-task training paradigm with capability-oriented objectives and adaptive data scheduling. This approach strengthens both semantic comprehension and visual generation performance. The results show that Lance outperforms existing unified models in image and video generation while maintaining strong multimodal understanding capabilities.

Overall, Lance presents a practical approach to unified multimodal modeling, demonstrating that collaborative multi-task training and a dual-stream architecture can lead to improved performance in multiple tasks without requiring large model capacity. The model has the potential to be applied to various applications that require multimodal understanding, generation, and editing capabilities.

📅 Published on May 18

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.18678
• PDF: https://arxiv.org/pdf/2605.18678
• Project Page: https://lance-project.github.io/

🤖 Models citing this paper:
• https://huggingface.co/bytedance-research/Lance

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/Nayefleb/Lance

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalModeling #MultitaskLearning #DualStreamArchitecture #MixtureOfExperts #UnifiedModelingApproach

🔥 Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent

💡 The paper introduces Agents-A1, a 35 billion parameter Mixture-of-Experts Agentic Model that achieves performance comparable to trillion-parameter models by scaling the agent horizon instead of the parameters. The problem addressed is how to improve the performance of large language models on long-horizon tasks without increasing the number of parameters. The method used is a three-stage training approach, which includes supervised fine-tuning, domain-level teacher models, and multi-teacher distillation. The model is trained on a long-horizon knowledge-action infrastructure that connects external knowledge, actions, observations, and verifier outcomes, producing agentic trajectories with an average length of 45,000 tokens. The results show that Agents-A1 achieves strong and broad performance on long-horizon agent benchmarks, outperforming or matching the results of 1 trillion parameter models on several tasks, including SEAL-0, IFBench, HiPhO, FrontierScience-Olympiad, and MolBench-Bind. The paper provides a practical path for scaling the horizon using a smaller model that can reach or match the performance of larger models on long-horizon tasks.

📅 Published on Jun 29

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.30616
• PDF: https://arxiv.org/pdf/2606.30616
• Project Page: https://internscience.github.io/Agents-A1/

🤖 Models citing this paper:
• https://huggingface.co/InternScience/Agents-A1
• https://huggingface.co/InternScience/Agents-A1-FP8-dynamic
• https://huggingface.co/Abiray/Agents-A1-Q4_K_M-GGUF

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MixtureOfExperts #AgenticModels #LongHorizonTasks #LargeLanguageModels #ParameterEfficientTraining