🔥 MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

💡 The paper introduces MODUS, a novel any-to-any modeling approach that enables the prediction of any modality from any combination of others within a single network. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, which can impact their performance and prevent them from leveraging strong pre-trained decoder-only models. In contrast, MODUS treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. This is achieved through a decoder-only architecture that allows every modality to be both an input and an output of the same model. The resulting model, MODUS, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model's own outputs with another generated modality. The authors demonstrate that MODUS exhibits strong out-of-the-box performance and is competitive with specialist and multi-task baseline models using a single model across various benchmarks. Overall, MODUS provides a flexible and effective approach to any-to-any modeling, and all materials are open-sourced for further research and development.

📅 Published on Jul 28

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.25948
• PDF: https://arxiv.org/pdf/2607.25948
• Project Page: https://modus-multimodal.epfl.ch/

📊 Datasets citing this paper:
• https://huggingface.co/datasets/epfl-vilab-modus/MODUS-15Modality

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalLearning #DecoderOnlyArchitectures #AnyToAnyModeling #ModalityTransfer #CrossModalPrediction