🔥 Lance: Unified Multimodal Modeling by Multi-Task Synergy

💡 The paper introduces Lance, a unified multimodal model that combines understanding, generation, and editing capabilities for images and videos. The goal is to develop a model that can handle multiple tasks without relying on large model capacity or focusing on specific modalities like text or images. Lance achieves this through a dual-stream architecture and collaborative multi-task training, which enables joint context learning while separating the pathways for understanding and generation.

The model uses a mixture-of-experts architecture on shared multimodal sequences, allowing it to learn from both images and videos simultaneously. To address interference among different visual tokens, the model employs modality-aware rotary positional encoding, which helps to align tasks across different modalities.

During training, Lance uses a staged multi-task training paradigm with capability-oriented objectives and adaptive data scheduling. This approach strengthens both semantic comprehension and visual generation performance. The results show that Lance outperforms existing unified models in image and video generation while maintaining strong multimodal understanding capabilities.

Overall, Lance presents a practical approach to unified multimodal modeling, demonstrating that collaborative multi-task training and a dual-stream architecture can lead to improved performance in multiple tasks without requiring large model capacity. The model has the potential to be applied to various applications that require multimodal understanding, generation, and editing capabilities.

📅 Published on May 18

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.18678
• PDF: https://arxiv.org/pdf/2605.18678
• Project Page: https://lance-project.github.io/

🤖 Models citing this paper:
• https://huggingface.co/bytedance-research/Lance

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/Nayefleb/Lance

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalModeling #MultitaskLearning #DualStreamArchitecture #MixtureOfExperts #UnifiedModelingApproach

🔥 ViQ: Text-Aligned Visual Quantized Representations at Any Resolution

💡 The paper introduces ViQ, a visual quantization framework that aims to balance semantic richness and detail preservation in discrete representations of images. The goal is to create a unified representation for text and vision that enables simpler multimodal modeling and more efficient training. Existing methods struggle to balance low-level details and high-level semantics in discrete representations, often resulting in severe information loss.

The ViQ framework addresses this issue by structuring quantization learning into two stages: text-aligned pre-training and feature discretization. In the first stage, the visual encoder is pre-trained with semantic-rich supervision from a pre-trained language model, allowing it to process native-resolution visual inputs. In the second stage, a proximal representation learning strategy is used to progressively compact the feature space, along with a position-aware head-wise quantization mechanism that enables flexible processing of arbitrary resolutions.

The results show that ViQ achieves competitive performance compared to state-of-the-art multimodal vision encoders with continuous and high-dimensional visual features, while maintaining high precision in low-level reconstruction. Additionally, multimodal training with visual quantized representations using ViQ leads to significant efficiency improvements, with up to 20-70 percent acceleration in training time compared to different base language models and training recipes. Overall, the paper presents a novel approach to visual quantization that balances semantics and details in discrete representations, enabling more efficient and effective multimodal modeling.

📅 Published on Jun 25

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.27313
• PDF: https://arxiv.org/pdf/2606.27313

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#VisualQuantization #MultimodalModeling #TextVisionAlignment #DiscreteRepresentations #ImageQuantization