🔥 Lance: Unified Multimodal Modeling by Multi-Task Synergy

💡 The paper introduces Lance, a unified multimodal model that combines understanding, generation, and editing capabilities for images and videos. The goal is to develop a model that can handle multiple tasks without relying on large model capacity or focusing on specific modalities like text or images. Lance achieves this through a dual-stream architecture and collaborative multi-task training, which enables joint context learning while separating the pathways for understanding and generation.

The model uses a mixture-of-experts architecture on shared multimodal sequences, allowing it to learn from both images and videos simultaneously. To address interference among different visual tokens, the model employs modality-aware rotary positional encoding, which helps to align tasks across different modalities.

During training, Lance uses a staged multi-task training paradigm with capability-oriented objectives and adaptive data scheduling. This approach strengthens both semantic comprehension and visual generation performance. The results show that Lance outperforms existing unified models in image and video generation while maintaining strong multimodal understanding capabilities.

Overall, Lance presents a practical approach to unified multimodal modeling, demonstrating that collaborative multi-task training and a dual-stream architecture can lead to improved performance in multiple tasks without requiring large model capacity. The model has the potential to be applied to various applications that require multimodal understanding, generation, and editing capabilities.

📅 Published on May 18

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.18678
• PDF: https://arxiv.org/pdf/2605.18678
• Project Page: https://lance-project.github.io/

🤖 Models citing this paper:
• https://huggingface.co/bytedance-research/Lance

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/Nayefleb/Lance

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalModeling #MultitaskLearning #DualStreamArchitecture #MixtureOfExperts #UnifiedModelingApproach

🔥 ViQ: Text-Aligned Visual Quantized Representations at Any Resolution

💡 The paper introduces ViQ, a visual quantization framework that aims to balance semantic richness and detail preservation in discrete representations of images. The goal is to create a unified representation for text and vision that enables simpler multimodal modeling and more efficient training. Existing methods struggle to balance low-level details and high-level semantics in discrete representations, often resulting in severe information loss.

The ViQ framework addresses this issue by structuring quantization learning into two stages: text-aligned pre-training and feature discretization. In the first stage, the visual encoder is pre-trained with semantic-rich supervision from a pre-trained language model, allowing it to process native-resolution visual inputs. In the second stage, a proximal representation learning strategy is used to progressively compact the feature space, along with a position-aware head-wise quantization mechanism that enables flexible processing of arbitrary resolutions.

The results show that ViQ achieves competitive performance compared to state-of-the-art multimodal vision encoders with continuous and high-dimensional visual features, while maintaining high precision in low-level reconstruction. Additionally, multimodal training with visual quantized representations using ViQ leads to significant efficiency improvements, with up to 20-70 percent acceleration in training time compared to different base language models and training recipes. Overall, the paper presents a novel approach to visual quantization that balances semantics and details in discrete representations, enabling more efficient and effective multimodal modeling.

📅 Published on Jun 25

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.27313
• PDF: https://arxiv.org/pdf/2606.27313

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#VisualQuantization #MultimodalModeling #TextVisionAlignment #DiscreteRepresentations #ImageQuantization

🔥 TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training

💡 The paper presents TorchUMM, a unified codebase for evaluating and analyzing multimodal models that can understand, generate, and edit across visual and textual modalities. The development of unified multimodal models has been hindered by the diversity of model architectures and training paradigms, making it challenging to compare and analyze different models. To address this issue, TorchUMM provides a comprehensive framework that supports a wide range of models and tasks, including multimodal understanding, generation, and editing. The codebase includes a broad spectrum of models and integrates both established and novel datasets to evaluate perception, reasoning, compositionality, and instruction-following abilities. By providing a unified interface and standardized evaluation protocols, TorchUMM enables fair and reproducible comparisons across heterogeneous models, facilitating the development of more capable unified multimodal systems. The codebase is designed to foster deeper insights into the strengths and limitations of different models, and its availability is expected to accelerate the development of unified multimodal models. The TorchUMM codebase is available for use, allowing researchers to evaluate and analyze their models in a standardized and reproducible manner. Overall, the paper contributes to the development of unified multimodal models by providing a unified framework for evaluation, analysis, and post-training, which can help to advance the field of multimodal modeling.

📅 Published on Apr 12

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2604.10784
• PDF: https://arxiv.org/pdf/2604.10784
• Project Page: https://aifrontierlab.github.io/TorchUMM/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalModeling #UnifiedCodebase #MultimodalUnderstanding #VisualTextualModalities #MultimodalGeneration

🔥 Hunyuan3D-Buffalo 1.0: A Unified Multimodal Model for Scalable 3D Generation, Understanding, and Editing

💡 The paper introduces Hunyuan3D-Buffalo 1.0, a unified multimodal model that integrates 3D understanding, generation, editing, and part generation into a single architecture. The main problem addressed is the lack of large-scale and geometrically consistent editing data for 3D modeling, which hinders the development of unified 3D models. To overcome this limitation, the authors propose a framework that combines two main components: Hunyuan3D-VLM for semantic, structural, and spatial understanding, and Hunyuan3D DiT for high-fidelity 3D synthesis.

The method involves training the model on an 87M-scale 3D multimodal corpus, which includes 25M understanding samples, 50M text-to-3D pairs, and 12M editing pairs generated using Nano3D-v2. The model uses multimodal semantic conditions for generation and conditions the diffusion process on the source object representation for editing and part generation, preserving the overall structure and unedited regions.

The results show that Hunyuan3D-Buffalo 1.0 achieves state-of-the-art or leading performance on text-to-3D generation and 3D editing benchmarks, while also exhibiting strong understanding and part-generation capabilities. The analysis demonstrates that unified 3D multimodal training is effective, as both generation and understanding improve editing. The paper contributes to the development of scalable and unified 3D modeling by providing a framework that can perform multiple tasks within a single architecture, enabling more efficient and effective 3D modeling and editing.

📅 Published on Aug 3

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2608.02711
• PDF: https://arxiv.org/pdf/2608.02711
• Project Page: https://tencent-hunyuan.github.io/Hunyuan3D-Buffalo1.0/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#3DGeneration #MultimodalModeling #3DUnderstanding #GeometricEditing #3DModelingTechniques