🔥 Semantic Generative Tuning for Unified Multimodal Models

💡 The paper addresses the issue of unified multimodal models where visual understanding and generation are not well aligned due to separate training objectives. The prevailing approach of optimizing understanding through text signals and generation through pixel objectives leads to isolated representation spaces. To bridge this gap, the authors propose a novel approach called Semantic Generative Tuning, which uses semantic segmentation as a generative proxy to align and synergize multimodal capabilities.

The method involves formulating hierarchical visual tasks as generative proxies, with a focus on high-level semantic tasks like image segmentation. The authors find that segmentation provides structural semantics that enhance both vision-centric perception and generative layout fidelity. Unlike low-level tasks, segmentation does not distract models with texture details, making it an optimal proxy.

The results show that Semantic Generative Tuning fundamentally improves feature linear separability and optimizes visual-textual attention allocation patterns. Extensive evaluations demonstrate that this approach consistently improves both multimodal comprehension and generative fidelity across mainstream benchmarks. The authors provide a systematic investigation into generative post-training and introduce a new paradigm that leverages segmentation to align multimodal capabilities. The code for the proposed method is made available for further research and development. Overall, the paper presents a significant contribution to the field of unified multimodal models by introducing a novel approach that enhances multimodal alignment and performance.

📅 Published on May 18

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.18714
• PDF: https://arxiv.org/pdf/2605.18714
• Project Page: https://song2yu.github.io/SGT/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalLearning #SemanticSegmentation #GenerativeModels #UnifiedMultimodalModels #MultimodalRepresentationLearning

🔥 LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence

💡 The paper introduces LLaVA-OneVision-2, a vision-language model that achieves superior performance across various multimodal benchmarks. The problem addressed is the need for a more capable model that can efficiently process and understand video content. The method used to achieve this is codec-stream tokenization, which treats compressed video as a continuous bit-cost stream and allocates a limited token budget to event-bearing content. This approach enables more stable long-video token compression than fixed groups of pictures. The model also incorporates windowed attention for efficient local computation and a shared 3D RoPE to place codec canvases, sampled frames, and images in a unified spatiotemporal coordinate system.

The model was trained using large-scale open supervision, with approximately 8 million re-captioned video samples for pretraining and a 4 million sample spatial corpus for fine-tuning. The paper also introduces JumpScore, a temporal-localization benchmark that targets fine-grained grounding in high-frequency, densely repeated motion. The results show that LLaVA-OneVision-2 outperforms existing models, including Qwen3-VL-8B, by a significant margin. On the JumpScore benchmark, LLaVA-OneVision-2-8B reaches 74.9 JumpScore mAP, surpassing Qwen3-VL-8B by 44.8 points. The model also outperforms Qwen3-VL-8B by 4.3 average points on video tasks, 5.3 on spatial tasks, and 15.6 average J&F on tracking tasks.

The key contributions of the paper are the introduction of codec-stream tokenization, windowed attention, and large-scale open supervision, which enable the model to achieve superior performance across a broad range of multimodal benchmarks. The paper also highlights the importance of unified perception across video understanding, temporal grounding, spatial grounding, and manipulation-trace reasoning. Overall, the paper demonstrates the effectiveness of LLaVA-OneVision-2 in achieving next-generation perceptual intelligence.

📅 Published on May 25

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.25979
• PDF: https://arxiv.org/pdf/2605.25979
• Project Page: https://evolvinglmms-lab.github.io/LLaVA-OneVision-2/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalLearning #VisionLanguageModels #VideoContentUnderstanding #PerceptualIntelligence #CodecStreamTokenization

🔥 WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent

💡 The paper introduces WebWatcher, a multimodal agent designed to improve visual-language reasoning in deep research tasks. The problem addressed is that most existing research agents are text-centric and overlook visual information, making multimodal deep research challenging. To solve this, WebWatcher is equipped with enhanced visual-language reasoning capabilities, leveraging synthetic multimodal trajectories for efficient training, utilizing various tools for deep reasoning, and enhancing generalization through reinforcement learning.

The method involves using high-quality synthetic multimodal trajectories for cold start training, which allows the agent to learn from both visual and textual information. The agent is also designed to work with various tools to improve its reasoning abilities. Additionally, the paper proposes a new benchmark called BrowseComp-VL, which is used to evaluate the capabilities of multimodal agents in complex information retrieval tasks involving both visual and textual information.

The results show that WebWatcher significantly outperforms existing baseline agents, including proprietary and open-source agents, in four challenging visual question answering benchmarks. This demonstrates the effectiveness of WebWatcher in solving complex multimodal information-seeking tasks and paves the way for further research in this area. Overall, the paper contributes to the development of multimodal agents with stronger reasoning abilities, which can handle both visual and textual information, and provides a new benchmark for evaluating the performance of such agents.

📅 Published on Aug 7, 2025

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2508.05748
• PDF: https://arxiv.org/pdf/2508.05748
• Project Page: https://tongyi-agent.github.io/blog/introducing-tongyi-deep-research/

🤖 Models citing this paper:
• https://huggingface.co/Alibaba-NLP/WebWatcher-32B
• https://huggingface.co/Alibaba-NLP/WebWatcher-7B

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalLearning #VisionLanguageReasoning #DeepResearchAgents #SyntheticMultimodalTrajectories #ReinforcementLearningForVision

🔥 M^3Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

💡 The paper introduces M3Eval, a comprehensive evaluation framework and benchmark for assessing the memory capabilities of multi-modal models in video understanding systems. The problem addressed is that current multi-modal models have significant limitations in their memory capabilities, particularly in maintaining disentangled representations and demonstrating human-like interference patterns. To address this gap, the authors designed M3Eval, which is grounded in cognitive psychology and features carefully constructed tasks that isolate key aspects of memory.

The method involves conducting extensive experiments across representative multi-modal models using the M3Eval framework, which evaluates different memory dimensions such as what models retain, how faithfully information is preserved, and how robust memory remains under interference. The framework includes tasks that test the models' ability to maintain disentangled representations, exhibit human-like interference patterns, and demonstrate symbolic memory.

The results of the experiments reveal consistent weaknesses and distinctive behaviors in the models. The models struggle to maintain disentangled representations when processing parallel video streams, exhibit interference patterns that differ substantially from those observed in human memory, and ground memory sources more reliably in the spatial domain than the temporal domain. Additionally, the models demonstrate limited symbolic memory.

The paper's contributions include providing a valuable resource for future research in the form of the M3Eval benchmark and highlighting memory as a fundamental yet underexplored capability in multi-modal models. The findings offer insights for designing more effective memory mechanisms in multi-modal models, which can advance the field of video understanding systems. The code and dataset are made available to facilitate future research.

📅 Published on Jun 3

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.05008
• PDF: https://arxiv.org/pdf/2606.05008
• Project Page: https://pku-value-lab.github.io/m3eval-homepage/

📊 Datasets citing this paper:
• https://huggingface.co/datasets/PKU-VaLuE-Lab/m3eval

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalLearning #VideoUnderstanding #CognitiveArchitectures #MemoryEvaluation #MultimodalModels

🔥 UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervision

💡 The paper introduces UniCorn, a self-improvement framework for unified multimodal models that addresses the generation gap in these models. The generation gap refers to the discrepancy between a model's ability to understand multimodal inputs and its ability to generate high-quality outputs. This gap is formalized as Conduction Aphasia, where models can accurately interpret multimodal inputs but struggle to translate that understanding into faithful and controllable synthesis.

To address this issue, UniCorn proposes a simple yet elegant self-improvement framework that eliminates the need for external data or teacher supervision. The framework partitions a single unified multimodal model into three collaborative roles: Proposer, Solver, and Judge. The Proposer generates initial outputs, the Solver refines these outputs, and the Judge evaluates the quality of the refined outputs. Through self-play and cognitive pattern reconstruction, UniCorn generates high-quality interactions and distills latent understanding into explicit generative signals.

The authors introduce UniCycle, a cycle-consistency benchmark based on a Text to Image to Text reconstruction loop, to validate the restoration of multimodal coherence. The results demonstrate that UniCorn achieves comprehensive and substantial improvements over the base model across six general image generation benchmarks. Notably, it achieves state-of-the-art performance on several benchmarks, including TIIF, DPG, CompBench, and UniCycle, and delivers substantial gains on WISE and OneIG.

The contributions of the paper are significant, as UniCorn enhances text-to-image generation while maintaining robust comprehension, demonstrating the scalability of fully self-supervised refinement for unified multimodal intelligence. The results highlight the effectiveness of the self-improvement framework in addressing the generation gap in unified multimodal models, and the potential of UniCorn to improve the performance of these models in various applications. Overall, the paper presents a novel approach to self-improving unified multimodal models, with significant implications for the development of more advanced and effective multimodal models.

📅 Published on Jan 6

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2601.03193
• PDF: https://arxiv.org/pdf/2601.03193
• Project Page: https://costaliya.github.io/UniCorn.github.io/

🤖 Models citing this paper:
• https://huggingface.co/CostaliyA/UniCorn

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalLearning #SelfImprovingModels #UnifiedModels #SelfGeneratedSupervision #MultimodalSynthesis

🔥 Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding?

💡 The paper proposes a novel framework called Robust-U1 to enhance the robustness of multimodal large language models against visual corruptions. The problem addressed is that existing models perform poorly when faced with real-world visual corruptions such as noise or blur. Current approaches to improve robustness have limitations, either lacking interpretability or being unable to restore lost pixel-level details.

The Robust-U1 framework is designed to equip models with explicit visual self-recovery capability, allowing them to recover corrupted visual content by themselves. The approach consists of three stages: supervised fine-tuning for initial reconstruction, reinforcement learning with dual rewards to align high visual quality, and multimodal reasoning that considers both the corrupted input and the recovered image.

The results show that Robust-U1 achieves state-of-the-art robustness on a real-world corruption benchmark and maintains superior performance under adversarial corruptions on general visual question answering benchmarks. The analysis confirms that high-quality visual recovery directly enhances reasoning performance, establishing self-recovery as a critical mechanism for robust visual understanding. Overall, the paper demonstrates that multimodal large language models can self-recover corrupted visual content, leading to improved robustness and performance in visual understanding tasks.

📅 Published on Jun 6

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.08063
• PDF: https://arxiv.org/pdf/2606.08063
• Project Page: https://huggingface.co/spaces/Jiaqi-hkust/Robust-U1

🤖 Models citing this paper:
• https://huggingface.co/Jiaqi-hkust/Robust-U1-SFT
• https://huggingface.co/Jiaqi-hkust/Robust-U1-RL
• https://huggingface.co/Jiaqi-hkust/Robust-U1

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/Jiaqi-hkust/Robust-U1

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalLearning #VisualContentRecovery #RobustLanguageModels #SelfRecoveryMechanisms #CorruptionResistantAI

🔥 OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains

💡 The paper introduces a new dataset and method for improving audio-visual question answering systems. Current systems typically process videos in short clips and generate separate descriptions for audio and visual modalities, which can lead to inconsistent descriptions and a lack of cross-modal reasoning. To address this, the authors propose a two-part approach: entity-anchored video scripting, which transforms videos into structured scripts with summaries, main entity lists, and segment-wise audio-visual descriptions, and clue-guided QA generation, which prompts models to mine cross-segment clues from the script and generate QA pairs based on these clues.

The entity-anchored video scripting mechanism ensures cross-segment referential consistency and reconstructs audio-visual associations, while the clue-guided QA generation mechanism encourages models to generate questions that require long-term temporal connections and deep cross-modal reasoning. The authors use this pipeline to construct a new dataset called OmniVideo-100K, which consists of structured scripts and QA pairs, as well as a human-verified test set called OmniVideo-Test.

The results show that fine-tuning models on OmniVideo-100K yields significant performance gains, with improvements of up to 20.59% on the OmniVideo-Test set. The models also demonstrate strong generalization, with improvements of up to 12.64% on established benchmarks such as Daily-Omni and JointAVBench. Overall, the paper contributes a new dataset and method for improving audio-visual question answering systems, with a focus on cross-modal reasoning and temporal consistency.

📅 Published on Jun 12

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.14702
• PDF: https://arxiv.org/pdf/2606.14702
• Project Page: https://yzlmhzz.github.io/OmniVideo-100K/

📊 Datasets citing this paper:
• https://huggingface.co/datasets/MiG-NJU/OmniVideo-100K
• https://huggingface.co/datasets/MiG-NJU/OmniVideo-Test

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#AudioVisualReasoning #MultimodalLearning #VideoUnderstanding #CrossModalReasoning #AudioVisualQuestionAnswering

🔥 Orchestra-o1: Omnimodal Agent Orchestration

💡 The paper presents Orchestra-o1, an omnimodal agent orchestration framework that enables efficient collaboration across multiple modalities such as text, image, audio, and video. The existing agent orchestration frameworks are limited to a narrow set of modalities and struggle to generalize to complex settings where heterogeneous modalities coexist and interact. To address this limitation, Orchestra-o1 introduces a unified orchestration mechanism that enables modality-aware task decomposition, online sub-agent specialization, and parallel sub-task execution. This allows agent systems to effectively tackle complex real-world tasks involving heterogeneous information sources. The framework is trained using decision-aligned group relative policy optimization, an efficient agentic reinforcement learning approach. The results show that Orchestra-o1 achieves superior performance on complex multimodal benchmarks, surpassing the second-best approach by 10.3 percent accuracy on the OmniGAIA benchmark. Additionally, the trained Orchestra-o1-8B model achieves state-of-the-art performance against all existing open-source omnimodal agents, demonstrating the effectiveness of the proposed framework. Overall, the paper contributes to the development of omnimodal agent orchestration frameworks that can efficiently collaborate across multiple modalities, enabling the creation of more complex and powerful agent systems.

📅 Published on Jun 10

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.13707
• PDF: https://arxiv.org/pdf/2606.13707

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#OmnimodalAgentOrchestration #MultimodalLearning #AgentCollaborationFrameworks #ModalityAwareTaskDecomposition #HeterogeneousModalitiesIntegration

🔥 UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation

💡 The paper introduces UniverSat, a new approach to applying Vision Transformers to Earth Observation data. The problem with current Vision Transformers is that they rely on rigid patch projectors, which makes it difficult to transfer them to Earth Observation tasks where the input data can vary widely in terms of modality, scale, and resolution. To address this issue, the authors propose a Universal Patch Encoder that can map patches from arbitrary spatial, spectral, and temporal resolutions, and from both optical and non-optical sensors, into a shared embedding space using a shared set of weights. This allows a single model to be trained on heterogeneous multimodal data using self-supervision, resulting in robust and sensor-agnostic spatial features. The authors validate their approach by achieving strong results on classification and segmentation tasks using standard Earth Observation benchmarks. The key contribution of UniverSat is its ability to enable resolution- and modality-agnostic spatial feature extraction, making it a versatile and effective tool for Earth Observation tasks. The authors make their code and models available for further research and development.

📅 Published on Jun 22

🔗 Links:
• GitHub: https://github.com/huggingface
• Project Page: https://huggingface.co/papers?q=patch%20projectors
• arXiv: https://arxiv.org/abs/2606.23503
• PDF: https://arxiv.org/pdf/2606.23503

🤖 Models citing this paper:
• https://huggingface.co/g-astruc/UniverSat

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#EarthObservation #VisionTransformers #MultimodalLearning #RemoteSensing #GeospatialAnalysis

🔥 Training Video Foundation Models with NVIDIA NeMo

💡 The paper addresses the challenges of training large scale high quality video foundation models that can generate high quality videos. Video foundation models have been used to simulate the real world and develop creative visual experiences but training them is difficult due to the complexity and size of video datasets. To overcome this the authors present a scalable open source pipeline using NVIDIA NeMo for training and inference of video foundation models. The pipeline provides accelerated video dataset curation multimodal data loading and parallelized video diffusion model training and inference. The authors also provide a comprehensive performance analysis highlighting best practices for efficient video foundation model training and inference. The pipeline is designed to address the challenges of training large scale video foundation models and provides a scalable and efficient solution for generating high quality videos. The results of the paper demonstrate the effectiveness of the pipeline in training video foundation models and provide insights into the best practices for efficient training and inference. Overall the paper contributes to the development of video foundation models by providing a scalable and efficient pipeline for training and inference.

📅 Published on Mar 17, 2025

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2503.12964
• PDF: https://arxiv.org/pdf/2503.12964

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#VideoFoundationModels #NVIDIANeMo #VideoDatasetCuration #MultimodalLearning #VideoDiffusionModels