🔥 TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training

💡 The paper presents TorchUMM, a unified codebase for evaluating and analyzing multimodal models that can understand, generate, and edit across visual and textual modalities. The development of unified multimodal models has been hindered by the diversity of model architectures and training paradigms, making it challenging to compare and analyze different models. To address this issue, TorchUMM provides a comprehensive framework that supports a wide range of models and tasks, including multimodal understanding, generation, and editing. The codebase includes a broad spectrum of models and integrates both established and novel datasets to evaluate perception, reasoning, compositionality, and instruction-following abilities. By providing a unified interface and standardized evaluation protocols, TorchUMM enables fair and reproducible comparisons across heterogeneous models, facilitating the development of more capable unified multimodal systems. The codebase is designed to foster deeper insights into the strengths and limitations of different models, and its availability is expected to accelerate the development of unified multimodal models. The TorchUMM codebase is available for use, allowing researchers to evaluate and analyze their models in a standardized and reproducible manner. Overall, the paper contributes to the development of unified multimodal models by providing a unified framework for evaluation, analysis, and post-training, which can help to advance the field of multimodal modeling.

📅 Published on Apr 12

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2604.10784
• PDF: https://arxiv.org/pdf/2604.10784
• Project Page: https://aifrontierlab.github.io/TorchUMM/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalModeling #UnifiedCodebase #MultimodalUnderstanding #VisualTextualModalities #MultimodalGeneration

🚀 Stop Maintaining Scrapers. Start Shipping Products.

Build AI products, not scraping infrastructure.

CoreClaw provides ready-to-use Workers & APIs for 1000+ websites — including Google Maps, Instagram, Facebook, YouTube, Amazon, Tiktok and Google Search Scraper.

✔️ No infrastructure
✔️ No proxy management
✔️ No scraper maintenance
✔️ JSON / CSV / REST API

🎁 Create a free account. Get free credits. Explore every Worker.

👉 https://coreclaw.com

🔥 HOMIE: Human-object Centric Video Personalization via Multimodal Intelligent Enchancement

💡 The paper proposes a framework called HOMIE for human-object centric video personalization, which is a core task in subject-driven video generation. Existing methods suffer from two key limitations: they struggle to balance high subject fidelity and accurate interaction patterns between humans and diverse objects, and they lack mechanisms to understand latent correspondence in intra-subject references. To address these challenges, HOMIE tackles both inter- and intra-subject input settings in a unified manner. The framework proposes a better integration strategy to extract knowledge of reference-level relationships without compromising the controllability of text encoders or incurring costly re-alignment. Specifically, it introduces global multimodal guidance within self-attention to better align semantic features derived from large language models with tokens from variational autoencoders. Additionally, it proposes modality-reference embedding to differentiate tokens from large language model features and variational autoencoder tokens and associate intra-subject reference image tokens. Extensive experiments validate that the method achieves state-of-the-art performance across various human-object centric video personalization tasks. The results demonstrate the effectiveness of HOMIE in addressing the limitations of existing methods and providing a unified framework for video personalization.

📅 Published on Jul 20

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.18217
• PDF: https://arxiv.org/pdf/2607.18217
• Project Page: https://yiyangcai.github.io/homie-page.github.io/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#HumanCentricVideoPersonalization #MultimodalIntelligentEnhancement #SubjectDrivenVideoGeneration #HumanObjectInteraction #VideoPersonalizationFrameworks

🔥 TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

💡 The paper introduces TimeLens2, a generalist video temporal grounding model that can predict variable-cardinality sets of evidence intervals across video lengths, domains, query forms, and viewpoints. Existing training strategies are misaligned with this set-valued task, often relying on brittle one-pass annotation or reinforcement learning that fails to distinguish non-overlapping predictions or requires fragile segment matching.

TimeLens2 treats temporal evidence as an interval set throughout supervision and optimization. It constructs reliable multi-span supervision through caption-derived proposals, independent localization, cross-agent consensus, semantic verification, and boundary refinement. The temporal Wasserstein reward computes exact one-dimensional distances between uniform distributions over merged interval supports, providing dense, matching-free feedback under unequal cardinalities and equivalent fragmentation. Temporal IoU complements it with precise overlap feedback.

The results show that TimeLens2 outperforms all size-matched baselines on every benchmark, while the 4B and 8B variants achieve state-of-the-art performance, surpassing open-source models with up to 397B parameters. The 2B, 4B, and 8B variants improve over their Qwen3-VL backbones by 14.2, 13.0, and 18.1 mIoU points, respectively. Overall, TimeLens2 demonstrates significant advancements in video temporal grounding, offering a robust and effective solution for this complex task.

📅 Published on Jul 19

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.17423
• PDF: https://arxiv.org/pdf/2607.17423
• Project Page: https://mcg-nju.github.io/TimeLens2/

🤖 Models citing this paper:
• https://huggingface.co/MCG-NJU/TimeLens2-8B
• https://huggingface.co/MCG-NJU/TimeLens2-4B
• https://huggingface.co/MCG-NJU/TimeLens2-2B

📊 Datasets citing this paper:
• https://huggingface.co/datasets/MCG-NJU/TimeLens2-93K

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/hugging-apps/timelens2-8b-temporal-grounding
• https://huggingface.co/spaces/hugging-apps/timelens2-2b-temporal-grounding

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#VideoTemporalGrounding #MultimodalLLMs #TemporalEvidenceLocalization #VideoAnalysisModels #GeneralistAIModels

🔥 Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

💡 The paper introduces Apple-PI, a benchmark for evaluating video models based on their ability to reason about physical laws. Current video models are often evaluated only on their output, without considering whether they arrive at that output through a faithful, law-grounded reasoning process. Apple-PI addresses this limitation by explicitly anchoring video model evaluation in physical laws.

The benchmark consists of three components: Orchard, a dataset of 400 videos covering ten canonical tasks in classical mechanics, which separates single-law tasks for confounder-free diagnosis from multi-law tasks for probing generalization; Benchmark Protocol, a three-stage protocol based on scientific reasoning, including Perception, Formulation, and Deduction, which uses chain-of-frames prompting on infographic-annotated first frames to treat the generated video as the model's visible reasoning trace; and Evaluation Suite, a hybrid evaluation suite that combines machine learning-based subjective scoring with physics-law-grounded objective measures.

The authors used Apple-PI to benchmark 11 video models and found that current models remain far from reliable law-grounded world simulators, with the best video models scoring only 0.473. The stage-, pillar-, and source-resolved analyses further exposed a Perception-to-Formulation-to-Deduction bottleneck, weak multi-law state transfer, and a persistent Sim-to-Real gap.

The paper positions Apple-PI as a diagnostic foundation for guiding future video models towards world models with law-grounded physical intelligence. The benchmark provides a comprehensive evaluation framework for video models, enabling researchers to identify areas for improvement and develop more robust and physically informed models.

📅 Published on Jul 17

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.16401
• PDF: https://arxiv.org/pdf/2607.16401
• Project Page: https://21yrm.github.io/Apple-PI-homepage

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/yaorunmao/apple-pi-leaderboard

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#VideoModelEvaluation #PhysicalIntelligence #LawGroundedReasoning #ClassicalMechanics #BenchmarkingAI

🔥 Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and Editing

💡 The paper introduces Mage-Flow, a compact 4B-scale generative stack for efficient text-to-image generation and instruction-based image editing. The stack is built from two co-designed components: Mage-VAE, a lightweight high-fidelity latent tokenizer, and a Native-Resolution Multimodal Diffusion Transformer trained with rectified flow matching. Mage-VAE uses one-step diffusion-style encoding and decoding with anchor-latent regularization, preserving the reconstruction quality of strong public VAEs while reducing tokenization cost by more than an order of magnitude.

The stack supports flexible-resolution training and improves end-to-end training throughput by about 2.5 times, thanks to native-resolution packing and stack-level CUDA kernel fusion. Built on this foundation, the authors develop a complete model family with Base, RL-aligned, and Turbo variants for both generation and editing. The results show that Mage-Flow and Mage-Flow-Edit achieve competitive performance across standard generation and editing benchmarks, despite their compact scale.

The Turbo variants make high-resolution generation and editing practical for interactive use, generating an image in 0.59 seconds and editing an image in 1.02 seconds at 1024^2 resolution on a single NVIDIA A100 GPU, while maintaining a small memory footprint. The results demonstrate that careful tokenizer-backbone-system co-design can deliver strong high-resolution generation and editing within an efficient 4B model family. The paper addresses the problem of large-scale visual generators being costly to train, fine-tune, and deploy, and provides a solution with Mage-Flow, which is a more efficient and compact alternative.

📅 Published on Jul 21

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.19064
• PDF: https://arxiv.org/pdf/2607.19064
• Project Page: https://microsoft.github.io/Mage/

🤖 Models citing this paper:
• https://huggingface.co/microsoft/Mage-Flow
• https://huggingface.co/microsoft/Mage-Flow-Edit-Turbo
• https://huggingface.co/microsoft/Mage-Flow-Base

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/microsoft/mage-flow
• https://huggingface.co/spaces/hugging-apps/mage-flow-base

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#ImageGeneration #TextToImageSynthesis #DiffusionTransformers #LatentTokenization #MultimodalLearning

🔥 AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report

💡 The paper presents AlayaWorld, an interactive long-horizon video world model that generates interactive environments from user inputs instantly. Unlike conventional video game development, which relies on labor-intensive pipelines for asset production, animation, physics, and programming, video world models like AlayaWorld enable the creation of customized, explorable, and continuously evolving virtual worlds from text, image, or video.

To achieve this, AlayaWorld requires four tightly coupled capabilities: interaction, persistent spatiotemporal consistency, stable long-horizon generation, and efficient response. The model is built on a 15B video diffusion transformer and generates 24-fps video at 540p and 720p. It produces short latent chunks autoregressively under camera trajectories and switchable text prompts.

The model's bounded visual context combines a persistent sink frame, compressed temporal history, geometry-aligned spatial memory, and recent-frame conditioning. To reduce long-term drift, the model is trained with corrupted histories and prediction residuals collected from its own rollouts. The authors also introduce a discrete autoregressive distillation formulation that combines distribution-matching distillation, self-forcing++, and consistency distillation, reducing inference from approximately 30 sampling steps to four steps per chunk.

The results show that AlayaWorld achieves the best performance over long-horizon generation on the iWorld-Bench dataset. The model is conceived as a full-stack, open-source, and long-term project, intended to provide an extensible foundation for future research on interactive video world models. Overall, AlayaWorld's contributions include its ability to generate interactive and stable long-horizon video, its efficient response to user inputs, and its potential to enable the creation of complex and evolving virtual worlds.

📅 Published on Jul 20

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.18367
• PDF: https://arxiv.org/pdf/2607.18367
• Project Page: https://alaya-lab.github.io/AlayaWorld/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#InteractiveWorldModeling #LongHorizonVideoGeneration #VideoWorldModels #VirtualWorldCreation #AIpoweredGameDevelopment

🔥 Masked Visual Actions for Unified World Modeling

💡 The paper introduces Masked Visual Actions for unified world modeling, a method that enables video models to learn how the visual world moves, interacts, and responds to contact, making them promising substrates for robotic world modeling. The central challenge addressed is how to communicate action to such models in a form aligned with the visual space in which they learned interaction priors, yet still grounded in physical manipulation.

The proposed method, Masked Visual Actions, expresses action as a partially revealed trajectory of an arbitrary entity in a video, using a pixel-space control interface. This allows the model to act as a forward dynamics model that predicts the scene's response to low-level robot actions, while also recovering robot behavior consistent with a desired outcome.

The method is fine-tuned with only 15 hours of masked examples from real videos and simulation, and achieves strong visual fidelity and controllability across diverse scenes and multiple embodiments. The model produces imagined rollouts whose outcomes correlate with real-world execution for policy evaluation, improves decision-making by ranking candidate futures in model-based planning, and supports inverse modeling by synthesizing robot motion from desired object motion.

The contributions of the paper include a novel method for communicating action to video models, a pixel-space control interface for expressing action, and a model that can predict scene responses to robot actions and recover robot behavior consistent with desired outcomes. The results demonstrate the effectiveness of the method in achieving strong visual fidelity and controllability, and its potential applications in downstream manipulation settings, such as policy evaluation, model-based planning, and inverse modeling.

📅 Published on Jul 21

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.19343
• PDF: https://arxiv.org/pdf/2607.19343
• Project Page: https://masked-visual-actions.github.io/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MaskedVisualActions #UnifiedWorldModeling #VideoModels #RoboticWorldModeling #ForwardDynamicsModel

🔥 Generative World Renderer at the Speed of Play

💡 The paper introduces Alaya Renderer Flash, a real-time generative forward world renderer that significantly improves the speed of the original Alaya Renderer. The original Alaya Renderer was too computationally expensive for real-time deployment, running at only 0.56 frames per second. In contrast, Alaya Renderer Flash achieves 31.54 frames per second, making it suitable for interactive world modeling and user-controllable play.

The key contribution of Alaya Renderer Flash is its reformulation of the original renderer as a few-step autoregressive streaming model, which enables efficient latent encoding and frame reconstruction. This approach preserves the scene structure without altering the underlying world dynamics, unlike models that generate frames from text or control hints. Additionally, Alaya Renderer Flash retains the teacher model's G-buffer and text-prompt interfaces while enabling continuous rendering over input streams of unbounded length.

The authors evaluate Alaya Renderer Flash on G-buffer streams across various metrics, including content preservation, temporal consistency, cross-window stability, prompt controllability, and runtime efficiency. The results show that Alaya Renderer Flash substantially reduces inference cost while preserving the core rendering capabilities of the teacher model. By integrating Alaya Renderer Flash with a physics engine, the authors build a fully playable generative world running at 30 frames per second, demonstrating the potential of this approach for real-time interactive applications. Overall, Alaya Renderer Flash offers a promising alternative path towards interactive world modeling and user-controllable play.

📅 Published on Jul 21

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.18703
• PDF: https://arxiv.org/pdf/2607.18703
• Project Page: https://alaya-renderer-flash.alayalab.ai/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#GenerativeRendering #RealTimeWorldModeling #AutoregressiveStreaming #ForwardRenderingTechniques #InteractiveWorldModeling

🚀 Stop Maintaining Scrapers. Start Shipping Products.

Build AI products, not scraping infrastructure.

CoreClaw provides ready-to-use Workers & APIs for 1000+ websites — including Google Maps, Instagram, Facebook, YouTube, Amazon, Tiktok and Google Search Scraper.

✔️ No infrastructure
✔️ No proxy management
✔️ No scraper maintenance
✔️ JSON / CSV / REST API

🎁 Create a free account. Get free credits. Explore every Worker.

👉 https://coreclaw.com