🔥 Toward Native Multimodal Modeling: A Roadmap

💡 The paper presents a roadmap for native multimodal modeling, which integrates different modalities within a unified transformer framework, enabling seamless understanding and generation across diverse input-output configurations. Traditional approaches rely on late-fusion, where encoders and language backbones are assembled with output heads, but recent efforts have shifted towards native multimodal modeling for superior performance. However, the design space of native architectures remains poorly defined.

To address this, the authors formally define architectural nativity, distinguishing mid-fusion and early-fusion from non-native paradigms. They categorize existing native models into three categories: Multi-to-Text for cross-modal comprehension with text-only output, Multi-to-Target for scenario-oriented generation such as image, audio, and video generation, and Multi-to-Multi for unified modeling with symmetric input-output.

The authors provide a comprehensive investigation into the transition towards a definitive native multimodal modeling framework, where understanding and generation coexist within a unified transformer paradigm. They systematically examine the end-to-end pipeline, including architectural coordination, massive data curation, full-stack training recipes, inference and deployment, and comprehensive evaluation for truly native modeling.

The paper's contributions include a formalized roadmap for native multimodal modeling, a categorization of existing native models, and a comprehensive investigation into the transition towards a unified transformer framework. The results provide a foundation for the development of native multimodal models that can seamlessly understand and generate across diverse input-output configurations, representing a significant step towards world modeling and modality-agnostic reasoning.

📅 Published on May 25

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.25343
• PDF: https://arxiv.org/pdf/2605.25343
• Project Page: https://nmm-roadmap.github.io/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#NativeMultimodalModeling #MultimodalTransformerArchitectures #EarlyFusionTechniques #MidFusionApproaches #UnifiedTransformerFrameworks

🔥 Reinforcing Few-step Generators via Reward-Tilted Distribution Matching

💡 The paper proposes a new framework called Reward-Tilted Distribution Matching Distillation, or RTDMD, to improve the alignment of few-step image generation models with human preferences. The problem addressed is that current few-step diffusion distillation methods can generate images efficiently but struggle to align with human preferences. RTDMD is a two-stage approach that combines distribution matching distillation with reward-guided reinforcement learning.

In the first stage, the authors introduce Ambient-Consistent Distribution Matching Distillation, which performs distribution matching and uses a consistency regularizer to help the model track the generator distribution.

In the second stage, the authors jointly optimize two terms: a distribution matching term and a reward maximization term. They derive a hybrid policy gradient that combines a gradient-based estimator with direct reward backpropagation to reduce variance.

The authors also introduce step-subset GRPO to further reduce variance. The experiments demonstrate that RTDMD achieves state-of-the-art results across preference, aesthetic, and compositional metrics with only 4 inference steps, outperforming previous few-step text-to-image generation methods.

The RTDMD framework is tested on several datasets, including SD3, SD3.5, and FLUX.2, and the results show that it can generate high-quality images that align with human preferences. The code and models are made available for further research and development. Overall, the paper contributes a new framework for few-step image generation that can efficiently generate high-quality images that align with human preferences.

📅 Published on May 25

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.26108
• PDF: https://arxiv.org/pdf/2605.26108

🤖 Models citing this paper:
• https://huggingface.co/Harahan/FLUX2-4B-RTDMD
• https://huggingface.co/Harahan/SD35M-RTDMD

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#FewStepGenerators #RewardTiltedDistributionMatching #ImageGenerationModels #DiffusionDistillationMethods #ReinforcementLearningForGenerativeModels

🔥 Coloring the Noise: Adversarial Sobolev Alignment for Faithful Image Super Resolution

💡 The paper addresses the issue of spectral misalignment in image super resolution, which occurs when the generated images do not accurately represent the original image structure and details. This is often due to the limitations of generative models that prioritize isotropic objectives over the natural image manifold. The authors propose a new framework called Adversarial Sobolev Alignment for image Super Resolution, or ASASR, which leverages Riemannian geometry and adversarial training to improve the structural fidelity of the generated images.

The method involves recasting the generative flow into a Sobolev-induced Riemannian geometry, which allows the model to capture the natural spectral decay of images. This is achieved by explicitly coloring the noise transition kernel to mirror the natural spectral decay. The authors also integrate a parametric adversary that synthesizes targeted negative samples to direct optimization along the tangent space of plausible structural failures.

The results of the paper demonstrate that ASASR outperforms leading generative baselines in preserving spectral consistency and structural fidelity, and effectively mitigates artifacts. The evaluations show that ASASR is a robust solution that can accurately restore high-frequency details and maintain the natural image structure. Overall, the paper contributes a new framework for image super resolution that addresses the limitations of existing generative models and provides a more faithful restoration of images.

📅 Published on May 22

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.23264
• PDF: https://arxiv.org/pdf/2605.23264

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#AdversarialTraining #ImageSuperResolution #SobolevAlignment #RiemannianGeometry #FaithfulImageGeneration

🔥 SpatialBench: Is Your Spatial Foundation Model an All-Round Player?

💡 The paper introduces SpatialBench, a comprehensive benchmark for evaluating spatial foundation models across various domains and tasks. The goal is to assess whether these models can generalize robustly across different tasks, viewpoints, scene domains, input densities, and hardware constraints. Current models are mainly evaluated on specific domains they were designed for, which limits their assessment. SpatialBench addresses this gap by featuring 19 datasets and 546 scenes across 5 diverse spatial domains, evaluating 41 models across 6 paradigms on 5 task suites under 4 different input density settings.

The evaluation reveals that current models are not all-round players, and the authors identify key insights for future advancement. They find that full-context attention maximizes accuracy, while bounded-memory strategies enable long-sequence scalability. The authors also demonstrate that domain alignment and data quality are more critical to performance than dataset size. To address the largest data gap, they introduce a large-scale dataset, DA-Next-5M, and a strong baseline model, DA-Next, to advance spatial representation learning.

The paper's contributions include a holistic assessment of spatial foundation models, a comprehensive benchmark with unprecedented scale and rigorous design, and the introduction of a new dataset and model to push the boundaries of spatial representation learning. The results provide valuable insights for the development of more robust and generalizable spatial foundation models. Overall, the paper highlights the limitations of current models and provides a foundation for future research to create more all-round players in the field of spatial representation learning.

📅 Published on May 26

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.27367
• PDF: https://arxiv.org/pdf/2605.27367
• Project Page: https://ropedia.github.io/SpatialBench/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#SpatialFoundationModels #SpatialBench #GeospatialArtificialIntelligence #ComputerVisionBenchmarks #SpatialModelEvaluation

🔥 A Very Big Video Reasoning Suite

💡 The paper introduces a large scale video reasoning dataset and benchmark to study video intelligence capabilities beyond visual quality. The problem addressed is that current video models have focused on visual quality and their reasoning capabilities have been underexplored. Video reasoning involves understanding spatiotemporal structure such as continuity, interaction, and causality, which is essential for intelligent systems. However, the lack of large scale training data has hindered systematic study of video reasoning.

To address this gap, the authors introduce the Very Big Video Reasoning Dataset, which is an unprecedentedly large scale resource consisting of 200 curated reasoning tasks and over one million video clips. This dataset is approximately three orders of magnitude larger than existing datasets. The authors also present VBVR-Bench, a verifiable evaluation framework that incorporates rule-based, human-aligned scorers to enable reproducible and interpretable diagnosis of video reasoning capabilities.

The results of the study show early signs of emergent generalization to unseen reasoning tasks, indicating that the proposed dataset and benchmark can be used to develop more generalizable video reasoning models. The dataset, benchmark toolkit, and models are publicly available, laying a foundation for the next stage of research in generalizable video reasoning. The contributions of the paper are the introduction of a large scale video reasoning dataset and benchmark, and the demonstration of their effectiveness in studying video reasoning capabilities and enabling the development of more generalizable models.

📅 Published on Feb 23

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2602.20159
• PDF: https://arxiv.org/pdf/2602.20159
• Project Page: https://video-reason.com/

🤖 Models citing this paper:
• https://huggingface.co/Video-Reason/VBVR-Wan2.2
• https://huggingface.co/Video-Reason/VBVR-LTX2.3-diffsynth
• https://huggingface.co/Video-Reason/VBVR-Wan2.1-diffsynth

📊 Datasets citing this paper:
• https://huggingface.co/datasets/Video-Reason/VBVR-Dataset
• https://huggingface.co/datasets/Video-Reason/VBVR-Bench-Data
• https://huggingface.co/datasets/Video-Reason/video-mcp

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/Video-Reason/VBVR-Bench-Leaderboard

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#VideoIntelligence #VideoReasoning #SpatiotemporalAnalysis #CausalityInAI #ComputerVision

🔥 Geometry-Aware Representation Denoising for Robust Multi-view 3D Reconstruction

💡 The paper addresses the challenge of multi-view 3D reconstruction under degraded conditions, where real-world observations often contain degradations that differ significantly from ideal settings. Traditional feed-forward 3D reconstruction models are typically trained and evaluated under ideal conditions and are not robust to degradations. To improve robustness, the authors propose a novel framework called Geometry-Aware Representation Denoising, or GARD.

GARD is a diffusion-based framework that operates in the feature space of a 3D reconstructor, exploiting the geometry-aware feature representations to effectively recover accurate scene geometry. The framework also employs an additional RGB image decoder to restore high-quality RGB images, enabling the simultaneous recovery of 3D scene geometry and high-quality imagery.

The authors evaluate the effectiveness of the proposed GARD framework through comprehensive experiments on the Depth Anything 3 benchmark. The results demonstrate that GARD can restore both scene geometry and high-quality imagery from degraded inputs, outperforming traditional methods. The paper contributes a novel approach to robust multi-view 3D reconstruction, improving the accuracy and quality of 3D scene geometry and imagery under degraded conditions.

📅 Published on May 25

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.26230
• PDF: https://arxiv.org/pdf/2605.26230
• Project Page: https://cvlab-kaist.github.io/GARD/

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#GeometryAwareReconstruction #3DReconstructionRobustness #MultiViewGeometry #DiffusionBasedReconstruction #Robust3DSceneUnderstanding

🔥 CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents

💡 The paper addresses the problem of training computer-use agents using reinforcement learning with verifiable rewards, which is limited by the scarcity of scalable training data with deterministic rewards. To solve this, the authors propose CUA-Gym, a scalable pipeline that generates task instructions, environment states, and reward functions. The pipeline consists of a generator agent, a discriminator agent, and an orchestrator agent that work together to create high-quality training data. The generated data is then filtered using a combination of large language model majority voting and agent rollouts to ensure quality.

To further address the scarcity of training environments, the authors create CUA-Gym-Hub, a suite of high-fidelity mock web applications that mimic real-world software-use distributions. Using this pipeline, the authors construct a dataset of 32,112 verified training tuples grounded in 110 environments. They then train two models, CUA-Gym-A3B and CUA-Gym-A17B, using the dataset and achieve state-of-the-art performance on the OSWorld-Verified benchmark, with scores of 62.1% and 72.6% respectively.

The results demonstrate that the proposed pipeline and dataset can be used to train computer-use agents that outperform prior models at comparable scales. Additionally, the models show transferability beyond the training environments, as they also improve on the held-out WebArena benchmark. The authors plan to open-source the full synthesis pipeline, dataset, environments, and models, making it possible for others to build upon their work and further advance the field of computer-use agents. Overall, the paper presents a significant contribution to the field of reinforcement learning and computer-use agents, providing a scalable and effective way to train agents that can perform complex tasks in a variety of environments.

📅 Published on May 25

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.25624
• PDF: https://arxiv.org/pdf/2605.25624
• Project Page: https://cua-gym.xlang.ai

📊 Datasets citing this paper:
• https://huggingface.co/datasets/xlangai/CUA-Gym

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#ComputerUseAgents #VerifiableRewards #ReinforcementLearning #TaskInstructionGeneration #ScalableTrainingEnvironments

🔥 ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents

💡 The paper presents ProRL Agent, a scalable infrastructure for reinforcement learning training of multi-turn large language model agents. The problem addressed is the difficulty in generating and managing large numbers of sandboxed rollout trajectories required for reinforcement learning, which is a key component for improving the long-horizon behavior of these agents. Existing infrastructures often combine rollout orchestration with the training loop, making systems hard to migrate and maintain.

To solve this problem, the authors propose a rollout-as-a-service approach, where ProRL Agent serves the full agentic rollout lifecycle through an API service. This allows for decoupling rollout orchestration from the training loop, making the system more flexible and easier to maintain. Additionally, ProRL Agent provides standardized and extensible sandbox environments that support diverse agentic tasks in high-performance computing settings.

The authors validate ProRL Agent by applying it to reinforcement learning training on various tasks, including software engineering, math, STEM, and coding. The results demonstrate the effectiveness of ProRL Agent in supporting scalable and efficient reinforcement learning training. Furthermore, ProRL Agent is open-sourced and integrated as part of NVIDIA NeMo Gym, making it accessible to the research community. Overall, the paper contributes a scalable and flexible infrastructure for reinforcement learning training of multi-turn large language model agents, which can facilitate advancements in complex, interactive tasks.

📅 Published on Mar 19

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2603.18815
• PDF: https://arxiv.org/pdf/2603.18815

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#ReinforcementLearning #LargeLanguageModels #MultiTurnDialogue #RolloutOptimization #RLTrainingInfrastructure

Found an easy way to learn math for ML: Mathematics for Machine Learning 🎓📚

This is a curated collection on GitHub, including books, research papers, video lectures, and basic materials on math for studying and reviewing the mathematical foundations of machine learning. 📖📊

It helps build a stronger knowledge base by bringing together trusted resources around topics that machine learning engineers constantly encounter: linear algebra, mathematical analysis, probability theory, statistics, information theory, matrix calculus, and deep learning mathematics. 🧮🤖

Free public repository on GitHub. 💻✨

https://github.com/dair-ai/Mathematics-for-ML

#MachineLearning #Mathematics #DataScience #Learning #GitHub #AI

🔥 Beyond Mode Collapse: Distribution Matching for Diverse Reasoning

💡 The paper addresses the problem of mode collapse in on-policy reinforcement learning, where methods like GRPO concentrate probability mass on a single solution and cease exploring alternative strategies. This is due to the reverse KL minimization method used, which reinforces the first high-reward trajectory found rather than maintaining a distribution over multiple diverse solutions. To solve this problem, the authors propose DMPO, a distribution-matching policy optimization method that uses forward KL minimization to maintain solution diversity and improve performance in combinatorial optimization and reasoning tasks. DMPO constructs a target distribution over sampled trajectories proportional to their rewards and aligns the policy distribution to this target, providing mode-covering behavior without requiring sampling from the intractable global target distribution. The authors validate DMPO on NP-hard combinatorial optimization tasks and achieve significant improvements over GRPO, with a 43.9 percent quality ratio on text-based tasks and 43.1 percent on vision-based tasks. These gains generalize to mathematical reasoning and out-of-domain tasks, demonstrating that diversity-preserving training enhances general reasoning capabilities across modalities. The results show that DMPO achieves consistent quality improvements and sustained exploration across diverse reasoning tasks, establishing distribution matching as a practical approach to preventing mode collapse in on-policy reinforcement learning.

📅 Published on May 19

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2605.19461
• PDF: https://arxiv.org/pdf/2605.19461

📊 Datasets citing this paper:
• https://huggingface.co/datasets/OliverLee/NP_MM

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#ModeCollapseMitigation #DistributionMatching #OnPolicyReinforcementLearning #DiverseReasoningTasks #CombinatorialOptimizationTechniques