💡 The paper introduces HumanNet, a large-scale human-centric video dataset that captures how humans interact with the physical world, with the goal of advancing embodied intelligence. The problem addressed is the lack of large, diverse, and richly annotated human activity data, which hinders progress in learning physical interaction. To solve this, the authors created a one-million-hour video corpus that spans first-person and third-person perspectives, covering various activities, human-object interactions, and long-horizon behaviors in diverse environments. The dataset is annotated with interaction-centric information, including captions, motion descriptions, and hand and body-related signals.

The method involves a systematic data curation paradigm that treats human-centric filtering, temporal structuring, viewpoint diversity, and annotation enrichment as key design principles. This approach transforms unstructured internet video into a scalable substrate for representation learning, activity understanding, motion generation, and human-to-robot transfer.

The results show that HumanNet can be used to train vision-language-action models, and that egocentric human video can effectively replace robot data for training. In a controlled experiment, the authors found that continued training with 1000 hours of egocentric video from HumanNet surpassed the performance of continued training with 100 hours of real-robot data. This suggests that human-centric video can be a scalable and cost-effective substitute for robot data, and that HumanNet can be used to explore the opportunity to scale embodied foundation models using human-centric videos. Overall, the paper contributes a large-scale dataset and a systematic data curation paradigm that can advance embodied intelligence and learning physical interaction.

💡 The paper introduces Cosmos 3, an omnimodal world model that can process and generate multiple data types, including language, image, video, audio, and action sequences, through a unified mixture-of-transformers architecture. This model is designed to jointly process and generate different modalities, effectively combining vision-language models, video generators, world simulators, and world-action models into a single framework. The Cosmos 3 model achieves state-of-the-art performance in various understanding and generation tasks, demonstrating its ability to serve as a scalable and general-purpose backbone for embodied agents.

The key contribution of the paper is the development of a unified architecture that can handle multiple input-output configurations, allowing for seamless integration of different modalities. The model is evaluated on a diverse suite of tasks, including text-to-image and image-to-video generation, and policy modeling, and achieves superior performance compared to existing models. The authors also make their code, model checkpoints, curated synthetic datasets, and evaluation benchmark available under an open-source license to accelerate research and deployment in Physical AI.

The results show that the Cosmos 3 model establishes a new state-of-the-art in various tasks, and its post-trained models were ranked as the best open-source models in text-to-image and image-to-video generation, as well as policy modeling. The availability of the code and model checkpoints is expected to facilitate further research and development in Physical AI, and the Cosmos 3 model has the potential to become a widely-used backbone for embodied agents. Overall, the paper presents a significant contribution to the field of Physical AI, demonstrating the effectiveness of omnimodal world models in achieving state-of-the-art performance in a wide range of tasks.

💡 The paper introduces Embodied-R1.5, a unified embodied foundation model that integrates various embodied reasoning capabilities, such as cognition, task planning, correction, and pointing, into a single architecture. The goal is to achieve general physical intelligence. To train the model, the authors developed three automated data construction pipelines, resulting in a large-scale data system of over 15 billion tokens. They also designed a multi-task balanced reinforcement learning approach to alleviate conflicts between different tasks.

The model consists of a Planner-Grounder-Corrector framework, which enables it to autonomously execute and self-correct over long-horizon tasks. With only 8 billion parameters, Embodied-R1.5 achieves state-of-the-art performance on 16 out of 24 embodied vision-language benchmarks, surpassing leading models. The model can also be fine-tuned into a vision-language agent with a small amount of data, outperforming leading models across popular manipulation benchmark suites.

The authors conducted extensive zero-shot real-robot experiments, demonstrating the model's strong generalization to the physical world. The experiments validated the model's performance in instruction following, affordance grounding, articulated object manipulation, and long-horizon complex tasks. The paper's contributions include the introduction of the Embodied-R1.5 model, the development of a large-scale data system, and the creation of an evaluation framework tailored for embodied tasks. The model weights, datasets, training code, and evaluation framework are open-sourced to facilitate future research in embodied foundation models.

The problem addressed in the paper is the development of a unified embodied foundation model that can achieve general physical intelligence. The method used to address this problem is the integration of various embodied reasoning capabilities into a single architecture, along with the development of a large-scale data system and a multi-task balanced reinforcement learning approach. The results show that Embodied-R1.5 achieves state-of-the-art performance on various benchmarks and demonstrates strong generalization to the physical world. Overall, the paper contributes to the development of embodied foundation models and has the potential to facilitate future research in this area.