💡 The paper introduces HumanNet, a large-scale human-centric video dataset that captures how humans interact with the physical world, with the goal of advancing embodied intelligence. The problem addressed is the lack of large, diverse, and richly annotated human activity data, which hinders progress in learning physical interaction. To solve this, the authors created a one-million-hour video corpus that spans first-person and third-person perspectives, covering various activities, human-object interactions, and long-horizon behaviors in diverse environments. The dataset is annotated with interaction-centric information, including captions, motion descriptions, and hand and body-related signals.

The method involves a systematic data curation paradigm that treats human-centric filtering, temporal structuring, viewpoint diversity, and annotation enrichment as key design principles. This approach transforms unstructured internet video into a scalable substrate for representation learning, activity understanding, motion generation, and human-to-robot transfer.

The results show that HumanNet can be used to train vision-language-action models, and that egocentric human video can effectively replace robot data for training. In a controlled experiment, the authors found that continued training with 1000 hours of egocentric video from HumanNet surpassed the performance of continued training with 100 hours of real-robot data. This suggests that human-centric video can be a scalable and cost-effective substitute for robot data, and that HumanNet can be used to explore the opportunity to scale embodied foundation models using human-centric videos. Overall, the paper contributes a large-scale dataset and a systematic data curation paradigm that can advance embodied intelligence and learning physical interaction.