✨CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Management

📝 Summary:
CurveStream enhances streaming video understanding in MLLMs via a curvature-aware hierarchical memory framework. It dynamically routes frames based on semantic intensity to prevent Out-of-Memory errors and achieve over 10 percent performance gains.

🔹 Publication Date: Published on Mar 20

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.19571
• PDF: https://arxiv.org/pdf/2603.19571
• Github: https://github.com/streamingvideos/CurveStream

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MLLMs #StreamingVideo #VideoUnderstanding #MemoryManagement #AI

✨StreamingClaw Technical Report

📝 Summary:
StreamingClaw is a unified framework for real-time streaming video understanding and embodied intelligence. It integrates real-time reasoning, multimodal long-term memory, and proactive interaction, enabling direct control of the physical world.

🔹 Publication Date: Published on Mar 23

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.22120
• PDF: https://arxiv.org/pdf/2603.22120

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#EmbodiedAI #VideoUnderstanding #RealTimeAI #Robotics #MultimodalAI

✨AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding

📝 Summary:
AdaptToken enables efficient long video understanding for MLLMs by using model uncertainty to dynamically select relevant tokens. It allocates a global token budget and supports early stopping, significantly improving accuracy and reducing inference time across benchmarks.

🔹 Publication Date: Published on Mar 30

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.28696
• PDF: https://arxiv.org/pdf/2603.28696
• Project Page: https://haozheqi.github.io/adapt-token
• Github: https://github.com/HaozheQi/AdaptToken

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MLLM #VideoUnderstanding #MachineLearning #AIResearch #TokenSelection

✨A Simple Baseline for Streaming Video Understanding

📝 Summary:
A simple sliding-window approach outperforms complex memory-based streaming video methods by using only recent frames. It demonstrates a trade-off between real-time perception and long-term memory, suggesting benchmarks should separate these abilities.

🔹 Publication Date: Published on Apr 2

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2511.16655
• PDF: https://arxiv.org/pdf/2604.02317
• Project Page: https://simple-stream.github.io/
• Github: https://simple-stream.github.io/

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#VideoUnderstanding #StreamingAI #ComputerVision #RealTimeAI #MachineLearning

✨AURA: Always-On Understanding and Real-Time Assistance via Video Streams

📝 Summary:
AURA is an end-to-end streaming visual interaction framework for continuous video understanding. It enables real-time question answering and proactive responses, improving on current VideoLLMs through integrated context management and optimized deployment.

🔹 Publication Date: Published on Apr 5

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2604.04184
• PDF: https://arxiv.org/pdf/2604.04184
• Project Page: https://aurateam2026.github.io
• Github: https://github.com/aurateam2026/AURA

🔹 Models citing this paper:
• https://huggingface.co/aurateam/AURA

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#VideoUnderstanding #RealTimeAI #VideoLLM #ComputerVision #DeepLearning

✨Watch Before You Answer: Learning from Visually Grounded Post-Training

📝 Summary:
VLMs struggle with video understanding due to text biases in benchmarks and training data. VidGround uses only visually grounded questions for post-training to eliminate these biases. This improves VLM performance and emphasizes the need for high-quality, visually grounded data.

🔹 Publication Date: Published on Apr 6

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2604.05117
• PDF: https://arxiv.org/pdf/2604.05117
• Project Page: http://vidground.etuagi.com
• Github: https://github.com/reacher-z/vidground

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#VLMs #VideoUnderstanding #AI #MachineLearning #ComputerVision

✨Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator

📝 Summary:
Uni-ViGU introduces a unified framework for video generation and understanding, uniquely building upon a video generator as its foundation. It uses unified flow matching and a bidirectional training mechanism to achieve competitive performance in both generation and understanding tasks.

🔹 Publication Date: Published on Apr 9

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2604.08121
• PDF: https://arxiv.org/pdf/2604.08121
• Project Page: https://fr0zencrane.github.io/uni-vigu-page/
• Github: https://fr0zencrane.github.io/uni-vigu-page/

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#VideoGeneration #VideoUnderstanding #DiffusionModels #AIResearch #DeepLearning

🔥 HumanNet: Scaling Human-centric Video Learning to One Million Hours

💡 The paper introduces HumanNet, a large-scale human-centric video dataset that captures how humans interact with the physical world, with the goal of advancing embodied intelligence. The problem addressed is the lack of large, diverse, and richly annotated human activity data, which hinders progress in learning physical interaction. To solve this, the authors created a one-million-hour video corpus that spans first-person and third-person perspectives, covering various activities, human-object interactions, and long-horizon behaviors in diverse environments. The dataset is annotated with interaction-centric information, including captions, motion descriptions, and hand and body-related signals.

The method involves a systematic data curation paradigm that treats human-centric filtering, temporal structuring, viewpoint diversity, and annotation enrichment as key design principles. This approach transforms unstructured internet video into a scalable substrate for representation learning, activity understanding, motion generation, and human-to-robot transfer.

The results show that HumanNet can be used to train vision-language-action models, and that egocentric human video can effectively replace robot data for training. In a controlled experiment, the authors found that continued training with 1000 hours of egocentric video from HumanNet surpassed the performance of continued training with 100 hours of real-robot data. This suggests that human-centric video can be a scalable and cost-effective substitute for robot data, and that HumanNet can be used to explore the opportunity to scale embodied foundation models using human-centric videos. Overall, the paper contributes a large-scale dataset and a systematic data curation paradigm that can advance embodied intelligence and learning physical interaction.

📅 Published on May 7

🔗 Links:
• arXiv: https://arxiv.org/abs/2605.06747
• PDF: https://arxiv.org/pdf/2605.06747
• Project Page: https://dagroup-pku.github.io/HumanNet/
• GitHub: https://github.com/DAGroup-PKU/HumanNet ⭐ 65

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#HumanCentricVideoLearning #EmbodiedIntelligence #LargeScaleVideoDatasets #HumanActivityRecognition #VideoUnderstanding

🔥 M^3Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

💡 The paper introduces M3Eval, a comprehensive evaluation framework and benchmark for assessing the memory capabilities of multi-modal models in video understanding systems. The problem addressed is that current multi-modal models have significant limitations in their memory capabilities, particularly in maintaining disentangled representations and demonstrating human-like interference patterns. To address this gap, the authors designed M3Eval, which is grounded in cognitive psychology and features carefully constructed tasks that isolate key aspects of memory.

The method involves conducting extensive experiments across representative multi-modal models using the M3Eval framework, which evaluates different memory dimensions such as what models retain, how faithfully information is preserved, and how robust memory remains under interference. The framework includes tasks that test the models' ability to maintain disentangled representations, exhibit human-like interference patterns, and demonstrate symbolic memory.

The results of the experiments reveal consistent weaknesses and distinctive behaviors in the models. The models struggle to maintain disentangled representations when processing parallel video streams, exhibit interference patterns that differ substantially from those observed in human memory, and ground memory sources more reliably in the spatial domain than the temporal domain. Additionally, the models demonstrate limited symbolic memory.

The paper's contributions include providing a valuable resource for future research in the form of the M3Eval benchmark and highlighting memory as a fundamental yet underexplored capability in multi-modal models. The findings offer insights for designing more effective memory mechanisms in multi-modal models, which can advance the field of video understanding systems. The code and dataset are made available to facilitate future research.

📅 Published on Jun 3

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.05008
• PDF: https://arxiv.org/pdf/2606.05008
• Project Page: https://pku-value-lab.github.io/m3eval-homepage/

📊 Datasets citing this paper:
• https://huggingface.co/datasets/PKU-VaLuE-Lab/m3eval

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#MultimodalLearning #VideoUnderstanding #CognitiveArchitectures #MemoryEvaluation #MultimodalModels

🔥 OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains

💡 The paper introduces a new dataset and method for improving audio-visual question answering systems. Current systems typically process videos in short clips and generate separate descriptions for audio and visual modalities, which can lead to inconsistent descriptions and a lack of cross-modal reasoning. To address this, the authors propose a two-part approach: entity-anchored video scripting, which transforms videos into structured scripts with summaries, main entity lists, and segment-wise audio-visual descriptions, and clue-guided QA generation, which prompts models to mine cross-segment clues from the script and generate QA pairs based on these clues.

The entity-anchored video scripting mechanism ensures cross-segment referential consistency and reconstructs audio-visual associations, while the clue-guided QA generation mechanism encourages models to generate questions that require long-term temporal connections and deep cross-modal reasoning. The authors use this pipeline to construct a new dataset called OmniVideo-100K, which consists of structured scripts and QA pairs, as well as a human-verified test set called OmniVideo-Test.

The results show that fine-tuning models on OmniVideo-100K yields significant performance gains, with improvements of up to 20.59% on the OmniVideo-Test set. The models also demonstrate strong generalization, with improvements of up to 12.64% on established benchmarks such as Daily-Omni and JointAVBench. Overall, the paper contributes a new dataset and method for improving audio-visual question answering systems, with a focus on cross-modal reasoning and temporal consistency.

📅 Published on Jun 12

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.14702
• PDF: https://arxiv.org/pdf/2606.14702
• Project Page: https://yzlmhzz.github.io/OmniVideo-100K/

📊 Datasets citing this paper:
• https://huggingface.co/datasets/MiG-NJU/OmniVideo-100K
• https://huggingface.co/datasets/MiG-NJU/OmniVideo-Test

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#AudioVisualReasoning #MultimodalLearning #VideoUnderstanding #CrossModalReasoning #AudioVisualQuestionAnswering