💡 The paper introduces OVO-S-Bench, a comprehensive benchmark for evaluating the ability of multimodal language models to understand spatial information from continuous video streams. The problem addressed is that existing benchmarks for spatial intelligence either evaluate models on full videos or focus on events rather than spatial structure, and do not account for the need to reason about places and layouts from partial information.

To address this, the authors created a benchmark consisting of 1680 human-annotated questions spanning 348 source videos, with each question having a query timestamp and an evidence interval. The questions cover four levels of abstraction, from basic perception to complex spatial reasoning and mapping. The annotation process involved 12 trained annotators who also served as cross-reviewers, ensuring high quality through multiple rounds of review.

The results show that even the best performing model, Gemini-3.1-Pro, trails human experts by 27 points, with the most challenging task being allocentric mapping. Interestingly, models that are specifically fine-tuned for streaming and spatial tasks actually perform worse than their original backbones, suggesting that these models may not be effectively using the spatial information in the video streams. The authors also found that using chain-of-thought reasoning can amplify spatial errors when the model is not grounded in the stream.

Overall, the OVO-S-Bench benchmark provides a challenging testbed for evaluating and improving the spatial intelligence of multimodal language models, and highlights the need for further research in this area to address the limitations of current models.