💡 The paper introduces a new paradigm for vision-language models, shifting from turn-based systems that require user prompting to a model that operates in real-time, making autonomous decisions about when to respond or delegate. The problem with current large models is that they only answer when addressed and do not interact in real-time, even in video-call apps. To address this, the authors propose a model that continuously watches what is happening and decides on its own whether to speak or stay silent.

The authors make two main contributions. First, they release JoyAI-VL-Interaction, an 8B-scale vision-first vision-language interaction model that makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model. The model excels at vision-triggered responsiveness and time awareness. They also provide a transferable training recipe that allows for capabilities to emerge that were not explicitly trained for, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck.

Second, they release a complete deployable system built around the model, which streams any ongoing video into the model, making it genuinely present in the world. The system has pluggable components, including ASR/TTS modules, memory, visualization UI, and a background brain that can connect to any API or agent.

The results show that human raters prefer JoyAI-VL-Interaction over in-app video-call assistants by a wide margin across six real-world scenarios. This is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system, making it a significant contribution to the field of interaction models.