🔥 UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating

💡 The paper presents UnityShots, a memory-driven audio-video generation system that can generate coherent multi-shot videos. The problem addressed is that existing approaches to generating multi-shot videos either cannot scale or do not maintain consistent subject appearance and audio across video cuts. To solve this, UnityShots uses a combination of fixed-size long-term and short-term memory slots, boundary-conditioned gates, and discrete cut-type priors to maintain consistency across shots. The system consists of two streams, video and audio, where the video stream uses two fixed-size slots to store information about the opening shot and the immediately preceding shot, and the audio stream uses a reference speaker token to preserve vocal timbre. The boundary-conditioned gate updates the memory slots at every cut, and the discrete cut-type prior allows for control over transition strength between shots. The system was trained on annotated cinematic and music-video shots and evaluated on a benchmark of 200 multi-cultural multi-shot sequences. The results show that UnityShots outperforms open-source baselines on every cross-shot coherence metric and matches the performance of the strongest closed-source system. The paper also releases a benchmark of multi-shot sequences with per-shot reference identities, reference audio, and per-boundary transition labels, which can be used for future research. Overall, UnityShots provides a new approach to generating coherent multi-shot videos that can maintain consistent subject appearance and audio across video cuts.

📅 Published on Jun 19

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.21661
• PDF: https://arxiv.org/pdf/2606.21661
• Project Page: https://jackailab.github.io/Projects/UnityShots/

📊 Datasets citing this paper:
• https://huggingface.co/datasets/KlingTeam/UnityShotsBench

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#AudioVideoGeneration #MultiShotVideo #BoundaryAwareGating #MemoryDrivenGeneration #CoherentVideoGeneration