✨UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions

📝 Summary:
UniAVGen uses dual Diffusion Transformers and Asymmetric Cross-Modal Interaction for unified audio-video generation. This framework ensures precise spatiotemporal synchronization and semantic consistency. It outperforms existing methods in sync and consistency with far fewer training samples.

🔹 Publication Date: Published on Nov 5

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2511.03334
• PDF: https://arxiv.org/pdf/2511.03334
• Project Page: https://mcg-nju.github.io/UniAVGen/
• Github: https://mcg-nju.github.io/UniAVGen/

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#GenerativeAI #AudioVideoGeneration #DiffusionModels #CrossModalAI #DeepLearning

✨Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation

📝 Summary:
Ovi is a unified audio-video generation model using twin-DiT modules with blockwise cross-modal fusion. This innovative design ensures natural synchronization and high-quality multimodal outputs, simplifying previous multi-stage approaches.

🔹 Publication Date: Published on Sep 30

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2510.01284
• PDF: https://arxiv.org/pdf/2510.01284
• Project Page: https://aaxwaz.github.io/Ovi
• Github: https://github.com/character-ai/Ovi

🔹 Models citing this paper:
• https://huggingface.co/chetwinlow1/Ovi
• https://huggingface.co/rkfg/Ovi-fp8_quantized

✨ Spaces citing this paper:
• https://huggingface.co/spaces/akhaliq/Ovi
• https://huggingface.co/spaces/deddytoyota/Ovi
• https://huggingface.co/spaces/alexnasa/Ovi-ZEROGPU

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#AudioVideoGeneration #MultimodalAI #DeepLearning #CrossModalFusion #AIResearch

🔥 LTX-2: Efficient Joint Audio-Visual Foundation Model

💡 The paper introduces LTX-2, an open-source audiovisual diffusion model that generates synchronized video and audio content. The problem addressed is that current text-to-video diffusion models can generate compelling video sequences but lack the semantic, emotional, and atmospheric cues that audio provides. To solve this, the authors propose a dual-stream transformer architecture with cross-modal attention and classifier-free guidance. The model consists of a 14 billion parameter video stream and a 5 billion parameter audio stream, coupled through bidirectional audio-video cross-attention layers. This architecture enables efficient training and inference of a unified audiovisual model, with more capacity allocated for video generation than audio generation.

The method used to achieve this includes employing a multilingual text encoder for broader prompt understanding and introducing a modality-aware classifier-free guidance mechanism for improved audiovisual alignment and controllability. The model is trained using a combination of video and audio data, with temporal positional embeddings and cross-modality AdaLN for shared timestep conditioning.

The results show that LTX-2 achieves state-of-the-art audiovisual quality and prompt adherence among open-source systems, while delivering results comparable to proprietary models at a fraction of their computational cost and inference time. The model is capable of generating high-quality, temporally synchronized audiovisual content, including rich and coherent audio tracks that follow the characters, environment, style, and emotion of each scene. The model weights and code are publicly released, making it accessible for further research and development. Overall, LTX-2 provides a significant contribution to the field of audiovisual generation, enabling the creation of more realistic and engaging multimedia content.

📅 Published on Jan 6

🔗 Links:
• arXiv: https://arxiv.org/abs/2601.03233
• PDF: https://arxiv.org/pdf/2601.03233
• Project Page: https://app.ltx.studio/ltx-2-playground/i2v
• GitHub: https://github.com/Lightricks/LTX-2 ⭐ 6.4k

🤖 Models citing this paper:
• https://huggingface.co/Lightricks/LTX-2
• https://huggingface.co/Lightricks/LTX-2.3
• https://huggingface.co/unsloth/LTX-2.3-GGUF

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/linoyts/LTX-2-3-First-Last-Frame
• https://huggingface.co/spaces/linoyts/LTX-2-3-sync
• https://huggingface.co/spaces/linoyts/LTX-2-3-outpaint

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#AudioVisualLearning #MultimodalDiffusionModels #CrossModalAttention #AudioVideoGeneration #JointFoundationModels

🔥 UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating

💡 The paper presents UnityShots, a memory-driven audio-video generation system that can generate coherent multi-shot videos. The problem addressed is that existing approaches to generating multi-shot videos either cannot scale or do not maintain consistent subject appearance and audio across video cuts. To solve this, UnityShots uses a combination of fixed-size long-term and short-term memory slots, boundary-conditioned gates, and discrete cut-type priors to maintain consistency across shots. The system consists of two streams, video and audio, where the video stream uses two fixed-size slots to store information about the opening shot and the immediately preceding shot, and the audio stream uses a reference speaker token to preserve vocal timbre. The boundary-conditioned gate updates the memory slots at every cut, and the discrete cut-type prior allows for control over transition strength between shots. The system was trained on annotated cinematic and music-video shots and evaluated on a benchmark of 200 multi-cultural multi-shot sequences. The results show that UnityShots outperforms open-source baselines on every cross-shot coherence metric and matches the performance of the strongest closed-source system. The paper also releases a benchmark of multi-shot sequences with per-shot reference identities, reference audio, and per-boundary transition labels, which can be used for future research. Overall, UnityShots provides a new approach to generating coherent multi-shot videos that can maintain consistent subject appearance and audio across video cuts.

📅 Published on Jun 19

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.21661
• PDF: https://arxiv.org/pdf/2606.21661
• Project Page: https://jackailab.github.io/Projects/UnityShots/

📊 Datasets citing this paper:
• https://huggingface.co/datasets/KlingTeam/UnityShotsBench

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#AudioVideoGeneration #MultiShotVideo #BoundaryAwareGating #MemoryDrivenGeneration #CoherentVideoGeneration

🔥 OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation

💡 The paper introduces OmniVAE, a joint audio-video variational autoencoder that learns fine-grained semantic alignment between audio and video latent representations. Recent generative models have moved beyond silent video or standalone audio synthesis towards joint generation of synchronized audio and video, but this remains challenging due to the fundamental structural differences between the two modalities. Most existing methods use audio and video VAEs trained separately, resulting in a lack of cross-modal alignment, which leaves the downstream generative model to learn cross-modal synchronization from scratch.

OmniVAE addresses this issue by jointly training an audio-video VAE that captures temporal-semantic correspondence and aligns the two latent spaces. It uses a segment-level audio-video contrastive objective to achieve this alignment and also distills features from pre-trained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces.

The results of extensive experiments show that OmniVAE consistently improves the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation tasks. The findings underscore the importance of learning unified representations as a foundation for omnimodal modeling. Overall, OmniVAE provides a novel approach to joint audio-video generation, enabling more effective and synchronized generation of audio and video content.

📅 Published on Jul 26

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2607.23855
• PDF: https://arxiv.org/pdf/2607.23855
• Project Page: https://openmoss.ai/OmniVAE.github.io/

🤖 Models citing this paper:
• https://huggingface.co/OpenMOSS-Team/OmniVAE

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#AudioVideoGeneration #CrossModalAlignment #VariationalAutoencoders #MultimodalLearning #JointGenerationModels