💡 The paper introduces LTX-2, an open-source audiovisual diffusion model that generates synchronized video and audio content. The problem addressed is that current text-to-video diffusion models can generate compelling video sequences but lack the semantic, emotional, and atmospheric cues that audio provides. To solve this, the authors propose a dual-stream transformer architecture with cross-modal attention and classifier-free guidance. The model consists of a 14 billion parameter video stream and a 5 billion parameter audio stream, coupled through bidirectional audio-video cross-attention layers. This architecture enables efficient training and inference of a unified audiovisual model, with more capacity allocated for video generation than audio generation.

The method used to achieve this includes employing a multilingual text encoder for broader prompt understanding and introducing a modality-aware classifier-free guidance mechanism for improved audiovisual alignment and controllability. The model is trained using a combination of video and audio data, with temporal positional embeddings and cross-modality AdaLN for shared timestep conditioning.

The results show that LTX-2 achieves state-of-the-art audiovisual quality and prompt adherence among open-source systems, while delivering results comparable to proprietary models at a fraction of their computational cost and inference time. The model is capable of generating high-quality, temporally synchronized audiovisual content, including rich and coherent audio tracks that follow the characters, environment, style, and emotion of each scene. The model weights and code are publicly released, making it accessible for further research and development. Overall, LTX-2 provides a significant contribution to the field of audiovisual generation, enabling the creation of more realistic and engaging multimedia content.