💡 The Qwen Image VAE 2.0 technical report presents a high compression Variational Autoencoder suite that improves reconstruction fidelity and diffusability. The problem addressed in this paper is the reconstruction bottleneck of high compression in Variational Autoencoders. To solve this problem, the authors propose an improved architecture featuring Global Skip Connections and expanded latent channels. They also scale training to billions of images and incorporate a synthetic rendering engine to improve performance in text rich scenarios.

The method used in this paper involves implementing an enhanced semantic alignment strategy to make the latent space highly amenable to diffusion modeling. The authors also leverage an asymmetric and attention free encoder decoder backbone to minimize encoding overhead. The performance of Qwen Image VAE 2.0 is evaluated on public reconstruction benchmarks and a new benchmark called OmniDoc TokenBench, which is a collection of real world documents with specialized OCR based evaluation metrics.

The results show that Qwen Image VAE 2.0 achieves state of the art reconstruction performance, demonstrating exceptional capabilities in both general domains and text rich scenarios at high compression ratio. Downstream DiT experiments reveal that the models possess superior diffusability, significantly accelerating convergence compared to existing high compression baselines. Overall, Qwen Image VAE 2.0 establishes itself as a leading model with high compression, superior reconstruction, and exceptional diffusability.

💡 The paper introduces OmniVAE, a joint audio-video variational autoencoder that learns fine-grained semantic alignment between audio and video latent representations. Recent generative models have moved beyond silent video or standalone audio synthesis towards joint generation of synchronized audio and video, but this remains challenging due to the fundamental structural differences between the two modalities. Most existing methods use audio and video VAEs trained separately, resulting in a lack of cross-modal alignment, which leaves the downstream generative model to learn cross-modal synchronization from scratch.

OmniVAE addresses this issue by jointly training an audio-video VAE that captures temporal-semantic correspondence and aligns the two latent spaces. It uses a segment-level audio-video contrastive objective to achieve this alignment and also distills features from pre-trained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces.

The results of extensive experiments show that OmniVAE consistently improves the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation tasks. The findings underscore the importance of learning unified representations as a foundation for omnimodal modeling. Overall, OmniVAE provides a novel approach to joint audio-video generation, enabling more effective and synchronized generation of audio and video content.