💡 The paper introduces UniVidX, a unified multimodal framework for versatile video generation using video diffusion model priors. The problem with existing methods is that they train separate models for each task, limiting the modeling of correlations across different modalities. UniVidX addresses this issue by formulating pixel-aligned tasks as conditional generation in a shared multimodal space, allowing it to adapt to modality-specific distributions while preserving the native priors of the video diffusion model.

The framework consists of three key designs: Stochastic Condition Masking, Decoupled Gated LoRA, and Cross-Modal Self-Attention. Stochastic Condition Masking enables omni-directional conditional generation by randomly partitioning modalities into clean conditions and noisy targets during training. Decoupled Gated LoRA preserves the strong priors of the video diffusion model by introducing per-modality LoRAs that are activated when a modality serves as the generation target. Cross-Modal Self-Attention facilitates information exchange and inter-modal alignment by sharing keys and values across modalities while keeping modality-specific queries.

The authors instantiate UniVidX in two domains: UniVid-Intrinsic for RGB videos and intrinsic maps, and UniVid-Alpha for blended RGB videos and their constituent RGBA layers. The results show that both models achieve performance competitive with state-of-the-art methods across distinct tasks and generalize robustly to in-the-wild scenarios, even when trained on fewer than 1000 videos. Overall, UniVidX provides a unified framework for versatile video generation, allowing for more efficient and effective modeling of correlations across different modalities.

💡 The paper presents LoomVideo, a unified architecture for video generation and editing that efficiently handles multimodal inputs. The problem addressed is that existing unified frameworks for video generation and editing rely on large models with high computational overhead, typically requiring 13 billion parameters or more. These models often incorporate source video conditions for editing by concatenating sequence tokens, which doubles the sequence length and quadruples the computational complexity of the self-attention mechanism.

To address this issue, LoomVideo introduces a novel 5 billion parameter architecture that replaces the standard text encoder with a Multimodal Large Language Model and employs a Deepstack injection mechanism to align multi-layer features with the Diffusion Transformer. The key contribution is the introduction of a zero-overhead Scale-and-Add conditioning approach for video editing, which eliminates the need for token concatenation and reduces computational cost. This approach scales and directly adds the clean source video latent to the noised target latent, allowing for complex and non-rigid edits.

Additionally, the model incorporates a Negative Temporal RoPE strategy to handle multiple reference images. The results demonstrate that LoomVideo achieves state-of-the-art or highly competitive performance across comprehensive benchmarks, with exceptional performance in e-commerce and fashion generation scenarios. The zero-overhead conditioning mechanism enables LoomVideo to achieve at least a 5.41x acceleration in inference speed compared to models of similar capabilities, making it a highly practical and efficient video foundation model. Overall, LoomVideo presents a significant advancement in unified video generation and editing models, offering a more efficient and effective solution for handling multimodal inputs.