💡 The paper proposes SwiftI2V, an efficient framework for high resolution image to video generation. The problem addressed is that existing methods for generating high resolution videos from images are either computationally expensive or lack fidelity to the input image. High resolution image to video generation aims to synthesize realistic temporal dynamics while preserving fine grained appearance details of the input image, but at 2K resolution, this becomes extremely challenging. Existing solutions suffer from weaknesses such as high memory and latency costs, or hallucinating details and drifting from input specific local structures.

The proposed method, SwiftI2V, addresses these weaknesses by using a two stage design. First, it generates a low resolution motion reference to reduce token costs and ease the modeling burden. Then, it performs a strongly image conditioned 2K synthesis guided by the motion to recover input faithful details with controlled overhead. To make generation more scalable, SwiftI2V introduces Conditional Segment wise Generation, which synthesizes videos segment by segment with a bounded per step token budget. It also adopts bidirectional contextual interaction within each segment to improve cross segment coherence and input fidelity.

The results show that SwiftI2V achieves performance comparable to end to end baselines while reducing total GPU time by 202 times on the VBench I2V benchmark at 2K resolution. This enables practical 2K image to video generation on a single datacenter GPU or consumer GPU, making it a significant contribution to the field of image to video generation. Overall, SwiftI2V provides an efficient and scalable solution for high resolution image to video generation, addressing the efficiency fidelity dilemma and achieving state of the art results.

💡 The paper introduces LongCat-Video, a 13.6 billion parameter video generation model based on the Diffusion Transformer framework. The model is designed to generate high-quality long videos efficiently, which is a crucial step towards creating world models. LongCat-Video has a unified architecture that can perform multiple tasks, including text-to-video, image-to-video, and video continuation, using a single model.

The model achieves efficient long video generation through a coarse-to-fine generation strategy and block sparse attention, allowing it to generate 720p, 30fps videos within minutes. The coarse-to-fine generation strategy works by gradually increasing the resolution and detail of the video, both in terms of time and space. Block sparse attention is a technique that reduces the computational cost of the model by only attending to certain parts of the input data.

The model was trained using a multi-reward reinforcement learning from human feedback approach, which enables it to achieve performance comparable to state-of-the-art models. The use of multi-reward reinforcement learning from human feedback allows the model to learn from human evaluators and improve its performance over time.

The results show that LongCat-Video excels in generating high-quality long videos, maintaining temporal coherence and quality even in videos that are several minutes long. The model's efficiency and performance make it a significant contribution to the field of video generation, and the fact that the code and model weights are publicly available will accelerate progress in this area. Overall, LongCat-Video is a foundational model that takes an important step towards creating world models, which are complex models that can simulate and generate realistic videos and other types of data.