💡 The paper LongLive-RAG addresses the challenge of generating long videos using autoregressive video diffusion models. The problem with existing methods is that they use sliding-window attention, which can lead to error accumulation and identity drift over time. This is because once the active window accumulates appearance errors, subsequent generations can only condition on this degraded trajectory and drift further away. To overcome this limitation, the authors propose a retrieval-augmented generation framework called LongLive-RAG.

In this framework, previously generated latents are treated as a dynamic and searchable history. At each new block, LongLive-RAG uses a query embedding to retrieve relevant historical latents, allowing the generator to condition on non-local context instead of only the recent window. This retrieval step adds only a small overhead relative to generation and helps reduce error accumulation.

To make retrieval more discriminative, the authors introduce the Window Temporal Delta Loss, which suppresses redundant local similarity and encourages embeddings to capture meaningful temporal changes. The LongLive-RAG framework is general and can be used with multiple autoregressive backbones and generation lengths.

The experiments show that LongLive-RAG improves long video quality and achieves the best average VBench-Long rank. The authors claim that LongLive-RAG is the first method to formulate self-generated latent history as content-addressable retrieval memory, making it a significant contribution to the field of long video generation. The code for LongLive-RAG is available, making it possible for others to build upon and extend this work. Overall, the paper presents a novel approach to long video generation that addresses the limitations of existing methods and achieves state-of-the-art results.

💡 The paper proposes a new method called Self Gradient Forcing for native long video extrapolation. Recent autoregressive video diffusion methods are built upon Self Forcing, where the student is trained on histories produced by its own rollout rather than ground-truth video contexts. However, this approach has a limitation known as the historical context-gradient gap, where future losses cannot supervise how earlier generated latents should be written into more useful keys and values for later video-latent generation.

To address this issue, the authors propose a two-pass training strategy called Self Gradient Forcing. The first pass performs a no-gradient autoregressive rollout matching inference and records both the self-generated context and the noisy latents fed to the model at a sampled denoising exit step. The second pass performs parallel context-gradient reconstruction for the recorded exit step. The generated context is used as a stop-gradient clean-latent input, while the model recomputes the context KV representations and future-to-context causal attention.

The proposed method provides the missing memory-writing supervision within the native autoregressive training objective, using losses on future video latents to train the model to encode context into more effective causal memory. The authors evaluate their method across extensive long-horizon frame-wise and chunk-wise experiments under different initializations and achieve stronger native long-video extrapolation than Self Forcing, especially in subject identity, background/layout consistency, and temporal stability. Notably, using only a 5-second training window, Self Gradient Forcing can extrapolate to videos lasting several minutes.