🔥 Packing Input Frame Context in Next-Frame Prediction Models for Video Generation

💡 The paper introduces FramePack, a neural network designed to improve video generation by enhancing next-frame prediction models. The main problem addressed is the limitation of transformer context length, which restricts the number of frames that can be processed. To overcome this, FramePack compresses input frames, allowing the transformer context length to be fixed regardless of the video length. This enables the processing of a large number of frames and increases the batch size, making it comparable to image diffusion training.

The method proposed by FramePack involves compressing input frames and using an anti-drifting sampling method to generate frames in inverted temporal order. This approach helps to avoid exposure bias, which occurs when errors accumulate over iterations. Additionally, FramePack can be used to fine-tune existing video diffusion models, allowing for more balanced diffusion schedulers with less extreme flow shift timesteps.

The results show that FramePack improves the visual quality of video generation by supporting more balanced diffusion schedulers. The increased batch size and improved frame prediction also enhance the overall performance of video diffusion models. Overall, FramePack provides a novel approach to video generation by addressing the limitations of transformer context length and improving the efficiency of next-frame prediction models.

📅 Published on Apr 17, 2025

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2504.12626
• PDF: https://arxiv.org/pdf/2504.12626
• Project Page: https://lllyasviel.github.io/frame_pack_gitpage/

🤖 Models citing this paper:
• https://huggingface.co/URWAIFU/framepack-eichi-f1

📊 Datasets citing this paper:
• https://huggingface.co/datasets/agreeupon/wrkspace-backup-ttl

🚀 Spaces citing this paper:
• https://huggingface.co/spaces/linoyts/FramePack-F1
• https://huggingface.co/spaces/makululinux/FramePack-F1
• https://huggingface.co/spaces/ObiJuanCodenobi/VidGen-Emilio

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#VideoGenerationModels #NextFramePrediction #TransformerContextLength #FrameCompressionTechniques #NeuralNetworkArchitecture