Title of paper:
Audio-Visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generation
Authors:
Fa-Ting Hong, Zunnan Xu, Zixiang Zhou, Jun Zhou, Xiu Li, Qin Lin, Qinglin Lu, Dan Xu
Description:
This paper introduces ACTalker, an end-to-end video diffusion framework designed for natural talking head generation with both multi-signal and single-signal control capabilities.
The framework employs a parallel Mamba structure with multiple branches, each utilizing a separate driving signal to control specific facial regions.
A gate mechanism is applied across all branches, providing flexible control over video generation.
To ensure natural coordination of the controlled video both temporally and spatially, the Mamba structure enables driving signals to manipulate feature tokens across both dimensions in each branch.
Additionally, a mask-drop strategy is introduced, allowing each driving signal to independently control its corresponding facial region within the Mamba structure, preventing control conflicts.
Experimental results demonstrate that this method produces natural-looking facial videos driven by diverse signals, and that the Mamba layer seamlessly integrates multiple driving modalities without conflict.
Link of abstract paper:
https://arxiv.org/abs/2504.00000
Link of download paper:
https://arxiv.org/pdf/2504.00000.pdf
Code:
https://github.com/harlanhong/actalker
Datasets used in paper:
The paper does not specify the datasets used.
Hugging Face demo:
No Hugging Face demo available.
#ACTalker #TalkingHeadGeneration #VideoDiffusion #MultimodalControl #MambaStructure #DeepLearning #ComputerVision #AI #OpenSource

🔥 LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation

💡 The paper introduces LeapTalk, a novel framework for real-time talking head generation that breaks the latency-quality trade-off. The problem addressed is that current methods for talking head generation either suffer from high latency due to multi-step diffusion or error accumulation and identity drift when using real-time autoregressive approaches. To solve this, LeapTalk proposes a single-step bridge distillation scheme that enables stable and real-time talking head generation for arbitrarily long videos.

The method involves a data-to-data transport formulation based on a Brownian bridge, which mitigates identity drift and enhances long-term temporal stability. Additionally, a heterogeneous distillation framework with an SNR-aligned time transformation is used to enable smooth knowledge transfer from a pre-trained diffusion teacher to the student bridge model. An audio-driven classifier-free guidance mechanism is also proposed to maintain fine-grained lip synchronization.

The results show that LeapTalk achieves high-fidelity and temporally consistent video generation with only one step, reaching up to 200 frames per second. This significantly outperforms existing approaches in both efficiency and stability, demonstrating the effectiveness of the proposed framework for real-time talking head generation. Overall, LeapTalk provides a solution to the latency-quality trade-off, enabling high-quality and real-time talking head generation for long-form videos.

📅 Published on Jul 29

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2608.00079
• PDF: https://arxiv.org/pdf/2608.00079
• Project Page: https://zhangrongxiang.github.io/leaptalk-page/

🤖 Models citing this paper:
• https://huggingface.co/z-rx/leaptalk

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#TalkingHeadGeneration #RealTimeVideoProcessing #LatencyReductionTechniques #DeepLearningForVideo #ComputerVisionApplications