Abstract
This thesis studies audio-driven talking avatar generation, with a focus on producing temporally coherent, identity-preserving, and controllable talking face videos from speech and a single reference image. Although recent methods have improved audio-lip synchronization, they remain limited in preserving identity, modeling expressive motion beyond the mouth region, supporting controllable appearance editing, and scaling reliably to long-form or more expressive avatar generation.
To address these limitations, this thesis presents a sequence of diffusion-based frameworks that progressively expand the scope of audio-driven avatar synthesis. First, StableTalk adapts diffusion models to generate coherent facial dynamics beyond lip synchronization while preserving identity without explicit 3D supervision. Second, PortraitTalk introduces identity-aware conditioning for customizable and editable talking face generation from a small number of reference images. Third, MAGIC-Talk extends this setting to one-shot generation, improving temporal stability and reducing identity drift in long-form video synthesis. Finally, SyncDreamer generalizes the problem beyond head-centric animation to controllable and expressive upper-body and full-body avatar generation using multimodal inputs, including speech and text.
Extensive experiments on standard benchmarks show that the proposed methods consistently improve visual quality, identity fidelity, audio-visual synchronization, temporal coherence, and controllability, achieving state-of-the-art or competitive performance across diverse evaluation settings. While challenges remain in real-time generation and long-term temporal stability, the results demonstrate that diffusion-based generative frameworks can jointly support identity-consistent, expressive, and controllable audio-driven avatar synthesis. Overall, this thesis advances the state of the art in talking face and avatar generation by moving from lip-centric animation toward more realistic, customizable, and expressive digital humans.