Method Pipeline
Given an input audio \( \boldsymbol{A_{t}} \), we first train a network to predict the plausible human motion of face, hands, and body, as denoted as SMPL-X mesh \( \boldsymbol{M_t} \) for each frame. Here \( \boldsymbol{t} \) is the frame index. Subsequently, we optimize the UV texture map \( \boldsymbol{U_{tex}} \) for each character using differentiable rendering so that we can reconstruct all video frames given the texture map. Finally, we design a conditional GAN-based video generator \( \boldsymbol{G_{frame}} \) to synthesize the final video given the current 2D rendered image \( \boldsymbol{I_{t}} \), the normal map \( \boldsymbol{N_t} \), the depth map \( \boldsymbol{D_t} \), as well as 2D rendered images from nearby frames. The video generator is trained with a combination of reconstruction loss and GAN loss.
