Supasorn Suwajanakorn | Synthesizing Obama: Learning Lip Sync from Audio @SupasornSuwajanakorn | Uploaded 7 years ago | Updated 3 hours ago
Synthesizing Obama: Learning Lip Sync from Audio
Supasorn Suwajanakorn, Steven M. Seitz, Ira Kemelmacher-Shlizerman
SIGGRAPH 2017
Given audio of President Barack Obama, we synthesize a high quality video of him speaking with accurate lip sync, composited into a target video clip. Trained on many hours of his weekly address footage, a recurrent neural network learns the mapping from raw audio features to mouth shapes. Given the mouth shape at each time instant, we synthesize high quality mouth texture, and composite it with proper 3D pose matching to change what he appears to be saying in a target video to match the input audio track
http://grail.cs.washington.edu/projects/AudioToObama/
Synthesizing Obama: Learning Lip Sync from Audio
Supasorn Suwajanakorn, Steven M. Seitz, Ira Kemelmacher-Shlizerman
SIGGRAPH 2017
Given audio of President Barack Obama, we synthesize a high quality video of him speaking with accurate lip sync, composited into a target video clip. Trained on many hours of his weekly address footage, a recurrent neural network learns the mapping from raw audio features to mouth shapes. Given the mouth shape at each time instant, we synthesize high quality mouth texture, and composite it with proper 3D pose matching to change what he appears to be saying in a target video to match the input audio track
http://grail.cs.washington.edu/projects/AudioToObama/
![[CVPR 2021] NeX: Real-time View Synthesis with Neural Basis Expansion](https://i.ytimg.com/vi/HyfkF7Z-ddA/hqdefault.jpg "[CVPR 2021] NeX: Real-time View Synthesis with Neural Basis Expansion
This is a supplementary video for
NeX: Real-time View Synthesis with Neural Basis Expansion
by Suttisak Wizadwongsa*, Pakkapon Phongthawee*, Jiraphon Yenphraphai*, Supasorn Suwajanakorn. (* first co-authors)
https://nex-mpi.github.io/
Abstract
We present NeX, a new approach to novel view synthesis based on enhancements of multiplane image (MPI) that can reproduce NeXt-level view-dependent effects in real time. Unlike traditional MPI that uses a set of simple RGBα planes, our technique models view-dependent effects by instead parameterizing each pixel as a linear combination of basis functions learned from a neural network. Moreover, we propose a hybrid implicit-explicit modeling strategy that improves upon fine detail and produces state-of-the-art results. Our method is evaluated on benchmark forward-facing datasets as well as our newly-introduced dataset designed to test the limit of view-dependent modeling with significantly more challenging effects such as rainbow reflections on a CD. Our method achieves the best overall scores across all major metrics on these datasets with more than 1000× faster rendering time than the state of the art.")
