IMU2Face: Real-time Gesture-driven Facial Reenactment @MatthiasNiessner
IMU2Face: Real-time Gesture-driven Facial Reenactment  @MatthiasNiessner
Uploaded January 2018 | Updated November 2026, 1 day ago
Website: niessnerlab.org
Arxiv Paper: arxiv.org/abs/1801.01446

We present IMU2Face, a gesture-driven facial reenactment system. To this end, we combine recent advances in facial motion capture and inertial measurement units (IMUs) to control the facial expressions of a person in a target video based on intuitive hand gestures. IMUs are omnipresent, since modern smart-phones, smart-watches and drones integrate such sensors; e.g., for changing the orientation of the screen content, counting steps, or for flight stabilization. Face tracking and reenactment is based on the state-of-the-art real-time Face2Face facial reenactment system. Instead of transferring facial expressions from a source to a target actor, we employ an IMU to track the hand gestures of a source actor and use its orientation to modify the target actor's expressions.
IMU2Face: Real-time Gesture-driven Facial ReenactmentDeleted videoMotion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and TrackingDeleted videoTUM AI Lecture Series - Perceiving Humans in the 3D World (Angjoo Kanazawa)3D Scanning & Motion Capture: 7. Non-Rigid Tracking & ReconstructionLocal Implicit Grid Representations for 3D Scenes (CVPR 2020)What are Deep Fakes? How do we detect them? (CVPR2020 Media Forensics Workshop)Introduction to Deep Learning - 9. Convolutional Neural Networks (Summer 2020)Deleted videoLearning 3D Scene Priors with 2D Supervision (CVPR2023)Deleted video
Matthias Niessner |

IMU2Face: Real-time Gesture-driven Facial Reenactment

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER