Name: Video-based Reconstruction of 3D People Models - CVPR 2018 Spotlight @christiantheobalt1053
Uploaded: 2024-10-15T08:17:00-05:00
Duration: 8 min 17 s
Description: Video-based Reconstruction of 3D People Models - CVPR 2018 Spotlight

Video-based Reconstruction of 3D People Models - CVPR 2018 Spotlight

Christian Theobalt 2024-10-15 | Video-based Reconstruction of 3D People Models - CVPR 2018 Spotlight

HQ3DAvatar: High Quality Controllable 3D Head Avatar, TOG 2024

Christian Theobalt 2024-06-04 | Paper Abstract:
Multi-view volumetric rendering techniques have recently shown great potential in modeling and synthesizing high-quality head avatars. A comm- on approach to capture full head dynamic performances is to track the underlying geometry using a mesh-based template or 3D cube-based graphics primitives. While these model-based approaches achieve promising results, they often fail to learn complex geometric details such as the mouth interior, hair, and topological changes over time. This paper presents a novel approach to building highly photorealistic digital head avatars. Our method learns a canonical space via an implicit function parameterized by a neural network. It leverages multiresolution hash encoding in the learned feature space, allowing for high-quality, faster training and high-resolution rendering. At test time, our method is driven by a monocular RGB video. Here, an image encoder extracts face-specific features that also condition the learnable canonical space. This encourages deformation-dependent texture variations during training. We also propose a novel optical flow based loss that ensures correspondences in the learned canonical space, thus encouraging artifact-free and temporally consistent renderings. We show results on challenging facial expressions and show free-viewpoint renderings at interactive real-time rates for medium image resolutions. Our method outperforms the related approaches, both visually and numerically. We will release our multiple-identity dataset to encourage further research.

Reference Publication: Kartik Teotia, Mallikarjun B R, Xingang Pan, Hyeongwoo Kim, Pablo Garrido, Mohamed Elgharib, Christian Theobalt

HQ3DAvatar: High Quality Controllable 3D Head Avatar, ACM Transactions on Graphics, 2024.

Project Page: https://vcai.mpi-inf.mpg.de/projects/HQ3DAvatar/

Showcase of Holoported Characters (CVPR 2024): Real-time 3D Avatar of Christian Theobalt

Christian Theobalt 2024-04-26 | Short video of a real-time 3D avatar of Christian Theobalt controlled in real-time by himself.

The avatar was created with the Holoported Characters method that will be presented at CVPR 2024.

https://vcai.mpi-inf.mpg.de/projects/holochar/

This work was done at the MPI for Informatics and the Saarbruecken Research Center for Visual Computing, Interaction and Artificial Intellligence (MPI for Informatics / Google)

https://www.mpi-inf.mpg.de/departments/visual-computing-and-artificial-intelligence

https://www.via-center.science/

A. Shetty, M. Habermann, G. Sun, D. Luvizon, V. Golyanik and C. Theobalt
Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras, Computer Vision and Pattern Recognition (CVPR), 2024

VINECS: Video-based Neural Character Skinning. In CVPR, 2024

Christian Theobalt 2024-04-25 | Paper Abstract:
Rigging and skinning clothed human avatars is a challenging task and traditionally requires a lot of manual work and expertise. Recent methods addressing it either generalize across different characters or focus on capturing the dynamics of a single character observed under different pose configurations. However, the former methods typically pre- dict solely static skinning weights, which perform poorly for highly articulated poses, and the latter ones either re- quire dense 3D character scans in different poses or can- not generate an explicit mesh with vertex correspondence over time. To address these challenges, we propose a fully automated approach for creating a fully rigged character with pose-dependent skinning weights, which can be solely learned from multi-view video. Therefore, we first acquire a rigged template, which is then statically skinned. Next, a coordinate-based MLP learns a skinning weights field parameterized over the position in a canonical pose space and the respective pose. Moreover, we introduce our pose- and view-dependent appearance field allowing us to differentiably render and supervise the posed mesh using multi-view imagery. We show that our approach outperforms state-of- the-art while not relying on dense 4D scans.

Reference Publication: Zhouyingcheng Liao, Vladislav Golyanik, Marc Habermann, Christian Theobalt.

VINECS: Video-based Neural Character Skinning., CVPR, 2024.

Project Page: https://people.mpi-inf.mpg.de/~mhaberma/projects/2023-Vinecs/

ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering. In CVPR, 2024

Christian Theobalt 2024-04-25 | Paper Abstract:
Real-time rendering of photorealistic and controllable human avatars stands as a cornerstone in Computer Vision and Graphics. While recent advances in neural implicit rendering have unlocked unprecedented photorealism for digital avatars, real-time performance has mostly been demonstrated for static scenes only. To address this, we propose ASH, an Animatable Gaussian Splatting approach for photorealistic rendering of dynamic Humans in real time. We parameterize the clothed human as animatable 3D Gaussians, which can be efficiently splatted into image space to generate the final rendering. However, naively learning the Gaussian parameters in 3D space poses a severe challenge in terms of compute. Instead, we attach the Gaussians onto a deformable character model, and learn their parameters in 2D texture space, which allows leveraging efficient 2D convolutional architectures that easily scale with the required number of Gaussians. We benchmark ASH with competing methods on pose-controllable avatars, demonstrating that our method outperforms existing real-time methods by a large margin and shows comparable or even better results than offline methods.

Reference Publication: Haokai Pang, Heming Zhu, Adam Kortylewski, Christian Theobalt, Marc Habermann.

ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering., CVPR, 2024.

Project Page: https://vcai.mpi-inf.mpg.de/projects/ash/

ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis. In CVPR, 2024.

Christian Theobalt 2024-04-25 | Paper Abstract:
Gestures play a key role in human communication. Recent methods for co-speech gesture generation, while managing to generate beat-aligned motions, struggle generating gestures that are semantically aligned with the utterance. Compared to beat gestures that align naturally to the audio signal, semantically coherent gestures require modeling the complex interactions between the language and human motion, and can be controlled by focusing on certain words. Therefore, we present ConvoFusion, a diffusion-based approach for multi-modal gesture synthesis, which can not only generate gestures based on multi-modal speech inputs, but can also facilitate controllability in gesture synthesis. Our method proposes two guidance objectives that allow the users to modulate the impact of different conditioning modalities (e.g. audio vs text) as well as to choose certain words to be emphasized during gesturing. Our method is versatile in that it can be trained either for generating monologue gestures or even the conversational gestures. To further advance the research on multi-party interactive gestures, the DnD Group Gesture dataset is released, which contains 6 hours of gesture data showing 5 people interacting with one another. We compare our method with several recent works and demonstrate effectiveness of our method on a variety of tasks.

Reference Publication: Muhammad Hamza Mughal, Rishabh Dabral, Ikhsanul Habibie, Lucia Donatelli, Marc Habermann, Christian Theobalt.

ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis., CVPR, 2024.

Project Page: https://vcai.mpi-inf.mpg.de/projects/ConvoFusion/

Egocentric Whole Body Motion Capture with FisheyeViT and Diffusion Based Motion Refinement.

Christian Theobalt 2024-04-25 | Paper Abstract:
In this work, we explore egocentric whole-body motion capture using a single fisheye camera, which simultaneously estimates human body and hand motion. This task presents significant challenges due to three factors, the lack of high-quality datasets, fisheye camera distortion, and human body self-occlusion. To address these challenges, we propose a novel approach that leverages FisheyeViT to extract fisheye image features, which are subsequently converted into pixel-aligned 3D heatmap representations for 3D human body pose prediction. For hand tracking, we incorporate dedicated hand detection and hand pose estimation networks for regressing 3D hand poses. Finally, we develop a diffusion-based whole-body motion prior model to refine the estimated whole-body motion while accounting for joint uncertainties. To train these networks, we collect a large synthetic dataset, EgoWholeBody, comprising 840,000 high-quality egocentric images captured across a diverse range of whole-body motion sequences. Quantitative and qualitative evaluations demonstrate the effectiveness of our method in producing high-quality whole-body motion estimates from a single egocentric camera.

Reference Publication: Jian Wang, Zhe Cao, Diogo Luvizon, Lingjie Liu, Kripasindhu Sarkar, Danhang Tang, Thabo Beeler, Christian Theobalt.

Egocentric Whole Body Motion Capture with FisheyeViT and Diffusion Based Motion Refinement., CVPR, 2024.

Project Page: https://people.mpi-inf.mpg.de/~jianwang/projects/egowholemocap/index.html

3D Human Pose Perception from Egocentric Stereo Videos, In CVPR, 2024

Christian Theobalt 2024-04-25 | Paper Abstract:
While head-mounted devices are becoming more compact, they provide egocentric views with significant self-occlusions of the device user. Hence, existing methods often fail to accurately estimate complex 3D poses from egocentric views. In this work, we propose a new transformer-based framework to improve egocentric stereo 3D human pose estimation, which leverages the scene information and temporal context of egocentric stereo videos. Specifically, we utilize 1) depth features from our 3D scene reconstruction module with uniformly sampled windows of egocentric stereo frames, and 2) human joint queries enhanced by temporal features of the video inputs. Our method is able to accurately estimate human poses even in challenging scenarios, such as crouching and sitting. Furthermore, we introduce two new benchmark datasets, i.e., UnrealEgo2 and UnrealEgo-RW (RealWorld). The proposed datasets offer a much larger number of egocentric stereo views with a wider variety of human motions than the existing datasets, allowing comprehensive evaluation of existing and upcoming methods. Our extensive experiments show that the proposed approach significantly outperforms previous methods. We will release UnrealEgo2, UnrealEgo-RW, and trained models on our project page.

Reference Publication: Hiroyasu Akada, Jian Wang, Vladislav Golyanik, Christian Theobalt.
3D Human Pose Perception from Egocentric Stereo Videos., CVPR, 2024.

Project Page: https://4dqv.mpi-inf.mpg.de/UnrealEgo2/

EventEgo3D 3D Human Motion Capture from Egocentric Event Streams. In CVPR, 2024

Christian Theobalt 2024-04-25 | Paper Abstract:
Monocular egocentric 3D human motion capture is a challenging and actively researched problem. Existing methods use synchronously operating visual sensors (e.g. RGB cameras) and often fail under low lighting and fast motions, which can be restricting in many applications involving head-mounted devices. In response to the existing limitations, this paper 1) introduces a new problem, i.e. 3D human motion capture from an egocentric monocular event camera with a fisheye lens, and 2) proposes the first approach to it called EventEgo3D (EE3D). Event streams have high temporal resolution and provide reliable cues for 3D human motion capture under high-speed human motions and rapidly changing illumination. The proposed EE3D framework is specifically tailored for learning with event streams in the LNES representation, enabling high 3D reconstruction accuracy. We also design a prototype of a mobile head-mounted device with an event camera and record a real dataset with event observations and the ground-truth 3D human poses (in addition to the synthetic dataset). Our EE3D demonstrates robustness and superior 3D accuracy compared to existing solutions across various challenging experiments while supporting real-time 3D pose update rates of 140Hz.

Reference Publication: Christen Millerdurai, Hiroyasu Akada, Jian Wang, Diogo Luvizon, Christian Theobalt, Vladislav Golyanik .

EventEgo3D 3D Human Motion Capture from Egocentric Event Streams., CVPR, 2024.

Project Page: https://4dqv.mpi-inf.mpg.de/EventEgo3D/

Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras.

Christian Theobalt 2024-04-25 | Paper Abstract:
We present the first approach to render highly realistic free-viewpoint videos of a human actor in general apparel, from sparse multi-view recording to display, in real-time at an unprecedented 4K resolution. At inference, our method only requires four camera views of the moving actor and the respective 3D skeletal pose. It handles actors in wide clothing, and reproduces even fine-scale dynamic detail, e.g. clothing wrinkles, face expressions, and hand gestures. At training time, our learning-based approach expects dense multi-view video and a rigged static surface scan of the actor. Our method comprises three main stages. Stage 1 is a skeleton-driven neural approach for high-quality capture of the detailed dynamic mesh geometry. Stage 2 is a novel solution to create a view-dependent texture using four test-time camera views as input. Finally, stage 3 comprises a new image-based refinement network rendering the final 4K image given the output from the previous stages. Our approach establishes a new benchmark for real-time rendering resolution and quality using sparse input camera views, unlocking possibilities for immersive telepresence.

Reference Publication: Ashwath Shetty, Marc Habermann, Guoxing Sun, Diogo Luvizon, Vladislav Golyanik, Christian Theobalt.

Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras., CVPR, 2024.

Project Page: https://vcai.mpi-inf.mpg.de/projects/holochar/

ROAM: Robust and Object-aware Motion Generation using Neural Pose Descriptors, 3DV, 2024

Christian Theobalt 2024-01-16 | Paper Abstract:
Existing automatic approaches for 3D virtual character motion synthesis supporting scene interactions do not generalise well to new objects outside training distributions, even when trained on extensive motion capture datasets with diverse objects and annotated interactions. This paper addresses this limitation and shows that robustness and generalisation to novel scene objects in 3D object-aware character synthesis can be achieved by training a motion model with as few as one reference object. We leverage an implicit feature representation trained on object-only datasets, which encodes an SE(3)-equivariant descriptor field around the object. Given an unseen object and a reference pose-object pair, we optimise for the object-aware pose that is closest in the feature space to the reference pose. Finally, we use l-NSM, i.e., our motion generation model that is trained to seamlessly transition from locomotion to object interaction with the proposed bidirectional pose blending scheme. Through comprehensive numerical comparisons to state-of-the-art methods and in a user study, we demonstrate substantial improvements in 3D virtual character motion and interaction quality and robustness to scenarios with unseen objects.

Reference Publication: Wanyue Zhang, Rishabh Dabral, Thomas Leimkühler, Vladislav Golyanik, Marc Habermann, Christian Theobalt.
ROAM: Robust and Object-aware Motion Generation using Neural Pose Descriptors, International Conference on 3D Vision (3DV), 2024.

Project Page: https://vcai.mpi-inf.mpg.de/projects/ROAM/

SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes, 3DV, 2024

Christian Theobalt 2024-01-16 | Paper Abstract:
Existing methods for the 4D reconstruction of general, non-rigidly deforming objects focus on novel-view synthesis and neglect correspondences. However, time consistency enables advanced downstream tasks like 3D editing, motion analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. It then reconstructs the deformations of an estimated canonical model of the geometry and appearance in an online fashion. Since this canonical model is time-invariant, we obtain correspondences even for long-term, long-range motions. We employ neural scene representations to parametrize the components of our method. Like prior dynamic-NeRF methods, we use a backwards deformation model. We find non-trivial adaptations of this model necessary to handle larger motions: We decompose the deformations into a strongly regularized coarse component and a weakly regularized fine component, where the coarse component also extends the deformation field into the space surrounding the object, which enables tracking over time. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.

Reference Publication: Edith Tretschk, Vladislav Golyanik, Michael Zollhoefer, Aljaž Božič, Christoph Lassner, Christian Theobalt. SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes
International Conference on 3D Vision (3DV), 2024

Project Page: https://vcai.mpi-inf.mpg.de/projects/scenerflow/

Our Neural Rendering technology applied to The Matrix 4 movie data.

Christian Theobalt 2023-10-27 | The following shows results of an exciting experiment we did with our friends from Volucap in Berlin who did the special effects for
the movie Matrix Resurrections. Input to the result above was a multi-view video sequence filmed with a seven camera rig. We then applied our NR-NeRF algorithm to it to use neural rendering for creating a virtual bullet time camera path through the scene. Also, we could extract structure and motion information from the scene.
The shot was not used like this in the movie, but the result shows the exciting potential of our neural rendering techniques also in cutting-edge movie productions.
See also volucap.com/portfolio-items/the-matrix-resurrections

Decaf: Monocular Deformation Capture for Face and Hand Interactions. In SIGGRAPH ASIA, 2023.

Christian Theobalt 2023-09-28 | Paper Abstract:
We introduces the first single view motion capture method that regresses 3D hand and face motions along with deformations arising from their interactions. We model hands as articulated objects inducing non-rigid face deformations during an active interaction. Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system. As a pivotal step in its creation, we process the reconstructed raw 3D shapes with position-based dynamics and an approach for non-uniform stiffness estimation of the head tissues, which results in plausible annotations of the surface deformations, hand-face contact regions and head-hand positions. At the core of our neural approach are a variational auto-encoder supplying the hand-face depth prior and modules that guide the 3D tracking by estimating the contacts and the deformations. Our final 3D hand and face reconstructions are realistic and more plausible compared to several baselines applicable in our setting, both quantitatively and qualitatively.

Reference Publication:Soshi Shimada, Vladislav Golyanik, Patrick Pérez, Christian Theobalt: Decaf: Monocular Deformation Capture for Face and Hand Interactions. In SIGGRAPH ASIA 2023 (ACM TOG).

Project Page:https://vcai.mpi-inf.mpg.de/projects/Decaf/

NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction. In ICCV, 2023.

Christian Theobalt 2023-09-28 | Paper Abstract:
Recent methods for neural surface representation and rendering, for example NeuS, have demonstrated remarkably high-quality reconstruction of static scenes. However, the training of NeuS takes an extremely long time (8~hours), which makes it almost impossible to apply them to dynamic scenes with thousands of frames. We propose a fast neural surface reconstruction approach, called NeuS2, which achieves two orders of magnitude improvement in terms of acceleration without compromising reconstruction quality. To accelerate the training process, we integrate multi-resolution hash encodings into a neural surface representation and implement our whole algorithm in CUDA. We also present a lightweight calculation of second-order derivatives tailored to our networks (i.e., ReLU-based MLPs), which achieves a factor two speed up. To further stabilize training, a progressive learning strategy is proposed to optimize multi-resolution hash encodings from coarse to fine. In addition, we extend our method for reconstructing dynamic scenes with an incremental training strategy. Our experiments on various datasets demonstrate that NeuS2 significantly outperforms the state-of-the-arts in both surface reconstruction accuracy and training speed.

Reference Publication:
Yiming Wang, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, Lingjie Liu : NeuS2: Fast Learning of Neural Implicit Surfaces
for Multi-view Reconstruction. In ICCV, 2023.

Project Page:
https://vcai.mpi-inf.mpg.de/projects/NeuS2/

LiveHand: Real-time and Photorealistic Neural Hand Rendering. In ICCV, 2023.

Christian Theobalt 2023-09-28 | Paper Abstract:
The human hand is the main medium through which we interact with our surroundings. Hence, its digitization is of uttermost importance, with direct applications in VR/AR, gaming, and media production amongst other areas. While there are several works modeling the geometry of hands, little attention has been paid to capturing photo-realistic appearance. Moreover, for applications in extended reality and gaming, real-time rendering is critical. We present the first neural- implicit approach to photo-realistically render hands in real-time. This is a challenging problem as hands are textured and undergo strong articulations with pose-dependent effects. However, we show that this aim is achievable through our carefully designed method. This includes training on a low- resolution rendering of a neural radiance field, together with a 3D-consistent super-resolution module and mesh-guided sampling and space canonicaliza- tion. We demonstrate a novel application of perceptual loss on the image space, which is critical for learning details accurately. We also show a live demo where we photo-realistically render the human hand in real-time for the first time, while also modeling pose- and view-dependent appearance effects. We ablate all our design choices and show that they optimize for rendering speed and quality.

Reference Publication:
Akshay Mundra, Mallikarjun B R, Jiayi Wang, Marc Habermann, Christian Theobalt, Mohamed Elgharib : LiveHand: Real-time and Photorealistic Neural Hand Rendering. In ICCV, 2023.

Project Page:
https://vcai.mpi-inf.mpg.de/projects/LiveHand/

AvatarStudio: Text-driven Editing of 3D Dynamic Human Head Avatars. In SIGGRAPH ASIA 2023.

Christian Theobalt 2023-09-28 | Paper Abstract:
Capturing and editing full head performances enables the creation of virtual characters with various applications such as extended reality and media production. The past few years witnessed a steep rise in the photorealism of human head avatars. Such avatars can be controlled through different input data modalities, including RGB, audio, depth, IMUs and others. While these data modalities provide effective means of control, they mostly focus on editing the head movements such as the facial expressions, head pose and/or camera viewpoint. In this paper, we propose AvatarStudio, a text-based method for editing the appearance of a dynamic full head avatar. Our approach builds on existing work to capture dynamic performances of human heads using neural radiance field (NeRF) and edits this representation with a text-to-image diffusion model. Specifically, we introduce an optimization strategy for incorporating multiple keyframes representing different camera viewpoints and time stamps of a video performance into a single diffusion model. Using this personalized diffusion model, we edit the dynamic NeRF by introducing view-and-time-aware Score Distillation Sampling (VT-SDS) following a model-based guidance approach. Our method edits the full head in a canonical space, and then propagates these edits to remaining time steps via a pretrained deformation network. We evaluate our method visually and numerically via a user study, and results show that our method outperforms existing approaches. Our experiments validate the design choices of our method and highlight that our edits are genuine, personalized, as well as 3D- and time-consistent.

Reference Publication:
Mohit Mendiratta, Xingang Pan, Mohamed Elgharib, Kartik Teotia, Mallikarjun B R, Ayush Tewari, Vladislav Golyanik, Adam Kortylewski, Christian Theobalt : AvatarStudio: Text-driven Editing of 3D Dynamic Human Head Avatars. In SIGGRAPH ASIA 2023 (ACM TOG).

Project Page:
https://vcai.mpi-inf.mpg.de/projects/AvatarStudio/

HDHumans: A Hybrid Approach for High-fidelity Digital Humans, In SCA, 2023

Christian Theobalt 2023-07-03 | Paper Abstract:
Photo-real digital human avatars are of enormous importance in graphics, as they enable immersive communication over the globe, improve gaming and entertainment experiences, and can be particularly beneficial for AR and VR settings. However, current avatar generation approaches either fall short in high-fidelity novel view synthesis, generalization to novel motions, reproduction of loose clothing, or they cannot render characters at the high resolution offered by modern displays. To this end, we propose HDHumans, which is the first method for HD human character synthesis that jointly produces an accurate and temporally coherent 3D deforming surface and highly photo-realistic images of arbitrary novel views and of motions not seen at training time. At the technical core, our method tightly integrates a classical deforming character template with neural radiance fields (NeRF). Our method is carefully designed to achieve a synergy between classical surface deformation and NeRF. First, the template guides the NeRF, which allows synthesizing novel views of a highly dynamic and articulated char- acter and even enables the synthesis of novel motions. Second, we also leverage the dense pointclouds resulting from NeRF to further improve the deforming surface via 3D-to-3D supervision. We outperform the state of the art quantitatively and qualitatively in terms of synthesis quality.

Reference Publication:
M. Habermann, L. Liu, W. Xu, G. Pons-Moll, M. Zollhoefer, C.Theobalt. HDHumans: A Hybrid Approach for High-fidelity Digital Humans. In SCA, 2023.

Project Page:
https://people.mpi-inf.mpg.de/~mhaberma/projects/2023-hdhumans/

EgoLocate, In SIGGRAPH 2023.

Christian Theobalt 2023-05-30 | Paper Abstract:
Human and environment sensing are two important topics in Computer Vision and Graphics. Human motion is often captured by inertial sensors (left), while the environment is mostly reconstructed using cameras (right). We integrate the two techniques together in EgoLocate (middle), a system that simultaneously performs human motion capture (mocap), localization, and mapping in real time from sparse body-mounted sensors, including 6 inertial measurement units (IMUs) and a monocular phone camera. On one hand, inertial mocap suffers from large translation drift due to the lack of the global positioning signal. EgoLocate leverages image-based simultaneous localization and mapping (SLAM) techniques to locate the human in the reconstructed scene. On the other hand, SLAM often fails when the visual feature is poor. EgoLocate involves inertial mocap to provide a strong prior for the camera motion. Experiments show that localization, a key challenge for both two fields, is largely improved by our technique, compared with the state of the art of the two fields.

Reference Publication:
X. Yi, Y. Zhou, M. Habermann, V. Golyanik, S. Pan, C. Theobalt, F. Xu. EgoLocate Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensors. In SIGGRAPH, 2023.

Project Page:
xinyu-yi.github.io/EgoLocate/#

Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold. SIGGRAPH 2023

Christian Theobalt 2023-05-30 | Paper Abstract:
Synthesizing visual content that meets users' needs often requires flexible and precise controllability of the pose, shape, expression, and layout of the generated objects. Existing approaches gain controllability of generative adversarial networks (GANs) via manually annotated training data or a prior 3D model, which often lack flexibility, precision, and generality. In this work, we study a powerful yet much less explored way of controlling GANs, that is, to "drag" any points of the image to precisely reach target points in a user-interactive manner, as shown in Fig.1. To achieve this, we propose DragGAN, which consists of two main components including: 1) a feature-based motion supervision that drives the handle point to move towards the target position, and 2) a new point tracking approach that leverages the discriminative GAN features to keep localizing the position of the handle points. Through DragGAN, anyone can deform an image with precise control over where pixels go, thus manipulating the pose, shape, expression, and layout of diverse categories such as animals, cars, humans, landscapes, etc. As these manipulations are performed on the learned generative image manifold of a GAN, they tend to produce realistic outputs even for challenging scenarios such as hallucinating occluded content and deforming shapes that consistently follow the object's rigidity. Both qualitative and quantitative comparisons demonstrate the advantage of DragGAN over prior approaches in the tasks of image manipulation and point tracking. We also showcase the manipulation of real images through GAN inversion.

Reference Publication:
X. Pan, A. Tewari, T. Leimkühler, L. Liu, A. Meka, C. Theobalt. Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold. In SIGGRAPH 2023 Conference Proceedings

Project Page:
https://vcai.mpi-inf.mpg.de/projects/DragGAN/

General Neural Gauge Fields. In ICLR, 2023

Christian Theobalt 2023-04-20 | Paper Abstract:
The recent advance of neural fields, such as neural radiance fields, has significantly pushed the boundary of scene representation learning. Aiming to boost the computation efﬁciency and rendering quality of 3D scenes, a popular line of research maps the 3D coordinate system to another measuring system, e.g., 2D manifolds and hash tables, for modeling neural fields. The conversion of coordinate systems can be typically dubbed as gauge transformation, which is usually a pre-defined mapping function, e.g., orthogonal projection or spatial hash function. This begs a question: can we directly learn a desired gauge transformation along with the neural field in an end-to-end manner? In this work, we extend this problem to a general paradigm with a taxonomy of discrete and continuous cases, and develop an end-to-end learning framework to jointly optimize the gauge transformation and neural fields. To counter the problem that the learning of gauge transformations can collapse easily, we derive a general regularization mechanism from the principle of information conservation during the gauge transformation. To circumvent the high computation cost in gauge learning with regularization, we directly derive an information-invariant gauge transformation which allows to preserve scene information inherently and yield superior performance.

Reference Publication:
F. Zhan, L. Liu, A. Kortylewski , C. Theobalt. General Neural Gauge Fields. In ICLR, 2023

IMoS Intent Driven Full Body Motion Synthesis for Human Object Interactions. In EUROGRAPHICS, 2023.

Christian Theobalt 2023-04-20 | Paper Abstract:
Can we make virtual characters in a scene interact with their surrounding objects through simple instructions? Is it possible to synthesize such motion plausibly with a diverse set of objects and instructions? Inspired by these questions, we present the first framework to synthesize the full-body motion of virtual human characters performing specified actions with 3D objects placed within their reach. Our system takes as input textual instructions specifying the objects and the associated `intentions' of the virtual characters and outputs diverse sequences of full-body motions. This contrasts existing works, where full-body action synthesis methods generally do not consider object interactions and human-object interaction methods focus mainly on synthesizing hand or finger movements for grasping objects. We accomplish our objective by designing an intent-driven full-body motion generator, which uses a pair of decoupled conditional variational auto-regressors to learn the motion of the body parts in an autoregressive manner. We also optimize the 6DoF pose of the objects such that they plausibly fit within the hands of the synthesized characters. We compare our proposed method with the existing methods of motion synthesis and establish a new and stronger state-of-the-art for the task of intent-driven motion synthesis.

Reference Publication:
A. Ghosh, R. Dabral, V. Golyanik, C. Theobalt and P. Slusallek. IMoS: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions. EUROGRAPHICS 2023.

Scene Aware 3D Multi Human Motion Capture from a Single Camera. In EUROGRAPHICS, 2023

Christian Theobalt 2023-04-20 | Paper abstract:
In this work, we consider the problem of estimating the 3D position of multiple humans in a scene as well as their body shape and articulation from a single RGB video recorded with a static camera. In contrast to expensive marker-based or multi-view systems, our lightweight setup is ideal for private users as it enables an affordable 3D motion capture that is easy to install and does not require expert knowledge. To deal with this challenging setting, we leverage recent advances in computer vision using large-scale pre-trained models for a variety of modalities, including 2D body joints, joint angles, normalized disparity maps, and human segmentation masks. Thus, we introduce the first non-linear optimization-based approach that jointly solves for the absolute 3D position of each human, their articulated pose, their individual shapes as well as the scale of the scene. In particular, we estimate the scene depth and person unique scale from normalized disparity predictions using the 2D body joints and joint angles. Given the per-frame scene depth, we reconstruct a point-cloud of the static scene in 3D space. Finally, given the per-frame 3D estimates of the humans and scene point-cloud, we perform a space-time coherent optimization over the video to ensure temporal, spatial and physical plausibility. We evaluate our method on established multi-person 3D human pose benchmarks where we consistently outperform previous methods and we qualitatively demonstrate that our method is robust to in-the-wild conditions including challenging scenes with people of different sizes.

Reference for publication:
D. C. Luvizon, M. Habermann, V. Golyanik, A. Kortylewski and C. Theobalt. Scene-Aware 3D Multi-Human Motion Capture from a Single Camera. EUROGRAPHICS 2023.

EventNeRF: Neural Radiance Fields from a Single Colour Event Camera. In CVPR, 2023.

Christian Theobalt 2023-04-05 | Paper abstract:
Asynchronously operating event cameras find many applications due to their high dynamic range, vanishingly low motion blur, low latency and low data bandwidth. The field saw remarkable progress during the last few years, and existing event-based 3D reconstruction approaches recover sparse point clouds of the scene. However, such sparsity is a limiting factor in many cases, especially in computer vision and graphics, that has not been addressed satisfactorily so far. Accordingly, this paper proposes the first approach for 3D-consistent, dense and photorealistic novel view synthesis using just a single colour event stream as input. At its core is a neural radiance field trained entirely in a self-supervised manner from events while preserving the original resolution of the colour event channels. Next, our ray sampling strategy is tailored to events and allows for data-efficient training. At test, our method produces results in the RGB space at unprecedented quality. We evaluate our method qualitatively and numerically on several challenging synthetic and real scenes and show that it produces significantly denser and more visually appealing renderings than the existing methods. We also demonstrate robustness in challenging scenarios with fast motion and under low lighting conditions. We release the newly recorded dataset and our source code to facilitate the research field, see our project page link below.

Reference for publication:
V. Rudnev, M. Elgharib, C. Theobalt, V. Golyanik
EventNeRF: Neural Radiance Fields from a Single Colour Event Camera.
Computer Vision and Pattern Recognition (CVPR), 2023

Project page link:
https://4dqv.mpi-inf.mpg.de/EventNeRF/

MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis. In CVPR, 2023

Christian Theobalt 2023-03-31 | Paper abstract:
Conventional methods for human motion synthesis have either been deterministic or have had to struggle with the trade-off between motion diversity vs~motion quality. In response to these limitations, we introduce MoFusion, i.e., a new denoising-diffusion-based framework for high-quality conditional human motion synthesis that can synthesise long, temporally plausible, and semantically accurate motions based on a range of conditioning contexts (such as music and text). We also present ways to introduce well-known kinematic losses for motion plausibility within the motion diffusion framework through our scheduled weighting strategy. We also present ways to introduce well-known kinematic losses for motion plausibility within the motion diffusion framework through our scheduled weighting strategy. The learned latent space can be used for several interactive motion-editing applications like in-betweening, seed-conditioning, and text-based editing, thus, providing crucial abilities for virtual-character animation and robotics. Through comprehensive quantitative evaluations and a perceptual user study, we demonstrate the effectiveness of MoFusion compared to the state-of-the-art on established benchmarks in the literature.

Reference for publication:
R. Dabral, M. Hamza Mughal, V. Golyanik, C. Theobalt
MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis.
Computer Vision and Pattern Recognition (CVPR), 2023

Project page link:
https://vcai.mpi-inf.mpg.de/projects/MoFusion/

Scene-aware Egocentric 3D Human Pose Estimation. In CVPR, 2023.

Christian Theobalt 2023-03-30 | Paper abstract:
Egocentric 3D human pose estimation with a single head-mounted fisheye camera has recently attracted attention due to its numerous applications in virtual and augmented reality. Existing methods still struggle in challenging poses where the human body is highly occluded or is closely interacting with the scene. To address this issue, we propose a scene-aware egocentric pose estimation method that guides the prediction of the egocentric pose with scene constraints. To this end, we propose an egocentric depth estimation network to predict the scene depth map from a wide-view egocentric fisheye camera while mitigating the occlusion of the human body with a depth-inpainting network. Next, we propose a scene-aware pose estimation network that projects the 2D image features and estimated depth map of the scene into a voxel space and regresses the 3D pose with a V2V network. The voxel-based feature representation provides the direct geometric connection between 2D image features and scene geometry, and further facilitates the V2V network to constrain the predicted pose based on the estimated scene geometry. To enable the training of the aforementioned networks, we also generated a synthetic dataset, called EgoGTA, and an in-the-wild dataset based on EgoPW, called EgoPW-Scene. The experimental results of our new evaluation sequences show that the predicted 3D egocentric poses are accurate and physically plausible in terms of human-scene interaction, demonstrating that our method outperforms the state-of-the-art methods both quantitatively and qualitatively.

Reference for publication:
J. Wang, D. Luvizon, W. Xu, L. Liu, K. Sarkar, C. Theobalt
Scene-aware Egocentric 3D Human Pose Estimation.
Computer Vision and Pattern Recognition (CVPR), 2023

Project page link:
https://people.mpi-inf.mpg.de/~jianwang/projects/sceneego/

Estimating Egocentric 3D Human Pose in the Wild with External Weak Supervision. In CVPR, 2022.

Christian Theobalt 2022-07-27 | Paper abstract:
Egocentric 3D human pose estimation with a single fisheye camera has drawn a significant amount of attention recently. However, existing methods struggle with pose estimation from in-the-wild images, because they can only be trained on synthetic data due to the unavailability of large-scale in-the-wild egocentric datasets. Furthermore, these methods easily fail when the body parts are occluded by or interacting with the surrounding scene. To address the shortage of in-the-wild data, we collect a large-scale in-the-wild egocentric dataset called Egocentric Poses in the Wild (EgoPW). This dataset is captured by a head-mounted fisheye camera and an auxiliary external camera, which provides an additional observation of the human body from a third-person perspective during training. We present a new egocentric pose estimation method, which can be trained on the new dataset with weak external supervision. Specifically, we first generate pseudo labels for the EgoPW dataset with a spatio-temporal optimization method by incorporating the external-view supervision. The pseudo labels are then used to train an egocentric pose estimation network. To facilitate the network training, we propose a novel learning strategy to supervise the egocentric features with the high-quality features extracted by a pretrained external-view pose estimation model. The experiments show that our method predicts accurate 3D poses from a single in-the-wild egocentric image and outperforms the state-of-the-art methods both quantitatively and qualitatively.

Reference for publication:
J. Wang, L. Liu, W. Xu, K. Sarkar, D. Luvizon and C. Theobalt
Estimating Egocentric 3D Human Pose in the Wild with External Weak Supervision.
Computer Vision and Pattern Recognition (CVPR), 2022

φ-SfT: Shape-from-Template with a Physics-Based Deformation Model. In CVPR, 2022.

Christian Theobalt 2022-07-27 | Paper abstract:
Shape-from-Template (SfT) methods estimate 3D surface deformations from a single monocular RGB camera while assuming a 3D state known in advance (a template). This is an important yet challenging problem due to the underconstrained nature of the monocular setting. Existing SfT techniques predominantly use geometric and simpliﬁed deformation models, which often limits their reconstruction abilities. In contrast to previous works, this paper proposes a new SfT approach explaining 2D observations through physical simulations accounting for forces and material properties. Our differentiable physics simulator regularises the surface evolution and optimises the material elastic properties such as bending coefﬁcients, stretching stiffness and density. We use a differentiable renderer to minimise the dense reprojection error between the estimated 3D states and the input images and recover the deformation parameters using an adaptive gradient-based optimisation. For the evaluation, we record with an RGB-D camera challenging real surfaces exposed to physical forces with various material properties and textures. Our approach signiﬁcantly reduces the 3D reconstruction error compared to multiple competing methods.

Reference for publication:
N. Kairanda, E. Tretschk, M. Elgharib, C. Theobalt and V. Golyanik
φ-SfT: Shape-from-Template with a Physics-Based Deformation Model.
Computer Vision and Pattern Recognition (CVPR), 2022.

Advances in Neural Rendering - EUROGRAPHICS 2022 STAR Report

Christian Theobalt 2022-07-25 | Title: Advances in Neural Rendering. In Eurographics STAR 2022.

Paper abstract:
Synthesizing photo-realistic images and videos is at the heart of computer graphics and has been the focus of decades of research. Traditionally, synthetic images of a scene are generated using rendering algorithms such as rasterization or ray tracing, which take specifically defined representations of geometry and material properties as input. Collectively, these inputs define the actual scene and what is rendered, and are referred to as the scene representation (where a scene consists of one or more objects). Example scene representations are triangle meshes with accompanied textures (e.g., created by an artist), point clouds (e.g., from a depth sensor), volumetric grids (e.g., from a CT scan), or implicit surface functions (e.g., truncated signed distance fields). The reconstruction of such a scene representation from observations using differentiable rendering losses is known as inverse graphics or inverse rendering. Neural rendering is closely related, and combines ideas from classical computer graphics and machine learning to create algorithms for synthesizing images from real-world observations. Neural rendering is a leap forward towards the goal of synthesizing photo-realistic image and video content. In recent years, we have seen immense progress in this field through hundreds of publications that show different ways to inject learnable components into the rendering pipeline. This state-of-the-art report on advances in neural rendering focuses on methods that combine classical rendering principles with learned 3D scene representations, often now referred to as neural scene representations. A key advantage of these methods is that they are 3D-consistent by design, enabling applications such as novel viewpoint synthesis of a captured scene. In addition to methods that handle static scenes, we cover neural scene representations for modeling non-rigidly deforming objects and scene editing and composition. While most of these approaches are scene-specific, we also discuss techniques that generalize across object classes and can be used for generative tasks. In addition to reviewing these state-of-the-art methods, we provide an overview of fundamental concepts and definitions used in the current literature. We conclude with a discussion on open challenges and social implications.

Reference for publication:
A. Tewari, J. Thies, B. Mildenhall, P. Srinivasan, E. Tretschk, Y. Wang, C. Lassner, V. Sitzmann, R.M. Brualla, S. Lombardi, T. Simon, C. Theobalt, M. Nießner, J. Barron, G. Wetzstein, M. Zollhoefer and V. Golyanik
Advances in Neural Rendering.
Computer Graphics Forum (Eurographics STAR report), 2022.

EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream. In ICCV, 2021

Christian Theobalt 2022-06-01 | Paper abstract:
3D hand pose estimation from monocular videos is a long-standing and challenging problem, which is now seeing a strong upturn. In this work, we address it for the first time using a single event camera, i.e., an asynchronous vision sensor reacting on brightness changes. Our EventHands approach has characteristics previously not demonstrated with a single RGB or depth camera such as high temporal resolution at low data throughputs and real-time performance at 1000 Hz. Due to the different data modality of event cameras compared to classical cameras, existing methods cannot be directly applied to and re-trained for event streams. We thus design a new neural approach which accepts a new event stream representation suitable for learning, which is trained on newly-generated synthetic event streams and can generalise to real data. Experiments show that EventHands outperforms recent monocular methods using a colour (or depth) camera in terms of accuracy and its ability to capture hand motions of unprecedented speed. Our method, the event stream simulator and the dataset are publicly available.

Reference for publication:
V. Rudnev, V. Golyanik, J. Wang, H-P. Seidel, F. Mueller, M. Elgharib and C. Theobalt.
EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream.
International Conference on Computer Vision (ICCV) 2021.

link to publication website:
https://4dqv.mpi-inf.mpg.de/EventHands/

Gravity-Aware Monocular 3D Human-Object Reconstruction. In ICCV, 2021

Christian Theobalt 2022-06-01 | Paper abstract:
This paper proposes GraviCap, i.e., a new approach for joint markerless 3D human motion capture and object trajectory estimation from monocular RGB videos. We focus on scenes with objects partially observed during a free flight. In contrast to existing monocular methods, we can recover scale, object trajectories as well as human bone lengths in meters and the ground plane’s orientation, thanks to the awareness of the gravity constraining object motions. Our objective function is parametrized by the object’s initial velocity and position, gravity direction and focal length, and jointly optimised for one or several free flight episodes. The proposed human-object interaction constraints ensure geometric consistency of the 3D reconstructions and improved physical plausibility of human poses compared to the unconstrained case. We evaluate GraviCap on a new dataset with ground-truth annotations for persons and different objects undergoing free flights. In the experiments, our approach achieves state-of-the-art accuracy in 3D human motion capture on various metrics. We urge the reader to watch our supplementary video. Both the source code and the dataset are released.

Reference for publication:
R. Dabral, S. Shimada, A. Jain, C. Theobalt and V. Golyanik
Gravity-Aware Monocular 3D Human-Object Reconstruction.
In International Conference on Computer Vision (ICCV), 2021.

link to publication website:
https://4dqv.mpi-inf.mpg.de/GraviCap/

High-Fidelity Neural Human Motion Transfer from Monocular Video. In CVPR, 2021

Christian Theobalt 2022-06-01 | Paper abstract:
Video-based human motion transfer creates video animations of humans following a source motion. Current methods show remarkable results for tightly-clad subjects. However, the lack of temporally consistent handling of plausible clothing dynamics, including fine and high-frequency details, significantly limits the attainable visual quality. We address these limitations for the first time in the literature and present a new framework which performs high-fidelity and temporally-consistent human motion transfer with natural pose-dependent non-rigid deformations, for several types of loose garments. In contrast to the previous techniques, we perform image generation in three subsequent stages, synthesizing human shape, structure, and appearance. Given a monocular RGB video of an actor, we train a stack of recurrent deep neural networks that generate these intermediate representations from 2D poses and their temporal derivatives. Splitting the difficult motion transfer problem into subtasks that are aware of the temporal motion context helps us to synthesize results with plausible dynamics and pose-dependent detail. It also allows artistic control of results by manipulation of individual framework stages. In the experimental results, we significantly outperform the state-of-the-art in terms of video realism. Our code and data will be made publicly available.

Reference for publication:
M. Kappel, V. Golyanik, M. Elgharib, J-O. Henningson, H-P. Seidel, S. Castillo, C. Theobalt and M. Magnor
High-Fidelity Neural Human Motion Transfer from Monocular Video.
Computer Vision and Pattern Recognition (CVPR), 2021 (Oral).

link to publication website:
https://vcai.mpi-inf.mpg.de/projects/NHMT/

HumanGAN: A Generative Model of Humans Images. In 3DV, 2021

Christian Theobalt 2022-06-01 | Paper abstract:
Generative adversarial networks achieve great performance in photorealistic image synthesis in various domains, including human images. However, they usually employ latent vector inputs encoding sampled output images globally. This does not allow convenient control of semantically-relevant individual parts of the image, and is not able to draw samples that only differ in partial aspects, such asclothing style. We address these limitations and present a generative model for images of dressed humans offering control over pose, local body part appearance and garment style. This is the first method to solve various aspect of human image generation such as global appearance sampling, pose transfer, parts and garment transfer, and parts ampling jointly in an unified framework. As our model encodes part-based latent appearance vectors in a normalized pose-independent space and warps them to different poses, it preserves body and clothing appearance under varying posture. Experiments show that our flexible and general generative method outperforms task-specific baselines for pose-conditioned image generation, pose transfer and part sampling in terms of realism and output resolution.

Reference for publication:
K. Sarkar, L. Liu, V. Golyanik and C. Theobalt.
HumanGAN: A Generative Model of Humans Images.
International Conference on 3D Vision (3DV), 2021 (Oral), 2021.

link to publication website:
https://people.mpi-inf.mpg.de/~ksarkar/humangan/

Learning Complete 3D Morphable Face Models from Images and Videos. In CVPR, 2021.

Christian Theobalt 2022-06-01 | Paper abstract:
Most 3D face reconstruction methods rely on 3D morphable models, which disentangle the space of facial deformations into identity geometry, expressions and skin reflectance. These models are typically learned from a limited number of 3D scans and thus do not generalize well across different identities and expressions. We present the first approach to learn complete 3D models of face identity geometry, albedo and expression just from images and videos. The virtually endless collection of such data, in combination with our self-supervised learning-based approach allows for learning face models that generalize beyond the span of existing approaches. Our network design and loss functions ensure a disentangled parameterization of not only identity and albedo, but also, for the first time, an expression basis. Our method also allows for in-the-wild monocular reconstruction at test time. We show that our learned models better generalize and lead to higher quality image-based reconstructions than existing approaches.

Reference for publication:
M. B R, A. Tewari, H-P. Seidel, M. Elgharib and C. Theobalt
Learning Complete 3D Morphable Face Models from Images and Videos.
Computer Vision and Pattern Recognition (CVPR), 2021.

link to publication website:
https://vcai.mpi-inf.mpg.de/projects/LeMoMo/

Learning Meaningful Controls for Fluids. In SIGGRAPH, 2021.

Christian Theobalt 2022-06-01 | Paper abstract:
While modern fluid simulation methods achieve high-quality simulation results, it is still a big challenge to interpret and control motion from visual quantities, such as the advected marker density. These visual quantities play an important role in user interactions: Being familiar and meaningful to humans, these quantities have a strong correlation with the underlying motion. We propose a novel data-driven conditional adversarial model that solves the challenging, and theoretically ill-posed problem of deriving plausible velocity fields from a single frame of a density field. Besides density modifications, our generative model is the first to enable the control of the results using all of the following control modalities: obstacles, physical parameters, kinetic energy, and vorticity. Our method is based on a new conditional generative adversarial neural network that explicitly embeds physical quantities into the learned latent space, and a new cyclic adversarial network design for control disentanglement. We show the high quality and versatile controllability of our results for density-based inference, realistic obstacle interaction, and sensitive responses to modifications of physical parameters, kinetic energy, and vorticity.

Reference for publication:
M. Chu, N. Thuerey, H-P. Seidel, C. Theobalt and R. Zayer.
Learning Meaningful Controls for Fluids.
ACM Transactions on Graphics (Proc. of SIGGRAPH 2021).

link to publication website:
https://people.mpi-inf.mpg.de/~mchu/gvv-den2vel/den2vel.html

Monocular Real-time Full Body Capture with Inter-part Correlations. In CVPR, 2021

Christian Theobalt 2022-06-01 | Paper abstract:
We present the first method for real-time full body capture that estimates shape and motion of body and hands together with a dynamic 3D face model from a single color image. Our approach uses a new neural network architecture that exploits correlations between body and hands at high computational efficiency. Unlike previous works, our approach is jointly trained on multiple datasets focusing on hand, body or face separately, without requiring data where all the parts are annotated at the same time, which is much more difficult to create at sufficient variety. The possibility of such multi-dataset training enables superior generalization ability. In contrast to earlier monocular full body methods, our approach captures more expressive 3D face geometry and color by estimating the shape, expression, albedo and illumination parameters of a statistical face model. Our method achieves competitive accuracy on public benchmarks, while being significantly faster and providing more complete face reconstructions.

Reference for publication:
Y. Zhou, M. Habermann, I. Habibie, A. Tewari, C. Theobalt and F. Xu
Monocular Real-time Full Body Capture with Inter-part Correlations.
Computer Vision and Pattern Recognition (CVPR), 2021.

link to publication website:
https://people.mpi-inf.mpg.de/~mhaberma/projects/2021-cvpr-full-body-capture/

Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control. In SIGGRAPH Asia 2021

Christian Theobalt 2022-06-01 | Paper abstract:
We propose Neural Actor (NA), a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses. Our method is built upon recent neural scene representation and rendering works which learn representations of geometry and appearance from only 2D images. While existing works demonstrated compelling rendering of static scenes and playback of dynamic scenes, photo-realistic reconstruction and rendering of humans with neural implicit methods, in particular under user-controlled novel poses, is still difficult. To address this problem, we utilize a coarse body model as the proxy to unwarp the surrounding 3D space into a canonical pose. A neural radiance field learns pose-dependent geometric deformations and pose- and view-dependent appearance effects in the canonical space from multi-view video input. To synthesize novel views of high fidelity dynamic geometry and appearance, we leverage 2D texture maps defined on the body model as latent variables for predicting residual deformations and the dynamic appearance. Experiments demonstrate that our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses. Furthermore, our method also supports body shape control of the synthesized results.

Reference for publication:
L. Liu, M. Habermann, V. Rudnev, K. Sarkar, J. Gu and C. Theobalt
Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control.
ACM Transactions on Graphics (Proc. of SIGGRAPH Asia 2021).

link to publication website:
https://vcai.mpi-inf.mpg.de/projects/NeuralActor/

Non-Rigid Neural Radiance Fields. In ICCV, 2021

Christian Theobalt 2022-06-01 | Paper abstract:
We present Non-Rigid Neural Radiance Fields (NR-NeRF), a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. Our approach takes RGB images of a dynamic scene as input (e.g., from a monocular video recording), and creates a high-quality space-time geometry and appearance representation. We show that a single handheld consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views, e.g. a `bullet-time' video effect. NR-NeRF disentangles the dynamic scene into a canonical volume and its deformation. Scene deformation is implemented as ray bending, where straight rays are deformed non-rigidly. We also propose a novel rigidity network to better constrain rigid regions of the scene, leading to more stable results. The ray bending and rigidity network are trained without explicit supervision. Our formulation enables dense correspondence estimation across views and time, and compelling video editing applications such as motion exaggeration. Our code will be open sourced.

Reference for publication:
E. Tretschk, A. Tewari, V. Golyanik, M. Zollhoefer, C. Lassner and C. Theobalt.
Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video.
International Conference on Computer Vision (ICCV), 2021.

link to publication website:
https://vcai.mpi-inf.mpg.de/projects/nonrigid_nerf/

PhotoApp: Photorealistic Appearance Editing of Head Portraits. In SIGGRAPH, 2021

Christian Theobalt 2022-06-01 | Paper abstract:
Photorealistic editing of head portraits is a challenging task as humans are very sensitive to inconsistencies in faces. We present an approach for high-quality intuitive editing of the camera viewpoint and scene illumination (parameterised with an environment map) in a portrait image. This requires our method to capture and control the full reflectance field of the person in the image. Most editing approaches rely on supervised learning using training data captured with setups such as light and camera stages. Such datasets are expensive to acquire, not readily available and do not capture all the rich variations of in-the-wild portrait images. In addition, most supervised approaches only focus on relighting, and do not allow camera viewpoint editing. Thus, they only capture and control a subset of the reflectance field. Recently, portrait editing has been demonstrated by operating in the generative model space of StyleGAN. While such approaches do not require direct supervision, there is a significant loss of quality when compared to the supervised approaches. In this paper, we present a method which learns from limited supervised training data. The training images only include people in a fixed neutral expression with eyes closed, without much hair or background variations. Each person is captured under 150 one-light-at-a-time conditions and under 8 camera poses. Instead of training directly in the image space, we design a supervised problem which learns transformations in the latent space of StyleGAN. This combines the best of supervised learning and generative adversarial modeling. We show that the StyleGAN prior allows for generalisation to different expressions, hairstyles and backgrounds. This produces high-quality photorealistic results for in-the-wild images and significantly outperforms existing methods. Our approach can edit the illumination and pose simultaneously, and runs at interactive rates.

Reference for publication:
M. B R, A. Tewari, A. Dib, T. Weyrich, B. Bickel, H-P. Seidel H. Pfister, W. Matusik, L. Chevalier, M. Elgharib and C. Theobalt
PhotoApp: Photorealistic Appearance Editing of Head Portraits.
ACM Transactions on Graphics (Proc. of SIGGRAPH 2021).

link to publication website:
http://vcai.mpi-inf.mpg.de/projects/PhotoApp/

StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis. In ICLR, 2021

Christian Theobalt 2022-06-01 | Paper abstract:
We propose StyleNeRF, a 3D-aware generative model for photo-realistic high-resolution image synthesis with high multi-view consistency, which can be trained on unstructured 2D images. Existing approaches either cannot synthesize high-resolution images with fine details or yield noticeable 3D-inconsistent artifacts. In addition, many of them lack control over style attributes and explicit 3D camera poses. StyleNeRF integrates the neural radiance field (NeRF) into a style-based generator to tackle the aforementioned challenges, i.e., improving rendering efficiency and 3D consistency for high-resolution image generation. We perform volume rendering only to produce a low-resolution feature map and progressively apply upsampling in 2D to address the first issue. To mitigate the inconsistencies caused by 2D upsampling, we propose multiple designs, including a better upsampler and a new regularization loss. With these designs, StyleNeRF can synthesize high-resolution images at interactive rates while preserving 3D consistency at high quality. StyleNeRF also enables control of camera poses and different levels of styles, which can generalize to unseen views. It also supports challenging tasks, including zoom-in and-out, style mixing, inversion, and semantic editing.

Reference for publication:
J. Gu, L. Liu, P. Wang and C. Theobalt
StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis.
International Conference on Learning Representations (ICLR),2021.

link to publication website:
http://jiataogu.me/style_nerf

StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN. In BMVC, 2021

Christian Theobalt 2022-06-01 | Paper abstract:
Generative adversarial models (GANs) continue to produce advances in terms of the visual quality of still images, as well as the learning of temporal correlations. However, few works manage to combine these two interesting capabilities for the synthesis of video content: Most methods require an extensive training dataset in order to learn temporal correlations , while being rather limited in the resolution and visual quality of their output frames. In this paper, we present a novel approach to the video synthesis problem that helps to greatly improve visual quality and drastically reduce the amount of training data and resources necessary for generating video content. Our formulation separates the spatial domain, in which individual frames are synthesized, from the temporal domain, in which motion is generated. For the spatial domain we make use of a pre-trained StyleGAN network, the latent space of which allows control over the appearance of the objects it was trained for. The expressive power of this model allows us to embed our training videos in the StyleGAN latent space. Our temporal architecture is then trained not on sequences of RGB frames, but on sequences of StyleGAN latent codes. The advantageous properties of the StyleGAN space simplify the discovery of temporal correlations. We demonstrate that it suffices to train our temporal architecture on only 10 minutes of footage of 1 subject for about 6 hours. After training, our model can not only generate new portrait videos for the training subject, but also for any random subject which can be embedded in the StyleGAN space.

Reference for publication:
G. Fox , A. Tewari, M. Elgharib, G. Pons-Moll and C. Theobalt
StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN.
British Mashine Vision conference (BMVC), 2021.

link to publication website:
https://vcai.mpi-inf.mpg.de/projects/stylevideogan/

VideoForensicsHQ: Detecting High-quality Manipulated Face Videos. In ICME, 2021

Christian Theobalt 2022-06-01 | Paper abstract:
There are concerns that new approaches to the synthesis of high quality face videos may be misused to manipulate videos with malicious intent. The research community therefore developed methods for the detection of modified footage and assembled benchmark datasets for this task. In this paper, we examine how the performance of forgery detectors depends on the presence of artefacts that the human eye can see. We introduce a new benchmark dataset for face video forgery detection, of unprecedented quality. It allows us to demonstrate that existing detection techniques have difficulties detecting fakes that reliably fool the human eye. We thus introduce a new family of detectors that examine combinations of spatial and temporal features and outperform existing approaches both in terms of detection accuracy and generalization.

Reference for publication:
G. Fox, W. Liu, H. Kim, H-P. Seidel, M. Elgharib and C. Theobalt
VideoForensicsHQ: Detecting High-quality Manipulated Face Videos
International Conference on Multimedia and Expo (ICME), 2021.

link to publication website:
http://vcai.mpi-inf.mpg.de/projects/VForensicsHQ/

Estimating Egocentric 3D Human Pose in Global Space. ICCV 2021 (Oral).

Christian Theobalt 2022-05-24 | Paper abstract:
Egocentric 3D human pose estimation using a single fisheye camera has become popular recently as it allows capturing a wide range of daily activities in unconstrained environments, which is difficult for traditional outside-in motion capture with external cameras. However, existing methods have several limitations. A prominent problem is that the estimated poses lie in the local coordinate system of the fisheye camera, rather than in the world coordinate system, which is restrictive for many applications. Furthermore, these methods suffer from limited accuracy and temporal instability due to ambiguities caused by the monocular setup and the severe occlusion in a strongly distorted egocentric perspective. To tackle these limitations, we present a new method for egocentric global 3D body pose estimation using a single head-mounted fisheye camera. To achieve accurate and temporally stable global poses, a spatio-temporal optimization is performed over a sequence of frames by minimizing heatmap reprojection errors and enforcing local and global body motion priors learned from a mocap dataset. Experimental results show that our approach outperforms state-of-the-art methods both quantitatively and qualitatively.

Reference for publication:
J. Wang, L. Liu, W. Xu, K. Sarkar and C. Theobalt
Estimating Egocentric 3D Human Pose in Global Space.
International Conference on Computer Vision (ICCV), 2021 (Oral).

link to publication website:
https://vcai.mpi-inf.mpg.de/projects/globalegomocap/

Efficient and Differentiable Shadow Computation for Inverse Problems. In ICCV, 2021.

Christian Theobalt 2022-05-24 | Paper abstract:
Differentiable rendering has received increasing interest for image-based inverse problems. It can benefit traditional optimization-based solutions to inverse problems, but also allows for self-supervision of learning-based approaches for which training data with ground truth annotation is hard to obtain. However, existing differentiable renderers either do not model visibility of the light sources from the different points in the scene, responsible for shadows in the images, or are too slow for being used to train deep architectures over thousands of iterations. To this end, we propose an accurate yet efficient approach for differentiable visibility and soft shadow computation. Our approach is based on the spherical harmonics approximations of the scene illumination and visibility, where the occluding surface is approximated with spheres. This allows for a significantly more efficient shadow computation compared to methods based on ray tracing. As our formulation is differentiable, it can be used to solve inverse problems such as texture, illumination, rigid pose, and geometric deformation recovery from images using analysis-by-synthesis optimization.

Reference for publication:
L. Lyu, M. Habermann, L. Liu, M. B R, A. Tewari and C. Theobalt
Efficient and Differentiable Shadow Computation for Inverse Problems.
International Conference on Computer Vision (ICCV), 2021.

link to publication website:
https://vcai.mpi-inf.mpg.de/projects/DifferentiableShadows/

Deep Physics-aware Inference of Cloth Deformation for Monocular Human Performance Capture (3DV21)

Christian Theobalt 2022-05-24 | Recent monocular human performance capture approaches have shown compelling dense tracking results of the full body from a single RGB camera. However, existing methods either do not estimate clothing at all or model cloth deformation with simple geometric priors instead of taking into account the underlying physical principles. This leads to noticeable artifacts in their reconstructions, \eg baked-in wrinkles, implausible deformations that seemingly defy gravity, and intersections between cloth and body. To address these problems, we propose a person-specific, learning-based method that integrates a simulation layer into the training process to provide for the first time physics supervision in the context of weakly supervised deep monocular human performance capture. We show how integrating physics into the training process improves the learned cloth deformations, allows modeling clothing as a separate piece of geometry, and largely reduces cloth-body intersections. Relying only on weak 2D multi-view supervision during training, our approach leads to a significant improvement over current state-of-the-art methods and is thus a clear step towards realistic monocular capture of the entire deforming surface of a clothed human.

Y. Li, M. Habermann, B. Thomaszewski, S. Coros, T. Beeler and C. Theobalt
Deep Physics-aware Inference of Cloth Deformation for Monocular Human Performance Capture.
International Conference on 3D Vision (3DV), 2021.

link to publication website:
https://people.mpi-inf.mpg.de/~mhaberma/projects/2021-deepphyscloth/

Differentiable Event Stream Simulator for Non-Rigid 3D Tracking. In CVPRW, 2021.

Christian Theobalt 2022-05-24 | Paper abstract:
This paper introduces the first differentiable simulator of event streams, i.e., streams of asynchronous brightness change signals recorded by event cameras. Our differentiable simulator enables non-rigid 3D tracking of deformable objects (such as human hands, isometric surfaces and general watertight meshes) from event streams by leveraging an analysis-by-synthesis principle. So far, event-based tracking and reconstruction of non-rigid objects in 3D, like hands and body, has been either tackled using explicit event trajectories or large-scale datasets. In contrast, our method does not require any such processing or data, and can be readily applied to incoming event streams. We show the effectiveness of our approach for various types of non-rigid objects and compare to existing methods for non-rigid 3D tracking. In our experiments, the proposed energy-based formulations outperform competing RGB-based methods in terms of 3D errors. The source code and the new data are publicly available.

Reference for publication:
J. Nehvi, V. Golyanik, F. Mueller, H-P. Seidel, M. Elgharib and C. Theobalt
Differentiable Event Stream Simulator for Non-Rigid 3D Tracking.
Computer Vision and Pattern Recognition Workshops (Event-based Vision Workshop) (CVPRW), 2021.

link to publication website:
https://vcai.mpi-inf.mpg.de/projects/Event-based_Non-rigid_3D_Tracking/

Real-time Deep Dynamic Characters (SIGGRAPH 2021)

Christian Theobalt 2021-05-24 | We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery. In contrast to previous work, our controllable 3D character displays dynamics, e.g., the swing of the skirt, dependent on skeletal body motion in an efficient data-driven way, without requiring complex physics simulation. Our character model also features a learned dynamic texture model that accounts for photo-realistic motion-dependent appearance details, as well as view-dependent lighting effects. During training, we do not need to resort to difficult dynamic 3D capture of the human; instead we can train our model entirely from multi-view video in a weakly supervised manner. To this end, we propose a parametric and differentiable character representation which allows us to model coarse and fine dynamic deformations, e.g., garment wrinkles, as explicit space-time coherent mesh geometry that is augmented with high-quality dynamic textures dependent on motion and view point. As input to the model, only an arbitrary 3D skeleton motion is required, making it directly compatible with the established 3D animation pipeline. We use a novel graph convolutional network architecture to enable motion-dependent deformation learning of body and clothing, including dynamics, and a neural generative dynamic texture model creates corresponding dynamic texture maps. We show that by merely providing new skeletal motions, our model creates motion-dependent surface deformations, physically plausible dynamic clothing deformations, as well as video-realistic surface textures at a much higher level of detail than previous state of the art approaches, and even in real-time.

Marc Habermann, Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt, Real-time Deep Dynamic Characters, to appear ACM Transactions on Graphics (Proc. SIGGRAPH 2021)

Neural Monocular 3D Human Motion Capture With Physical Awareness (SIGGRAPH 2021)

Christian Theobalt 2021-05-04 | We present a new trainable system for physically-plausible markerless 3D human motion capture, which achieves state-of-the-art results in a broad range of challenging scenarios. Unlike most neural methods for human motion capture, our approach, which we dub “physionical”, is aware of physical and environmental constraints. It combines in a fully-differentiable way several key innovations,i.e., 1) a proportional-derivative controller, with gains predicted by a neural network, that reduces delays even in the presence of fast motions, 2) an explicit rigid body dynamics model and 3) a novel optimisation layer that prevents physically implausible foot-floor penetration as a hard constraint. The inputs to our system are 2D joint keypoints, which are canonicalised in a novel way so as to reduce the dependency on intrinsic camera parameters---both at train and test time. This enables more accurate global translation estimation without generalisability loss. Our model can be trained with 2D annotations only when the 3D annotations are not available. It produces smooth and physically-principled 3D motions in an interactive frame rate in a wide variety of challenging scenes, including newly recorded ones. Its advantages are especially noticeable on in-the-wild sequences that significantly differ from common 3D pose estimation benchmarks such as Human 3.6M and MPI-INF-3DHP. Qualitative results are provided in the supplementary video.

Neural Monocular 3D Human Motion Capture With Physical Awareness
S. Shimada, V. Golyanik, W. Xu, P. Pérez and C. Theobalt, ACM Transactions on Graphic (Proc. of SIGGRAPH), 2021.

http://gvv.mpi-inf.mpg.de/projects/PhysAware/

Real-time Global Illumination Decomposition of Videos (ACM TOG 2021)

Christian Theobalt 2021-02-12 | We propose the first approach for the decomposition of a monocular color
video into direct and indirect illumination components in real time. We
retrieve, in separate layers, the contribution made to the scene appearance
by the scene reflectance, the light sources and the reflections from various
coherent scene regions to one another. Existing techniques that invert global
light transport require image capture under multiplexed controlled lighting,
or only enable the decomposition of a single image at slow off-line frame
rates. In contrast, our approach works for regular videos and produces
temporally coherent decomposition layers at real-time frame rates. At the
core of our approach are several sparsity priors that enable the estimation
of the per-pixel direct and indirect illumination layers based on a small
set of jointly estimated base reflectance colors. The resulting variational
decomposition problem uses a new formulation based on sparse and dense
sets of non-linear equations that we solve efficiently using a novel alternating data-parallel optimization strategy. We evaluate our approach qualitatively and quantitatively, and show improvements over the state of the art in this field, in both quality and runtime. In addition, we demonstrate various real-time appearance editing applications for videos with consistent illumination.

ABHIMITRA MEKA, MOHAMMAD SHAFIEI, MICHAEL ZOLLHÖFER, CHRISTIAN RICHARDT, CHRISTIAN THEOBALT, Real-Time Global Illumination Decomposition of Videos, ACM Transactions on Graphics, 2021

i3DMM: Deep Implicit 3D Morphable Model of Human Heads

Christian Theobalt 2020-12-15 | We present the first deep implicit 3D morphable model (i3DMM) of full heads. Unlike earlier morphable face models it not only captures identity-specific geometry, texture, and expressions of the frontal face, but also models the entire head, including hair. We collect a new dataset consisting of 64 people with different expressions and hairstyles to train i3DMM. Our approach has the following favorable properties: (i) It is the first full head morphable model that includes hair. (ii) In contrast to mesh-based models it can be trained on merely rigidly aligned scans, without requiring difficult non-rigid registration. (iii) We design a novel architecture to decouple the shape model into an implicit reference shape and a deformation of this reference shape. With that, dense correspondences between shapes can be learned implicitly. (iv) This architecture allows us to semantically disentangle the geometry and color components, as color is learned in the reference space. Geometry is further disentangled as identity, expressions, and hairstyle, while color is disentangled as identity and hairstyle components. We show the merits of i3DMM using ablation studies, comparisons to state-of-the-art models, and applications such as semantic head editing and texture transfer. We will make our model publicly available.

Tarun Yenamandra, Ayush Tewari, Florian Bernard, Hans-Peter Seidel, Mohamed Elgharib, Daniel Cremers, Christian Theobalt, i3DMM: Deep Implicit 3D Morphable Model of Human Heads, arXiv

Pose-Guided Human Animation from a Single Image in the Wild

Christian Theobalt 2020-12-11 | We present a new pose transfer method for synthesizing a human animation from a single image of a person controlled by a sequence of body poses. Existing pose transfer methods exhibit significant visual artifacts when applying to a novel scene, resulting in temporal inconsistency and failures in preserving the identity and textures of the person. To address these limitations, we design a compositional neural network that predicts the silhouette, garment labels, and textures. Each modular network is explicitly dedicated to a subtask that can be learned from the synthetic data. At the
inference time, we utilize the trained network to produce a unified representation of appearance and its labels in UV coordinates, which remains constant across poses. The unified representation provides an incomplete yet strong guidance to generating the appearance in response to the pose change. We use the trained network to complete the appearance and render it with the background. With these strategies, we are able to synthesize human animations that can preserve the identity and appearance of the person in a temporally coherent way without any fine-tuning of the network on the testing scene. Experiments show that our method outperforms the state-of-the-arts in terms of synthesis quality, temporal coherence, and generalization ability.

Jae Shin Yoon, Lingjie Liu, Vladislav Golyanik, Kripasindhu Sarkar, Hyun Soo Park, and Christian Theobalt, Pose-Guided Human Animation from a Single Image in the Wild, Arxiv 2020