Cyber Valley Days 2024: A look back. A look forward. A call to action. Michael Black 2024-06-20 | This video provides a little history about the field of AI from 2009 to the present that follows the trajectory of the Max Planck Institute for Intelligent Systems and the development of Cyber Valley. It attempts to put AI developments into the context of the German economy and to lay out what needs to be done going forward. It describes the headwinds facing Germany in developing an AI economy and ends with a "call to action".
VAREN: Very Accurate and Realistic Equine Network (CVPR 2024, HD) Michael Black 2024-06-07 | Data-driven three-dimensional parametric shape models of the human body have gained enormous popularity both for the analysis of visual data and for the generation of synthetic humans. Following a similar approach for animals does not scale to the multitude of existing animal species, not to mention the difficulty of accessing subjects to scan in 3D. However, we argue that for domestic species of great importance, like the horse, it is a highly valuable investment to put effort into gathering a large dataset of real 3D scans, and learn a realistic 3D articulated shape model. We introduce VAREN, a novel 3D articulated parametric shape model learned from 3D scans of many real horses. VAREN bridges synthesis and analysis tasks, as the generated model instances have unprecedented realism, while being able to represent horses of different sizes and shapes. Differently from previous body models, VAREN has two resolutions, an anatomical skeleton, and interpretable, learned pose-dependent deformations, which are related to the body muscles. We show with experiments that this formulation has superior performance with respect to previous strategies for modeling pose-dependent deformations in the human body case, while also being more compact and allowing an analysis of the relationship between articulation and muscle deformation during articulated motion.Project: https://varen.is.tue.mpg.de/Code: github.com/silviazuffi/varenPaper: https://ps.is.mpg.de/uploads_file/attachment/attachment/764/zuffi_varen_CVPR2024.pdf@inproceedings{Zuffi:CVPR:2024, title = {{VAREN}: Very Accurate and Realistic Equine Network}, author = {Zuffi, Silvia and Mellbin, Ylva and Li, Ci and Hoeschle, Markus and Kjellström, Hedvig and Polikovsky, Senya and Hernlund, Elin and Black, Michael J.}, booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, pages = {}, month = Jun, year = {2024}, url = {https://varen.is.tue.mpg.de} }
VAREN: Very Accurate and Realistic Equine Network (CVPR 2024) Michael Black 2024-06-05 | Data-driven three-dimensional parametric shape models of the human body have gained enormous popularity both for the analysis of visual data and for the generation of synthetic humans. Following a similar approach for animals does not scale to the multitude of existing animal species, not to mention the difficulty of accessing subjects to scan in 3D. However, we argue that for domestic species of great importance, like the horse, it is a highly valuable investment to put effort into gathering a large dataset of real 3D scans, and learn a realistic 3D articulated shape model. We introduce VAREN, a novel 3D articulated parametric shape model learned from 3D scans of many real horses. VAREN bridges synthesis and analysis tasks, as the generated model instances have unprecedented realism, while being able to represent horses of different sizes and shapes. Differently from previous body models, VAREN has two resolutions, an anatomical skeleton, and interpretable, learned pose-dependent deformations, which are related to the body muscles. We show with experiments that this formulation has superior performance with respect to previous strategies for modeling pose-dependent deformations in the human body case, while also being more compact and allowing an analysis of the relationship between articulation and muscle deformation during articulated motion.Project: https://varen.is.tue.mpg.de/Code: github.com/silviazuffi/varenPaper: https://ps.is.mpg.de/uploads_file/attachment/attachment/764/zuffi_varen_CVPR2024.pdf@inproceedings{Zuffi:CVPR:2024, title = {{VAREN}: Very Accurate and Realistic Equine Network}, author = {Zuffi, Silvia and Mellbin, Ylva and Li, Ci and Hoeschle, Markus and Kjellström, Hedvig and Polikovsky, Senya and Hernlund, Elin and Black, Michael J.}, booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, pages = {}, month = Jun, year = {2024}, url = {https://varen.is.tue.mpg.de} }
AMUSE: Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion (CVPR 2024) Michael Black 2024-06-03 | Existing methods for synthesizing 3D human gestures from speech have shown promising results, but they do not explicitly model the impact of emotions on the generated gestures. Instead, these methods directly output animations from speech without control over the expressed emotion. To address this limitation, we present AMUSE, an emotional speech-driven body animation model based on latent diffusion. Our observation is that content (i.e., gestures related to speech rhythm and word utterances), emotion, and personal style are separable. To account for this, AMUSE maps the driving audio to three disentangled latent vectors: one for content, one for emotion, and one for personal style. A latent diffusion model, trained to generate gesture motion sequences, is then conditioned on these latent vectors. Once trained, AMUSE synthesizes 3D human gestures directly from speech with control over the expressed emotions and style by combining the content from the driving speech with the emotion and style of another speech sequence. Randomly sampling the noise of the diffusion model further generates variations of the gesture with the same emotional expressivity. Qualitative, quantitative, and perceptual evaluations demonstrate that AMUSE outputs realistic gesture sequences. Compared to the state of the art, the generated gestures are better synchronized with the speech content and better represent the emotion expressed by the input speech.Project: https://amuse.is.tue.mpg.de/Code: github.com/kiranchhatre/amusePaper: arxiv.org/abs/2312.04466@inproceedings{chhatre2023emotional, title = {{AMUSE}: Emotional Speech-driven {3D} Body Animation via Disentangled Latent Diffusion}, author = {Chhatre, Kiran and Daněček, Radek and Athanasiou, Nikos and Becherini, Giorgio and Peters, Christopher and Black, Michael J. and Bolkart, Timo}, booktitle = {Proceedings IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = jun, year = {2024}, eprint = {2312.04466}, url = {https://amuse.is.tue.mpg.de}, month_numeric = {6}}
SCULPT: Shape-Conditioned Unpaired Learning of Pose-dependent Clothed and Textured Human Meshes Michael Black 2024-05-15 | We present SCULPT, a novel 3D generative model for clothed and textured 3D meshes of humans. Specifically, we devise a deep neural network that learns to represent the geometry and appearance distribution of clothed human bodies. Training such a model is challenging, as datasets of textured 3D meshes for humans are limited in size and accessibility. Our key observation is that there exist medium-sized 3D scan datasets like CAPE, as well as large-scale 2D image datasets of clothed humans and multiple appearances can be mapped to a single geometry. To effectively learn from the two data modalities, we propose an unpaired learning procedure for pose-dependent clothed and textured human meshes. Specifically, we learn a pose-dependent geometry space from 3D scan data. We represent this as per vertex displacements w.r.t. the SMPL model. Next, we train a geometry conditioned texture generator in an unsupervised way using the 2D image data. We use intermediate activations of the learned geometry model to condition our texture generator. To alleviate entanglement between pose and clothing type, and pose and clothing appearance, we condition both the texture and geometry generators with attribute labels such as clothing types for the geometry, and clothing colors for the texture generator. We automatically generated these conditioning labels for the 2D images based on the visual question answering model BLIP and CLIP. We validate our method on the SCULPT dataset, and compare to state-of-the-art 3D generative models for clothed human bodies.arXiv: arxiv.org/pdf/2308.10638v2Code: github.com/soubhiksanyal/SCULPT_releaseProject: https://sculpt.is.tue.mpg.de/Citation:@inproceedings{SCULPT:CVPR:2024, title = {{SCULPT}: Shape-Conditioned Unpaired Learning of Pose-dependent Clothed and Textured Human Meshes}, author = {Sanyal, Soubhik and Ghosh, Partha and Yang, Jinlong and Black, Michael J. and Thies, Justus and Bolkart, Timo}, booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)} , year = {2024},}
[CVPR24 Highlight] HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects Michael Black 2024-05-06 | HOLD is the first method that jointly reconstructs articulated hands and objects from monocular videos without assuming a pre-scanned object template and 3D hand-object training data.See more details at: github.com/zc-alexfan/holdAbstract:Since humans interact with diverse objects every day, the holistic 3D capture of these interactions is important to understand and model human behaviour. However, most existing methods for hand-object reconstruction from RGB either assume pre-scanned object templates or heavily rely on limited 3D hand-object data, restricting their ability to scale and generalize to more unconstrained interaction settings. To this end, we introduce HOLD -- the first category-agnostic method that reconstructs an articulated hand and object jointly from a monocular interaction video. We develop a compositional articulated implicit model that can reconstruct disentangled 3D hand and object from 2D images. We also further incorporate hand-object constraints to improve hand-object poses and consequently the reconstruction quality. Our method does not rely on 3D hand-object annotations while outperforming fully-supervised baselines in both in-the-lab and challenging in-the-wild settings. Moreover, we qualitatively show its robustness in reconstructing from in-the-wild videos.Authors: Zicong Fan, Maria Parelli, Maria Eleni Kadoglou, Muhammed Kocabas, Xu Chen, Michael J. Black, Otmar HilligesarXiv: arxiv.org/abs/2311.18448project: zc-alexfan.github.io/hold
BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion (CVPR 2023) Michael Black 2023-06-19 | We show, for the first time, that neural networks trained only on synthetic data achieve state-of-the-art accuracy on the problem of 3D human pose and shape (HPS) estimation from real images. Previous synthetic datasets have been small, unrealistic, or lacked realistic clothing. Achieving sufficient realism is non-trivial and we show how to do this for full bodies in motion. Specifically, our BEDLAM dataset contains monocular RGB videos with ground-truth 3D bodies in SMPL-X format. It includes a diversity of body shapes, motions, skin tones, hair, and clothing. The clothing is realistically simulated on the moving bodies using commercial clothing physics simulation. We render varying numbers of people in realistic scenes with varied lighting and camera motions. We then train various HPS regressors using BEDLAM and achieve state-of-the-art accuracy on real-image benchmarks despite training using with synthetic data. We use BEDLAM to gain insights into what model design choices are important for accuracy. With good synthetic training data, we find that a basic method like HMR approaches the accuracy of the current SOTA method (CLIFF). BEDLAM is useful for a variety of tasks and all images, ground truth bodies, 3D clothing, support code, and more are available for research purposes. Additionally, we provide detailed information about our synthetic data generation pipeline, enabling others to generate their own datasets.Project & Data: https://bedlam.is.tue.mpg.de/Code: github.com/pixelite1201/BEDLAMpdf: https://bedlam.is.tuebingen.mpg.de/media/upload/BEDLAM_CVPR2023.pdfBibtex:@inproceedings{Black_2023_CVPR, title = {{BEDLAM}: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion}, author = {Black, Michael J. and Patel, Priyanka and Tesch, Joachim and Yang, Jinlong}, booktitle = {IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)}, pages = {8726--8737}, month = jun, year = {2023}}
3D Human Pose Estimation via Intuitive Physics (CVPR 2023) Michael Black 2023-06-19 | Estimating 3D humans from images often produces implausible bodies that lean, float, or penetrate the floor. Such methods ignore the fact that bodies are typically supported by the scene. A physics engine can be used to enforce physical plausibility, but these are not differentiable, rely on unrealistic proxy bodies, and are difficult to integrate into existing optimization and learning frameworks. In contrast, we exploit novel intuitive-physics (IP) terms that can be inferred from a 3D SMPL body interacting with the scene. Inspired by biomechanics, we infer the pressure heatmap on the body, the Center of Pressure (CoP) from the heatmap, and the SMPL body’s Center of Mass (CoM). With these, we develop IPMAN, to estimate a 3D body from a color image in a “stable” configuration by encouraging plausible floor contact and overlapping CoP and CoM. Our IP terms are intuitive, easy to implement, fast to compute, differentiable, and can be integrated into existing optimization and regression methods. We evaluate IPMAN on standard datasets and MoYo, a new dataset with synchronized multi-view images, ground-truth 3D bodies with complex poses, body-floor contact, CoM and pressure. IPMAN produces more plausible results than the state of the art, improving accuracy for static poses, while not hurting dynamic ones. Project: https://ipman.is.tue.mpg.de/Code: github.com/sha2nkt/MoYoPaper: arxiv.org/pdf/2303.18246.pdfMOYO dataset: https://moyo.is.tue.mpg.de/Bibtex:@inproceedings{tripathi2023ipman, title = {{3D} Human Pose Estimation via Intuitive Physics}, author = {Tripathi, Shashank and M{\"u}ller, Lea and Huang, Chun-Hao P. and Taheri Omid and Black, Michael J. and Tzionas, Dimitrios}, booktitle = {Conference on Computer Vision and Pattern Recognition ({CVPR})}, year = {2023}, url = {https://ipman.is.tue.mpg.de}}
Reconstructing Signing Avatars From Video Using Linguistic Priors (CVPR 2023) Michael Black 2023-06-19 | Sign language (SL) is the primary method of communication for the 70 million Deaf people around the world. Video dictionaries of isolated signs are a core SL learning tool. Replacing these with 3D avatars can aid learning and enable AR/VR applications, improving access to technology and online media. However, little work has attempted to estimate expressive 3D avatars from SL video. This task is difficult due to occlusion, noise, and motion blur. We address this by introducing novel linguistic priors that are universally applicable to SL and provide constraints on 3D hand pose that help resolve ambiguities within isolated signs. Our method, SGNify, captures fine-grained hand pose, facial expression, and body movement fully automatically from in-the-wild monocular SL videos. We evaluate SGNify quantitatively by using a commercial motion-capture system to compute 3D avatars synchronized with monocular video. SGNify outperforms state-of-the-art 3D body-pose- and shape-estimation methods on SL videos. A perceptual study shows that SGNify’s 3D reconstructions are significantly more comprehensible and natural than those of previous methods and are on par with the source videos.Project and Data: https://sgnify.is.tue.mpg.de/Code: github.com/MPForte/SGNifyPaper PDF: arxiv.org/pdf/2304.10482.pdfBibtex:@inproceedings{Forte23-CVPR-SGNify, title = {Reconstructing Signing Avatars from Video Using Linguistic Priors}, author = {Forte, Maria-Paola and Kulits, Peter and Huang, Chun-Hao Paul and Choutas, Vasileios and Tzionas, Dimitrios and Kuchenbecker, Katherine J. and Black, Michael J.}, booktitle = {IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)}, month = jun, year = {2023}, pages = {12791-12801}}
TRACE: 5D Temporal Regression of Avatars With Dynamic Cameras in 3D Environments (CVPR 2023) Michael Black 2023-06-07 | Although the estimation of 3D human pose and shape (HPS) is rapidly progressing, current methods still cannot reliably estimate moving humans in global coordinates, which is critical for many applications. This is particularly challenging when the camera is also moving, entangling human and camera motion. To address these issues, we adopt a novel 5D representation (space, time, and identity) that enables end-to-end reasoning about people in scenes. Our method, called TRACE, introduces several novel architectural components. Most importantly, it uses two new "maps" to reason about the 3D trajectory of people over time in camera, and world, coordinates. An additional memory unit enables persistent tracking of people even during long occlusions. TRACE is the first one-stage method to jointly recover and track 3D humans in global coordinates from dynamic cameras. By training it end-to-end, and using full image information, TRACE achieves state-of-the-art performance on tracking and HPS benchmarks. The code and dataset are released for research purposes.arXiv: http://arxiv.org/abs/2306.02850Code: https://www.yusun.work/TRACE/TRACE.htmlBibtex:@InProceedings{Sun_2023_CVPR, author = {Sun, Yu and Bao, Qian and Liu, Wu and Mei, Tao and Black, Michael J.}, title = {TRACE: 5D Temporal Regression of Avatars With Dynamic Cameras in 3D Environments}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2023}, pages = {8856-8866}}
TEMPEH: Instant Multi-View Head Capture through Learnable Registration (CVPR 2023) Michael Black 2023-05-08 | Existing methods for capturing datasets of 3D heads in dense semantic correspondence are slow, and commonly address the problem in two separate steps; multi-view stereo (MVS) reconstruction followed by non-rigid registration. To simplify this process, we introduce TEMPEH (Towards Estimation of 3D Meshes from Performances of Expressive Heads) to directly infer 3D heads in dense correspondence from calibrated multi-view images. Registering datasets of 3D scans typically requires manual parameter tuning to find the right balance between accurately fitting the scans’ surfaces and being robust to scanning noise and outliers. Instead, we propose to jointly register a 3D head dataset while training TEMPEH. Specifically, during training we minimize a geometric loss commonly used for surface registration, effectively leveraging TEMPEH as a regularizer. Our multi-view head inference builds on a volumetric feature representation that samples and fuses features from each view using camera calibration information. To account for partial occlusions and a large capture volume that enables head movements, we use view- and surface-aware feature fusion, and a spatial transformer-based head localization module, respectively. We use raw MVS scans as supervision during training, but, once trained, TEMPEH directly predicts 3D heads in dense correspondence without requiring scans. Predicting one head takes about 0.3 seconds with a median reconstruction error of 0.26 mm, 64% lower than the current state-of-the-art. This enables the efficient capture of large datasets containing multiple people and diverse facial motions. Code, model, and data are publicly available.FaMoS dataset:FaMoS is a dynamic 3D head dataset from 95 subjects, each performing 28 motion sequences. The sequences comprise of six prototypical expressions (i.e., Anger, Disgust, Fear, Happiness, Sadness, and Surprise), two head rotations (left/right and up/down), and diverse facial motions, including extreme and asymmetric expressions. Each sequence is recorded at 60 fps. In total, FaMoS contains around 600K 3D head meshes (i.e., ~225 frames per sequence). For each frame, we compute a registration in FLAME mesh topology, which are downloadable here for research purposes. You must sign up and agree to the license to download the data.Project: https://tempeh.is.tue.mpg.de/Paper: https://ps.is.mpg.de/uploads_file/attachment/attachment/711/CVPR2023_Multiview_Face_Capture.pdfDataset: https://tempeh.is.tue.mpg.de/#datasetCitation:@inproceedings{TMPEH:CVPR:2023, title = {Instant Multi-View Head Capture through Learnable Registration}, author = {Bolkart, Timo and Li, Tianye and Black, Michael J.}, booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)}, year = {2023}}
Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguation Michael Black 2022-10-27 | Virtual facial avatars will play an increasingly important role in immersive communication, games and the metaverse, and it is therefore critical that they be inclusive. This requires accurate recovery of the albedo, regardless of age, sex, or ethnicity. While significant progress has been made on estimating 3D facial geometry, appearance estimation has received less attention. The task is fundamentally ambiguous because the observed color is a function of albedo and lighting, both of which are unknown. We find that current methods are biased towards light skin tones due to (1) strongly biased priors that prefer lighter pigmentation and (2) algorithmic solutions that disregard the light/albedo ambiguity. To address this, we propose a new evaluation dataset (FAIR) and an algorithm (TRUST) to improve albedo estimation and, hence, fairness. Specifically, we create the first facial albedo evaluation benchmark where subjects are balanced in terms of skin color, and measure accuracy using the Individual Typology Angle (ITA) metric. We then address the light/albedo ambiguity by building on a key observation: the image of the full scene –as opposed to a cropped image of the face– contains important information about lighting that can be used for disambiguation. TRUST regresses facial albedo by conditioning on both the face region and a global illumination signal obtained from the scene image. Our experimental results show significant improvement compared to state- of-the-art methods on albedo estimation, both in terms of accuracy and fairness. The evaluation benchmark and code are available for research purposes at https://trust.is.tue.mpg.de.PDF: ecva.net/papers/eccv_2022/papers_ECCV/papers/136730072.pdfProject: https://trust.is.tue.mpg.de/index.htmlCode: github.com/HavenFeng/TRUSTDataset: https://trust.is.tue.mpg.de/login.phpReference:@inproceedings{TRUST:ECCV2022, title = {Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguation}, author = {Feng, Haiwen and Bolkart, Timo and Tesch, Joachim and Black, Michael J. and Abrevaya, Victoria}, booktitle = {European Conference on Computer Vision (ECCV)}, publisher = {Springer International Publishing}, month = oct, year = {2022}, doi = {}, month_numeric = {10}}
BARC: Learning to Regress 3D Dog Shape from Images by Exploiting Breed Information (CVPR 2022) Michael Black 2022-06-21 | Our goal is to recover the 3D shape and pose of dogs from a single image. This is a challenging task because dogs exhibit a wide range of shapes and appearances, and are highly articulated. Recent work has proposed to directly regress the SMAL animal model, with additional limb scale parameters, from images. Our method, called BARC (Breed-Augmented Regression using Classification), goes beyond prior work in several important ways. First, we modify the SMAL shape space to be more appropriate for representing dog shape. But, even with a better shape model, the problem of regressing dog shape from an image is still challenging because we lack paired images with 3D ground truth. To compensate for the lack of paired data, we formulate novel losses that exploit information about dog breeds. In particular, we exploit the fact that dogs of the same breed have similar body shapes. We formulate a novel breed similarity loss consisting of two parts: One term encourages the shape of dogs from the same breed to be more similar than dogs of different breeds. The second one, a breed classification loss, helps to produce recognizable breed-specific shapes. Through ablation studies, we find that our breed losses significantly improve shape accuracy over a baseline without them. We also compare BARC qualitatively to WLDO with a perceptual study and find that our approach produces dogs that are significantly more realistic. This work shows that a-priori information about genetic similarity can help to compensate for the lack of 3D training data. This concept may be applicable to other animal species or groups of species.Project: https://barc.is.tue.mpg.de/Code: github.com/runa91/barc_releasePaper: https://barc.is.tue.mpg.de/media/upload/barc_main.pdfReference:@inproceedings{BARC:2022, title = {{BARC}: Learning to Regress {3D} Dog Shape from Images by Exploiting Breed Information}, author = {Rueegg, Nadine and Zuffi, Silvia and Schindler, Konrad and Black, Michael J.}, booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)}, pages={3876--3884}, year = {2022}}
MOVER: Human-Aware Object Placement for Visual Environment Reconstruction (CVPR 2022) Michael Black 2022-06-20 | Humans are in constant contact with the world as they move through it and interact with it. This contact is a vital source of information for understanding 3D humans, 3D scenes, and the interactions between them. In fact, we demonstrate that these human-scene interactions (HSIs) can be leveraged to improve the 3D reconstruction of a scene from a monocular RGB video. Our key idea is that, as a person moves through a scene and interacts with it, we accumulate HSIs across multiple input images, and optimize the 3D scene to reconstruct a consistent, physically plausible and functional 3D scene layout. Our optimization-based approach exploits three types of HSI constraints: (1) humans that move in a scene are occluded or occlude objects, thus, defining the depth ordering of the objects, (2) humans move through free space and do not interpenetrate objects, (3) when humans and objects are in contact, the contact surfaces occupy the same place in space. Using these constraints in an optimization formulation across all observations, we significantly improve the 3D scene layout reconstruction. Furthermore, we show that our scene reconstruction can be used to refine the initial 3D human pose and shape (HPS) estimation. We evaluate the 3D scene layout reconstruction and HPS estimation qualitatively and quantitatively using the PROX and PiGraphs datasets. Project: https://mover.is.tue.mpg.de/Paper: arxiv.org/pdf/2203.03609.pdfCode: github.com/yhw-yhw/moverReference:@inproceedings{yi2022mover,title = {Human-Aware Object Placement for Visual Environment Reconstruction},author = {Yi, Hongwei and Huang, Chun-Hao P. and Tzionas, Dimitrios and Kocabas, Muhammed and Hassan, Mohamed and Tang, Siyu and Thies, Justus and Black, Michael J.},booktitle = {Computer Vision and Pattern Recognition (CVPR)},month = jun,year = {2022},pages={3959--3970},month_numeric = {6}}
Meta-commerce in the Age of Avatars Michael Black 2022-06-19 | In this short 20 minute talk, I introduce the idea of meta-commerce and the role of avatars in enabling a blurring between the physical and virtual worlds. In doing so, I summarize recent work from my department on creating and animating full-body and facial avatars from images and video. The talk is aimed at a general (not expert) audience that is familiar with computer vision and machine learning.This talk was given at the Retail Vision workshop at CVPR 2022, retailvisionworkshop.github.ioLinks to the CVPR'22 work described here:BEV: github.com/Arthur151/ROMPSHAPY: https://shapy.is.tue.mpg.de/EMOCA: https://emoca.is.tue.mpg.de/gDNA: xuchen-ethz.github.io/gdnaIMAvatar: https://ait.ethz.ch/projects/2022/IMavatar/ICON: https://icon.is.tue.mpg.de/
SHAPY: Accurate 3D Body Shape Regression using Semantic Attributes (CVPR 2022) Michael Black 2022-06-17 | While methods that regress 3D human meshes from images have progressed rapidly, the estimated body shapes often do not capture the true human shape. This is problematic since, for many applications, accurate body shape is as important as pose. The key reason that body shape accuracy lags pose accuracy is the lack of data. While humans can label 2D joints, and these constrain 3D pose, it is not so easy to “label” 3D body shape. Since paired data with images and 3D body shape are rare, we exploit two sources of partial information: (1) we collect internet images of diverse models together with a small set of measurements; (2) we collect semantic shape attributes for a wide range of 3D body meshes and model images. Taken together, these datasets provide sufficient constraints to infer metric 3D shape. We exploit this partial and semantic data in several novel ways to train a neural network, called SHAPY, that regresses 3D human pose and shape from an RGB image. We evaluate SHAPY on public benchmarks but note that they either lack significant body shape variation, ground-truth shape, or clothing variation. Thus, we collect a new dataset for 3D human shape estimation, containing photos of people in the wild for whom we have ground-truth 3D body scans. On this new benchmark, SHAPY significantly outperforms recent state-of-the-art methods on the task of 3D body shape estimation. This is the first demonstration that a 3D body shape regressor can be trained from sparse measurements and easy-to-obtain semantic shape attributes. Our model and data are available for research.PDF: https://ps.is.mpg.de/uploads_file/attachment/attachment/691/00928.pdfProject: https://shapy.is.tue.mpg.de/Code: github.com/muelea/shapyReference:@inproceedings{Shapy:CVPR:2022, title = {Accurate 3D Body Shape Regression using Metric and Semantic Attributes}, author = {Choutas, Vasileios and M{\"u}ller, Lea and Huang, Chun-Hao P. and Tang, Siyu and Tzionas, Dimitrios and Black, Michael J.}, booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)}, year = {2022}}
GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping (CVPR22) Michael Black 2022-06-16 | Generating digital humans that move realistically has many applications and is widely studied, but existing methods focus on the major limbs of the body, ignoring the hands and head. Hands have been separately studied but the focus has been on generating realistic static grasps of objects. To synthesize virtual characters that interact with the world, we need to generate full-body motions and realistic hand grasps simultaneously. Both sub-problems are challenging on their own and, together, the state-space of poses is significantly larger, the scales of hand and body motions differ, and the whole-body posture and the hand grasp must agree, satisfy physical constraints, and be plausible. Additionally, the head is involved because the avatar must look at the object to interact with it. For the first time, we address the problem of generating full-body, hand and head motions of an avatar grasping an unknown object. As input, our method, called GOAL, takes a 3D object, its position, and a starting 3D body pose and shape. GOAL outputs a sequence of whole-body poses using two novel networks. First, GNet generates a goal whole-body grasp with a realistic body, head, arm, and hand pose, as well as hand-object contact. Second, MNet generates the motion between the starting and goal pose. This is challenging, as it requires the avatar to walk towards the object with foot-ground contact, orient the head towards it, reach out, and grasp it with a realistic hand pose and hand-object contact. To achieve this the networks exploit a representation that combines SMPL-X body parameters and 3D vertex offsets. We train and evaluate GOAL, both qualitatively and quantitatively, on the GRAB dataset. Results show that GOAL generalizes well to unseen objects, outperforming baselines. A perceptual study shows that GOAL’s generated motions approach the realism of GRAB’s ground truth. GOAL takes a step towards synthesizing realistic full-body object grasping. The models and code are available for research.Project: https://goal.is.tue.mpg.de/PDF: arxiv.org/pdf/2112.11454Code: github.com/otaheri/GOALReference:@inproceedings{taheri2021goal, title = {{GOAL}: {G}enerating {4D} Whole-Body Motion for Hand-Object Grasping}, author = {Taheri, Omid and Choutas, Vasileios and Black, Michael J. and Tzionas, Dimitrios}, booktitle = {Conference on Computer Vision and Pattern Recognition ({CVPR})}, year = {2022}, url = {https://goal.is.tue.mpg.de}}
EMOCA: Emotion Driven Monocular Face Capture and Animation (CVPR 2022) Michael Black 2022-04-26 | As 3D facial avatars become more widely used for communication, it is critical that they faithfully convey emotion. Unfortunately, the best recent methods that regress parametric 3D face models from monocular images are unable to capture the full spectrum of facial expression, such as subtle or extreme emotions. We find the standard reconstruction metrics used for training (landmark reprojection error, photometric error, and face recognition loss) are insufficient to capture high-fidelity expressions. The result is facial geometries that do not match the emotional content of the input image. We address this with EMOCA (EMOtion Capture and Animation), by introducing a novel deep perceptual emotion consistency loss during training, which helps ensure that the reconstructed 3D expression matches the expression depicted in the input image. While EMOCA achieves 3D reconstruction errors that are on par with the current best methods, it significantly outperforms them in terms of the quality of the reconstructed expression and the perceived emotional content. We also directly regress levels of valence and arousal and classify basic expressions from the estimated 3D face parameters. On the task of in-the-wild emotion recognition, our purely geometric approach is on par with the best image-based methods, highlighting the value of 3D geometry in analyzing human behavior. Project: https://emoca.is.tue.mpg.de/Code: github.com/radekd91/emocaPaper: https://ps.is.mpg.de/uploads_file/attachment/attachment/686/EMOCA__CVPR22.pdfarXiv: arxiv.org/abs/2204.11312Reference:@inproceedings{EMOCA:CVPR:2021, title = {{EMOCA}: {E}motion Driven Monocular Face Capture and Animation}, author = {Danecek, Radek and Black, Michael J. and Bolkart, Timo}, booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)}, year = {2022}}
OSSO: Obtaining Skeletal Shape from Outside (CVPR 2022) Michael Black 2022-04-22 | We address the problem of inferring the anatomic skeleton of a person, in an arbitrary pose, from the 3D surface of the body; i.e. we predict the inside (bones) from the outside (skin). This has many applications in medicine and biomechanics. Existing state-of-the-art biomechanical skeletons are detailed but do not easily generalize to new subjects. Additionally, computer vision and graphics methods that predict skeletons are typically heuristic, not learned from data, do not leverage the full 3D body surface, and are not validated against ground truth. To our knowledge, our system, called OSSO (Obtaining Skeletal Shape from Outside), is the first to learn the mapping from the 3D body surface to the internal skeleton from real data. We do so using 1000 male and 1000 female dual-energy X-ray absorptiometry (DXA) scans. To these, we fit a parametric 3D body shape model (STAR) to capture the body surface and a novel part-based 3D skeleton model to capture the bones. This provides inside/outside training pairs. We model the statistical variation of full skeletons using PCA in a pose-normalized space. We then train a regressor from body shape parameters to skeleton shape parameters and refine the skeleton to satisfy constraints on physical plausibility. Given an arbitrary 3D body shape and pose, OSSO predicts a realistic skeleton inside. In contrast to previous work, we evaluate the accuracy of the skeleton shape quantitatively on held out DXA scans, outperforming the state-of-the art. We also show 3D skeleton prediction from varied and challenging 3D bodies. The code to infer a skeleton from a body shape is available for research, and the dataset of paired outer surface (skin) and skeleton (bone) meshes are available as a Biobank Returned Dataset. This research has been conducted using the UK Biobank Resource.Project: https://osso.is.tue.mpg.de/Code: github.com/MarilynKeller/OSSODataset: https://osso.is.tue.mpg.de/#DatasetPaper: https://download.is.tue.mpg.de/osso/OSSO.pdfReference:@inproceedings{Keller:CVPR:2022, title = {{OSSO}: Obtaining Skeletal Shape from Outside}, author = {Keller, Marilyn and Zuffi, Silvia and Black, Michael J. and Pujades, Sergi}, booktitle = {Proceedings IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)}, month = jun, year = {2022}, month_numeric = {6}}
BEV: Monocular Regression of Multiple 3D People in Depth (CVPR 2022) Michael Black 2022-04-17 | Given an image with multiple people, we directly regress the pose and shape of all the people as well as their relative depth. Inferring the depth of a person in an image, however, is fundamentally ambiguous without knowing their height. This is particularly problematic when the scene contains people of very different sizes, e.g. from infants to adults. To solve this, we need several things. First, we develop a novel method to infer the poses and depths of multiple people in a single image. While previous work that estimates multiple people does so by reasoning in the image plane, our method, called BEV, adds an additional imaginary Bird's-Eye-View representation to explicitly reason about depth. BEV reasons simultaneously about body centers in the image and in depth and, by combing these, estimates 3D body position. Unlike prior work, BEV is a single-shot method that is end-to-end differentiable. Second, height varies with age, making it impossible to resolve depth without also estimating the age of people in the image. To do so, we exploit a 3D body model space that lets BEV infer shapes from infants to adults. Third, to train BEV, we need a new dataset. Specifically, we create a "Relative Human" (RH) dataset that includes age labels and relative depth relationships between the people in the images. Extensive experiments on RH and AGORA demonstrate the effectiveness of the model and training scheme. BEV outperforms existing methods on depth reasoning, child shape estimation, and robustness to occlusion. The code and dataset are released for research purposes.Code: github.com/Arthur151/ROMPRH dataset: github.com/Arthur151/Relative_HumanProject: arthur151.github.io/BEV/BEV.htmlarXiv: arxiv.org/abs/2112.08274Reference:@InProceedings{BEV,author = {Sun, Yu and Liu, Wu and Bao, Qian and Fu, Yili and Mei, Tao and Black, Michael J.},title = {Putting People in their Place: Monocular Regression of 3D People in Depth},booktitle = {IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)},month = jun,year = {2022}}
Perceiving Systems: 2016-2021 Michael Black 2022-02-17 | Every few years our institute (the Max Planck Institute for Intelligent Systems) is evaluated by our Scientific Advisory Board (SAB). I gave this talk at our recent evaluation, which covered the research in the Perceiving Systems department over the last 6 years. Since it summarizes our research in about 20 minutes, I thought It might be interesting to a broader audience. I couldn't cover everything and everyone who was part of our journey so please visit our website for a full description of our research: https://ps.is.mpg.de/researchAll the publications mentioned are available here: https://ps.is.mpg.de/publicationsThe code and data is here: https://ps.is.mpg.de/codeFull citations of the work I talk about:SMPL: Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M. J., “SMPL: A Skinned Multi-Person Linear Model”, ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1-248:16, Oct. 2015SMPLify: Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., Black, M. J., “Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image,” In Computer Vision – ECCV 2016, pages: 561-578, Oct. 2016HMR: Kanazawa, A., Black, M. J., Jacobs, D. W., Malik, J., “End-to-end Recovery of Human Shape and Pose,” In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages: 7122-7131, 20183DPW: von Marcard, T.., Henschel, R., Black, M. J., Rosenhahn, B., Pons-Moll, G., Recovering Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera,” In European Conference on Computer Vision (ECCV), Sept. 2018SPIN: Kolotouros, N., Pavlakos, G., Black, M. J., Daniilidis, K., “Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop,” Proceedings International Conference on Computer Vision (ICCV), pages: 2252-2261, Oct. 2019SMPL-X: Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A. A. A., Tzionas, D., Black, M. J., “Expressive Body Capture: 3D Hands, Face, and Body from a Single Image,” In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages: 10975-10985, June 2019VIBE: Kocabas, M., Athanasiou, N., Black, M. J., “VIBE: Video Inference for Human Body Pose and Shape Estimation,” In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages: 5252-5262, June 2020AMASS: Mahmood, N., Ghorbani, N., Troje, N. F., Pons-Moll, G., Black, M. J., “AMASS: Archive of Motion Capture as Surface Shapes,” Proceedings International Conference on Computer Vision, pages: 5442-5451, October 2019MANO: Romero, J., Tzionas, D., Black, M. J., “Embodied Hands: Modeling and Capturing Hands and Bodies Together,” ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6):245:1-245:17, 245:1–245:17, ACM, Nov. 2017FLAME: Li, T., Bolkart, T., Black, M. J., Li, H., Romero, J., “Learning a model of facial shape and expression from 4D scans,” ACM Transactions on Graphics, 36(6):194:1-194:17, Nov. 2017PROX: Hassan, M., Choutas, V., Tzionas, D., Black, M. J., “Resolving 3D Human Pose Ambiguities with 3D Scene Constraints,” In Proceedings International Conference on Computer Vision, pages: 2282-2292, Oct. 2019eXpose: Choutas, V., Pavlakos, G., Bolkart, T., Tzionas, D., Black, M. J., “Monocular Expressive Body Regression through Body-Driven Attention,” In Computer Vision – ECCV 2020, LNCS 12355, pages: 20-40, Aug. 2020PIXIE: Feng, Y., Choutas, V., Bolkart, T., Tzionas, D., Black, M., “Collaborative Regression of Expressive Bodies using Moderation,” International Conference on 3D Vision (3DV), December 2021BABEL: Punnakkal, A. R., Chandrasekaran, A., Athanasiou, N., Quiros-Ramirez, A., Black, M. J., “BABEL: Bodies, Action and Behavior with English Labels,” In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages: 722-731, June 2021GRAB: Taheri, O., Ghorbani, N., Black, M. J., Tzionas, D., “GRAB: A Dataset of Whole-Body Human Grasping of Objects,” In Computer Vision – ECCV 2020, LNCS 12355, pages: 581-600, Aug. 2020ObMan: Hasson, Y., Varol, G., Tzionas, D., Kalevatykh, I., Black, M. J., Laptev, I., Schmid, C., “Learning Joint Reconstruction of Hands and Manipulated Objects,” In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages: 11807-11816, June 2019POSA: Hassan, M., Ghosh, P., Tesch, J., Tzionas, D., Black, M. J., “Populating 3D Scenes by Learning Human-Scene Interaction,” In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages: 14708-14718, June 2021SAMP: Hassan, M., Ceylan, D., Villegas, R., Saito, J., Yang, J., Zhou, Y., Black, M., Stochastic Scene-Aware Motion Prediction,” In Proc. International Conference on Computer Vision (ICCV), pages: 11374-11384, October 2021SCANimate: Saito, S., Yang, J., Ma, Q., Black, M. J., “SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks,” In Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages: 2886-2897, June 2021
PIXIE: Collaborative Regression of Expressive Bodies using Moderation (3DV21) Michael Black 2021-12-15 | Recovering expressive humans from images is essential for understanding human behavior. Methods that estimate 3D bodies, faces, or hands have progressed significantly, yet separately. Face methods recover accurate 3D shape and geometric details, but need a tight crop and struggle with extreme views and low resolution. Whole-body methods are robust to a wide range of poses and resolutions, but provide only a rough 3D face shape without details like wrinkles. To get the best of both worlds, we introduce PIXIE, which produces animatable, whole-body 3D avatars with realistic facial detail, from a single image. For this, PIXIE uses two key observations. First, existing work combines independent estimates from body, face, and hand experts, by trusting them equally. PIXIE introduces a novel moderator that merges the features of the experts, weighted by their confidence. All part experts can contribute to the whole, using SMPL-X’s shared shape space across all body parts. Second, human shape is highly correlated with gender, but existing work ignores this. We label training images as male, female, or non-binary, and train PIXIE to infer “gendered” 3D body shapes with a novel shape loss. In addition to 3D body pose and shape parameters, PIXIE estimates expression, illumination, albedo and 3D facial surface displacements. Quantitative and qualitative evaluation shows that PIXIE estimates more accurate whole-body shape and detailed face shape than the state of the art. Paper: https://ps.is.mpg.de/uploads_file/attachment/attachment/667/PIXIE_3DV_CR.pdfCode: github.com/YadiraF/PIXIEProject: https://pixie.is.tue.mpg.de/Bibtex:@inproceedings{PIXIE:2021, title={Collaborative Regression of Expressive Bodies using Moderation}, author={Yao Feng and Vasileios Choutas and Timo Bolkart and Dimitrios Tzionas and Michael J. Black}, booktitle={International Conference on 3D Vision (3DV)}, year={2021}}Voice: fiverr.com/christianhmiles
Virtuelle Tour durch die Capture Hall im Perzeptive Systeme (Deutsch) Michael Black 2021-10-28 | Willkommen in der Capture Hall der Abteilung Perzeptive Systeme am Max-Planck-Institut für Intelligente Systeme.Diese virtuelle Tour stellt die einzigartige Ausrüstung vor, mit der wir die Form und Bewegung des menschlichen Körpers erfassen. Wir geben einige Einblicke, wie und warum wir die von uns erhobenen Daten erfassen und wie diese Daten es uns ermöglichen, Computern beizubringen, Menschen zu „sehen“.Sie erfahren mehr über unsere4D-Körperscanner: https://ps.is.tuebingen.mpg.de/pages/4d-capture4D-Gesichtsscanner: https://ps.is.tuebingen.mpg.de/pages/4d-dynamic-face-scanner4D-Handscanner: https://ps.is.tuebingen.mpg.de/pages/4d-hand-scannermocap-System: https://ps.is.tuebingen.mpg.de/pages/motion-capture
Perceiving Systems Capture Hall Virtual Tour (English) Michael Black 2021-10-28 | Welcome to the Capture Hall in the Perceiving Systems Department at the Max Planck Institute for Intelligent Systems.This virtual tour introduces the unique equipment that we use to capture human body shape and motion. We give some insights into how and why we capture the data we do and how this data allows us to train computers to "see" people.You'll learn about our4D body scanner: https://ps.is.tuebingen.mpg.de/pages/4d-capture4D face scanner: https://ps.is.tuebingen.mpg.de/pages/4d-dynamic-face-scanner4D hand scanner: https://ps.is.tuebingen.mpg.de/pages/4d-hand-scannermocap system: https://ps.is.tuebingen.mpg.de/pages/motion-capture
ROMP: Monocular, One-stage, Regression of Multiple 3D People (ICCV 2021) Michael Black 2021-10-13 | This paper focuses on the regression of multiple 3D people from a single RGB image. Existing approaches predominantly follow a multi-stage pipeline that first detects people in bounding boxes and then independently regresses their 3D body meshes. In contrast, we propose to Regress all meshes in a One-stage fashion for Multiple 3D People (termed ROMP). The approach is conceptually simple, bounding box-free, and able to learn a per-pixel representation in an end-to-end manner. Our method simultaneously predicts a Body Center heatmap and a Mesh Parameter map, which can jointly describe the 3D body mesh on the pixel level. Through a body-center-guided sampling process, the body mesh parameters of all people in the image are easily extracted from the Mesh Parameter map. Equipped with such a fine-grained representation, our one-stage framework is free of the complex multi-stage process and more robust to occlusion. Compared with state-of-the-art methods, ROMP achieves superior performance on the challenging multi-person benchmarks, including 3DPW and CMU Panoptic. Experiments on crowded/occluded datasets demonstrate the robustness under various types of occlusion. The released code is the first real-time implementation of monocular multi-person 3D mesh regression.arXiv: http://arxiv.org/abs/2008.12272Code: github.com/Arthur151/ROMPReference:@inproceedings{ROMP:ICCV:2021, title = {Monocular, One-Stage, Regression of Multiple {3D} People}, author = {Sun, Yu and Bao, Qian and Liu, Wu and Fu, Yili and Black, Michael J. and Mei, Tao}, booktitle = {Proc. International Conference on Computer Vision (ICCV)}, pages = {11179--11188}, month = oct, year = {2021}, doi = {}, month_numeric = {10}}
Learning to Regress Bodies using Differentiable Semantic Rendering (ICCV 2021) Michael Black 2021-10-13 | Learning to regress 3D human body shape and pose (e.g. SMPL parameters) from monocular images typically exploits losses on 2D keypoints, silhouettes, and/or part-segmentation when 3D training data is not available. Such losses, however, are limited because 2D keypoints do not supervise body shape and segmentations of people in clothing do not match projected minimally-clothed SMPL shapes. To exploit richer image information about clothed people, we introduce higher-level semantic information about clothing to penalize clothed and non-clothed regions of the image differently. To do so, we train a body regressor using a novel ‘’Differentiable Semantic Rendering - DSR" loss. For Minimally-Clothed regions, we define the DSR-MC loss, which encourages a tight match between a rendered SMPL body and the minimally-clothed regions of the image. For clothed regions, we define the DSR-C loss to encourage the rendered SMPL body to be inside the clothing mask. To ensure end-to-end differentiable training, we learn a semantic clothing prior for SMPL vertices from thousands of clothed human scans. We perform extensive qualitative and quantitative experiments to evaluate the role of clothing semantics on the accuracy of 3D human body estimation. We outperform all previous state-of-the-art methods on 3DPW and Human3.6M and obtain on par results on MPI-INF-3DHP.arXiv: tinyurl.com/j5wka6khProject: https://dsr.is.tue.mpg.de/Code: github.com/saidwivedi/DSRReference:@inproceedings{DSR:ICCV:2021, title = {Learning To Regress Bodies From Images Using Differentiable Semantic Rendering}, author = {Dwivedi, Sai Kumar and Athanasiou, Nikos and Kocabas, Muhammed and Black, Michael J.}, booktitle = {Proc. International Conference on Computer Vision (ICCV)}, pages = {11250--11259}, month = oct, year = {2021}, doi = {}, month_numeric = {10}}
PARE: Part Attention Regressor for 3D Human Body Estimation (ICCV 2021) Michael Black 2021-10-13 | Despite significant progress, state of the art 3D human pose and shape estimation methods remain sensitive to partial occlusion and can produce dramatically wrong predictions although much of the body is observable. To address this, we introduce a soft attention mechanism, called the Part Attention REgressor (PARE), that learns to predict body-part-guided attention masks. We observe that state-of-the-art methods rely on global feature representations, making them sensitive to even small occlusions. In contrast, PARE's part-guided attention mechanism overcomes these issues by exploiting information about the visibility of individual body parts while leveraging information from neighboring body-parts to predict occluded parts. We show qualitatively that PARE learns sensible attention masks, and quantitative evaluation confirms that PARE achieves more accurate and robust reconstruction results than existing approaches on both occlusion-specific and standard benchmarks. arXiv: arxiv.org/abs/2104.08527Project: https://pare.is.tue.mpg.de/Code: github.com/mkocabas/PAREReference:@inproceedings{Kocabas_PARE_2021, title = {{PARE}: Part Attention Regressor for {3D} Human Body Estimation}, author = {Kocabas, Muhammed and Huang, Chun-Hao P. and Hilliges, Otmar and Black, Michael J.}, booktitle = {Proc. International Conference on Computer Vision (ICCV)}, pages = {11127--11137}, month = oct, year = {2021}, doi = {}, month_numeric = {10}}
SPEC: Seeing People in the Wild with an Estimated Camera (ICCV 2021) Michael Black 2021-10-13 | Due to the lack of camera parameter information for in-the-wild images, existing 3D human pose and shape (HPS) estimation methods make several simplifying assumptions: weak-perspective projection, large constant focal length, and zero camera rotation. These assumptions often do not hold and we show, quantitatively and qualitatively, that they cause errors in the reconstructed 3D shape and pose. To address this, we introduce SPEC, the first in-the-wild 3D HPS method that estimates the perspective camera from a single image and employs this to reconstruct 3D human bodies more accurately. First, we train a neural network to estimate the field of view, camera pitch, and roll given an input image. We employ novel losses that improve the calibration accuracy over previous work. We then train a novel network that concatenates the camera calibration to the image features and uses these together to regress 3D body shape and pose. SPEC is more accurate than the prior art on the standard benchmark (3DPW) as well as two new datasets with more challenging camera views and varying focal lengths. Specifically, we create a new photorealistic synthetic dataset (SPEC-SYN) with ground truth 3D bodies and a novel in-the-wild dataset (SPEC-MTP) with calibration and high-quality reference bodies. Both qualitative and quantitative analysis confirm that knowing camera parameters during inference regresses better human bodies.arXiv: arxiv.org/abs/2110.00620Project page: https://spec.is.tue.mpg.de/Code: github.com/mkocabas/SPECReference:@inproceedings{Kocabas_SPEC_2021, title = {{SPEC}: Seeing People in the Wild with an Estimated Camera}, author = {Kocabas, Muhammed and Huang, Chun-Hao P. and Tesch, Joachim and M\"uller, Lea and Hilliges, Otmar and Black, Michael J.}, booktitle = {Proc. International Conference on Computer Vision (ICCV)}, pages = {11035--11045}, month = oct, year = {2021}, doi = {}, month_numeric = {10}}
SOMA: Solving Optical Marker-Based MoCap Automatically (ICCV2021) Michael Black 2021-10-07 | Marker-based optical motion capture (mocap) is the "gold standard" method for acquiring accurate 3D human motion in computer vision, medicine, and graphics. The raw output of these systems are noisy and incomplete 3D points or short tracklets of points. To be useful, one must associate these points with corresponding markers on the captured subject; i.e. "labelling". Given these labels, one can then "solve" for the 3D skeleton or body surface mesh. Commercial auto-labeling tools require a specific calibration procedure at capture time, which is not possible for archival data. Here we train a novel neural network called SOMA, which takes raw mocap point clouds with varying numbers of points, labels them at scale without any calibration data, independent of the capture technology, and requiring only minimal human intervention. Our key insight is that, while labeling point clouds is highly ambiguous, the 3D body provides strong constraints on the solution that can be exploited by a learning-based method. To enable learning, we generate massive training sets of simulated noisy and ground truth mocap markers animated by 3D bodies from AMASS. SOMA exploits an architecture with stacked self-attention elements to learn the spatial structure of the 3D body and an optimal transport layer to constrain the assignment (labeling) problem while rejecting outliers. We extensively evaluate SOMA both quantitatively and qualitatively. SOMA is more accurate and robust than existing state of the art research methods and can be applied where commercial systems cannot. We automatically label over 8 hours of archival mocap data across 4 different datasets captured using various technologies and output SMPL-X body models. The model and data is released for research purposes.Project/Code: https://soma.is.tue.mpg.de/pdf: https://download.is.tue.mpg.de/soma/SOMA_ICCV21.pdfReference: @inproceedings{SOMA:ICCV:2021, title = {{SOMA}: Solving Optical Marker-Based MoCap Automatically}, author = {Ghorbani, Nima and Black, Michael J.}, booktitle = {Proc. International Conference on Computer Vision (ICCV)}, publisher = {IEEE}, month = oct, year = {2021}, month_numeric = {10}}
DECA: Siggraph 2021 talk (full version) Michael Black 2021-09-09 | Full conference presentation of:DECA: Learning an Animatable Detailed 3D Face Model from In-the-Wild ImagesWhile current monocular 3D face reconstruction methods can recover fine geometric details, they suffer several limitations. Some methods produce faces that cannot be realistically animated because they do not model how wrinkles vary with expression. Other methods are trained on high-quality face scans and do not generalize well to in-the-wild images. We present the first approach that regresses 3D faces shape and animatable details that are specific to an individual but change with expression. Our model, DECA (Detailed Expression Capture and Animation), is trained to robustly produce a UV displacement map from a low-dimensional latent representation that consists of person-specific detail parameters and generic expression parameters, while a regressor is trained to predict detail, shape, albedo, expression, pose and illumination parameters from a single image. To enable this, we introduce a novel detail-consistency loss that disentangles person-specific details from expression-dependent wrinkles. This disentanglement allows us to synthesize realistic person-specific wrinkles by controlling expression parameters while keeping person-specific details unchanged. DECA is learned from in-the-wild images with no paired 3D supervision and achieves state-of-the-art shape reconstruction accuracy on two benchmarks. Qualitative results on in-the-wild data demonstrate DECA's robustness and its ability to disentangle identity- and expression-dependent details enabling animation of reconstructed faces. Authors: Yao Feng*, Haiwen Feng*, Michael J. Black, and Timo Bolkart (*authors contributed equally)Project: https://deca.is.tue.mpg.deCode: github.com/YadiraF/DECApdf: http://files.is.tue.mpg.de/black/papers/SIGGRAPH21_DECA.pdfSupplemental: http://files.is.tue.mpg.de/black/papers/SIGGRAPH21_DECA_SupMat.pdfBibtex:@article{Feng:SIGGRAPH:2021, title = {Learning an Animatable Detailed {3D} Face Model from In-the-Wild Images}, author = {Feng, Yao and Feng, Haiwen and Black, Michael J. and Bolkart, Timo}, journal = {ACM Trans. Graphics (ToG), Proc. SIGGRAPH}, volume = {40}, number = {4}, pages = {88:1--88:13}, month = aug, year = {2021}, month_numeric = {8}}
SAMP: Scene-Aware Motion Prediction (ICCV 2021) Michael Black 2021-08-17 | A long-standing goal in computer vision is to capture, model, and realistically synthesize human behavior. Specifically, by learning from data, our goal is to enable virtual humans to navigate within cluttered indoor scenes and naturally interact with objects. Such embodied behavior has applications in virtual reality, computer games, and robotics, while synthesized behavior can be used as training data. The problem is challenging because real human motion is diverse and adapts to the scene. For example, a person can sit or lie on a sofa in many places and with varying styles. We must model this diversity to synthesize virtual humans that realistically perform human-scene interactions. We present a novel data-driven, stochastic motion synthesis method that models different styles of performing a given action with a target object. Our Scene-Aware Motion Prediction method (SAMP) generalizes to target objects of various geometries while enabling the character to navigate in cluttered scenes. To train SAMP, we collected mocap data covering various sitting, lying down, walking, and running styles. We demonstrate SAMP on complex indoor scenes and achieve superior performance than existing solutions. Project page: https://samp.is.tue.mpg.de/pdf: https://ps.is.tuebingen.mpg.de/uploads_file/attachment/attachment/652/samp.pdfReference:@inproceedings{hassan_samp_2021, title = {Stochastic Scene-Aware Motion Prediction}, author = {Hassan, Mohamed and Ceylan, Duygu and Villegas, Ruben and Saito, Jun and Yang, Jimei and Zhou, Yi and Black, Michael}, booktitle = {Proceedings of the International Conference on Computer Vision 2021}, month = oct, year = {2021}, event_name = {International Conference on Computer Vision 2021}, event_place = {virtual (originally Montreal, Canada)}, month_numeric = {10}}
SCALE (CVPR21 Talk Version) Michael Black 2021-07-22 | SCALE: Modeling Clothed Humans with a Surface Codec of Articulated Local Elements Learning to model and reconstruct humans in clothing is challenging due to articulation, non-rigid deformation, and varying clothing types and topologies. To enable learning, the choice of representation is the key. Recent work uses neural networks to parameterize local surface elements. This approach captures locally coherent geometry and non-planar details, can deal with varying topology, and does not require registered training data. However, naively using such methods to model 3D clothed humans fails to capture fine-grained local deformations and generalizes poorly. To address this, we present three key innovations: First, we deform surface elements based on a human body model such that large-scale deformations caused by articulation are explicitly separated from topological changes and local clothing deformations. Second, we address the limitations of existing neural surface elements by regressing local geometry from local features, significantly improving the expressiveness. Third, we learn a pose embedding on a 2D parameterization space that encodes posed body geometry, improving generalization to unseen poses by reducing non-local spurious correlations. We demonstrate the efficacy of our surface representation by learning models of complex clothing from point clouds. The clothing can change topology and deviate from the topology of the body. Once learned, we can animate previously unseen motions, producing high-quality point clouds, from which we generate realistic images with neural rendering. We assess the importance of each technical contribution and show that our approach outperforms the state-of-the- art methods in terms of reconstruction accuracy and inference time. Code: github.com/qianlim/SCALEPaper: arxiv.org/abs/2104.07660Citation:@inproceedings{Ma:CVPR:2021, title = {{SCALE}: Modeling Clothed Humans with a Surface Codec of Articulated Local Elements}, author = {Ma, Qianli and Saito, Shunsuke and Yang, Jinlong and Tang, Siyu and Black, Michael J.}, booktitle = {Proceedings IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)}, month = jun, year = {2021}}
Putting SMPL into Scenes Michael Black 2021-06-22 | Part of the SMPL made Simple tutorial at CVPR 2021Presenter: Siyu Tang
SMPL made Simple Introduction Michael Black 2021-06-22 | Part of the SMPL made Simple Tutorial at CVPR 2021Instructor: Michael BlackContents: history of body models, scanning, registration, PCA, linear blend skinning, corrective blend shapes, SMPL, faces, hands, SMPL-X, dynamics of soft tissue, future directions like implicit surfaces and neural rendering.
SMPL: Frequently Asked Questions Michael Black 2021-06-21 | Part of the SMPL made Simple tutorial at CVPR 2021Presenters: members of Perceiving Systems and Meshcapade
Problems with SMPL and fixing them with STAR Michael Black 2021-06-21 | Part of the SMPL made Simple tutorial at CVPR 2021Presenter: Ahmed Osman
Visual Imitation with SMPL Michael Black 2021-06-21 | Part of the SMPL made Simple tutorial at CVPR 2021Presenter: Angjoo Kanazawa
Clothing SMPL Michael Black 2021-06-21 | Part of the SMPL made Simple tutorial at CVPR 2021Presenter: Gerard Pons-Moll
SMPL in Human Health Michael Black 2021-06-21 | Part of the SMPL made Simple tutorial at CVPR 2021Presenter: Sergi Pujades
SMPL at Meshcapade: Digital Humans made Easy Michael Black 2021-06-21 | Part of the SMPL made Simple tutorial at CVPR 2021Presenter: Naureen Mahmood
SMPLpix: Combining SMPL and Neural Rendering Michael Black 2021-06-21 | Part of the SMPL made Simple tutorial at CVPR 2021Presenter: Sergey Prokudin
VIBE: Video Inference for human Body pose and shape Estimation Michael Black 2021-06-21 | Part of the SMPL made Simple tutorial at CVPR 2021Presenter: Mohammed Kocabas
Robots and SMPL: Combining ROS and Unreal using Julia Michael Black 2021-06-21 | Part of the SMPL Made Simple tutorial.Instructor: Nitin Saini, MPI for Intelligent Systems.
SMPL-X Application Integrations. Using SMPL-X in Blender, Unity and Unreal Michael Black 2021-06-21 | Part of the SMPL Made Simple Tutorial at CVPR 2021Instructor: Joachim Tesch, MPI for Intelligent Systems
SMPL in VR and AR Michael Black 2021-06-21 | Part of the SMPL Made Simple Tutorial at CVPR 2021Instructor: Joachim Tesch, MPI for Intelligent Systems
Datasets of and for SMPL and related models Michael Black 2021-06-21 | Various datasets that use SMPL or can be used to train SMPL and related models like SMPL-X and FLAME.Instructors: VariousAGORA: https://agora.is.tue.mpg.deAMASS: https://amass.is.tue.mpg.deBABEL: https://babel.is.tue.mpg.deBUFF: https://buff.is.tue.mpg.deCAPE: https://cape.is.tue.mpg.deCoMA: https://coma.is.tue.mpg.deDIP: https://dip.is.tue.mpg.deGRAB: https://grab.is.tue.mpg.dePROX: https://prox.is.tue.mpg.de3DPW: https://virtualhumans.mpi-inf.mpg.de/3DPWSURREAL: https://www.di.ens.fr/willow/research/surrealTUCH: https://tuch.is.tue.mpg.deVOCASET: https://voca.is.tue.mpg.de
SMPL from Images via Optimization Michael Black 2021-06-21 | Part of the SMPL made Simple tutorial at CVPR 2021Instructor: Dimitris Tzionas
Regressing SMPL from Images Michael Black 2021-06-21 | Part of the SMPL made Simple tutorial at CVPR 2021Instructor: Timo Bolkart
SMPL in Maya Michael Black 2021-06-21 | Part of the SMPL made Simple tutorial at CVPR 2021Instructor: Naureen Mahmood
TUCH: On Self-Contact and Human Pose (CVPR 2021) Michael Black 2021-06-19 | People touch their face 23 times an hour, they cross their arms and legs, put their hands on their hips, etc. While many images of people contain some form of self-contact, current 3D human pose and shape (HPS) regression methods typically fail to estimate this contact. To address this, we develop new datasets and methods that significantly improve human pose estimation with self-contact. First, we create a dataset of 3D Contact Poses (3DCP) containing SMPL-X bodies fit to 3D scans as well as poses from AMASS, which we refine to ensure good contact. Second, we leverage this to create the Mimic-The-Pose (MTP) dataset of images, collected via Amazon Mechanical Turk, containing people mimicking the 3DCP poses with self-contact. Third, we develop a novel HPS optimization method, SMPLify-XMC, that includes contact constraints and uses the known 3DCP body pose during fitting to create near ground-truth poses for MTP images. Fourth, for more image variety, we label a dataset of in-the-wild images with Discrete Self-Contact (DSC) information and use another new optimization method, SMPLify-DC, that exploits discrete contacts during pose optimization. Finally, we use our datasets during SPIN training to learn a new 3D human pose regressor, called TUCH (Towards Understanding Contact in Humans). We show that the new self-contact training data significantly improves 3D human pose estimates on withheld test data and existing datasets like 3DPW. Not only does our method improve results for self-contact poses, but it also improves accuracy for non-contact poses. The code and data are available for research purposes at https://tuch.is.tue.mpg.de.Pdf: arxiv.org/pdf/2104.03176.pdfProject: https://tuch.is.tue.mpg.de@inproceedings{Mueller:CVPR:2021, title = {On Self-Contact and Human Pose}, author = {M{\"u}ller, Lea and Osman, Ahmed A. A. and Tang, Siyu and Huang, Chun-Hao P. and Black, Michael J.}, booktitle = {Proceedings IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)}, month = jun, year = {2021}, doi = {}, month_numeric = {6}}
