Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking @MatthiasNiessner
Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking  @MatthiasNiessner
Uploaded January 2024 | Updated November 2026, 3 days ago
Project: vveicao.github.io/projects/Motion2VecSets
Paper: arxiv.org/abs/2401.06614

We introduce Motion2VecSets, a 4D diffusion model for dynamic surface reconstruction from point cloud sequences. While existing state-of-the-art methods have demonstrated success in reconstructing non-rigid objects using neural field representations, conventional feed-forward networks encounter challenges with ambiguous observations from noisy, partial, or sparse point clouds. To address these challenges, we introduce a diffusion model that explicitly learns the shape and motion distribution of non-rigid objects through an iterative denoising process of compressed latent representations. The diffusion-based prior enables more plausible and probabilistic reconstructions when handling ambiguous inputs. We parameterize 4D dynamics with latent vector sets instead of using a global latent. This novel 4D representation allows us to learn local surface shape and deformation patterns, leading to more accurate non-linear motion capture and significantly improving generalizability to unseen motions and identities. For more temporal-coherent object tracking, we synchronously denoise deformation latent sets and exchange information across multiple frames. To avoid the computational overhead, we design an interleaved space and time attention block to alternately aggregate deformation latents along spatial and temporal domains. Extensive comparisons against the state-of-the-art methods demonstrate the superiority of our Motion2VecSets in 4D reconstruction from various imperfect observations, notably achieving a 19% improvement in Intersection over Union (IoU) compared to CaDex for reconstructing unseen individuals from sparse point clouds on the DeformingThings4D-Animals dataset.
Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and TrackingDeleted videoTUM AI Lecture Series - Perceiving Humans in the 3D World (Angjoo Kanazawa)3D Scanning & Motion Capture: 7. Non-Rigid Tracking & ReconstructionPiGraphs: Learning Interaction Snapshots from Observations (SIGGRAPH 2016)Local Implicit Grid Representations for 3D Scenes (CVPR 2020)What are Deep Fakes? How do we detect them? (CVPR2020 Media Forensics Workshop)Introduction to Deep Learning - 9. Convolutional Neural Networks (Summer 2020)Deleted videoLearning 3D Scene Priors with 2D Supervision (CVPR2023)Deleted videoDeleted video
Matthias Niessner |

Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER