World Reconstruction from Inconsistent Views @MatthiasNiessner
World Reconstruction from Inconsistent Views  @MatthiasNiessner
Uploaded March 2026 | Updated November 2026, 1 day ago
lukashoel.github.io/video_to_world

Abstract: Video diffusion models generate high-quality and diverse worlds; however, individual frames often lack 3D consistency across the output sequence, which makes the reconstruction of 3D worlds difficult. To this end, we propose a new method that handles these inconsistencies by non-rigidly aligning the video frames into a globally-consistent coordinate frame that produces sharp and detailed pointcloud reconstructions. First, a geometric foundation model lifts each frame into a pixel-wise 3D pointcloud, which contains unaligned surfaces due to these inconsistencies. We then propose a tailored non-rigid iterative frame-to-model ICP to obtain an initial alignment across all frames, followed by a global optimization that further sharpens the pointcloud. Finally, we leverage this pointcloud as initialization for 3D reconstruction and propose a novel inverse deformation rendering loss to create high quality and explorable 3D environments from inconsistent views. We demonstrate that our 3D scenes achieve higher quality than baselines, effectively turning video models into 3D-consistent world generators.
World Reconstruction from Inconsistent ViewsTUM AI Lecture Series - Deep Surface Meshes (Pascal Fua)SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans (CVPR)Deleted videoNeural Voice Puppetry: Audio-driven Facial Reenactment (ECCV 2020)IGNOR: Image-guided Neural Object Rendering (ICLR 2020)NPGA: Neural Parametric Gaussian Avatars (SIGGRAPH Asia24)ViewAL: Active Learning with Viewpoint Entropy for Semantic Segmentation (CVPR 2020)3DGS-LM: Faster Gaussian-Splatting Optimization with Levenberg-MarquardtClipFace: Text-guided Editing of Textured 3D Morphable Models (SIGGRAPH23)ICCV 2023 Paper Compilation - TUM Visual Computing Lab & CollaboratorsEnd2End Multi-View Feature Matching with Differentiable Pose Optimization (ICCV2023)
Matthias Niessner |

World Reconstruction from Inconsistent Views

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER