ViewDiff: 3D-Consistent Image Generation with Text-To-Image Models @MatthiasNiessner
ViewDiff: 3D-Consistent Image Generation with Text-To-Image Models  @MatthiasNiessner
Uploaded March 2024 | Updated November 2026, 2 days ago
Project: lukashoel.github.io/ViewDiff

3D asset generation is getting massive amounts of attention inspired by the recent success on text-guided 2D content creation. Existing text-to-3D methods use pretrained text-to-image diffusion models in an optimization problem or fine-tune them on synthetic data, which often results in non-photorealistic 3D objects without backgrounds. In this paper, we present a method that leverages pretrained text-to-image models as a prior, and learn to generate multi-view images in a single denoising process from real-world data. Concretely, we propose to integrate 3D volume-rendering and cross-frame-attention layers into each block of the existing U-Net network of the text-to-image model. Moreover, we design an autoregressive generation that renders more 3D-consistent images at any viewpoint. We train our model on real-world datasets of objects and showcase its capabilities to generate instances with a variety of high-quality shapes and textures in authentic surroundings. Compared to the existing methods, the results generated by our method are consistent, and have favorable visual quality (-30% FID, -37% KID).
ViewDiff: 3D-Consistent Image Generation with Text-To-Image ModelsMesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation (ECCV24)CVPR 2020 Paper Compilation - TUM Visual Computing Lab & CollaboratorsScanRefer: 3D Object Localization in RGB-D Scans using Natural Language (ECCV 2020)FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human FacesGenRecon: Bridging Generative Priors for Multi-View 3D Scene ReconstructionPrEditor3D: Fast and Precise 3D Shape EditingIntroduction to Deep Learning - 5.Scaling Optimization and Stochastic Gradient Descent (Summer 2020)Matthias Niessner Live StreamL3DG: Latent 3D Gaussian Diffusion (SIGGRAPH Asia24)MeshGPT: Generating Triangle Meshes with Decoder-Only TransformersIMU2Face: Real-time Gesture-driven Facial Reenactment
Matthias Niessner |

ViewDiff: 3D-Consistent Image Generation with Text-To-Image Models

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER