LightIt: Illumination Modeling and Control for Diffusion Models (CVPR24) @MatthiasNiessner
LightIt: Illumination Modeling and Control for Diffusion Models (CVPR24)  @MatthiasNiessner
Uploaded March 2024 | Updated November 2026, 2 hours ago
Project: peter-kocsis.github.io/LightIt

We introduce LightIt, a method for explicit illumination control for image generation. Recent generative methods lack lighting control, which is crucial to numerous artistic aspects of image generation such as setting the overall mood or cinematic appearance. To overcome these limitations, we propose to condition the generation on shading and normal maps. We model the lighting with single bounce shading, which includes cast shadows. We first train a shading estimation module to generate a dataset of real-world images and shading pairs. Then, we train a control network using the estimated shading and normals as input. Our method demonstrates high-quality image generation and lighting control in numerous scenes. Additionally, we use our generated dataset to train an identity-preserving relighting model, conditioned on an image and a target shading. Our method is the first that enables the generation of images with controllable, consistent lighting and performs on par with specialized relighting state-of-the-art methods.
LightIt: Illumination Modeling and Control for Diffusion Models (CVPR24)3D Scanning & Motion Capture: 0. IntroductionDeleted video3D Scanning & Motion Capture: 9. Face Tracking & ReconstructionIntroduction to Deep Learning (I2DL 2023) - 3. Intro Neural NetworksDeleted videoPBR-SR: Mesh PBR Texture Super Resolution from 2D Image PriorsLiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D ScansDeleted videoIntroduction to Deep Learning - 7. Training Neural Networks Part 2  (Summer 2020)Deleted videoText2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (ICCV2023)
Matthias Niessner |

LightIt: Illumination Modeling and Control for Diffusion Models (CVPR'24)

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER