Weakly-supervised single-view image relighting

National University of Defense Technology

CVPR 2023

teaser image

Our method relights real objects into new scenes from single images, which also enables editing materials from diffuse to low-frequency specular with non-Lambertian rendering layers.

Abstract

We present a learning-based approach to relight a single image of Lambertian and low-frequency specular objects. Our method enables inserting objects from photographs into new scenes and relighting them under the new environment lighting, which is essential for AR applications. To relight the object, we solve both inverse rendering and re-rendering. To resolve the ill-posed inverse rendering, we propose a weakly-supervised method by a low-rank constraint. To facilitate the weakly-supervised training, we contribute Relit, a large-scale (750K images) dataset of videos with aligned objects under changing illuminations. For re-rendering, we propose a differentiable specular rendering layer to render low-frequency non-Lambertian materials under various illuminations of spherical harmonics. The whole pipeline is end-to-end and efficient, allowing for a mobile app implementation of AR object insertion. Extensive evaluations demonstrate that our method achieves state-of-the-art performance.

Pipeline

pipeline image

Overview of our method. At training time, Spec-Net separates input images into specular and diffuse branches. Spec-Net, Normal-Net and Light-Net are trained in a self-supervised manner by the Relit dataset. At inference time, inverse rendering properties are predicted to relight the object under novel lighting and material. The non-Lambertian render layers produce realistic relit images.

Videos

Here are relighting demos for videos, App demos and sample videos from the Relit dataset, whose link is at the top of this page. You are also welcome to check a 7-minute presentation of the paper at this link .

BibTeX

@InProceedings{Yi_2023_CVPR,
    author    = {Yi, Renjiao and Zhu, Chenyang and Xu, Kai},
    title     = {Weakly-Supervised Single-View Image Relighting},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {8402-8411}
}