3D Scene Painting via Semantic Image Synthesis

Abstract

We propose a novel approach to 3D scene painting using a configurable 3D scene layout. Our approach takes a 3D scene with semantic class labels as input and trains a 3D scene painting network that synthesizes color values for the input 3D scene. We exploit an off-the-shelf 2D seman-tic image synthesis method to teach the 3D painting net-work without explicit color supervision. Experiments show that our approach produces images with geometrically cor-rect structures and supports scene manipulation, such as the change of viewpoint, object poses, and painting style. Our approach provides rich controllability to synthesized images in the aspect of 3D geometry.