View synthesis using image segmentation

Abstract

These days, 3D video has been gathering considerable interest and actively researched. Free-viewpoint television (FTV) is one of the latest 3D video systems allowing the user to interactively control the viewpoint and synthesize a new scene of a dynamic scene from any 3D position. To display multiple views, the virtual view synthesis is applied in FTV, which generates a synthetic image as taken from a virtual camera placed in a different point with given settings. Previous view synthesis methods, however, have limitation of virtual cameras position since a missing area appeared during synthesizing. To handle this missing area, many researchers have assumed that the virtual camera placed between a pair of reference cameras. From this assumption, it cannot synthesize virtual view which is located outside of the stereo cameras. Therefore, the aim of the present paper is to give a new approach in virtual view synthesis without these limitations. The data for the test is given at Microsoft Research. It consists of an image set captured with multiple cameras and camera matrix and depth map generated using a stereo algorithm. The algorithms contained two main modules: Separation between background and objects, and layered inpainting completing occluded areas with the layer of background. First of all, the layer separation is applied to separate the background from the images. Because the missing parts are occluded by object, it should be filled with background layers. In the method, it follows four steps. The occlusion region is detected in a depth map in the virtual camera port in the first step. Then K-mean clustering is applied for labeling the background and the foreground and searches the pixels along to the x-line from the occlusion region. Next, mean shift segmentation is used in the texture image of the virtual camera port. Finally, each segment from mean shift segmentation is labeled using step two labels. The layered inpainting is the technique to complete the missing area with only background layer from the first procedure. Indeed, the approach is based on the exemplar inpainting consist of mainly setting the priority and copying from the highest priority patch. In each step, the layer of background is considered to defined priority and search the best-matched patch. Our paper shows the result is dramatically improved compared with the previous inpainting method. Based on a ground truth database of 50 images,the average of psnr are 86.7072 for our work compared with 83.6550 for exemplar inpainting method. The quality of the results, yet, significantly depends on the layer separation. When the layer separation has an error in the background, it causes problem in layer inpainting. Therefore, the first process is very important in that reason. In conclusion, we have presented a new approach in view synthesis based on the concept missing area being filled with the background parts. With the algorithm, we can synthesize any virtual view freely without the limitation. Previous virtual view synthesis is a kind of a view