Correspondence Matching Algorithm Based on Mutual Information Similarity for Multi-View Video Sequences

Abstract

The multi-view video sequences are essentially used for many computer vision applications such as surveillance system. For these applications, the correspondence matching that identifies the corresponding positions of one view to another is essentially required. The correspondence matching has been fundamentally researched for a long time, however, it is still challenging for multi-view video sequences. In this dissertation, the correspondence matching algorithm and its applications for the multi-view video sequences are presented.

First, an accurate and robust similarity measure for the correspondence matching of multi-view video sequences captured by arbitrarily positioned cameras is proposed. We use an activity vector, which represents the temporal occurrence pattern of moving foreground objects at a pixel position, as an invariant feature for correspondence matching. Activity vectors are derived from a moving object detection algorithm, so it is robust to illumination changes and additive noises. Then, we devise a novel similarity measure between two activity vectors by considering the joint and individual behavior of the activity vectors. Specifically, we define random variables associated with the activity vectors and represent the similarity between them using the mutual information based similarity (MIBS) measure. Because the MIBS measure adaptively explains the behaviors between two activity vectors, it outperforms other conventional similarity measures of binary vectors especially for a correspondence matching problem.

Then, the framework for finding correspondence matching between two multi-view surveillance sequences is proposed. In order to achieve a more accurate and robust inter-view homography, three practical techniques are utilized. The first technique is the adaptive activity area refinement which represents actual ground regions touched by foreground objects moving on the ground plane. It reduces the discrepancy between objects areas and actual ground surfaces, so that the activity vectors can effectively feature geometry surfaces in the scenes. In addition, we propose the consistent pixel positions on which the MIBS measure is reliably evaluated. At consistent pixel positions, the maximum MIBS criterion is satisfied backward and forward, therefore, we can yield more accurate correspondence matchings. Finally, the correspondence at multiple pixel positions are determined by minimizing a matching cost function associated with the MIBS measure and structure preservation terms.

The proposed correspondence matching algorithm is robust to various positions of cameras and illumination/color differences between cameras. Moreover, the proposed MIBS measure reliably represents the similarity of two binary vectors even under the additive noises. Therefore, the results of proposed algorithm demonstrate the correspondences between two different views are more accurately and reliably estimated than the conventional state-of-the-art algorithm with a relatively small computational complexity. These results indicate that the proposed algorithm is a very promising technique for various multi-view video applications for a visual surveillance such as homograpy estimation and panoramic view synthesis.