Analysis of Visual and Musical Tempo Congruence in Commercial Motion Pictures

Abstract

The amount of multimedia including movies in existence has become so extensive that dealing with these data cannot be performed easily. Previous studies have explored the relationship between visual and auditory signals in multimedia data using CV (Computer Vision) and MIR (Music Information Retrieval) techniques. This thesis focuses on the temporal correlation between music and video signals in commercial motion pictures. Tempo means the speed of music the rate of activity. Musical tempo is created by temporal periodicity and visual speed is created through spatial variance or salient event intervals. This means another indicator to combine musical and visual tempo scale is needed. We refer to this as Tempo Index. In this research, multimodal tempo relationship is analyzed by estimating each tempo index of training data. Visual tempo derived from combination of motion vector magnitude, shot length, and visual activity index. Musical tempo is estimated by calculating beat intervals. To match visual and musical tempo, two indices are converted to tempo indices referring to study of Motion tempo and multisensory integration. All features used as the ground truth are compensated manually after automated computation. The experimental result shows that there exists a linear relationship between proposed two tempo indicators. This global tempo congruency is evaluated by analyzing other film data and user survey. The result of this study is expected to provide clues for further multimedia contents research.