Understanding Mutual Information in Contrastive Representation Learning

Abstract

Contrastive learning has played a pivotal role in the recent success of unsupervised representation learning. It has been commonly explained with instance discrimination and a mutual information loss, and some of the fundamental explanations are based on mutual information analysis. An analysis based on mutual information, however, can be misleading. First of all, an exact quantification of mutual information over a real-world dataset is challenging. It has not been solved because we cannot access the true joint distribution function of real-world dataset before. Second, previous studies have equated the limitations of contrastive learning with them of mutual information estimation in the absence of the rigorous investigation for a relationship between them. Third, what information is actually being shared by the two views is overlooked. Without carefully examining what information is actually being shared, the interpretation can be completely misleading. In this work, we develop new methods that enable rigorous analysis of mutual information in contrastive learning. We also evaluate the accuracy of variational MI estimators across various data domains, including images and texts. Using the methods, we investigate three existing beliefs and show that they are incorrect. Based on the investigation results, we address two issues in the discussion section. In particular, we question if contrastive learning is indeed an unsupervised representation learning method because the current framework of contrastive learning relies on validation performance for tuning the augmentation design.