Detailed Information

Weakly Supervised Referring Image Segmentation with Intra-Chunk and Inter-Chunk Consistency

Cited 0 time in Web of Science Cited 0 time in Scopus

Lee, Jung Beom; Lee, Sung Jin; Nam, Jin Seok; Yu, Seung Hak; Do, Jae Young; Taghavi, Tara

Issue Date
Institute of Electrical and Electronics Engineers Inc.
Proceedings of the IEEE International Conference on Computer Vision, pp.21813-21824
Referring image segmentation aims to localize the object in an image referred by a natural language expression. Most previous studies learn referring image segmentation with a large-scale dataset containing segmentation labels, but they are costly. We present a weakly supervised learning method for referring image segmentation that only uses readily available image-text pairs. We first train a visual-linguistic model for image-text matching and extract a visual saliency map through Grad-CAM to identify the image regions corresponding to each word. However, we found two major problems with Grad- CAM. First, it lacks consideration of critical semantic relationships between words. We tackle this problem by modeling the relationship between words through intra-chunk and inter-chunk consistency. Second, Grad-CAM identifies only small regions of the referred object, leading to low recall. Therefore, we refine the localization maps with self-attention in Transformer and unsupervised object shape prior. On three popular benchmarks (RefCOCO, RefCOCO+, G-Ref), our method significantly outperforms recent comparable techniques. We also show that our method is applicable to various levels of supervision and obtains better performance than recent methods.
Files in This Item:
There are no files associated with this item.
Appears in Collections:

Related Researcher

  • College of Engineering
  • Department of Electrical and Computer Engineering
Research Area AI 애플리케이션을 위한 알고리즘-시스템 공동 설계, AI-powered Big Data Management, Generative AI, Large Language Model, ML, 고성능 대규모 AI 데이터 분석 및 처리, 모달 AI


Item View & Download Count

  • mendeley

Items in S-Space are protected by copyright, with all rights reserved, unless otherwise indicated.