Query generation for multimodal documents

Abstract

© 2021 Association for Computational LinguisticsThis paper studies the problem of generating likely queries for multimodal documents with images. Our application scenario is enabling efficient first-stage retrieval of relevant documents, by attaching generated queries to documents before indexing. We can then index this expanded text to efficiently narrow down to candidate matches using inverted index, so that expensive reranking can follow. Our evaluation results show that our proposed multimodal representation meaningfully improves relevance ranking. More importantly, our framework can achieve the state of the art in the first-stage retrieval scenarios.