Coarse-to-Fine Scene Graph Similarity Reasoning for Image-text Retrieval

Image-text retrieval is a crucial task, which targets at finding the counterparts from the opposing modalities. Scene graph based image-text retrieval methods leverage the object and predicate features to reason the cross-modal similarity, therefore increasing the retrieval accuracy. However, existing scene graph based image-text retrieval methods simply fuse the similarity calculations for features at each granularity in a single network, which only brings a slight improvement in the retrieval performance. The features of the scene graph fail to be effectively utilized. Therefore, this paper proposes a Coarse-to-Fine Scene Graph Similarity Reasoning CFSGR method to conduct coarse-grained and fine-grained cross-modal similarity reasoning, separately. CFSGR includes two networks: coarse-grained similarity reasoning network for graphs, fine-grained similarity reasoning network for objects and predicates. Moreover, CFSGR conducts local and global alignments for each feature, ensuring that the similarities at each granularity of visual and textual scene graphs are fully exploited. The evaluation and ablation study on Flickr30K demonstrates the superiority of CFSGR among the SOTA State-Of-The-Art image-text retrieval methods, and CFSGR achieves competitive results with Rsum as 506. The source code is available at https: github.com okeike CFSGR.