MT-Net: A heterogeneous image matching method based on modality transformation

This paper focuses on the challenges in heterogeneous image matching. The matching accuracy of heterogeneous image pairs is lower than that of homogeneous image pairs. Existing methods have attempted adaptive improvements to address the challenges in heterogeneous image matching, but accuracy still needs improvement. This is because heterogeneous image pairs exhibit significant differences, primarily due to their distinct imaging mechanisms. Regarding this issue, we propose an end-to-end hybrid framework that employs modality transformation for heterogeneous image matching. First, a modality transformation method based on style transfer is proposed to convert heterogeneous image pairs into pseudo-homologous image pairs. Second, we extract multiscale and multilevel discriminative features from the pseudo-homologous image pairs to enhance the repeatability and discrimination of keypoints. Third, a unified matching loss is proposed to optimize the method for generating pseudo- homologous images. This loss function improves the performance of the modality transformation module and even the entire network. The experiments indicate that the proposed MT-Net improves the mean matching result by 0.9 –3.5 .