An overview of deep learning methods for image registration with focus on feature-based approaches

References

• Altwaijry, H., et al., 2016. Learning to detect and match keypoints with deep architectures. In: Proceedings of the British Machine Vision Conference (BMVC), BMVA Press, New York, 49.1–49.12.
   Google Scholar
• Andrade, N., Augusto Faria, F., and Cappabianco, F., 2018. A practical review on medical image registration: from rigid to deep learning based approaches., In: 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Parana, 10, 463–470.
   Google Scholar
• Audebert, N., et al., 2017. Deep learning for urban remote sensing. In: 2017 Joint Urban Remote Sensing Event (JURSE). IEEE, Dubai, United Arab Emirates, 1–4.
   Google Scholar
• Bai, M., et al., 2016. Exploiting semantic information and deep matching for optical flow. In: European Conference on Computer Vision. Springer, 154–170.
   Google Scholar
• Balakrishnan, G., et al., 2018. An unsupervised learning model for deformable medical image registration. In: Proceedings of the IEEE conference on computer vision and pattern recognition., Salt lake city, Utah, 9252–9260.
   Google Scholar
• Balntas, V., et al., 2016. Pn-net: conjoined triple deep network for learning local image descriptors., ArXiv, abs/1601.05030.
   Google Scholar
• Baruch, E.B. and Keller, Y., 2018. Multimodal matching using a hybrid convolutional neural network., ArXiv , abs/1810.12941.
   Google Scholar
• Bay, H., et al., 2008. Speeded-up robust features (surf). Computer Vision and Image Understanding, 110 (3), 346–359. doi:10.1016/j.cviu.2007.09.014.
   Web of Science ®Google Scholar
• Brandao, P., Mazomenos, E.B., and Stoyanov, D., 2018. Widening siamese architectures for stereo matching. Pattern Recognition Letters, 120, 75–81. doi:10.1016/j.patrec.2018.12.002
   Web of Science ®Google Scholar
• Brown, M., Hua, G., and Winder, S., 2010. Discriminative learning of local image descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33 (1), 43–57. doi:10.1109/TPAMI.2010.54.
   Web of Science ®Google Scholar
• Chen, S., et al., 2017. Local deep hashing matching of aerial images based on relative distance and absolute distance constraints. Remote Sensing, 9 (12), 1244. doi:10.3390/rs9121244.
   Web of Science ®Google Scholar
• Cheng, X., Zhang, L., and Zheng, Y., 2018. Deep similarity learning for multimodal medical images. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 6 (3), 248–252.
   Web of Science ®Google Scholar
• Chopra, S., et al., 2005. Learning a similarity metric discriminatively, with application to face verification. CVPR' 05, IEEE, 01, 539–546.
   Google Scholar
• Costante, G., et al., 2015. Exploring representation learning with cnns for frame-to-frame ego-motion estimation. IEEE Robotics and Automation Letters, 1 (1), 18–25. doi:10.1109/LRA.2015.2505717.
   Web of Science ®Google Scholar
• Dalal, N. and Triggs, B., 2005. Histograms of oriented gradients for human detection. In: international Conference on computer vision & Pattern Recognition (CVPR’05). IEEE Computer Society, vol. 1, 886–893. doi:10.1109/IEMBS.2005.1616557.
   Google Scholar
• De Vos, B.D., et al., 2017. End-to-end unsupervised deformable image registration with a convolutional neural network. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, Québec City, QC, Canada, 204–212.
   Google Scholar
• DeTone, D., Malisiewicz, T., and Rabinovich, A., 2016. Deep image homography estimation. ArXiv, abs/1606.03798.
   Google Scholar
• Dosovitskiy, A., et al., 2015. Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision., Santiago, Chile, 2758–2766.
   Google Scholar
• Eigen, D., Puhrsch, C., and Fergus, R., 2014. Depth map prediction from a single image using a multi-scale deep network. Advances in neural information processing systems, Montreal, Canada, 2366–2374.
   Google Scholar
• Elbaz, G., Avraham, T., and Fischer, A., 2017. 3d point cloud registration for localization using a deep neural network auto-encoder. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., Honolulu, HI, 2472–2481.
   Google Scholar
• Eppenhof, K.A., et al., 2018. Deformable image registration using convolutional neural networks. In: Medical Imaging 2018: Image Processing. International Society for Optics and Photonics, Houston, Texas, vol. 10574, 105740S.
   Google Scholar
• Erlik Nowruzi, F., Laganiere, R., and Japkowicz, N., 2017. Homography estimation from image pairs with hierarchical convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision., Venice, Italy, 913–920.
   Google Scholar
• Fischer, B. and Modersitzki, J., 2008. Ill-posed medicinean introduction to image registration. Inverse Problems, 24 (3), 034008. doi:10.1088/0266-5611/24/3/034008.
   Web of Science ®Google Scholar
• Fischer, P., Dosovitskiy, A., and Brox, T., 2014. Descriptor matching with convolutional neural networks: a comparison to sift. arXiv, abs/1405.5769.
   Google Scholar
• Fortun, D., Bouthemy, P., and Kervrann, C., 2015. Optical flow modeling and computation: A survey. Computer Vision and Image Understanding, 134, 1–21. doi:10.1016/j.cviu.2015.02.008
   Web of Science ®Google Scholar
• Fu, Z. and Fard, M.A., 2018. Learning confidence measures by multi-modal convolutional neural networks. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, Lake Tahoe, NV, 1321–1330.
   Google Scholar
• Garcia-Gasulla, D., et al., 2018. On the behavior of convolutional nets for feature extraction. Journal of Artificial Intelligence Research, 61, 563–592. doi:10.1613/jair.5756.
   Web of Science ®Google Scholar
• Guo, Q., Xiao, J., and Hu, X., 2018. New keypoint matching method using local convolutional features for power transmission line icing monitoring. Sensors, 18 (3), 698. doi:10.3390/s18030698.
   Web of Science ®Google Scholar
• Han, X., et al., 2015. Matchnet: unifying feature and metric learning for patch-based matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., Boston, MA, 3279–3286.
   Google Scholar
• Haskins, G., Kruger, U., and Yan, P., 2019. Deep learning in medical image registration: A survey., ArXiv, abs/1903.02026.
   Google Scholar
• He, K., et al., 2016. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition., Las Vegas, NV, 770–778.
   Google Scholar
• Herb, M., 2015. Computing the stereo matching cost with a convolutional neural network, seminar recent trends in 3 d computer vision., Technical University of Munich.
   Google Scholar
• Hu, Y., et al., 2018. Weakly-supervised convolutional neural networks for multimodal image registration. Medical Image Analysis, 49, 1–13. doi:10.1016/j.media.2018.07.002.
   PubMed Web of Science ®Google Scholar
• Huang, X., et al., 2004. Hybrid image registration based on configural matching of scale-invariant salient region features. In: 2004 Conference on Computer Vision and Pattern Recognition Workshop. IEEE, Washington, DC, 167.
   Google Scholar
• Ilg, E., et al., 2017. Flownet 2.0: evolution of optical flow estimation with deep networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition., Honolulu, HI, 2462–2470.
   Google Scholar
• Ioannidou, A., et al., 2017. Deep learning advances in computer vision with 3d data: A survey. ACM Computing Surveys (CSUR), 50 (2), 20. doi:10.1145/3071073.
   Web of Science ®Google Scholar
• Jain, P., et al., 2012. Metric and kernel learning using a linear transformation. Journal of Machine Learning Research, 13 (Mar), 519–547.
   Google Scholar
• Jeon, D.S., et al., 2018. Enhancing the spatial resolution of stereo images using a parallax prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., Salt Lake City, UT, 1721–1730.
   Google Scholar
• Jia, Y. and Darrell, T., 2011. Heavy-tailed distances for gradient based image descriptors. Advances in neural information processing systems., Granada, Spain: Curran Associates Inc., NIPS 2011, 397–405.
   Google Scholar
• Joung, S., et al., 2019. Unsupervised stereo matching using confidential correspondence consistency. IEEE Transactions on Intelligent Transportation Systems. doi:10.1109/TITS.2019.2917538.
   Google Scholar
• Kendall, A., et al., 2017. End-to-end learning of geometry and context for deep stereo regression. In: Proceedings of the IEEE International Conference on Computer Vision., Venice, Italy, 66–75.
   Google Scholar
• Keszei, A.P., Berkels, B., and Deserno, M.T., 2017. Survey of non-rigid registration tools in medicine. Journal of Digital Imaging, 30 (1), 102–116. doi:10.1007/s10278-016-9915-8.
   PubMed Web of Science ®Google Scholar
• Krizhevsky, A., Sutskever, I., and Hinton, G.E., 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems., Lake Tahoe, Nevada: Curran Associates Inc., 1097–1105.
   Google Scholar
• Kumar, B., et al., 2016. Learning local image descriptors with deep siamese and triplet convolutional networks by minimising global loss functions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., Las Vegas, NV, 5385–5394.
   Google Scholar
• Lai, W.S., Huang, J.B., and Yang, M.H., 2017. Semi-supervised learning for optical flow with generative adversarial networks. Advances in neural information processing systems., Long Beach, CA: Curran Associates Inc., 354–364.
   Google Scholar
• LeCun, Y., et al., 1995. Comparison of learning algorithms for handwritten digit recognition. In: International conference on artificial neural networks. Perth, Australia, vol. 60, 53–60.
   Google Scholar
• Li, H. and Fan, Y., 2017. Non-rigid image registration using fully convolutional networks with deep self-supervision., arXiv, abs/1709.00799.
   Google Scholar
• Liao, R., et al., 2017. An artificial agent for robust image registration. In: Thirty-First AAAI Conference on Artificial Intelligence., San Francisco, California.
   Google Scholar
• Litjens, G., et al., 2017. A survey on deep learning in medical image analysis. Medical Image Analysis, 42, 60–88. doi:10.1016/j.media.2017.07.005.
   PubMed Web of Science ®Google Scholar
• Lowe, D.G., 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60 (2), 91–110. doi:10.1023/B:VISI.0000029664.99615.94.
   Web of Science ®Google Scholar
• Lucas, B.D., et al., 1981. An iterative image registration technique with an application to stereo vision., Proceedings of the 7th International Joint Conference on Artificial Intelligence (IJCAI '81), vol. 81
   Google Scholar
• Luo, W., Schwing, A.G., and Urtasun, R., 2016. Efficient deep learning for stereo matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., Las Vegas, 5695–5703.
   Google Scholar
• Luo, Y., et al., 2018. Single view stereo matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., Salt Lake City, UT, 155–163.
   Google Scholar
• Ma, L., et al., 2019. Deep learning in remote sensing applications: A meta-analysis and review. ISPRS Journal of Photogrammetry and Remote Sensing, 152, 166–177. doi:10.1016/j.isprsjprs.2019.04.015.
   Web of Science ®Google Scholar
• Mambo, S., et al., 2018. A review on medical image registration techniques. International Journal of Computer and Information Sciences, vol. 5.
   Google Scholar
• Maurer, D. and Bruhn, A., 2018. Proflow: learning to predict optical flow. arXiv.org, cs, arXiv:1806.00800.
   Google Scholar
• Meister, S., Hur, J., and Roth, S., 2018. Unflow: unsupervised learning of optical flow with a bidirectional census loss. In: Thirty-Second AAAI Conference on Artificial Intelligence., New Orleans, Louisiana.
   Google Scholar
• Merkle, N., et al., 2017. Exploiting deep matching and sar data for the geo-localization accuracy improvement of optical satellite images. Remote Sensing, 9 (6), 586. doi:10.3390/rs9060586.
   Web of Science ®Google Scholar
• Miao, S., Wang, Z.J., and Liao, R., 2016. A cnn regression approach for real-time 2d/3d registration. IEEE Transactions on Medical Imaging, 35 (5), 1352–1363. doi:10.1109/TMI.2016.2521800.
   PubMed Web of Science ®Google Scholar
• Mikolajczyk, K. and Schmid, C., 2005. A performance evaluation of local descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27 (10), 1615–1630. doi:10.1109/TPAMI.2005.188.
   PubMed Web of Science ®Google Scholar
• Moo, Y.K., et al., 2018. Learning to find good correspondences. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., Salt Lake City, UT, 2666–2674.
   Google Scholar
• Mur-Artal, R., Montiel, J.M.M., and Tardos, J.D., 2015. Orb-slam: a versatile and accurate monocular slam system. IEEE Transactions on Robotics, 31 (5), 1147–1163. doi:10.1109/TRO.2015.2463671.
   Web of Science ®Google Scholar
• Nassar, A., et al., 2018. A deep cnn-based framework for enhanced aerial imagery registration with applications to uav geolocalization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops., Salt Lake City, UT, 1513–1523.
   Google Scholar
• Nguyen, T., et al., 2018. Unsupervised deep homography: A fast and robust homography estimation model. IEEE Robotics and Automation Letters, 3 (3), 2346–2353. doi:10.1109/LRA.2018.2809549.
   Web of Science ®Google Scholar
• Öfverstedt, J., Lindblad, J., and Sladoje, N., 2019. Fast and robust symmetric image registration based on distances combining intensity and spatial information. IEEE Transactions on Image Processing, 28 (7), 3584–3597. doi:10.1109/TIP.83.
   Web of Science ®Google Scholar
• Ono, Y., et al., 2018. Lf-net: learning local features from images. In: Advances in neural information processing systems, Montréal, Quebec, Canada: Curran Associates, Inc., 6234–6244.
   Google Scholar
• Ren, Z., et al., 2017. Unsupervised deep learning for optical flow estimation. In: Thirty-First AAAI Conference on Artificial Intelligence., San Francisco, California.
   Google Scholar
• Rocco, I., Arandjelovic, R., and Sivic, J., 2017. Convolutional neural network architecture for geometric matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., Honolulu, HI, 6148–6157.
   Google Scholar
• Salakhutdinov, R. and Hinton, G., 2007. Learning a nonlinear embedding by preserving class neighbourhood structure. Artificial intelligence and statistics., PMLR, San Juan, Puerto Rico, vol. 2, 412–419.
   Google Scholar
• Schonberger, J.L., Sinha, S.N., and Pollefeys, M., 2018. Learning to fuse proposals from multiple scanline optimizations in semi-global matching. In: Proceedings of the European Conference on Computer Vision (ECCV). 739–755. doi:10.1177/1753193417753261.
   Google Scholar
• Shan, S., et al., 2017. Unsupervised end-to-end learning for deformable medical image registration. arXiv, abs/1711.08608.
   Google Scholar
• Sheikhjafari, A., et al., 2018. Unsupervised deformable image registration with fully connected generative neural network., In MIDL 2018 Conference, Amsterdam.
   Google Scholar
• Simonovsky, M., et al., 2016. A deep metric for multimodal registration. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Athens, Greece,10–18.
   Google Scholar
• Simonyan, K., Vedaldi, A., and Zisserman, A., 2014. Learning local feature descriptors using convex optimisation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36 (8), 1573–1585. doi:10.1109/TPAMI.2014.2301163.
   PubMed Web of Science ®Google Scholar
• Simonyan, K. and Zisserman, A., 2014. Very deep convolutional networks for large-scale image recognition. In ICLR 2014, CORR, abs/1409.1556.
   Google Scholar
• Sloan, J.M., Goatman, K.A., and Siebert, J.P., 2018. Learning rigid image registration-utilizing convolutional neural networks for medical image registration., In Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 2: BIOIMAGING, ISBN 978-989-758-278-3, 89–99. doi:10.5220/0006543700890099
   Google Scholar
• Soh, Y., et al., 2014. A feature area-based image registration. Int J Comput Theory Eng, 6, 407–411. doi:10.7763/IJCTE.2014.V6.899.
   Google Scholar
• Sotiras, A., Davatzikos, C., and Paragios, N., 2013. Deformable medical image registration: A survey. IEEE Transactions on Medical Imaging, 32 (7), 1153. doi:10.1109/TMI.2013.2265603.
   PubMed Web of Science ®Google Scholar
• Sun, S., et al., 2018. Robust multimodal image registration using deep recurrent reinforcement learning. In: Asian Conference on Computer Vision. Springer, 511–526. doi:10.5021/ad.2018.30.4.511.
   Google Scholar
• Suzuki, K., 2017. Survey of deep learning applications to medical image analysis. Med Imaging Technol, 35 (4), 212–226.
   Google Scholar
• Szegedy, C., et al., 2015. Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition., Boston, MA, 1–9.
   Google Scholar
• Trzcinski, T., et al., 2012. Learning image descriptors with the boosting-trick. In: Advances in neural information processing systems. 269–277. doi:10.1177/1753193411419945.
   Google Scholar
• Tu, Z., et al., 2019. A survey of variational and cnn-based optical flow techniques. Signal Processing: Image Communication, 72, 9–24.
   Web of Science ®Google Scholar
• Uzunova, H., et al., 2017. Training cnns for image registration from few samples with model-based data augmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Quebec City, QC, Canada: Springer, 223–231.
   Google Scholar
• Vakalopoulou, M., et al., 2016. Simultaneous registration, segmentation and change detection from multisensor, multitemporal satellite image pairs. In: 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS). IEEE, Beijing, China, 1827–1830.
   Google Scholar
• Verdie, Y., et al., 2015. Tilde: a temporally invariant learned detector. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., Boston, MA, 5279–5288.
   Google Scholar
• Wang, C.W. and Chen, H.C., 2013. Improved image alignment method in application to x-ray images and biological images. Bioinformatics, 29 (15), 1879–1887. doi:10.1093/bioinformatics/btt309.
   PubMed Web of Science ®Google Scholar
• Wang, Y., et al., 2018. Occlusion aware unsupervised learning of optical flow. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., Salt Lake City, UT, 4884–4893.
   Google Scholar
• Weinzaepfel, P., et al., 2013. Deepflow: large displacement optical flow with deep matching. In: Proceedings of the IEEE International Conference on Computer Vision. 1385–1392. doi:10.1097/ALN.0b013e31828744c0.
   Google Scholar
• Wu, G., et al., 2013. Unsupervised deep feature learning for deformable registration of mr brain images. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Nagoya, Japan: Springer, 649–656.
   Google Scholar
• Wu, G., et al., 2016. Correction to scalable high-performance image registration framework by unsupervised deep feature representations learning [jul 16 1505-1516]. IEEE Transactions on Biomedical Engineering, 64 (1), 250. doi:10.1109/TBME.2016.2633139.
   Web of Science ®Google Scholar
• Xing, C., Ma, L., and Yang, X., 2016. Stacked denoise autoencoder based feature extraction and classification for hyperspectral images. Journal of Sensors, Hindawi, vol. 2016.
   Web of Science ®Google Scholar
• Yadati, P. and Namboodiri, A.M., 2017. Multiscale two-view stereo using convolutional neural networks for unrectified images. In: 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA). IEEE, 346–349. doi:10.1177/1753193417690669.
   Google Scholar
• Yang, X., et al., 2017. Quicksilver: fast predictive image registration–a deep learning approach. NeuroImage, 158, 378–396. doi:10.1016/j.neuroimage.2017.07.008.
   PubMed Web of Science ®Google Scholar
• Yang, Z., Dan, T., and Yang, Y., 2018. Multi-temporal remote sensing image registration using deep convolutional features. IEEE Access, 6, 38544–38555. doi:10.1109/ACCESS.2018.2853100
   Web of Science ®Google Scholar
• Yi, K.M., et al., 2016. Lift: learned invariant feature transform. In: European Conference on Computer Vision. Amsterdam, The Netherlands: Springer, 467–483.
   Google Scholar
• Yu, L., et al., 2018. Deep stereo matching with explicit cost aggregation sub-architecture. In: Thirty-Second AAAI Conference on Artificial Intelligence., New Orleans, LA, 7517–7524.
   Google Scholar
• Zagoruyko, S. and Komodakis, N., 2015. Learning to compare image patches via convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition., Boston, MA, 4353–4361.
   Google Scholar
• Zampieri, A., Charpiat, G., and Tarabalka, Y., 2018. Coarse to fine non-rigid registration: a chain of scale-specific neural networks for multimodal image alignment with application to remote sensing., arXiv, abs/1802.09816.
   Google Scholar
• Zbontar, J., et al., 2016. Stereo matching by training a convolutional neural network to compare image patches. Journal of Machine Learning Research, 17 (1–32), 2.
   Google Scholar
• Zhang, F., et al., 2019. Ga-net: guided aggregation net for end-to-end stereo matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)., Long Beach, CA.
   Google Scholar
• Zhang, L., et al., 2018. Temporal interpolation via motion field prediction. arXiv, abs/1804.04440.
   Google Scholar
• Zhang, X., et al., 2017. Learning discriminative and transformation covariant local feature detectors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, 6818–6826.
   Google Scholar
• Zhao, L. and Jia, K., 2015. Deep adaptive log-demons: diffeomorphic image registration with very large deformations. Computational and Mathematical Methods in Medicine, vol. 2015, 1–6, doi:10.1155/2015/836202.
   Web of Science ®Google Scholar
• Zhong, Y., Dai, Y., and Li, H., 2017. Self-supervised learning for stereo matching with self-improving ability. arXiv, abs/1709.00930.
   Google Scholar
• Zhu, A.Z., et al., 2018. Ev-flownet: self-supervised optical flow estimation for event-based cameras., In Robotics: Science and Systems 2018, Pittsburgh, Pennsylvania, doi: 10.15607/RSS.2018.XIV.062.
   Google Scholar
• Zhu, X., et al., 2017. Deep learning in remote sensing: A comprehensive review and List of Resources., IEEE Geoscience and Remote Sensing Magazine (GRSM)., 5(4), 8–36, doi:10.1109/mgrs.2017.2762307
   Google Scholar
• Zitová, B. and Flusser, J., 2003. Image registration methods: a survey. Image and Vision Computing, 21, 977–1000. doi:10.1016/S0262-8856(03)00137-9
   Web of Science ®Google Scholar