A news picture geo-localization pipeline based on deep learning and street view images

References

• Arandjelović, Relja, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. 2016. “NetVLAD: CNN Architecture for Weakly Supervised Place Recognition.” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, 5297–5307. IEEE. doi:10.1109/CVPR.2016.572.
   Google Scholar
• Arandjelović, Relja, and Andrew Zisserman. 2013. “All About VLAD.” 2013 IEEE Conference on Computer Vision and Pattern Recognition, 1578–1585. IEEE. doi:10.1109/CVPR.2013.207.
   Google Scholar
• Babenko, Artem, and Victor Lempitsky. 2015. “Aggregating Deep Convolutional Features for Image Retrieval.” 2015 IEEE International Conference on Computer Vision (ICCV), 1269–1277. IEEE. doi:10.1109/ICCV.2015.150.
   Google Scholar
• Berton, Gabriele, Riccardo Mereu, Gabriele Trivigno, Carlo Masone, Gabriela Csurka, Torsten Sattler, and Barbara Caputo. 2022. “Deep Visual Geo-Localization Benchmark.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, 5396–5407. IEEE.
   Google Scholar
• Bhuvana, N., and I. Arul Aram. 2019. “Facebook and Whatsapp as Disaster Management Tools During the Chennai (India) Floods of 2015.” International Journal of Disaster Risk Reduction 39: 101135. doi:10.1016/j.ijdrr.2019.101135.
   Web of Science ®Google Scholar
• Bruno, Arcanjo, Ferrarini Bruno, Milford Michael, McDonald-Maier Klaus D., and Ehsan Shoaib. 2022. “An Efficient and Scalable Collection of Fly-Inspired Voting Units for Visual Place Recognition in Changing Environments.” IEEE Robotics and Automation Letters 7 (2): 2527–2534. doi:10.1109/lra.2022.3140827.
   Web of Science ®Google Scholar
• Cheng, Liang, Sensen Chu, Wenwen Zong, Shuyi Li, Jie Wu, and Manchun Li. 2017. “Use of Tencent Street View Imagery for Visual Perception of Streets.” ISPRS International Journal of Geo-Information 6 (9): 265. doi:10.3390/ijgi6090265.
   Web of Science ®Google Scholar
• Cheng, Liang, Yi Yuan, Nan Xia, Song Chen, Yanming Chen, Kang Yang, Lei Ma, and Manchun Li. 2018. “Crowd-Sourced Pictures Geo-Localization Method Based on Street View Images and 3D Reconstruction.” ISPRS Journal of Photogrammetry and Remote Sensing 141: 72–85. doi:10.1016/j.isprsjprs.2018.04.006.
   Web of Science ®Google Scholar
• Chu, Tianyou, Yumin Chen, Liheng Huang, Zhiqiang Xu, and Huangyuan Tan. 2020. “A Grid Feature-Point Selection Method for Large-Scale Street View Image Retrieval Based on Deep Local Features.” Remote Sensing 12 (23): 3978. doi:10.3390/rs12233978.
   Web of Science ®Google Scholar
• Feng, Yansong, and Mirella Lapata. 2013. “Automatic Caption Generation for News Images.” IEEE Transactions on Pattern Analysis and Machine Intelligence 35 (4): 797–812. doi:10.1109/tpami.2012.118.
   PubMed Web of Science ®Google Scholar
• Ge, Yixiao, Haibo Wang, Feng Zhu, Rui Zhao, and Hongsheng Li. 2020. “Self-Supervising Fine-Grained Region Similarities for Large-Scale Image Localization.” In Computer Vision – ECCV 2020, edited by Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, 369–386. London: Springer. doi:10.1007/978-3-030-58548-8_22.
   Google Scholar
• He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. “Deep Residual Learning for Image Recognition.” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778. IEEE. doi:10.1109/CVPR.2016.90.
   Google Scholar
• Jégou, Hervé, Matthijs Douze, Cordelia Schmid, and Patrick Pérez. 2010. “Aggregating Local Descriptors Into a Compact Image Representation.” 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 3304–3311. IEEE. doi:10.1109/CVPR.2010.5540039.
   Google Scholar
• Jing, Fengrui, Lin Liu, Suhong Zhou, Jiangyu Song, Linsen Wang, Hanlin Zhou, Yiwen Wang, and Ruofei Ma. 2021. “Assessing the Impact of Street-View Greenery on Fear of Neighborhood Crime in Guangzhou, China.” International Journal of Environmental Research and Public Health 18 (1): 311. doi:10.3390/ijerph18010311.
   PubMed Web of Science ®Google Scholar
• Johnson, Jeff, Matthijs Douze, and Hervé Jégou. 2019. “Billion-Scale Similarity Search with GPUs.” IEEE Transactions on Big Data 7 (3): 535–547. doi:10.1109/tbdata.2019.2921572.
   Web of Science ®Google Scholar
• Kalantidis, Yannis, Clayton Mellina, and Simon Osindero. 2016. “Cross-Dimensional Weighting for Aggregated Deep Convolutional Features.” In Computer Vision – ECCV 2016 Workshops, edited by Gang Hua and Hervé Jégou, 685–701, Amsterdam, The Netherlands. ECCV. doi:10.1007/978-3-319-46604-0_48.
   Google Scholar
• Kelly, Cheryl M., Jeffrey S. Wilson, Elizabeth A. Baker, Douglas K. Miller, and Mario Schootman. 2013. “Using Google Street View to Audit the Built Environment: Inter-Rater Reliability Results.” Annals of Behavioral Medicine 45 (1): 108–112. doi:10.1007/s12160-012-9419-9.
   Web of Science ®Google Scholar
• Kim, Hyo Jin, Enrique Dunn, and Jan-Michael Frahm. 2017. “Learned Contextual Feature Reweighting for Image Geo-Localization.” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3251–3260. IEEE. doi:10.1109/CVPR.2017.346.
   Google Scholar
• Lehtola, Ville V., Juho-Pekka Virtanen, Antero Kukko, Harri Kaartinen, and Hannu Hyyppä. 2015. “Localization of Mobile Laser Scanner Using Classical Mechanics.” ISPRS Journal of Photogrammetry and Remote Sensing 99: 25–29. doi:10.1016/j.isprsjprs.2014.10.008.
   Web of Science ®Google Scholar
• Li, Xiaojiang. 2021. “Examining the Spatial Distribution and Temporal Change of the Green View Index in New York City Using Google Street View Images and Deep Learning.” Environment and Planning B: Urban Analytics and City Science 48 (7): 2039–2054. doi:10.1177/2399808320962511.
   Google Scholar
• Li, Xin, Liang Li, Xiangrong Wang, Qing Lin, Danzi Wu, Yang Dong, and Shuang Han. 2021. “Visual Quality Evaluation Model of an Urban River Landscape Based on Random Forest.” Ecological Indicators 133: 108381. doi:10.1016/j.ecolind.2021.108381.
   Web of Science ®Google Scholar
• Li, Xiaojiang, Carlo Ratti, and Ian Seiferling. 2017. “Mapping Urban Landscapes Along Streets Using Google Street View.” In Advances in Cartography and Giscience, edited by M. P. Peterson, 341–356. doi:10.1007/978-3-319-57336-6_24.
   Google Scholar
• Lin, Binbin, Lei Zou, Nick Duffield, Ali Mostafavi, Heng Cai, Bing Zhou, Jian Tao, Mingzheng Yang, Debayan Mandal, and Joynal Abedin. 2022. “Revealing the Linguistic and Geographical Disparities of Public Awareness to Covid-19 Outbreak Through Social Media.” International Journal of Digital Earth 15 (1): 868–889. doi:10.1080/17538947.2022.2070677.
   Web of Science ®Google Scholar
• Liu, Liu, and Hongdong Li. 2019. “Lending Orientation to Neural Networks for Cross-View Geo-Localization.” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5617–5626. IEEE. doi:10.1109/CVPR.2019.00577.
   Google Scholar
• Lock, Oliver, Tomasz Bednarz, and Christopher Pettit. 2021. “The Visual Analytics of Big, Open Public Transport Data – A Framework and Pipeline for Monitoring System Performance in Greater Sydney.” Big Earth Data 5 (1): 134–159. doi:10.1080/20964471.2020.1758537.
   Google Scholar
• Meng, Lingchao, Kuo-Hsun Wen, Zhijie Zeng, Richard Brewin, Xiaolei Fan, and Qiong Wu. 2020. “The Impact of Street Space Perception Factors on Elderly Health in High-Density Cities in Macau—Analysis Based on Street View Images and Deep Learning Technology.” Sustainability 12 (5): 1799. doi:10.3390/su12051799.
   Web of Science ®Google Scholar
• Ning, Huan, Zhenlong Li, Xinyue Ye, Shaohua Wang, Wenbo Wang, and Xiao Huang. 2022. “Exploring the Vertical Dimension of Street View Image Based on Deep Learning: A Case Study on Lowest Floor Elevation Estimation.” International Journal of Geographical Information Science 36 (7): 1317–1342. doi:10.1080/13658816.2021.1981334.
   Web of Science ®Google Scholar
• Nizzoli, Leonardo, Marco Avvenuti, Maurizio Tesconi, and Stefano Cresci. 2020. “Geo-Semantic-Parsing: AI-Powered Geoparsing by Traversing Semantic Knowledge Graphs.” Decision Support Systems 136: 113346. doi:10.1016/j.dss.2020.113346.
   Web of Science ®Google Scholar
• Noh, Hyeonwoo, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han. 2017. “Large-Scale Image Retrieval with Attentive Deep Local Features.” 2017 IEEE International Conference on Computer Vision (ICCV), 3476–3485. IEEE. doi:10.1109/ICCV.2017.374.
   Google Scholar
• Park, Chanho, Hyunwoo Cho, Sangheon Park, Sung-Uk Jung, and Suwon Lee. 2022. “Strategy for Creating AR Applications in Static and Dynamic Environments Using SLAM- and Marker Detector-Based Tracking.” CMES-Computer Modeling in Engineering & Sciences 131 (1): 529–549. doi:10.32604/cmes.2022.019214.
   Web of Science ®Google Scholar
• Perronnin, Florent, and Christopher Dance. 2007. “Fisher Kernels on Visual Vocabularies for Image Categorization.” 2007 IEEE Conference on Computer Vision and Pattern Recognition, 1–8. IEEE. doi:10.1109/CVPR.2007.383266.
   Google Scholar
• Radenović, Filip, Giorgos Tolias, and Ondřej Chum. 2019. “Fine-Tuning CNN Image Retrieval with No Human Annotation.” IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (7): 1655–1668. IEEE. doi:10.1109/tpami.2018.2846566.
   PubMed Web of Science ®Google Scholar
• Sarlin, Paul-Edouard, Ajaykumar Unagar, Mans Larsson, Hugo Germain, Carl Toft, Viktor Larsson, Marc Pollefeys, Vincent Lepetit, Lars Hammarstrand, and Fredrik Kahl. 2021. “Back to the Feature: Learning Robust Camera Localization from Pixels to Pose.” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3246–3256. IEEE. doi:10.1109/CVPR46437.2021.00326.
   Google Scholar
• Sattler, Torsten, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, and Josef Sivic. 2018. “Benchmarking 6dof Outdoor Visual Localization in Changing Conditions.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8601–8610. IEEE. doi:10.1109/CVPR.2018.00897.
   Google Scholar
• Shang, Lanyu, Yang Zhang, Christina Youn, and Dong Wang. 2022. “SAT-Geo: A Social Sensing Based Content-Only Approach to Geolocating Abnormal Traffic Events Using Syntax-Based Probabilistic Learning.” Information Processing & Management 59 (2): 16. doi:10.1016/j.ipm.2021.102807.
   Web of Science ®Google Scholar
• Shi, Yujiao, Xin Yu, Liu Liu, Dylan Campbell, Piotr Koniusz, and Hongdong Li. 2022. “Accurate 3-DoF Camera Geo-Localization via Ground-to-Satellite Image Matching.” arXiv preprint arXiv:2203.14148.
   Google Scholar
• Sivic, Josef, and Andrew Zisserman. 2003. “Video Google: A Text Retrieval Approach to Object Matching in Videos.” Proceedings of the Ninth IEEE International Conference on Computer Vision. Vol. 2, 1470–1477. IEEE. doi:10.1109/ICCV.2003.1238663.
   Google Scholar
• Skripnikov, A., N. Wagner, J. Shafer, M. Beck, E. Sherwood, and M. Burke. 2021. “Using Localized Twitter Activity to Assess Harmful Algal Bloom Impacts of Karenia Brevis in Florida, USA.” Harmful Algae 110: 102118. doi:10.1016/j.hal.2021.102118.
   PubMed Web of Science ®Google Scholar
• Song, Chenguang, Nianwen Ning, Yunlei Zhang, and Bin Wu. 2021. “A Multimodal Fake News Detection Model Based on Crossmodal Attention Residual and Multichannel Convolutional Neural Networks.” Information Processing & Management 58 (1): 102437. doi:10.1016/j.ipm.2020.102437.
   Web of Science ®Google Scholar
• Tancik, Matthew, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. 2022. “Block-NeRF: Scalable Large Scene Neural View Synthesis.” arXiv preprint arXiv:2202.05263. doi:10.48550/arXiv.2202.05263.
   Google Scholar
• Tian, Yurun, Vassileios Balntas, Tony Ng, Axel Barroso-Laguna, Yiannis Demiris, and Krystian Mikolajczyk. 2020. “D2D: Keypoint Extraction with Describe to Detect Approach.” arXiv preprint arXiv:2005.13605. doi:10.48550/arXiv.2005.13605.
   Google Scholar
• Toker, Aysim, Qunjie Zhou, Maxim Maximov, and Laura Leal-Taixe. 2021. “Coming Down to Earth: Satellite-to-Street View Synthesis for Geo-Localization.” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6484–6493. IEEE. doi:10.1109/CVPR46437.2021.00642.
   Google Scholar
• Tolias, Giorgos, Yannis Avrithis, and Hervé Jegou. 2016. “Image Search with Selective Match Kernels: Aggregation Across Single and Multiple Images.” International Journal of Computer Vision 116 (3): 247–261. doi:10.1007/s11263-015-0810-4.
   Web of Science ®Google Scholar
• Tolias, Giorgos, Sicre Ronan, and Jégou Hervé. 2015. “Particular Object Retrieval with Integral Max-Pooling of CNN Activations.” arXiv preprint arXiv:1511.05879. doi:10.48550/arXiv.1511.05879.
   Google Scholar
• Tolias, Giorgos, Jenicek Tomas, and Chum Ondřej. 2020. “Learning and Aggregating Deep Local Descriptors for Instance-Level Recognition.” Computer Vision – ECCV 2020, edited by Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, 460–477. ECCV. doi:10.1007/978-3-030-58452-8_27.
   Google Scholar
• Torii, Akihiko, Josef Sivic, Masatoshi Okutomi, and Tomas Pajdla. 2015. “Visual Place Recognition with Repetitive Structures.” IEEE Transactions on Pattern Analysis and Machine Intelligence 37 (11): 2346–2359. doi:10.1109/tpami.2015.2409868.
   PubMed Web of Science ®Google Scholar
• Wang, Tingyu, Zhedong Zheng, Chenggang Yan, Jiyong Zhang, Yaoqi Sun, Bolun Zheng, and Yi Yang. 2022. “Each Part Matters: Local Patterns Facilitate Cross-View Geo-Localization.” IEEE Transactions on Circuits and Systems for Video Technology 32 (2): 867–879. doi:10.1109/tcsvt.2021.3061265.
   Web of Science ®Google Scholar
• Wilson, Daniel, Xiaohan Zhang, Waqas Sultani, and Safwan Wshah. 2021. “Visual and Object Geo-localization: A Comprehensive Survey.” arXiv preprint arXiv:2112.15202. doi:10.48550/arXiv.2112.15202.
   Google Scholar
• Xia, Yixi, Nobuyoshi Yabuki, and Tomohiro Fukuda. 2021. “Development of a System for Assessing the Quality of Urban Street-Level Greenery Using Street View Images and Deep Learning.” Urban Forestry & Urban Greening 59: 126995. doi:10.1016/j.ufug.2021.126995.
   Web of Science ®Google Scholar
• Yang, Kehua, Shaosong Long, Wei Zhang, Jiqing Yao, and Jing Liu. 2020. “Personalized News Recommendation Based on the Text and Image Integration.” Computers, Materials & Continua 64 (1): 557–570. doi:10.32604/cmc.2020.09907.
   Web of Science ®Google Scholar
• Yang, Tsun-Yi, Duy-Kien Nguyen, Huub Heijnen, and Vassileios Balntas. 2020. “Ur2kid: Unifying Retrieval, Keypoint Detection, and Keypoint Description without Local Correspondence Supervision.” arXiv preprint arXiv: 2001.07252. doi:10.48550/arXiv.2001.07252.
   Google Scholar
• Zhang, Fan, and Yu Liu. 2021. “Street View Imagery: Methods and Applications Based on Artificial Intelligence.” Journal of Remote Sensing 25 (5): 1043–1054. doi:10.11834/jrs.20219341.
   Google Scholar
• Zheng, Xin, Jialong Han, and Aixin Sun. 2018. “A Survey of Location Prediction on Twitter.” IEEE Transactions on Knowledge and Data Engineering 30 (9): 1652–1671. doi:10.1109/tkde.2018.2807840.
   Web of Science ®Google Scholar
• Zheng, Zhedong, Yunchao Wei, and Yi Yang. 2020. “University-1652: A Multi-View Multi-Source Benchmark for Drone-Based Geo-Localization.” Proceedings of the 28th ACM International Conference on Multimedia, 1395–1403. ACM. doi:10.1145/3394171.3413896.
   Google Scholar
• Zhi, Lanyue, Zhifeng Xiao, Yonggang Qiang, and Linjun Qian. 2021. “Street-Level Image Localization Based on Building-Aware Features via Patch-Region Retrieval Under Metropolitan-Scale.” Remote Sensing 13 (23): 4876. doi:10.3390/rs13234876.
   Web of Science ®Google Scholar
• Zhu, Yingying, Bin Sun, Xiufan Lu, and Sen Jia. 2022. “Geographic Semantic Network for Cross-View Image Geo-Localization.” IEEE Transactions on Geoscience and Remote Sensing 60: 1–14. doi:10.1109/tgrs.2021.3121337.
   Web of Science ®Google Scholar
• Zou, Lei, Nina S. N. Lam, Shayan Shams, Heng Cai, Michelle A. Meyer, Seungwon Yang, Kisung Lee, Seung-Jong Park, and Margaret A. Reams. 2019. “Social and Geographical Disparities in Twitter Use During Hurricane Harvey.” International Journal of Digital Earth 12 (11): 1300–1318. doi:10.1080/17538947.2018.1545878.
   Web of Science ®Google Scholar