Identifying disaster related social media for rapid response: a visual-textual fused CNN architecture

References

• Ashktorab, Z., C. Brown, M. Nandi, and A. Culotta. 2014. “Tweedr: Mining Twitter to Inform Disaster Response.” ISCRAM, May.
   Google Scholar
• Avgerinakis, K., A. Moumtzidou, S. Andreadis, E. Michail, I. Gialampoukidis, S. Vrochidis, and I. Kompatsiaris. 2017. “Visual and Textual Analysis of Social Media and Satellite Images for Flood Detection@ Multimedia Satellite Task MediaEval 2017.” Working Notes Proc. MediaEval Workshop, p. 2.
   Google Scholar
• Avvenuti, M., S. Cresci, F. Del Vigna, T. Fagni, and M. Tesconi. 2018. “CrisMap: A Big Data Crisis Mapping System Based on Damage Detection and Geoparsing.” Information Systems Frontiers, 1–19.
   Web of Science ®Google Scholar
• Avvenuti, M., S. Cresci, F. Del Vigna, and M. Tesconi. 2016. “Impromptu Crisis Mapping to Prioritize Emergency Response.” Computer 49 (5): 28–37.
   Web of Science ®Google Scholar
• Avvenuti, M., S. Cresci, A. Marchetti, C. Meletti, and M. Tesconi. 2014. “EARS (Earthquake Alert and Report System): A Real Time Decision Support System for Earthquake Crisis Management.” Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, August, 1749–1758.
   Google Scholar
• Avvenuti, M., S. Cresci, L. Nizzoli, and M. Tesconi. 2018. “GSP (Geo-Semantic-Parsing): Geoparsing and Geotagging with Machine Learning on Top of Linked Data.” European Semantic Web Conference, Springer, Cham, June, 17–32.
   Google Scholar
• Bakillah, M., R. Y. Li, and S. H. Liang. 2015. “Geo-located Community Detection in Twitter with Enhanced Fast-greedy Optimization of Modularity: The Case Study of Typhoon Haiyan.” International Journal of Geographical Information Science 29 (2): 258–279.
   Web of Science ®Google Scholar
• Bar, Y., I. Diamant, L. Wolf, and H. Greenspan. 2015. “Deep Learning with Non-medical Training used for Chest Pathology Identification.” Medical Imaging 2015: Computer-Aided Diagnosis, International Society for Optics and Photonics, March, Vol. 9414, 94140V.
   Google Scholar
• Bischke, B., P. Bhardwaj, A. Gautam, P. Helber, D. Borth, and A. Dengel. 2017. “Detection of Flooding Events in Social Multimedia and Satellite Imagery Using Deep Neural Networks.” Working Notes Proc. MediaEval Workshop, 2.
   Google Scholar
• Bojanowski, P., E. Grave, A. Joulin, and T. Mikolov. 2017. “Enriching Word Vectors with Subword Information.” Transactions of the Association for Computational Linguistics 5: 135–146.
   Google Scholar
• Bosch, A., A. Zisserman, and X. Munoz. 2007. “Image Classification Using Random Forests and Ferns.” Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference, IEEE, October, 1–8.
   Google Scholar
• Chapelle, O., P. Haffner, and V. N. Vapnik. 1999. “Support Vector Machines for Histogram-Based Image Classification.” IEEE Transactions on Neural Networks 10 (5): 1055–1064.
   PubMed Web of Science ®Google Scholar
• Cheng, Z., J. Caverlee, and K. Lee. 2010. “You are Where You Tweet: A Content-based Approach to Geo-locating Twitter Users.” Proceedings of the 19th ACM International Conference on Information and Knowledge Management, ACM, October, 759–768.
   Google Scholar
• Ciresan, D. C., U. Meier, J. Masci, L. Maria Gambardella, and J. Schmidhuber. 2011. “Flexible, High Performance Convolutional Neural Networks for Image Classification.” IJCAI Proceedings-International Joint Conference on Artificial Intelligence, Vol. 22, 1237, July.
   Google Scholar
• Collobert, R., J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. 2011. “Natural Language Processing (Almost) from Scratch.” Journal of Machine Learning Research 12: 2493–2537.
   Web of Science ®Google Scholar
• Crampton, J. W. 2009. “Cartography: Maps 2.0.” Progress in Human Geography 33 (1): 91–100.
   Web of Science ®Google Scholar
• Dredze, M. 2012. “How Social Media Will Change Public Health.” IEEE Intelligent Systems 27 (4): 81–84.
   Web of Science ®Google Scholar
• Earle, P. S., D. C. Bowden, and M. Guy. 2012. “Twitter Earthquake Detection: Earthquake Monitoring in a Social World.” Annals of Geophysics 54 (6): 708–715.
   Web of Science ®Google Scholar
• Feng, Y., and M. Sester. 2018. “Extraction of Pluvial Flood Relevant Volunteered Geographic Information (VGI) by Deep Learning from User Generated Texts and Photos.” ISPRS International Journal of Geo-Information 7 (2): 39.
   Web of Science ®Google Scholar
• Fohringer, J., D. Dransch, H. Kreibich, and K. Schröter. 2015. “Social Media as an Information Source for Rapid Flood Inundation Mapping.” Natural Hazards and Earth System Sciences 15 (12): 2725–2738.
   Web of Science ®Google Scholar
• Gao, Y., S. Wang, A. Padmanabhan, J. Yin, and G. Cao. 2018. “Mapping Spatiotemporal Patterns of Events Using Social Media: A Case Study of Influenza Trends.” International Journal of Geographical Information Science 32 (3): 425–449.
   Web of Science ®Google Scholar
• Gao, Y., M. Wang, Z. J. Zha, J. Shen, X. Li, and X. Wu. 2013. “Visual-textual Joint Relevance Learning for Tag-based Social Image Search.” IEEE Transactions on Image Processing 22 (1): 363–376.
   PubMed Web of Science ®Google Scholar
• Goodchild, M. F. 2007. “Citizens as Sensors: The World of Volunteered Geography.” GeoJournal 69 (4): 211–221.
   Google Scholar
• Gupta, A., H. Lamba, P. Kumaraguru, and A. Joshi. 2013. “Faking Sandy: Characterizing and Identifying Fake Images on Twitter during Hurricane Sandy.” Proceedings of the 22nd International Conference on World Wide Web, ACM, May, 729–736.
   Google Scholar
• Hannak, A., E. Anderson, L. F. Barrett, S. Lehmann, A. Mislove, and M. Riedewald. 2012. “Tweetin’ in the Rain: Exploring Societal-Scale Effects of Weather on Mood.” ICWSM, June.
   Google Scholar
• He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep Residual Learning for Image Recognition.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778.
   Google Scholar
• Hershey, S., S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, and M. Slaney. 2017. “CNN Architectures for Large-scale Audio Classification.” 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, March, 131–135.
   Google Scholar
• Hochreiter, S., and J. Schmidhuber. 1997. “Long Short-term Memory.” Neural Computation 9 (8): 1735–1780.
   PubMed Web of Science ®Google Scholar
• Hu, F., G. S. Xia, J. Hu, and L. Zhang. 2015. “Transferring Deep Convolutional Neural Networks for the Scene Classification of High-resolution Remote Sensing Imagery.” Remote Sensing 7 (11): 14680–14707.
   Web of Science ®Google Scholar
• Huang, X., C. Wang, and Z. Li. 2018a. “Reconstructing Flood Inundation Probability by Enhancing Near Real-Time Imagery with Real-Time Gauges and Tweets.” IEEE Transactions on Geoscience and Remote Sensing. doi:10.1109/TGRS.2018.2835306.
   PubMed Web of Science ®Google Scholar
• Huang, X., C. Wang, and Z. Li. 2018b. “A Near Real-Time Flood-Mapping Approach by Integrating Social Media and Post-Event Satellite Imagery.” Annals of GIS 24 (2): 113–123.
   Web of Science ®Google Scholar
• Huang, X., C. Wang, Z. Li, and H. Ning. 2018c. “A Visual–Textual Fused Approach to Automated Tagging of Flood-Related Tweets During a Flood Event.” International Journal of Digital Earth. doi:10.1080/17538947.2018.1523956.
   Web of Science ®Google Scholar
• Imran, M., C. Castillo, F. Diaz, and S. Vieweg. 2015. “Processing Social Media Messages in Mass Emergency.” ACM Computing Surveys 47 (4): 1–38.
   Web of Science ®Google Scholar
• Kalchbrenner, N., E. Grefenstette, and P. Blunsom. 2014. “A Convolutional Neural Network for Modelling Sentences.” arXiv preprint arXiv:1404.2188.
   Google Scholar
• Kent, J. D., and H. T. Capello. 2013. “Spatial Patterns and Demographic Indicators of Effective Social Media Content During TheHorsethief Canyon Fire of 2012.” Cartography and Geographic Information Science 40 (2): 78–89.
   Web of Science ®Google Scholar
• Kim, Y. 2014. “Convolutional Neural Networks for Sentence Classification.” arXiv preprint arXiv:1408.5882.
   Google Scholar
• Krizhevsky, A., I. Sutskever, and G. E. Hinton. 2012. “Imagenet Classification with Deep Convolutional Neural Networks.” In Advances in Neural Information Processing Systems, 1097–1105. Lake Tahoe, NV: Neural Information Processing Systems (NIPS).
   Google Scholar
• Lampos, V., and N. Cristianini. 2012. “Nowcasting Events from the Social Web with Statistical Learning.” ACM Transactions on Intelligent Systems and Technology (TIST) 3 (4): 72.
   Web of Science ®Google Scholar
• Laura, L. F., W. Joost, B. Marc, and S. Harald. 2017. “Multi-modal Deep Learning Approach for Flood Detection.” Proceedings of the MediaEval. Workshop, Dublin Google Scholar.
   Google Scholar
• Lew, M. S., N. Sebe, C. Djeraba, and R. Jain. 2006. “Content-based Multimedia Information Retrieval: State of the Art and Challenges.” ACM Transactions on Multimedia Computing, Communications, and Applications 2 (1): 1–19.
   Web of Science ®Google Scholar
• Li, Z., C. Wang, C. T. Emrich, and D. Guo. 2018. “A Novel Approach to Leveraging Social Media for Rapid Flood Mapping: A Case Study of the 2015 South Carolina Floods.” Cartography and Geographic Information Science 45 (2): 97–110.
   Web of Science ®Google Scholar
• Lin, Z., H. Jin, B. Robinson, and X. Lin. 2016. “Towards an Accurate Social Media Disaster Event Detection System Based on Deep Learning and Semantic Representation.” Proceedings of the Australasian Data Mining Conference.
   Google Scholar
• McCann, S., and D. G. Lowe. 2012. “Local Naive Bayes Nearest Neighbor for Image Classification.” Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference, IEEE, June, 3650–3656.
   Google Scholar
• Middleton, S. E., L. Middleton, and S. Modafferi. 2014. “Real-time Crisis Mapping of Natural Disasters Using Social Media.” IEEE Intelligent Systems 29 (2): 9–17.
   Web of Science ®Google Scholar
• Mikolov, T., K. Chen, G. Corrado, and J. Dean. 2013. “Efficient Estimation of Word Representations in Vector Space.” arXiv preprint arXiv:1301.3781.
   Google Scholar
• Mikolov, T., M. Karafiát, L. Burget, J. Černocký, and S. Khudanpur. 2010. “Recurrent Neural Network Based Language Model.” Eleventh annual conference of the International Speech Communication Association.
   Google Scholar
• Muralidharan, S., L. Rasmussen, D. Patterson, and J. H. Shin. 2011. “Hope for Haiti: An Analysis of Facebook and Twitter Usage During the Earthquake Relief Efforts.” Public Relations Review 37 (2): 175–177.
   Web of Science ®Google Scholar
• Nguyen, D. T., F. Ofli, M. Imran, and P. Mitra. 2017. “Damage Assessment from Social Media Imagery Data during Disasters.” Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, ACM, July, 569–576.
   Google Scholar
• Ofli, F., P. Meier, M. Imran, C. Castillo, D. Tuia, N. Rey, and S. Joost. 2016. “Combining Human Computing and Machine Learning to Make Sense of Big (Aerial) Data for Disaster Response.” Big Data 4 (1): 47–59.
   PubMed Web of Science ®Google Scholar
• Ouyang, X., P. Zhou, C. H. Li, and L. Liu. 2015. “Sentiment Analysis Using Convolutional Neural Network.” In Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing (CIT/IUCC/DASC/PICOM), 2015 IEEE International Conference, IEEE, October, 2359–2364.
   Google Scholar
• Pennington, J., R. Socher, and C. Manning. 2014. “Glove: Global Vectors for Word Representation.” Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1532–1543.
   Google Scholar
• Peters, M. E., M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. “Deep Contextualized Word Representations.” arXiv preprint arXiv:1802.05365.
   Google Scholar
• Resch, B., F. Usländer, and C. Havas. 2018. “Combining Machine-Learning Topic Models and Spatiotemporal Analysis of Social Media Data for Disaster Footprint and Damage Assessment.” Cartography and Geographic Information Science 45 (4): 362–376.
   Web of Science ®Google Scholar
• Ruiz Sinoga, J. D., and T. León Gross. 2013. “Droughts and Their Social Perception in the Mass Media (Southern Spain).” International Journal of Climatology 33 (3): 709–724.
   Web of Science ®Google Scholar
• Sakaki, T., M. Okazaki, and Y. Matsuo. 2010. “Earthquake Shakes Twitter Users: Real-time Event Detection by Social Sensors.” In Proceedings of the 19th International Conference on World Wide Web, ACM, April, 851–860.
   Google Scholar
• Schmidt, C. W. 2012. “Trending Now: Using Social Media to Predict and Track Disease Outbreaks.” Environmental Health Perspectives 120 (1): a30.
   PubMed Web of Science ®Google Scholar
• Schnebele, E., and G. Cervone. 2013. “Improving Remote Sensing Flood Assessment Using Volunteered Geographical Data.” Natural Hazards and Earth System Sciences 13: 669–677.
   Web of Science ®Google Scholar
• Schnebele, E., G. Cervone, and N. Waters. 2014. “Road Assessment After Flood Events Using Non-Authoritative Data.” Natural Hazards and Earth System Sciences 14 (4): 1007–1015.
   Web of Science ®Google Scholar
• Serengil, S. L. 2018. “Transfer Learning in Keras Using Inception V3.” March 12. Accessed June 12, 2018. https://sefiks.com/2017/12/10/transfer-learning-in-keras-using-inception-v3/.
   Google Scholar
• Simonyan, K., and A. Zisserman. 2014. “Very Deep Convolutional Networks for Large-scale Image Recognition.” arXiv preprint arXiv:1409.1556.
   Google Scholar
• Slavkovikj, V., S. Verstockt, S. Van Hoecke, and R. Van de Walle. 2014. “Review of Wildfire Detection Using Social Media.” Fire Safety Journal 68: 109–118.
   Web of Science ®Google Scholar
• Sloan, L., and J. Morgan. 2015. “Who Tweets with Their Location? Understanding the Relationship Between Demographic Characteristics and the Use of Geoservices and Geotagging on Twitter.” PloS one 10 (11): e0142209.
   PubMed Web of Science ®Google Scholar
• Sloan, L., J. Morgan, W. Housley, M. Williams, A. Edwards, P. Burnap, and O. Rana. 2013. “Knowing the Tweeters: Deriving Sociologically Relevant Demographics from Twitter.” Sociological Research Online 18 (3): 1–11.
   Web of Science ®Google Scholar
• Srivastava, N., G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. 2014. “Dropout: A Simple Way to Prevent Neural Networks from Overfitting.” The Journal of Machine Learning Research 15 (1): 1929–1958.
   Google Scholar
• Stefanidis, A., A. Crooks, and J. Radzikowski. 2013. “Harvesting Ambient Geospatial Information from Social Media Feeds.” GeoJournal 78 (2): 319–338.
   Google Scholar
• Sutskever, I., O. Vinyals, and Q. V. Le. 2014. “Sequence to Sequence Learning with Neural Networks.” In Advances in Neural Information Processing Systems, 3104–3112. Montréal: Neural Information Processing Systems Foundation, Inc.
   Google Scholar
• Sutton, J. N., L. Palen, and I. Shklovski. 2008. “Backchannels on the Front Lines: Emergency Uses of Social Media in the 2007 Southern California Wildfires.” University of Colorado, 624–632.
   Google Scholar
• Szegedy, C., S. Ioffe, V. Vanhoucke, and A. A. Alemi. 2017. “Inception-v4, Inception-resnet and the Impact of Residual Connections on Learning.” AAAI, February, Vol. 4, p. 12.
   Google Scholar
• Szegedy, C., W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, and A. Rabinovich. 2015. “Going Deeper with Convolutions.” Proceedings of the IEEE conference on computer vision and pattern recognition, 1–9.
   Google Scholar
• Szegedy, C., V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. 2016. “Rethinking the Inception Architecture for Computer Vision.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2818–2826.
   Google Scholar
• Tai, K. S., R. Socher, and C. D. Manning. 2015. “Improved Semantic Representations from Tree-structured Long Short-term Memory Networks.” arXiv preprint arXiv:1503.00075.
   Google Scholar
• Tang, Z., L. Zhang, F. Xu, and H. Vo. 2015. “Examining the Role of Social Media in California’s Drought Risk Management in 2014.” Natural Hazards 79 (1): 171–193.
   Web of Science ®Google Scholar
• Twitter MAU in the United States 2018 | Statistic. 2018. Accessed August 21, 2018. https://www.statista.com/statistics/274564/monthly-active-twitter-users-in-the-united-states/.
   Google Scholar
• Vadicamo, L., F. Carrara, A. Cimino, S. Cresci, F. Dell’Orletta, F. Falchi, and M. Tesconi. 2017. “Cross-media Learning for Image Sentiment Analysis in the Wild.” Proceedings of the IEEE International Conference on Computer Vision, 308–317.
   Google Scholar
• Verma, S., S. Vieweg, W. J. Corvey, L. Palen, J. H. Martin, M. Palmer, and K. M. Anderson. 2011. “Natural Language Processing to the Rescue? Extracting’ Situational Awareness’ Tweets during Mass Emergency.” Fifth International AAAI Conference on Weblogs and Social Media, July.
   Google Scholar
• Vieweg, S., A. L. Hughes, K. Starbird, and L. Palen. 2010. “Microblogging during Two Natural Hazards Events: What Twitter May Contribute to Situational Awareness.” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, April, ACM, 1079–1088.
   Google Scholar
• Wang, T., D. J. Wu, A. Coates, and A. Y. Ng. 2012. “End-to-end Text Recognition with Convolutional Neural Networks.” Pattern Recognition (ICPR), 2012 21st International Conference, IEEE, November, 3304–3308.
   Google Scholar
• Xu, Z., Y. Liu, N. Yen, L. Mei, X. Luo, X. Wei, and C. Hu. 2016. “Crowdsourcing Based Description of Urban Emergency Events Using Social Media big Data.” IEEE Transactions on Cloud Computing. doi:10.1109/TCC.2016.2517638.
   Google Scholar
• Yates, D., and S. Paquette. 2010. “Emergency Knowledge Management and Social Media Technologies: A Case Study of the 2010 Haitian Earthquake.” Proceedings of the 73rd ASIS&T Annual Meeting on Navigating Streams in an Information Ecosystem-Volume 47, American Society for Information Science, October, p. 42.
   Google Scholar
• Yih, W. T., X. He, and C. Meek. 2014. “Semantic Parsing for Single-relation Question Answering.” Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Vol. 2, 643–648.
   Google Scholar
• Yin, J., A. Lampert, M. Cameron, B. Robinson, and R. Power. 2012. “Using Social Media to Enhance Emergency Situation Awareness.” IEEE Intelligent Systems 27 (6): 52–59.
   Web of Science ®Google Scholar
• Yosinski, J., J. Clune, Y. Bengio, and H. Lipson. 2014. “How Transferable are Features in Deep Neural Networks?” Advances in Neural Information Processing Systems, 3320–3328.
   Google Scholar
• You, Q., J. Luo, H. Jin, and J. Yang. 2016. “Cross-modality Consistent Regression for Joint Visual-textual Sentiment Analysis of Social Multimedia.” Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, ACM, February, 13–22.
   Google Scholar
• Yue-Hei Ng, J., M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici. 2015. “Beyond Short Snippets: Deep Networks for Video Classification.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4694–4702.
   Google Scholar
• Zhang, Y., and B. Wallace. 2015. “A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification.” arXiv preprint arXiv:1510.03820.
   Google Scholar
• Zikopoulos, P., and C. Eaton. 2011. Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data. New York: McGraw-Hill Osborne Media.
   Google Scholar