![Sample our Politics & International Relations journals, sign in here to start your FREE access for 14 days]({"type":"image" , "src" : "/sda/5947/TOC-Subject-Sample-2021-Politics-International-Relations.jpg"})
ABSTRACT
This article presents a mixed methods approach for analysing text and image relations in violent extremist discourse. The approach involves integrating multimodal discourse analysis with data mining and information visualisation, resulting in theoretically informed empirical techniques for automated analysis of text and image relations in large datasets. The approach is illustrated by a study which aims to analyse how violent extremist groups use language and images to legitimise their views, incite violence, and influence recruits in online propaganda materials, and how the images from these materials are re-used in different media platforms in ways that support and resist violent extremism. The approach developed in this article contributes to what promises to be one of the key areas of research in the coming decades: namely the interdisciplinary study of big (digital) datasets of human discourse, and the implications of this for terrorism analysis and research.
Glossary of key terms
Big data analytics: The study of large amounts of data to uncover hidden patterns, correlations and other insights; Data mining: Computational processes involving methods of artificial intelligence, machine learning, statistics and database systems to discern patterns in large data sets; Information visualisation: The study of interactive visual representations of abstract data to enhance human understanding and cognition; Multimodal discourse analysis: Study of the meaning arising from the integration of multiple semiotic resources (e.g., language, image, gesture, and sound resources) in texts, interactions, and events; Semiotic resource: Sign systems (e.g. language, images, gesture, sound resources) which are used to create meaning contextually in texts, interactions, and events; Social semiotics: The branch of semiotics which focuses on the study of human sign systems and processes, conceptualised as interacting systems of meaning which constitute society and culture; Systemic Functional Theory: The approach developed by Michael A. K. Halliday that conceptualises language, images and other resources as systems of meaning which have evolved to fulfill different functions in society and culture; Text and image relations: The meanings arising from the interaction of linguistic text and images.
Notes
1. C. Jewitt, ed., The Routledge Handbook of Multimodal Analysis, 2nd ed. (London: Routledge, 2014).
2. J. Bateman, Text and Image: A Critical Introduction to the Visual/Verbal Divide (London: Rouledge, 2014).
3. K. L. O’Halloran, “Multimodal Digital Humanities,” in International Handbook of Semiotics, edited by P. Trifonas (Dordrecht: Springer, 2015), 383–409; K. L. O’Halloran, S. Tan, D.-S. Pham, J. Bateman and A. Vande Moere, “A Digital Mixed Methods Research Design: Integrating Multimodal Analysis with Data Mining and Information Visualization for Big Data Analytics”, Journal of Mixed Methods Research (2016). Advance online publication. doi:10.1177/1558689816651015.
4. A. Kovács, “The ‘New Jihadists’ and the Visual Turn from Al-Qa’ida to ISIL / ISIS / Da’ish,” BiztPol Affairs 2, no. 3 (2015): 67.
5. J. Klausen, “Tweeting the Jihad: Social Media Networks of Western Foreign Fighters in Syria and Iraq,” Studies in Conflict and Terrorism 38, no. 1 (2015): 3.
6. J. M. Berger and B. Strathearn, Who Matters Online: Measuring Influence, Evaluating Content and Countering Violent Extremism in Online Social Networks (London: International Centre for the Study of Radicalisation and Political Violence, 2013), http://icsr.info/wp-content/uploads/2013/03/ICSR_Berger-and-Strathearn.pdf; J. A. Carter, S. Maher, and P. R. Neumann, #Greenbirds: Measuring Importance and Influence in Syrian Foreign Fighter Networks (London: International Centre for the Study of Radicalisation and Political Violence, 2014), http://icsr.info/wp-content/uploads/2014/04/ICSR-Report-Greenbirds-Measuring-Importance-and-Infleunce-in-Syrian-Foreign-Fighter-Networks.pdf; M. Sageman, Leaderless Jihad: Terror Networks in the 21st Century (Philadelphia: University of Pennsylvania Press, 2014); I. von Behr, A. Reding, C. Edwards, and L. Gribbon, Radicalisation in the Digital Era: The Use of the Internet in 15 Cases of Terrorism and Extremism (Cambridge, UK: RAND Europe, 2013), http://www.rand.org/content/dam/rand/pubs/research_reports/RR400/RR453/RAND_RR453.pdf; G. Weimann, Terror on the Internet: The New Arena, the New Challenges (Washington, DC: United States Institute of Peace Press, 2006).
7. D. Murthy, “Towards a Sociological Understanding of Social Media: Theorizing Twitter,” Sociology 46, no. 6 (2013): 1059–73; K. Z. Sands, “Muslims, Identity and Multimodal Communication on the Internet,” Contemporary Islam 4, (2010): 139–55.
8. See for example Berger and Strathearn, Who Matters Online (see note 6 above); Carter, Maher, and Neumann, #Greenbirds: (see note 6 above); Klausen, “Tweeting the Jihad” (see note 5 above).
9. Z. Papacharissi, ed., A Networked Self: Identity, Community and Culture on Social Network Sites (New York and London: Routledge, 2011).
10. M. Castells, Communication Power (Oxford, UK: Oxford University Press, 2009).
11. C. Jewitt, “An Introduction to Multimodality,” in The Routledge Handbook of Multimodal Analysis, edited by C. Jewitt, 2nd ed. (London and New York: Routledge, 2014), 15–30.
12. See for example J. Blascovich and C. McCall, “Social Influence in Virtual Environments,” in The Oxford Handbook of Media Psychology, edited by K. E. Dill (Oxford, UK: Oxford University Press, 2013), 305–15.
13. Ibid., 312.
14. See for example Jewitt, “An Introduction to Multimodality,” 24–25 (see note 11 above); G. Kress, “What Is Mode?” in The Routledge Handbook of Multimodal Analysis, edited by C. Jewitt, 2nd ed. (London and New York: Routledge, 2014), 54–67, 55–58.
15. M. A. K. Halliday, Language as Social Semiotic: The Social Interpretation of Language and Meaning (London: Edward Arnold, 1978); T. van Leeuwen, Introducing Social Semiotics (London: Routledge, 2015).
16. G. Kress and T. van Leeuwen, Multimodal Discourse: The Modes and Media of Contemporary Communication Discourse (London: Arnold, 2001).
17. J. L. Lemke, “Multimedia and Discourse Analysis” in The Routledge Handbook of Discourse Analysis, edited by J. P. Gee and M. Handford (London & New York: Routledge, 2013).
18. M. Hynes and S. Sharpe, “Affected with Joy: Evaluating the Mass Actions of the Anti-Globalisation Movement,” Borderlands 8, no. 3 (2009): 1–21.
19. Jewitt, The Routledge Handbook of Multimodal Analysis (see note 1 above).
20. J. L. Lemke, “Place, Pace, and Meaning: Multimedia Chronotopes,” in Discourse in Action: Introducing Mediated Discourse Analysis, edited by S. Norris and R. H. Jones (London: Routledge, 2005), 110–22.
22. M. Lombardi, “Islamic State Communication Project,” Sicurezza, Terrorismo e Società 1, no. 1 (2015): 99–133; A. Y. Zelin, “Picture or It Didn’t Happen: A Snapshot of the Islamic State’s Official Media Output,” Perspectives on Terrorism 9, no. 4 (2015).
23. M. Vergani and A.-M. Bliuc, “The Evolution of the ISIS’ Language: A Quantitative Analysis of the Language of the First Year of Dabiq Magazine,” Sicurezza, Terrorismo e Società 1, no. 2 (2015): 7–20, 8.
24. Jewitt, The Routledge Handbook of Multimodal Analysis (see note 1 above).
25. Bateman, Text and Image (see note 2 above).
26. O’Halloran, “Multimodal Digital Humanities,” (see note 3 above).
27. J. Bateman, “Methodological and Theoretical Issues for the Empirical Investigation of Multimodality” in Sprache Im Multimodalen Kontext / Language and Multimodality, edited by N.-M Klug and H. Stöckl (Berlin: de Gruyter Mouton, 2016).
28. K. L. O’Halloran, A. Chua, and A. Podlasov, “The Role of Images in Social Media Analytics: A Multimodal Digital Humanities Approach,” in Visual Communication, edited by D. Machin (Berlin: Gruyter, 2014), 565–88.
29. A. Podlasov and K. L. O’Halloran, “Japanese Street Fashion for Young People: A Multimodal Digital Humanities Approach for Identifying Socio-Cultural Patterns and Trends,” in Critical Multimodal Studies of Popular Culture, edited by E. Djonov & S. Zhao (New York: Routledge, 2014), 71–90.
30. J. Bartlett and L. Reynolds, The State of the Art 2015: A Literature Review of Social Media Intelligence Capabilities for Counterterrorism (London: Demos, 2015), 12.
31. Ibid.
32. Zelin, “Picture or It Didn’t Happen” (see note 22 above).
33. B. Al-Hejin, “Covering Muslim Women: Semantic Macrostructures in BBC News,” Discourse & Communication 9, no. 1 (2015): 19–46.
34. K. Bhui and Y. Ibrahim, “Marketing the ‘Radical’: Symbolic Communication and Persuasive Technologies in Jihadist Websites,” Transcultural Psychiatry 50, no. 2 (2013): 216–34.
35. F. Jan, The Muslim Extremist Discourse: Constructing Us Versus Them (London: Lexington Books, 2015).
36. L. E. Wyszomierski, “Boko Haram and the Discourse of Mimicry: A Critical Discourse Analysis of Media Explanations for Boko Haram’s Improved Video Propaganda Quality,” Critical Studies on Terrorism 8, no. 3 (2015): 503–15.
37. W. H. Allendorfer and S. C. Herring, “ISIS vs the U.S. Government: A War of Online Video Propaganda,” First Monday 20, no. 12 (2015): doi:10.5210/fm.v20i12.6336.
38. J. E. Wright and M. Bachmann, “Al Qaida’s Persuasive Devices in the Digital World,” Journal of Terrorism Research 6, no. 2 (2015): 70–82.
39. D. B. Skillicorn, “Empirical Assessment of Al Qaeda, ISIS, and Taliban Propaganda,” Intelligence and Security Informatics (ISI), 2015 IEEE International Conference (2015): 61–66.
40. Vergani and Bliuc, “The Evolution of the ISIS’ Language” (see note 23 above).
41. Kovács, “The ‘New Jihadists’ and the Visual Turn from Al-Qa’ida to ISIL / ISIS / Da’ish” (see note 4 above).
42. A. Aly, D. Weimann-Saks, and G. Weimann, “Making ‘Noise’ Online: An Analysis of the Say No to Terror Online Campaign,” Perspectives on Terrorism 8, no. 5 (2014): 33–47.
43. C. Archetti, “Terrorism, Communication and New Media: Explaining Radicalization in the Digital Age,” Perspectives on Terrorism 9, no. 1 (2015): 49–59.
44. M. Grossman and H. Tahiri, “Community Perceptions of Radicalisation and Violent Extremism: An Australian Perspective,” Journal of Policing, Intelligence and Counter Terrorism 10, no. 1 (2015): 14–24.
45. M. Grossman, “Disenchantments: Counter-Terror Narratives and Conviviality,” Critical Studies on Terrorism 7, no. 3 (2014): 319–35.
46. K. O’Hara and D. Stevens, “Echo Chambers and Online Radicalism: Assessing the Internet’s Complicity in Violent Extremism,” Policy & Internet 7, no. 4 (2015): 401–22.
47. Bartlett and Reynolds, The State of the Art 2015 (see note 30 above).
48. Ibid.
49. Ibid., 22–24.
50. For example, van Leeuwen, Introducing Social Semiotics (see note 15 above).
51. Bateman, “Methodological and Theoretical Issues” (see note 27 above); Jewitt, The Routledge Handbook of Multimodal Analysis (see note 1); K. L. O’Halloran et al., “Multimodal Digital Semiotics: The Interaction of Language with Other Resources,” Text and Talk: Special Edition for Michael Halliday (edited by Geoff Thompson) 33, no. 4–5 (2013): 665–90.
52. Halliday, Language as Social Semiotic (see note 15 above).
53. K. L. O’Halloran, S. Tan, and P. Wignell, “SFL and Multimodal Discourse Analysis”, in The Cambridge Handbook of Systemic Functional Linguistics, edited by G. Thompson et al. (Cambridge UK: Cambridge University Press, in press).
54. M. A. K. Halliday and C. M. I. M. Matthiessen, Halliday’s Introduction to Functional Grammar, 4th ed., revised by C. M. I. M. Matthiessen (London and New York: Routledge, 2014).
55. J. R. Martin and D. Rose, Working with Discourse: Meaning Beyond the Clause, 2nd ed. (London: Continuum, 2007).
56. M. O’Toole, The Language of Displayed Art, 2nd ed. (London & New York: Routledge, 2011).
57. For example, see E. Margolis and L. Pauwels, eds., Sage Handbook of Visual Research Methods (London: SAGE, 2011).
58. Zelin, “Picture or It Didn’t Happen” (see note 22 above).
59. Bateman, Text and Image (see note 2 above).
60. Ibid., 206.
61. Martin and Rose, Working with Discourse: Meaning Beyond the Clause (see note 55 above).
62. J. R. Martin and P. R. R. White, The Language of Evaluation: Appraisal in English (London and New York: Palgrave Macmillan, 2005).
67. http://scrapy.org.
73. Bartlett and Reynolds, The State of the Art 2015 (see note 30 above).
75. K. L. O’Halloran, M. K. L. E, and S. Tan, “Multimodal Semiosis and Semiotics” in The Bloomsbury Companion to M.A.K. Halliday, edited by J. Webster (London: Bloomsbury, 2015); O’Halloran, Tan, and Wignell, “SFL and Multimodal Discourse Analysis” (see note 53 above).
76. O’Halloran, Chua, and Podlasov, “The Role of Images in Social Media Analytics” (see note 28 above).
77. See also O’Halloran et al., “A Digital Mixed Methods Research Design” (see note 3 above).
79. J. Lehman, R. Isele, M. Jakob, A. Jentzsch, D. Kontokostas, P. N. Mendes, S. Hellmann, M. Morsey, P. van Kleef, S. Auer, & C. Bizer, “DBpedia—A Large-Scale, Multilingual Knowledge Base Extracted from Wikipedia,” Semantic Web 6, no. 2 (2015): 167–95.
80. M. Szczuka, A. Janusz, and K. Herba, “Semantic Clustering of Scientific Articles with Use of DBpedia Knowledge Base,” in Intelligent Tools for Building a Scientific Information Platform, edited by R. Bembenik, L. Skonieczny, H. Rybinski, and M. Niezgodka (Berlin: Springer-Verlag, 2012), 61–76.
81. B. Pang and L. Lee, “Opinion Mining and Sentiment Analysis,” Foundations and Trends in Information Retrieval 2, no. 1–2 (2008): 1–135.
82. D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent Dirichlet Allocation,“ Journal of Machine Learning Research 3 (2003): 993–1022.
83. H. M. Wallach, “Topic Modeling: Beyond Bag-of-Words,” in Proceedings of the 23rd International Conference on Machine Learning (New York: ACM, 2006), 977–84.
84. F. Feng, X. Wang, and R. Li, “Cross-Modal Retrieval with Correspondence Autoencoder,” in Proceedings of the 22nd ACM International Conference on Multimedia (New York: ACM, 2014), 7–16.
85. T. Munzner, Visualization Analysis & Design (Boca Raton, London and New York: CRC Press, 2014).
86. A. Vande Moere, M. Tomitsch, C. Wimmer, C. Boesch and T. Grechenig, “Evaluating the Effect of Style in Information Visualization,” IEEE Transactions on Visualization and Computer Graphics December 2012 (Proceedings IEEE InfoVis) 18, no. 12 (2012): 2739–48.
87. O’Halloran et al., “A Digital Mixed Methods Research Design” (see note 3 above).
88. R. Conte, N. Gilbert, G. Bonelli, C. Cioffi-Revilla, G. Deffuant, J. Kertesz, V. Loreto et al., “Manifesto of Computational Social Science,” European Physical Journal Special Topics 214 (2012): 325–46; D. Lazer, A. Pentland, L. Adamic, S. Aral, A.-L. Barabási, D. Brewer, N. Christakis et al., “Computational Social Science,” Science 323 (2009): 721–3; T. Preis, H. S. Moat, S. R. Bishop, P. Treleaven, and H. E. Stanley, “Quantifying the Digital Traces of Hurricane Sandy on Flickr,” Scientific Reports 3 (2013): Article no. 3141. doi:10.1038/srep03141.
89. L. Byrne, D. Angus, and J. Wiles, “Acquired Codes of Meaning in Data Visualization and Infographics: Beyond Perceptual Primitives,” IEEE Transactions on Visualization and Computer Graphics 22, no. 1 (2016): 509–18.
90. M. Craglia, K. de Bie, D. Jackson, M. Pesaresi, G. Remetey-Fulopp, C. Wang, A. Annoni et al., “Digital Earth 2020: Towards the Vision for the Next Decade,” International Journal of Digital Earth 5, no. 1 (2012): 4–21.
91. Conte et al., “Manifesto of Computational Social Science,” 342 (see note 88 above).
92. Ibid., 341.
93. Ibid., 341.
94. C. P. Snow, The Two Cultures and the Scientific Revolution (Cambridge, UK: Cambridge University Press, 1961).
95. P. Raghavan, “It’s Time to Scale the Science in the Social Sciences,” Big Data & Society 1, no. 4 (2014): 1. doi: 10.1177/2053951714532240.
96. Ibid., 4.
97. Lazer et al., “Computational Social Science,” 723 (see note 88 above).
Additional information
Notes on contributors
Kay L. O’Halloran
Kay L. O’Halloran is affiliated with the Curtin Institute for Computation, and is a research academic in the School of Education, Faculty of Humanities, at Curtin University, Perth, Western Australia. Sabine Tan is a Research Fellow at the School of Education, Faculty of Humanities, Curtin University, Perth, Western Australia. Peter Wignell is a Research Fellow at the School of Education, Faculty of Humanities, Curtin University, Perth, Western Australia. John A. Bateman is Professor of Applied Linguistics in the Department of English, Bremen University, Bremen, Germany. Duc-Son Pham is a research-active lecturer in the Department of Computing, School of Electrical Engineering and Computing, Curtin University, Perth, Western Australia. Michele Grossman is Director of the Centre for Cultural Diversity and Wellbeing, Victoria University, Melbourne, Australia. Andrew Vande Moere is Co-Director of the Research[x]Design group at the Department of Architecture of KU Leuven, Belgium.