Crowdsourcing Samples for Research on Violent Extremism: A Research Note

ABSTRACT

As research on violent extremism continues to progress beyond some of the field’s earlier challenges, new ways of gathering primary source data are becoming increasingly popular. One such data collection methodology implemented widely across parallel fields is crowdsourcing: the process of gathering information, or input, from large numbers of people, either for payment or not, online. In this research note, we present a brief introduction to crowdsourcing, highlight a popular platform for gathering samples online, Prolific, and present four studies conducted by the research team to demonstrate the unique benefits and challenges of crowdsourcing samples online for research on violent extremism.

KEYWORDS:

• Crowdsourcing
• terrorism
• violent extremism
• Prolific
• Qualtrics

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. J. C. Palmer and J. Strickland, “A Beginner’s Guide to Crowdsourcing,” Psychological Science Agend (June, 2016) http://www.apa.org/science/about/psa/2016/06/changing-minds.

2. T. S. Behrend, D. J. Sharek, and A. W. Meade, et al., “The Viability of Crowdsourcing for Survey Research,” Behavior Research 43 (2011): 800, https://doi.org/10.3758/s13428-011-0081-0.

3. A. Silke, “The Devil You Know: Continuing Problems with Research on Terrorism” Journal of Terrorism and Political Violence 13, no. 4 (2001): 1–14, https://doi.org/10.1080/09546550109609697; A. Silke, Research on Terrorism: Trends, Achievements and Failures (London, England: Routledge, 2004); A. Silke, “Research on Terrorism,” in Terrorism Informatics, ed. H. Chen, E. Reid, J. Sinai, A. Silke, and B. Ganor, Vol. 18. (Boston, MA: Springer, 2008), 27–50.

4. E. Peer, L. Brandimarte, S. Samat, and A. Acquisti, “Beyond the Turk: Alternative Platforms for Crowdsourcing Behavioral Research,” Journal of Experimental Social Psychology 70 (2017): 153–63, https://doi.org/10.1016/j.jesp.2017.01.006.

5. S. Palan and C. Schitter, “Prolific.ac—A Subject Pool for Online Experiments,” Journal of Behavioral and Experimental Finance 17 (2018): 22–27, https://doi.org/10.1016/j.jbef.2017.12.004.

6. E. Peer, D. Rothschild, A. Gordon, Z. Evernden, and E. Damer, “Data Quality of Platforms and Panels For Online Behavioral Research,” Behavior Research Methods, (September 29, 2021): 1–20, https://doi.org/10.3758/s13428-021-01694-3. Epub ahead of print. PMID: 34590289; PMCID: PMC8480459.

7. Ibid.

8. Peer et al., “Beyond the Turk.”

9. J. Adam-Troian, E. Bonetto, M. Araujo, O. Baidada, E. Celebi, M. Dono Martin, F. Eadeh, A. Godefroidt, S. Halabi, and Y. Mahfud, et al., “Positive Associations between Anomia and Intentions to Engage in Political Violence: Cross-Cultural Evidence From Four Countries,” Peace and Conflict: Journal of Peace Psychology 26, no. 2 (2020): 217.

10. J. J. Bélanger, B. G. Robbins, H. Muhammad, M. Moyano, C. F. Nisa, B. M. Schumpe, and M. Blaya-Burgo, “Supporting Political Violence: The Role of Ideological Passion and Social Network,” Group Processes & Intergroup Relations 23, no. 8 (2020): 1187–203.

11. M. Obaidi, J. R. Kunst, N. Kteily, L. Thomsen, and J. Sidanius, “Living Under Threat: Mutual Threat Perception Drives Anti‐Muslim and Anti‐Western Hostility in the Age of Terrorism,” European Journal of Social Psychology 48, no. 5 (2018): 567–84.

12. O. Gøtzsche-Astrup, “Dark Triad, Partisanship and Violent Intentions in the United States,” Personality and Individual Differences 173 (2021): 110633.

13. C. Clemmow, S. Schumann, N. L. Salman, and P. Gill, “The Base Rate Study: Developing Base Rates for Risk Factors and Indicators for Engagement in Violent Extremism,” Journal of Forensic Sciences 65, no. 3 (2020): 865–81, https://doi.org/10.1111/1556-4029.14282.

14. H. S. Gomes, D. P. Farrington, and Â. Maia, et al., “Measurement Bias in Self-Reports of Offending: A Systematic Review of Experiments,” Journal of Experimental Criminology 15 (2019): 313–39, https://doi.org/10.1007/s11292-019-09379-w.

15. Ibid.

16. A. O’Connor and T. A. Gannon, “An Examination of the Prevalence and Characteristics of UK Community Males Who Hold a Sexual Interest in Children using the Revised Interest in Child Molestation Scale,” Psychology, Crime & Law (2021): 1-22, https://doi.org/10.1080/1068316X.2021.1876049.

17. I. van der Vegt, B. Kleinberg, and P. Gill, “Too Good to be True? Predicting Author Profiles from Abusive Language,” arXiv preprint arXiv:2009.01126 (2020).

18. I. van der Vegt, M. Mozes, B. Kleinberg, and P. Gill, “The Grievance Dictionary: Understanding Threatening Language Use,” Behavior Research Methods (2021): 1-15, https://doi.org/10.3758/s13428-021-01536-2.

19. B. Rottweiler and P. Gill, “Individual Differences in Personality Moderate the Effects of Perceived Group Deprivation on Violent Extremism: Evidence From a United Kingdom Nationally Representative Survey,” Frontiers in Psychology (2022): 99.

20. E. Ruel, W. E. Wagner III, and B. J. Gillespie, The Practice of Survey Research: Theory and Applications (Sage Publications, 2015).

21. E. Anduiza and C. Galais, “Answering Without Reading: IMCs and Strong Satisficing in Online Surveys,” International Journal of Public Opinion Research 29, no. 3 (2016): 497–519, https://doi.org/10.1093/ijpor/edw007; A. J. Berinsky, M. F. Margolis, and M. W. Sances, “Separating the Shirkers from the Workers? Making sure Respondents Pay Attention on Self‐Administered Surveys,” American Journal of Political Science 58, no. 3 (2014): 739–53, https://doi.org/10.1111/ajps.12081; D. J. Hauser, A. Sunderrajan, M. Natarajan, and N. Schwarz, “Prior Exposure to Instructional Manipulation checks does not Attenuate Survey Context Effects Driven by Satisficing or Gricean Norms,” Methods, Data, Analyses: A Journal for Quantitative Methods and Survey Methodology (mda) 10, no. 2 (2016): 195–220, https://doi.org/10.12758/mda.2016.008; J. Miller and J. Baker-Prewitt, “Beyond “Trapping” the Undesirable Panellist: The Use of Red Herrings to Reduce Satisficing” (paper presented at the CASRO Panel Quality Conference, New Orleans, LA, February, 2009).

22. L. Zmigrod, P. J. Rentfrow, and T. W. Robbins, “Cognitive Inflexibility Predicts Extremist Attitudes,” Frontiers in Psychology 10 (2019): 989, https://doi.org/10.3389/fpsyg.2019.00989; S. Schumann, N. L. Salman, C. Clemmow, and P. Gill, “Does Cognitive Inflexibility Predict Violent Extremist Behaviour Intentions? A Registered Direct Replication Report of Zmigrod, Rentfrow, & Robbins, 2019,” Legal and Criminological Psychology 26, no. 2 (2021): 145–157, https://doi.org/10.1111/lcrp.12186.

23. S. A. Mednick, “The Remote Associates Test,” The Journal of Creative Behavior (1968).

24. D. A. Grant and E. Berg, “A Behavioral Analysis of Degree of Reinforcement and Ease of Shifting to New Responses in a Weigl-Type Card-Sorting Problem,” Journal of Experimental Psychology 38, no. 4 (1948): 404. https://doi.org/10.1037/h0059831.

25. G. Stoet, “PsyToolkit: A Software Package for Programming Psychological Experiments using Linux,” Behavior Research Methods 42, no. 4 (2010): 1096–104, https://doi.org/10.3758/BRM.42.4.1096; G. Stoet, “PsyToolkit: A Novel Web-Based Method For Running Online Questionnaires and Reaction-Time Experiments,” Teaching of Psychology 44, no. 1 (2017): 24–31, https://doi.org/10.1177/0098628316677643.

26. V. Williamson, “On the Ethics of Crowdsourced Research,” PS: Political Science & Politics 49, no. 1 (2016): 77–81, https://doi.org/10.1017/S104909651500116X.

27. R. Luong and A. M. Lomanowska, “Evaluating Reddit as a Crowdsourcing Platform for Psychology Research Projects,” Teaching of Psychology 49, no. 4 (2022): 329–337, https://doi.org/10.1177/00986283211020739.

Additional information

Funding

This study received funding from The European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant 758834).

Notes on contributors

Caitlin Clemmow

Caitlin Clemmow leads the Jill Dando Institute Research Lab and is a member of UCL’s Counterterrorism Research Group. Her research focusses on risk and protective factors for grievance-fuelled violence.

Isabelle van der Vegt

Isabelle van der Vegt holds a PhD in Security and Crime Science from UCL. Her research focuses on understanding targeted violence using computational linguistics. She currently works as a scientific project manager at the Research and Documentation Centre of the Ministry of Justice and Security in The Netherlands.

Bettina Rottweiler

Bettina Rottweiler is a Research Fellow in the Department of Security and Crime Science at University College London. Her research examines risk and protective factors for violent extremist intentions using general population surveys.

Sandy Schumann

Sandy Schumann is a Lecturer (Assistant Professor) and the Postgraduate Tutor at University College London, Department of Security and Crime Science. Her research examines risk factors of radicalisation, extreme political attitudes, as well as hate crime prevention. Schumann studies these phenomena “online” as well as “offline” and is keen to translate her research into policy and practice. Doing so, she advises governments (e.g., U.K. Commission on Countering Extremism, UAE Ministry of Culture and Knowledge Development) and collaborates with civil society organisations to conduct impact evaluations.

Paul Gill

Paul Gill is a Professor of Security and Crime Science at UCL. His research focuses on terrorist behaviour. He has conducted research funded by the Office for Naval Research, the Department of Homeland Security, the European Union, the National Institute of Justice, Public Safety Canada, CREST and MINERVA. These projects focused upon various aspects of terrorist behavior including the IED development, creativity, terrorist network structures, violent extremist risk assessment and lone-actor terrorism. He currently leads a European Research Council Starter grant for a project entitled GRIEVANCE.