Confronting Selection Bias: The Normative and Empirical Risks of Data Collection in Violent Contexts

ABSTRACT

The data collection strategies we employ affect the quality of our findings. This is particularly true for field researchers of violence and human rights. Working in high-risk, low-information contexts, these researchers are at greater risk of methodological missteps and the accompanying shortfalls for their findings and policy recommendations. We interrogate one methodological challenge particularly common to research in violent contexts: selection bias. While methodology textbooks address this topic generally, little space is dedicated to the unique challenges scholars face in their attempts to avoid selection bias during fieldwork amidst violence. Using survey and interview data from the field, we demonstrate how such a methodological misstep not only biases results, but further marginalises the already marginalised by privileging some voices over others. Asymmetrical power relationships and the normative consequences of selection bias are emphasised. We suggest how scholars of several positivist methodological traditions can address selection bias in the field. Specifically, we urge critically assessing received insight about a fieldsite, multi-method research, long-term engagement in and with fieldsites, and acknowledging biases. We draw on our fieldwork in Mexico and Colombia, referencing our data gathering strategies, quantitative and qualitative evidence, and missteps.

Acknowledgments

The authors are grateful for comments from Rob Blair, Noelle Brigden, Kai Thaler, Rebecca Weitz-Shapiro, and participants in the Challenges for Researchers in Violent Environments Workshop at the Cristosal Global School. We also benefited greatly from the comments of two anonymous reviewers. All errors are our own.

Notes

1. Author One refers to Bell-Martin and Author Two to Marston.

2. Author Two’s survey data are available from the author upon reasonable request.

3. For more on non-response error in surveys due to migration from conflict settings, see Haer and Becher (2012, 8).

4. By ‘flight,’ we mean fleeing their homes due to violence.

5. This qualitative research was part of a larger multi-method project that also leveraged quantitative and experimental methods.

6. There were, for example, objectively high-risk neighbourhoods, often (not always) in lower socioeconomic districts.

7. Wealthier citizens were at greater risk of kidnapping for ransom, for example.

8. We have little reason to think that respondents who had already revealed to enumerators that they had fled violence (and answered a series of questions on the topic) would then be unwilling to share that they had registered the flight. In other words, we do not think the 65% discrepancy between registering and not is an artefact of the survey, but rather a generally accurate reflection of reality in those neighbourhoods.

9. Author One field notes (15 March 2017).

10. Jacobsen and Landau make a similar observation in their work with refugees in Johannesburg, South Africa (2003, 195).

11. We consider ‘local insight’ distinct from ‘local knowledge’ (see Antweiler 1998 for a review of the ‘local knowledge’ concept). Local knowledge refers to the collection of understandings, skills, facts, practices and philosophies of everyday life developed over time by a community. Local insight refers strictly to the information about the local context (usually from individuals) that informs our research strategies. While ‘local knowledge’ generally does not refer to knowledge or information provided by elites or experts (and in fact often resists such knowledge), ‘local insight’ frequently comes from elites in addition to others.

12. ‘Local insight’ is often held by ‘gatekeepers’ (Campbell et al. 2006). For a discussion of gatekeepers to datasets, see Chambers-Ju (2014).

13. For more on researchers developing ‘ethnographic safety,’ see Baird (2018).

14. For examples of local research brokers’ privilege and its impact on research, see Baaz and Utas (Forthcoming).

15. Author One field notes (3 April 2017).

16. Author One Interview, March 2017, Monterrey.

17. Participant in Author Two’s survey, Medellín, September 2017.

18. Both authors were fortunate to secure funding for preliminary research trips. These trips were instrumental for us to identify the hazards of selection bias in our projects and to design high quality, engaged research. By supporting preliminary fieldwork, universities and other funding sources improve not only research findings, but also ensure scholars gain the in-depth contextual knowledge necessary to make challenging ethical judgements when designing and carrying out the research, some of which we discuss in this article.

19. Jacoby, for example, argues that the victim concept relies on ‘categories of victim and perpetrator [that] are generally treated as ‘prior or external to analysis’ instead of as propositions for further enquiry’ (2015, 511).

20. Mneimneh et al. (2014) make a similar suggestion.

21. The Justice in Mexico Project, for example, measures drug-related deaths via homicide counts provided by the Mexican state and major Mexican news outlets.

22. Although using more than one data collection strategy may be time-consuming and costly, strategies exist for effectively using time in the field and prioritizing data sources (see e.g., Chambers-Ju 2014). Barakat et al. (2002) and Thaler (2017) also recommend multi-method research in violent contexts, but do not focus on selection bias.

23. For more on respondents’ refusal to participate due to security concerns in conflict zones or because researchers cannot find them, resulting in non-response error, see Haer and Becher (2012, 8).

24. For details on the safety measures taken by enumerators in Author Two’s study, see Marston (2019b). For more on interviewer effects when hiring local survey enumerators in conflict settings, see Haer and Becher (2012, 10).

25. Author One Interview, July 2017, Monterrey.

26. For more on the ethical tradeoffs of engaged scholarship, see Pacheco-Vega and Parizeau (2018).

27. Readers may discern an important tension between our recommendation to critically engage local insight on one hand and pursue engaged scholarship on the other. In fact, these recommendations are complementary. Our recommendation is not to eschew local insight but to engage it critically and to ensure that it includes a multitude of perspectives. In fact, increasing the diversity of local insight requires that we engage local insight more and more diversely. Engaged scholarship facilitates our ability to do this.

28. We urge scholars to be attentive to the potential of retraumatisation. However, we also acknowledge that some witnesses to violence and victims find sharing their stories cathartic (Wood 2006, 377–378). This largely rings true in our own research.

29. For instance, Henry encourages graduate students not to research sexual violence in war, because for many aspects of the topic and in many contexts, ‘It’s been done already!’ (2013, emphasis in original). See also Cronin-Furman and Lake (2018, 609).

30. Cronin-Furman and Lake offer further discussion of the ethical dilemmas presented when research assistants put themselves at risk for employment or training opportunities (2018, 610–611). Meanwhile, Baird offers a brief discussion of the risks gatekeepers face and researchers’ ethical obligations to them (2018, 350).

31. For more details on the safety measures Author Two took, see Marston (2019b).

Additional information

Funding

Jerome F. Marston Jr. acknowledges support from the Fulbright-Hays Doctoral Dissertation Research Abroad. The authors’ research has been supported by the Graduate Program in Development at the Watson Institute, the Cogut Center Brown-in-the-World Travel Grant, the Joukouwsky Dissertation Research and Travel Funds, and the Populations Studies Training Center at Brown University.