Quantifying Qualitative Scholarship: A Fool’s Errand or a Methodological Necessity?

Abstract

This paper describes our efforts to systematically generate knowledge from ethnographic and other qualitative scholarship as a way of bridging intellectual gaps found in quantitative research. We ground this scholarly endeavor in the study of gangs, which has a lengthy history in the field of criminology, yet lacks a coherent, coordinated, and comparative understanding of gang dynamics, behavior, and impacts. We discuss the importance of acknowledging the multiple and mixed methodologies used in gang scholarship and detail the inductive, deductive, and iterative process that helped us (1) develop a comprehensive coding instrument for coding a range of qualitative works, (2) establish the instrument’s inter-rater reliability, and (3) remain cognizant and reflexive about the strengths of both quantitative and qualitative research. Finally, we share the critical yet informative challenges we faced throughout the project and conclude with reflections on why the project was unsuccessful in the way we originally envisioned.

Acknowledgments

This project was funded in part by the University of California, Irvine’s Academic Senate Council on Research, Computing and Libraries and the School of Social Ecology’s Dean’s Fund for Excellence in Undergraduate Research. Additionally, support from the UCI Visiting Fellows Program brought Jody Miller to UCI during the winter of 2015 when the project was initiated. UCI graduate students Jason Gravel, Kristen Maziarka, Amanda Petersen, and James Pratt made substantial contributions to earlier stages of the project. The authors also recognize and appreciate the research assistants who provided helpful insight and support throughout different stages of the project. Research assistants include Ntasha Bhardwaj, Kelsie Chesnut, Christina Elias, Martin Gomez, Sadaf Hashimi, Andrew Kind, Jose Sandoval, Caroline Shaver, Carissa Singh, Brandi Taylor, and Rachel Tran.

Notes

¹ The Eurogang Ethnography Guidelines include participant observation, interviews with gang members and non-gang community members, and/or examination of key events; topics to be addressed, at minimum, include history and local setting, group descriptions and group culture, and the life experiences of individual members (see: https://www.umsl.edu/ccj/Eurogang/pdf/Ethnography%20Guidelines.pdf).

² The closest comparison is the Global Terrorism Database (https://www.start.umd.edu/gtd/), which codes reports from open media sources for a large-scale longitudinal data-set on the attributes of terrorist incidents.

³ The phrase “detached workers” refers to individuals who act as conduits between gang members and other social groups and organizations.

⁴ See Hughes (2005) for an overview of the strengths and weaknesses of qualitative and quantitative gang research.

⁵ Open coding is a qualitative data analysis technique that enables researchers to produce grounded theory. Rather than using already-existing codes, the researcher allows the data source to dictate the natural emerging themes. The themes can then be used as findings or the starting point for systematic observation in other data sources, such as the development of the coding instrument used during the Comparative Qualitative Gang Project (Strauss & Corbin, 1990; Esterberg, 2002).

⁶ Coded projects at this point in the project included: Dead End Kids (Fleisher, 1998); One of the Guys (Miller, 2001); Unwanted: Muslim Immigrants, Dignity, and Drug Dealing (Bucerius, 2014); People and Folks (Hagedorn & Macon, 1988); Real Gangstas (Lauger, 2012).

⁷ Focused coding is a data analysis procedure that more directly centers the open codes in the data. Instead of a grounded approach to data analysis, focused coding is intended to verify the already established open codes and determine their prevalence in the data (Esterberg, 2002).

⁸ To test reliability, we looked at the percentage of agreement between coders. For each item, the percentage of agreement had to be at least 66% (typically 2/3 of coders) for the team to retain the variable. Agreement between 67% and 79% indicated the item needed some revision in order to yield greater reliability. Agreement above 80% suggested that reliability was strong and that the item could remain unchanged.

⁹ Undergraduate students were enrolled in directed study credits under Professor Cheryl Maxson during the spring 2016 quarter, with the ultimate goal of the undergraduate team yielding eight independently completed instruments for Dead End Kids. Lead graduate research assistants Amy Magnus and Daniel Scott designed a syllabus under the supervision of Professor Maxson that included training in content analyses, memo-writing, coding/organization strategies, and practice coding with qualitative material formerly used by the graduate team, including Miller’s One of the Guys and Bucerius’ Unwanted: Muslim Immigrants, Dignity, and Drug Dealing. The graduate and undergraduate students met once per week and engaged in lecture-, discussion-, and application-based classes. They were asked to maintain an ongoing research memo (somewhat like a journal) to document their challenges while coding, questions about particular items, and their experiences with coder fatigue and time management. The excerpts presented in the proceeding section of this paper have been extracted from the students’ journals with their permission. A copy of the spring course syllabus is available upon request from the lead author.

¹⁰ Sykes et al. (2017) reveal how incongruent sampling approaches in mixed methods research can bias conclusions about important relationships and thereby undermine the perceived advantages of combining such strategies.

¹¹ Although the instrument is lengthy and could not yield perfect reliability across all items, the research team finds it to still be useful in specific ways. For example, researchers hoping to learn more about particular topics, such as the use of weapons during violent incidents or the common ways in which gang members are initiated, can use specific items in the instrument to answer particular research questions. While the instrument is too long to include in an appendix, those interested may contact the first author to view a copy.

¹² All undergraduate research assistants’ names are pseudonyms.

¹³ For our purposes, “incidents” are defined as events involving at least one group member who is under study by the researcher. The event must take place while the group member is in the group - for example, an event from their childhood would not be coded—though the event does not need to occur during data collection. Additionally, particular variables were pre-selected that must be codeable in order for the incident to qualify as such. These variables included whether the researcher was present for event, type of event, the people directly involved and responsible for the incident occurring, whether someone was killed or not, and whether someone was injured or not. Combined with these, at least three other variables must be able to be coded for the event to qualify as an incident. If an incident is mentioned multiple times throughout the publication, it should be coded as one incident.