Is replication possible in qualitative research? A response to Makel et al. (2022)

ABSTRACT

There has been much debate in recent years about how open research practices, which have been promoted in efforts to improve research robustness, may (not) be appropriate for qualitative methodologies, particularly in educational research. Among these is a concern for replication efforts. Makel et al. (2022) make the case that “replication is relevant to qualitative research”. The authors argue that concerns surrounding the transferability, intentionality, and transparency of qualitative research may be eased, or responded to, by replication studies. Here, I argue that there are three fundamental questions that need unpacking before declarative claims can be made about the relevance of replication to qualitative research. This includes critical questioning of what we mean by: replication, qualitative research, and rigour. I address each of these issues and encourage a more nuanced appraisal of how replication may, or may not, be epistemologically, ontologically, or methodologically compatible with the goals of qualitative research.

KEYWORDS:

• Qualitative research
• replication
• replicability
• epistemology

The advent of replication studies is a necessary and timely mechanism to continually assess the validity, robustness, and transferability of claims made in the literature, particularly in the context of education (Brown, 2022). While this paradigm has been used extensively to verify claims made in quantitative research, including in education, psychology, and beyond, Makel et al. (2022) advocate for the adoption of replication as a necessary pursuit for qualitative research paradigms. Here, I argue that replication is not (always) possible with the majority of qualitative research approaches. The author’s commentary is framed around the claim “replication is relevant to qualitative research” and articulates the ways in which qualitative methodologies may benefit from replication attempts, drawing upon issues such as increased transparency, intentionality, transferability, and reflexivity. Key to the author’s argument is the notion that “the emphasis on context and perspective within phenomenological inquiry [that replication encourages] is highly compatible with the broad goals of replication” (p. 2). In this response commentary, I attempt to offer a broader, more nuanced view of this issue, questioning whether replication is even possible for qualitative research. I draw upon the critical questions of compatibility that have been applied to other aligned issues, related to the rigour and robustness of qualitative research (e.g., see Pownall et al., 2023), including the relevance of standardised reporting standards (see Barbour, 2001; Clarke, 2021), compatibility of preregistration (e.g., Pratt et al., 2020), and open data sharing (see Branney et al., 2023; Joyce et al., 2022).

What do we mean by replication?

To begin with, I think it is important to note here that I am sympathetic to the overall value of replication and agree wholly that replicability is an important cornerstone to many research paradigms, particularly in terms of quantitative, experimental work. However, in order to assess the compatibility with qualitative research, it is important to first delineate the concept of replication. Replication, through which replicability can be assessed, refers broadly to the process of re-testing a theory or hypothesis through a collection of new data. In classic direct replication paradigms, researchers seek to test a theory published in the literature by re-running the methods reported in the original papers with new participants, calling into question how robust the original findings are (Brandt et al., 2014). Replication has thus been thought to be a useful tool in determining the robustness or rigour of theory and is an important part of cumulative science (Asendorpf et al., 2016). The aims of replication efforts in quantitative paradigms are thus well established and clear. However, it is unclear what replication efforts actually look like when applied to the context of qualitative research. Makel et al. (2022) discuss how replication may be a useful assessment of “the transferability of qualitative findings” (p. 2). It is not clear, however, whether this claim refers to direct replications (i.e., the process of taking an exact method and replicating it) or conceptual replication (i.e., testing the same claim or theory with an altered methodology). If the authors advocate for the value of direct replication in advancing the understanding of qualitative transferability, I question the methodological and epistemological utility of this approach, which I expand upon throughout this paper. Relatedly, it is important to note too that while “transferability” of findings may be aligned with “replicability”, the two are quite distinct (Tuval-Mashiach, 2021). Therefore, this warrants a careful consideration of terminology.

In order to begin to consider what replication may “look like” in qualitative research, let’s consider an example. Imagine an educational researcher conducting a qualitative interview study, which assesses student’s lived experiences of an intervention. There are methodological analytical decisions to be made at every step of this research process, from the identification of participants to recruitment, interview style, type of analysis, and approach to the textual data as a whole. Therefore, can any study that attempts to “recreate” this approach ever, really, be considered a true “replication”? Indeed, as Makel et al. (2022) rightly describe, the analyst of qualitative data largely informs and influences the process and product of the analysis itself. That is not to say that there is a “potential risk of bias” (p. 3) but rather, this subjectivity is “baked in” to the research process. Bias, subjectivity, and influence of researcher positionality are inescapable, intentional, and by-design, for the majority of qualitative approaches, especially those that take social constructivist approaches to their study (i.e., those who come from the position that human experience is socially situated and constructed). Therefore, with this in mind, I question the extent to which replication is possible for qualitative research, not least “relevant”. In light of this, the authors may be (mis)appropriating the term replication to instead refer to the basis of qualitative research itself, which involves exploring (and often re-exploring) theories and questions using diverse forms of data collection. This issue points to my second point, which refers to a lack of definition and overly broad conceptualisation of “qualitative research” itself.

What do we mean by qualitative research?

In order to say something meaningful about the relevance of replication to qualitative research, we first need to establish not only what we mean by replication, but also how we define “qualitative research” itself. It is important to recognise that there is no one “qualitative research”; it is not a static, neat concept. As Clarke (2021) argues, qualitative research is a “messy swamp”. Indeed, even within the paradigm itself, there have been widely contested uses of methodological and analytical terminology to describe the process and product of qualitative research (see Varpio et al., 2017). Therefore, when making bold and concrete claims about the “relevance” of a process, principle, or mechanism to qualitative research, it is important to have a good handle on what specific flavour, approach, or type of qualitative research is being centred. Positivist-leaning content analysis will look and feel very different, in a replicability sense, to interpretative phenomenological analysis coming from a social constructionist approach, for example. Positivism, in this context, refers to research that comes from an assumption that scientific verification is possible, and that validity is determined by scientific evidence that uncovers a true “reality”. This may be best understood through the “small q/Big Q” distinction by Kidder and Fine (1987; see also Braun & Clarke, 2019). Small q qualitative research typically operates in a positivist framework and is generally descriptive, whereas Big Q is more interpretive, nuanced, and rich.

With this in mind, it appears that the kind of qualitative research that Makel et al. (2022) refer to derives from an implicitly positivist, or post-positivist epistemological and ontological position (see Carter & Little, 2007). Put simply, this kind of “small q” approach to qualitative research comes from the assumption that there is one “single identifiable reality” that researchers can objectively access through their methodology (see Lincoln et al., 2011). In other words, in re-testing the claims made of qualitative research researchers can collectively work towards establishing a transferable “truth” that can be (in)directly observed through qualitative methods. Although Makel et al. (2022) do not grapple with epistemological or ontological tensions directly in their paper, their implicit approach can be seen through their use of examples of where replication may be a useful mechanism by which to view transferability and uncover a “truth” or “answer”:

For example, an educational intervention that works in one context may be ineffective in another context, or at one time but not another. Qualitative evaluation findings are highly contextualised and their replication can give more detail about the nature and effectiveness of the intervention across contexts … . (p. 2)

While I appreciate Makel et al.’s (2022) notion of contextually-situated research, I would argue that a study that uses rich, qualitative methodologies to establish whether an intervention “works” will only produce findings that are tightly situated within the specific context of the intervention (and are relevant for the specific students within the intervention, when analysed by the same researcher). Attempts to locate “the same” results using qualitative methods are futile, given the contextual, personal, and situational influences of this approach. This calls into question the relevance of “replication” as a framework.

Further, to extend Makel et al.’s (2022) case study, if another educational researcher was to examine the very same intervention and assess its utility using the same broad qualitative methodologies (e.g., interviews with students or staff, focus groups, etc.), this would be a separate reappraisal of the theory or underlying research question. The extent to which this would constitute a “replication study” is not clear. Indeed, it may also be argued that the label of replication is unnecessary, given how qualitative research already has established principles and practices for examining how theories and research questions operate across data, researchers, and contexts. Issues including triangulation and crystallisation, for example, are more ontologically appropriate for qualitative methods and may offer a more nuanced perspective on this process. As an alternative to the “replication” discourse, triangulation refers broadly to the notion of enhancing the depth of understanding and rigour by considering and consulting multiple methods, theories, perspectives, and types of data (e.g., Shenton, 2004), and crystallisation, by extension, aims to enhance rigour through “comprehensiveness rather than convergence” (Richardson & St. Pierre, 2005). I argue that this terminology offers a more appropriate framework to consider how qualitative researchers can use varied sources of data collection to improve comprehensiveness or, to use Makel et al.’s (2022) terminology, “transferability” of qualitative findings. Indeed, while the argument that Makel et al. (2022) outline is, in places, useful and provocative, there is an issue of semantics that is troubling.

What do we mean by “rigour”?

The issue of using appropriate and epistemologically relevant terminology leads me to my final point. Beyond the issues of ontological and epistemological compatibility, it concerns me that, in the approach taken in their paper, Makel et al. (2022) are adopting values, principles, and tools derived from quantitative research to examine the robustness (defined by the authors as transferability, intentionality, and transparency) of qualitative research. This is not a new concern; Pratt et al. (2020) refer to “the inappropriate transfer of quantitative logics to qualitative research” which serves to (a) promote the long-standing idea that quantitative research is inherently more rigorous than qualitative approaches, (b) disregard the promising progress that qualitative researchers have made in establishing their own appropriate standards and practices (see also Carter & Little, 2007; Pownall et al., 2023). Aligned with this, while I appreciate the notion that rigour and transferability should be continually assessed and revisited, I am concerned about the discourse in which this operates. That is, the transferability of “replication concerns” to qualitative research proposed by Makel et al. (2022) may be contributing to the ongoing inappropriate transfer of quantitative practices and tools to qualitative research. This perpetuates the notion that quantitative practices are “gold standard”, and that qualitative work must mould itself around these standards (see, for example, Harley & Cornelissen, 2020, on inappropriate adoption of reporting protocols). As Jarzabkowski et al. (2021) neatly argue, efforts to standardise markers of “quality” may also be in direct tension with some of the hallmarks of qualitative research, such as pluralism, positionality, and creativity (see also Easterby-Smith et al., 2008). Therefore, “replication” may not be the most appropriate framework to consider the notion of (re)exploring a theory grounded in qualitative approaches. Future thinking should also, however, consider how replication may not always be possible in all quantitative approaches. The implications for mixed methods research could also be considered more carefully.

Conclusion

Overall, it is important for me to reiterate here that I do not disagree with some of the claims made by Makel et al. (2022). I think one of the most promising ways forward for scientific rigour is to centre and celebrate researcher reflexivity and positionality across research paradigms (see Jamieson et al., 2022) and I appreciate the notion that reflexive engagement should be considered in the context of replication. However, I hope this commentary adds nuance to the issues articulated by Makel et al. (2022) by critically questioning how, why, and when replication “makes sense” for qualitative research. I encourage a broader and more thoughtful engagement with how we, as researchers, define and conceptualise terms such as “replication”, “qualitative research”, and “rigour”, in light of these concerns.

Disclosure statement

No potential conflict of interest was reported by the author.

Additional information

Notes on contributors

Madeleine Pownall

Dr. Madeleine Pownall is a Lecturer in the School of Psychology, University of Leeds. She is an advocate for critical approaches to open research and a feminist social psychologist.