Is Critical Human Geography Research Replicable?

Abstract

The debate concerning replicable scientific research has reached geography’s shores. This has exposed old fault lines in our discipline, because some forms of geographical inquiry are more amenable to replicability than others. If there is a corner of the discipline that seems especially ill-suited to replicability, it is critical human geography. Almost no work in the subfield exhibits the combination of qualities—explicit and replicable methods; large, numerical data sets; full reporting—that enable reproducibility. Should we care? Although the inability of critical human geographers to reproduce our research results does not constitute a crisis, it is a matter worthy of reflection. Even if it proves difficult to realize, the challenge of designing replicable research promises to generate insights into the relative rigor of our disciplinary practices. Moreover, by clarifying the limits on replicability in social inquiry, we should be better positioned to weigh and mediate between competing values, for instance, the potential conflict between the principle of scientific integrity and the protection of vulnerable research subjects. I contend that producing a rigorous and reproducible geographical research, while also respecting the dignity of subaltern social groups, would require significant changes to standard research practice. To flesh out these claims, I offer concise reflections on the literature from critical human geography research with subaltern social groups.

科学研究可重复性的讨论已经涉及到地理学界, 也触及地理学科的软肋, 因为一些地理学研究比其它地理学研究更具有可重复性。如果地理学有一个不适合重复性的角落, 那就是批判人文地理学。这个分领域几乎不具备可再现性的特点：明确并可重复的方法, 数字化的大数据, 完整的报告。我们需要担心吗？批判人文地理学者不能再现其研究的结果, 尽管还构不成危机, 但值得我们思考。设计可重复的研究（即使实践证明这很难实现）, 可以让我们了解地理学方法的相对严谨性。此外, 通过澄清社会问题在可重复性上的局限性, 能更好地权衡、协调各种矛盾的价值（例如, 科学诚信原则和保护弱势研究对象之间潜在的冲突）。本文主张, 建立严谨的、可再现的、尊重社会群体尊严的地理学研究, 可能需要对研究的标准方法进行重大变革。为了充实这些观点, 本文简短地讨论了涉及社会群体的比较人文地理研究的文献。

El debate relacionado con el carácter replicable de la investigación científica ha llegado a las playas de la geografía. Esto ha puesto al descubierto los afallamientos de nuestra disciplina, por cuanto algunas formas de la indagación geográfica son más propicias a la replicación que otras. Si hubiere una esquina de la disciplina que pareciere inadecuada para la replicabilidad, esa sería la geografía humana crítica. Poco menos que ningún trabajo en el subcampo exhibe la combinación de cualidades –métodos explícitos y replicables; grandes conjuntos de datos numéricos; y reportaje completo– que posibiliten la reproducibilidad. ¿Debería importarnos eso? Si bien la inhabilidad de los geógrafos humanos críticos para reproducir los resultados de nuestra investigación constituye una crisis, si es materia digna de reflexión. Aun en el caso de que eso resultare difícil de llevar a cabo, el reto de diseñar investigación que sea replicable promete generar perspectivas en el rigor relativo de nuestras prácticas disciplinarias. Todavía más, clarificando los límites de replicabilidad en la indagación social, será mejor que nos apostemos mejor para sopesar y mediar entre valores en rivalidad, por ejemplo, el conflicto potencial entre el principio de la integridad científica y la protección de sujetos de investigación vulnerables. Sostengo que producir una investigación geográfica rigurosa y reproducible, mientras se respete también la dignidad de grupos sociales subalternos, requeriría introducir cambios significativos en la práctica investigativa estándar. Para dar sustancia a estas pretensiones, ofrezco reflexiones concisas sobre la literatura desde la investigación en geografía humana crítica con grupos sociales subalternos.

Key Words:

• critical human geography
• replicability
• reproducible geographical research

关键词:

• 批判人文地理
• 可重复性
• 可再现地理学研究

Palabras clave::

• geografía humana crítica
• investigación geográfica reproducible
• replicabilidad

Has the whole progress of science not up to now been manifested in the fact that new experiments and observations have corrected and extended previous experiments and observations? How could this happen if a given experiment were not reproducible, and if, with another observer, it could not be checked and extended, thereby giving rise to new and original connections?

—Antonio Gramsci (1995, 287)

T he debate concerning replicable scientific research has reached geography’s shores (Brunsdon 2016; O’Sullivan et al. 2018; Kedron et al. 2019).¹ Many engagements about reproducible and replicable research in the social sciences have taken the form of an oppositional debate: for or against replicability, very concerned about the crisis versus not concerned, and so on. This article attempts to transcend these oppositions. If we do indeed face a reproducibility crisis, then, as with any crisis, the first step should be to find one’s bearings.

Before wading in further, it might be useful to define key terms and show my cards on the basic questions. By reproducible I mean research that could be redone by another scholar with the same materials and produce the same results. By replicable I mean research that could be repeated by another scholar using identical procedures with new data. I use R&R as shorthand for the pairing of the two as normative goals. In adopting this language, I reiterate the recommendation of the National Academy of Sciences (2019) and the proposal by Kedron et al. (2019) to extend these protocols to geographical analysis.

Like most social science (Goodman, Fanelli, and Ioannidis 2016; Servick 2018; National Academy of Sciences 2019), very little contemporary research in human geography is reproducible or, indeed, reproduced. Yet I do not think it makes sense to claim that this lack of replicability presents a crisis for geography² in general or for critical human geography in particular. One could argue that we lack sufficient attempts at replicability (as I will later), but then we face more serious problems. To be sure, we want to be able to repeat and replicate experiments when it is possible and sensible to do so. Who wants to take a medication that has not been tested by rigorous research? Who doesn’t know what this means: studies executed as objectively as possible; replicable and, in fact, replicated; subjected to blind expert evaluation; published for public review, preferably with open access to original data and methods. Simply stated, replication means doing things methodically, repeatedly, and socially—so that data and results are shared with others. Put this way, replication describes best practice for much honest work. There is also the pedagogic function of replication, often lost in this discussion. Anyone reading this article has had a basic scientific education and will recall experiences in educational settings where passing science courses meant conducting preplanned experiments for which the teacher already knew the correct result. The point of such practices (which are necessary for the preparation of a healthy conception of scientific understanding) has nothing to do with the creation of new knowledge but with the cultivation and assimilation of a “spirit of re-creation” (Gramsci 1916).

Taken together, this means that the contemporary drive for R&R reaffirms the inherently social character of science. It amounts to saying that it would be good if more scientists engaged in open, social processes through which we could collectively test our knowledge. This is a claim that Gramsci emphasized in the notebooks he wrote while imprisoned by fascists—a source of abiding inspiration for critical human geographers (e.g., Wainwright 2010; Ekers et al. 2012). Gramsci treated science as a distinctively powerful conception of the world, capable of producing enduring insights due to its particularly social character (see epigraph). This line of reasoning carries potentially radical implications at a time when open, publicly supported, scientific inquiry is not enjoying great support.

Let us not, therefore, get bogged down in debate over the merits of R&R in the abstract. The difficult questions lie upstream from methodology, where the debate over R&R raises fundamental questions anew. What sort of geography do we need, and what sort needs replication? What differentiates geography from other disciplines, and does this distinctiveness carry any normative weight when we reflect on the need for research replication? How different are the objectives of divergent branches of geography? Does meeting these objectives require different forms of knowledge production?

The timing here is perhaps auspicious because we are living through times of momentous change. If our discipline has a second law, it must be that as the world changes, so, too, does the discipline. Perhaps R&R will allow us to track these changes—in our world and in our thinking—more rigorously. If so, then we should celebrate the challenge.

What Do We Want to Replicate and Why?

The drive for R&R exposes old fault lines in our discipline because some forms of geographical inquiry are more amenable to replicability than others. Brunsdon (2016) acknowledged this implicitly in a review paper on reproducibility in geographical scholarship: “Clearly, this idea is more practical in some areas of study than others, and resources are an important factor” (689). Brunsdon is correct about the differential practicality of R&R but does not explore the causes or implications of this unevenness. They are worth considering.

If there is a corner of the discipline that seems especially ill-suited to replicability, it is critical human geography. Of course, much depends on how one defines critical human geography as a discipline, but there is no general agreement on this question. When the subfield initially emerged, it was called radical geography. Radical carries a double meaning of getting to the roots and radical politics from the Left. A recent collection of essays on the historical geography of the subdiscipline (Barnes and Sheppard 2019) treats radical and critical almost synonymously, and to my reading, neither term is defined precisely. It might be useful to speak of radical or critical geography and to return to the writings of the German philosopher Max Horkheimer, cofounder of the Frankfurt School and a major influence on radical or critical geography at its inception. For Horkheimer (1972), like Gramsci, critical theory takes the totality of social life as its object.³ The aim of this critical theory is not a perfect description of social life, nor a better functioning of society, but the transformation of social relations in toto. Its measure is not, therefore, the perfection of truth about society but the realization of freedom and equality. As Horkheimer (1972) explained:

[Critical theory’s] opposition to the traditional concept of theory springs in general from a difference not so much of objects as of subjects. … The scholarly specialist “as” scientist regards social reality as extrinsic, … and “as” citizen exercises [an] interest in them through political articles, membership in political parties or social service organizations, and participation in elections. But … these two activities [are not unified]. … Critical thinking, on the contrary, is motivated today by the effort really to transcend the tension between the individual’s purposefulness, spontaneity and rationality, and those work relationships on which society is built. (209–10)⁴

It follows that critical theory seeks to overcome itself and this tension—not through the adjustment of the individual to particular circumstances but through the collective transformation of social life.

However the subfield might be defined, I think it is fair to say that very little critical social research in geography today exhibits the combination of qualities—explicit and replicable methods; large, numerical data sets; full reporting—that enable reproducibility. I conducted an informal search through the literature on critical human geography to try to find illustrations of works that met the standards for R&R research; I found very little. I reached out to some colleagues for suggestions. None could point to specific works. For his part, Brunsdon cited one exception: Bergmann’s (2013) study of the geographies of embodied carbon emissions. Bergmann’s paper is unusual not only for its blend of methods and critical sensibility (which he characterized as “postpositivist”) but also for an appendix to the paper that provides the data and code for his input–output models. In theory, another scholar could rerun Bergmann’s analyses, with either his data or different data.⁵ The lack of retesting is not necessarily a problem—the virtue of replicability is not dependent on actual replication of each research project—yet it confirms the general pattern. Given the rarity of replication of critical geographical research, it is not surprising that the one study that could be replicated has not been. No doubt this reflects in part disciplinary training and labor power: Not many critical human geographers possess Bergmann’s disposition and training. To be sure, Bergmann’s is not the only geographical study to court reproducibility. Yet it reflects a gesture in a direction where few other critical human geographers are prepared to follow.

This is by no means the only factor shaping the paucity of R&R work in radical or critical geography. Reporting on data from 1,576 researchers who completed an online survey, Baker (2016) pointed out that a majority (52 percent) believe that there is a “reproducibility crisis.” The survey identifies eight factors that at least 40 percent of the respondents believe “always” or “often” contribute to irreproducible research. At least five of these eight factors pertain to critical social research. They are as shown in Table 1, with brief illustrations of why they might legitimately occur.

Table 1. Factors inhibiting reproducibility and replicability in critical social science (drawing from Baker 2016)

Factor	Some reasons it may occur in critical social science
Selective reporting	Concern for the well-being of human subjects
Pressure to publish^a	Dynamics inherent to the market for academic labor power
Low statistical power	Small N research; for example, textual (close reading, archival work) or social (interviews, focus groups, participant observation)
Methods or code are unavailable	Difficulty in explaining the method—the thought—from which novel social insights derive
Raw data unavailable	Many forms of social data are protected and cannot be shared ethically

^a But see Fanelli, Costas, and Larivière (2015).

Note that these factors may persist even where we somehow bracket the question of ideology or the interpretative framing of events and data that are inherent to social analysis. However, we cannot bracket these matters forever.

As presently practiced, R&R will be most readily enacted when the object of research—the phenomenon under study—can be subdivided in a replicable fashion.⁶ If an object can be defined precisely (with attributes that can be directly, reliably, and repeatedly measured) and the research questions limited to measuring interactions among these attributes, then R&R should be possible. Nevertheless, doing so would not necessarily be worthwhile unless study of these attributes and their spatial relations was sufficient to establish meaningful knowledge of that phenomenon.

These basic conditions do not typically apply to the object of radical or critical research, namely, society grasped as a complex, differentiated totality. Broadly speaking, most critical social research is concerned with understanding how different people, or groups of people, understand some social phenomena—typically because the research would like to change those very conceptions. Indeed, we might wish to eliminate them. For instance, we might study racism to find better ways of ending it. Hence, we are concerned not only with the interactions of given attributes but with the way the meanings of these attributes and their interactions are conceived, or grasped, by particular subjects under given historical and geographical conditions. To say the least, this complicates the challenge of defining the objects of research. It follows that one must address two or more levels of analysis simultaneously: the object but also the perception of the object (both varying in time and space). This leads to the much discussed problem of the subjectivity of social knowledge (or what Marxists sometimes call the problem of ideology; see Harvey 1972, 1974). Because social researchers are always already implicated with the phenomena typically studied—some subset of language, social relations, and history—it is difficult to define objects of social research in ways amenable to definite, precise, and reliable boundaries. The basic meaning of historical events, even their recognition as events, often differs strikingly between people. You and I might live through the same event but have radically different understandings of what has occurred. For that matter, you and I might speak the same language, but my language is not yours (nor is it really mine). The effort to formalize social concepts, which has its virtue, finds its ultimate limits in language and history, which none of us can escape. Consider instead a more conventionally narrow and quantitative method, such as a survey. Two surveys, with the same questions, given at two different points in time, produce comparable data (panel data); variations between results do not necessarily imply that the previous round was wrong, nor do they necessarily prove that change has occurred (as some variation could be explained by random variations from a mean, which could fluctuate over time). Thus, even if we assume the best intentions on the part of the researcher, a reasonable attempt to carefully study a social problem (with adequate self-reflection and appropriate mentoring and oversight), as practiced by most critical social researchers, does not lend itself to reproducible research.

Well, should we care? Although the inability of radical or critical human geographers to reproduce our results does not constitute a crisis, it is a matter worthy of reflection. Even if it proves difficult to realize, the challenge of designing replicable research promises to generate insights into the relative rigor of our disciplinary practices. Moreover, by clarifying the limits on replicability in social inquiry, we should be better positioned to weigh and mediate between competing values; for instance, the potential conflict between the principle of scientific integrity and the protection of vulnerable research subjects. R&R raises a special complication for critical human geography, where a great deal of our data are ultimately derived from the experiences and knowledge of diverse, subaltern social groups.

There is a strong case to be made that reproducibility presupposes that data are freely available for scholars to evaluate (Al-Quraishi and Sorger 2016): No data, no replication. By implication, critical human geographers whose data concern subaltern groups face a dilemma: Make the data available to others or fail to meet the protocols of R&R (on provenance and R&R, see Tullis and Kar 2020).

Arguably the two most fundamental normative standards to which critical social scientists adhere are tell the truth and do no harm (i.e., do not put those subjects or coparticipants in knowledge production at risk). The former might require making data fully transparent; the latter could well be compromised by openness.⁷ In addition, logistical complications can arise with sharing qualitative data and its interpretation (activities fundamental to critical social science). Interview data that were transcribed and translated are not typically made freely available to the public because of potential risk to human subjects in the future. Consider, too, the challenges entailed with defining and sharing experiential data from singular events, such as participant observation of particular activity at a given time and space. Even a video recording of the same activity would not be equivalent to the existential experience of the researcher who was present; moreover, the interpretation of the events from the video would vary. Thus, there are commonsensical reasons for the paucity of reproducible social science.

Conclusions

I conclude by drawing four concise arguments out of this analysis. First, we should neither fear nor fetishize reproducibility. Just because research can be replicated does not mean that its results are infallible, nor that the research is inherently important and valuable. Trivial questions can be studied with great rigor; essential problems are often difficult to grasp. The absence of R&R in human geography is not a crisis. To avoid hyperbole, the word crisis should be reserved for genuinely threatening problems, such as the crisis of planetary climate change. This is a real crisis and one about which critical human geographers have contributed an outstanding body of scholarship. Practically none of that work would meet the standards of R&R protocols. Thank goodness the work was published nonetheless. Still, we should not look at this matter one-sidedly. The planetary climate crisis can be seen as the largest irreplicable scientific experiment in history. It would be helpful if we had more prior experience with what lies ahead. This intuition points again to the value of R&R.

Second, to recognize that there are competing sides to this discussion does not condemn us to an endless oscillation between the two poles, nor to pointless academic debate. Tensions exist between critical and R&R approaches, but these should not be essentialized or ontologized into warring camps. Creative attempts at transcending differences should be welcomed. Still, there are also some serious logistical and practical complications that make it difficult to meet the two challenges (genuine social critique and protocols of R&R) simultaneously. Although we should affirm Fanelli’s (2017) contention that the “only real antidote to bad science and misguided policies [lies in] open and transparent scholarly debate,” we also need to bear in mind two points. First, debate is necessary but in itself insufficient to change “misguided policies.” Second, the conditions of possibility for critical and replicable social research extend beyond open scholarly debate.

Third, the conditions of possibility for R&R in radical and critical human geography do not presently exist. I have already noted several issues; others could be added. For instance, the program outlined by Gertler, Galiani, and Romero (2018) to “make replication the norm” focuses on the publishing system. Brunsdon (2016) similarly concluded his review by arguing that “the adoption of reproducible approaches” in human geography will imply “some changes in the practice of both researchers … and publishers—in providing a medium where reproducible documents may be easily submitted, handled and distributed” (694). His focus lies with quantitative data and code. The problem is more complex with qualitative social research. Simply put, this entails converting the raw material of social analysis into reproducible documents freely available for public distribution and discussion. I will make a general assertion that this is a principle opposed by every undemocratic state.

Finally, this is an article on critical social science, so, appropriately, my final point concerns the capitalist character of the industry in which this debate plays out. Please do not think that the for-profit publishing industry is going to be our helpful friend in the pursuit of replication. The largest publishing corporations, including Springer Nature and Elsevier, have published many editorials calling for replication. True, they want to avoid fraud, but they also want your data—not only the raw data of your research (so that future scientists go to them, not a public repository, for access); they also want your metadata, which is the basis of their move into data analytics. I would love to see my fieldwork tested by replication, but I would not trust Elsevier to hold my data in their digital archive (see Chan 2019; Chen, Posada, and Chan 2019; Büscher 2020; Wainwright and Bervejillo 2020). The movement for publicly funded, not-for-profit open access publishing, which finds a natural ally in the R&R ethos, has a long way to go.

Thus, I do not believe that radical or critical human geographers—or any other social scientists—should oppose scientific replicability. We should embrace its inherently social ethos but without illusions that the conditions of possibility presently exist for replicable critical social research. Among other things, it would require trustworthy, public, and open—hence, radically democratic—institutions that would guard the materials (texts and data, broadly defined) necessary for collective reappraisal of social interpretation. Many such institutions have existed, and some still do, but in the world in which we live, like many good things, they are presently under attack. Where the drive for replicable research arrives at the shore of social research, it becomes an inherently political demand.

Additional information

Notes on contributors

Joel Wainwright

JOEL WAINWRIGHT is Professor in the Department of Geography at the Ohio State University, Columbus, OH 43210. E-mail: [email protected]. His research concerns political economy, social theory, and environmental change.

Notes

1 It would be more precise to say that the debate has returned to geography, because the present discussion recapitulates questions raised during the early years of the quantitative revolution (Schaefer 1953; Harvey 1969; on the Schaefer–Hartshorne debate and R&R, see Sui and Kedron 2020). Pursued rigorously, this article would historicize the debate and its terms (see Harvey 1972, 1974; Gregory 1978; Bowen 1981; Gramsci 1995; Mercer and Wainwright 2018; Barnes and Sheppard 2019). Space does not permit this.

2 For reasons identified by Baker (2016) and Fanelli (2017, 2018), elaborated with respect to critical human geography later.

3 See also Horkheimer (1987). For a useful review of critical theory’s place in philosophy, see Bohman (2019), who wrote, citing Horkheimer (1972), that a theory is “critical to the extent that it seeks human ‘emancipation from slavery’, acts as a ‘liberating … influence,’ and works ‘to create a world which satisfies the needs and powers’ of human beings” (246).

4 Translation modified to address Horkheimer’s gendered language. Many feminists have made similar arguments; see, for example, Mies (1993).

5 Bergmann’s paper has been cited thirty-five times (per Google Scholar, 16 October 2019). By reading the titles and abstracts of those papers, I found nothing suggesting any retests. I emailed Bergmann to ask and learned that although no one has retested his data with the model, a PhD student is using the model for another purpose. See also O’Sullivan et al. (2018) and Kedron et al. (2019).

6 This paragraph is indebted to Peter Kedron, whom I thank.

7 Richardson (2019) described one attempt to address this issue.