Ignorance and the paradoxes of evidence-based global health: the case of mortality statistics in India’s million death study

ABSTRACT

Quantitative evidence and metrics play a central role in contemporary global health. Mortality statistics, for example, are considered essential for improving health in the global South. Yet, many observers lament that reliable cause of death data is not available for many low- and middle-income countries. The Million Death Study (MDS) in India forms an effort to address this issue, seeking to reduce ignorance around mortality by generating representative statistics by combining an existing, representative demographic sample with an innovative diagnostic method called verbal autopsy. Yet, ignorance is more than the absence of reliable mortality statistics in this study. Social science perspectives on institutionalized ignorance can help unpack how certain paradoxes of evidence-based global health manifest through three different articulations of ignorance in the MDS. First, the study’s simultaneously national and global ambitions intersect in arguments that present ignorance as legitimation for the study. Second, ignorance is presented as instrumental in balancing the need for expertise with the risk of bias in diagnosing causes of death. Third, MDS researchers dismiss remaining ignorance or uncertainty about diagnoses, by claiming it is relative compared to the ‘actionability’ of study results for improving public health. In exploring these various manifestations of institutionalized ignorance, several paradoxes of the MDS as an evidence-based global health project become visible. By exploring these paradoxes, this analysis suggests that studies of institutionalized ignorance can provide novel perspectives on how deliberate articulations and mobilization of ignorance helps constitute evidence-based global health.

KEYWORDS:

• Mortality statistics
• India
• evidence
• global health
• data
• million death study

Introduction

We sit in front of a laptop screen in an office in a nondescript hospital building, somewhere in an Indian metropolis. My interview partner has opened a file that contains the scanned version of a form, on which a government official has noted down how a relative of a recently deceased person describes the events of the days and weeks preceding that person’s death. This description of ‘signs and symptoms,’ as my interview partner calls them, is what is called a verbal autopsy in the Million Death Study (MDS) in India. This study has the aim of producing more accurate and representative mortality statistics than those the allegedly deficient Indian medical system provides. Our look at the laptop screen serves as an illustration of the process by which a medical doctor like my interview partner can deduce a cause of death from notes taken during a proverbial, yet highly structured, household conversation. I get to see how a trained physician might identify and highlight key terms used in the short narrative in front of us. These terms may suggest a particular medical history to the physician, yet the software platform on which the study runs simultaneously uses the highlighted terms to guide the diagnostic process. It suggests certain diagnoses to consider and blocks others that are highly unlikely or impossible in light of the available data. The combination of the physician’s expertise and software’s suggestions are expected to generate an accurate diagnosis for this death; this exercise applied to thousands of forms from all around the country ought to result in aggregate quantitative evidence of the conditions people in India die from.

Quantitative evidence and metrics play a central role in contemporary global health (Sangaramoorthy and Benton, 2012; Adams, 2013, 2016). Researchers, government agencies and international aid organizations collect quantitative data to describe the scale of the most pressing health problems and to evaluate the effects of health programs. Such efforts position numbers as the backbone of the highly influential paradigm of evidence-based global health (McInnes et al., 2012). Quantification in the context of global health is increasingly concerned with the burdens and (economic) costs of disease. This has resulted in the development and use of increasingly sophisticated calculative models and devices for estimating the global impacts of ill health (Murray and Lopez, 1996; Gaudillière and Gasnier, 2020). Advocates of numerical evidence point to the (still) widespread absence of cause-specific data on mortality as a major deficiency for health policy in the global South (Deaton, 2013). Alan Lopez, one of the major voices in efforts to track the global burden of disease, for example, has recently argued for the importance of civil registration and the collection of data on causes of mortality. Together with his co-authors, he highlights the importance of causal mortality data as vital for assuring that ‘efforts to improve health are truly based on evidence, not ignorance’ (Lopez, McLaughlin and Richards, 2020, p. 5).

The MDS forms an exemplary effort to generate representative national mortality statistics, using a low-cost method that is presented as easily applicable elsewhere and thus instructive for evidence-based global health (Birbeck et al., 2013). Rather than establishing a national mortality registration system, the MDS pursues nationally representative mortality estimates by combining an existing, representative demographic sample with the verbal autopsy method. In a nutshell, the MDS works with the so-called Sample Registration System (SRS), which consists of nearly 1% of India’s population. This sample was established to trace population dynamics (births, deaths, and migrations) during the ten-year intervals of the full census. The MDS added the verbal autopsy method to the SRS, which entails asking relatives of the deceased questions about symptoms occurring around the time of death. Diagnoses of causes of death based on the interview data can be aggregated to produce a (supposedly) representative picture of the major causes of mortality in India. In light of the alleged shortcomings of existing estimates, the MDS can thereby address what one senior researcher described to me as the ‘extraordinary widespread degree of ignorance about how people die’ (Interview 1b, epidemiologist and study initiator) in India.

Ignorance thus comes to the fore as a major problem in the context of both public health in India and global efforts to improve health. However, social science work describing ignorance as ‘a productive force in itself, as the twin and not the opposite of knowledge’ (McGoey, 2012, p. 3) would suggest that more is at stake than merely not knowing what people die from. Social scientists have described how ignorance can be both a problem and a resource for the description and solution of social problems. This symmetrical view of ignorance will be adopted here to explore how the MDS is positioned vis-à-vis both evidence-based global health and the Indian state. The MDS is based on a critique of established methods for estimating mortality that are widespread in global health, while it simultaneously bears many of the institutional and financial traces characteristic of contemporary global health initiatives – including their central focus on numbers, and key roles for private financiers and research institutions from the global North (Brown et al., 2006; Birn, 2009; Crane, 2010).

The study’s use of the SRS as its sample means it is deeply integrated in the Indian state bureaucracy, while it simultaneously entails a critique of the state’s inability to produce reliable causal mortality estimates. The MDS is therefore positioned in between the ambivalent role of the state and multiple references to ignorance in evidence-based global health. An analysis of the generation of mortality statistics in the MDS can help to answer some key questions regarding institutionalized ignorance in relation to data and evidence in global health. What role do different articulations of ignorance play in the study? How are they institutionalized in the study? How does the institutionalization of ignorance help position the study in the global health landscape, both in India and internationally? How can social science perspectives on institutionalized ignorance provide insight into evidence-based global health?

I answer these questions by tracing the different manifestations of ignorance researchers and government officials referred to in discussions of the MDS. To explore how different references to ignorance reflect paradoxes within data-centric approaches to global health in India and beyond, I first explore different conceptualizations of institutionalized ignorance in sociology and science and technology studies (STS). I then illustrate how MDS researchers articulate different forms of ignorance to argue for the necessity and particular configuration of the study. Several paradoxes of data-centric, evidence-based global health become visible in their narratives. First, the study’s simultaneously national and global ambitions intersect in arguments that present ignorance as legitimation for the study. Second, ignorance is presented as instrumental in balancing the need for expertise with the risk of bias in diagnosing causes of death. Third, researchers dismiss remaining ignorance or uncertainty about diagnoses, by claiming it is relative compared to the ‘actionability’ of study results for improving public health. Each of these manifestations demonstrates how ignorance is critical to the particular configuration of the MDS and points to the paradoxes at the heart of evidence-based global health. This analysis thereby suggests that studies of institutionalized ignorance can provide novel perspectives on how ignorance helps constitute evidence-based global health.

Conceptualizing ignorance and the contingencies of data-centric global health

Literature on the emergence of ‘global health’ has emphasized how this notion emerged from, but is defined differently than pre-existing notions of ‘international’ health. Although various authors stress the heterogeneity of the global health concept, widely shared key properties include a focus on health issues in the global South and its institutionalization outside of established institutions of international collaboration (Brown et al., 2006; Kaplan et al., 2009; Lakoff, 2010). While various ‘frames’ shape global health governance (McInnes et al., 2012), epidemiological evidence and quantification are among the most influential approaches to global health issues, as they allow evaluation and comparability of interventions across space and time (McGoey et al., 2011; Reubi, 2018). Many global health programs and interventions are therefore formulated in ways that allow for the collection and assessment of numerical data.

The widespread use of such metrics and other numbers in global health has far-reaching consequences. Quantification shapes what is considered worth knowing, how diseases are understood and how health interventions are evaluated (Adams, 2016). Numbers thereby affect many aspects of global public health, ranging from individual disease experiences to governance and funding decisions (Sangaramoorthy and Benton, 2012). Moreover, the insistence on quantitative evidence is closely aligned with evaluation practices of donors and academic institutions in the global North, which contributes to the persistence of unequal relations in global health partnerships (Crane, 2010) and narrow problem definitions (Birn, 2009; Biehl and Petryna, 2013).

Scholars in the social sciences have therefore widely critiqued the effects of the centrality of numbers in global health. Such critiques emphasize both how quantitative approaches provide a reductionist view of the complex lived realities of health and disease and how they obscure the messiness of knowledge production practices (Biruk, 2012). Moreover, the (often implicit) expectation in global health discourse that interventions in particular places can unproblematically be transferred elsewhere ignores how global health activities are always geographically situated, both shaping and shaped by their environments (Herrick, 2016). For the MDS, the widely discussed deficiencies in data generation and overall delivery of public health services in the Indian health system add particular urgency to the study (Global Health Watch, 2011; Drèze and Sen, 2013; Economic and Political Weekly, 2015). At the same time, the demographic SRS sample used for data collection is unique to India, and efforts to collect data on a population this large has particular effects on how a population gets perceived and defined. Along these lines, Singh observes for the Indian biometric database project Aadhaar how the size and diversity of India’s population presents particular data generation challenges (Singh, 2019). In the case of Aadhaar, and to a lesser degree in the MDS, this problem was solved by combining the state’s ambition to make a population legible with the ability of non-state actors to generate and process data in new ways. Such efforts create, as Singh points out, India anew in the form of a database (ibid.).

In the context of global health, the intersections between its institutional and epistemic dimensions can productively be explored in terms of ignorance. In social science research ‘ignorance or non-knowledge are conceptualized and constructed as a fundamental part of social life’ (Gross, 2007, p. 743), rather than a mere absence of knowledge. Ignorance holds a moral significance that is quite pronounced in the use of this notion in the context of global health. Yet, it can have multiple meanings. Ignorance as a category can be deliberately and strategically employed, for example, as ‘a state which people attribute to others [that] is laden with moral judgement’ (Hobart, 1993, p. 1). Drawing on long-standing engagements in science and technology studies (STS), we can take a symmetrical approach to knowledge and non-knowledge (Paul and Haddad, 2019), understanding both as constitutive of and constituted by social and political realities. Like knowledge, ignorance is a socially constituted and constitutive phenomenon that has different dimensions across time, degree, and scale (Croissant, 2018). Paul and Haddad give various examples of how ignorance is constitutive to public health policies. They show how responsibility for vaccination program improvement can be avoided by not collecting data on uptake (Paul and Haddad, 2019). Likewise, we will see how ignorance plays a variety of roles – both implicitly and explicitly – in constituting the MDS and in arguments for its relevance.

Ignorance, which I describe, for the time being, as ‘not knowing something’ can form both a positive and a negative aspect of social life – a problem or an opportunity. This paradoxical character of ignorance intersects with the paradoxical institutional configuration of global health, both within the MDS and beyond. To make sense of these paradoxes of institutionalized ignorance – that is, ignorance embedded in the organizational and operational set-up of the MDS – I propose that Spivak’s reflections on historiography and asymmetries between different ways of knowing may be instructive. According to Spivak, some forms of (not) knowing are seen as more legitimate than others in the writing of history. In this context, she proposes the concept of sanctioned ignorance (Spivak, 1988b) to describe how historiography, and social theory in general have marginalized and disqualified non-Western ways of knowing (Spivak, 1988a). She describes such disqualifications as a form of epistemic violence that fails to consider other ways of knowing to successfully create versions of history that exclude subaltern perspectives (Spivak, 1988a). Such paradoxical ‘successful failure’ is linked to a particular perspective on what counts as historical evidence. As the ‘cornerstone of the edifice of historical truth’ (Spivak, 1988a, p. 7), evidence thus legitimates the paradox of sanctioned ignorance.

Spivak’s reflections on the paradoxical status of different forms of ignorance and the asymmetries embedded in ideas of evidence provide a productive entry point for considering institutionalized ignorance in the context of global health. The idea of successful failures suggests that the meanings of ‘not knowing’ within a study like the MDS can be diverse. To explore how ignorance is institutionalized in the MDS and in global health more widely, it is therefore instructive to follow Spivak to ask which paradoxes may be revealed when tracing different articulations of ignorance, especially in relation to metrics and quantification. Studies of metrics and quantification in global health have made clear that numbers both make some dimensions of health and disease visible, while obscuring others, often highlighting biological and diagnostic perspectives at the expense of social and structural ones (Sangaramoorthy and Benton, 2012; Adams, 2016). They have further shown how such epistemic paradoxes are co-produced (Jasanoff, 2004) with institutional paradoxes between the increasing role of both public and private non-state institutions such as research organizations or philanthropies and the continued relevance of the state, which is often still a gateway for delivering health services (Brown et al., 2006). We will see how in the context of the MDS, particular understandings of disease and interactions between state and non-state actors shape the study. At the same time, detailed study of different articulations of ignorance in the MDS can make visible the mechanics of more specific paradoxes between national and global ambitions, between knowledge and bias, and between accuracy and actionability in this specific case. Below I will explore how a focus on institutionalized ignorance can help identify these paradoxes, and what that may tell us about the broader idea of evidence-based global health.

Methods

In my analysis of the paradoxes of evidence-based global health expressed in the MDS, I trace how researchers map the causal distribution of mortality in India, focusing on how different perceptions of and references to ignorance contributed to the institutional shape of the study and its knowledge production processes. My analysis is based on interviews with study researchers, government officials and a health activist, as well as the analysis of (scientific) publications and internal documents (e.g. data collection and diagnostic manuals). I interviewed thirteen persons in total, most of them at research institutes and government agencies in major cities in India during a six-week visit in December 2013 and January 2014. One interview took place with three participants at the same time and I interviewed another person twice. To identify possible interview partners, I first contacted (and interviewed) study director Prabhat Jha, who suggested six further individuals who were or had been involved in setting up and operating the study. Further respondents were either suggested by this first group of people, or identified via literature research on public health in India.

All interview partners are anonymized, but interviews are numbered and the respondents characterized by a short indicator of their status in relation to the study. On a few occasions, descriptive details may make participants identifiable for people familiar with the MDS and its personnel. However, in most cases where a chance of identification is unavoidable, interview partners and their positions were already well-known within the study, and these participants often very outspoken in their opinions. This, to me, reduces most of the ethical risks associated with potential identification. Academic publications reporting on the aims, methods and results of the MDS were identified via PubMed; study protocols and guidelines are largely available via the study homepage (https://www.cghr.org/projects/million-death-study-project/, last accessed 10 October 2022). Together, interviews and documents allow for a reconstruction of the study’s institutional configuration and its mode of knowledge production vis-à-vis both the alleged dire state of public health in India and the promise of quantitative global health approaches.

While it was not my initial research focus, I identified various references to ignorance and similar ideas during my analysis, which I decided to pursue further. I began the analysis by applying the qualitative mapping techniques of situational analysis, which serve to identify the various elements (e.g. human actors, institutions, technologies) involved in a ‘situation’ (the unit of analysis), as well as these elements’ positions and relations (Clarke, 2005; Clarke et al., 2015). This mapping helped in generating an overview over the organization and operations of the MDS and to identify how researchers and administrators described their work. Over the course of this analysis, multiple (sometimes implicit) references to and articulations of ignorance came forward as an important part of the repertoire respondents and documents used for describing the study. I explored and theorized themes related to ignorance further by drawing on the tenets of abductive analysis (Tavory and Timmermans, 2014). Abductive analysis suggests an iterative approach that looks for commonalities and differences between theory and data. In this case, this approach for example involved further exploration of statements in the literature on how references to how ignorance can be used as a ‘resource’ to the particular ways MDS researchers would talk about the value of lack of medical knowledge among data collectors (see below for details). In sum, I adopted the idea that ignorance can be used strategically from the literature, but ultimately drew the characterizations of different forms of ignorance that structure the discussion below from the material I analysed. The characterizations of ignorance as legitimation, as instrument and as a relative property below are thus primarily empirically grounded. Yet, they complement other ways of thinking about the social life of ignorance in relation to evidence-based global health.

Ignorance in the causal quantification of mortality in India

The motivations for, and particular configuration of the MDS are perhaps best characterized by recounting the study’s origin story. Several interview respondents located the study’s origins in the mid – to late 1990s, in conversations between an epidemiologist working for the World Bank and a demographer. Both were concerned about the absence of reliable cause-of-death information for the Indian population. As the demographer who was one of the study’s initiators told me, ‘[w]hat we found in vital registration is that at national levels, at state levels, these estimates [of causes of death] were not possible’ (Interview 3, demographer and study initiator). As the epidemiologist moved on from the World Bank to the Center for Global Health Research (CGHR) at the University of Toronto, he and his demographer colleague at the Office of the Registrar General of India (RGI), part of the Ministry of Home Affairs, continued to think about ways to produce better mortality data.

They ultimately combined their expertise and resources to use an existing infrastructure for monitoring population dynamics, the Sample Registration System (SRS) combined with an interview-based diagnostic method called verbal autopsy. They thereby established an infrastructure for data generation and analysis located outside India’s public health system, while involving the Indian state through the SRS. They also engaged a broader network of domestic and foreign research institutions in a configuration typical of asymmetric collaborations in global health (Crane, 2010). Since data collection is already largely covered by the existing SRS infrastructure, one of the study initiators described the study as cheap, suggesting it costs less than one dollar a day, money mostly spent on staff for data collection and analysis. Nevertheless, this person joked that the MDS is ‘underfunded by several organisations’ (Interview 1a, epidemiologist and study initiator), having its operational costs covered by RGI, and relying on small grants from medical research funding agencies in Canada, India and the United States as well as philanthropic organizations such as the Bill and Melinda Gates Foundation.

The institutional configuration of the study suggests that it is located at the intersection between state-centered population monitoring and the public-private, diffuse international networks of global health research and intervention. This configuration is further reflected in the study’s modes of knowledge production and articulations of ignorance. These each contain a paradox associated with the specific ways that governance, knowledge and health come together in (this particular instance of) evidence-based global health.

A basic conviction informing the study is that reliable knowledge on causes of death for the majority of people in low- and middle-income countries is currently not available. Such ignorance serves as legitimation for developing an alternative approach to estimating mortality. This approach involves the use of the SRS for data collection. Described as the ‘largest demographic survey in the world’ (Office of the Registrar General of India, 2013), the RGI uses the SRS to collect data on migration, births and deaths from a representative sample of eight million people (0,8% of the population), at six-month intervals.

For the estimated 50,000 deaths that occur within this sample annually, additional data collection techniques are applied. These provide the diagnostic data for the MDS. Study initiators have developed the verbal autopsy method, which employs ignorance as an instrument for data collection and analysis. Government officials who collect data for the SRS ask relatives of recently deceased individuals additional questions about symptoms occurring prior to a person’s death. On the basis of descriptive data about these symptoms, medical doctors assign causes of death.

The aggregate diagnoses from all cases provide insight into the country’s mortality profile, and are supposed to inform public health policy. Researchers acknowledge that the statistics generated in this manner have some shortcomings. Yet, they argue that any remaining ignorance is a relative property compared to the alternatives of not having any, or only largely inaccurate, data. Different forms of ignorance point to paradoxes in knowledge production and the institutional configuration of the study vis-à-vis its claims to contribute to improvements in health policy. I will now turn towards how these articulations of ignorance reflect the paradoxical institutional configuration of evidence-based global health in India.

Ignorance as legitimation of a localized global health study

Initiators of the MDS indicated that they see the unavailability of reliable, cause-specific mortality statistics for the whole of India as the main legitimation for the study. They attribute the absence of these statistics primarily to shortcomings in the Indian public health system and civil registration. Although several systems for collecting cause of death statistics were already in place when the MDS was developed, the study’s initiators considered none of these reliable. Some of these systems relied on causes of death assigned by family members with insufficient medical knowledge to be able to accurately diagnose the deceased. Others were based on diagnoses by clinical professionals, resulting in a strong bias towards the small minority of deaths, primarily occurring within the urban population, that received medical attention. This means that the majority of deaths are not included in the statistics. Moreover, the statistics have a bias towards causes of death that are more common among urban populations with access to medical care. MDS researchers therefore argue that extrapolations and estimates based on these cases paint a misleading picture. Rural, medically unattended populations who die from different causes are not included.

One of the study initiators emphasized how incomplete and inaccurate mortality registration is a major public health problem, because ‘people don’t really understand how important understanding causes of death is to improving health of the living’ (Interview 1b, epidemiologist and study initiator). This person further emphasized that accurate mortality data can help correct or confirm assumptions about which public health threats are most prevalent. Their fellow initiator refers to the so-called epidemiological transition, which describes the shift from infectious to chronic diseases as the major source of mortality, as a specific example. This person argues that ‘[t]here are many states in the country where infectious diseases are under control, there is a gradual shift now.’ (Interview 3, demographer and study initiator). This observation suggests that existing mortality estimates at the time remained ignorant of this changing reality, thus requiring an alternative approach.

MDS initiators and researchers blame this ignorance on institutional deficiencies. Since there are no appropriate mechanisms or routines for collecting mortality data in place, an entire alternative set-up is required. This conviction has given rise to what researchers claim to be one of the main methodological innovations within the MDS. One researcher indicated how the institutional deficiencies mean that there is ‘no alternative for India,’ since, ‘in the immediate future, there will be no clinically certified deaths in India’ (Interview 8, researcher). This perspective emphasizes that the MDS is a response to structural shortcomings in the medical system’s ability to produce comprehensive and reliable mortality data. Researchers seek to mobilize the established demographic infrastructure of the SRS as an alternative. The basic idea of the MDS was to use a system that had already been in place for decades to gather representative data on changes to India’s population and to see ‘whether for the deaths in SRS we could assign a particular cause of death’ (Interview 2, study coordinator). Yet, due to the size of the SRS sample and the need for data collection at regular intervals, a former study collaborator observed that the MDS ‘can’t be done like a research methodology. It has to be incorporated in the system’ (Interview 6, former study coordinator and current government official). The institutional set-up of the MDS thereby reflects how a demographic infrastructure came to be considered a more reliable source of data than the medical system.

MDS researchers’ ambitions do not stop at the border, however. Even the government agency involved touts the study as an initiative that is ‘not only of national interest but also watched globally’ (Office of the Registrar General of India, 2009, p. 52). One senior researcher in the study likewise began explaining the importance of the study in terms of the overall global scale of mortality and the number of undiagnosed deaths in all low- and middle-income countries (see also Jha, 2014; Gomes et al., 2017). Still others argued that the study was relevant for public health elsewhere, an argument brought both in terms of the applicability of the methods and in terms of the diversity of India’s population and mortality profile, supposedly reflecting large parts of the global South. Nevertheless, it is an open question how easily experiences and approaches from India can be transferred elsewhere. Interview respondents described the SRS as a unique infrastructure that only exists in this form in India and Bangladesh. Paradoxically, the reference to ignorance about causes of death as legitimation for this study is based on the particular institutional deficiencies in data collection that affect mortality estimates in India. Yet, in taking on the SRS as an alternative, the MDS’ approach does not consider that the institutional preconditions for representative population-wide data collection may be different elsewhere.

Ignorance as legitimation for the MDS reflects both a critique of Indian state institutions that aligns with prevalent approaches in contemporary global health, while the study’s set-up is simultaneously dependent on a state-based infrastructure unique to India. As such, the study’s legitimation decontextualizes its local situatedness to position the MDS in a global health context.

Ignorance as instrument between expertise and bias

According to MDS researchers, efforts to produce more reliable mortality statistics for India involve several methodological innovations in data generation and processing. These involve the use of the demographic monitoring infrastructure SRS, collection of symptomatic data by way of verbal autopsy and the coding of diagnostic information by a panel of physicians. Interview respondents did not explicitly refer to these methodological novelties in terms of ignorance. However, the lack of medical knowledge on the part of data collectors is deliberately mobilized for the study. Similarly, potential ignorance or bias towards certain diagnoses on the part of coders is addressed through further technological intervention. The study’s edifice of verbal autopsy and subsequent diagnosis was piloted and adjusted in the late 1990s, before the MDS formally started collecting and aggregating mortality data from 2001 onwards. Verbal autopsies rely on so-called enumerators, who are employees of the Registrar General’s Office that visit households in SRS units (e.g. city blocks or villages) twice a year. During those visits, they collect data on births, deaths and migrations, as well as symptomatic data on any deaths for the MDS. This symptomatic data is collected on a form that includes a questionnaire and space for a ‘diagnostic narrative.’ Aim of the verbal autopsy interview, and the narrative in particular, is to accurately describe ‘the train of events and circumstances at the onset and during the cause of illness leading to death’ (SRS Collaborators of the RGI-CGHR, 2011, p. 2).

The narratives should record ‘signs and symptoms’ (Interview 2, study coordinator) of medical conditions preceding death as accurately as possible. These form the most important data for the study. It is therefore critical that narrative descriptions of symptoms are accurate, specific and significant. Working groups that included representatives of various research institutions have therefore developed sets of diagnostic criteria that can realistically be identified by people without medical training (i.e. the relatives being interviewed) and can be described verbally. These working groups produced a list of so-called ‘cardinal symptoms’ (including, for example, fever, weight loss, urinary problems or seizures). SRS enumerators receive training to inform them about the procedure they should follow to write a diagnostically useful narrative. This procedure involves five steps that urges enumerators to listen carefully and identify the presence, sequence, and duration of cardinal symptoms. This specific procedure for verbal autopsy interviews does not only institutionalize mechanisms for dealing with the lack of medical training among respondents. It simultaneously mobilizes the ignorance of enumerators as an instrument for accurate data collection. As a senior researcher explained:

the staff is consciously not medically trained, since they are comparable to medical students who are taught to listen and record without drawing conclusions […] They ask and describe what the symptoms are and try to capture the information, getting accurate information without a bias toward a particular diagnosis. (Interview 1a, epidemiologist and study initiator)

Although other quality control measures, such as standardized re-sampling of 5% of all records, are in place, a central mechanism for securing data quality thus relies on reducing enumerators to ‘human sensors’ (Aarden, 2021) that are able to engage in conversation on the one hand, but can reduce such conversations to (largely) decontextualized bits of data on the other.

Enumerators are expected to produce accurate symptomatic data that is not influenced by diagnostic presumptions. The subsequent diagnostic procedure for turning this data into mortality statistics starts with navigating between the expertise and potential bias of physicians who ‘code’ the data. It is considered important that physicians do not jump to conclusions on the basis of biases regarding causes of death they consider likely, or interesting, but that they carefully consider the data first. The study coordinator at the time therefore described the relation between enumeration and diagnosis in terms of how ‘these signs and symptoms [described in the narratives], doctors use to assign cause of death’ (Interview 2, study coordinator). For the purpose of diagnosis, doctors who are part of the study’s diagnostic panel are instructed to simultaneously use their medical expertise, while remaining agnostic regarding causes of death in order to base their assessment solely on the available data. Physicians receive a range of diagnostic instructions for coding individual ‘records.’ They are, for example, urged to ‘[u]se common sense and best clinical judgement’ (SRS Collaborators of the RGI-CGHR, 2011, p. 30), keeping in mind that the study is looking for the most common causes of death. They are encouraged to assign a single cause of death, distinguishing between main causes, terminal events and comorbidities. These instructions serve to stimulate physicians to postpone judgement, in a way simulating ignorance about potential causes of death until they can confidently confirm them with the available symptomatic data.

Similar to the quality control measures for data collection, there are mechanisms in place to secure the quality of diagnoses. These involve the use of multiple opinions and of a software platform that is intended to assist diagnostic decision-making. Each ‘record’ is ‘coded’ (diagnosed) by two physicians who are randomly selected from a panel of around 300 Indian doctors. The diagnostic procedure consists of up to three steps. In the first step, the two physicians code independently. If they arrive at different diagnoses, a second step, called ‘reconciliation’ takes place. Here, the physicians receive each other’s diagnosis and are asked to reconsider their initial diagnosis. If they continue to disagree, a final ‘adjudication’ step takes place, in which a more senior physician decides. In addition to these different steps, physicians are also supported by a software platform that suggests or rejects certain diagnoses. The software’s suggestions are based on all records coded to date, which a software developer referred to as ‘machine learning’ (Interview 8, software developer). While the vision is that the software can ultimately replace coding by human physicians, one researcher indicated that ‘the current conclusion is: none of these [algorithms] are – not yet – well enough developed to replace the physician’ (Interview 7, study researcher; see also Desai et al., 2014; Leitao et al., 2014). For the time being, the MDS therefore continues to rely on human coders to diagnose individual deaths. At the same time, it prescribes specific instructions and procedures that seek to make physicians’ medical knowledge operational for the study.

The individual diagnoses based on the verbal autopsy records are subsequently aggregated into mortality statistics for India as a whole, as well as for particular regions, gender and age groups. These statistics are first published by the Registrar General’s office and subsequently processed further in disease- or medical domain-specific working groups that author scientific papers. The MDS uses the authoritative International Classification of Diseases (ICD-10) as its diagnostic standard. As we will see, there are some remaining uncertainties and frictions between this classification system and diagnostics based on verbal autopsy, for which the MDS has developed its own solutions. Beyond these issues, there are more fundamental questions regarding the particular historical background, narrow descriptions of disease and scientific and political priorities reflected in the ICD-10, which may affect its value for describing the burden of illness in India in context (Bowker and Star, 1999). Yet, the particular ways enumerators’ and physicians’ ignorance is mobilized and simulated in the MDS show how the generation of diagnoses in terms of ICD-10 codes is institutionalized in the first place. What is remarkable in this context is how its data generation and processing procedures navigate ignorance, knowledge and bias in order to arrive at diagnostic results. Paradoxically, not knowing is sometimes required in order to let the dead speak in accurate epidemiological terms.

Ignorance serves as an instrument for knowledge production in the MDS, since it aims to avoid premature and potentially biased determinations of cause of death during both data collection and the diagnostic process. Responses to the enumerators’ questions in household interviews thereby get redefined as symptomatic data and this data in turn needs to be captured in a single diagnosis. Collectively, these moves decontextualize descriptions of mortality as (scientific) data.

Ignorance as a relative property between accuracy and intervention

The MDS was set up to provide better evidence of the causal distribution of mortality in India, positioning ‘widespread ignorance’ as the issue to be resolved through the use of the verbal autopsy method. Researchers are quick to point out that the aim of generating better evidence was not merely academic. Rather, one of the study initiators emphasized, ‘we produced the data for advocacy purposes […] changing the mindset of the administrators’ (Interview 3, demographer and study initiator) – or in other words, to improve health policy-making in India. Researchers consequently claim to value what we may call the ‘actionability’ of the study results over their accuracy, suggesting that any remaining ignorance regarding causes of death is only of relative importance vis-à-vis the alternative of ill-informed health policy. In particular, researchers relativized the shortcomings of the sample and data collection method and of a lack in diagnostic precision. A few respondents pointed out that the SRS sample is not based on disease incidence, which means it may not be able to generate statistically reliable mortality estimates for certain smaller regions or rarer causes of death (see also Bhutta, 2006). Others saw limitations in data collection, where ‘there are some doubts about the narrative details sometimes’ (Interview 7, study researcher). Another interview respondent likewise lamented the shortcomings of relatives’ recollections, suggesting that ‘if we could do some biological markers, you could find out what is really happening’ (Interview 6, former study coordinator/government official).

Yet, uncertainties about the study’s results and remaining ignorance compared to an elusive diagnostic ‘gold standard’ are considered acceptable in light of the particular challenges the study responds to. Verbal autopsy is considered the best option in the context of a persistent lack of knowledge regarding mortality in low- and middle-income countries. Comparison to the clinically confirmed hospital statistics that are available are considered problematic, since ‘[p]eople who go to hospitals and die are very different from those who stay home and die […]. If you base the national estimates on hospital statistics, you get really misleading numbers.’ (Interview 1b, epidemiologist and study initiator). Instead, mortality statistics produced by the MDS also include what one researcher described as ‘non-users’ (interview 5, study researcher) of medical services. MDS researchers therefore feel confident in assuming that their statistics are more accurate than those generated by government health programs (but see Butler (2010) for a critique).

Researchers do not only evaluate persisting unknowns and uncertainties in relation to other methods for assessing mortality, but also question which degree of accuracy is required for the study’s purpose of providing evidence for health policy-making. One researcher indicated how ‘a lot of the time, you don’t have to get the exact medical diagnosis’ (interview 4, former study coordinator). They went on to claim that even though it is often ‘not possible to distinguish the final subcategories [of an ICD-code] by [verbal autopsy], this stratification is not required from the population point of view.’ In many cases where detailed differentiation on the basis of verbal autopsy is not possible, the study therefore treats certain codes as equivalent. One former study coordinator explained:

[codes] J44 and J45, between asthma and COPD, even though they are separate elements, if I correlate, say, 44, and [my colleague] has 45, the system takes it as equivalent cause. Etiologically they may be different, but as far as the burden of disease is concerned it makes hardly any difference. (Interview 4, former study coordinator)

The software that supports diagnostic work comes into the picture here again, registering both codes, without requesting reconciliation. In later (scientific) analysis, a choice between the codes may be made on the basis of additional criteria, but a lesser degree of certainty and accuracy is accepted at this stage of analysis. Arriving at a diagnosis one can work with is considered most important.

This emphasis on ‘actionability’ at the expense of an exact diagnosis reflects the importance attached to the study’s ‘policy impact.’ A health program manager who was formerly involved with the MDS, for example, maintained not to find the question of accuracy particularly interesting – and explained:

Why are we getting into this philosophy of whether the numbers are right? For a program implementer, it is more important – do the same study after a few years and see what happens […] You use a methodology, ten years down the line you use the same methodology and tell us the results […]. That is where the importance of MDS comes. (Interview 6, former study coordinator and government official)

The health program manager considers the possibility of evaluating interventions and informing policy to be the true value of the study. In their eyes, as well as in those of others, a degree of relative ignorance can be accepted in light of the purpose to inform health policy in India more accurately than had hitherto been the case.

The relativization of accuracy as a factor in health policy-making stands in a somewhat paradoxical relation to the terms in which study staff narrated the MDS’ impact on health policy. They decisively positioned the quantitative descriptions of health problems that the study generates as the basis for more tailored public health policy. The study coordinator at the time of my interviews, for example, illustrated their understanding of the evidence-policy nexus on the basis of findings regarding snakebites. The reasoning goes as follows:

One of the significant studies has been about snakebites. We found that 50.000 people are dying each year. The government statistics is 900. Snakebites are happening at home, so nobody knows. So now the government has changed their policies. (Interview 2, study coordinator)

Another researcher in the study took this observation a step further. They suggested that, since the study found that death from HIV/AIDS is less common than previously thought, and from snakebites more so, government resources could be shifted from HIV/AIDS programs to the procurement of anti-venom. The point is not to ridicule this somewhat reductive sequence of cause and effect, or study staff’s naivety regarding health policy making. Rather, I consider the logic of this narrative illustrative of the status of quantitative evidence in the study, in spite of recognition of the evidence’s limitations.

The paradoxical relation between accuracy and actionability in part reflects a long-standing area of concern in global health. This relates to the limits of disease-specific (so-called ‘vertical’) programs and the difficulty of addressing issues at a more systematic (or ‘horizontal’) level. MDS researchers recognize how disease-specific data do not tell the whole story. When explaining the merits of the study, a senior researcher indicated that verbal autopsy can help identify

[t]he history of what happened to the person. You can identify risk factors, you can identify the train of events they had, including obvious cases of medical neglect or malpractice in India, which are common. Or just poor public services. (Interview 1b, epidemiologist and study initiator)

This description lists several structural factors contributing to disease and death. Still, cause-specific data and interventions remain central to how health policy improvement is imagined in the MDS. Study director Prabhat Jha makes this explicit in a book he co-authored to propose improvements to health service delivery in India (Jha and Laxminarayan, 2009). The proposal refers to many structural problems in terms of India’s burden of disease, lack of public funding and deficient governance of health service. Yet it resurrects evidence-based health policy as the solution. It focuses on interventions that are ‘clearly designed to set priorities for tackling specific health problems rather than improving a health system more broadly’ (Jha and Laxminarayan, 2009, p. 77). In spite of widespread critique of ‘vertical’ approaches in social science perspectives on global health, such an approach is not necessarily without merit in light of the impervious structural problems of public health in India. Nevertheless, it once again illustrates how the paradox between evidence and intervention remains unresolved.

Ignorance as a relative property illustrates the difficult relation between generating more accurate statistics and making these statistics actionable in the MDS. Although accurate quantitative insight into causes of death is not always possible, researchers claim this is not a major issue since accuracy is not always necessary either. Difficulties in knowledge production are thereby detached from difficulties in policy making and implementation, thereby decontextualizing interventions from their potentially unruly context.

Conclusion

In the preceding analysis, we have seen how ignorance is critical to shaping the particular configuration of the MDS as it is. Although the Million Death Study’s proclaimed aim has been to reduce ignorance about causes of death in India, institutionalized ignorance manifests as both a problem and a resource. On the one hand, the study aims to reduce ignorance. More precisely, its aim is to generate more accurate and representative statistics of the major causes of mortality in India. These statistics are currently not available due to alleged shortcomings in data collection within the medical system. On the other hand, ignorance also provides an opportunity. The ignorance of people collecting symptomatic data, which is actively cultivated in the study, is seen as an advantage for collecting precise descriptions of symptoms that are not influenced by premature efforts to determine causes of death. In between its status as a problem and as a chance, ignorance is institutionalized in various forms in the MDS. It plays an important role as motivation for the study, which was set up to develop an alternative method for collecting mortality data and, thus, for reducing ignorance about mortality, both in India and in the global South more generally. Furthermore, ignorance is an important element in the study’s methodology. Both the collection of symptomatic data and diagnosis on the basis of this data rely on enumerators and coding physicians not presuming to know a cause of death prematurely. Since the generation of more reliable statistics is a critical aim of the study, the reliability of data and its processing is treated accordingly. Finally, ignorance is also institutionalized in relation to the study outcomes. MDS initiators and researchers consider a degree of remaining ignorance or uncertainty acceptable, since diagnostic accuracy is considered secondary to actionability. Along the entire pathway of the MDS, different forms of ignorance are thus reflected in its institutional architecture and operations.

In exploring these various manifestations of institutionalized ignorance, several paradoxes of the MDS as an evidence-based global health project become visible. One paradox concerns the study’s aim to both produce representative national mortality statistics and serve as a (methodological) exemplar for global efforts to identify causes of death. Thus, particular problems and possibilities in India, particularly the SRS demographic sample, were foregrounded, while the study was simultaneously presented as transferrable elsewhere. A second paradox entails the balancing of knowledge and expertise of study collaborators, with their potential biases. Enumerators’ ability to carefully record family members’ observations and physicians’ medical knowledge thus were valued as possible contributions to generating mortality statistics. At the same time, the study had several mechanisms and instructions in place to avoid diagnostic biases in the generation and interpretation of data. Finally, there was a paradox between aiming for accurate statistics and accepting limits in order to obtain numbers that were good enough for improving health policy. The idea that accuracy is secondary to actionability shows how, at least in the MDS, scientific aspirations ultimately give way to the practical use of findings. Collectively, these paradoxes show how a focus on institutionalized ignorance can make visible how the oft-critiqued mechanisms of reductionism and asymmetries in global health work. The different articulations of ignorance are not straightforward, but navigate a contested landscape within which the question what global health is, which role numbers (can) play, and how the field should be institutionalized are at stake (Lakoff, 2010; McInnes et al., 2012).

Social science analyses of (institutionalized) ignorance can show how certain forms of not knowing are constitutive of quantitative, evidence-based global health. As the preceding analysis has shown, the MDS, for example, is built on multiple articulations of ignorance as more than just the absence of knowledge (Gross, 2007; McGoey, 2012). Treating ignorance as analytically symmetrical to knowledge demonstrates how both are constitutive of particular political framings of health and disease (Paul and Haddad, 2019). Yet, we may extend the analytical valence of this classic STS notion of symmetry by drawing on Spivak’s concept of ‘sanctioned ignorance’ (Spivak, 1988b). She argues that knowledge and ignorance are also mutually constitutive, with particular forms of knowledge and evidence depending on ignorance and exclusion of others. By applying these ideas to the MDS, we could see how institutionalized ignorance contributed to the decontextualization of the study’s local situatedness, of the disease data being produced and the health policy interventions pursued. These dimensions of the decontextualization of disease experience in quantitative approaches to global health have been widely critiqued in the social sciences (Biehl and Petryna, 2013; Adams, 2016), yet without considering the ‘productive’ potential of ignorance (McGoey, 2012).

While Spivak’s reflections pertain to what counts as evidence in historiography, they hold important lessons for tracing (epistemic) asymmetries in evidence-based global health. The case of the MDS showed how the production of mortality numbers requires active engagement with, and ‘sanctioning’ of, different forms of ignorance – as legitimation, as instrument, and as relative in relation to the study’s overall objectives. This approach opens up a novel pathway for the critical study of metrics and quantification in global health, asking how the institutionalization of certain forms of ignorance enables the creation of particular kinds of knowledge. Furthermore, it invites analysts to interrogate the limitations and marginalisations embedded in such sanctioned ignorance. What Spivak (1988a) then calls the ‘successful failure’ of not knowing certain things in order to generate credible evidence can further be instructive for STS analyses beyond global health, to help interrogate the unequally configured worlds knowledge and ignorance jointly create.

Acknowledgements

I would like to thank all interview respondents for providing their time and insight, and Ashawari Chaudhury, Poonam Pandej and Ambuj Sagar for their fieldwork support. Journal reviewers and a list of colleagues too long to mention have helped me clarify and improve the paper’s argument. Should any lack of clarity or other inadequacies remain, only I am to blame.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

Research for this paper was funded by a European Commission Marie Curie fellowship (grant number PIOF-GA-2010-272996), hosted by the Harvard Program on Science, Technology and Society and Maastricht University.

Notes on contributors

Erik Aarden

Erik Aarden is assistant professor at the Department of Science, Technology & Society Studies of the University of Klagenfurt, Austria. He has done research on the intersections between medical research and innovation, governance and social values in Asia, Europe, and the United States, focusing on genetic diagnostics and biobanking.