Standards of evidence for institutional review board decision-making

ABSTRACT

A standard of evidence is a rule or norm pertaining to the type or amount of evidence that is required to prove or support a conclusion. Standards of evidence play an important role in institutional review board (IRB) decision-making, but they are not mentioned in the federal research regulations. In this article, I examine IRB standards of evidence from a normative, epistemological perspective and argue that IRBs should rely on empirical evidence for making decisions, but that other sources of evidence, such as intuition, emotion, and rational reflection, can also play an important role in decision-making, because IRB decisions involve an ethical component which is not reducible to science. I also argue that an IRB should approve a study only if it has clear and convincing evidence that the study meets all the approval criteria and other relevant, ethical considerations; and that for studies which expose healthy volunteers to significant risks, an IRB should require that evidence be more than clear and convincing as a condition for approval. Additional empirical research is needed on how IRBs use evidence to make decisions and how standards of evidence influence IRB decision-making at the individual and group level.

KEYWORDS:

• Institutional review boards
• decision-making
• standards of evidence
• empirical evidence
• intuition

Acknowledgments

This research was supported by the Intramural Program of the National Institute of Environmental Health Sciences (NIEHS), National Institutes of Health (NIH). It does not represent the views of the NIEHS, NIH, or US government. I am grateful to Emily Anderson, Ramin Karbasi, Elise Smith and David Wendler for helpful discussions. On 9 November 2020, I presented a version of this paper to the Consortium to Advance Effective Research Ethics Oversight. I am also grateful to members of this audience for helpful comments and discussions.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1. Other countries have institutional structures which are similar to IRBs but go by different names, such as research ethics boards (REBs) or research ethics committees (RECs).

2. In this article, I will cite regulations found in the Common Rule (45 CFR 46), which is accepted by 17 federal agencies. Regulations pertaining to IRBs adopted by the FDA and EPA are virtually identical to the Common Rule with respect to the IRB’s structure, function, and operations.

3. Over half of the nations in Africa have national IRBs or REBs (Klitzman 2012a).

4. An advantage of local IRB review is the local IRBs are likely to have better knowledge of the qualifications of investigators, the institutional research environment, and local values and cultural practices than national IRBs (Moon 2009).

5. As far as I know, the federal regulations are not unique in this regard. That is, I know of no major ethical regulation of guideline that discusses standards of evidence for approving human subjects research.

6. It is worth noting that the federal policy on research misconduct includes a standard of evidence. Research misconduct must be proven by preponderance of evidence. The federal policy applies to research funded by federal agencies or conducted to support application to the FDA (Office of Science and Technology Policy 2000).

7. In theory, oversight agencies, such as the Office of Human Research Protections (OHRP) and the FDA, could provide standards of evidence for IRBs by issuing interpretative guidance for applying approval criteria to proposed research. However, these agencies have not done so.

8. The federal regulations allow IRBs to make some decisions on an expedited basis. New studies can be reviewed on an expedited basis if they determined to be no more than minimal risk. Expedited reviews are conducted by the IRB chair or a designated IRB member (Department of Health and Human Services 2017).

9. The federal regulations state that some types of research are exempt from IRB review. Some common exemptions include research involving surveys, interviews, or focus groups, and research on de-identified data or samples (Department of Health and Human Services 2017).

10. The Office of Human Research Protections (OHRP), which oversees research funded by the Department of Health and Human Services, enforces the federal regulations by issuing determination letters to institutions for noncompliance. Institutions must take appropriate steps to comply with the regulations or OHRP may temporarily halt their funding. The FDA oversees research conducted on FDA-regulated products and enforces its regulations by issuing determination letters to IRBs. The FDA can enforce its regulations by withdrawing approval of IRBs that have registered with the agency. The FDA can also enforce its regulations by informing the sponsor that its research cannot be used in FDA decision-making because it is non-compliant (Resnik 2018).

11. In the study by Green et al. (2006) discussed above, it is likely that IRB that did not approve the health services research protocol was being overprotective.

12. By empirical evidence, I mean evidence from observation, testing, or experimentation (Goldman 1986; Chisholm 1989).

13. An intuition is a judgment or belief that is formed without conscious awareness of any reasoning process at work (Resnik 2017).

14. I use the term “empirically-based” rather than the more popular term “evidence-based” because evidence could, in theory, come from non-empirical sources, such as intuition, logical proofs, or rational arguments. Evidence in mathematics, for example, comes from logical proofs and mathematical intuitions (Resnik 2000).

15. In October 2020, the UK approved a COVID-19 vaccine human challenge trial to be conducted by Open Orphan, a commercial research organization. The UK government will also fund the study. The study is scheduled to begin in January 2021 (Callaway 2020).

16. I discuss some of this literature below.

17. For more on the relationship between descriptive and normative statements in ethics and epistemology, see Kon (2009), Goldman (1986, 2002).

18. By “normative” I also mean “evaluative” or “prescriptive.” A normative claim says what ought to be the case or what we ought to do.

19. By “descriptive” I also mean “factual,” “predictive,” or “explanatory.”

20. Schneider (2015) applies legal concepts to IRB decision-making and argues that IRBs do not adhere to important legal requirements, such as due process and accountability.

21. An IRB must apply the same regulatory criteria to approve new studies, to reapprove (or renew) previously approved studies, and to approve proposed changes (or amendments) to studies (Department of Health and Human Services 2017).

22. I am paraphrasing the regulations. For the exact wording, see Department of Health and Human Services (2017). I have omitted approval criteria for limited IRB review or broad consent.

23. Whether an IRB should exceed its regulatory mandate is ethically controversial. Some commentators argue that to avoid “mission creep” IRBs should stay within their regulatory mandate and make decisions based only on the approval criteria (Gunsalus et al. 2006; Schrag 2010). Others argue that to protect human subjects (and others) from research risks, IRBs sometimes need to make decisions based on rules or guidelines not explicitly stated in the federal regulations (Emanuel, Wendler, and Grady 2000; Resnik 2018).

24. The Belmont Report’s principles are respect for persons, beneficence, and justice (National Commission for the Protection of Human Subjects in Biomedical and Behavioral Research 1979).

25. Many academic institutions have developed protocol templates. See Council for International Organizations of Medical Sciences (2016) for a detailed list of information to include in a protocol.

26. The federal research regulations include special protections for pregnant women and fetuses in research but do not classify pregnant women as vulnerable subjects (Department of Health and Human Services 2017).

27. Many academic institutions have consent templates. See, for example, Northwestern University Institutional Review Board (2020).

28. Interesting epistemological and ethical questions arise here concerning the IRB’s reliance on expert testimony, but I will not address them in this paper. See Selinger and Crease (2006) for further discussion of expertise.

29. Anderson and DuBois (2012) argue that IRB decision-making is like a black box because for many years it has been shrouded in secrecy since IRB meetings are confidential and many IRBs are reluctant to allow outside researchers to study their deliberations.

30. Pritchard has been a senior advisor to OHRP since 2004.

31. We could also say that we are justified in believing the conclusion, the evidence (Goldman 1986). A conclusion could be a statement or belief that supports a decision. For example, if an IRB concludes that a study meets all the approval criteria it could decide to approve a study.

32. Deductive arguments are common in the formal sciences, such as mathematics, logic, statistics, and decision theory, but not very common in the natural sciences, medicine, and engineering, because these disciplines obtain knowledge from observation and experiment rather than logical argumentation and proof (Resnik 2000). Since deductive arguments probably play only a minimal role in IRB decision-making, I will focus on inductive arguments in this paper.

33. We could construct a valid, deductive argument using the axioms of probability theory to estimate the probability of flipping a fair coin and getting heads ten times in a row. We assume that the probability of flipping the coin and getting heads is 0.5, and that each coin flip is independent of the other flips. We also know, from probability theory, that the probability of two independent events is the product of their individual probabilities (Weiss 2011). Given these premises, we can conclude, deductively, that the probability of flipping a fair coin and getting heads ten times in a row is: (0.5)(0.5)(0.5)(0.5)(0.5)(0.5)(0.5)(0.5)(0.5)(0.5) = 0.00098.

34. In US criminal cases, the prosecution has the burden of proof. In civil cases, the person bringing the matter before the court (i.e., the plaintiff) has the burden of proof (Loevinger 1992).

35. While there is little dispute that science is an empirical enterprise, non-empirical methods, such as mathematical modeling, logical and mathematical proof, conceptual arguments, and thought experiments have played and continue to play an important role in scientific theorizing and discovery (Kuhn 1977; Haack 2003). For example, in 1846 Urbain Le Verrier (1811–77) used information about the perturbations in the orbit of Uranus and mathematical equations from celestial mechanics to predict the position a new planet, later named Neptune. Johann Galle (1812–1910) pointed his telescope at the coordinates provided by Le Verrier and discovered the planet .

36. These same criticisms would apply to using p-values for statistical significance.

37. The formula is: p (H/E) = [p (E/H) x p (H)]/p (E).

38. I realize I am making assumptions here about the relationship between science and moral values which are beyond the scope of this paper. Some philosophers, known as ethical naturalists, hold that there are moral facts which can be reduced to facts about the natural world and that we can obtain knowledge about moral facts in the same way that we acquire knowledge about scientific or natural facts (see, for example, Brink 1989; Jackson 1998; Foot 2001). Other philosophers, known as ethical non-naturalists, hold that moral facts cannot be reduced to scientific facts because there are no moral facts; or, if there are moral facts, they are facts about properties or phenomena that supervene on or transcend the natural world. We cannot, therefore, obtain moral knowledge (if there is such as thing) in the same way that we would obtain scientific knowledge. Non-naturalists hold that morality is based on intuition (Moore 1903; Audi 2005), emotion (Gibbard 1990; Greene 2013), culture (Harman 1996), religion (Adams 1987), or rational reflection (Kant [1785] 1981). I would describe my approach as a limited form of non-naturalism that recognizes that some of our moral values, including human life, freedom from pain, health, have a solid basis in nature but that others, such as justice and human rights, do not. For further discussion of naturalism and non-naturalism, see Lenman (2018) and Ridge (2019).

39. Wendler et al. (2005) attempt to quantify the concept of minimal risk for pediatric research in terms of the average daily life risks faced by US children. This attempt to quantify risk, even if successful, would not prove that risk/benefit determinations are quantifiable because these determinations involve judging the risks are reasonable (or morally acceptable) in relation to the benefits, and saying that a risk is reasonable is a value judgment (Resnik 2017).

40. See note 40.

41. A legal analogy may be useful here: all members of a jury, regardless of their background or education, are obligated to use the same standard of evidence.

42. Human subjects often still fail to understand that they may not benefit from participation, however (Appelbaum et al. 1987; Joffe et al. 2001).

43. One might also ask whether the IRB should require less evidence for some types of research, such as research that imposes minimal risks on human subjects. While it makes some sense to say that evidence for the reasonableness of risks not need to meet the clear and convincing standard, I would argue that evidence pertaining to other criteria, such as consent or equitable subject selection, would still need to meet the clear and convincing evidence standard, because these criteria involve issues related to human rights.

44. The research may offer these subjects other types of benefits, such as immunity to COVID-19, priority access to COVID-19 therapeutics if they become ill, or the satisfaction that they are contributing to a worthy cause, but these benefits are usually not considered to be direct benefits of the interventions under investigation (King 2000; Friedman, Robbins, and Wendler 2012). Moreover, these other benefits do not outweigh the risks to subjects, so the primary justification for this research is that it is likely to offer significant benefits to society (Shah et al. 2020).

45. I am tempted to suggest that evidence should meet the beyond reasonable doubt standard, but that would probably be too restrictive.

Additional information

Funding

This work was supported by the National Institute of Environmental Health Sciences [ZIAES102646-10].