![Sample our Humanities journals, sign in here to start your FREE access for 14 days]({"type":"image" , "src" : "/sda/5939/TOC-Subject-Sample-2021-Humanities.jpg"})
ABSTRACT
In the biomedical, behavioural and social sciences, the leading method used to estimate causal effects is commonly randomised controlled trials (RCTs) that are generally viewed as both the source and justification of the most valid evidence. In studying the foundation and theory behind RCTs, the existing literature analyses important single issues and biases in isolation that influence causal outcomes in trials (such as randomisation, statistical probabilities and placebos). The common account of biased causal inference is described in a general way in terms of probabilistic imbalances between trial groups. This paper expands the common account of causal bias by distinguishing between the range of biases arising between trial groups but also within one of the groups or across the entire sample during trial design, implementation and analysis. This is done by providing concrete examples from highly influential RCT studies. In going beyond the existing RCT literature, the paper provides a broader, practice-based account of causal bias that specifies the between-group, within-group and across-group biases that affect the estimated causal results of trials – impacting both the effect size and statistical significance. Within this expanded framework, we can better identify the range of different types of biases we face in practice and address the central question about the overall validity of the RCT method and its causal claims. A study can face several smaller biases (related simultaneously to a smaller sample, smaller estimated effect, greater unblinding etc.) that generally add up to greater aggregate bias. Though difficult to measure precisely, it is important to assess and provide information in studies on how much different sources of bias, combined, can explain the estimated causal effect. The RCT method is thereby often the best we have to inform our policy decisions – and the evidence is strengthened when combined with multiple studies and other methods. Yet there is room for continually improving trials and identifying ways to reduce biases they face and to increase their overall validity. Implications are discussed.
Acknowledgements
I am grateful for comments from Nancy Cartwright, John Worrall, Federica Russo, Carl Hoefer, Bennett Holman, Stephan Guettinger, Federica Malfatti, Saana Jukola, Corinna Peters, Camille Lassale, Johannes Findl, Adrià Segarra, and anonymous journal reviewers. I received funding from the Marie Curie programme of the European Commission and the Beatriu de Pinós programme of the Government of Catalonia.
Notes
1 For example, Andrew et al. Citation1994; Sackett et al. Citation1996; Djulbegovic et al. Citation2013.
2 For example, Seligman Citation1996; Duflo, Glennerster, and Kremer Citation2007; Banerjee Citation2007.
3 Holman (Citation2017) also acknowledges some of the limitations of theoretical and idealised approaches to studying RCTs.
4 The examples provided throughout this paper are all found within the top ten cited RCT studies worldwide in any scientific journal. Each of these ten trials has been cited by at least 6,500 or more articles as of 2016 based on the Scopus database. They include world-leading trials on the topics of breast cancer (Slamon et al. Citation2001), colorectal cancer (Hurwitz et al. Citation2004), stroke (Marler Citation1995), postmenopause (Rossouw et al. Citation2002), insulin therapy (Van den Berghe et al. Citation2001), two separate trials on cholesterol (Shepherd et al. Citation1995; SSSSG Citation1994) and three separate trials on diabetes (Turner Citation1998; DCC Citation1993; Knowler et al. Citation2002). The issues discussed here, while these RCTs are a sample of highly influential trials and fall within medicine, biology and neurology, generally apply to any RCT across the medical, behavioural and social sciences and beyond; though there are differences in the design features of trials between fields like medicine, psychology and economics.
5 The RCT method yields what is often called an interventionist or manipulationist form of causation (Woodward Citation2003; see also Russo and Williamson Citation2011).
6 For a discussion on the meaning of causal claims in biomedical contexts, see Russo and Williamson (Citation2011).
7 Cartwright refers here to balance ‘in both wings’, though balance is needed between all wings as trials in practice at times employ multiple treatment groups (to test different treatments or dosages against each other) and multiple control groups (to test the treatment against both a placebo and the common treatment at present, for example).
8 The distinction of internal-external validity can also at times be problematic for another reason. There are many biases that often go beyond both, including industry sponsorship bias, reporting bias, reference bias (citing only studies favouring a given outcome), publication bias etc.
9 The majority of RCTs in medicine do not thus have what Clarke et al. (Citation2014, 343) call the ‘realistic chance of stumbling across coincidental correlations’.
10 In addition, computerised randomisation algorithms make the assumption that numbers can actually be selected entirely at random – but such algorithms, as Fallis (Citation2000) argues, in fact tend to use a deterministic sequence in which initial values shape later values.
11 Researchers conducting trials in one field are not always aware of different design features in other fields. For example, in trials within economics all participants are commonly randomized in the entire sample before a trial begins, while participants are generally randomized on a roll-in basis in medical trials which can lead to greater imbalances in background traits of participants.
12 See Teira (Citation2010) for a discussion on frequentist versus Bayesian clinical trials.
13 For a discussion on meta-analyses, see Moher et al. (Citation1998), Stegenga (Citation2011) and Holman (Citation2018).
14 Most of the ten most cited trials were furthermore published after the Consolidated Standards of Reporting Trials guidelines were adopted (Andrew et al. Citation1994) – though these guidelines need to be extended to include the broader range of issues and constraints facing trials (cf. Moher et al. Citation2010; Rennie Citation2001).
15 Ankeny Citation2014.