Walking the talk: the need for a trial registry for development interventions

Abstract

Recent advances in the use of randomised control trials to evaluate the effect of development interventions promise to enhance our knowledge of what works and why. A core argument supporting randomised studies is the claim that they have high internal validity. The authors argue that this claim is weak as long as a trial registry of development interventions is not in place. Without a trial registry, the possibilities for data mining, created by analyses of multiple outcomes and subgroups, undermine internal validity. Drawing on experience from evidence-based medicine and recent examples from microfinance, they argue that a trial registry would also enhance external validity and foster innovative research.

Keywords:

• impact assessment
• randomised control trials
• trial registry

Acknowledgements

The authors thank two anonymous referees for helpful comments and suggestions.

Notes

1. Requiring authors reporting on RCTs to follow an augmented CONSORT would also lead to a much needed improvement in reporting on other issues; for example, the method used for the actual randomisation (Bruhn and McKenzie 2009).

2. In this study smoking is a confounder only in the sense that smoking might have been correlated with asbestos levels by chance and we do not know this. It is difficult to imagine, as Freedman (1999) seems to suggest, that smoking causes higher levels of asbestos in drinking water as this comes from the level of natural occurring rock, serpentine, in the water reservoirs. Also, it is unlikely that areas with high asbestos in the drinking water would attract more smokers, as the level of asbestos is not known.

3. With dependence between the variables, the picture becomes more complicated (and depends upon assumptions about the involved test statistics). Still, α_overall > α, unless the m variables are all perfectly correlated. As an example, assume that m = 2 and that the two test-statistics follow a bivariate standard normal distribution with a correlation coefficient of 0.5, then α_overall = 0.09; and with a correlation coefficient of 0.9, α_overall still exceeds 0.07.

4. To some extent, this is of course always the case. Any randomised experiment in social science must at some point rely on the subjective judgement of ‘experts’ with local and/or context specific information (Cartwright 2007).

5. In principle, the registry could be open to any question involving statistical analysis, whether or not the question involves a survey. But since there is no way of assuring whether or not a researcher started the analysis prior to registration, this type of registration would not be credible.

6. ClinicalTrials.gov terms this interventional and observational trials.

7. These registries include The Cochrane Central Register of Controlled Trials (CENTRAL) and the Campbell Collaboration Social, Psychological, Educational and Criminological Trials Register (C2-SPECTR).

8. When it comes to results, a trial registry will not necessarily make data collection easier as many existing trial registries do not contain results; for example, ClinicalTrials.gov, which is the largest existing registry.

9. The study by Coleman (1999) is a case in point. Using a pipeline method approach for identification, Coleman finds no effect of microcredit in Northern Thailand. He then notes that this negative finding may be due to an abundance of credit in the region, for which reason the results may have little external validity.

10. This list was inspired by the World Health Organisation Trial Registration Data Set version 1.2.1.