Abstract
School-based curricula have become a mainstay of drug prevention policy in the United States and are increasing in popularity in other parts of the world. The promotion and dissemination of these interventions has been driven in large part by the creation of lists of programmes which, it is claimed, are grounded in scientific evidence demonstrating their effectiveness. Recently concerns have been raised about the data analysis and presentation practices used in evaluations of a number of programmes that appear on these lists. Here we examine a series of papers from an evaluation of an intervention that combined the Strengthening Families Program 10 – 14 and Life Skills Training Program, each of which is among the most widely advocated universal drug prevention programmes. The data analysis and presentation practices employed in the evaluation of this combined programme include one-tailed significance testing, alpha levels of 0.10, changes in outcome variables across publications and use of the post-test data as the baseline when assessing change over time. Taken together, these practices severely limit the claims that can be made about the results presented in the evaluation. Specifically, we believe that far from supporting the evaluators' claims concerning the rigour of the findings and their generalisability and public health significance, the results are very fragile, of little practical significance and quite possibly analysis-dependent. [Gorman DM, Conde E, Huber JC Jr. The creation of evidence in ‘evidence-based’ drug prevention: a critique of the Strengthening Families Program Plus Life Skills Training evaluation. Drug Alcohol Rev 2007;26:585 – 593]
Notes
There is a slight discrepancy between the data reported in the 2006 publication Citation[25], p. 879] cited here and the 2005 publication Citation[24], p. 374]. The latter states that 228 families (not 226) were actively recruited for the SFP10−14 + LST group, and that 129 (not 130) attended at least one session.
of the methamphetamine paper Citation[25], p. 879] shows that 190 subjects in the SFP10−14 + LST group and 196 subjects in the control group were followed up at 11th grade. The figures for the 12th grade follow-up were 191 and 197, respectively. However, the sample sizes are slightly smaller in the results section, due presumably to missing data (e.g. 189 for the SFP10−14 + LST group and 196 for the control group for past year use at 12th grade) Citation[25], p. 880]. The latter were used in this re-analysis.
Spoth et al. cite four of their earlier papers in support of their contention that differential attrition can be ruled out Citation[25], p. 880]. However, only one of these actually pertains to this study (the others are from a study of the SFP and the Preparing for the Drug Free Years Program). The attrition data they present in the one relevant paper are from the 8th and 9th grade follow-ups Citation[23]. This is of limited use in estimating the effects of attrition at 11th and 12th grades as it is entirely possible that there was greater differential attrition later in the study than there was early on.
of Spoth's methamphetamine paper Citation[25], p. 879] shows that 226 families were recruited in the SFP10−14 + LST (arm 1, box 3 of the figure). The next box in the figure states that 130 of these agreed to participate and 89 declined. We have assumed that the latter is a misprint and should instead read 86, as 130 and 89 do not sum to 226. All the other boxes in arm 1 of the figure sum to 226.