Sorry everyone, but it didn't work (p = 0.06)

Abstract

The more or less ubiquitous use of Fisher-type statistics in quantitative/numeric evaluations of drug and alcohol education initiatives takes place within the context of a literature of long standing which suggests there are areas in which null hypothesis testing and the use of conventional cut-off points (i.e. the 1% and 5% probability levels) are inappropriate. This literature is largely ignored. The paper identifies some of these issues in terms of their relevance to the problems of evaluation of drug/alcohol programmes. The paper argues that the qualitative approach does not solve these problems but merely by-passes them, as well as being unsatisfactory in a number of ways. Thus, it is concluded that there is a pressing need for change in the type of quantitative approach adopted. Suggestions are made for a variety of exploratory methods, still involving broadly numerical analysis, which have a philosophical rather than a merely technical base, and which shed light on ‘what is going on’ rather than merely providing a binary decision (it worked/it did not work) derived from an arbitrary criterion for statistical significance and a null hypothesis which is usually known to be false from the start.

• Statistics
• programme evaluation
• methodology