Optimising the ingredients for evaluation of the effects of intervention

Abstract

Background: In Howard, Best, and Nickels (2015, Optimising the design of intervention studies: Critiques and ways forward. Aphasiology, 2015.), we presented a set of ideas relevant to the design of single-case studies for evaluation of the effects of intervention. These were based on our experience with intervention research and methodology, and a set of simulations. Our discussion and conclusions were not intended as guidelines (of which there are several in the field) but rather had the aim of stimulating debate and optimising designs in the future. Our paper achieved the first aim—it received a set of varied commentaries, not all of which felt we were optimising designs, and which raised further points for debate.

Aims: This paper responds to the commentaries and in the context of recent guidelines for evaluation of the design of single-case studies. We aim to further the discussion our target article has started and extend the scope of the discussion more broadly to issues that were not raised in our target article (e.g., replication).

Main Contributions and Conclusions: It is clear that there is a strong consensus that adequately designed single-case studies of intervention are an appropriate and important tool in our quest for effective interventions with people with cognitive disorders. It is also the case that many agree that there is no single design that is appropriate for every intervention, every participant or every question. However, whichever design is used it must be able to discriminate between the true effect of an intervention on behaviour, and other potential reasons for change (e.g., practice effects, spontaneous recovery, Hawthorne effects, and placebo effects). We have suggested that, depending on the conditions and question to be addressed, this can be achieved using a combination of design features. These may include: multiple pre-treatment baselines, treated and untreated (or subsequently treated) items/processes/tasks, control tasks (not predicted to be affected by treatment even when generalisation is expected), and a cross-over phase (replication across items/tasks). In addition, the outcome of treatment should be evaluated statistically.

We note that generalisation, which is clinically desirable, can lead to particular difficulties in attributing change to intervention unless appropriate controls have been included, and that when items are selected on the basis of poor pre-treatment performance, apparent treatment-related gains may in fact be due to regression to the mean and discuss the implications of this for future research.

Keywords:

• intervention
• effectiveness
• single-case experimental design
• treatment effectiveness
• therapy study design

This article refers to:

Establishing the effects of treatment for aphasia using single-subject-controlled experimental designs

Primary and secondary analyses of single-subject data have complementary value—commentary in response to “Optimising the design of intervention studies: critiques and ways forward”

Facing the challenges of single-case experimental methodology

Preserving the flexibility of single-subject experimental design—a commentary on “Optimising the design of intervention studies: critiques and ways forward”

Mythology and shape shifters in clinical research: false taxonomies and erroneous conclusions—commentary on “Optimizing the design of intervention studies: critiques and ways forward”

Throwing the baby out with the bathwater: pitfalls of misrepresenting single-case experimental designs

Steam, broil, or bake: good recipes for language treatment studies

The case for single-case studies in treatment research—comments on Howard, Best and Nickels “Optimising the design of intervention studies: critiques and ways forward”

The curse of serial dependency in single-case data—commentary on Howard, Best, and Nickels’ “Optimising the design of intervention studies: critiques and ways forward”

How much better? The challenge of interpreting interactions in intervention studies

Optimising the design of intervention studies: should we test whether people require treatment or which intervention has the best outcome?

Notes

1. Figure 1 in Howard et al. only provides the mean of three pre-therapy baselines and two post-therapy baselines. This was done in order to focus attention on the problem of regression to the mean. If a figure like this were to be presented in a report of a single-case treatment study, it would be essential that raw data for every testing point was also provided (most probably in a table).

2. We believe that pre-post designs represent a minimal ABA design: performance is assessed once during a no treatment phase (A), then treatment is applied (B) but performance is not probed, followed by assessment once at the beginning of the second no treatment phase (A). There could be an argument, however, that they represent an AB design: The post-test could be the last (and only) point in the treatment phase (see e.g., Laganaro, this volume). However, the terminology itself is not of importance, what is vital is that there is an understanding of the strengths and weaknesses of a particular design.

3. For the sake of illustration we have not used statistical analysis here, however, we are committed to the view that these conclusions could only be drawn if (1) the treated items showed significant improvement and (2) the difference in any change was significantly greater for the treated than the untreated items.

4. However, as noted by McDonald (this volume), not all those who in general use Approach A design principles use the criterion of stability of baseline (e.g., Kiran & Johnson, 2008).

5. Thompson (this volume) suggests that in Approach B stable pre-treatment is desired. This is not the case. While a stable pre-treatment performance provides the most straightforward pattern to evaluate change as a result of treatment, the use of statistics comparing rate of change across treated and untreated phases (WEST-ROC) means that the effectiveness of treatment can be evaluated even when there is an upward trend over baseline.

6. Of course, in some situations this may not be a question of interest: What may be important is that the treatment as a whole has been effective, and whether that is the naming component or the other aspects of the intervention may be unimportant.

7. By item-level dependency we mean that for any item the probability correct on trial n depends on the performance for that item on trial n-1.

Additional information

Funding

During the preparation of this paper Lyndsey Nickels was funded by an Australian Research Council Future Fellowship (FT120100102). Wendy Best was in receipt of a grant from the Economic and Social Research Council (RES-062-23-2721) and a visitor’s travel grant from the Australian Research Council Centre for of Excellence in Cognition and Its Disorders, Macquarie University.