Quantifying differences between conditions in single-case designs: Possible analysis and meta-analysis

ABSTRACT

The current paper is a call for and illustration of a way of closing the gap between basic research and professional practice in the field of neurorehabilitation. Methodologically, single-case experimental designs and the guidelines created regarding their conduct are highlighted. Statistically, we review two data analytical options, namely (a) indices quantifying the difference between pairs of conditions in the same metric as the target behavior and (b) a formal statistical procedure offering a standardized overall quantification. The paper provides guidance in the analysis and suggests free software in order to illustrate, in the context of data from behavioral interventions with children with developmental disorders, that informative analyses are feasible. We also show how the results of individual studies can be made eligible for meta-analyses, which are useful for establishing the evidence basis of interventions. Nevertheless, we also point at decisions that need to be made during the process of data analysis.

KEYWORDS:

• Effect size
• meta-analysis
• single-case designs
• standardized mean difference

Declaration of interest

The authors report no conflicts of interest.The authors alone are responsible for the content and writing of the article.

Notes

¹ We use the term “quasi-statistical” here given that it is employed in the methodological quality scale developed by Tate et al (2013). This term refers to procedures that offer quantitative summaries of the data (i.e., a statistical description), but lack the possibility to obtain inferential results (e.g., confidence intervals) on the basis of standard errors that quantify the uncertainty in the point estimate. Thus, such quasi-statistical procedures do not meet the requirement of reporting confidence intervals about the effect size measures (Wilkinson & The Task Force on Statistical Inference, 1999), although the correctness of the standard errors and confidence intervals of inferential statistical techniques is subjected to the completion of their underlying assumptions.

² Bulté and Onghena (2012) for a discussion of visual aids such as lines presenting phase means or medians, trend lines fitted to each phase, range lines representing the amount of data variability and also for software tools.

³ Note that neither the MPD nor the SLC, in this application, take into account the fact that the six AB comparisons belong to three (rather than six) participants. Such nesting is taken into account by the d-statistic and would also be taken into account by a multilevel model. The same results for MPD and SLC we obtained would have been obtained via the following steps: (1) obtain a weighted average per participant, with the weight being the number of measurements per AB comparison; (2) obtain a weighted average per study out of the effects per participant, with the weight being the number of measurements per participant; (3) obtain the weighted average across studies, using series length and the inverse of the coefficient of variation, as explained.

⁴ A preliminary study that we carried out showed that another possible weight giving greater importance to , actually lead to too large differences between the weights assigned to different AB comparisons.