Reducing Threats to Validity by Design in a Nonrandomized Experiment of a School-Wide Prevention Model

Abstract

This paper reviews literature on the rationale, challenges, and recommendations for choosing a nonequivalent comparison (NEC) group design when evaluating intervention effects. After reviewing frequently addressed threats to validity, the paper describes recommendations for strengthening the research design and how the recommendations were implemented in a current study. Next, tests were performed in order to investigate whether the added design components had contributed to increased internal validity by creating comparable cluster groups at baseline. In order to estimate the external validity or the potential generalizability of results, the statistical power of the study was calculated up front and representativeness of the participating schools in relation to other schools in Norway was examined. Baseline comparisons indicated that the addition of several design elements, such as the random invitation of multiple, stratified, and active comparison groups, may have reduced significant validity threats stemming from selection bias. The need for more accurate and reliable effect estimates in school-based evaluation research is discussed.

Keywords::

• Nonrandomized design
• validity
• school intervention
• effectiveness
• problem behavior

Acknowledgements

The APA ethical guidelines were followed. Likewise, the standards of the Regional Committee for Medical Research Ethics, South Norway, were followed throughout the conduct of the study. The study was approved by the Norwegian Social Science Data Services (NSD).

Additional information

Funding

This research was supported by the Norwegian Center for Child Behavioral Research and in part by a grant from the Norwegian Directorate for Education and Training.

Notes on contributors

Mari-Anne Sørlie

Mari-Anne Sørlie, Cand.Paed.Spec., has background in special education and is a senior researcher at the Norwegian Center for Child Behavioral Development in Oslo, Norway. Her primary research interests are in evaluation of school-based interventions, and in the development of behavior problems and social competence in children and youths. She is the author of several research articles and has contributed to pedagogical books and reports, written in both Norwegian and English.

Terje Ogden

Terje Ogden, PhD, holds the position of research director at the Norwegian Center for Child Behavioral Development, Unirand, and is also a professor at the Institute of Psychology, University of Oslo, Norway. Research interests include clinical trials and implementation of empirically supported interventions (ESI) targeting antisocial children and youth, and the longitudinal development of social competence and externalizing problem behavior in early childhood. Ogden has published several articles on interventions like Multisystemic Therapy, Parent Management Training—The Oregon Model, Positive Behavior and Learning Support in schools (N-PALS), and Early Interventions for Children at Risk (the TIBIR project). Publications also include articles on moderators and mediators of interventions, and the large-scale implementation of ESIs. Ogden has recently authored the book Adolescent Mental Health, Interventions and Prevention together with K. Amlund-Hagen (Routledge, 2013). See also www.ogden.no.