Evidential Value of Intervention Studies in Two School Psychology Journals: A p-Curve Analysis of Research from 2011 to 2021

Abstract

This study uses p-curve analysis to examine the evidential value of intervention studies published in two major journals in the field of school psychology. We undertook this study in the context of the growing concerns about publication bias, replication crisis, and questionable research practices. We conducted our primary analysis on p values from 37 intervention studies published from 2011 to 2021 in the journals of two national school psychology professional organizations. The analysis shows that, overall, this set of studies does contain evidential value. However, while the subset of academic and mental health and well-being intervention studies contains strong evidential value, the subset of social-emotional and behavioral interventions does not and the results of those studies are unlikely to be reproduced if replicated. The present study demonstrates how p-curve analysis can be used to assess the evidential value of intervention studies in school psychology and to identify areas of research that may be more subject to questionable research practices and publication bias. Embracing open science and open science practice are the keys to enhancing the science foundation for intervention practices and promoting evidence-based practices to improve student outcomes.

Impact Statement

This study uses p-curve analyses to examine the evidential value of 37 intervention study results published in two major school psychology journals. The findings provide researchers, journals, training programs, and practitioners with critical information about how to determine if intervention studies contain evidential value, and if different types of interventions have different levels of evidential power.

Keywords:

• Intervention study
• questionable research practice
• p-curve analysis
• evidential value

Associate Editor:

• Ryan L. Farmer

Additional information

Notes on contributors

Hong Ni

Hong Ni earned her Ed.S. in school psychology and Ph.D. in developmental psychology from the University of Nebraska, Lincoln. She worked as a school psychologist in Iowa for four years and then joined the faculty in the Department of Psychology at California State University, Fresno. Her research interests include resilience promotion in schools, cultural adaptation of multi-tiered systems of support, and systematic review research regarding culturally diverse students and evidence-based intervention.

Paul C. Price

Paul C. Price earned his Ph.D. in cognitive psychology from the University of Michigan and has been on the psychology faculty at California State University, Fresno for 25 years. He teaches courses in research methods, statistics, and judgment and decision making, and is the director of the Judgment and Reasoning Lab at Fresno State. His research interests include quantitative thinking, risk perception and communication, college teaching intervention effectiveness, and open science practices.

Constance J. Jones

Constance Jones earned her Ph.D. in human development and family studies from Pennsylvania State University. She worked as a postdoctoral researcher at the Institute of Human Development, University of California, Berkeley, then joined the faculty in the Department of Psychology at California State University, Fresno. Her research interests include change in personality and psychological health across the lifespan, statistical methods best able to capture individual differences in developmental change, and transformational teaching practices.