A Multiple-Design, Experimental Strategy: Academic Probation Warning Letter's Impact on Student Achievement

Abstract

We aimed to compare the findings of three research designs to bracket effect estimates of a strongly worded warning letter delivered by certified mail to students on academic probation.

We embedded an experiment within a regression discontinuity design and calculated two achievement estimates, average GPA and percentage of students remaining on probation. Study participants attended a large Midwestern college. Cohen's d experimental effect size was .45. Regression discontinuity design results were validated by our experimental evidence, and outcome measures were generally statistically significant. We provided additional supportive evidence using comparative RD control group design logic. Regression point displacement design results were successfully replicated using a within-study comparison inside the experiment. In the context of probation, a diverse design, replicative approach provided considerable promise for more precise estimation of intervention effectiveness. We found no deleterious impact on reenrollment and concluded that the certified letter represents an inexpensive probation policy.

Keywords:

• Academic probation
• college student achievement
• comparative regression discontinuity
• randomized experiment
• regression discontinuity design
• regression point displacement design
• within-study comparison

Acknowledgments

We would like to express gratitude to Ben Kelcey and Amy Krentzman for valuable feedback on previous drafts of this manuscript.

Notes

1 We included details for each analytic model in the Appendix. The coefficient associated with β₂ provided an estimate of the treatment effect for each respective design.

2 As noted by Steiner and Wong (2016), when design comparisons contain the same intervention or counterfactual group, the statistical contrasts are not independent and should be regarded with some caution.