Money Well Spent? The Cost-Effectiveness of a Texting Intervention Targeting College Persistence

Abstract

Past studies of texting interventions designed to promote college student persistence have given little attention to assessing both the fixed and variable costs of such initiatives. In addition, they typically have not assessed costs relative to the desired outcome(s). In this case study, we assessed the costs of implementing a text messaging program relative to persistence outcomes using a randomized field experiment in one college. The intervention’s cost per student was $18.76 and the cost per text was $0.85 per student. Mean cost-effectiveness ratios per student retained ranged from $1,312 to $750. Sensitivity analyses revealed increases in the number of students texted or gains in academic advisors’ efficiencies both contributed to sizable declines in the cost-effectiveness ratios. We conclude the texting intervention was cost-effective relative to other published interventions that have targeted college persistence.

Keywords:

• Cost effectiveness
• college persistence
• texting intervention
• randomized field experiment
• academic advising

Acknowledgments

The research reported here was undertaken with the support of an intramural Degree Completion Challenge Grant at the University of Utah.

Declaration of Interest Statement

No potential conflict of interest was reported by the author(s).

Notes

1 For relatively recent reviews of the effectiveness of a range of educational nudges, including those involving texting programs, see Damgaard and Nielsen (2018) and Bird et al. (2021).

2 It is unclear as to whether the cost per student calculation was based on only the high-risk sample size or the sample size for the full intervention.

3 The college does not have the ability to send mass communication messages to students who have yet to declare a major or to students who are majoring in one of the many other colleges at the university.

4 Approximately 40 percent of the majors in the college transfer from a nearby two-year institution or other four-year institutions in the state. Thus, it was not surprising that approximately one-third of all majors had completed 100+ credit hours.

5 Ideally, in addition to identifying the costs, one would identify the benefits that accrue to the university and/or the college for each student that persists. But, with tuition discounting it is virtually impossible to come up with a firm estimate of the tuition benefit from the university’s accounting perspective. At the college level, the enrollment benefits are assessed through a complex formula where the variables include the number of majors, the number of graduating seniors, and the number of courses taken within the college (by all undergraduates, not just majors) averaged over two years. Thus, evaluating the benefits for either the university or the college is beyond the scope of this project.

6 This second measure is especially relevant to the college as a portion of the college’s budget is linked to number of undergraduates majoring in the college. Also, it should be noted that over the course of the semester, there were many students who declared new majors in the college. However, because our criteria for inclusion in the study required that a student be a declared major at the beginning of the semester, these students did not receive text messages and therefore are not included in our calculations.

7 It should be noted that these are the predicted probabilities for students in the treatment and control groups evaluated at the means for the covariates. The nonlinear nature of logistic regression leads to different probabilities for specific sub-groups. For example, focusing on transfer students whose GPAs are not in the top quartile, the predicted probabilities of persisting in the college are .9123 and .8856 for the treatment and control groups, respectively.

8 Over the past five years, undergraduate major enrollment in the college has fluctuated from a low of 3,254 to a high of 4,441.

9 Analysis of advising appointment data revealed that during spring 2020 there were no statistically significant differences in advising appointments between students in the treatment and control groups. Specifically, 52.03% of students in the treatment group made an advising appointment and 51.43% of the students in the control group made an advising appointment during spring 2020.

10 While number of students was fixed for the intervention semester, the total number typically varies across time and thus its implications for advising time should be treated as a variable cost.

11 Advisors plan one hour for each appointment with 15 minutes devoted to reviewing the student’s file prior to the meeting, 30 minutes allocated to the meeting, and finally, another 15 minutes spent on entering notes into the student’s file after the meeting. The average hourly wage for academic advisors in the center is approximately $22. Benefits are 36% of wages, making the total hourly cost $29.92.

12 It is important to note forecasting of advisors’ time costs is also dependent on advisors’ salaries which vary considerably across institutions. The typical advisor in our college has an annual salary of approximately $45,000. Data from various employment websites suggest our salaries are near the national average, however, the Bureau of Labor Statistics reports a median annual salary across all years of experience of $50,050, with the lowest 10 percent earning less than $34,000 and the highest 10% earing almost $95,000 (Bruens, 44 2020).

13 We can get a very rough idea of the pandemic’s impact by comparing spring 2019 and spring 2020 persistence rates. Specifically, using the university’s accounting perspective and focusing on only those majors in the college with fewer than 100 credit hours, 96.6 percent persisted to the end of the spring 2019 semester. When the college’s accounting perspective is used, the persistence rate is 94.1 percent. In spring 2020, the overall persistence rates were 95.7 percent and 91.6 percent, respectively. These figures suggest that the pandemic may have had a modest, negative impact on persistence.