Shining the Spotlight on Those outside Florida’s Reform Limelight: The Impact of Developmental Education Reform for Nonexempt Students

ABSTRACT

Since the 2000s, states have experimented with reforms to improve success among underprepared students traditionally assigned to developmental education (DE). Florida’s reform under Senate Bill 1720 has been among the most comprehensive and wide-reaching. Recent public high school graduates and military personnel became exempt from DE, but nearly one-third of students, including those without a Florida standard high school diploma, were still required to take a placement test and enroll in DE if they scored below college-ready. The legislation also required colleges to offer accelerated instructional strategies for students remaining in DE, and provide enhanced advising and support services. Focusing specifically on nonexempt students, we use statewide data to conduct a difference-in-regression discontinuity analysis to examine differences in first-year math coursetaking outcomes for students on the margins of college readiness before and after the reform. While students narrowly assigned to DE tend to have a lower likelihood of taking and passing college-level courses relative to their college-ready peers, these students experienced larger gains after the reform when DE courses were offered in accelerated formats accompanied by support services. The reform also improved outcomes for students scoring above college-ready, which suggests that nonexempt students benefited from enhanced advising and support services too.

KEYWORDS:

• Developmental education
• state policy
• student success

Students at community colleges often face a complex web of challenges involving inadequate academic preparation, competing family and work obligations, financial barriers, and difficulties navigating complex college systems and requirements (Bettinger et al., 2013). These challenges may be further exacerbated by difficulties in the classroom, thus increasing the risk of early departure prior to credential completion. Developmental education courses are widely used by community colleges to help support underprepared students from a wide variety of backgrounds, including recent high school graduates with inadequate academic preparation as well as returning adults who have been away from the classroom for many years. Yet developmental education has been criticized for failing to sufficiently develop skills among underprepared students and discouraging persistence among students potentially misassigned to these courses who likely would have succeeded in college-level courses (Scott-Clayton & Rodriguez, 2015).

In traditional developmental education programs, incoming college students take a placement test and these scores are used to place students into one or more levels of developmental education in reading, writing, and/or mathematics. In the mid- to late 2000s, the first wave of developmental education reforms began which included isolated practices such as providing additional tutoring or supplemental instruction, and implementing student success courses (Jaggars & Bickerstaff, 2018). These early reforms were intended to improve students’ progression to and success in introductory college-level courses, and focused on short-term interventions which tended to be small in scope. The second wave of developmental education reforms began around 2010 and consisted of more wide-scale adoption of practices including more comprehensive placement strategies, the use of accelerated options such as compressed courses to allow for faster completion of developmental education requirements, and changes to the pedagogy and content of developmental education courses. Many of these second-wave reforms have demonstrated positive impacts on short-term outcomes like taking and passing college-level courses, but there are still challenges to improving longer-term outcomes like graduation rates for students who enter college underprepared (Jaggars & Bickerstaff, 2018).

In 2014, Florida implemented one of the most comprehensive developmental education reform efforts under Senate Bill (SB) 1720. This reform required all 28 institutions in the Florida College System (FCS) to make three significant changes simultaneously: (a) remove requirements for placement testing and developmental education courses for the exempt students, (b) offer different instructional strategies for students remaining in developmental education, and (c) provide additional advising and academic support services to all incoming students, regardless of their initial course placement. The first component of the reform that made developmental educational optional for the majority of students was the most controversial, as critics feared that it would harm those students at the lowest levels of preparation (e.g., Flannery, 2014; Tierney & Duncheon, 2013). Yet Florida’s reform did not merely eliminate developmental education, rather it gave students the option of enrolling directly into a college-level course if they believed they were ready, or the option of enrolling in a developmental education course using new instructional modalities if they thought that they needed additional preparation. These developmental courses were designed in a way to accelerate students’ progression to college-level coursework among both exempt students opting into developmental education and nonexempt students scoring below college-ready who were still required to take these courses. Colleges were required to offer options for modularized, compressed, co-requisite, or contextualized developmental courses that could often be completed more quickly than traditional semester-longer courses. The additional advising services were intended to help students make good decisions about which courses to take, while the enhanced academic support services were designed to provide students with extra help so that they could succeed in their courses.

However, not all students could take advantage of the option of enrolling directly into college-level courses. Students who entered a Florida public school in 2003/04 or later and completed a standard high school diploma, as well as activity duty military personnel, became exempt from developmental education. In contrast, students who had not earned Florida standard high school diplomas in this timeframe were still required to take a placement test and enroll in one to two developmental courses in math, reading and/or writing, depending on their test scores. The years selected for the exemption criteria coincide with increased academic standards in Florida’s public schools following the implementation of Governor Jeb Bush’s “A+ Plan” in 1999. Therefore, the exemption criteria were likely intended to address concerns about students entering college without secondary preparation under the reformed Florida public schools.

Nonexempt students make up nearly one-third of all first-time-in-college (FTIC) students and include students from out-of-state schools, private schools, home schools, older students who attended public schools under prior state standards, and those who did not complete a standard high school diploma. Given the large number of students affected by these nonexemption criteria, it is important to examine how this population may have been affected by the reform. Additionally, the results provide context for understanding whether the benefits of Florida’s reform may be attributed solely to removing mandatory assignment to developmental education for exempt students, or whether there may also be benefits to changing the ways in which developmental education is offered for nonexempt students. Specifically, we address the following research questions:

1. How do first-year coursetaking outcomes differ based on assignment to developmental education for nonexempt students in the pre-policy years (under traditional developmental education) and in the post-policy years (under reformed developmental education)?

2. What is the effect of the developmental education reform on first-year coursetaking outcomes for nonexempt students?

3. Does the effect of the reform differ depending on nonexempt students’ assignment to developmental education?

Prior research indicates the reform has led to an overall increase in student success as measured by completion rates in the first college-level course and college-level credit hours attempted and earned during the first year of enrollment among all first-time-in-college students (Hu et al., 2019; Park-Gaghan et al., 2020). Further, examining the effectiveness of Florida’s developmental education reform among the population of nonexempt students allows us to extend this research in several important ways. First, it allows us to examine whether concerns were valid that nonexempt students not completing standard high school diplomas under more rigorous Florida state standards may not be adequately prepared to enroll directly into college-level coursework. Second, since nonexempt students were still required to take a placement exam, we can use more rigorous methods through a difference-in-regression discontinuity (DiRD) design to compare first-year coursetaking outcomes before and after the reform for students narrowly assigned to developmental education courses relative to those who narrowly avoided placement in these courses. Students scoring just below the college-ready cutoff would be subject to the greatest changes under the reform, as they would switch from placement in a traditional semester-long developmental education course to placement in an accelerated developmental course with additional support services. These new instructional strategies for developmental education went into effect after the reform, so comparing outcomes among students below this cutoff before and after the reform allows us to see if the effectiveness of developmental education changes. We can also examine whether the effects of the reform differed for students above or below the cutoff. The students scoring just above the cutoff would only experience a change in the support services provided after the reform while continuing to enroll in the first college-level math course. The students scoring below the cutoff would experience changes in instructional strategies in developmental education courses and the change in the support services. Using a rigorous empirical approach allows us to begin to tease out the extent to which outcomes may have changed due to enhanced advising and support services alone (based on students above the cutoff), relative to also being assigned to take a developmental education course under new accelerated instructional strategies (based on students below the cutoff). We focus specifically on placement in math, an area of critical importance given that math is the most commonly needed developmental education course and tends to have the lowest success rates, making it a “gatekeeper” to higher education success (e.g., Bryk & Treisman, 2010; Huang, 2018). We also examine whether the effects differ by academic preparation depending on whether students are near the cutoff for assignment to one or two levels of developmental math courses.

The next section describes previous literature examining the impacts of both traditional and reformed developmental education courses on first-year coursetaking outcomes, as well as how the results may differ for various subgroups of students such as lower-performing students and older adults. Next, we describe our analytic approach that employs a difference-in-regression discontinuity analysis to examine differences in first-year coursetaking outcomes for nonexempt students on the margins of college readiness before and after the reform. Our data uses longitudinal student records from the population of nonexempt students at all 28 FCS institutions (the former community colleges). Lastly, we conclude with a discussion of the implications for this unique student population, and important considerations for policymakers in Florida and other states to consider when determining who may benefit from being allowed to opt-out of developmental education.

Literature review

Much of the rigorous research to-date on the impact of developmental education has been conducted using regression discontinuity (RD) designs to compare postsecondary outcomes for students who score just below-college ready and are assigned to developmental education courses relative to students who score just above-college ready and are assigned to college-level courses. Valentine et al. (2017) conducted a meta-analysis of studies using RD to examine the effects of placement into developmental education and found statistically significant negative effects on several postsecondary outcomes, and the effect sizes were often quite large in magnitude. In particular, students assigned to developmental education were approximately 8 percentage points less likely to ever complete a college-level course in the same subject area in which remediation was needed compared to similar peers who were college-ready. However, a major limitation of these RD studies is that the estimates are only for students on the margins of college readiness, and may not be generalizable to students with varying levels of academic preparation.

To further explore this issue of whether the effects of developmental education differ by students’ levels of preparation, several studies have examined the impacts for lower-performing students who are placed into more than one level of developmental education. One study by Dadger (2012) found that students in Virginia community colleges who were assigned to three developmental math courses would have been more likely to complete a certificate or associate’s degree if they had been allowed to take only two developmental math courses. Another study using data from four California community colleges found similar results, as students assigned to an extended two-semester sequence of developmental math were significantly less likely to complete subsequent courses and earned fewer degree-applicable credits relative to students in a single semester-long developmental math course (Ngo & Kosiewicz, 2017). A third study by Melguizo et al. (2016) examined placement in four distinct levels of math in a large urban community college system and found that students assigned to lower level math courses had a lower likelihood of progressing to the next course in the sequence relative to students placed in higher-level courses. Only one study by Boatman and Long (2018) has demonstrated more positive evidence of the impacts of developmental education for lower-performing students assigned to multiple levels of developmental courses. While there were negative, and often large, effects for students assigned to a single level of developmental education, most effects were null for students assigned to multiple levels of developmental math and there were some positive effects for students assigned to multiple levels of reading or writing.

While we are not aware of any studies that have examined differences in the effects of developmental education by the type of high school attended (public, private, homeschool, in-state, or out-of-state), there has been some research on the subgroup of adult undergraduates (typically 25 or older) who have been out of high school for a number of years. Older adults are more likely to be placed into developmental education courses upon college enrollment and tend to be less likely to complete the developmental sequence to which they are assigned relative to traditional college-age students (Bailey et al., 2010). On one hand, they may be at a disadvantage because they are more likely to have additional work and family obligations than their younger peers, are more likely to enroll part-time, and may be more likely to have forgotten math skills learned in high school (Bettinger et al., 2013). Yet, on the other hand, they may have higher levels of motivation because they often return to school due to life transitions, and they may have a more realistic understanding of their skills based on their work experience. Calcagno et al. (2007) examined postsecondary outcomes for students 25 and older in Florida two-year colleges. Compared to their younger counterparts, these students were more likely to have obtained an alternative secondary credential like a GED rather than a standard high school diploma and tended to score lower on the math placement exam upon college entry. The older students were less likely to complete a degree overall, but this trend reversed when taking into account differences in incoming math achievement levels, suggesting that “old dogs can learn new tricks—after they refresh their math skills” (p. 231). Another study by Hawley and Chiang (2017) looked specifically at the effects of developmental education for adult undergraduate students at Ohio public community colleges and found more mixed results. While participation in developmental education was associated with some positive effects on short-term outcomes like persistence, there were statistically significant and negative effects on degree completion for adult students in developmental education.

Developmental education reform and Florida’s SB 1720

Over the past two decades, states and postsecondary education systems have responded to the challenges of low success rates in traditional developmental education programs by experimenting with various types of reforms. Early reform efforts in the 2000s tended to be small in scope and only provided short-term support, while more recent efforts have struggled with challenges such as scaling up initiatives so that they are available to the majority of underprepared students (Jaggars & Bickerstaff, 2018). Florida’s developmental education reform under SB 1720 has been one of the most comprehensive, by requiring all FCS institutions to make three major changes system-wide relating to course placement, advising and support services, and the instructional strategies used in developmental education courses.

The first major change was that all students entering grade 9 after 2003/04 and completing a standard high school diploma, as well as active-duty military personnel, became exempt from placement testing and developmental education courses. These students still had the option to take developmental education courses if they felt that they needed additional support, or they could enroll directly in the first college-level math course (Intermediate Algebra). There is very little evidence available about the outcomes of reforms that simply remove the requirement for developmental education, and the results are mixed. Kosiewicz and Ngo (2019) were able to take advantage of a natural experiment that occurred when a community college mistakenly allowed students to self-place into a developmental or college-level math course. They found that there were some improvements in student outcomes such as the likelihood of completing degree requirements in math, but the self-placement almost exclusively benefitted White, Asian, and male students. The authors suggest that these differences are likely due to women and students from underrepresented racial/ethnic groups being more likely to underestimate their abilities and opt into developmental education, and also a lack of adequate advising to help students make good decisions about placement.

Florida’s reform differed in that it required all colleges to develop a plan to provide enhanced advising and academic support services for all incoming students, regardless of their initial course placement. Institutional leaders reported making advising changes such as adding more online orientation resources, increasing the duration of advising sessions, and spending additional time with at-risk students identified through early alert systems (Hu et al., 2017). To provide additional academic support, colleges made changes such as adopting online tutoring programs, and adding or increasing faculty time in success centers or tutoring. These types of advising and academic support services have been shown to support student success in other contexts (e.g., Bettinger et al., 2013; Hatch & Garcia, 2017; Tampke, 2013).

The final change under Florida’s SB 1720 was that colleges could no longer offer traditional semester-long developmental education courses and were instead required to offer developmental education courses using one or more instructional strategies including modularized, compressed, contextualized, and co-requisite. Modularized courses allow students to complete customized modules on content relating to only those skills that they have not yet mastered. Compressed courses contain the same amount of content as a semester-long developmental education course, but they meet for a greater number of hours per week for fewer total weeks, potentially allowing students to complete two compressed courses in a single semester. Contextualized courses present developmental content in an applied manner related to students’ major course pathways, or meta-majors. Co-requisite courses allow students to take a developmental education course and a gateway course in the same subject area concurrently in the same semester. Prior research on these types of accelerated developmental education strategies has demonstrated positive impacts on short-term outcomes such as gateway course enrollment and completion rates, as well as longer-term outcomes including degree attainment (Denley, 2015; Hodara & Jaggars, 2014; Jaggars et al., 2014; Kalamkarian et al., 2015; Okimoto & Heck, 2015).

There is a growing body of evidence that Florida’s SB 1720 has yielded significant improvements in student success overall. Beginning in the first year of the reform in 2014, fewer students enrolled in developmental education, particularly in math, which declined from enrollment rates of 38% of incoming students in fall 2013 to 22% in fall 2014 (Hu et al., 2016). When examining the overall impact of the reform for both exempt and nonexempt students, the likelihood of enrolling in a college-level math course in the first year increased by 8 percentage points and the likelihood of completing these courses increased by about 4 percentage points (Park-Gaghan et al., 2020). Moreover, the improvements in first-year coursetaking outcomes were greater for Black and Hispanic students relative to White students, thus reducing (and in some cases even eliminating) prior racial/ethnic achievement gaps. The impacts of the reform also differed by students’ high school preparation, with greater gains for students on the lowest two academic tracks (Park-Gaghan et al., 2019). A similar trend was seen in the impact of SB 1720 on credit accumulation, with significant gains in the number of credits attempted and earned in the first year, particularly for Black, Hispanic, low-income, and underprepared student (Mokher et al., in press). Taken together, these findings suggest that it is important to examine the heterogeneous impacts of SB1720 among student subgroups, as the reform seems to have benefitted some students more than others.

The focus of the present study is the subgroup of nonexempt students who were still required to participate in placement testing and enroll in developmental education if they scored below college-ready. These nonexempt students made up about one-third of FTIC students, so a substantial number of students were affected by these nonexemption criteria. Focusing on nonexempt students allows us to look at the performance of students assigned to new accelerated instructional strategies for developmental education relative to those in traditional developmental education courses prior to the reform. We also examine how outcomes for students in these new accelerated strategies compare to similar students who narrowly avoided developmental education and could enroll directly into college-level courses with additional support services.

Data

The data consist of student-level records for all first-time-in-college (FTIC) students at Florida’s 28 state colleges who were nonexempt from developmental education under SB 1720. Our data include college enrollment records and transcripts, as well as demographic characteristics. We also have high school records for students who attended a Florida public high school, but these records are missing for the majority of nonexempt students so we are unable to account for high school variables in our analyses. Table 1 describes the reasons that students were classified as nonexempt and the share of students classified as nonexempt for each reason. The largest group is students who attended a non-Florida public high school (52%), followed by students who attended a Florida public high school but did not complete a standard diploma (26%). Only about 5% of students were nonexempt because they entered grade 9 in a Florida public high school prior to 2003/04. Additionally, about 3% were nonexempt for multiple reasons—such as entering grade 9 prior to 2003/04 and not completing a standard high school diploma. The reasons for exemption are unknown for the remaining 15% of students because their administrative records are incomplete.

Table 1. Number and percent of nonexempt students, by reason for nonexemption

Reason for nonexemption	N	Percent
Non-FL public school	47,344	52.02
FL public school, but no standard diploma	23,220	25.51
FL public school, but in grade 9 prior to 03/04	4,571	5.02
Multiple reasons	2,459	2.70
Unknown	13,423	14.75
Total	89,892	100.00

Our sample includes two cohorts of FTIC students prior to the reform (Fall 2012 and 2013) and two cohorts post-reform (Fall 2014 and 2015). We limit the analytic sample to students who have scores on the college placement test, which is over 70% of nonexempt students in both the pre- and post-policy years. Students may be missing test scores due to incomplete administrative records, or if they have scores on alternate placement tests including the SAT, ACT, or Accuplacer (which are not observed in our data). Students missing PERT scores are less likely to be Black and more likely to be Hispanic (standardized mean differences of 0.28 and 0.27, respectively), and also tend to be about 2.3 years older (standardized mean difference of 0.27). Demographic characteristics for White, other race, and female are similar for students with missing and non-missing PERT scores.

Each cohort has approximately 25,000 nonexempt students. For the post-reform years, the data includes an indicator for whether the student met the exemption criteria under SB 1720. In order to create a comparable variable for the pre-policy years, we classified students as nonexempt if they (a) did not have a transcript record from a Florida public high school, (b) did not complete a standard high school diploma, or (c) completed a standard high school diploma prior to 2006/07. We do not have a valid indicator for active-duty military personnel status so we are unable to include this in the pre-policy exemption variable. However, this should have little impact on the sample given that less than 1% of students enrolled in FCS institutions are classified as active duty (Florida College System, 2018).

The sequence of math courses available to students at FCS institutions consists of lower-level developmental education, upper-level developmental education, the first college-level math, and gateway math. While the specific courses taken may vary somewhat by institutions, we provide a description of a typical sequence based on information provided by Valencia College (n.d.). The lower-level course, Developmental Mathematics I (MAT 0018), is the first course in a college-preparatory two-course sequence emphasizing fundamental operations with applications to beginning algebra. The upper-level course, Developmental Mathematics II (MAT 0028), is the second course in the college-preparatory sequence which provides algebraic background on topics including fundamental operations with polynomials, linear equalities, factoring, and an introduction to radicals. The first college-level math course, Intermediate Algebra (MAT 1033) counts for elective credit only, and develops algebraic skills on topics such as systems of linear equations and inequalities, introduction to functions, complex numbers, and quadric equations.¹ Next in the sequence is the first gateway math course, which fulfills the associate’s degree requirements in math. Students have the option to choose from among several different gateway courses including College Algebra (MAC1105), Elementary Statistics (STA2023), Liberal Arts Math I (MGFX106), and Liberal Arts Math II (MGFX107).

The outcomes of interest for this study include dichotomous indicators for whether students took and passed the first college-level math course (Intermediate Algebra), and a gateway math course. These outcomes are measured by the end of the first year of college enrollment. Control variables for student background characteristics include a series of dichotomous variables for race (White, Black, Hispanic, other), a continuous variable for age (in years), and a dichotomous variable for sex (1 = female, 0 = male). There was no missing data for any of these variables. On average, nonexempt students tend to be similar to exempt students on all demographic characteristics except for age (Table 2). While nonexempt students tend to be about 6 years older than exempt students (25.0 versus 19.3 years), it is important to note that many nonexempt students do not meet the criteria of 25 years or older typically used to define adult undergraduates. The racial/ethnic distribution of nonexempt students is 40.6% White, 20.0% Black, 32.1% Hispanic, and 7.3% other. Approximately 53% of nonexempt students are female.

Table 2. Mean (and standard deviation) of student background characteristics, by exempt status and policy status

	Exempt status				Policy status (exempt only)
	Nonexempt	Exempt	Std. mean difference		Pre-policy	Post-Policy	Std. mean difference
Percent White	40.6	38.8	−0.04		40.9	40.3	−0.01
	(40.9)	(48.7)			(49.2)	(49.1)
Percent Black	20.0	21.3	0.03		21.7	18.0	−0.09
	(40.0)	(40.9)			(41.2)	(38.5)
Percent Hispanic	32.1	33.6	0.03		30.7	33.8	0.07
	(46.7)	(47.3)			(46.1)	(47.3)
Percent other race	7.3	6.00	−0.05		6.7	7.9	0.04
	(26.0)	(23.9)			(25.1)	(27.0)
Percent female	53.0	51.8	−0.02		52.8	53.3	0.01
	(49.9)	(50.0)			(49.9)	(50.0)
Average age	25.0	19.3	−0.91		26.2	23.7	−0.27
	(9.4)	(3.1)			(10.0)	(8.6)

Standardized (Std.) mean differences are calculated as the difference between each group divided by the pooled standard deviation.

We also compared the demographic characteristics of the nonexempt students in the pre-policy years to the post-policy years. The standardized mean differences were less than 0.10 for all demographic characteristics except for age. The pre-policy cohort of exempt students was slightly older, with an average age of 26.2 years compared to 23.7 years in the post-policy cohort.

Methods

We examine the impact of assignment to developmental education coursework on enrollment and passing rates in first-year college courses in math for nonexempt students, and whether this relationship changes with the implementation of Florida’s SB 1720. We use a regression discontinuity (RD) design since assignment to the treatment (in this case, developmental education) is determined by a continuous running variable (the Postsecondary Education Readiness Test, or PERT). Scaled scores on the PERT range from 50 to 150, with students assigned to lower developmental education for scores below 96, upper developmental education for scores between 96 and 112, and Intermediate Algebra for scores of 113 or above.² We model the likelihood of taking and passing various math courses in the first year as a function of whether an individual’s score on the PERT fell above or below the cutoff for lower-level developmental courses in the first set of models, and for the cutoff for upper-level development courses in the second set of models.

Not all students who are assigned to developmental education courses proceed to take these courses. Many students delay enrollment in their first math course and may take no math in the first year. Additionally, some students may have concordance scores on other tests like ACT or SAT, which are not observed in our data. However, Figure 1 shows that there are sharp discontinuities in enrollment in developmental education courses near the placement score cutoffs. At the upper developmental cutoff, less than 2% of students scoring above the cutoff take any developmental math course in the first year compared to nearly 70% of students below the cutoff. At the lower developmental cutoff, less than 2% of students scoring above the cutoff take a lower developmental math course compared to 50% of students below the cutoff. Since this is a sharp RD analysis, we are estimating the intent-to-treat effect based on assignment to developmental education courses. Therefore, this estimate represents the overall impact of the reform for all nonexempt students, regardless of whether or not they complied with an assignment to their treatment condition (placement into developmental education or college-level courses). This estimate may be particularly relevant from a policy perspective, as it represents the impacts of the reform under actual implementation rather than under ideal conditions. The primary advantage of RD is that it provides a rigorous estimate of the program impact, because any differences in student outcomes should be attributable to the treatment if assignment to the treatment is the only thing that changes at the cutoff. However, one limitation of this approach is that the treatment effects are only estimated for students near the cutoff, and may not be generalizable to the full sample of nonexempt students.

Figure 1. Compliance with assignment to treatment based on the relationship between PERT scores (centered around the cutoff for assignment to developmental education) and enrollment in any developmental course (upper-level cutoff) and enrollment in a lower-level developmental course (lower-level cutoff)

Using 2 years of data before and after the policy, we employ a Difference-in-Regression Discontinuity (DiRD) to empirically determine whether the RD estimates change following the policy implementation (Murnane & Willett, 2011). This model allows us to combine pre- and post-policy cohorts to test whether the RD estimates for assignment to developmental education change after the reform. Following the approach of Cattaneo et al. (2017, 2018), we identify optimal bandwidths by minimizing the mean squared error (MSE) of the local polynomial RD point estimator, while allowing the left and right bandwidths to differ. We estimate the following logit model for student i at college j in cohort (year) t:

$Logit(y_{ijt})=\alpha +\beta (DE{\hspace{0.17em}}_{ijt})+\tau \left(POS{T}_t\right)+\varsigma \left(DE{\hspace{0.17em}}_{ijt}\ast POST\right)+\rho \left(PER{T}_{ijt}\right)+\phi \left(S_{jt}\right)+y_j+{\lambda }_t+{\epsilon }_{ijt}$

where y is the outcome of interest (e.g., enrolling in a college-level math course), DE is an indicator of whether the student was assigned to DE based on the placement test score (1 = yes, 0 = no), POST is an indicator for the post-policy cohorts (1 = yes, 0 = no), and DE*POST is an interaction between the two. We also control for the placement test score (PERT) centered at the cut score, measures of student background characteristics (S), year fixed effects (λ_t), and college fixed effects (γ_j) to control for unobserved heterogeneity across institutions, with standard errors clustered at the institutional level. For ease of interpretation, we also present predicted probabilities from the regression models for students above and below the cutoffs in the pre-reform and post-reform periods.

The DiRD estimates for the DE coefficient address our first research question of whether first-year coursetaking outcomes differ based on assignment to developmental education for nonexempt students. Based on numerous prior studies of developmental education, we expect that the likelihood of taking and passing college-level courses in the first year would be lower for students scoring below the college-level cut score who are placed into developmental education, relative to similar students scoring just above this cut score who narrowly avoid developmental education. The estimates for the POST coefficient address our second research question about the effect of the developmental education reform on first-year coursetaking outcomes for nonexempt students. For example, a statistically significant positive coefficient on POST would indicate that students tend to have a greater likelihood of taking or passing college-level courses in the post-reform years relative to the pre-reform years.

The coefficient for DE*POST addresses our third research question about whether the effect of the reform differs depending on nonexempt students’ assignment to developmental education. This is important because it allows us to examine whether the new instructional strategies—coupled with enhanced support services—tend to be more effective than traditional developmental education courses offered in the pre-reform period. While prior studies on Florida’s reform have focused on what happens when developmental education is made optional for the majority of students, this study is the first to examine what happens to the nearly one-third of remaining students who could not opt out. The results provide context for understanding whether the benefits of Florida’s reform may be attributed solely to removing mandatory assignment to developmental education for exempt students, or whether there may also be benefits to changing the ways in which developmental education is offered for nonexempt students.

Another benefit of these analyses is that we can examine whether the effects of assignment to developmental education before and after the reform change differentially depending on students’ level of preparation. For example, we might anticipate that students who are close to college-ready may perform well in an accelerated developmental course accompanied by additional support services. But would students who are far from college-ready also benefit, or would they be better off in a traditional semester-long course? We run separate DiRD models for two different PERT cutoffs, in which each has a different sample. The first cutoff is for students who are assigned to a lower-level developmental education course. Prior to the reform, these students would have been required to complete one lower-level developmental course and one upper-level developmental course before enrolling in the first college-level course. Given that these courses were designed to be taken in a sequence, this means that most students would not be ready to enter a college-level math course until at least their third semester of college. However, after the reform many colleges combined both lower- and upper-level developmental education in a single course with additional credit hours, so that students could potentially fulfill the developmental education requirements within one semester. The second cutoff is for students who are assigned to an upper-level developmental education course. These students would have been assigned to a traditional semester-long developmental education course prior to the reform. After the reform, they would have been assigned to take a developmental course using a new instructional strategy, such as a compressed or co-requisite course, which would allow them to complete the developmental education requirements concurrently with the first college-level math course in the same semester. By including a separate set of models for both the upper and lower developmental cutoffs in our analyses, we are able to examine whether the effects of the reform may differ for students with different levels of academic preparation.

We also conduct several sensitivity tests to assess the robustness of our results. First, we estimate a baseline model with no covariates to determine whether the results differ depending on whether we control for other factors including student demographic characteristics and college fixed effects. Second, we estimate our models using a variety of different functional forms for the relationship between the dependent variable and the running variable, including quadratic, cubic, and quartic specifications. If the underlying relationship is incorrectly specified then the treatment effect may be biased, as the functional form can create an apparent shift that suggests a discontinuity when there is none (Jacob et al., 2012). Third, we assess the sensitivity of the results in the bandwidth specification by using coverage error rate (CER) optimal bandwidths in addition to the MSE optimal bandwidths in our preferred specification (Cattaneo et al., 2017, 2018). Fourth, we estimate models including interaction terms between developmental education assignment and PERT scores (DE*PERT), as well as developmental education assignment and PERT scores and the post-policy indicator (DE*PERT*POST) to determine whether the treatment may impact the slope of the regression line. Overall, we find that our results are highly robust across all of these sensitivity tests, as shown in the Appendix Tables A1-A8.

Validity of RD estimates

We begin by assessing whether our analyses satisfy the four standards for sharp RD studies using the criteria specified by the What Works Clearinghouse (What Works Clearinghouse, 2017). The first standard is to satisfy the integrity of the running variable, which means demonstrating that students’ scores have not been manipulated. The PERT has institutional integrity because it is scored by an external provider (McCann Associates) with no incentive to modify students’ placement status. Additionally, if the student took the PERT more than once, we use the first test score since this cannot be manipulated due to selective retesting. We further test the statistical integrity of the running variable by conducting an RD manipulation test using local polynomial density estimation (Cattaneo et al., 2017). For both cutoffs, we fail to reject the null hypothesis of no difference in the density of the treated and control observations (p = .255 for the upper-cutoff and p = .206 for the lower-cutoff). These findings are further supported by a graphical analysis of the distribution of PERT scores, which shows no strong evidence of any discontinuities in students’ test scores (Figure 2).

Figure 2. Density plot of the running variable, PERT math scores

The second standard is to ensure low levels of attrition, both overall and by treatment status. We have outcome data for all students, so no observations are lost due to attrition. If students drop out of college before taking or completing a college-level math course, then they are assigned a value of 0 for the dichotomous outcomes indicating whether students took and passed college-level math courses.

The third standard is to satisfy the continuity of the relationship between the outcomes and the running variable. The first criterion within this standard is to demonstrate baseline equivalence on key covariates for students above and below the cutoffs. In our sample, all values have a standardized mean difference of less than 0.15 standard deviation units, which is well below the WWC maximum of 0.25 (Table 3). We also control for the full set of covariates in our models since some standardized mean differences were greater than 0.05. The second criterion is to provide graphical evidence that there are no discontinuities in the running variable at any values other than the cutoff. Figure 3 provides scatterplots of the outcomes and running variables, and the only discontinuities are at the scores for assignment to upper- and lower-level developmental courses.

Table 3. Baseline equivalence on student background characteristics for students above and below the cutoffs for assignment to lower-level developmental education and upper-level developmental education

	Below cutoff			Above cutoff
	Mean	Std. Dev.		Mean	Std. Dev.		Std. mean difference
	Assignment to lower-level developmental courses
White	0.46	0.50		0.43	0.49		0.06
Black	0.24	0.43		0.19	0.39		0.13
Hispanic	0.27	0.45		0.28	0.45		−0.03
Other race	0.06	0.23		0.07	0.25		−0.04
Female	0.50	0.50		0.53	0.50		−0.06
Age	23.94	7.71		23.21	7.25		0.10
Cohort	2013.38	1.09		2013.54	1.08		−0.14
	Assignment to upper-level developmental courses
White	0.43	0.49		0.42	0.49		0.01
Black	0.15	0.36		0.14	0.34		0.05
Hispanic	0.34	0.47		0.33	0.47		0.00
Other race	0.09	0.28		0.10	0.30		−0.04
Age	21.25	5.78		21.22	5.91		0.01
Female	0.50	0.50		0.52	0.50		−0.04
Cohort	2013.61	1.11		2013.72	1.10		−0.10

Standardized (Std.) mean differences are calculated as the difference between each group divided by the pooled standard deviation.

Figure 3. Scatterplots of the relationship between the outcome variables and the running variable (PERT math scores). Solid lines indicate the cut scores for assignment to lower- and upper-level developmental education

The fourth standard is to satisfy the functional form and bandwidth selections. The first criterion is that the statistical model must control for the running variable, which is done in all of our models. The second criterion is that a local regression or related nonparametric approach is used with a justified bandwidth. Our regression models are estimated using MSE-optimal bandwidths, and sensitivity tests using CER-optimal bandwidths yield similar results. Given that we have a large sample that pools across multiple cohorts, our bandwidths are relatively small, ranging from 3 to 6 points above or below the cut score (on a scale of 50 to 150) depending on the model. The next criterion is evidence that the findings are robust to varying bandwidths and functional forms. In the appendix (Tables A1-A8), we demonstrate that our results are robust to the choice of bandwidth (MSE or CER), to the choice of functional form (linear, quadratic, cubic, and quartic), and to the inclusion of interaction terms between the treatment variable and the forcing variable. The next criterion is a graphical analysis displaying the relationship between the outcome and the running variable. The graph must demonstrate consistency with the choice of bandwidth and the functional form specification. As shown in Figure 3, the relationship appears to approximate a linear trend, although we do conduct sensitivity tests using other functional forms. The last criterion is that the relationship between the running variable and the outcome must not be constrained to be the same on both sides of the cutoff. Our optimal bandwidths allow for variation on the left and right of the cutoff.

Findings

Impact of assignment to upper-level developmental education

We begin by examining differences in first-year coursetaking outcomes in math among students narrowly assigned into a single developmental education math course (upper DE) relative to those who are narrowly assigned to the first college-level course (Intermediate Algebra). As a reminder, these are local average treatment effects and are not necessarily generalizable to the whole population of nonexempt students. The results for the DiRD models are presented in Table 4, with predicted probabilities and marginal effects in Table 5. The statistically significant negative coefficients on the upper DE variable indicate that students assigned to upper DE were less likely to enroll in and complete both Intermediate Algebra (the first college-level math course) and gateway math courses (the next college-level math course in the sequence) relative to students who were directly placed into Intermediate Algebra. For example, the predicted probability of enrolling in Intermediate Algebra prior to the reform was 60.1% for students assigned to upper DE versus 72.8% for students directly assigned to this course. Enrollment in gateway courses was much lower for students on both sides of the cutoff, with predicted probabilities of 20.6% for students assigned to upper DE versus 31.2% for students assigned to Intermediate Algebra in the pre-reform period. These findings suggest that in response to our first research question about how outcomes differ based on assignment to developmental education, nonexempt students assigned to upper DE tend to perform worse on first-year coursetaking outcomes compared to similar students not assigned to developmental education courses.

Table 4. Impact of assignment to upper DE (relative to college-level) on first-year coursetaking outcomes for nonexempt students

	(1) Enrollment in Intermediate Algebra (or higher)	(2) Completion of Intermediate Algebra (or higher)	(3) Enrollment in gateway math (or higher)	(4) Completion of gateway math (or higher)
Upper DE	−0.572***	−0.487***	−0.558***	−0.579***
	(0.107)	(0.092)	(0.104)	(0.116)
Post	0.383***	0.320**	0.532***	0.515***
	(0.104)	(0.113)	(0.105)	(0.100)
Upper DExPost	−0.103	−0.069	0.407***	0.348***
	(0.093)	(0.086)	(0.084)	(0.084)
Bandwidth	[−6, +5]	[−6, +5]	[−6, +5]	[−6, +5]
N	8,968	8,968	8,968	8,968

Notes. Results are from a Difference-in-Regression Discontinuity model using optimal bandwidths by minimizing the mean squared error (MSE) of the local polynomial RD point estimator. Models include controls for the placement test score centered at the cut score, measures of student background characteristics, year fixed effects, and college fixed effects to control for unobserved heterogeneity across institutions, with standard errors clustered at the institutional level. ***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10.

Table 5. Predicted probabilities for first-year math coursetaking outcomes for students assigned to upper DE relative to college-level math

	Pre-Reform	Post-Reform	Difference
Enrollment in Intermediate Algebra (or higher)
Upper DE	60.1%	66.6%	6.5%***
College-level	72.8%	79.6%	6.9%***
Marginal Effects
Upper DE vs. college-level			−0.4%
Completion of Intermediate Algebra (or higher)
Upper DE	43.6%	49.8%	6.2%***
College-level	55.6%	63.2%	7.6%***
Marginal Effects
Upper DE vs. college-level			−1.4%
Enrollment in gateway math (or higher)
Upper DE	20.6%	39.9%	19.3%***
College-level	31.2%	43.6%	12.4%***
Marginal Effects
Upper DE vs. college-level			6.9%***
Completion of gateway math (or higher)
Upper DE	14.1%	28.0%	13.9%***
College-level	22.6%	32.8%	10.2%***
Marginal Effects
Upper DE vs. college-level			3.7%***

Results are predicted probabilities from a Difference-in-Regression Discontinuity model using optimal bandwidths by minimizing the mean squared error (MSE) of the local polynomial RD point estimator. Models include controls for the placement test score centered at the cut score, measures of student background characteristics, year fixed effects, and college fixed effects to control for unobserved heterogeneity across institutions, with standard errors clustered at the institutional level. ***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10.

In response to our second research question concerning the effects of the reform, we find that the developmental education reform tends to have a positive effect on first-year coursetaking outcomes for nonexempt students. The statistically significant positive coefficients on the POST variable indicate that the reform is associated with improvements on all four outcomes (enrollment in Intermediate Algebra, completion of Intermediate Algebra, enrollment in gateway math, and completion of gateway math) for all students near the margins of college-readiness. In Intermediate Algebra, both enrollment and passing rates increased by about 7 percentage points after the reform for students both above and below the upper DE cutoff.

For research question three about whether there are differential effects of the reform, we find some evidence that the reform had greater effects for students assigned to upper DE. On the outcomes of enrollment and completion of gateway math courses, the statistically significant and positive coefficients on the UpperDE X POST interaction term indicates that students who were assigned to upper DE experienced larger gains under the reform relative to students assigned to Intermediate Algebra. In other words, the new developmental education approach as stipulated in SB 1720 has better results as measured by student first-year course outcomes than the traditional approach before the reform. The likelihood of enrolling in a gateway math course increased by 19.3 percentage points for students assigned to upper DE, relative to an increase of 12.4 percentage points for students assigned to Intermediate Algebra (a difference of 6.9 percentage points). The likelihood of passing a gateway math course increased by 13.9 percentage points for students assigned to upper DE, relative to an increase of 10.2 percentage points for students assigned to Intermediate Algebra (a difference of 3.7 percentage points). This indicates that the gap in performance on gateway outcomes by developmental education placement status narrowed after the reform.

Impact of assignment to lower-level developmental education

The second set of results in Tables 6 and 7 examines the same first-year math coursetaking outcomes for students narrowly assigned to two developmental math courses (lower DE) relative to students narrowly assigned to a single developmental math course (upper DE). In response to the first research question about how outcomes differ based on assignment to developmental education, there is some evidence that first-year coursetaking outcomes are worse for students assigned to lower DE. The statistically significant negative coefficients on the lower DE variable in the first two models indicate that students below the cutoff who were assigned to lower DE were significantly less likely to enroll in or complete Intermediate Algebra (the first college-level math course) relative to students who were assigned to upper DE. Prior to the reform, the predicted probabilities of enrolling in Intermediate Algebra were 15.8% for students assigned to lower DE and 33.3% for students assigned to upper DE, and the predicted probabilities of passing this course were 8.4% for lower DE students and 17.5% for upper DE students. The coefficients on the lower DE variable were not significant in models 3 and 4, indicating there were no statistically significant differences by placement status in the likelihood of taking and passing gateway math courses, with very low predicted probabilities of 3% or less on these outcomes for both groups of students prior to the reform. These low enrollment and passing rates are not surprising, given that gateway math would have been the fourth course in the math sequence under a traditional developmental program (lower DE, upper DE, Intermediate Algebra, and gateway math) so it would have been very difficult for lower-performing students to progress this far in their first year.

Table 6. Impact of assignment to lower DE (relative to upper DE) on first-year coursetaking outcomes for nonexempt students

	Enrollment in Intermediate Algebra (or higher)	Completion of Intermediate Algebra (or higher)	Enrollment in gateway math (or higher)	Completion of gateway math (or higher)
Lower DE	−0.971***	−0.841***	−0.253	−0.375
	(0.149)	(0.146)	(0.213)	(0.293)
Post	0.107	0.433***	0.898***	0.948***
	(0.129)	(0.109)	(0.204)	(0.272)
LowerXPost	0.722***	0.606***	0.200	0.173
	(0.115)	(0.134)	(0.175)	(0.183)
Bandwidth	[−4, +6]	[−4, +6]	[−4, +6]	[−4, +6]
N	13,772	13,772	13,772	13,772

Notes. Results are from a Difference-in-Regression Discontinuity model using optimal bandwidths by minimizing the mean squared error (MSE) of the local polynomial RD point estimator. Models include controls for the placement test score centered at the cut score, measures of student background characteristics, year fixed effects, and college fixed effects to control for unobserved heterogeneity across institutions, with standard errors clustered at the institutional level***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10.

Table 7. Predicted probabilities for first-year math coursetaking outcomes for students assigned to lower DE relative to upper DE math

	Pre-Reform	Post-Reform	Difference
Enrollment in Intermediate Algebra (or higher)
Lower DE	15.8%	30.2%	14.3%
Upper DE	33.2%	35.7%	2.5%
Marginal Effects
Lower DE vs. Upper DE			11.8%***
Completion of Intermediate Algebra (or higher)
Lower DE	8.4%	20.6%	12.2%***
Upper DE	17.5%	24.7%	7.2%***
Marginal Effects
Lower DE vs. Upper DE			5.0%***
Enrollment in gateway math (or higher)
Lower DE	2.7%	7.7%	5.0%***
Upper DE	3.4%	8.0%	4.6%***
Marginal Effects
Lower DE vs. Upper DE			0.4%
Completion of gateway math (or higher)
Lower DE	1.5%	4.5%	3.0%***
Upper DE	2.2%	5.4%	3.3%***
Marginal Effects
Lower DE vs. Upper DE			−0.2%

Results are predicted probabilities from a Difference-in-Regression Discontinuity model using optimal bandwidths by minimizing the mean squared error (MSE) of the local polynomial RD point estimator. Models include controls for the placement test score centered at the cut score, measures of student background characteristics, year fixed effects, and college fixed effects to control for unobserved heterogeneity across institutions, with standard errors clustered at the institutional level. ***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10.

Our second research question examines whether the reform has an effect on first-year coursetaking outcomes for nonexempt students assigned to lower DE, and as with the results for the upper DE cutoff, we find some evidence that the reform did result in gains in student outcomes. The statistically significant positive coefficients on the POST variable in models 2, 3, and 4 indicate the reform was associated with an increase in three of the four coursetaking outcomes (completion of Intermediate Algebra, enrollment in gateway math, and completion of gateway math).

The third research question addresses whether the effects of the reform differ depending on nonexempts students’ assignment to developmental education. The statistically significant and positive coefficients on the LowerDE X POST interaction term in models 1 and 2 indicate that the effects of the reform on the Intermediate Algebra outcomes were greater for students’ assigned to lower DE. The predicted probability of enrolling in Intermediate Algebra increased after the reform by 14.3 percentage points for students assigned to lower DE, relative to only 2.5 percentage points for students assigned to upper DE (a difference of 11.8 percentage points). Additionally, the predicted probability of completing Intermediate Algebra after the reform increased by 12.2 percentage points for students assigned to lower DE relative to 7.2 percentage points for students assigned to upper DE. This indicates that the gap in performance on Intermediate Algebra outcomes by developmental education placement status narrowed after the reform. There were no statistically significant effects of the LowerDE X POST interaction terms in models 3 and 4, which suggests that the effects of the reform were similar by DE placement status for the gateway coursetaking outcomes. For students on both sides of the placement cutoff, the predicted probabilities increased after the reform for enrollment in gateway math by about 5 percentage points and for completion of gateway math by about 3 percentage points.

Discussion

While several states have experimented with developmental education reforms such as using multiple measures for placement decisions, Florida’s reform went a step further by letting most students decide whether to opt out of developmental education courses. However, there were concerns that some students who had not recently graduated with a standard diploma from a Florida public high school may not be adequately prepared for college-level courses, and these students were not exempt from prior requirements for placement testing and developmental education (for those scoring below college-ready). This study provides additional insight into the policy decision to leave this group of students subject to these traditional requirements, and has important implications for policymakers in Florida and other states as they consider who may benefit from being allowed to opt-out of developmental education. This is a policy issue of increasing relevance, as California has recently eliminated all free-standing developmental education courses in the California State University system, and is moving toward doing the same in the community college system beginning in the 2020/21 academic year (Mangan, 2019). There are concerns that this approach could harm some students in the absence of greater efforts to improve the academic preparation provided in K-12 schools to incoming students (e.g., Kurlaender, 2018).

Similar to other RD studies examining the effects of placement into developmental education (e.g., Valentine et al., 2017), we find that first-year math coursetaking outcomes tend to be worse for nonexempt students assigned to additional developmental education requirements relative to similar students scoring just above college-ready. This suggests that more nonexempt students may have benefitted from the option to enroll directly into college-level courses like their exempt peers, although the results may not be generalizable to all nonexempt students. Policymakers may want to consider removing exemption criteria and providing all students with the choice of opting out of developmental education. This finding is also consistent with results from the larger study of the impact of Florida’s developmental education reform for all students (Hu et al., 2019; Park-Gaghan et al., 2020), which finds that removing developmental education requirements tends to be beneficial for student outcomes when additional support services are provided.

We also find that the reform has positive effects on first-year coursetaking outcomes for students above and below both cutoffs for assignment to developmental education in math. The reform is beneficial for students just above the developmental cutoffs who are assigned to college-level courses and have additional supports, but overall the effects tend to be greater for students who have both additional support plus accelerated instructional strategies for developmental education. These gains were often quite large in magnitude, particularly for a statewide reform. This is consistent with the majority of prior research examining differential effects of developmental education by students’ academic preparation, which have found that lower-performing students who are assigned to multiple levels of developmental education tend to be less likely to progress to the next course in the sequence relative to students with fewer developmental requirements (Dadger, 2012; Melguizo et al., 2016; Ngo & Kosiewicz, 2017).

For the lower DE cutoff, the reform has greater effects on taking and completing Intermediate Algebra (the first college-level course) for students who would have been assigned to two levels of developmental education relative to those assigned to one level of developmental education. For the upper DE cutoff, the reform has greater effects on taking and completing gateway math courses (the next course in the math sequence) for students who would have been assigned to upper-level DE relative to those assigned to college-level math. These post-reform gains have narrowed preexisting gaps in coursetaking outcomes by developmental education placement status. Pre-post differences in outcomes for students placed into developmental education also suggest that providing students with developmental education courses through accelerated strategies with additional support services is more effective than assigning students to traditional developmental education courses. Despite these gains, we still do not know whether some students may benefit from enrolling in optional developmental education courses, or if it is more effective to eliminate developmental education completely. At least one prior study (Boatman & Long, 2018) has found that some lower-performing students benefit from assignment to multiple levels of developmental courses. If policymakers decide to continue offering developmental education courses, they should consider alternate instructional strategies, such as those used in Florida, to help students progress through developmental courses more quickly. Other states like Texas have taken a similar approach by implementing legislation effectively mandating that co-requisites be the primary developmental education model at all public institutions.

Further, while it is difficult (if not impossible) for us to attribute gains in student outcomes to any single component of the Florida reform, we argue that Florida’s reform demonstrates the importance of thinking about how to package complementary reform efforts together synergistically. These types of comprehensive reform efforts may have a greater effect than if each component was implemented on its own as an isolated practice. Particularly for students at the lower DE cutoff, it may be important to provide additional support services so that students have a greater chance of succeeding in more accelerated course formats. Colleges were also given some flexibility in designing their plans about how to implement changes to developmental education courses and advising services. This aspect of Florida’s reform recognizes that what works in one setting with a specific population of students, may not be as effective in a different context. Allowing colleges to modify implementation based on their own unique contexts also likely contributed to the overall success of Florida’s reform. Policymakers in other states should also consider how to offer a package of complementary reform initiatives in a way that supports local adaptations to meet the needs of unique student populations.

There were also some smaller gains after the reform for nonexempt students scoring above college-ready who were assigned to college-level classes. Even though most of these students did not participate in the accelerated instructional strategies for developmental education, they may have benefitted from the additional advising and academic support services that were available to all incoming students. Further, there is some evidence that Florida’s developmental education reform led to institutional transformation, which resulted in extensive changes to operational procedures, and cultural changes in the values and beliefs of diverse stakeholders throughout FCS institutions (Mokher et al., 2020). Even though the legislation did not require institutions to make changes to college-level and gateway courses, many institutional leaders reported making changes to instructional practices and curriculum in these courses as part of a larger effort to improve student success in the midst of the developmental education reform.

Both before and after the reform, enrollment rates in college-level and gateway math courses tend to be higher than the subsequent completion rates in these courses, which suggests that despite overall improvements in these outcomes post-reform, there are still some students who may not be adequately prepared to succeed in these courses. Future research could further explore whether there are certain subgroups of students who are more likely to be successful by beginning in an accelerated developmental course rather than enrolling directly into a college-level work. For example, prior research has found that older returning adults may be more in need of a “refresher” of math skills from high school (Bailey et al., 2010; Bettinger et al., 2013; Calcagno et al., 2007). Additionally, interviews with students themselves may provide insights into the extent to which students in accelerated developmental courses found these courses to be helpful. This type of information would also be useful for advisors who are tasked with helping students to make wise choices about course placement, often in the absence of test score information about students’ level of academic preparation. While the overall positive effects of Florida’s developmental education reform are encouraging, it is important to ensure, both in Florida and elsewhere, that there are no students outside of a given reform’s limelight who might be inadvertently left behind.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant [number] to [institution], and in part by a grant from the Bill & Melinda Gates Foundation. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education, or the Gates Foundation.

Notes

1. In some states, Intermediate Algebra is considered to be a developmental education course. However, in Florida, it is the first college-level math course and is a pre-requisite for almost all gateway math courses. SB 1720 only removed the requirement for developmental education courses for exempt students, and all students were still required to take Intermediate Algebra unless they had placement test scores indicating that they were eligible to enroll in a gateway math course.

2. In 2014 the cutoff for Intermediate Algebra increased by 1 point to 114. For the 2014 and 2015 cohorts, students with a score of 113 are categorized as assigned to upper-level developmental courses.

Appendix

DiRD sensitivity tests with alternate bandwidths, functional forms, and interaction terms

Table A1. Sensitivity analyses with alternate bandwidths and functional forms for impact of assignment to upper DE (relative to college-level) on enrollment and completion of Intermediate Algebra (or higher) for nonexempt students

	(1) Preferred specification w/MSE optimal bandwidths	(2) No covariates	(3) Quadratic	(4) Cubic	(5) Quartic	(6) CER-optimal bandwidths
	Enrollment in Intermediate Algebra
Upper DE	−0.572***	−0.520***	−0.833***	−0.814***	−0.805***	−0.659**
	(0.107)	(0.110)	(0.141)	(0.189)	(0.214)	(0.254)
Post	0.383***	0.460***	0.380***	0.380***	0.380***	0.280
	(0.104)	(0.104)	(0.103)	(0.103)	(0.102)	(0.174)
Upper DE X Post	−0.103	−0.129	−0.100	−0.100	−0.099	−0.168
	(0.093)	(0.087)	(0.092)	(0.092)	(0.093)	0.130
	Completion of Intermediate Algebra
Upper DE	−0.487***	−0.467***	0.543***	−0.734***	−0.726***	0.563**
	(0.092)	(0.093)	(0.057)	(0.137)	(0.174)	(0.111)
Post	0.320**	0.359**	1.374**	0.318**	0.318**	1.299
	(0.113)	(0.108)	(0.155)	(0.113)	(0.113)	(0.213)
Upper DE X Post	−0.069	−0.080	0.935	−0.068	−0.068	1.043
	(0.086)	(0.082)	(0.080)	(0.086)	(0.086)	(0.126)
Bandwidth	[−6, +5]	[−6, +5]	[−6, +5]	[−6, +5]	[−6, +5]	[−3, +3]
N	8,968	8,968	8,968	8,968	8,968	3,912

Results are from Difference-in-Regression Discontinuity models with alternate bandwidths and functional forms. ***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10.

Table A2. Sensitivity analyses with alternate bandwidths and functional forms for the impact of assignment to upper DE (relative to college-level) on enrollment and completion of gateway math (or higher) for nonexempt students

	(1) Preferred specification w/MSE optimal bandwidths	(2) No covariates	(3) Quadratic	(4) Cubic	(5) Quartic	(6) CER-optimal bandwidths
	Enrollment in Gateway Math
Upper DE	-0.558***	-0.514***	-0.636***	-0.589***	-0.606**	-0.602**
	(0.104)	(0.107)	(0.113)	(0.155)	(0.190)	(0.220)
Post	0.532***	0.576***	0.528***	0.528***	0.528***	0.643**
	(0.105)	(0.110)	(0.106)	(0.106)	(0.106)	(0.114)
Upper DE X Post	0.407***	0.369***	0.413***	0.414***	0.414***	0.402**
	(0.084)	(0.080)	(0.083)	(0.084)	(0.084)	(0.137)
	Completion of Gateway Math
Upper DE	-0.579***	-0.549***	-0.624***	-0.582**	-0.646**	-0.615*
	(0.116)	(0.115)	(0.119)	(0.207)	(0.236)	(0.276)
Post	0.515***	0.515***	0.512***	0.512***	0.512***	0.734***
	(0.100)	(0.113)	(0.100)	(0.100)	(0.100)	(0.164)
Upper DE X Post	0.348***	0.326***	0.352***	0.352***	0.351***	0.300
	(0.084)	(0.083)	(0.084)	(0.084)	(0.084)	(0.167)
Bandwidth	[-6, +5]	[-6, +5]	[-6, +5]	[-6, +5]	[-6, +5]	[-3, +3]
N	8,968	8,968	8,968	8,968	8,968	3,912

Results are from Difference-in-Regression Discontinuity models with alternate bandwidths and functional forms. ***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10.

Table A3. Sensitivity analyses with alternate bandwidths and functional forms for the impact of assignment to lower DE (relative to upper DE) on enrollment and completion of Intermediate Algebra (or higher) for nonexempt students

	(1) Preferred specification w/MSE optimal bandwidths	(2) No covariates	(3) Quadratic	(4) Cubic	(5) Quartic	(6) CER-optimal bandwidths
	Enrollment in Intermediate Algebra
Lower DE	-0.971***	-0.945***	-0.884***	-0.890***	-0.785***	-0.951***
	(0.149)	(0.152)	(0.165)	(0.157)	(0.219)	(0.196)
Post	0.107	0.083	0.108	0.108	0.109	0.205
	(0.129)	(0.138)	(0.129)	(0.129)	(0.129)	(0.183)
Lower DE X Post	0.722***	0.707***	0.720***	0.720***	0.720***	0.583**
	(0.115)	(0.116)	(0.115)	(0.115)	(0.115)	(0.167)
	Completion of Intermediate Algebra
Lower DE	-0.841***	-0.832***	-0.637***	-0.620***	-0.347	-0.426*
	(0.146)	(0.145)	(0.161)	(0.138)	(0.221)	(0.173)
Post	0.433***	0.396***	0.435***	0.435***	0.436***	0.334^~
	(0.109)	(0.117)	(0.108)	(0.108)	(0.108)	(0.189)
Lower DE X Post	0.606***	0.591***	0.600***	0.600***	0.601***	0.434*
	(0.134)	(0.133)	(0.134)	(0.135)	(0.134)	(0.192)
Bandwidth	[-4, +6]	[-4, +6]	[-4, +6]	[-4, +6]	[-4, +6]	[-2, +3]
N	13,772	13,772	13,772	13,772	13,772	5,073

Results are from Difference-in-Regression Discontinuity models with alternate bandwidths and functional forms. ***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10.

Table A4. Sensitivity analyses with alternate bandwidths and functional forms for the impact of assignment to lower DE (relative to upper DE) on enrollment and completion of gateway math (or higher) for nonexempt students

	(1) Preferred specification w/MSE optimal bandwidths	(2) No covariates	(3) Quadratic	(4) Cubic	(5) Quartic	(6) CER-optimal bandwidths
	Enrollment in Gateway Math
Lower DE	-0.253	-0.209	-0.357	-0.341	0.089	0.255
	(0.213)	(0.223)	(0.236)	(0.225)	(0.349)	(0.321)
Post	0.898***	0.753**	0.896***	0.896***	0.897***	1.320***
	(0.204)	(0.247)	(0.203)	(0.202)	(0.203)	(0.305)
Lower DE X Post	0.200	0.185	0.203	0.203	0.203	-0.197
	(0.175)	(0.172)	(0.174)	(0.174)	(0.174)	(0.384)
	Completion of Gateway Math
Lower DE	-0.375	-0.339	-0.684	-0.920*	-0.596	0.014
	(0.293)	(0.294)	(0.432)	(0.411)	(0.463)	(0.399)
Post	0.948***	0.827**	0.943***	0.948***	0.950***	1.546*
	(0.272)	(0.296)	(0.270)	(0.268)	(0.269)	(0.376)
Lower DE X Post	0.173	0.154	0.185	0.180	0.180	-0.466
	(0.183)	(0.184)	(0.181)	(0.181)	(0.180)	0.371
Bandwidth	[-4, +6]	[-4, +6]	[-4, +6]	[-4, +6]	[-4, +6]	[-2, +3]
N	13,772	13,772	13,772	13,772	13,772	5,073

Results are from Difference-in-Regression Discontinuity models with alternate bandwidths and functional forms. ***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10.

Table A5. Sensitivity analyses with interaction terms for the impact of assignment to upper DE (relative to college-level) on enrollment and completion of Intermediate Algebra (or higher) for nonexempt students

	(1) No interaction	(2) Interaction for DE*PERT	(3) Interaction for DE*PERT and DE*PERT*POST
	Enrollment in Intermediate Algebra
Upper DE	-0.572***	-0.746***	-0.715***
	(0.107)	(0.133)	(0.121)
Post	0.383***	0.457***	0.457***
	(0.104)	(0.103)	(0.102)
Upper DE X Post	-0.103	-0.126	-0.189
	(0.093)	(0.086)	(0.144)
	Completion of Intermediate Algebra
Upper DE	-0.487***	-0.586***	-0.613***
	(0.092)	(0.105)	(0.124)
Post	0.320**	0.318**	0.317**
	(0.113)	(0.113)	(0.113)
Upper DE X Post	-0.069	-0.067	-0.185
	(0.086)	(0.085)	(0.124)
Bandwidth	[-6, +5]	[-6, +5]	[-6, +5]
N	8,968	8,968	8,968

Results are from Difference-in-Regression Discontinuity models with different specifications of interaction terms. ***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10.

Table A6. Sensitivity analyses with interaction terms for the impact of assignment to upper DE (relative to college-level) on enrollment and completion of gateway math (or higher) for nonexempt students

	(1) No interaction	(2) Interaction for DE*PERT	(3) Interaction for DE*PERT and DE*PERT*POST
	Enrollment in Gateway Algebra
Upper DE	-0.558***	-0.619***	-0.512***
	(0.104)	(0.109)	(0.154)
Post	0.532***	0.528***	0.528***
	(0.105)	(0.106)	(0.106)
Upper DE X Post	0.407***	0.414***	0.246
	(0.084)	(0.083)	(0.172)
	Completion of Gateway Algebra
Upper DE	-0.579***	-0.61***	-0.449**
	(0.116)	(0.12)	(0.189)
Post	0.515***	0.51***	0.512***
	(0.100)	(0.10)	(0.100)
Upper DE X Post	0.348***	0.35***	0.104
	(0.084)	(0.08)	(0.196)
Bandwidth	[-6, +5]	[-6, +5]	[-6, +5]
N	8,968	8,968	8,968

Results are from Difference-in-Regression Discontinuity models with different specifications of interaction terms. ***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10.

Table A7. Sensitivity analyses with interaction terms for the impact of assignment to lower DE (relative to upper DE) on enrollment and completion of Intermediate Algebra (or higher) for nonexempt students

	(1) No interaction	(2) Interaction for DE*PERT	(3) Interaction for DE*PERT and DE*PERT*POST
	Enrollment in Intermediate Algebra
Upper DE	-0.971***	-0.887***	-0.798***
	(0.149)	(0.163)	(0.209)
Post	0.107	0.109	0.108
	(0.129)	(0.129)	(0.129)
Upper DE X Post	0.722***	0.720***	0.568*
	(0.115)	(0.115)	(0.238)
	Completion of Intermediate Algebra
Upper DE	-0.841***	-0.673***	-0.582**
	(0.146)	(0.159)	(0.236)
Post	0.433***	0.436***	0.436***
	(0.109)	(0.108)	(0.084)
Upper DE X Post	0.606***	0.600***	0.451
	(0.134)	(0.135)	(0.279)
Bandwidth	[-4, +6]	[-4, +6]	[-4, +6]
N	13,772	13,772	13,772

Results are from Difference-in-Regression Discontinuity models with different specifications of interaction terms. ***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10.

Table A8. Sensitivity analyses with interaction terms for the impact of assignment to lower DE (relative to upper DE) on enrollment and completion of gateway math (or higher) for nonexempt students

	(1) No interaction	(2) Interaction for DE*PERT	(3) Interaction for DE*PERT and DE*PERT*POST
	Enrollment in Gateway Algebra
Upper DE	-0.253	-0.324	-0.523
	(0.213)	(0.230)	(0.424)
Post	0.898***	0.896***	0.897***
	(0.204)	(0.203)	(0.203)
Upper DE X Post	0.200	0.202	0.456
	(0.175)	(0.174)	(0.452)
	Completion of Gateway Algebra
Upper DE	-0.375	-0.531	-0.386
	(0.293)	(0.392)	(0.726)
Post	0.948***	0.944***	0.944***
	(0.272)	(0.180)	(0.269)
Upper DE X Post	0.173	0.062	-0.005
	(0.183)	(0.037)	(0.607)
Bandwidth	[-4, +6]	[-4, +6]	[-4, +6]
N	13,772	13,772	13,772

Results are from Difference-in-Regression Discontinuity models with different specifications of interaction terms. ***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10.