From SIGNALS to success? The effects of an online advising system on course grades

ABSTRACT

Online advising tools have the potential to reach a large number of students with the goal of increasing course performance and learning with relatively modest time investment from the instructor. Course SIGNALS provides students with online, real-time feedback on course performance based on an algorithm using the student’s effort, achievement, academic history, and demographic characteristics. Using roughly 25,000 student-course level observations from a large, research-intensive university in the Midwest, results from fixed effects difference-in-differences models indicate that SIGNALS helps improves grades of lower-achieving students and decreases the probability that students will fail the course.

KEYWORDS:

• Instructional technologies
• grades
• advising
• gender
• race/ethnicity

JEL Codes:

• I21

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 For the importance of grades on persistence in a field, please see our discussion in the literature review below. References most relevant include Rask and Tiefenthaler (2008), Ost (2010), Owen (2010), and Rask (2010).

2 Due to the proprietary nature of the algorithm and program, it is not possible to provide any more in-depth information on how the program functions. It is important to note that instructors can use the program differently and therefore the ‘treatment’ may differ across instructors. More details are provided in the discussion section.

3 To construct a match when there are no sections of the same course number that never adopted SIGNALS, we chose the nearest course numbers with the closest course characteristics (level and content) to the course material in the SIGNALS course of interest.

4 For first-time students observed in the fall of 2008, we do not have a previous measure of GPA to use as a control. In this case, we set the previous GPA to zero and include a dummy variable ‘New Student’ to control for the missing data.

5 Although we do have access to SAT composite scores, the data are missing for many of the students in our sample, so we do not include it in our main models. However, our results are robust to its inclusion with a flag to indicate missing values.

6 Students are considered to have received an A if they earned an A- through A+, a B if they earned a B- through B+, and so on.

7 When the sample is restricted to courses for which there are at least 4 observations, the results remain qualitatively the same, although significance is lower given the smaller sample size. These results are available from the corresponding author upon request, but are omitted here for the sake of brevity.

8 Due to the proprietary nature of the advising program, we were unable to calculate an estimate of the cost of providing SIGNALS. However, once the program is developed (or purchased) the majority of costs come in time of staff and instructors, manipulating the algorithm and delivering the nudges to the students. Therefore, if deployed widely across many large classes, the cost appears to be very low, for a reasonable gain in grades for lower-ability students. However, this should be weighed against the possibility of less of an impact in larger classes.