Do student evaluations of teaching depend on the distribution of expected grade?

Abstract

Prior research suggests that student evaluations of teaching may depend on the average grade expected in a class. We hypothesize that, because of risk aversion, student ratings also depend on the distribution of expected grades. As predicted, student ratings at the University of Puerto Rico at Bayamón are significantly and negatively related to the variance of expected grades, implying that faculty may be able to boost their student evaluations of teaching ratings by narrowing the grade distribution. Findings are also consistent with the hypothesis that weak students place the highest value on a tight distribution of expected grades.

Keywords:

• student evaluations
• expected grade
• grade distribution
• risk aversion

Acknowledgements

The authors are deeply grateful to Gilberto Calderón for compiling data for this project. They thank Dong Li and an anonymous referee for their valuable advice; Lance Bachmeier, Hamilton Fout, and Dennis Weisman for helpful comments; and Renfeng Xiao for research assistance. H. Matos‐Díaz acknowledges the support of the University of Puerto Rico at Bayamón through a sabbatical leave that allowed him to work on this project.

Notes

1. According to Algozzine et al. (2004), Remmers (1927) created the precursor of SET with the ‘Purdue Rating Scale for Instructors.’ Sailor, Worthen, and Shin (1997) report that, within a year of its creation, members of the academic community were challenging the validity and adequacy of student ratings. Among the issues still debated are the proper use of SET ratings and the potential ability of instructors to obtain higher ratings through more lenient grading.

2. Among the studies to conclude that expected grade is not a significant determinant of SET are Seiver (1983), DeCanio (1986), and Marsh and Roche (2000).

3. In Nowell’s model, SET = a EG_i + b EG_i / EG, where EG is the average expected grade of the class. According to Nowell’s empirical estimates, both statistically significant, a > 0 and b < 0. The latter result implies that, for a given value of EG_i , ∂SET / ∂EG > 0.

4. An alternative measure of the reward structure is the variance of students’ actual course grade at the time the class is evaluated, but this information is missing. The two measures should be highly correlated, so we proceed with the available data, but future work might consider alternative proxies for a teacher’s reward structure.

5. See Feldman (1978), Marsh and Duncan (1992), Centra (1993), Braskamp and Ory (1994), and Sixbury and Cashin (1995).

6. See EViews (2004, 887), for details.

7. According to the Hausman test, the fixed‐effects model is preferable to random effects (the assumption of no correlation between µ_i and the explanatory variables is rejected at the 0.01 level). But, as illustrated in the empirical section, results are largely invariant to the estimation technique adopted.

8. The university computes the average value of SET based on the questions for which the student assigns a value between one and four; missing values are ignored.

9. For each GPA category listed on the SET form, we assign the midpoint. For example, students who report a GPA > 3.0 are assigned a value of 3.5. As a referee noted, because students self‐report both expected grade and GPA, students may link the two variables more closely, which would inflate the correlation between these two variables. But the correlation between the two variables is only 0.38, which suggests that collinearity between the variables is not a severe problem.

10. We also allowed for the possibility that the payoff to a higher expected grade or the penalty for greater variance in expected grade might vary by field by interacting both variables with field dummies. Results pointed to a common structure; the hypothesis of identical coefficients could not be rejected at even the 0.20 level (F(10, 1196) = 1.23). These findings justify our decision to pool results across fields.

11. The effect of gender of the instructor has been studied extensively (for reviews, see Feldman 1993; Braskamp and Ory 1994). As a rule, gender is not significantly related to SET. Typically, the effect of age, degree, and rank are also insignificant, although Feldman (1983) and McPherson, Jewell, and Kim (2007) find that SET is negatively related to age.

12. Most other studies also conclude that the time a class is taught is not important (for example, Feldman 1978; Aleamoni 1981; Nelson and Lynch 1984). The effect of class size in the literature is mixed. Often, class size turns out to be insignificant (as in Nelson and Lynch 1984; Krautmann and Sander 1999). Some studies find an inverse relationship between size and student ratings (for example, Isely and Singh 2005; Bedard and Kuhn 2008; and, in a study of the quality of mathematics classes in Sweden, Westerlund 2008). In other cases, the relationship is positive, as in Mirus (1973) and, for core economics courses, Boex (2000).

13. Nowell (2007) is an exception. Based on a population of 1016 students, which includes those who do not complete the SET survey, he is able to estimate the probability that students who complete the course also answer the survey questions. He finds that sample selection is not a significant determinant of SET ratings. Grimes (1994) and Becker and Powers (2001) also control for sample selectivity, but their dependent variables are economic understanding and student performance, respectively, rather than student assessment of teaching.

14. When we experimented with models in which expected grade was estimated though two‐stage least squares, we too were unable to reject the assumption that expected grade is exogenous. We do not report these results because we do not believe that the identifying variables available to us in the data‐set are valid. A key challenge in modeling expected grade as endogenous is to find identifying variables that are highly correlated with expected grade but not correlated with the unobserved error term in the SET equation.