A framework for using SET when evaluating faculty

ABSTRACT

This paper presents a framework for utilizing Student Evaluations of Teaching (SET) in faculty evaluations. Recognizing the ongoing debate about the validity of SET as a measure of teaching effectiveness, the authors agree with scholars who propose viewing SET as a tool for gauging ‘student perceptions of learning’. They present a method that combines post-estimation residuals from an Ordinary Least Squares (OLS) model, which accounts for key confounding factors, with simple mean SET scores. This approach provides a more nuanced perspective on each faculty member's individual ratings. The authors draw on SET data from courses taught by tenured faculty in the Faculty of Arts and Sciences at a large, private university in the Northeastern United States. Their findings suggest that the proposed residual framework can help address some of the concerns associated with the use of SET in personnel decision-making. However, they caution that this approach requires careful explanation and should be used in conjunction with a comprehensive review of a professor's full teaching portfolio. The study contributes to the ongoing discourse on the use of SET in faculty evaluations, offering a more equitable and nuanced approach to interpreting SET data.

KEYWORDS:

• Student evaluations of teaching
• postestimation predictions
• faculty evaluation
• bias

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 We include only courses classified as “Primarily for undergraduates” or “undergraduate/graduate” because our model is used as one piece of information in determining undergraduate teaching awards. However, institutions could also include graduate courses based on the needs of the institution.

2 An additional 21 tenured faculty who were active on January 1, 2022, have no undergraduate SET data between Fall 2010 and Fall 2021. A majority of these 21 faculty have only recently started their appointments.

3 Due to confidentiality concerns, the data used are not publicly available. However, aggregated data and code are available upon reasonable request.

4 For data from Fall 2006 to Fall 2010, we found that female faculty members received statistically significantly lower scores than male faculty members, indicating potential bias against women. From data since Fall 2010, we have found no evidence of systematic gender bias against women in instructor overall effectiveness scores (Zipser, Mincieli, and Kurochkin 2021).

5 The “instructor overall” question asks students to “Evaluate your instructor overall,” with a scale of 1=unsatisfactory, 2=fair, 3=good, 4=very good, 5=excellent.

6 This methodology is similar to that used in the educational value-added literature. Please see Topping and Sanders (2000) for a description of this methodology.

7 Due to confidentiality concerns, the data used are not publicly available. However, aggregated data and code are available upon request.

8 Although neither the female instructor dummy variable nor the variable measuring the proportion of female students in the course is statistically significant in variation D of the model [or column 13], we also run a variation of the model (E) where we interact the gender of the instructor with the fraction of female students in the course. Using this model, we find that the expected score difference between female and male faculty at the mean proportion of female students in the course (47% female) is 0.01 of a point (holding the other variables in the model constant at the sample means). At one standard deviation below the mean (22% female), the difference is 0.05 of a point, and at one standard deviation above the mean (69% female), the difference is -0.02 of a point. In other words, as the fraction of male students increases, SET scores for female instructors increase, and as the fraction of female students increases, SET scores for female instructors decrease. This finding is the reverse of what is usually found in the literature, in which female instructors receive higher SET scores when more female students are enrolled in a course.