An Elephant in the Room: Bias in Evaluating a Required Quantitative Methods Course

Abstract

Undergraduate Political Science programs often require students to take a quantitative research methods course. Such courses are typically among the most poorly rated. This can be due, in part, to the way in which courses are evaluated. Students are generally asked to provide an overall rating, which, in turn, is widely used by students, faculty, and administrators to assess a course. Unfortunately, even questions composed with the best of intentions have the potential to bias the results. In this article, we evaluate the global rating question used at our university and show that it introduces bias into the measure by cuing extraneous considerations. It artificially inflates the number of negative reactions to the course by leading students to think about its required status and their initial level of enthusiasm rather than their level of accomplishment and its value as a learning experience. By locating our results in the course evaluation and framing literature, we suggest an approach to evaluating overall rating questions that can be adapted for use at other institutions.

Keywords:

• course evaluations
• framing
• quantitative methods

Notes

Note: χ² = 8.46 with one df; p = .004.

χ² = 75.6 with one df; p = .000; McNemar Exact significance = .000.

*p < .05. **p < .01. ***p < .001.

All classes were taught by the same instructor precluding examination of instructor effects.

For example, specialists are required to take half of their 20 credits in political science courses. Political Science majors must take seven. Roughly 90% of those who take the course are required to do so.

As part of their requirements, students are, at times, given a pool of courses from which they need to pick one. These courses were treated as required and considered nonquantitative.

The 2011–2012 findings suggest that the required, nonquantitative courses eclipsed nonrequired courses in retake rate. One explanation for this is that a popular first-year course became required in 2011–2012.

The revised question appears to go some distance toward closing the ratings gaps shown in Figure 1, but any such comparison should be based on similar not different questions.

The McNemar test, specifically designed to determine whether the same respondents differ in their answers on two measures, produces an exact significance of .000.

The multivariate models exclude approximately 27% of the respondents due to missing data on at least one of the independent variables. Those excluded do not differ significantly from students included on any of the variables in the equation (p values range from .21 to .99 with an average of .61). Green (1991) suggests a minimum sample size of 50. To increase the sample size responses for 17 cases on the learning experience question were imputed based on a close examination of the answer sheets. Their inclusion does not substantially affect the analysis nor does constraining the analysis to only those required to take the course.

Preliminary effort at path analysis suggests that the effect of required status on the traditional retake question may be mediated by initial level of enthusiasm.

In selecting items for inclusion in the models, we sought to balance concerns over bias and efficiency. A number of inefficient factors such as questions about the workload, the quality of the readings as well as ratings of the tutorials and laboratory sessions that are part of the course are insignificant as predictors; eliminating them does not substantially affect the model or the conclusions drawn here.

This makes sense in that neither the traditional nor revised question evoke considerations of matters such as degree of difficulty, workload, or time commitment entailed by the course. It would be possible, of course, to design a question that does cue such considerations by asking respondents to consider how much they had to do for the class, how many Saturday afternoons and evenings they had to devote to the course, etc.

All other variables in the model are held constant.

The gap between them narrows considerably from .33 (.58 - .25) on the left hand side to .11 on the right (.95 - .84).

It moves from .21 on the left (.77 - .56) to .70 on the right (.95 - .88).