Proposed metrics for summarizing student evaluation of teaching data from balanced Likert scale surveys

Abstract

Student Evaluations of Teaching (SETs) gather crucial feedback on student experiences of teaching and learning and have been used for decades to evaluate the quality of teaching and student experience of instruction. In this paper, we make the case for an important improvement to the analysis of SET data that can further refine its interpretation. An extensive literature explores the reliability and consistency of SETs, and factors hypothesized to affect student evaluation of instructors. Much less attention has been given to the statistics used by institutions of higher education to summarize and report SET data—historically, arithmetic means and standard deviations computed from Likert scale scores. Here, we argue that a combination of two different statistics—interpolated median (IM) and percent favourable (PF) rating—together with a measure of dispersion suitable for ordinal data offer a fairer and more meaningful set of metrics for summarizing and interrogating SET data collected using balanced Likert scale surveys. To illustrate the power of these proposed metrics, we present an analysis of SET data from two academic years at a major Canadian university, and we also highlight an interesting and previously unreported relationship that we identified between the interpolated median and percent favourable rating.

Keywords:

• interpolated median
• dispersion index
• student evaluation of teaching
• percent favourable rating
• Likert scale

1. Introduction

Student evaluations of teaching (SET), while not uncontroversial (see, for example, Aleamoni (1999); Feldman (2007)), are instruments that have been employed in higher education since at least the 1920s. Originally introduced for the purpose of improving teaching practice, over time they have increasingly become tools of “performance management” (Galbraith et al., 2012). Marsh (2007) explains, for example, that while SETs are commonly applied to offer diagnostic feedback to teachers, facilitate certain kinds of pedagogical research, develop measures of teaching effectiveness, and/or give students additional information that may guide future course selections, they are most frequently employed for purposes of “quality assurance”—including monitoring and management of the performance of instructional staff.

The widespread use of SET, the reliability and consistency of this approach for evaluation of teaching effectiveness, and the factors that may affect student rating of their instructors, have been the subject of numerous studies over decades (for an overview, see, for example, Blackmore, 2009; Clayson, 2009; Spooren et al., 2013). Few studies, however, have examined the statistics used to summarize and present SET data, or whether they are fit for purpose. In particular, little attention has been given to the appropriateness of the different descriptive statistics regularly computed in SET reports.

As long ago as 1987, and referencing the extensive body of work interrogating the validity and application of SETs, Marsh noted that SETs are probably “the most thoroughly studied of all forms of personnel evaluation” (p. 369) and instead urged that future SET research should focus on (among other things) methodology. Since then, numerous studies have proposed different approaches to analysing SET data, such as Item Response Theory (IRT) and/or multilevel modelling (Bacci & Caviezel, 2011; La Rocca et al., 2017; Rampichini et al., 2004). These model-based analyses allow for comparisons across course/instructor ratings, considering, among other things, variations in student characteristics and course attributes. However, in practice—and perhaps because of the perceived complexity of these latter approaches?—institutions of higher education in North America by and large use the mean and standard deviation to report individual instructor SET results (Stark & Freishtat, 2014). Parametric analyses of this kind can be appropriate for analysis of ordinal data, if certain distributional assumptions are met (Harris, 2001; Tabachnick & Fidell, 2021)—principally that the data are normally or near normally distributed. However, SET data is often negatively skewed and not normally distributed, because students tend to rate their experience of instruction favourably more often than not (Nulty, 2008; Zumrawi et al., 2014).

Kitto et al. (2019) build on this observation and offer an excellent overview of the widespread problem of inappropriate analysis and the resulting ill-informed interpretation of SET data. Given that the statistics used in summarizing and reporting SET data, and their interpretation, can dramatically affect career progression of academic staff (hiring, (re)appointment, assignment of rewards and awards, promotion, tenure) this is a critical problem.

2. Background

Typical SET surveys in higher education offer students a set of statements about their learning experience (for example, “The instructor made it clear what I was expected to learn”) and invite them to indicate their level of agreement or disagreement with each statement on a 5- or 7-point Likert scale. Student responses to these survey items are then converted to numerical scores on scales of 1–5 or 1–7, depending on the scale used. Such scores are ordinal—that is to say, they represent a series of categories of response that exist in a particular order, but the distance between the categories (say, between a score of 3 and a score of 4) is not known. Nonetheless, it is common practice to treat such scores as if they are interval data and assume that such scores are distributed along a measurable scale and spaced equidistantly from each other. Because there is a desire to compute a single and simple representative composite score that somehow represents the “central tendency” of scores for each instructor, university analysts typically then calculate an “average (arithmetic mean) score” from the set of Likert scale scores an instructor receives. As Stark and Freishtat (2014) note, “Averages of SET ratings have an air of objectivity simply by virtue of being numerical” (p. 1). These are often then simplistically compared with, say, the departmental average score. As noted earlier, the problem with this approach is that SET data does not meet the distributional assumptions required for parametric analysis.

More importantly, and as Kitto et al. (2019) recently demonstrated through the use of hypothetical examples, very different distributions of student responses (scores) can result in identical average scores (for an illustration of this, see Figure 2, below). That is to say, SET surveys for two different educators may receive two very different sets of scores—representing significant differences in student sentiment or satisfaction—but their average scores may be identical. The use of an average can therefore mask great underlying variability in evaluation of teaching effectiveness and can lead to simplistic comparisons and conclusions regarding an individual educator’s performance. Stark and Freishtat (2014) similarly highlight the challenges of meaningfully capturing, summarizing, and presenting SET data, and argue that distribution of scores—the % of scores in each category—gives more meaningful insight.

2.1. The interpolated median: a better approach to representing central tendency in SET

The preferred and more interpretatively valuable measure of central tendency for ordinal data such as Likert scale SET scores is the simple median (Manikandan, 2011). The simple median (referred to here as the customary median) is the value which divides the data into two halves. It is the middle data point, or the 50th percentile. However, while the customary median is preferred over the mean as a measure of central tendency for ordinal data, we propose that an even more appropriate measure is the “special variant” of median obtained by interpolating or adjusting the customary median upwards or downwards to better reflect the distribution of individual scores. This special variant of the median has been known as the interpolated median.

In their book Fundamental statistics in psychology and education, Guilford and Fruchter (1978) described an interpolation method for calculating the median for grouped data (that is to say, data “binned” into frequencies per category, where individual data items are not available). They extended the method to calculate the median of discrete ungrouped data such as SET scores. Some commercially available systems that process and report on SET surveys (for example, Blue from Explorance¹) do now compute and report the IM.

The value and relevance of the IM for fuzzy data that represent ordinal sentiment/satisfaction responses on an SET survey can best be explained if we think of the set of discrete responses on a Likert scale as representing a distribution of perspectives held by the respondents that actually vary in degree much more finely than by the simple scale units offered. Each numerical score assigned to represent a value label is therefore understood as representing a “cluster” of responses that span a range. For example, the group of students who select 4 (“Agree”) in response to a SET question can be understood to encompass a range of perceptions, from those bordering on “Neutral” (a score of 3) to those leaning towards “Strongly Agree” (a score of 5). The numerical representation of 4 can therefore be better understood as the mid-point of an actual perspective interval ranging from 3.5 to 4.5. This is true for each “score” on a Likert scale.

How can we determine whether more students are at the 3.5 or the 4.5 end of that spectrum, when they selected a score of 4? This is where the IM can help. Consider an example in which the simple median of a set of SET responses on a 5-point Likert scale is 4 (“Agree”). Recall that for interval data of this kind, this simple median (midpoint) of 4 may actually represent student sentiment closer to 3.5 or closer to 4.5. Interpolation allows the simple median to be meaningfully adjusted upwards or downwards, depending on the overall distribution of scores in the set of responses, to better represent the overall distribution of student perspective. Where the overall proportion of scores above the value of the simple median score is higher, it is meaningful and valid to adjust the median score upwards; where the overall proportion of scores below the value of the simple median is higher, it is meaningful and valid to adjust the median score downwards. Such adjustments to the median therefore accommodate a measure of overall sentiment distribution into the IM as a measure of central tendency.

2.2. The need to represent ‘percent favourable rating’

A balanced Likert-type rating scale contains equal number of favourable and unfavourable response categories, often, but not necessarily, centred around a “neutral” category. Thus, the bounds of what constitutes favourable rating are well defined. For example, in a balanced 5-point rating scale, and within the context of interpolation (as shown above), the lower bound for a favourable rating is 3.5. On the other hand, an unbalanced rating scale, would have unequal number of favourable and unfavourable categories (Friedman et al., 1981). Furthermore, which end of an unbalanced scale has more response categories depends on the choices a researcher makes based on their knowledge of the target audience. For this reason, the bounds of what constitutes a favourable rating can vary depending on which end of the scale has more response categories. The use of percent favourable ratings in this paper therefore applies to balanced rating scales.

For a balanced Likert-type rating scale, percent favourable (PF) rating is the percentage of responses that are higher than “neutral”. PF is a simple yet meaningful statistic and is a good indicator of overall student satisfaction with their learning experience.

Students do in fact tend to rate their instructors favourably more often than not (Nulty, 2008). For example, in a 2014 study of 4 years of SET data from the University of British Columbia (UBC), Canada, Zumrawi et al. (2014) found that, overall, greater than 78% of student responses were favourable (defined as responses of “Agree” (4) or “Strongly Agree” (5) on a 5-point Likert scale).

2.3. Dispersion index: an appropriate measure of score distribution

If we accept that the distribution of student response scores on Likert scale SET surveys is significant and meaningful, how can we best represent it numerically? While Stark and Freishtat (2014) propose the use of bar charts, these clearly do not offer an efficient or reportable metric. Historically, standard deviation (a measure of score variability in relation to the mean) has been employed as a measure of data dispersion. But as with the mean itself, standard deviation is not an appropriate measure of distribution of ordinal data such as SET scores (Rampichini et al., 2004). Instead, we concur with these authors and make use of a measure of dispersion which is more appropriate for categorical data. This particular Dispersion Index (DI) ranges in value from 0 to 1.0. A value of 0 is obtained when all student respondents agree on the same rating. A value of 1.0, on the other hand, occurs when respondents split 50/50 between the two extreme responses, e.g., “Strongly Disagree” and “Strongly Agree” in the case of a 5-point Likert scale.

2.4. New metrics for meaningful representation of instructor SET scores

To overcome the statistical challenge described above, we therefore propose that SET scores for educators are more appropriately and meaningfully represented using the interpolated median (IM) as a measure of central tendency and the percent favourable (PF) rating. The use of these two statistics, along with a measure of dispersion suitable for ordinal data, provides much more appropriate metrics for reporting of SET data than the commonly used arithmetic mean and standard deviation. Through investigation of a sample set of SET data, we illustrate, below, the properties of the IM as a measure of central tendency for SET data collected using a balanced, 5- or 7-point, Likert scale survey instrument. Moreover, we demonstrate a previously unreported relationship between the IM and PF and show how these two statistics can be combined with an appropriate measure of dispersion to summarize and interrogate SET data in a more meaningful way that can reveal nuanced differences in student perspectives.

3. Methods

In this study, we used SET data collected at the University of British Columbia, Canada, during the 2018 and 2019 academic years, to illustrate the implementation of the proposed metrics. (The analysis examines anonymized data from standard annual university surveys and is exempted from Research Ethics Board review as institutional quality assurance research). Standard UBC evaluation of teaching surveys is released to students a few weeks before final exams. The survey items are modular in nature and include university module items (UMI), and academic unit-specific items. The UMI and most academic unit-specific items use a 5-point Likert scale; some academic unit-specific items use a 7-point scale. In the UMI items, students are invited to use the 5-point Likert scale to indicate their level of agreement with the item statements: 1 = “Strongly Disagree”, 2 = “Disagree”, 3 = “Neutral”, 4 = “Agree”, 5 = “Strongly Agree”.

For the purposes of this study, we focused on SET data from five UMIs, gathered by 16,982 SET surveys in the two academic years. While the vast majority of these surveys employed a 5-point Likert scale, a small subsample of surveys from one academic unit that made use of a 7-point scale, was used to illustrate the extension of the proposed methodology from a 5-point scale to a 7-point scale. For each course/instructor survey, we computed summary statistics, including the mean, median, IM, DI, and PF rating.

It is important to note that the relationships between the statistics in the proposed metrics are based on the properties of these statistics, within the context of a balanced rating scale. For this reason, these metrics are applicable to survey data from balanced Likert-type rating scales. UBC SET data is used for illustrative purposes.

3.1. Calculation of interpolated median, percent favourable rating and dispersion index

3.1.1. Percent favourable (PF) rating

PF rating represents the proportion of responses that are strictly greater than a “Neutral” response on a balanced Likert-type rating scale (see, for example, Nulty (2008)). For example, in the case of a 5-point scale, PF rating is calculated by dividing the sum of “Agree” and “Strongly Agree” responses by the total number of responses, expressed as a percentage.

3.1.2. Interpolated median (IM)

We used the following form of the IM formula (Lorenz, 2018):

(1) $\mathrm{I}\mathrm{M}=\mathrm{M}+\left({\mathrm{N}}_{\mathrm{g}}-{\mathrm{N}}_{\mathrm{s}}\right)/\left(2{\mathrm{N}}_{\mathrm{e}}\right)\mathrm{i}\mathrm{f}{\mathrm{N}}_{\mathrm{e}}\mathrm{i}\mathrm{s}\mathrm{n}\mathrm{o}\mathrm{n}-\mathrm{z}\mathrm{e}\mathrm{r}\mathrm{o}$(1)

(2) $\mathrm{I}\mathrm{M}=\mathrm{M}\mathrm{i}\mathrm{f}{\mathrm{N}}_{\mathrm{e}}\mathrm{i}\mathrm{s}\mathrm{z}\mathrm{e}\mathrm{r}\mathrm{o}$(2)

where:

M = the standard or customary median of responses

N_s = number of responses less than M

N_e = number of responses equal to M

N_g = number of responses greater than M

Equation (1)(1) $\mathrm{I}\mathrm{M}=\mathrm{M}+\left({\mathrm{N}}_{\mathrm{g}}-{\mathrm{N}}_{\mathrm{s}}\right)/\left(2{\mathrm{N}}_{\mathrm{e}}\right)\mathrm{i}\mathrm{f}{\mathrm{N}}_{\mathrm{e}}\mathrm{i}\mathrm{s}\mathrm{n}\mathrm{o}\mathrm{n}-\mathrm{z}\mathrm{e}\mathrm{r}\mathrm{o}$(1) is employed in cases where at least one response is equal to the median. In this case, and depending on the distribution of scores, the median is adjusted upwards or downwards by the quantity [(N_g - N_s)/(2N_e)]. The range of this adjustment quantity is (−0.5, 0.5), i.e., IM is calculated by adjusting the median downwards by a (negative) value that is >−0.5 or upwards by a (positive) value of <0.5.

Equation (2)(2) $\mathrm{I}\mathrm{M}=\mathrm{M}\mathrm{i}\mathrm{f}{\mathrm{N}}_{\mathrm{e}}\mathrm{i}\mathrm{s}\mathrm{z}\mathrm{e}\mathrm{r}\mathrm{o}$(2) is employed in cases where no response is equal to the median and no interpolation is required, i.e., IM is equal to the customary median.

3.1.3. Dispersion index (DI)

The DI is calculated using the following formula, as described by Rampichini et al. (2004):

(3) $D=\frac{\left\{{\sum }_{k=1}^{K-1}{F}_k\left(1-F_k\right)\hspace{0.17em}\right\}}{\frac{K-1}{4}},\hspace{0.17em}0\le D\le 1,\hspace{0.17em}$(3)

Where:

D = Dispersion Index

K = 1, 2, …; K is the categories of possible responses for each item

F_k = the relative cumulative distribution of responses

DI ranges in value from 0 to 1.0. A value of 0 is obtained when all student respondents agree on the same rating. A value of 1.0, on the other hand, occurs when respondents split 50/50 between the two extreme responses, e.g., “Strongly Disagree” and “Strongly Agree” in the case of a 5-point Likert scale.

4. Results

4.1. The differentiating power of the interpolated median

To demonstrate that the IM offers a simple measure of central tendency in a set of Likert scale scores that represents score distribution better than the arithmetic mean, Figure 1 gives a scatter plot of 83 SET ratings from the 2019 academic year. All 83 of these individual instructor surveys yielded the same mean score (3.4) but are found to possess a wide range of IM and PF ratings: IM ranges from 3.2 to 3.8, and the associated PF rating ranges from 37% to 76%.

Figure 1. Percent favourable ratings and interpolated medians for sets of instructor SET scores with the same mean score.

Scatter plot of mean scores versus PF showing a positive linear relationship. Three sample individual scores with low, medium, and high IM are labelled a, b, and c.

To further demonstrate the differentiating power of the IM and PF rating, Figure 2 shows raw score distribution for three exemplar sets of instructor SET scores with (a) the lowest, (b) intermediate, and (c) highest IM, selected from the set shown in Figure 1. While the set of raw scores for each individual yields a mean of 3.4, they display very different score distributions (reflected in the different IMs and PF ratings).

Figure 2. SET score distribution for the highest, middle, and lowest points (instructors) drawn from Figure 1 (mean of each = 3.4).

Sample histograms showing percent distributions of Likert scale score values for three different instructors. The distributions for each are different, but each has a mean value of 3.4.

Given that students tend to rate their instructors favourably more often than not (Nulty, 2008) and since the mean SET score achieved by each of these educators was 3.4, all three ratings would have been deemed “below average” in terms of teaching performance. However, the IMs of the SET score distributions were 3.2, 3.6, and 3.8, respectively; the corresponding PF ratings were 37%, 55%, and 76%, respectively, making it clear that two of these three instructors achieved majority favourable scores from respondents, and in one case more than three out of four students rated their instructor favourably. This example illustrates the power of the IM as a measure of central tendency to more appropriately capture and represent the real distribution of respondent perspectives.

4.2. Relationship between the interpolated median and percent favourable rating

Recall that in equation (1)(1) $\mathrm{I}\mathrm{M}=\mathrm{M}+\left({\mathrm{N}}_{\mathrm{g}}-{\mathrm{N}}_{\mathrm{s}}\right)/\left(2{\mathrm{N}}_{\mathrm{e}}\right)\mathrm{i}\mathrm{f}{\mathrm{N}}_{\mathrm{e}}\mathrm{i}\mathrm{s}\mathrm{n}\mathrm{o}\mathrm{n}-\mathrm{z}\mathrm{e}\mathrm{r}\mathrm{o}$(1) of the IM formula, IM is calculated by adjusting the median downwards or upwards by value that is <0.5. Thus, a median (50th percentile) of 3 or less, on a 5-point scale, would yield an IM of <3.5 and would necessarily be associated with a PF < 50%, whereas, on the favourable end of the scale, a median of 4 or more would yield an IM > 3.5 and would necessarily be associated with a PF > 50%.

Recall that equation (2)(2) $\mathrm{I}\mathrm{M}=\mathrm{M}\mathrm{i}\mathrm{f}{\mathrm{N}}_{\mathrm{e}}\mathrm{i}\mathrm{s}\mathrm{z}\mathrm{e}\mathrm{r}\mathrm{o}$(2) is used when there is an even number of data points or responses and no response is equal to the median. In this case, no interpolation is needed, and the median would be the midpoint between two observed scores. If the median falls in the favourable end of the scale (between scores of 4 and 5), PF will necessarily be >50%. On the other hand, if the median falls in the unfavourable end of the scale, PF will necessarily be <50%. If, however, the median is between favourable and unfavourable responses, e.g., between scores of 2 (disagree) and 4 (agree) …, etc., PF will be exactly 50%. Table 1 shows all-possible outcomes, for a balanced 5-point scale, where the median is the midpoint between a favourable and unfavourable response. The expected range of DI (equation 3(3) $D=\frac{\left\{{\sum }_{k=1}^{K-1}{F}_k\left(1-F_k\right)\hspace{0.17em}\right\}}{\frac{K-1}{4}},\hspace{0.17em}0\le D\le 1,\hspace{0.17em}$(3) ) is also shown. It could be seen in Table 1 that for IM < 3.5, a PF of exactly 50% is only possible at IM of 2.5 or 3 and will be associated with high dispersion. For example, when the median, and hence IM, is the midpoint between a score of 1 (strongly disagree) and a score 5 (strongly agree), the data will be split evenly between the two extreme responses, resulting in maximum DI, indicative of polarized rating.

Table 1. All possible outcomes (for a 5-point scale) where the median is the midpoint between a favourable and unfavourable response (IM equation 2) resulting in a PF of exactly 50%

Middle scores		Median/IM	Dispersion index (DI)^1
3	4	3.5	[0.25, 1.0)
3	5	4	[0.5, 1.0)
2	4	3	[0.5, 1.0)
2	5	3.5	[0.75, 1.0)
1	4	2.5	[0.75, 1.0)
1	5	3	1.0

¹Expected range of DI.

It follows then, that for a 5-point balanced scale, an IM of 3.5, with a PF of exactly 50%, is the pivot point in the IM/PF relationship, such that an IM < 3.5 is associated with PF not exceeding 50% and an IM > 3.5 is associated with a PF of at least 50%.

Similarly, for a 7-point balanced scale, and in the case of equation (1)(1) $\mathrm{I}\mathrm{M}=\mathrm{M}+\left({\mathrm{N}}_{\mathrm{g}}-{\mathrm{N}}_{\mathrm{s}}\right)/\left(2{\mathrm{N}}_{\mathrm{e}}\right)\mathrm{i}\mathrm{f}{\mathrm{N}}_{\mathrm{e}}\mathrm{i}\mathrm{s}\mathrm{n}\mathrm{o}\mathrm{n}-\mathrm{z}\mathrm{e}\mathrm{r}\mathrm{o}$(1) , a median of 4 or less would yield an IM of <4.5 and would necessarily be associated with a percent favourable <50%, whereas, on the favourable end of the scale, a median of 5 or more would yield an IM > 4.5 and would necessarily be associated with a PF exceeding 50%. Furthermore, in the case of equation 2(2) $\mathrm{I}\mathrm{M}=\mathrm{M}\mathrm{i}\mathrm{f}{\mathrm{N}}_{\mathrm{e}}\mathrm{i}\mathrm{s}\mathrm{z}\mathrm{e}\mathrm{r}\mathrm{o}$(2) , and for IM < 4.5, a PF of exactly 50% is only possible at IMs of 3, 3.5, or 4, with a maximum DI occurring at IM of 4, when it is the midpoint between the extreme scores (1 and 7).

It follows then, that for a balanced 7-point scale, an IM of 4.5, with a PF of exactly 50%, is the pivot point for the relationship between these two statistics.

This relationship underpins our submission that IM and PF rating, along with a suitable measure of dispersion (DI), provide a more meaningful approach to summarizing, interrogating, and presenting SET data from a balanced rating scale.

4.3. An illustration of the implementation of the proposed metrics

We used SET data from two academic years at UBC (16,982 surveys) to illustrate this interesting and previously unreported relationship between IM and PF rating.

A tabular representation of this relationship is shown in Table 2. This example shows SET data for the UMI that reads “The instructor helped inspire interest in learning the subject matter”. The table shows the distribution of surveys when clustered into five categories by IM and eight categories by DI (recall that DI represents a measure of “score variability per survey”; the higher the DI, the greater the level of “disagreement” between respondents). The number of surveys in each grouping is given per cell, and the average PF rating is given in parentheses.

Table 2. Relationship between the interpolated median and percent favourable rating

	Dispersion index
	0	<0.2	0.2–0.3	0.3–0.4	0.4–0.55	0.55–0.70	0.7–0.85	>0.85	Total
IM	Number of Evaluations (% Favourable Rating in Parenthesis)
5.0	1,055(100%)	1,756 (99%)	2,420 (97%)	1,694 (89%)	1,217 (81%)	237 (74%)	40 (70%)	12 (67%)	8,431
4.5	197(100%)	83 (100%)	443 (97%)	992 (86%)	2,030 (76%)	646 (71%)	93 (67%)	21 (61%)	4,505
4.0		26 (79%)	127 (60%)	246 (65%)	873 (62%)	887 (58%)	191 (55%)	16 (54%)	2,366
3.5	65(0%)	12 (23%)	54 (32%)	80 (34%)	297 (38%)	470 (41%)	133 (42%)	19 (43%)	1,130
3.0	15(0%) (0%)	5	28 (1%)	17 (10%)	111 (16%)	260 (24%)	94 (33%)	20 (35%)	550
									16,982

As would be expected, by and large the PF rating decreases as dispersion increases in the first three rows (IM of >3.5) but increases with dispersion in the lower two rows (IM < 3.5). Thus, surveys in the upper left cells of Table 2 have high IM values with low variability in student scores, resulting in higher PF ratings. Cells in the lower left-hand area of Table 2, on the other hand, show low IM values with low variability in student scores, resulting in low PF ratings. Furthermore, surveys in the bottom two rows, corresponding to an IM < 3.5, have PF ratings not exceeding 50%.

The relationship between these two statistics can be illustrated further by means of a simple scatter plot. Figure 3 offers a graphical depiction of the IM and PF ratings from the 16,982 surveys presented in Table 2.

Figure 3. Relationship between the interpolated median and percent favourable rating on a 5-point Likert scale.

Scatter plot of IM scores against PF for scores collected using a 5-point Likert scale. The relationship is positive and linear. Most instructor scores fall in the top right quadrant.

In Figure 3, data points in the upper right quadrant represent instructors whom at least half of the respondents rated favourably (PF rating ≥50%); data points in the lower left quadrant represent instructors whom at least half of SET survey respondents did not rate favourably (PF rating =<50%).

As shown above, for a 5-point balanced rating scale, no SET ratings with an IM below 3.5 would have a PF rating above 50% nor would ratings with an IM above 3.5 ever have PF ratings below 50%. Thus, no data points could possibly fall in the upper left or lower right quadrants of Figure 3. Recall that within the context of interpolation, a score of “Agree” (4), on a 5-point scale, is understood to hold a range of perceptions, from those bordering on “Neutral” (3) to those leaning towards “Strongly Agree” (5). In this context, the IM value at the pivot point (3.5) is the lower bound for an “Agree” response, thus it is the lower bound for favourable ratings. As an aside, this allows us to immediately understand that the instructors whose SET survey score distributions are shown in Figures 2(b,c) had PF ratings of >50%, based on their IMs of 3.6 and 3.8.

Another property of this relationship is that PF ratings approach 50%, with decreasing variance, as the IM approaches 3.5, from above or below.2. Furthermore, for IM > 3.5 (upper right quadrant of Figure 3), and for a given IM, PF rating decreases as dispersion increases (Figure 4). However, in the lower left quadrant (IM < 3.5), PF increases as dispersion increases.

Figure 4. Relationship between percent favourable rating and dispersion index for IM > 3.5, on a 5-point Likert scale.

The top right quadrant of Figure 3. Above IM of 3.5, percent favourable decreases as dispersion increases.

As discussed above, this relationship extends to situations where SET surveys use a balanced 7-point scale, with the lower bound for an “Agree” response (and thus, the pivot point) being 4.5. An example from the UBC data is shown in Figure 5.

Figure 5. Relationship between the interpolated median and percent favourable rating on a 7-point Likert scale.

Scatter plot of IM scores against PF for scores collected using a 7-point Likert scale. The relationship is positive and linear. Most instructor scores fall in the top right quadrant. Above IM of 3.5, percent favourable decreases as dispersion increases. Below IM of 3.5, percent favourable increases as dispersion increases.

5. Discussion

Some decades ago, McIntosh (1979) noted that the use of statistical information is one of the several barriers to the successful implementation of real insights from educational research in higher education, especially when implementation is in the hands of “non-numerate decision-makers” (p. 81). Educator and educational administrator data literacy continues to complicate effective use of educational data (Macfadyen et al., 2014).

We recognize that many educational leaders default to using means and standard deviations to analyse SET data, because these metrics are relatively simple and well understood. However, the use of parametric tests (based on the arithmetic mean and standard deviation) in student evaluations of teaching has been the subject of much criticism, because (as discussed above) students tend to rate their instruction experience favourably more often than not, resulting in the data being negatively skewed (not meeting distributional assumptions for parametric analysis). Given the centrality of SETs in contemporary academic management and quality assurance processes in higher education, we argue that it is now increasingly important to ensure the appropriateness and fairness of metrics used to summarize and report on instructor teaching evaluations.

By definition, the IM reflects the distribution of scores above and below the customary or standard median (50% percentile) and, as such, is a better measure of location for SET data than the mean.

Moreover, in this paper, we have demonstrated in theory, and illustrated through analysis of exemplar SET survey data from UBC, an interesting and previously unreported relationship between the interpolated median and percent favourable rating, defined as the proportion (percentage) of responses higher than neutral in the total set of SET survey responses received by an instructor, on a balanced rating scale.

As a general rule, for surveys employing a 5-point Likert scale, an IM of 3.5 corresponds to a PF rating of 50% and divides the evaluations into two distinct classes: above and below 50% favourable rating. In the neighbourhood of this pivot point (IM 3.5, PF 50%), the relationship is fairly linear. However, for higher or lower IM values (away from the pivot point), the DI provides a good insight into the variability of PF rating for a given range of IM (Figure 4). For example, it is evident from Figure 4 that an instructor survey with an IM as high as 4.7 (on a 5-point scale) could have favourable rating of only 60%, due to high dispersion. This confirms the insight gained by using IM, PF, and dispersion measure suitable for ordinal data.

6. Conclusion

Students have the capacity to offer critical feedback on their experiences of teaching and learning, helping to close the loop between educators and learners. We affirm that student evaluations can and should be implemented as part of an overall teaching evaluation system, which may also include regular peer review of teaching, educator self-assessment, and other forms of assessment and learner feedback. However, we and others (see, for example, Kitto et al., 2019) have made a clear case that parametric analysis of SET data is a poor choice and can result in inappropriate interpretations and judgments of teaching performance that may have lasting impact on educators’ professional lives. Parametric analyses, using the mean and standard deviation, can be appropriate for analysis of ordinal data, if certain distributional assumptions are met, but SET data do not meet such distributional assumptions. We acknowledge that implementing and explaining more complex statistical methods adds an additional burden to effective use of SET feedback. We assert, nonetheless, that by definition, and as we have demonstrated here, the interpolated median provides more and better information about the real distribution of student responses to evaluation questions, and as such is a better representation of the centre of the data.

We have also presented an interesting and previously unreported relationship between the interpolated median and percent favourable rating and elaborated on how data dispersion affects the relationship. Furthermore, we have demonstrated through the use of simple graphics how these three statistics can allow a more meaningful interrogation of SET data, offering better insight into student feedback. This simple approach is a significant improvement in process compared to the use of a single statistic (usually the mean).

Based on these findings, it is our recommendation that SET data be summarized, reported, and interpreted using the IM and PF rating, combined with a measure of dispersion suitable for ordinal data (such as DI).

Only through the use of appropriate metrics to represent the outcomes of student evaluation of teaching can we ensure that the real and nuanced distribution of student feedback on their different experiences of instruction is as complete and representative as possible, that interpretation of teaching evaluations is appropriate and fully informed, and that judgments made about the teaching effectiveness of academic staff are meaningful and fair.

Correction

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Abdel Azim Zumrawi

Abdel Azim Zumrawi is a statistician with the Planning and Institutional Research (PAIR) Office at the University of British Columbia. He has a diverse research background in applied statistics, including work on decision-support tools used in natural resources management. More recently, his work has focused exclusively on quantitative methods and measurements related to learning analytics and student evaluation of teaching.

Leah P. Macfadyen

Leah P. Macfadyen is an Assistant Professor of Teaching in the Faculty of Education at the University of British Columbia. Her work concerns teaching and learning in online learning environments, and she has contributed to the emerging field of learning analytics in the past decade, inspired by its potential of learning data to offer actionable insights into our teaching and learning contexts and practice.

Notes

1. https://explorance.com/products/blue/