Consideration of student evaluations of teaching (SET) and learning: perspectives of learning and teaching leaders through the ethical lens of “first, do no harm”

Abstract

This paper explores institutional leaders’ perceptions of learning and teaching involved in facilitating and assessing Student Evaluations of Teaching (SET) survey instruments across Australian regional universities. It focuses on how they understand the function of SET, strategies used to mitigate bias, and potential residual harm. Through adopting a combination of inductive and deductive research processes and a thematic analysis through the ethical lens of nonmaleficence (first, do no harm), we report that leaders in learning and teaching perceive SET as a form of surveillance and Quality Assurance ‘performance’, recognise inherent biases inhabited in SET reports, and identify how these biases negatively impact academics through a lack of systematic harm mitigation strategies. The paper’s critical – and novel – contributions include an increased understanding of how SET inflicts harm towards women and other marginalised academic groups through systematic and authorised microaggressions and how SET contravenes universities’ duty of care to employees. It recommends an expansion of the principle of nonmaleficence beyond potential harm to research subjects, including those who undertake research or evaluation (such as academics), particularly if these impact them.

Keywords:

• Student evaluation of teaching
• academics
• mental health
• wellness
• gender bias
• emotional trauma
• psychological harm
• nonmaleficence

Introduction

Student Evaluations of Teaching (SET) is the most widely used assessment instrument for collecting and interpreting higher education student feedback to inform course and teaching improvement, staff tenure and promotion, and recruitment processes. University administrators use SET in higher education to make academic retention, pay increases, tenure, and promotion decisions. However, like all survey instruments, SET ‘are not impervious to subjective factors and biases (both explicit and implicit) that students may possess’ (Cone et al. 2022, 1085) and which articulate into evaluative responses. Students may hold socially constructed assumptions about what constitutes effective teaching and maintain certain gendered stereotypes of male and female faculty (Khazan et al. 2019). Consequently, student perception of the course and teaching quality may be influenced by the gender (and/or other personal or cultural features) of the academic convening the course or unit. It is important to pause here and note that we understand male and female are biological terms that are imbued with socially constructed connotation. We try not to use ‘female’ as it implies that all females are women, and all women are female. However, we lean on scholarship and survey instruments that often use these terms. We recognise the complexity of gendered terminology and positionalities and hope that our work supports people who also live outside of gendered binaries.

When students’ expectations remain unfulfilled, they may respond with critical and, at times, hostile reactions, which place some teachers at a disadvantage by influencing perceptions of their teaching effectiveness (Cunningham et al. 2023; Heffernan 2022a, 2022b, 2023). Yet, despite the increasing recognition of the volume and impact of biased and, at times, abusive feedback in SET surveys, minimal research pays attention to the emotional trauma and psychological harm they pose to academic staff (Cunningham et al. 2023; Heffernan 2023), or which considers the ethical implications of SET (McCormack 2005), or more specifically the ethical implications of the impacts and implications of administering SET more generally.

This paper addresses these research gaps by exploring the perceptions and experiences of leaders of learning and teaching involved in facilitating and assessing SET assessment instruments across Australian regional universities, focusing mainly on how they understand the function of SET and strategies to mitigate bias and any residual harm it may cause. We contextualise the study by first offering an overview of scholarship concerned with the recent history of SET, its main functionality, and how it has been received/perceived by academics. We also consider literature that explores biases made manifest within student responses and the effect of stereotyping. The paper then provides an overview of the research methodology employed in this study, which combines inductive and deductive analytical approaches overlayed with the principle of nonmaleficence (first, do no harm), before presenting a summary and thematic analysis of the research findings.

The critical contribution of this research includes an increased understanding of how SET operates as a surveillance mechanism that inflicts harm towards women and other marginalised academic groups – such as those from non-white racial groups, culturally and linguistically diverse backgrounds, older academics, and academics with diverse genders and sexualities – through systematic and authorised microaggressions. We also consider the ethical implications of administering SET and universities’ duty of care to its employees. The paper recommends an expansion of the principle of nonmaleficence beyond potential harm to research subjects to include the people undertaking research – particularly if the research or evaluation directly impacts them.

Literature review

Recent history of SET

In the 1970s, the SET instrument became the primary method for formative assessment to improve the quality of courses and measure the capability/competence of teachers and teaching practice (Kayas, Assimakopoulos, and Hines 2022). Over the years, SET has evolved into a summative evaluation tool, the primary performance management tool, and an indicator for the promotion and tenure of academic staff (see Heffernan 2018). Estimates suggest that over 16,000 higher education institutions’ teaching staff collect SET results at the end of each teaching period (Cunningham-Nelson, Baktashmotlagh, and Boles 2019). In many cases, SET is used as the sole performance indicator of teaching competence (Spooren, Brockx, and Mortelmans 2013).

Adopting the SET tool became popular because of the potential perceived benefits for stakeholders – university administrators/managers, students, and academics. SET provides managers with insights to validate new programs; provides academics/instructors with feedback to enhance courses and improve individual teaching practice, leading to promotion and tenure (Johnson 2000); empowers students with a source of information to aid their choice of where and what to study (Collini 2012), and a voice for their student experience. However, Heffernan (2018) raised concerns over six years ago about the current use of SET, including critiques of how student biases are embedded within SET responses.

Gender bias and the effect of stereotyping

One of the critical issues emerging within the literature is the presence and impact of gender bias on SET results. According to social psychological theory, gender biases in SET assessment may occur because of a lack of fit between gender stereotypes and individuals’ professional roles, leading to negative evaluations (Kwok and Potter 2022). This position supports Khazan et al. (2019) investigation into the relationship between bias, prejudice, and gender in student evaluations of an extensive, asynchronous online course. The research argues women faculty in male-dominated disciplines ‘face prejudice like that experienced by women leaders, resulting in similar consequences. The incongruity of these perceived roles can result in fewer women entering male-dominated disciplines, and women who do enter these disciplines receive poorer performance evaluations’ (Khazan et al. 2019, 423).

As far back as the 1980s, research indicated the presence and impact of gender bias in student evaluations of teaching responses (Bennett 1982) – a trend that continues. A more contemporary examination of the literature on bias in student evaluations by Kreitzer and Sweet-Cushman (2021) produced a novel dataset of over 100 articles, providing a nuanced review of this broad but established literature. The review indicated researchers and scholars, using different data and methodologies, routinely identify that women faculty and those from other marginalised groups face significant biases and are thus disadvantaged in SET responses.

Student comments, language, the volume, and impact of abusive and malicious responses

Mitchell and Martin (2018) identified students appropriated different language to evaluate men’s and women’s academic performance in their qualitative comments. They applied more subjective criteria when considering women than their male counterparts. Students frequently commented about women academics’ personalities and appearance, referring to them as ‘teachers’. In contrast, other comments referred to male instructors as ‘professors’ and focused on their professional objectives, such as intelligence (Mitchell and Martin 2018).

Other studies are concerned with the use of negative student comments in SET assessment by highlighting the use of offensive, unprofessional, abusive, and malicious language, exhibiting gender bias, homophobic prejudice, sexism, and racism in Australian universities (see Cunningham et al. 2023; Heffernan 2022a, 2022b, 2023). Moreover, Heffernan (2022b) found that student evaluations ‘openly prejudiced against the sector’s most underrepresented academics … contribute to further marginalising the same groups universities declare to protect and value and aim to increase their workforces’ (199).

Heffernan’s (2023) ongoing research on the volume, type, and impact of anonymous student comments on 674 academics highlights the amount and type of abusive comments academics receive. The research outlines how women and marginalised groups received the highest volume, most derogatory, and most threatening abuse. Equally troubling is that the research underestimates the magnitude and severity of abusive comments academics receive. Related research by Cunningham et al. (2023) documented a machine-learning approach to screening 100,000 student comments in one institution in 2021. This project prevented 100 abusive comments (within one institution only) from reaching academic staff members. While some might dismiss the small percentage of unacceptable comments made by students as minimal and thus insignificant (statistically), Cunningham et al. (2023) recommend capturing inappropriate comments to ensure students know when their comments are unacceptable to the university.

The growing body of research focusing on using SET is welcomed. Yet, there is a lack of understanding of how the leaders of learning and teaching within universities perceive SET and mitigate any biases inherent within survey responses. Accordingly, negligible research exists on the impact of SET on the mental health and wellness of academic staff about whom SET responses are based.

Research methodology

We explore perceptions of academics and learning and teaching leaders who facilitate or lead SET in regional universities, considered through the lens of ethical practice, particularly the principle of ‘first, do no harm’. While healthcare practices and research often employ nonmaleficence, they rarely engage in educational research (Buchanan and Warwick 2021). Additionally, when applied to the planning and evaluating of a research project within the Human Ethics application processes, consideration is usually limited to the wellbeing of research participants without concern for the people (or the work of people) who are the focus of research or evaluation. We thus expand the frame of reference to include a consideration of ‘first, do no harm’ to the people who undertake evaluations or research (particularly teachers through the SET process) and to whom research, or evaluation reports are presented. This novel methodological approach is informed by the scholarship summarised above, which suggests SET practices inhabit biases that can harm academics, particularly women academics and those from other marginalised groups.

The methodology incorporates elements of both inductive and deductive research processes. Our initial literature review provides a map of the central discourses within and around SET, which sensitised us to the possibility that participants may perceive SET as biased and unfair. Yet, while we were aware of current discourses, we attempted to review the research data without prejudice by adopting a deductive approach.

Methods

Survey

Upon obtaining ethical clearance (Ethics Number: de-identified) between November 2022 and June 2023, the research team invited directors and senior staff of learning and teaching units across the Regional Universities Network to complete and distribute an online survey amongst their relevant staff. Regional universities in Australia are typically located in non-metropolitan areas, often serving cities and towns that are distant from the major urban centres. These locations are typically categorized as regional, rural, or remote based on various classification systems such as the Australian Bureau of Statistics’ (ABS) remoteness areas. Regional Universities Network/RUN is a network of universities primarily from regional Australia. Open questions were designed to garner participants’ understanding of the role of SET, the relationship between student evaluation and teaching quality, and if and how they undertake measures to optimise the completion. of SET and mitigate bias in the SET process. The survey was designed to garner responses that would be thematically analysed.

Thematic analysis

Thematic analysis is a versatile qualitative research method to explore and analyse relevant patterns, meanings, and themes within textual/interview data rooted in various theoretical frameworks (Braun and Clarke 2023). The approach allows researchers to strategically explore and understand the lived experiences, perspectives, and meanings attributed by participants to those experiences (see Braun and Clarke 2023).

Results and findings

Survey outcomes

A total of 55 leaders in learning and teaching completed the survey. The results are tabled below (Table 1).

Table 1. Leaders in learning and teaching responses.

	Number
Deputy Vice-Chancellor Academic	1
Dean Academic (or equivalent)	1
Chair of learning and teaching committee	2
Director or Manager of a learning and teaching unit	3
Academic Developer	2
A Senior Executive role (not identified above) responsible for student engagement or learning and teaching across an institution	1
Dean, Deputy Dean, Head of School, or Deputy Head of School (note, all learning and teaching)	8
Discipline Lead (overseeing a series of programs within a Discipline)	4
Program Coordinator (understood in this context as a sequence of study customarily awarded with a qualification such as a degree, diploma, or certificate).	18
Principal or Senior Advance HEA Fellow	14
Delegate from Human Resources or People and Culture	1
Total	55

Note on terms: Deputy Vice Chancellor Academic, commonly abbreviated as DVCA, is a senior executive position within a university, responsible for shaping the educational strategies, policies, and quality assurance processes of the university. Administered by Advance HE, HEA Fellowships are awarded to learning and teaching professionals who can demonstrate they meet The UK Professional Standards Framework (UKPSF).

From the data, 12 survey respondents identified as men and 42 as women, with the remainder preferring not to disclose. A total of 46 respondents identified as white, five as non-specified other racial identity, one as Asian, and one as an Aboriginal or Torres Strait Islander. Of the respondents, 40 identified they did not have a disability or long-term condition, 13 identified that they did, and four preferred not to disclose. To ensure participant anonymity, we have not linked these demographic features to the positions or roles held by the participants.

The number of respondents did not allow for statistically significant quantitative findings or generalisable claims to be made. Instead, we present the demographic breakdown of participants to contextualise the following overview of the qualitative data from the survey instrument. As the richest data was provided through qualitative comments, we focus this paper on the thematic analysis of the qualitative responses provided within the survey instrument.

Thematic analysis

The thematic analysis process we undertook, post-initial read and identification of broad themes, was layered by a lens of nonmaleficence. We maintained a concern for how leaders of learning and teaching perceive the function of SET, potential risks associated with SET and harm mitigation strategy. These were not used as hard and fast categories but were considerations that coloured our close reading of the results.

Dominant themes identified in the survey responses include how leaders in learning and teaching perceive SET to operate as a form of surveillance and Quality Assurance ‘performance’. Respondents also recognised inherent biases in SET reports, detailing how these biases impact academics’ mental and emotional health and result in career impacts. Despite some individualised risk mitigation strategies, it was perceived that the systematic harm mitigation strategy was not consistently employed and that the current use of SET contravenes universities’ ‘duty of care’ to their employees.

Surveillance

Through a thematic analysis of open-text survey responses, respondents perceived SET predominantly as a form of managerial surveillance and subsequently as a ‘punitive measure’ to rank and punish academics considered underperforming. Survey responders specifically described the deployment of SET as ‘a surveillance mechanism’ or ‘blunt instrument’ used to ‘monitor of staff by management’ [with] potential for the data to be used as a ‘big stick’’. These themes were repeated across several responses (n = 15), which included descriptions of SET as employed ‘to reprimand faculty for bad teaching’ and ‘used against academics in performance reviews or promotion applications.’ Respondents also indicated that despite SET responses generally being too low to represent the student body, management used the results to assess academics’ learning and teaching quality with potential impacts on their employment and career success. Relatedly, managerialism was similarly inherent in the second theme of the results – Quality Assurance performance.

Performativity of quality assurance

Several respondents (n = 6) indicated they perceived SET as a ‘compliance measure’, while others perceived SET as a disingenuous approach to the quality assurance process: ‘[SET is] required to demonstrate accountability’, so ‘universities can be seen to be looking after and valuing the paying customer’ and a ‘performance management mechanism rather than as a continuous improvement mechanism’.

Of note is that the results did not provide significant commentary relating to the benefits of SET to the quality assurance of learning and teaching; instead, they focused on perceived performativity.

In the interest of nonmaleficence, please note the following section of survey results contains references to various biases communicated via SET and to the psycho-emotional harms they can create. To protect yourself from vicarious trauma, you may wish to bypass this section of text and move to the ‘Discussion’ section below.

Inherent bias

The significant impact of perceived biases embedded within SET responses was explained in terms of a small sample size of responses and student anonymity inherent in the survey design. Examples include:

‘They can be biased because only a few students complete them’; ‘A small, biased sample of students complete the SET. This sample is not representative of the population’; ‘The feedback from such a small, unrepresentative sample should not be used to determine the strengths or limitations of a course’; ‘The fact that it is anonymous means that students can say what they like and get away with things they could never say if they were identifiable and accountable’; ‘[biases are] fostered through the anonymity of the instrument’.

Common biases perceived in SET responses were based on the teacher’s gender, race, disability, cultural and linguistic background, and age. Gendered bias was the most prevalent form perceived:

Students use these as opportunities to discriminate and speak rudely and derogatory about the staff - especially women staff and bias against women with other marginalised status: ‘Student bias - especially against women and people of colour’; ‘If you are female, international, Indigenous, old, etc., etc. then chances are your SET scores will be lower’; ‘I have seen comments which speak negatively about academic staff based on race, gender and perceived sexual identity.’

Respondents indicated academics with disability were also the victims of bias within SET surveys:

‘I have personally received comments that my disability made me “incompetent”’; that culturally and linguistically diverse academics were evaluated less favourably than white, western counterparts: ‘Other colleagues from cultural and linguistically diverse backgrounds regularly receive negatively biased data and discriminatory comments… comments that they were less intelligent than other staff… about their accents and needing to take ‘speech therapy’, and that age discrimination informed some SET responses: ‘I have experienced age-related bias on a few occasions’.

Finally, the lack of student awareness of their biases was also raised as a flaw within the SET process: ‘Students do not know (necessarily) that they have a bias’.

Impacts on mental health

When asked what impact SET might have on course/unit coordinators, the most common response related to adverse effects on mental health (n = 29), including comments that:

‘it ends up being harmful to staff well-being’, ‘it is a major stressor for academic staff that undermines their wellbeing’, ‘staff have been psychologically affected by student comments’, and SET are ‘psychologically harmful’.

More specific respondents perceived negative mental health impacts, leading to:

‘academics feeling helpless and disheartened’, ‘paranoia’, ‘feeling sick’, ‘makes you feel inadequate’, ‘undue stress and emotional trauma’, and ‘distress, rumination, anxiety, and worry that the evaluation will impact future opportunities’.

Similarly, highly emotive language relating to significant psycho-emotional harm and trauma was used to describe the impact of SET:

‘can be devastating to a teacher’, ‘The impact on the teacher can be catastrophic’ …, ‘Trauma to the Course Convenor/teaching staff’, ‘It severely damages staff mental health’.

Given the damaging impact of the common use of SET, respondents queried its use on ethical grounds, including whether the use of SET transgresses an institution’s duty of care to its employees.

Career impacts

In addition to the harm SET comments can inflict upon academics, respondents indicated that low SET scores might negatively impact opportunities for tenure, academic promotion, and academic self-efficacy. Four respondents identified that harmful SET comments prompted academics to leave the profession. For example:

‘A professor I had as an undergrad who was amazing and so supportive … has quit academia because she was sick of the negative comments in SET… such a loss to the discipline’; ‘two older and very good lecturers that I know have left [the profession] because of the unwarranted abuse via SET’; and a ‘female academic is now leaving’.

All four academics who resigned from, or are in the process, were described as women.

Risk mitigation

The final theme emerging from thematic data analysis focused on the university strategy employed to mitigate the risk of SET associated harm. While most respondents could not identify a harm mitigation strategy, when described, it was often presented as haphazard and individually facilitated rather than systematically employed. A total of 14 leaders in learning and teaching described filtering:

‘out the insults before showing the feedback to colleagues’, or ‘review[ing] data, remov[ing] comments that may be discriminatory’, or scanning ‘for expletives and other ‘flagged’ phrases’, and ‘modifying’ inappropriate comments’.

However, one respondent identified an inconsistent approach ‘the Head of School … is required to review the feedback before it can be released to staff’. Yet, a different respondent identified she was unaware of a process for mitigating bias or harm,

‘None that I am aware of. Sporadically, we have been asked to nominate the SET data/responses for individual courses that we feel the Head of School should review before releasing them to the academic responsible… But this is patchy and does not always happen’.

SET contravene universities’ duty of care

Ten survey responses reflected that the implementation of SET transgresses universities’ ethical due diligence and harm mitigation, including descriptions of SET as:

‘bordering on bullying’, a ‘workplace psychosocial hazard’, and that ‘if SET were given a risk assessment or had to pass ethics, it would fail the do no harm test because some students use it as a means to personally harass academic staff through vindictive scoring and comments when they have done poorly in the course’.

Other responses identified that it offered:

‘no psychological safety’, advising that there ‘should have been a trigger warning or a help number’ and that the release of SET results require care-full scheduling and support, A BIG [sic] problem last year was that SET report was released to staff on a Friday at 6:00 pm.

This situation meant that many staff were home and read their feedback without the support. Many reported feelings of being ‘crushed’ by some of the comments received. Finally, some comments suggested that SET processes were abusive. For example, SET was described as ‘a domestic violence relationship’. The impact of SET on academics’ careers was not only discussed (as above) concerning a perceived sense of surveillance but also identified that SET outcomes resulted in some academics choosing to leave the profession.

Discussion

The role of SET as a means of top-down surveillance, with significant implications for academics’ career opportunities and navigation, emerged as a central theme within the data. This concern aligns with Kayas’ et al. (2022) research, which found that top-down vertical surveillance, imbued with disciplinary procedures, involved university managers scrutinising academics through SET surveys. They further propose that management utilise the panopticon of power ‘to capture the knowledge of employee performance under a totalising instrumental rationalism that marginalises dissent and resistance’ (Kayas, Assimakopoulos, and Hines 2022, 2). This theorising is salient to our study, as external surveillance, when inhabited, can transform into internal surveillance, where people come to self-regulate themselves in response to perceived scrutiny and judgement by others and accept individual responsibility and self-remonstration even for systematic or external failings (Selwyn and Grant 2019). Evidence of such self-surveillance in several responses was made manifest in reports of individualised mental health impacts, with some academics choosing to leave the profession rather than withstand the further personal and emotional assault. In comparison, others took personal responsibility for mitigating the harm (to others) created by SET (by removing biases or comments that might cause harm). Yet, when systematic protections are not generally identified, self-surveillance and individualising harm mitigation processes might inadvertently invisibilise and propagate the harmful SET process.

Surveillance is often co-opted as using power to regulate, define, and control people to maintain the existing social order (Nakamura 2015). Yet, maintenance of the status quo is ethically problematic when the current order (and structure) of higher education is highly gendered, raced, and ableist. It appears, therefore, that SET, as a means of surveillance, serves to reproduce rather than diversify the structures and the actors of higher education. For example, structural and cultural factors negatively impact academic women’s career progression across all nations and disciplines, race significantly affects academic careers (Tsouroufli 2023), and people with disabilities experience persistent ableism, social exclusion, non-accommodating environments, and a lack of opportunities, leading to their significant under-representation within faculty (Lindsay and Fuentes 2022). Therefore, SET, with an inherent surveillance functionality (that seeks to maintain the existing social order of the institution), undermines universities’ cl/aims to support diversity and inclusion.

Furthermore, academic practice is not simply institutionally managed but increasingly state-mandated and controlled. Additionally, Lloyd and Wright-Brough (2023) indicate the university/higher education sector employs SET, as evidenced in national and international awards within and beyond individual institutions, where comparisons are frequently made between institutions. It is perhaps unsurprising within this context that several research participants noted how SET operated as a performativity of Quality Assurance. Key phrases, such as ‘seen to be looking after and valuing the paying customer’ and ‘a compliance measure’, reflect the cynical perspective that the purpose of SET is ‘institutional peacocking’ (Yarrow and Johnston 2022) rather than ensuring quality learning and teaching.

Similarly, Sachs (1994, 22) posited that notions of quality in higher education have dichotomised into ‘two distinct categories: quality assurance (QA) and quality improvement (QI)’ where ‘a tension has arisen … between quality as a measure for accountability and quality as a means for transformation and improvement’ (Rowlands 2012, 99). This discourse raises concerns about whether SET is employed to fulfil a public-facing governance agenda or functions to assess and actively improve learning and teaching. The concern is of significant ethical value as the academy must question whether SET's potential risks outweigh perceived benefits. Thus, higher education institutions must identify and actively mitigate all identified potential risks and harms, even if this means ceasing the administration of traditional SETs.

The findings also relate to the harmful role of SET that appears to operationalise microaggressions. Initially coined by Pierce et al. (1977) to describe derogatory representations of Black Americans, the concept of microaggression has since expanded to encompass other marginalised groups, including women (Sue 2010). Microaggressions are subtle, commonplace, verbal and non-verbal actions that communicate negative messages toward a targeted group or person (Sue 2010). According to Sue (2010), microaggressions can be categorised into microassaults, microinsults, and microinvalidations. Microassaults are categorised as explicit, conscious discrimination, which includes name-calling; microinsults exclude, negate, or nullify the thoughts or experiential reality of a person using subtle but persistent demeaning and disrespectful comments and behaviours, reflecting unconscious bias (Sue 2010) and microinvalidation occurs when participants’ experience or expertise is not acknowledged (Sue 2010).

SET can function to insult women academics and/or Black academics (of Colour) and/or those with a disability or who are older. It does so by having their learning and teaching design and facilitation deemed less valuable and legitimate in the academy than learning and teaching designed and delivered by white, male, able-bodied, and younger academics. The SET processes, which provide no recourse or opportunity for academics to respond to SET comments, essentially render marginalised academics without voice or contribution function as microinsults. In addition, denying an academic voice and ownership of their learning and teaching practice invalidates them by authorising academics to be spoken of and about but not with, thereby constituting microinvalidations.

Moreover, when evaluation outcomes impact marginalised academics’ career opportunities, SET-related microinvalidation is a matter of epistemic justice because when marginalised and minority peoples are excluded from or considered lesser in the academy, ‘the research questions they would raise are not asked, and the corresponding research is not undertaken’ (West 2006, 5). The silencing of marginalised academics within the SET process creates barriers to individuals’ career progress and might limit the development of new knowledge and ways of knowing. Therefore, the risk of harm from traditional SET processes is personal, epistemic, ubiquitous, and manifold.

Within this context, we posit that ethical considerations, using the lens of nonmaleficence (first, do no harm), are required when administering SET processes. We recognise a lack of consensus when describing what constitutes harm and anticipating risks, with harm usually referring to physically, morally, socially, or mentally injuring a person. The British Psychological Society (2014) defines risk as the potential physical or psychological harm, discomfort, or stress to human participants, including risk to someone’s social status, values, beliefs, and relationships.

More specifically, the potential emotional, psychological, economic, and social risks and harms that SET processes inflict upon women and academics from other marginalised groups have been known for over 20 years, and as Ray (2018) observes, ‘Usually when social scientists discover that a research instrument is biased, it is discontinued in favor of a more effective mechanism [sic]’ (139). Moreover, Ross (2023) warns that although aggrieved academics could not take legal action against students for discriminatory comments, they could act against their employers for allowing them to affect their careers. Related to this, student scores or the numerical value attributed to academics and their courses (provided by the students who make abusive, personal, and biased evaluative comments, which are often removed) are not removed along with the comments. Yet, tenure and promotion decisions are usually made based on these scores. The (ad hoc) removal of abusive/biased comments might protect the academic from immediate harm. However, the process fails to mitigate the impact of the biased perception of students on academics’ career standing and opportunities. The removal of comments might, therefore, serve to protect the institution from litigation more than protecting academics from bearing witness to offensive comments that have the potential to harm them psychologically or emotionally.

It is seemingly paradoxical that SET continues to be administered by central university learning and teaching units at the behest and/or explicit permission of senior university executives, especially when the leaders of learning and teaching we surveyed both identified known concerns with SET evaluation instruments and processes employed and deployed strategy to mediate some of their potential harm. We provide below, therefore, a brief theorisation on the complex context and position of learning and teaching leadership within universities before recommending a framework that they, and all those involved in developing or administering SET, can use to adopt an ethical approach to assessment and evaluation of learning and teaching.

Universities operate within an era of neoliberalism, which is inextricably linked to resource constraints and free market principles, with a focus on measurement and reporting. Financial resource constraints, for example, are reflected in Australia, where there was growth in student enrolment from 138,666 in 1985 to 1,192,657 in 2011, followed by a decline of 28% in public funding per student within higher education between 1995 and 2005 (Rea 2012). Additionally, while there is no consensus on what constitutes learning and teaching excellence or how it can be facilitated or measured, most international higher education institutions must assess and report on student outcomes and experience (Chalmers and Tucker 2018). In Australia, for example, the Tertiary Education Quality and Standards Agency (TEQSA) evaluates higher education institutions against the Higher Education Standards Framework (HESF) to ensure all students have opportunities to provide feedback on their educational experiences; that all teachers and supervisors have opportunities to review feedback on their teaching and are supported in enhancing these activities; and that the results of regular student feedback are used to mitigate future risks to the quality of the education (TEQSA (Tertiary Education Quality and Standards Agency) 2021). Thus, within financial constraints, the requirement to meet an ­institution’s academic obligations SET is often employed because of its advantage of speed (in administration), anonymity and/or confidentiality (of response), and standardisation (for purposes of comparison between cohorts) to fulfil the institution’s reporting requirements (De-identified 2024).

Tensions for leaders of learning and teaching

Notwithstanding external imperatives that result in the continued deployment of SET, many leaders of learning and teaching experience tension between what they are required to administer and to support their colleagues. As Maistry (2016) noted, academic leadership tensions occur between senior management and the academics who carry out the day-to-day work. This space ‘straddles the supervisor/foreman-like role of operationalising university policy and engaging “upward”’ with university structures (515). They continue that the tension becomes pronounced when there is a disjuncture between the belief systems of academic leaders and the ideology of the policies they must implement (Maistry 2016). In response, those required to operationalise strategies determined by senior executives are commonly referred to as ‘umbrella carriers’, a role that seeks to ‘protect colleagues from seemingly unnecessary and/or damaging initiatives and information from top management’ (Gjerde and Alvesson 2020, 124).

Similarly, Sims (2003) uses the term ‘sandwiched middle’ to describe how leaders in higher education institutions are vulnerable to attacks from both above and below. Finally, Gjerde and Alvesson (2020) found that the umbrella carrier regularly fends off the ‘shit’ coming from top management to protect the academics from being bombarded with excrement (132). Thus, a more productive understanding of the role of middle management in higher education institutions requires abandoning the managerialism/collegiate duality in favour of a dialectic where managerialism co-exists in tension with collegiality (Clegg and McAuley 2005).

The leaders’ position as ‘umbrella carriers’ (Gjerde and Alvesson 2020) appears to align with the survey data. The respondents identified their concern for SETs capacity to inflict harm, their ad hoc removal of abusive/biased comments, and recommendations of support provisions for academics, including when and how SET results are shared, are congruent with the analogy protecting academics from ‘being bombarded with excrement’ (Gjerde and Alvesson 2020, 132). It thus appears that the leaders required to administer and review set also seek to protect academics from potential harm caused by SET.

A proposed strategy to enhance SET processes

Within these ethical and potentially litigious contexts, and to provide a framework for leaders of learning and teaching to employ systematically, we recommend that universities radically reassess the use of SET beyond minor and ad hoc (and individualised) changes. Yet, this reassessment should not be limited to SET practice, as nonmaleficence is a fundamental principle upon which all processes and interactions should be based. To support such an evaluation, we have developed a decision-making tool (see Figure 1) to gauge the efficacy of all instruments and processes and assess the quality of learning, teaching design, and facilitation.

Figure 1. Decision-making Tool for evaluating learning and teaching in higher education.

We posit that the decision-making tool could either dissuade universities from engaging in SET (requiring them to employ other processes or tools) or encourage them to use the SET instrument/process cautiously. This approach would mean applying the principle of ‘first, do no harm’ (nonmaleficence) to all aspects of the evaluative process, including recruitment, data gathering, data analysis, dissemination of outcomes, and recommendations.

Conclusion

SET processes aim to improve course quality, but research reveals bias and abuse in feedback. Despite this, the ethical implications of their emotional, psychological, and social impacts are understudied. In response, this paper reported on a research project which employed the ethical framework of nonmaleficence (‘first, do no harm’) to learn how leaders in learning and teaching perceive the function of SET and strategies to mitigate inherent biases within SET data.

The findings revealed learning and teaching leaders recognise SET's surveillance role, the presence of biases (gender, racial, cultural, ableism, ageism), and the emotional and career impacts on academics. Mitigation strategies were ad-hoc, failing to systematically address biases, causing significant harm, particularly to marginalised groups, resulting in stress, anxiety, and trauma. Participants explicitly noted that SET ‘can be devastating to a teacher’. While these findings confirm those from previous studies, new findings emerged within this study, signalling that implementing SET transgresses universities’ ethical due diligence and duty of care to employees.

Analysis showed that SET-induced external surveillance transforms into self-surveillance, causing mental health impacts and some academics to leave. Individual harm mitigation hides systemic issues, perpetuating harmful SET processes. SET often operationalises microaggressions, undermining universities’ diversity and inclusion claims. In response, we recommend universities radically reassess SET use by employing nonmaleficence as a fundamental principle upon which all university assessment processes and interactions ought to be based. We devised a decision-making tool for evaluating learning and teaching in higher education (Figure 1) to support academics in gauging the efficacy of all instruments and processes to assess the quality of learning, teaching design, and facilitation and to support the application of the principle of ‘first, do no harm’ (nonmaleficence) to all aspects of the evaluative process. Traditional ethics often prioritise participant wellbeing over the subjects of research or evaluation. The decision-making tool addresses this by incorporating concern for the subjects, including teaching academics evaluated through SET. Using this tool could transform SET administration and its significance in universities.

Future research

While not all student feedback is abusive, it is essential to note online trolling, cyber hate, and harassment, what Jane (2006) calls ‘e-bile’, are on the rise, commonplace and almost expected in some other areas of mediated networked society. Thus, in keeping with the Zeitgeist, higher education institutions must remain cognisant of the severity of these issues that particularly frequently target marginalised groups. Accordingly, higher education institutions need to take this issue seriously and provide greater protection to keep up with this phenomenon that has bled into many aspects of everyday life. Thus, investigating any increase in online and anonymous abuse in SET and its relation to other types of cyberbullying – and any associated e-safety practices and protection of employees in different industries and sectors – is worthy of future research consideration.

Acknowledgements

The authors would like to acknowledge the following co-researchers for their commitment and contributions to the project: Dr Melissa Sullivan, University of the Sunshine Coast; Professor Graham K. Brown, Charles Sturt University; Professor Janelle Wheat, Charles Sturt University; Associate Professor David Smith, Charles Sturt University; Dr Carol Quadrelli, University of Southern Queensland; Professor Kate Ames, Central Queensland University; Professor Steven Warburton, University of Newcastle; Professor Ruth Greenaway, Southern Cross University.

Disclosure statement

No potential conflict of interest was reported by the author(s).