Are we answering the question that has been set? Exploring the gap between research and practice around examinations in higher education

ABSTRACT

Despite a large amount of critical research literature, traditional examinations continue to be widely used in higher education. This article reviews recent literature in order to assess the role played by the approaches adopted by researchers in the gap between research on exams, and the way exams are used. Viviane Robinson’s ‘problem-based methodology’ focuses on the need for researchers to engage with the challenges and priorities of practitioners. Drawing on Robinson’s approach, the article investigates how the strengths and weaknesses of exams and their alternatives are framed in the literature published between 2016 and 2021. The article concludes that there is an absence of evidence about how and why practitioners make decisions about assessment. This hinders the ability of assessment researchers to appropriately connect their work with the assessment challenges practitioners face. To make a difference, assessment research needs to live in the real world; a world which, at least as far as practitioners’ assessment decisions are concerned, we do not yet sufficiently understand.

KEYWORDS:

• Assessment
• examinations
• methodology
• educational research
• practice
• research impact

1. Introduction

Assessment is an aspect of higher education where the gap between research and practice is often believed to be particularly wide. There is a sense that while the historical shift in focus from what teachers do to what students do (Barr and Tagg 1995) has had considerable impact on pedagogy – whether that’s problem-based learning (Savin-Baden 2000) or flipped classrooms (O'Flaherty and Phillips 2015) – the hold of traditional transmission-focused approaches has lingered longer when it comes to assessment and feedback (Winstone and Carless 2020). It is common to see researchers lament the lack of innovation in assessment, and the use of outmoded and unhelpful ways of assessing students and providing feedback. In 1995, David Boud said ‘[t]here is probably more bad practice and ignorance of significant issues in the area of assessment than in any other aspect of higher education’ (Boud 1995, 35). Similar things are still said today. For example, relatively low scores on student survey items on assessment are cited – perhaps questionably (Buckley 2021a) – as evidence that assessment faces something of a crisis: ‘In contrast to the effective feedback possibilities expounded in the literature, students generally report in surveys that feedback is done poorly in higher education, compared with other aspects of their studies’ (Dawson et al. 2019, 25).

Formal examinations probably represent the epitome of this perceived disconnect between assessment research and the assessment methods that are actually used within universities. Chong and McArthur (2023) say that ‘traditional examinations remain commonplace in Western higher education, despite the scholarship on assessment for learning and the work of educational developers in this field’ (5). For anyone in doubt about the status of exams as the traditional bogeyman for those working in the field of assessment, the concerns stretch back a very long way indeed. In 1967, R. Cox (1967) could say ‘[i]t is now thirty years since serious doubts were raised about examinations, yet despite the fact that there has been no serious shortage of critics since then, very little has changed’ (352, quoted in Elton 2004, 43). If we want to find out why assessment research generally fails to have the impact on practice that we would like, then examinations seem like a good place to look. That isn’t necessarily to agree with the body of literature criticising the use of exams; the value of exams is a complex topic outside the scope of this paper. This paper is focused on a different issue, namely why there is this gap between the broadly critical literature on examinations and the ongoing and widespread use of examinations in higher education. The status of traditional exams as a standard assessment tool was of course undermined dramatically by the COVID-19 pandemic, which required universities around the world to shift at short notice to alternative forms of assessment that could be undertaken at a distance (Henderson et al. 2023). The longer-term implications of that emergency shift for the ongoing use of exams are still unclear; time will tell whether the departure from exams will be a temporary measure or the start of a broader shift. However, while the practices and influences may have changed as a result of COVID-19, there are still lessons to learn about how to close the distance between research and practice.

There has been some limited discussion of the research-practice gap in higher education. In 1985, George Keller infamously attacked HE research, stating that ‘hardly anyone in higher education pays attention to the research and scholarship about higher education’ (7). Colbeck (2000) describes the perverse situation whereby ‘university administrators implement major teaching and curriculum reform initiatives without so much as brief phone consultations with experts on their own campuses’ (35). More recently, Evans et al. (2021) lament the way in which higher education research can end up producing ‘work that is little value on the ground in informing teaching and learning’ (526).

In contrast to the muted debate about the research-practice gap in higher education, there has been vigorous discussion over many decades at school level, going back at least as far as John Dewey (1929). A wide range of explanations have been suggested for the lack of impact of research on learning, teaching and assessment in schools, some of which are unlikely to translate to any great extent into higher education. For example, given the dual research-teaching role of most academics, there is probably not a great divide between a ‘craft culture’ and ‘research culture’ as has been observed in school contexts (Hirschkorn and Geelan 2008). The relative lack of status afforded by HE research (Cotton, Miller, and Kneale 2018) probably means that there are fewer incentives than in school education for researchers to focus on ‘pure’ rather than practice-focused research (Hemsley-Brown and Sharp 2003). On the other hand, some explanations are likely to be relevant to higher education, such as a perception that the findings from educational research lack validity (Vanderline and van Braak 2010) and the fact that practitioners lack the time to engage with research (Hirschkorn and Geelan 2008).

The issue discussed in this article is whether researchers are using the right kinds of approaches in their work. That is not because the other factors contributing to the gap between research and practice are not important or serious, but because they are in a sense less fundamental. Even if HE researchers solely used words of one syllable, even if HE practitioners had all the time in the world, even if HE research was seen as highly rigorous; even with all those benefits, if the research itself does not engage with the concrete and practical challenges of assessment in the right way, it will not influence practice in any way. Practitioners simply won’t be able to make use of the research as part of their assessment practice.

In order to explore researchers’ approaches as a factor in the research-practice gap in assessment in higher education, the article will describe a theory of impactful educational research, and discuss its application to a particular aspect of assessment. The theory is Viviane Robinson’s ‘problem-based methodology’, and the aspect of assessment is the traditional examination – meaning a high-stakes, summative, in-person, invigilated, individual, written, closed-book assessment.

Over a number of years, Viviane Robinson has developed an influential (Haig 2014; McNaughton 2014) critique of research methodologies employed by school-focused educational researchers (Robinson, 1992, 1993, 1998, 2014; Robinson and Lai 1999). In her view, insufficient engagement by researchers with the challenges faced by practitioners is a key factor in the lack of uptake of educational research:

I am making the … radical claim that research may be ignored, regardless of how skilfully it is communicated, because it bypasses the problem-solving processes that sustain the practices that researchers seek to alter. (Robinson 1998, 17)

Robinson’s model is worth exploring because its underlying idea – that researchers don’t place enough emphasis on engaging with the problems practitioners take themselves to face – is widely felt among those commenting on higher education research. For example, Evans et al. (2021) caution against research that is ‘ignorant of the situational requirements’, and say that ‘researchers need to have a holistic understanding of issues impacting the field and an in-depth knowledge of the immediate context they are reporting on’ (526). Kezar (2000) is concerned that ‘practitioners’ assumptions and habits of mind are generally not taken into account’ (17) in higher education research. Cross (2000) reports the sense that researchers ‘are not interested in addressing the problems that practitioners must solve’ (65).

Beyond that, there are strong parallels between Robinson’s model and some recent work advocating for a ‘practice perspective’ on assessment (Boud et al. 2018). This work shares Robinson’s diagnosis that the gap between research and practice is partly due to researchers’ approaches, or ‘limitations in the ways in which research on assessment is conceived and framed’ (Boud et al. 2018, 1108). The proposed remedy is, as for Robinson, to focus on the reality of assessment practice: ‘an unrelentingly grounded picture of the everyday world in which assessment sits’ (Boud et al. 2018, 1116). The practice perspective also, like Robinson, has a focus on the decisions that practitioners make about assessment. For example, it has resulted in the development of the ‘Assessment Design Decisions Framework’, which is intended to help practitioners consider a range of different factors when making their decisions; factors such as the purpose and context of the assessment (Bearman et al. 2016). In aiming to help practitioners to ‘achieve and expand their aspirations’ (Bearman et al. 2017, 51) the approach maps nicely onto Robinson’s idea – discussed below – of addressing both the effectiveness with which assessment approaches satisfy practitioners’ existing constraints, and the coherence of those approaches with wider constraints of relevance to practitioners.

Nevertheless, there are differences between the ‘practice perspective’ and Robinson’s ‘problem-based methodology’. Most fundamentally, there is a disagreement about the value of framing decisions in terms of rationality. While Robinson’s approach ‘requires … a willingness on the part of researchers to recognize the rationality of those practices they seek to alter’ (Robinson 1998, 25), Bearman et al. (2017) reject the idea that assessment decisions are ‘based on a rational analysis of competing demands’ (50), and therefore focus on influences that go beyond those that practitioners would consciously consider when weighing up an assessment format – to include, for example, professional identity. So while there are crucial differences between these approaches, there are also key similarities, and the emergence of the ‘practice perspective’ provides additional encouragement that Robinson’s problem-based methodology would find a receptive audience in researchers exploring assessment in higher education.

To say that Robinson’s ‘problem-based methodology’ has value – and more broadly that it is worthwhile exploring how researchers engage with practitioners’ challenges – is not the same as saying that all research is aimed at changing practice; or that all research aimed at changing practice attempts to do so via persuading practitioners that there are better solutions to their challenges. There is much research that is not aimed at changing practice, and there is much research that aims to change practice through changing systems and processes rather than practitioners’ decisions. In those cases, the intended audience of the research will not be practitioners. Indeed, many assessment decisions are out of practitioners’ hands; where accrediting professional bodies have particular expectations, for example, or where institutions have certain kinds of assessment policies. And in understanding and influencing assessment practices, a whole range of factors beyond individual practitioners require exploration: structural, political, financial, etc. However, in many cases practitioners do have a significant element of choice about assessment methods. If those practitioners are going to be persuaded to make different choices (what Robinson calls ‘non-coercive change’) then the research ‘must square with what they already believe, or with what, as a result of dialogue with the researcher, they come to believe’ (Robinson 1992, 25). For that kind of research, the intended audience (directly or indirectly) will be practitioners themselves. As evidenced above, there is a concern from higher education researchers that there is insufficient engagement with the challenges practitioners take themselves to face; a concern which Robinson’s approach addresses head-on.

The article will explore how well the recent research literature on exams satisfies the demands of Robinson’s model, in order to develop some suggestions for how exam research – and assessment research in general – could have more impact on assessment practices. The issue of exams is a promising case study for the gap between research and practice in assessment in higher education, simply because the gap appears so large. The widespread and ongoing use of traditional exams seems to stubbornly ignore the volume of negative literature. By focusing on exams, this article attempts to shed some wider light on how the research-practice gap in assessment in higher education can be made at least a little narrower.

2. Closing the research-practice gap with a problem-based methodology

Drawing on the idea of ‘theories of practice’ developed by Chris Argyris and Donald Schon in the 1970s (Argyris and Schon 1974), Robinson proposes a ‘problem-based methodology’ for educational research, starting from the idea that educational practices that researchers may wish to challenge and change are not best seen as irrational spasms or unthinking traditions, but as rational attempts to solve particular problems. As a consequence, Robinson believes that researchers need to offer superior solutions to those very same problems if they want to contribute to the improvement of those practices. Just as students can perform poorly in exams because they answer the question they wish they’d been asked rather than the one they are set on the paper, the work of researchers can lack impact because they offer solutions to the problems they see, rather than the problems practitioners take themselves to face: ‘Researchers have little to offer by way of alternative solutions, when the problems they have been studying are not those of the practitioner’ (Robinson 1993, 4).

Central to Robinson’s approach is the idea of practitioners’ ‘constraints’, ‘the things that need to be taken into account in coming up with an adequate solution’ (Robinson 2014, 343). Constraints can include a very wide range of things: particular values and beliefs, time pressures, available resources, other existing practices, etc. Constraints need not necessarily be transparent to the practitioners themselves. Robinson uses the established idea of ‘espoused theories’ and ‘theories-in-use’ (Argyris and Schon 1974) to distinguish between those constraints that practitioners themselves would cite when defining the problem, and those constraints that guide practitioners’ activities without them being aware. The ‘constraint set’, consisting of both kinds of constraints, defines the problem to be solved.

Educational practices are conceptualised as attempts by practitioners to satisfy a relevant constraint set (Robinson 1993). Therefore, any research that aims to directly change practice needs to offer a solution that satisfies the constraint set in a superior way: researchers need to offer a better solution to the problem that practitioners are attempting to solve. Robinson offers three criteria for judging the value of such a solution:

• Accuracy. The factual claims that are part of the solution need to be accurate.

• Effectiveness. An effective solution needs to satisfy practitioners’ constraint set. This is not as simple as solving the problem as defined by practitioners, due to the inclusion in the constraint set of ‘constraints-in-use’, of which practitioners may not be overtly aware. This criterion captures how well a solution achieves the outcomes anticipated or desired by practitioners.

• Coherence. Not only does a solution need to satisfy practitioners’ constraint set, it needs to do so in a way that does not make it harder to solve other problems that the practitioners care about. Researchers can make the case that additional considerations are relevant to the problem. This criterion accommodates the need for researchers to ‘get outside’ the constraint set and challenge how practitioners frame the problem: ‘the coherence criterion … goes beyond what problem solvers themselves have currently taken as important and asks whether there are other values which should influence the choice of solution for the problem’ (Robinson 2014, 349)

Robinson links a number of familiar concerns about educational research to these three criteria. For example, researchers who privilege one or two constraints rather than addressing the set as a whole – who oversimplify the problem – are failing on the effectiveness criterion. Researchers who offer solutions without considering important knock-on impacts, or who emphasise constraints not included in practitioners’ own constraint set without explaining their relevance to the wider concerns of practitioners’ themselves, are failing on the coherence criterion.

Given that impactful research needs to engage with practitioners’ own ‘problem analysis’, what do we know about the factors that academics are constrained by when they design assessments? On this issue, there is a widely-recognised lack of evidence. Numerous researchers have commented on the absence of attention paid to academics’ decision-making processes when it comes to assessment (Bearman et al. 2017; Joughin, Dawson, and Boud 2017; Samuelowicz and Bain 2002). For example, in his review of the literature comparing exams with coursework, Richardson (2015) ‘found no research on the attitudes of academic staff to different forms of assessment’ (445). Some researchers have contrasted this lack of evidence about how academics develop assessments, with the very extensive research on students’ experiences of those assessments (Adachi, Tai, and Dawson 2018; Brady, Devitt, and Kiersey 2019). Of the research that does discuss assessment decisions, some focuses on non-rational factors such as the kinds of biases and traditions that can lead academics to ‘resist change’ (Joughin, Dawson, and Boud 2017; see also Chong and McArthur 2023; Harland and Wald 2021). And some of the research is limited in that it explores the experiences of academics who are unrepresentative in crucial ways, such as being particularly passionate about self and peer assessment (Adachi, Tai, and Dawson 2018) or because they were identified as using particularly innovative assessment methods (McLean 2018). Bearman et al. (2017) report the results of interviews with academic staff and identified a number of constraints (or ‘influences’), such as the need to reduce the time taken up by assessment implementation and marking. Citing the poor understanding of assessment decision-making, Dawson et al. (2013) ‘urge scholars of learning and teaching to move beyond the study of abstracted assessment practices to understand how assessment principles can translate to improved outcomes in local contexts’ (110). It is important to note that this lack of research into practitioners’ assessment decisions doesn’t imply that there is a lack of research into practitioners’ conceptions of assessment. There is a large and longstanding literature on issues such as practitioners’ assessment literacy (e.g. Chan and Luk 2022) and the gap between student and staff conceptions of assessment (e.g. MacLellan 2001). How practitioners conceptualise the assessment process is related to, but distinct from, the question of how they go about making assessment decisions and the factors they consider when they do so. It is the lack of research on the latter that has been highlighted by researchers.

While there is limited research exploring the constraints that frame the assessment decisions of academic staff, there appears to be effectively none that specifically looks at the constraints that are involved in the decisions around the use of exams. We are limited to glimpses such as an ‘illustrative story’ that Samuelowicz and Bain (2002) use to bring their findings to life, and an interesting but isolated quotation from Bearman et al. (2016):

Well I suppose, yeah, I suppose … Well, I don’t know actually. It depends whether it’s important to test whether they have any basic knowledge, I don’t know. I feel uncomfortable not having exams. (Science lecturer)

I don’t think that probably crossed our mind, not having an exam. (Science lecturer, 550)

Given the widespread concern about the persistence of exams in the face of critical assessment research, it is surprising that research has not explored the decision processes that lead academics to choose examinations as an assessment method. It is surprising in itself, and in the context of employing an appropriate methodology for practice-oriented research, it creates a significant obstacle to influencing practice.

3. Method

Literature on examinations in higher education published in the five-year period between 2016 and 2021 was reviewed to explore the extent of its alignment with Robinson’s problem-based methodology. (Some of the papers reviewed were originally available as advance online publications, and have been published since 2021 in print). In line with the procedures described by Richardson (2015) – who undertook a review of the literature comparing exams and coursework – a number of steps were taken to overcome the issue that the term ‘examination’ can refer to both an assessment format (‘open-book examination’) and as the process of investigation (‘a critical examination of … ’). An initial simple search was undertaken using Google Scholar, with backwards and forwards searching of cited sources. To provide a more systematic literature search this was supplemented by searches within individual journals. In his review of higher education journals, Tight (2018) provides a list of 28 ‘key journals’ based on age, size and ranking, and this list was used as a focus for the search. Within these 28 journals, searches were undertaken using Scopus to find papers published between 2016 and 2021 featuring the terms (and their plurals) ‘exam’ and ‘examination’ in the title, with phrases such as ‘an examination’ and ‘critical examination’ excluded. This yielded 142 papers. This was reduced to 31 papers following the elimination of papers not focused on the investigation of examinations as a way of assessing taught degree programmes in higher education (e.g. focused on school-level or university entry exams, examination of PhDs, physical medical examinations, and where ‘examination’ was used in the sense of an investigation). The number was reduced further to 17 when papers that discussed exams as data sources rather than exams as an assessment format were eliminated. In total (combining the initial and systematic searches) 43 relevant papers were found.

These 43 papers were first reviewed in order to ascertain which assessment constraints they focused on. The most common themes have been used to structure the findings: learning, incorporating both the activity of learning (studying) and the achievement of learning (student performance) (Buckley 2021b); student wellbeing (exam anxiety); assessment security and academic integrity; authenticity; inclusivity. The papers were then reviewed to determine how they framed those constraints with regard to practitioners’ concerns. Each paper was categorised as addressing the accuracy of practitioners’ beliefs (accuracy), how well practitioners’ existing constraints are satisfied (effectiveness), or the relevance of additional considerations to practitioners’ existing constraints (coherence). It was also noted where a paper did not provide any framing of practitioners’ concerns, i.e. where there was no mention at all of how the consideration under discussion features in practitioners’ assessment decisions.

In asserting that a particular paper frames a constraint in a certain way, it does not follow that the paper is necessarily focused substantially on addressing practitioners’ concerns. It indicates instead that, where the topic of the paper is discussed (particularly in the introduction and conclusion of papers), some claim is made about why the topic is of relevance to practitioners: because it corrects erroneous practitioner beliefs, because it helps practitioners achieve their existing goals, or because it helps them adjust their priorities to accommodate additional important considerations. In particular, where a concern is simply acknowledged as a consideration for practitioners, with no further discussion, that has been coded as addressing the effectiveness with which the assessment approach/es under discussion satisfy existing constraints.

4. Results

Table 1 provides a summary of the results of this review. Some papers feature more than once in the table, because they address more than one of the constraints, or because they provide more than one framing.

Table 1. Overview of reviewed papers.

	Accuracy Addresses the accuracy of practitioners’ beliefs	Effectiveness Addresses the effectiveness with which assessment approaches satisfy practitioners’ existing constraints	Coherence Addresses the impact of assessment approaches on wider concerns relevant to practitioners	None Does not address practitioners’ concerns
Assessment security		Baijnath and Singh (2019), Chirumamilla, Sindre, and Nguyen-Duc (2020), Denny et al. (2019), Manoharan (2019), van Halem, van Klaveren, and Cornelisz (2021)	Forkuor et al. (2019), Kapardis and Spanoudis (2022)	Al-Hakeem and Abdulrahman (2017)
Authenticity			Myyry and Joutsenvirta (2016), Villarroel et al. (2020)	Linden and Gonzalez (2021), Malone et al. (2021), Surry, Torre, and Durning (2017)
Inclusivity	O’Neill and Padden (2022)	Bygren (2020), Ellison and Jones (2019), Faisal, Shinwari, and Hussain (2017)	O’Neill and Padden (2022), Stegers-Jager, Brommet, and Themmen (2016)
Learning	Chong and McArthur (2023), Harland and Wald (2021), Panadero et al. (2019)	Bay and Pacharn (2017), Chong and McArthur (2023), Guo and Shi (2016), Harland and Wald (2021), Levant, Zückert, and Paolo (2018), Levy, Svoronos, and Klinger (2023), Nsor-Ambala (2020), Villarroel et al. (2020)		Ardolino et al. (2016), Efu (2019), Kilgour and Tayyaba (2016), Lindsley et al. (2016), Nardi and Ranieri (2019), Spiegel and Nivette (2023), Tam (2022), Zhan (2020)
Student wellbeing		Nsor-Ambala (2020)	Bulgan et al. (2017), Lynam and Cachia (2018), Myyry and Joutsenvirta (2016)	Bay and Pacharn (2017), Harley et al. (2021), Monrad et al. (2021), Spiegel and Nivette (2023), Turner and Briggs (2018)

4.1. Learning

The most common constraint considered in recent research on exams is – perhaps predictably – that of ‘learning’, with 17 of the 43 papers reviewed focusing on how well assessments support learning. These are split roughly equally between those that focus on student performance, and those that focus on the impact of assessment on studying. Those who discuss student performance as a constraint on assessment method tend to focus on the difference in student performance on traditional exams as compared to alternatives they advocate, such as ‘bring-your-own-device’ computer-based exams (Nardi and Ranieri 2019), two-stage (collaborative) exams (Efu 2019) and take-home exams (Spiegel and Nivette 2023). Empirical studies use different measures to evaluate student performance, such as average grade (Malone et al. 2021; Nardi and Ranieri 2019), distribution of marks (Linden and Gonzalez 2021), and self-reported grades and performance on a follow-up knowledge test (Spiegel and Nivette 2023).

Those who discuss the relationship between exams and studying, rather than performance, address a venerable topic. For many years, exams have been criticised as unevenly distributing student effort, leading to last-minute cramming before the exam period (e.g. Richardson 2015); as demotivating students who are interested in more than just passing the exam (e.g. Havnes 2004); and as encouraging surface learning and discouraging deep learning (e.g. Scouller 1998). In the papers reviewed for this study, there are still concerns about the impact of exams on the depth of student learning (Ardolino et al. 2016; Malone et al. 2021; Myyry and Joutsenvirta 2016; Spiegel and Nivette 2023; Villarroel et al. 2020), and students’ motivation and engagement more broadly (Guo and Shi 2016; Harland and Wald 2021; Lindsley et al. 2016; Tam 2022).

Of the 17 papers that focused on the idea that assessment should support student learning (whether in the activity or achievement sense) nine papers discuss – to different levels of detail – the relationship between learning and practitioners’ constraints. Six of those nine papers present the improvement of learning simply as an existing priority for practitioners. This corresponds to the effectiveness criterion. This can be very brief; for example Levant, Zückert, and Paolo (2018) open their paper with the sentence ‘Question response feedback is widely used in formative and summative testing to facilitate learning, especially in computer-based systems’ (996). This is a very brief indication that they believe that the impact of assessment on learning is a consideration for practitioners. Similarly, in their opening sentence Guo and Shi (2016) state that ‘[t]he quality of student learning and development has been widely accepted as the core of educational quality in the last three decades’ (642).

Three papers frame the consideration in terms of weaknesses in practitioners’ understanding of effective assessment practices – the accuracy criterion. In encouraging practitioners to move away from exams and towards assessments more aligned with the ‘Assessment for Learning’ movement, Chong and McArthur (2023) do partly frame their advice in terms of supporting practitioners to achieve their existing goals: practitioners in both western higher education and Confucian-influenced cultures value assessment for learning (though perhaps not highly enough, in their view). However, they also suggest that practitioners have faulty beliefs about assessment, by ‘not having sufficient knowledge of the varied forms of assessment’ (11). Similarly, Harland and Wald (2021) acknowledge (perhaps sarcastically) that ‘assur[ing] the desired patterns of student behaviour … seems to be a requirement for lecturers’ (114), but are concerned that practitioners possess flawed beliefs about assessment: ‘The new order of graded internal assessment (which is hardly “new” anymore) seems to have been uncritically accepted by most academics’ (115).

A narrow minority of the papers discussing the impact of exams on learning (eight papers) include no discussion of the relevance of student learning to practitioners’ own concerns. For example, Ardolino et al. (2016) explore the impact on student performance of ‘integrated’ exams, covering multiple modules. No discussion is included of how student performance does, or should, feature in practitioners’ decisions about assessment methods. Tam (2022) studied students’ perceptions of the impact of take-home examinations to their study practices, but does not discuss the role of study practices in practitioners’ assessment decisions.

It is perhaps not surprising that some authors omit any discussion of the importance of learning to practitioners’ assessment decisions. It is possible that they simply take it as read that whether or not a change in assessment format would impact on students’ performance, or study practices, is a factor constraining practitioners. In which case, the absence of discussion of the relevance of the constraint to practitioners represents an implicit appeal to the effectiveness constraint: learning is so obviously an existing constraint of practitioners that discussion would be superfluous.

In line with the brief or absent explicit engagement with practitioners’ concerns, few papers offer any evidence about how practitioners perceive the importance of learning as a constraint on assessment methods. Of the 17 papers addressing the relationship between examinations and learning, only four purport to provide references to existing literature on how practitioners make assessment decisions. Chong and McArthur (2023) reference empirical studies about practitioners’ assessment literacy and use of formative assessment. Bay and Pacharn (2017) reference literature about the impact of professional bodies on academics’ assessment choices. In other cases, however, the appearance of evidence is sometimes not matched by reality. The reference that Panadero et al. (2019) provide to support the claim that ‘[t]here is a large literature on teachers’ conceptions about assessments and how they shape what they do in classrooms’ (381) relates to school teachers rather than university teachers. Similarly, while Nsor-Ambala (2020) claims that ‘the literature has been fairly consistent about the views of faculty about exam types’ (33), no reference is provided.

4.2. Stress, anxiety and wellbeing

Another prominent theme in the recent literature on examinations is the impact of assessment on student wellbeing, particularly in terms of the traditional topic of exam anxiety. Nine of the 43 papers reviewed included a focus on this topic. For example, Bulgan et al. (2017) explore levels of anxiety among students at a Turkish university during a set of final examinations, while Spiegel and Nivette (2023) compare the impact on student wellbeing of in-class closed-book exams and take-home open-book exams. As with the impact of exams on learning, a substantial portion of the papers (five of nine) do not include any explicit discussion of how wellbeing features in practitioner assessment choices. For example, in their investigation of students’ emotions in computer-based examination, Harley et al. (2021) do explicitly discuss wellbeing as a focus of assessment researchers (‘test anxiety is one of the most widely investigated areas of academic achievement emotion research’ (956)) but fail to discuss its importance to practitioners. In the case of Monrad et al. (2021) the situation is mixed. The first sentence of the abstract begins: ‘As educators seek to improve medical student well-being … ’ (872). However, that explicit claim about the assessment choices of practitioners is not repeated in the main body of the text (which refers only obliquely to practitioner priorities, by claiming that there is an ‘increasing imperative’ (872) to address negative impacts on student wellbeing).

Of the four papers that do discuss practitioner concerns, three seem to suggest that the importance of wellbeing in the context of exams is important because of its impact on a further consideration, student performance. For example, Lynam and Cachia (2018) state that, because the emotions of assessment have an impact on learning, ‘it is important that academics are aware of the elements of an assessment that are associated with negative or positive emotions and aim to adjust these accordingly’ (232). In framing wellbeing as an important consideration because of its impact on an existing priority, these papers seem to appeal to the coherence of exams as a solution to practitioners’ constraints. One final paper (Nsor-Ambala 2020) frames wellbeing as an existing practitioner constraint: ‘this detrimental effect of pre-exam anxiety on student scores is of concern for educators’ (39). This is an appeal to effectiveness.

Only one of the nine papers discussing wellbeing, Nsor-Ambala (2020), purports to provide evidence for its role in practitioner assessment choices. The paper includes a reference for the claim that ‘[t]here is growing interest among educators to choose assessment methods that enhance “active learning” as active learning … reduces pre-exam anxiety’ (35) (though a reference from 1994 perhaps provides limited evidence for a ‘growing interest’).

4.3. Assessment security and academic integrity

Eight of the papers reviewed discussed some aspect of academic misconduct in relation to assessment. For example, Forkuor et al. (2019) explore students’ perceptions of acceptable behaviour in exams, while van Halem, van Klaveren, and Cornelisz (2021) discuss barriers faced by students in the use of virtual proctoring of online exams. Of these eight papers, all but one comment on the relevance of the topic to practitioners. Five of the papers frame student academic misconduct in assessments as an issue of effectiveness, i.e. as an existing concern of practitioners. For example, Manoharan (2019) states that ‘almost all academic institutions [have] in place compulsory tuition on academic honesty’ (140), and Denny et al. (2019) justify focusing on how exams can be made less vulnerable to misconduct on the grounds that ‘[c]heating on tests and exams is viewed, by faculty and students alike, as a particularly serious act’ (463). Only one of those five papers offers evidence that misconduct in assessments is an existing practitioner concern; Denny et al. (2019) cite a study that found that practitioners viewed ‘cheating in tests and exams’ (463) as a particularly severe act of academic dishonesty.

Three of the papers seem to frame misconduct in assessments as something that practitioners should care about because of its impact on other considerations that they do prioritise, i.e. in terms of the coherence of assessments. All three (Baijnath and Singh 2019; Forkuor et al. 2019; Kapardis and Spanoudis 2022) cite academic misconduct in assessments as a risk for institutional reputation. Forkuor et al. (2019) also highlight the socio-economic consequences (in terms of the quality of graduates and national economic productivity), and Baijnath and Singh (2019) similarly flag the potential risks to the ‘moral fabric of the entire society’ (5). Kapardis and Spanoudis (2022) reference a study about practitioners’ perceptions of computer-based versus paper-based exams, but none of these three papers provide evidence that the wider impacts of cheating (on institutional reputation, for example) are existing concerns of practitioners.

4.4. Authenticity

A traditional criticism of exams is the lack of resemblance between a time-constrained, closed-book, handwritten, individual examination, and the kinds of tasks undertaken in contemporary professional workplaces. Going back several decades, Powell and Butterworth (1971) disparaged the belief that ‘[e]mployers are entitled to know if a man [sic] is going to crack up under pressure’ (5) and that ‘[e]xaminations are thus a psychological and moral test as well as an intellectual one’ (10). Of the literature reviewed between 2016 and 2021, five papers addressed the authenticity of examinations. Linden and Gonzalez (2021) and Malone et al. (2021) observe in passing that alternatives (open-book exams and open-book open-web exams, respectively) provide tasks that more closely resemble the world of work. Neither of those papers comment on how that resemblance features in practitioners’ considerations. Myyry and Joutsenvirta (2016), Surry, Torre, and Durning (2017) and Villarroel et al. (2020) discuss the issue in more depth. In the case of Myyry and Joutsenvirta (2016) and Villarroel et al. (2020), authenticity is framed in terms of its positive impact on an additional practitioner constraint (coherence), which in both cases is the benefits of assessment to learning: ‘[Authenticity] has an impact on the quality and depth of learning achieved by the student and the development of higher-order cognitive skills’ (Villarroel et al. 2020, 39). For Surry et al. (2017), the issue is one of the effectiveness of exams (specifically clinical-vignette multiple choice question exams) as a solution to the existing need to assess ‘real-world’ clinical reasoning. Only one of the five papers cites any literature that provides evidence about practitioner constraints: Linden and Gonzalez (2021) reference papers exploring staff perceptions of online examinations.

4.5. Inclusivity

Finally, five of the papers reviewed explore the inclusivity of exams and their alternatives. For example, Faisal, Shinwari, and Hussain (2017) analyse the differences in exam performance between male and female students at a medical college in Pakistan. Bygren (2020) studied the impact of student gender and the ‘foreignness’ of student names on non-blind examination marking. All five of the papers comment on the relevance of inclusivity to practitioner constraints. Three of the papers frame inclusivity as an existing concern (effectiveness). Bygren (2020) talks about ‘impartiality and tolerance’ (302) being particular priorities in a university setting, while Faisal, Shinwari, and Hussain (2017) say that ‘[g]ender issues are currently a matter of high concern worldwide, especially among academics and policy formulators’ (204). Both Faisal, Shinwari, and Hussain (2017) and Ellison and Jones (2019) provide some evidence for the importance of inclusivity as a constraint on assessment choices, with the latter citing the policies of governments and the English sector regulator.

Two of the papers frame the concern instead in terms of the coherence of exams: they argue that the importance of diverse assessments lies not in the diversity itself, but in the impact of that diversity on widening participation as a further goal. Both Stegers-Jager, Brommet, and Themmen (2016) and O’Neill and Padden (2022) are concerned about the impact of assessments on the retention of non-traditional students. The latter paper also seems to question the accuracy of practitioner beliefs, by claiming there is a lack of awareness that ‘equitable assessment practices’ (2) are important for supporting diverse student populations. Neither paper provides evidence that practitioners prioritise the widening participation agenda, or – for O’Neill and Padden (2022) – that practitioners have erroneous beliefs about the role of assessment in that agenda.

5. Discussion

Using Robinson’s ‘problem-based methodology’ as a guide to the kind of research that engages with the assessment challenges of practitioners, what lessons can we draw from the research on exams in higher education published between 2016 and 2021? The previous section featured research that matches all three elements of Robinson’s model. There is research that seems aimed at improving the accuracy of practitioner’s factual knowledge, such as Chong and McArthur (2023) who are concerned about the lack of understanding of the varied forms that assessment can take. There is research that attempts to help practitioners improve the effectiveness with which assessments satisfy their existing priorities, such as Manoharan (2019) who offers advice on how practitioners can adapt multiple-choice examinations to better meet their existing desire to reduce student academic misconduct. And there is research that seems aimed at supporting the coherence of practitioners’ assessment choices, by encouraging them to take into account wider considerations; an example of this is Villarroel et al. (2020) who argue for the value of authentic assessment because of its’ positive impact on student learning. While all three approaches are represented in the literature reviewed, they are not represented equally. The dominant strategy is to address the effectiveness of assessment solutions. 17 of the 43 papers reviewed address how well traditional exams, or their alternatives, satisfy the existing constraints of practitioners.

An equal number of papers (17), featured no explicit discussion of how the research relates to practitioners’ assessment decisions. This was common for papers addressing all the themes, except for assessment security, where seven of the eight papers included some information about its relevance to practitioners. In the papers lacking that discussion, a plausible interpretation is that researchers assume that the relevance to practitioners of the strengths and weaknesses they discuss is simply obvious. Of the 43 papers reviewed, six explored the impact of assessment on students’ approaches to their studies, yet failed to explain its relevance to practitioners. It may seem an obvious consideration to some (particularly those professionally involved in the study of assessment) but given the available evidence (e.g. MacLellan 2001) the importance of ‘assessment for learning’ to practitioners can’t simply be assumed.

A further issue is that even where researchers do explicitly discuss the relationship between their work and practitioners’ concerns it is rare for relevant evidence about practitioners’ concerns to then be cited; only nine of the 43 papers reviewed do so. That’s not to say that researchers don’t provide evidence for the importance of the issues they discuss: they do, as part of providing a rationale for the research. However what is usually missing is evidence that the issue is of importance to practitioners, rather than other researchers. This is perhaps not surprising, given the limited amount of research about how practitioners make assessment decisions in general; and the almost complete lack of research about how practitioners make decisions about using exams in particular. If we don’t know why practitioners choose to use exams in particular situations, it is hard for those conducting research on exams to provide evidence about how their research engages with those choices. Robinson herself is clear that the kind of practice-focused model she advocates depends on robust information about how practitioners frame their problems and make their choices: ‘the focus of data collection and analysis within problem based methodology is the discovery of the reasoning that sustains the practices under investigation’ (Robinson and Lai 1999, 199). More research about the decision-making processes that lead practitioners to use traditional exams, or to switch away to other kinds of assessments, is a prerequisite clearer links between research on exams and the practical concerns of front-line academic staff.

Another likely explanation for researchers failing to provide evidence for the relevance of their considerations to those of practitioners is the dual role of the researchers themselves. The vast majority of those undertaking research on exams will also be practitioners, in the sense that they make decisions about assessment methods in the course of their own teaching. They are likely to be academics in other subjects, educational developers, or other professionals with a teaching load. For assessment researchers rooted in the day-to-day reality of assessment choices, it may seem superfluous to justify how their research relates to that reality. While understandable, this is nevertheless problematic.

6. Conclusion

This article has discussed the research-practice gap in assessment and feedback in higher education. It has focused on the literature about examinations published between 2016 and 2021, and explored how the approaches adopted by researchers may influence the impact of that research. Using Viviane Robinson’s ‘problem-based methodology’ as a guide, it has sought to extract some lessons about how exam research could engage better with the challenges of those practitioners tasked with making decisions about assessments.

An important limitation of this review is that a reasonable proportion of research is not aimed at directly influencing practitioners’ decision-making at all, for which Robinson’s model is not applicable. However, a significant amount of research does have that goal, and as described earlier in the paper there is a clear desire from within HE research to promote that goal further. Another limitation of the review is the narrowness of the literature search. The systematic element of the search was limited to 28 key journals. In addition, the workload challenges created by the dual-meaning of the term ‘examination’ meant that the search for that term was restricted to the papers’ titles, rather than also encompassing the abstracts and keywords as would normally be the case.

Using Robinson’s problem-based methodology as a guide, this article has highlighted some important features of the research on exams. Many of the papers reviewed do not explicitly address how the issue under consideration relates to practitioners’ priorities, and those that do rarely provide any supporting evidence about those priorities. The general sense is that the priorities of the practitioners who ultimately make the key assessment choices are simply not an important part of the picture. There are of course exceptions, and this review has found numerous examples of papers that both explicitly engage with practitioners’ concerns, and provide evidence about those concerns. But it has also found that a substantial minority does not do the former, and a large majority does not do the latter.

This study connects importantly with contemporary ways of thinking about assessment. The approach taken in this paper – applying Robinson’s problem-based methodology to research on assessment in higher education – focuses on how practitioners balance constraints to arrive at assessment decisions. This approach intentionally centres practitioners’ rational decision-making processes. However, this does not imply that only rational processes are involved in those decisions, or that the only factors influencing assessment are practitioners’ decisions. What it does assume is that practitioners’ assessment decisions have an important role in determining assessment practices, and that it is important to engage directly with those decisions. This is common ground with the ‘practice perspective’ of Boud et al. (2018). The focus on practitioners’ decisions is not only compatible with, but complementary to contemporary approaches to assessment research that adopt broader perspectives that may be, for example, socio-material (Tai et al. 2023), relational (Gravett, Taylor, and Fairchild 2021) or socio-political (Nieminen 2022). If important factors relating to social structures, materiality, relationships etc. are revealed, there is value in encouraging practitioners to bring them into consideration in their assessment decisions. Robinson’s model offers a recipe for doing that: such research could indicate that practitioners need to review their assessment decisions because they have erroneous beliefs, because their assessment choices don’t achieve what they want them to, or because they are neglecting additional factors that bear on the goals they’re aiming for.

The findings reported in this paper indicate that, in recent years and in relation to examinations, assessment research has engaged with practitioners’ decisions only intermittently and superficially. For any single piece of research, that is not a flaw: there are many other aims to research than directly exploring or influencing practitioners’ decisions. However, as a judgement about the wider body of literature it is more concerning. And in the context of widespread concerns about a disconnect between the insights of those researching assessment and the assessment practices to be found in universities, it is a fundamental problem. Highlighting how researchers might better engage with practitioners’ decisions is not to lay the blame at researchers’ door for any negative effects of the ongoing use of exams. It is, however, to take seriously the concerns – expressed by researchers – about the gap between research and practice; and to explore ways in which we can improve the situation by bringing research a little closer to the reality of assessment practices.

A clear contributing factor is that more needs to be known about how practitioners see their assessment challenges. As other researchers have highlighted, there is little evidence about how assessment decisions are made; and there is virtually none on the decision-making processes that lead to the use of exams. If research is going to engage with how practitioners attempt to balance constraints, then we need to know what those constraints are and how they are weighted by practitioners. That is not to say that research shouldn’t aim to critique and challenge the way that practitioners frame the problem of assessment design – after all, shifting that framing may be a big part of how assessment can be improved – but there are benefits in motivating that challenge in terms of practitioners’ own wider concerns.

This article has focused on research on exams, but what goes for exams goes for other issues in assessment and feedback. If we want research to influence the decision-making processes of practitioners (and acknowledging that other approaches are legitimate) lamenting the lack of innovation and improvement in assessment while failing to engage with the thought-processes that lead to decisions about assessment is unlikely to succeed. Offering better solutions to the problems that practitioners face is a more promising path to the change we would all like to see.

Disclosure statement

No potential conflict of interest was reported by the author(s).