Educators as Judges: Applying Judicial Decision-Making Principles to High-Stakes Education Assessment Decisions

Abstract

Phenomenon: Programmatic assessment and competency-based education have highlighted the need to make robust high-stakes assessment decisions on learner performance from evidence of varying types and quality. Without guidance, lengthy deliberations by decision makers and competence committees can end inconclusively with unresolved concerns. These decisional dilemmas are heightened by their potential impacts. For learners, erroneous decisions may lead to an unjustified exit from a long-desired career, or premature promotion to clinical responsibilities. For educators, there is the risk of wrongful decision-making, leading to successful appeals and mistrust. For communities, ill-prepared graduates risk the quality and safety of care. Approaches such as psychometric analyses are limited when decision-makers are faced with seemingly contradictory qualitative and quantitative evidence about the same individual. Expertise in using such evidence to make fair and defensible decisions is well established in judicial practice but is yet to be practically applied to assessment decision-making. Approach: Through interdisciplinary exchange, we investigated medical education and judicial perspectives on decision-making to explore whether principles of decision-making in law could be applied to educational assessment decision-making. Using Dialogic Inquiry, an iterative process of scholarly and mutual critique, we contrasted assessment decision making in medical education with judicial practice to identify key principles in judicial decision-making relevant to educational assessment decisions. We developed vignettes about common but problematic high-stakes decision-making scenarios to test how these principles could apply. Findings: Over 14 sessions, we identified, described, and applied four principles for fair, reasonable, and transparent assessment decision-making. These were:

• The person whose interests are affected has a right to know the case against them, and to be heard.

• Reasons for the decision should be given.

• Rules should be transparent and consistently applied.

• Like cases should be treated alike and unlike cases treated differently.

Reflecting our dialogic process, we report findings by separately presenting the medical educator and judicial perspectives, followed by a synthesis describing a preferred approach to decision-making in three vignettes. Insights: Judicial principles remind educators to consider both sides of arguments, to be consistent, and to demonstrate transparency when making assessment decisions. Dialogic Inquiry is a useful approach for generating interdisciplinary insights on challenges in medical education by critiquing difference (e.g., the meaning of objectivity) and achieving synthesis where possible (e.g., fairness is not equal treatment of all cases). Our principles and exemplars provide groundwork for promoting good practice and furthering assessment research toward fairer and more robust decisions that will assist learning.

Keywords:

• Decision-making
• judgment
• educational measurement
• academic performance
• competency based medical education

Background

Erroneous education assessment decisions may lead to learners progressing when not yet ready (failure to fail) or being unduly held back when ready (failure to pass). For learners the stakes are often very high: from being allowed to continue study or work, to being excluded from further training and employment. If poorly made, such decisions risk legal appeals and much expenditure of time and angst. Ultimately there can be consequences for patients and public safety. To avoid such outcomes, much effort is invested in blueprinting programmes of assessment, training assessors, and ensuring reliability and validity when deciding whether learners are ready to progress and graduate. Yet these efforts count for little if assessment decisions and the processes used to reach them are not robust, in the same way that flawed clinical reasoning can result in sub-optimal and even harmful patient outcomes. Biases in clinical decision-making are well-known and parallels have been made with educational assessment decision-making.^1,2

Programmatic assessment and competency based medical education aim to increase robustness in high-stakes decision-making by using a broad range of evidence collected over time.³ Decisions are rarely based on single pieces of evidence (e.g., a single high-stakes exam) but from many sources. How then, should diverse, sometimes divergent and often incomplete evidence be evaluated so that it results in decisions that not only help learning but are fair and robust? In medical schools, high-stakes decisions are often made by a panel or committee, yet the evidence presented to panels may be of variable quality⁴ and panel members may be divided by different, implicit, or ambiguous intents.^1,2,4–6 It is no wonder that such panel discussions are frequently lengthy and effortful, and end without resolving concerns and uncertainties. The judiciary, however, can draw upon centuries of experience to collect, appraise, and synthesize a range of information and formalize decisions, either as individuals or in panels.

Weighing up disparate evidence for high-stakes decisions with substantial consequences for people’s lives is central to judicial practice. Judges interpret and apply laws, policies, and principles to cases, using expertise developed through years of training and experience.⁷ Public scrutiny, open hearings, and documentation

of judgements promote outcomes which are fair, defensible, and reflect community values and societal expectations.⁸ We therefore saw the opportunity to learn from interdisciplinary dialogue by investigating the principles of judicial decision-making for their relevance to assessment decision-making in medical education.

Comparing decision-making within law and in educational assessment

Grounding our study is a central concept in law that good process not only leads to the best or preferable decision but also clearly explains the path to that decision. What has been considered and how, or the “architecture of the argument” is transparent.⁹ Legal judgments have a common structure: these facts, viewed in the context of these laws, leads to this conclusion.⁹ Similarly, in assessment, these assessment results, viewed against these policies, leads to this outcome for the learner. Less robust decisions can arise where the path from the assessment results (the facts) to the outcome (decision) is not clear. For example, in decision making an assessor or a competence committee may not adhere to policy, nor be oriented toward being just or fair.⁵ In jurisprudence, four arguments can be made when questioning decisions: the facts are contested, the wrong law has been applied, the law has been misinterpreted, or the relationship between the facts and the law is ambiguous.⁹ In assessment, similar arguments can be mounted: the assessment results are contested, the program policy is not applied or ignored, the policy has been misinterpreted or the connection between the facts and the policy is unclear. Table 1 outlines parallels between judicial and assessment decision making related to facts, rules, and decisions.

Table 1. The architecture of an argument in law and assessment.

	Judiciary	Assessment
Facts	Provided by eyewitnesses, forensic evidence	Provided by assessors’ reports
Rules	Laws, regulations	Standards, policy
Decisions/outcome	Innocence or guilt	Progression or interruption to study

In addition to clearly explaining the reasons for decisions, another essential in fair judicial process is attention to fact finding.⁷ Fact finding requires investigation of both sides of the argument and may require preferring evidence from one side over the other. Corroboration, credibility, and a deliberate search for relevant facts are required. Documenting the reasons for decisions should show the process followed, the issues considered, why one version of the event was favored over another, and provides accountability to the affected parties and to the public.⁷ Such transparency guards against bias and reduces the risk of arbitrary or unreasonable decision-making.^10–12

One key difference between judicial and assessment decision-making is the distribution of responsibilities. In common law as practiced in English-speaking countries, those who gather and present the facts (e.g., advocates, police, witnesses) differ from those who make the rules (governments and lawmakers) who differ again from those who make the decisions (judges and juries). This is known, constitutionally, as “the doctrine of separation of powers.” In educational assessment there can be blurring of these boundaries; one person (the educator) can be observing learners’ performances, deciding the rules, and deciding the outcomes.¹³ This situation risks undetected error and bias and requires safeguards to be defensible and seen as independent.

Fairness is a foundational concept in both law and in assessment. Arguing that fairness has not been clearly defined, Valentine proposes a conceptual framework for fairness in assessment,¹⁴ comprising values (transparency, credibility, fitness for purpose, and defensibility), system and process factors (procedural fairness, documentation, validity evidence, multiple assessors, and multiple opportunities for assessment). Most of these qualities are well defined in judicial decision-making and are examined in this study. It is now accepted that a sole focus on “objectivity,” for example by strengthening psychometric data, will not solve assessment decision-making dilemmas.^4,14 Professional judgment is needed to design assessments that will yield sound evidence, and to search for, evaluate, and appraise available evidence, particularly when synthesizing qualitative and quantitative assessment information.¹⁵ The guidance and expertise to make such decisions are however less well developed in assessment than in judicial practice. We therefore asked:

• What are the judicial decision-making principles relevant to educational assessment?

• How transferrable are these principles to high stakes assessment decisions?

In answering these questions, our goal is to highlight and test key principles that would be relevant to a range of international contexts. We aimed to produce groundwork for further scholarly conversations by exploring whether a novel interdisciplinary approach could inform educational assessment decision making.

Methods

To critically examine and synthesize a judicial worldview with a health professions assessment worldview, we adopted Dialogic Inquiry (DI) and report our study according to accepted standards for qualitative research.¹⁶ DI makes disparate disciplinary perspectives and knowledges explicit through dialogue. Based on the “premise that scientific knowledge is jointly created…through egalitarian dialogue,”^17(p998) the DI process can be described as Socratic questioning aimed at unearthing insights through rigorous interdisciplinary exchange, a practice which resonates with health professions education practice.¹⁸ It is not assumed that agreement will be reached, but that examining and confronting differences can prompt progress on solutions which are “stuck.” Between disciplines such as law and education, for example, there may be irreconcilable differences about what constitutes knowledge and evidence. Identifying and articulating such differences through interdisciplinary dialogue widens the possibilities for response when knowledges and practices are contested, for example when an educator faces a legal challenge, or a lawyer represents a student in a university setting.

DI arose from the work of Bakhtin, which was influential in driving the “dialogic turn” in the social sciences. Through dialogue—communication that is embodied, relational, and unique to setting and context—“tensions and dilemmas are enacted and new understanding and knowledge are co-created.”^19(p2) In 1996, Baxter and Montgomery set out the DI process of dialectical inquiry: contrary arguments are sought and pitted against preliminary arguments and proposals, aiming to expose underlying assumptions through challenge and questioning.^20,21 Since then DI has been used largely in communication and education qualitative research, but not as yet to medical education research. It is not discipline specific, so is widely applicable to other fields, and can be used in less structured¹⁸ and more highly structured¹⁹ ways.

Reflexivity

In DI, explicit consideration of stance and positionality is needed throughout, and for generating transformative knowledge.¹⁹ Two authors (WCYH, TW) are medically qualified educators, with national and international experience in diverse nations on accreditation, quality assurance, and scholarship of undergraduate and postgraduate medical education and training. TW has extensive experience in quantitative research, design, and conduct of assessment in medicine, often as an external expert or consultant. WCYH’s experience is in qualitative interdisciplinary and participatory research, informed by the social sciences and critical theory. Within the medical discipline our backgrounds are considered varied, but we share biomedical and scientific epistemic traditions. One author (HD) has many years’ experience in judicial practice as a magistrate and coroner, involving thorough, transparent gathering and interrogation of case-based evidence, and expertise as a senior law academic and textbook author. In contrast to biomedicine, legal practice, and research is based on theory, reasoning, and logic rather than empirical or observable evidence, although it may draw upon scientific expertise in decision making. All three authors have been involved in medical student appeals against assessment decisions. While the assessment situations described in this study are based on direct experience in two countries with different educational and legal systems, our collective international experience also informed our choice of exemplars so as to be relevant to a broader range of countries and contexts.

To assist interdisciplinary dialogue, we examined each other’s languages, epistemic and discursive traditions, using difference to illuminate “biasing” effects when any perspective was inadvertently valued over others. Differences were evident in our use of discipline specific terms, or by using the same terms in discipline specific ways. We addressed potential misunderstandings by explaining our disciplinary understandings and agreeing on one interpretation for this study (See Table 2).

Table 2. Differences in discipline specific terms.

Term	Law perspective	Educational assessment perspective
Objective/subjective	Objective refers to what a “reasonable person” might do or to broad standards that are widely accepted. Subjective refers to actions that are idiosyncratic, apply only in a particular case, unique, peculiar to that person.	Objective was previously synonymous with reliable and valid; subjective with unreliable. Current understandings posit that reliability and validity are better determined by collective judgements based on a range of evidence, rather than over-standardisation of procedures and reliance on a few, or one, assessment tool.
Argument	Is a neutral proposition presenting a particular reasoning or framing of the issue to inform the decision-maker	In lay terms, can be interpreted as meaning confrontational, hostile or critical.
Case	Conceptualised as the theory or set of arguments explaining how the relevant evidence and law combine to produce a particular outcome or position.	Can be interpreted as depersonalized and dehumanizing, for example an anonymous patient case. Alternatively as a legal challenge or courtroom case.
Appeal	Can be seen as part of due process and a procedural safeguard against legal error and arbitrary or unreasonable decision-making.	Can be seen as implying wrongdoing or error and thus induce defensiveness.

The study was conducted in accordance with national guidelines for ethical human research;²² ethical review was not required.

Phase 1—Identifying the principles

We embarked on dialogue where each researcher was questioned by the others to explain their disciplinary practices, using real-life examples of decision-making.

Over 14 sessions of 1–2 hours duration, we described common decision-making scenarios, processes, intended and unintended outcomes, drawing upon our disciplinary literatures, knowledges, and experiences to illustrate commonalities and differences. For example, the judge [HD] shared experiences with appeals against a medical school’s process or where judicial decision-making was flawed, and the educators [WCYH, TW] reflected upon how this situation may have come about. In turn WCYH and TW recounted difficult assessment decisions while HD responded with how judges might approach such decisions. Between each session, discussions were summarized, and the key principles inductively developed by drafting descriptors. Further questioning and discourse refined successive drafts so that principles and descriptors were conceptually distinct, essential to good decision-making practice from a judicial perspective, and relevant to medical education. The resulting principles, referenced to the legal literature and legal precedents, formed our framework for guiding robust assessment decisions.

Phase 2—Applying the principles

To test the identified principles, we drew on our experiences and published case studies^23–25 to develop hypothetical vignettes depicting common assessment situations which could be challenged as evidence in decision-making by competence committees and illustrated the identified principles. Vignettes are commonly used in qualitative²⁶ and educational research²⁷ to explore beliefs, values, and practices. Basing our examples on known events ensured that they could reveal our disciplinary norms and practices during dialogic inquiries, and also have practical relevance. Designing our vignettes to illustrate specified principles aided coherence between our study aims, methods, findings, and practical implications, thus enhancing internal and external validity. All reviewed the vignettes drafted by TW and WCYH, evaluating against the intended principles and alternative interpretations. “What ifs”—alternative events and contexts—were applied to test transferability while covering the same principles and constructs. Redrafting occurred until agreement was reached on the version to be used.

Presenting results dialogically

Reporting interdisciplinary inquiry creates a tension between presenting integrated findings and presenting separate voices to represent disciplinary views and incommensurable differences.²⁸ Reflecting our dialogic process, we firstly present the principles with an integrated voice, then their application separately to illustrate our educational and judicial starting points, followed by an integrated synthesis and practical application of concepts from our dialogue:

• Vignette

• Medical Educators’ view—reflections from a programme design and delivery perspective

• The Judge’s view—reflections from a legal, independent perspective

• Integrated preferred approach—a practical application generated by synthesizing insights from dialogue informed by legal principles and the practicalities of delivering a medical program.

Through these worked examples of our interdisciplinary approach, we present our findings as if to educators responsible for high-stakes decisions, focusing on aspects of the decision and decision-making process which observers would likely question, and on changes that would make the decision more defensible and robust.

Findings

We identified four principles; these are explained below, followed by an application of the principles to three vignettes.

Principles for assessment decision-making

Underscoring the four identified principles is the requirement that robust decisions and decision processes must be fair, reasonable, and transparent (see Table 3).

Table 3. Principles for robust decision-making in assessment and progression.

Principle 1 The person whose interests are affected has a right to know the case against them, and for their perspective to be heard by an impartial decision-maker. Principle 2 Reasons for the decision should be given, including why one perspective is preferred over another. Principle 3 Rules should be transparent, consistently applied and cannot be changed retrospectively. Principle 4 Like cases should be treated alike and unlike cases treated differently from each other.

As well as being based on clear principles, the process of good decision making requires that decision-makers are independent and do not have predetermined or vested interests in the outcome.²⁹ They should bring open minds capable of actively listening to both sides of an argument. For example, Brill describes a US example of the injustice that can occur if the principle that a judge must be impartial is disregarded.³⁰

Adequate notice of concerns or allegations must be given to persons whose interests may be in jeopardy, with a reasonable opportunity to address those concerns or allegations. In common law this is known as the audi alterem partem (“hear the other side”) rule.^31–33 The “person” can refer to an individual, such as a learner or a teacher, or to an organization such as a medical school. It is the person or party whose interests are materially affected by the outcome of the decision, such as being able to continue their studies, or their employment, who has a right to know and be heard. In this study, we focus on the learner as the person whose interests may be affected.

Particular care should be taken where the same educator is both the assessor for a single assessment episode (e.g., observation of a clinical skills performance) and the judge of a collation of assessment episodes (e.g., member of a competence committee) as there is risk of conflict of interest; the judicial equivalent would be the judge who is also an eyewitness bringing evidence.

Principles applied to assessment vignettes

We present the three vignettes aligned to the identified principles, followed by findings from our interdisciplinary dialogues for each vignette.

Vignette 1: Education lingo (Principles 1, 2, and 3)

A final year student lodges an appeal against a fail grade for their long case, a threshold assessment for graduation. For this assessment, students independently gather interview and physical evidence from an actual patient, then synthesize and succinctly present their findings to a senior medical examiner. The student claims that no reason was provided and they cannot identify which part of the marking rubric they failed. As the assessment coordinator, you discover there are no comments or feedback on the marking sheet and approach the examiner. The examiner exclaims “That student was hopeless! They couldn’t even the recall the right drug dose.” You point out recall of doses is not a criterion on the rubric provided to students and examiners. The examiner retorts “I don’t need rubric educator lingo; I’ve taught trainees for decades and know who’s ready to be an intern on my team.”

Commentary on Vignette 1

Medical educators’ view

We rely on experienced assessors to provide evidence about whether learners have reached required standards, but we also want assessments to guide learning. Accordingly, clear descriptions of the purpose³⁴ and what is expected, which also identify any deficits in learning, are used. If consistent with medical program policy, global opinions have value, but their value is weakened if they are not explainable or at odds with learning expectations.

The judge’s view

The process by which the student was failed was fundamentally unfair. The purposes of marking rubrics are to (a) establish accepted criteria against which all candidates can be assessed; and (b) ensure consistency in assessor approach. Making a high stakes decision based on a single, idiosyncratic, assessor using criteria unique to that assessor, or biased by prejudice, undermines both the process and confidence in the process. The examiner has applied an irrelevant marking criterion, suggesting the examiner took little or no account of the rubric. The examiner’s irritated comment corroborates that impression, as does the statement that they rely on their own subjective judgment as to who is ready for internship. This is an unfair failure by omission.

The examiner mistakes fact and “law” by applying a criterion that in fact, is not part of the marking rubric, vitiating their decision.¹ No reasons were provided until the educator investigated. For good practice, reasons should be given, and must explain, by referring to both the marking rubric and the student’s result, why their performance (or parts of it) did or did not meet the required standard. A failure to provide such reasons, or providing irrelevant alternative reasons, is a fundamental failure of natural justice. The purposes of giving reasons are to (a) provide proper feedback to enable students to understand and learn from mistakes; (b) ensure the process is fair and seen to be fair, thereby supporting confidence in the system; and (c) enable any appeals body to understand the case and the reasoning process.⁷

Integrated preferred approach

All examiners (and students) would receive sufficient information on the substance and application of the marking rubric so they have a shared understanding of what is expected. Practice in applying the rubric could be provided, emphasizing principles of objectivity (see Table 2), consistency, and fairness. Fairness can be emphasized by explicitly acknowledging personal biases to neutralize any prejudice.

Standardizing examiners is not always feasible, or even desirable. Synthesizing a range of examiners’ views on a range of observations is preferred so that high stakes decisions are made only when enough evidence is available. Regardless of any requests, all learners should receive feedback. This is important both for patient care and learning. Providing reasons for a judgment justifies and makes transparent any decisions and maintains system integrity.

Vignette 2: Two sides to every story (Principles 1 and 2)

A first-year mature age student repeatedly comes to hospital tutorials casually and inappropriately dressed. When questioned by the clinical tutor, the student defensively replies, “I like to dress this way; patients haven’t complained.” After repeated exchanges, the student begins to dress like other students. A month later the student arrives late repeatedly, blaming traffic. After several weeks, the tutor challenges the “excuses.” The student protests, so the tutor reports the student for lapses of professionalism. This prevents the student from sitting end-of-semester examinations and subsequent progression.

In reality, the student is a single parent under financial stress, juggling study and care responsibilities with night shifts stacking supermarket shelves. The late arrivals were due to new temporary childcare arrangements. The tutor did not provide an opportunity for the student to explain their behavior further.

Commentary on Vignette 2

Medical educators’ view

Determining the motivations of observed behaviors is prone to unconscious biases, for example from previous interactions and early foreclosure. Being late can signal lack of motivation, stress from any cause or, in this case, a consequence of circumstances. Behaviors should be only interpreted after considering alternative perspectives. It can be easy, particularly with a “difficult” student, to overlook a formal opportunity for the student to present their case and assume that repeated observations are sufficient. The tutor appears to be sensitized by previous events, facilitating hasty conclusions. Once a sanction mindset is adopted, it can be difficult to shift to a supportive mindset, and vice versa.

The judge’s view

The tutor has overreacted and therefore has been unfair to the student. The tutor represents the medical program and the university; it is reasonable for the tutor to set tutorial standards and enforce them. It is, however, unfair and unjust to sanction a student without giving them adequate chance to be heard and without considering what they say, when given that chance (see the audi alterem partem (“Hear the other side”) rule introduced earlier).

The tutor has taken appropriate action regarding the student’s attire, even though this episode has soured the relationship between student and tutor.

It is a human and common scenario when the student provokes the tutor into acting precipitately. Nevertheless, to sanction the student without considering all relevant facts and circumstances remains unfair and unjust. The tutor has failed to conduct a reasonable investigation, and as a result, mitigating or exculpatory circumstances which should be considered when exercising discretion have not been considered.²

The student may have contributed to the failure of process by taking a combative stance, but the onus is on the tutor to ascertain all relevant facts and make decisions based on those, and not simply rely on facts already known to them. It is incumbent on the tutor, who has power in this situation, to remain balanced and judicious, notwithstanding any provocation on the part of the student.

Integrated preferred approach

What looks like unprofessional conduct may have multiple causes. Before making hasty judgements, educators should pause to consider other explanations for an observed behavior. Is the student unaware or incapable of understanding the concept of professionalism—or, are there external pressures, health concerns, or another cause? Can assistance be provided to help? Considering alternative explanations and adopting a problem-solving attitude should be the starting point of any investigations.

Unless the relationship has deteriorated so that prejudice or perception of unfairness cannot be put aside, the tutor, rather than a delegate, should speak informally to the student, outside the tutorial setting. The problem can be explained in terms of the university’s expectations, noting that a pattern appears to have emerged. An explanation should be sought—and listened to. Regardless of the lapse, if support is required, the tutor should discuss or refer the student to supports available.

Vignette 3: Rules are rules (Principles 3 and 4)

Laboratory practicals are a mandatory teaching activity, covered by the medical school’s attendance policy which states that being more than 5 minutes late is an absence. Unexplained absences are considered in progression decisions. Over time, the tutor begins to start the practical 10–15 minutes after the scheduled time. Observing this, students also start arriving 10 minutes late, but usually in time for the start. One day, the tutor overhears students talking about how disorganized the tutorials are. The next day, the tutor arrives and starts the tutorial on time. Nine students arrive 10 minutes late. The tutor berates these students, marking all nine as absent. The students claim they are scapegoats and their treatment is unfair; the tutor replies “rules are rules.”

Commentary on Vignette 3

Medical educators’ view

What we say we want is not always what we do. We do not always enforce rules consistently—practice may drift over time, so students learn to attribute more salience to actions than to words. With time, the limits of tolerance can be reached, and rules reinstated, but how? How much warning should students receive, if at all, if the rules have not formally changed?

The judge’s view

For this type of misdemeanor, the tutor is within their rights. However, a reasonable competence committee is likely to find the “offences” proven but admonish the students without recording a “conviction” or imposing further punishment. The “punishment must fit the crime” and should be in proportion to the magnitude and impact of the crime. The concept of retributive justice and proportionality of punishment well-established in history and traditions. The biblical principle of “an eye for an eye” (Exodus 21:23–27) places limitations on retribution. As noted in Vignette 2, the level of retribution can be moderated by personal or other mitigating factors. A committee might see the tutor’s performance as the root cause because it is the tutor’s primary responsibility to set and maintain standards. Students will have a justifiable sense of grievance if the tutor punishes for breaches of standards they undermine themselves, seeing it as an arbitrary and unfair exercise of power, and undermining their confidence in school administration. All parties are diminished in such a process.

Integrated preferred approach

Students have recognized that lack of punctuality is causing disorganization, diminishing the quality of teaching and learning. Rather than retrospectively applying rules that were inconsistently applied, this is an opportunity for the tutor to reset their relationship with the class. The tutor can call attention to the problem, reminding students of the standard—and of the reasons for the standard—warning them that the standard will be maintained in the future and infringements will lead to disciplinary action. Providing a date for re-implementation of the rules and change of practice offers certainty and promotes autonomy; students have sufficient information to act, for example, by changing travel plans.

Discussion

Our study adds to recent research on fairness in assessment^14,35 by demonstrating a rigorous approach to including interdisciplinary insights¹ on an enduring challenge; high stakes decision-making in competency based medical education.⁵ Below, we extend and further examine practice insights from the judicial principles identified and applied in our findings.

Practice insights from applying judicial principles

Proper consideration must be given to both sides of an argument (Principles 1 and 2)

These judicial principles are entirely consistent with learner-centred learning, and requires that reasons must be provided for decisions, especially if decisions are against the learner. Firstly, the learner’s interests are most at stake and fairness dictates they should know why they have failed an assessment or not progress. Second, if the decision is explained plausibly and reasonably, the student is more likely to accept the justice of the outcome. And third, they may appeal. A rigorous and transparent process concluding in a documented and well-reasoned decision will ensure that an appeals panel, competence committee, or court will fully understand what issues were considered and how a decision was reached.

Consistency is a hallmark of justice (Principle 3)

Changing or tightening rules or expectations mid-stream can lead to justifiable grievances from those adversely affected. The law is very clear; Lord Bingham wrote “You cannot be punished for something which was not criminal when you did it, and you cannot be punished more severely than you could have been punished at the time of the offence… is a rule which any child could understand.”³⁶ This may be less clear in situations where the rules are stated but have not been consistently applied. However, retrospectively changing how rules are applied can be viewed the same as retrospectively changing the rules. So it is not reasonable to argue that “the rules are the rules” as a defence if how the rules are applied is changed without warning.

Considerations when determining sanctions (Principle 4)

Together with the seriousness of the offense, the affected person’s personal circumstances must be considered when a sanction is imposed. If cases are significantly similar, they may be treated similarly. It is also reasonable to treat dissimilar cases differently, as it would be unfair to treat as if one size fits all. The factors to be considered must be relevant to the current issue; panels and committees may need direction about what should be considered, and what should not. In assessment and evaluation, “fairness” is not simply treating everyone the same³⁷; more importantly, judgements should not be biased or discriminatory.³⁸

It then follows that the more serious the consequences, the more careful and considered the process and use of evidence should be. In Briginshaw, the court concluded that “The seriousness of an allegation made… or the gravity of the consequences flowing from a particular finding are considerations which must affect the answer” which decision-makers must provide.³⁹ Consequently, ambiguous, inexact, or weak evidence should not determine outcomes with serious consequences for students. “The standard of proof must be responsive to the gravity of the facts…and the consequences of the ultimate decision.”³⁹ This aligns with a principle in programmatic assessment; the stakes of the assessment should be proportional to increases in the stakes of the decision, and to the number of data points included.³

Using the framework of the architecture of an argument, Table 4 compares judicial and educator approaches to system level practices necessary for fair assessment. Our findings add practical insights as to how transparency and procedural fairness, which Valentine described as “an amorphous concept” with “no clear definition….within education”¹⁴ can be achieved. Ensuring that students’ voices and perspectives are heard is one aspect; documentation is another. Our study specifies what must be documented in addition to the decision itself. In sum, judicial principles remind educators of the need to be explicit about the reasons for decisions—regardless of the law, providing reasons enables all to continue learning from the process and its outcomes.

Table 4. Comparison of judicial and assessment components for a fair process.

	Necessary components for a fair judicial process^7	Necessary components for a fair assessment process^14
Facts	Fact finding from both sides Corroboration of evidence. Credibility of witnesses. Fallibility of witnesses. Deliberate seeking of necessary facts	Credibility or “truthfulness” of the findings Multiple assessors Multiple opportunities Fitness for purpose or making judgements on observations that are deemed to be relevant to the work of a health care professional
Rules	Transparency regarding which rules are applicable	Transparency
Decisions/outcome	Transparent consideration and documentation of the reasons Explanation of how conflicting evidence was resolved. Exclusion of irrelevant evidence	Transparency Documentation Defensibility or a judgment made with good reason Validity evidence for judgements

Dialogic inquiry—methodological insights

Our use of Dialogic Inquiry suggests the potential of rigorous interdisciplinary critique to address persistent problems in medical education. Judicial practice is well-tested and transparent; newer systems of assessment need the confidence of the public and employers already concerned that medical school and training programs may “fail to fail” students who become incompetent doctors.⁴⁰ Outside law courts, it is rare to find examples where medical education practice is interrogated through the lens of a different, incommensurate, discipline.²⁰ Although not the primary aim of the study, scholarly engagement with legal practitioners and jurists may reduce the risk of legal action over progression decisions, and thus prevent their long lasting impacts for faculty, the medical school, and the student. These can include shattering of trust between learners and teachers, and harm to the careers, community standing, and reputation of individuals and universities.^41,42

Similar to Wells, our inquiries followed stages; from resisting to recognizing multiple voices, reconsidering our positions, appreciating epistemic difference, realizing our affective responses.¹⁹ As educators, the process was validating as the findings reinforced contemporary assessment principles highlighted in a recent consensus statement on good assessment,³⁴ in particular the need for assessment to be transparent, free from bias, purpose-driven, and coherent. Adopting the judicial emphasis of examining all perspectives, not assuming that the evidence at hand and existing practice norms are sufficient, and understanding how the law might view educator responsibilities, allays concerns about legal risk and appeals. From the judicial perspective, our discussions opened the “black box” of medical education decision-making, which mirror the self-regulating structures of the medical profession.⁴³ We discovered that policy and procedural aspects of universities and courts have much in common.

A barrier to dialogic inquiry is misunderstandings from using the same terms differently.²⁸ These were discovered when “stuck” in circular exchanges; to progress we learnt to pause and revisit foundational concepts and words, their meaning and application in each other’s disciplines (see Table 2). We suggest the same when communicating to students; to establish and continuously update a shared vocabulary from the outset. We benefited from greater awareness of how language can be received and became more precise in our language use. The focus on language and co-production of meaning, without assuming consensus can be found, is a strength of dialogic inquiry.^17,28

Dialogues can also founder if the purpose is not clear. For example, contested decisions may be viewed with educative and/or punitive lenses; education programs may aim for sanction (e.g., proven academic misconduct) and legal systems at remediation (e.g., community service or rehabilitation). Our use of vignettes grounded us as participants in the same contextual, if imagined, setting. This allowed clarification of purpose which can be useful when exploring abstract, yet-to-be defined concepts.⁴⁴

Although we have focused on how educators can be better judges, we do not wish to undervalue the other roles that educators have. Unlike educators, judges do not have the task of educating citizens about the law, writing laws and policies themselves, or giving formative feedback if citizens detour from the laws. Our focus in this paper is on ensuring that learners are treated fairly when they are at risk of failing a high-stakes assessment or of being excluded from further training. In such situations, educators may be acting quasi-judicially by dealing with a contested issue, assessing the evidence, listening to arguments from both sides and formulating a decision. In these respects, we sought to test the relevance and transferability of judicial practice, articulated as key decision-making principles, to education practice.

Limitations

We do not claim that our identified principles are the only principles that could be applied to decision-making. Other principles will be just as, or more, pertinent in different contexts. Our principles were chosen in relation to our experiences with contested assessment decisions. We aimed to demonstrate whether a novel interdisciplinary method could assist in assessment decision-making by identifying a few, key principles and applying them to exemplars which could be relevant in different contexts. Thus we did not seek to comprehensively identify all potentially relevant principles or the situations where they could apply, nor to provide a general introduction to the law, compare legal systems or to explain detailed legal implications of case law across different jurisdictions. Our approach is nevertheless based on the common law heritage of the English-speaking world, including UK, USA, Canada, Australia, and New Zealand and other former British colonies. The principles of fair hearings hold throughout these nations, ^12,31 and also underscore the principles we identified. Thus, while we make no claims for universality, our principles are transferrable to many settings, for example to US case law as outlined in a recent review of competency decision making.¹⁰ While invoking legal principles may have secondary benefits in reducing the risk of legal and courtroom appeals, this was not our primary intention. Given the different functions of medicine, education, and law in society, we also draw limits on analogies between legal case-based reasoning to the practice of clinical medicine and medical education, by focusing only on a few principles relevant to assessment decision making.

While our process for developing vignettes was consistent with the recommended steps ²⁷ and we aimed for authenticity by basing vignettes on experience and relevant literature, hypothetical vignettes are at best exemplars. Similar to real life, our vignettes are open to different and unintended interpretations and the information provided is incomplete. Our vignettes focus on the decisions of individuals, rather than group decision-making as occurs in competence committees, and as recommended for high-stakes decisions in contemporary assessment practice.¹ Individual events and decisions, however, reflect the range of evidence that competence committees must consider in reaching their decisions. Our purpose was to offer principles for how to evaluate and use such evidence. While vignettes may have limited transferability to different contexts, our aim was to show how principles can be applied in a specific context, similar to how judicial reasoning is documented and made explicit in common law for readers to consider in relation to their own situations.

Strengths of DI include the potential for transformation of epistemic traditions into new knowledge and for equal inclusion of multiple perspectives from disparate disciplines.¹⁹ We did not, however, test our findings with stakeholders such as students and educators, but DI has been used to engage students¹⁷ and improve group decision-making,⁴⁵ provided the questioning has structure and direction, there is trust, and power differentials between participants are minimized. Inclusion of these perspectives, and testing the principles and method on real-life scenarios in future research¹⁹ could advance understanding, and promote collaborative processes in panel and committee deliberations.

Conclusion

While many assessment decisions in medicine may be robust, what judicial principles reiterate is the need for transparent explanations of the reasoning behind decisions to all persons. That is, not just what was decided, but how was it decided, what views were considered and what processes were followed. In making assessment decisions, educators and judges share a belief in the importance of due process when applying professional judgment. Judicial practices remind educators of fundamentals such as seeking different perspectives, corroborating evidence where possible and to be timely, open and transparent when informing of options, consequences, and decisions. While theorization and practice in each discipline may diverge, both medical education and legal systems are founded on community standards and values such as fairness and accountability to individuals and to society, thus ensuring that decisions which are both fair and seen to be fair, can be made.

Declarations

Ethical review, consent to participate or publish and data availability are not applicable. The authors do not have financial or competing interests or benefits to declare. No external funding was provided.

Author’s contributions

WCYH and TW contributed to the design, conduct and drafting of the study. HD contributed to the conduct and drafting of the study. All authors revised the work, approved the final version, and undertake to be accountable for all aspects of the work.

Funding

The author(s) reported there is no funding associated with the work featured in this article.

Notes

1 See, for example, Minister for Immigration and Multicultural Affairs v Yusuf (2001) 206 CLR 323 at 351 in which the Australian High Court called mistakes of this kind “errors of law.”

2 See Hindi v Minister for Immigration and Ethnic Affairs (1988) 20 Federal Court Reports 1 at [13] which required relevant considerations to be considered; see also 18 U.S. Code § 3553 section (a)(1) which requires mitigating factors to be considered on sentence. (Similar provisions apply in other common law countries.)