Managing assessment during curriculum change: Ottawa Consensus Statement

Abstract

Curriculum change is relatively frequent in health professional education. Formal, planned curriculum review must be conducted periodically to incorporate new knowledge and skills, changing teaching and learning methods or changing roles and expectations of graduates. Unplanned curriculum evolution arguably happens continually, usually taking the form of “minor” changes that in combination over time may produce a substantially different programme. However, reviewing assessment practices is less likely to be a major consideration during curriculum change, overlooking the potential for unintended consequences for learning. This includes potentially undermining or negating the impact of even well-designed and important curriculum changes. Changes to any component of the curriculum “ecosystem “- graduate outcomes, content, delivery or assessment of learning – should trigger an automatic review of the whole ecosystem to maintain constructive alignment. Consideration of potential impact on assessment is essential to support curriculum change. Powerful contextual drivers of a curriculum include national examinations and programme accreditation, so each assessment programme sits within its own external context. Internal drivers are also important, such as adoption of new learning technologies and learning preferences of students and faculty. Achieving optimal and sustainable outcomes from a curriculum review requires strong governance and support, stakeholder engagement, curriculum and assessment expertise and internal quality assurance processes. This consensus paper provides guidance on managing assessment during curriculum change, building on evidence and the contributions of previous consensus papers.

Keywords:

• Assessment
• curriculum change
• health professional education

“For every assessment question there is often more than one answer that could be correct.”

Cees van der Vleuten, personal communication, circa 2000

Introduction

Consider how students discuss their health professional education (HPE) programme – “Our programme is mostly alright but some stuff seems irrelevant to what we have to do.” “Some of the clinical experiences are just watching, not doing, yet we have to get a lot of things signed off by people we hardly see”. “The assessment relies too much on end of year exams, not what we do through the year”. “We do not receive much feedback and most of it is unhelpful“. “The exam questions seem to be just a collection of random questions unrelated to what we do. “Dr X always writes questions about his/her pet interests rather than what is in the curriculum so we have to attend his/her lectures to work out what to study”. “A new curriculum was introduced to make the course more current and interesting but it is safer to focus learning on what is assessed”.

Practice points

• Curriculum outcomes, content, delivery and the assessment of learning are closely inter-related within a curriculum “ecosystem”.

• Understanding the ecosystem and engaging stakeholders is critical to the success of curriculum or assessment change.

• Change in any component of this ecosystem should consider the potential impact on other components as there may be unintended consequences that impede learning.

• Assessment practices within a curriculum change must respect and align with internal and external drivers of change.

• Assessment may be used as one quality assurance approach, but substantial curriculum changes may require substantial assessment re-design to guide achievement of graduate outcomes.

• Adequate and sustainable outcomes on assessment change relies on strong governance and support, stakeholder engagement, expertise and quality assurance processes.

What do such comments mean? Why and by whom are they perceived in that way? What is a “good programme”? Who is the best judge of what is “good” or “bad”?

Health professional education programmes are often complex because they provide more than a university degree. Graduates are expected to be capable of entering the workforce and providing care. Medical students are eligible to commence specialist training. Health care is important to the whole community, so funders, regulators, employers, profession groups, students and the general population are interested in programme design and quality. Central to each programme is the curriculum, defined as “learning opportunities provided by the institution to meet goals established by internal and external bodies” (Merriam-Webster online Dictionary, 2024). These goals are often expressed as graduate outcomes and reflect preparation for future roles. Central to the curriculum is how learning is assessed, because learners focus on curriculum components known to be assessed (Newble and Jaeger 1983; Bezuidenhout and Alt 2011). Assessment often dominates learner thinking and behaviour. More senior peers are asked for advice, past papers are sought and subscriptions to external examination banks provide practice, suggesting competing undercurrents in assessment that may skew learning towards passing exams rather than achieving graduate outcomes. Managing these undercurrents within a dynamic healthcare system can be difficult, depending on participants, faculty, students and the mix of clinical experiences (Hays et al. 2020). Even subtle or minor variations may mean the learning varies within and across learner cohorts and locations, endowing each programme with a “life” of its own. Contexts vary even more across national boundaries, where different cultures and healthcare systems may make apparently identical curricula quite different at delivery.

Previous Ottawa Consensus papers have provided guidance on how to design assessment practices (Norcini et al. 2011; 2018; Heenaman et al. 2021; Fuller et al. 2022, Boursicot et al. 2011), but not in the context of curriculum change. This paper focuses on strategy for why, how and when to apply available assessment design guidance when learning activities change.

We are therefore approaching curriculum design from an assessment perspective, explaining the importance of assessment in supporting changes to learning. We acknowledge that most of this is based on the literature concerning primary medical qualifications rather than other health professions and postgraduate programmes, where there is less documentation. However, while there is unlikely to be a single, correct set of assessments for all contexts, most health professional education (HPE) programmes are likely to reflect similar principles.

Purposeful instructional design

We start by explaining the close relationship between assessment and learning. A health professions education (HPE) programme is much more than a list of what must be learned to complete a course or programme. It is a complex, dynamic entity with three main components: the “content” to be learned, the process of delivering (teaching and learning strategies) and the assessment of learning achievement, all guided by planned graduate outcomes and dependent on substantial healthcare system experience. The content includes the knowledge, skills and behaviours expected of new graduates from the programme, often referred to as the syllabus (Burton and McDonald 2001). This content is usually organised within curriculum domains, which are conceptual groupings of related material (Behar and Ornstein 1994) that reflect the professional context in which learning is applied and the graduate outcomes to be achieved. Domains usually overlap to a variable extent because they combine components of the curriculum differently, according to graduate outcomes. Achieving graduate outcomes requires satisfactory achievement in all curriculum domains, with all curriculum content and intermediate learning outcomes mapped to graduate outcomes. This is known as constructive alignment (Biggs 1996). The focus of assessment should be for, of and as learning (Dann, 2014), moving through junior to senior years from “knows how” and “shows how” to “does” and “is” in the revised Miller”s Pyramid and from “remember” to “evaluate” and “create” in the revised Bloom”s Taxonomy (Anderson and Krathwohl 2001; Cruess et al. 2016). While there are many assessment methods and formats, this progression requires increasing use of workplace-based assessment of how competencies are chosen and applied, the basis of the concept of entrustable professional activities (EPAs) (Norcini and Burch 2007; Boursicot et al. 2011; ten Cate et al. 2015), particularly by graduation and into postgraduate education.

Although often considered independently, including listing as separate standards in most accreditation standards documents, curriculum content, teaching and learning methods and assessment practices are so closely interwoven that they should always be considered together as part of a curriculum ecosystem. Conceptually they form a “tripod” that supports learning the “right things” in the “right way” (Kibbe et al. 1994), to which could be added “at the right time” and “for the right purpose”. Change in only one or two legs of the tripod is likely to affect balance. This close relationship is depicted in Figure 1.

Figure 1. The curriculum ecosystem: content, teaching and learning (T&L), assessment contextualised with health professional roles, the healthcare system and the wider community.

Ideally, all three curriculum components provide learners with consistent messages about what and how to learn. This is the core of constructive alignment (Biggs 1996), a concept that may be extended to include a vision or mission for the programme, selecting certain kinds of students and programme evaluation, but those facets are not within the scope of this paper. The importance of constructive alignment is that it should be built into the entire curriculum ecosystem. In this paper the use of the term “curriculum” refers to that ecosystem, although our primary focus is on how learning is assessed.

Change in just one part of the curriculum ecosystem may have unintended consequences for learning. For example, a “hidden” curriculum may develop, where clinical experience, assessment or both drift away from alignment with curriculum content and/or graduate outcomes (Jackson 1968). This may have a negative impact on learning, with learners exposed to competing pressures that are difficult to control. The first pressure is workplace culture, which has a significant influence on student learning delivery and outcomes (Lempp et al. 2004). Regardless of what curriculum documents state, learning depends to an extent on where, when, how and by whom the curriculum is experienced. The second is how learning is assessed. This may exert more influence on learning than curriculum content or delivery – “assessment drives learning” (Newble and Jaeger 1983). Further, learning approaches may be different when scores contribute to progress decisions rather than “just” provide feedback on progress (Rapauch et al. 2013). A concrete example of this is the impact of standardised national examinations, which have shown in several contexts to improve factual knowledge more than higher cognitive processes such as understanding and application in practice (Bezuidenhout and Alt 2011; Jürges et al. 2012). While they may provide confidence that at a single moment in time examination (“SMITHEX”) all candidates meet the same benchmark standard (Dave et al. 2020), students may focus on short-term learning to pass assessment that “matters” (e.g. eligibility for a licence) rather than learning for future practice. Learning journeys are longitudinal and incremental; what may be more important is assessing progress and trajectory, which is more like a slope than a series of steps. Assessment should match and facilitate that learning progress. The third is that external pressures from accreditation agencies, employers or funders may request changes to produce “better” graduates and a more inclusive workforce, but responses do not necessarily penetrate to the point of programme delivery. Examples include the increasing emphasis placed on professionalism, work readiness and patient safety, all laudable attributes that are not easy to assess. Further, there is some evidence that clinical assessors of students close to graduation may focus on system factors, such as teamwork, safety and trustworthiness, rather than traditional individual clinical competencies (Malau-Aduli et al. 2021).

Health professional education programmes should not be static. Scientific development improves continuously and clinical practice evolves in response to new content, new learning experiences and new roles. Future community needs and professional roles must be considered, more likely reflecting the perspectives of funders, regulators and employers. Advances in technology improve access to learning resources, current clinical information andnew assessmentformats, but should be tools rather than main drivers of development. The “shelf life” of a curriculum is difficult to measure but all health professions curricula should be regarded as “living” documents that require regular review to maintain currency. All changes should be evidence-based, described clearly and evaluated. This is “healthy” curriculum development.

Not all curriculum changes are necessarily “healthy”, particularly if implemented on the run. Changes in teaching faculty may bring different perspectives to existing content, new ideas and energy, but sometimes directed towards personal interests rather than proven needs. Learners may request additional lectures on more complex topics, but perhaps learning resources should be reviewed rather than increasing didactic activities. A “new” assessment method or format may gain popularity and be introduced without empirical evidence or consideration of the broader curriculum ecosystem and potential unintended consequences. For example, even minor changes can result in “curriculum creep”. All curriculum changes should be developed appropriately and implemented uniformly across the programme. More substantial change may be sparked by new leadership that wants to motivate/bring together faculty around an important project, an accreditation report that recommends or requires curriculum reform, or a major change in circumstances that affects curriculum delivery. Examples of major changes include the recent COVID-19 pandemic, international conflicts and large natural disasters that disrupt “normal” healthcare and HPE delivery. These reasons are not mutually exclusive and may relate to both internal and external pressures. External pressures may be regarded as more urgent due to reputational risk.

An explanation commonly used for not changing assessment during curriculum change is that assessment can be used as quality assurance by showing that a curriculum change does not disadvantage learners. Many curriculum delivery initiatives, such as Longitudinal Integrated Clerkships (LICs) have taken this approach (Worley et al. 2001; El et al. 2019). While this assertion has some strengths in support of innovation, substantial curriculum changes may require substantial assessment re-design to guide learning.

In this paper we use scenarios that are based on real although de-identified examples (see Table 1) to demonstrate how the principles may be applied in a variety of contexts (Table 2). These principles are based on both the literature, which is skewed heavily towards primary medical education and the pooled experiences of the authors, which although broader is still medicine focused. Primary HPE qualification programmes tend to have more stringent external accreditation standards and processes, while postgraduate programmes tend to be more in the hands of the individual professions, producing greater variation both nationally and globally in curriculum and assessment approaches. Despite these limitations, we believe that the principles may be valid at all levels of education for all health professions and have included scenarios reflecting both specialty and continuing professional development contexts.

Table 1. Scenarios for consideration of assessment change.

A	A major curriculum change	A new Dean decides to modernise the existing, successful medical programme by changing the curriculum from a “traditional”, modular, subject-based pre-clinical and clinical curriculum to one with strong horizontal and vertical integration, early clinical skills teaching and early patient contact. While this curriculum looks sound and both teaching faculty and students look forward to its introduction, the separately managed assessment office plans to continue with the same assessment practices because they have “stood the test of time” and represent thousands of hours of development. Further, there may be need for a complete new accreditation cycle, with some reputational risk and substantial resource requirements.
B1	A “minor” change:  a. Faculty driven  b. Student driven  c. Profession driven	An academic surgeon with a strong personal interest in the use of point-of-care ultrasound machines (POCUS) believes that all medical graduates will need these skills. He has piloted this in his own surgical team and now wants all students to demonstrate competence to “pass” surgery. Students return from an international HPE conference inspired by presentations on curriculum and assessment practices adapted during the COVID-19 pandemic. Their own experience during the pandemic was poor because current resources were simply put on line, but usability was constrained by failure to deploy advances in interactive technology that engage younger adults. The concept of changing design rather than just delivery raised interest and students want their programme to adopt a similar design approach rather than reverting to pre-pandemic methods. To meet changing national competencies determined by the dental council, dental faculty reviewed both curriculum delivery and assessment of their undergraduate programme. As dental graduates are licensed for independent practice upon graduation, the team decided to use entrustable professional activities (EPAs) gained and assessed in a student-led group dental practice, anchored by competency-based assessments linked to daily professional tasks. In all three cases submissions were made to the Curriculum committee and accepted in principle. A review committee was appointed to explore this further and develop plans to manage change. Faculty and the students were appointed to a review team. These scenarios show that “bottom-up” creativity is welcome and best managed through governance structures
C	An adverse external accreditation report	Following a routine re-accreditation cycle with the relevant national authority, the Dean is advised that the curriculum requires updating. The 20-year-old curriculum was for many years regarded as one of the best in the nation – contemporary, innovative and popular with students and local employers. Other programmes are perceived now to have “caught up” or even surpassed its quality. The curriculum has evolved but changes have been ad hoc rather than strategic. Most of the innovators have moved on or retired and current faculty members have been so wedded to the “successful” model that there is resistance to change
D	A new specialty emerges	A new group develops from an existing specialty and believes strongly that members require training that is different. They work through professional groups and the regulator, requiring detailed submissions about these differences
E	A rapid evolution in clinical practice	In the late 1980s and early 1990s in the US, open cholecystectomy (OC) was the standard of care for the removal of a diseased gallbladder. In the late 1990s laparoscopic cholecystectomy (LC) was introduced and soon became the preferred method in uncomplicated cases and then spread to other abdominal procedures. However, surgeons in practice had limited exposure to laparoscopy and there was increased demand for CME specific to this procedure

Table 2. Application of principles demonstrated by scenarios.

	Scenarios
Principles	Major curriculum change	Minor curriculum change	Adverse accreditation report	New specialty	Rapid change in current practice
Governance	Systems change at school/health system level, led by leadership and supported by stakeholders. Consider Business Process Reengineering (BPR)	Faculty or student driven, taken to curriculum committee and endorsed	Clear documentation and approvals of curricular and assessment changes based on governance structure	A professional group approaches the regulator with a proposal	Initial pressure from technology developers, employers and funders. Invite representative into discussions.
Purpose	Enhance efficiency and alignment with national/global level healthcare needs. Reduce costs without reducing quality	Include specific ideas about technology enhanced learning and assessment to increase “remote” delivery	Address specific requirements and recommendations to produce a high quality programme based on current evidence and practice	Establish a new specialty recognised by regulators, funders, the professions and patients	Improve clinical efficiency without compromising safety
Constructive alignment	Create new curriculum map/ blueprint to ensure curriculum content, delivery and assessment are aligned	Revise curriculum map/ blueprint to ensure curriculum content, delivery and assessment are aligned	Revise curriculum and assessment map/ blueprint to ensure alignment of new assessment methods and overall assessment strategy	Create new curriculum map/ blueprint aligned with roles and healthcare practice	New CPD training modules developed and competence assessed, also introduced to specialty training programmes
Systematic and comprehensive	Focus on systems level change to assessment of curriculum outcomes, including appropriate methods across all Domains to assess breadth of graduate outcomes	Not all assessment can be remote. Re-design assessment to include minor change but maintain balance and breadth of assessment.	Revise assessment strategy to ensure more methods, judges and frequency. Consider more frequent assessments and feedback	Assessment must be comprehensive across curriculum Domains. Ideally included in purposeful curriculum design	While apparently isolated to specific procedures, also had impact on other procedures and on use as diagnostic tool
Methods	Start with clean sheet, considering more methods according to utility	Review potential methods and technology to choose the most appropriate	New assessment method introduced with resources and know-how to implement	A range of knowledge, skill and behaviour assessments throughout training with emphasis on WBA	Simulation with direct observation and then workplace supervision until assessed as competent.
Standards	Should conform to national standards. Consider exceeding to “future proof” and extend shelf-life	Should conform to healthcare or HPE standards	Should comply to all standards set by the regulatory boards	Developed based on other similar specialties	Minimally invasive surgery added to specialty training and incorporated in standards
Rules of progression	An opportunity to adopt evidence-based best practice across new, broader range of assessment data points	An opportunity to improve and make more transparent	Developed to ensure that student progress based on standards set in a fair and transparent manner	Jointly developed by the profession and the new specialty group and employers	All surgeons require training before permitted to adopt techniques
Quality Assurance	Strong internal QA processes will facilitate accreditation by providing comprehensive reports on how graduate outcomes are measured	Seek data to provide evidence of success at and graduation and into the workforce	Seek data to provide evidence for longitudinal evaluation with continuous quality improvement initiatives	The new specialty should gather data, as will healthcare institutions	Monitor patient outcomes (Length of stay, readmissions, complications), primarily within healthcare system

Principles to guide successful and sustainable assessment change during curriculum review

Leadership, governance and ownership

Clear governance structures and delegations are required to guide any curriculum review process from preparation through to completion. Change management is not covered in detail here but success requires leadership, engagement, organizational “readiness”, clear outcomes, clear decision-making and data collection to provide provenance of change and its evaluation (Gelaidan et al. 2017; Ford et al. 2021; Law 2022). Leaders should be independent of “vested” interests and focus on aspirations to improve healthcare outcomes and meet accreditation requirements as conduits for activity that unite teams. Clear, accepted, unequivocal procedures facilitate decisions that are more likely to be implemented. Respectful disagreement should be encouraged to refine ideas and strategies. Are decisions made by consensus or majority vote? If so, do all participants have voting rights? How do options fit with institutional regulations, professional norms and community expectations? Data collection to track changes and document reasons for change is essential to demonstrate curriculum provenance and evaluation of the impact of changes, particularly for innovative changes. In major reviews there is potential for distraction and disaffection, with students from the “old” course feeling neglected or disadvantaged. It may be wise to have two separate curriculum and assessment teams – one each for the “old” and “new” programmes to improve messaging clarity.

From an assessment perspective, a key issue is the appointment of a senior faculty member with, or with access to, assessment expertise to monitor the potential impact of any curriculum change on assessment practices. While there may not be a single answer to every assessment question, the focus should be on finding “better” ways to guide learning and measure achievement. Approaching curriculum or assessment change should consider what we call the 6 Ps, listed in Table 3.

Table 3. The 6 Ps of engagement in curriculum and assessment review.

Permission	Participants need explicit permission to think outside the square and engage in creative scholarship. Major curriculum or assessment changes may come along only once or twice during faculty tenure, so consideration of all options may facilitate better outcomes that make a difference. Clear direction and support from senior leaders may harness enthusiasm.
Promise	Although pressures for change may come from vested interests for different reasons, effort should be focused on aligning changes with improvement in patient care outcomes. All participants should support this goal. Negativity around the existing curriculum should be avoided.
Power	Educational and healthcare institutions are hierarchical, combining collegial and corporate management styles. Early reflection on power dynamics is critical. Power sources may not be obvious and may vary from individuals to cohesive teams or factions. A senior, respected faculty member should be responsible for monitoring the impact on assessment. Resistance may decrease where change does not affect influence and resourcing. The power differentials between senior faculty, junior faculty and students should be abandoned for change discussions.
People	Successful change requires constructive engagement with stakeholders who value the outcome but may not have assessment expertise. Community voices are increasingly important and so are students, especially on how curriculum changes may impact their progression. A clear mapping of stakeholders, their expectations and options for change may increase transparency about integration of their contributions into the final narrative, demonstrating co-creation and strengthening relationships. Investing in faculty development in HPE assessment may increase in-house expertise and assist with translation into practice
Practice	Participants need a shared understanding of the prevailing assessment practices to contribute to changes in scope or context of assessment that maintains balance and alignment. Relevant faculty development is helpful but access to additional HPE expertise is essential. Sending faculty to observe other well-respected programmes may foster greater understanding of how well different approaches work.
Provision of resources	Continuing quality improvement may be implicit in the roles of teaching faculty but is usually not explicit in job planning. Change requires resources for both curriculum and assessment development. faculty and students to commit time (“buying” additional faculty as backfill, a gap or project year for students) without detracting from other roles. A detailed, costed project plan with timelines should be developed and approved by the leadership.

Curriculum governance structures should include faculty members, senior students, recent graduates, employers of graduates and patient representatives to oversee processes and to provide opportunities for contribution. Considerable energy and effort are required to provide a “return on investment” for all stakeholders. Debate, sometimes robust, is normal, particularly early in the process as differing perspectives are voiced and mutual understanding is achieved. Respectful discussions in a contest of ideas should not hinder the prospects of participants. New ideas should be encouraged. Bringing in an external consultant with awareness of developments in other medical programmes may increase awareness of current trends. “Looking outside” may be useful learning: consider sending review team members to visit other programmes that are well regarded. Regular progress reports, particularly to students and faculty, should maintain engagement and encourage feedback. Consensus should be the goal but is not mandatory, so long as all work towards implementation. These steps should improve the acceptability, achievability and quality of changes.

The role of faculty is much broader than providing information to learners (Harden and Lilley. 2018) They possess the content, context, delivery and assessment expertise that provides the programme. Their engagement is critical to successful change (Cintra et al 2023). They need to engage in any review discussions and, ideally take some ownership of outcomes. However, they are busy people and continuing quality improvement (CQI) is usually not explicit in position descriptions. Participating faculty will need dedicated time to commit to the task. Additional faculty may be required to ease the workload of existing faculty. This may be an ideal time for faculty development activities that expand educational expertise and enhance career prospects.

The role of learners is much broader than receiving information (Karakitsiou et al. 2012). They are both the initial “consumers” of HPE programmes and the future clinical and educational workforce. Students can provide valuable perspectives and so should join review teams. Their participation may ensure that any new assessment strategy encourages learning towards achieving graduate outcomes, reducing resistance and facilitating implementation and sustainability. For a major review recruiting one or more students to the project team (e.g. as an intercalated HPE project) may add value to both the programme change and the student(s) future career.

Purpose

There are three levels of purpose. The purpose of the overall programme is unlikely to change, focusing on providing an appropriate healthcare workforce as defined by the regulator. Where changes are proposed for curriculum and/or assessment, the purpose of each change should be clear. Where changes potentially require different assessment methods, the purpose of each method should be clear. Curriculum change is an opportunity to re-visit the overall assessment philosophy and the purpose of the assessment, building on empirical evidence and reflecting contemporary trends to improve HPE programmes. Is the primary goal to guide learning, to demonstrate particular workforce outcomes or to achieve national accreditation status? All are important yet may require different applications of similar resources. Whether it is addition of contemporary content or an amendment to graduate outcomes, the whole curriculum ecosystem may be affected. Clarity and shared understanding of goals and objectives of the change are required to facilitate moving beyond the “what” and “why” to “how”. Is this “routine maintenance”, where a focused change is required in response to a specific trigger? Is it a major overhaul, such as introduction of a parallel curriculum (different sites or placement models) with identical graduate outcomes? Is the change effectively a new programme, such as changing from lower to higher integration (or vice versa). The more substantial the review, the more resources, educational expertise and faculty-wide engagement are necessary. Where possible, adding new content should be resisted due to the risk of curriculum or assessment overload. Replacing less important or older content is preferable.

Constructive alignment

Constructive alignment is central to purposeful design and any curriculum review should aim to correct or improve the consistency, balance and alignment of programme components to support and strengthen learning. Even “minor”, apparently “isolated” changes that address specific issues may impact other programme components. In particular, any changes to curriculum outcomes and delivery are likely to require revision of the assessment strategy. More substantial changes may require a programme-wide review of all curriculum components. For example, changing from lower to higher curriculum integration requires a matching change in assessment across the curriculum. Further, a review of any part of an assessment schedule requires a review of the whole schedule to maintain alignment and balance. The assessment blueprint and sampling should be based on the curriculum blueprint to demonstrate constructive alignment (Raymond and Grande 2018).

Systematic and comprehensive

Every aspect of the curriculum, across all domains, should be considered as eligible for assessment, guided by the graduate outcomes (Schuwirth and Van Der Vleuten 2004). This includes a wide range of cognitive, procedural and behavioural attributes. Assessment should go beyond relying on the “easier to measure” attributes, such as knowledge and simulated skills, to incorporate other important attributes, such as elements of professionalism and self-care. The current approach is to conduct more frequent assessment throughout programmes with multiple assessors, methods and contexts to “shine a light” on learning progress across the whole curriculum (Norcini et al. 2018). A more elaborate model is programmatic assessment, which includes consideration of the whole curriculum nexus to focus on learning (van der Vleuten et al. 2012; Heeneman et al. 2021; Torre et al. 2021). Just as delivering a curriculum is feasible for only a sample of the “universe” of requirements for early clinical practice, assessment is feasible for only a sample of the curriculum, based on a programme-wide assessment blueprint. Appropriate sampling strengthens validity and sends messages to learners and teachers that the whole curriculum is important. Learning portfolios are a suitable place to gather complementary information that builds a “profile” of graduates, although much depends on what is included and how it is evidenced. The role of learning portfolios in decision-making continues to be a work in progress (Buckley et al. 2009; Hong et al. 2021; Tan et al. 2022).

Selection of individual assessment methods

A wide range of assessment methods and formats is available. As with assessment development in general, the choice of individual methods should be based on the purpose. Content, delivery methods and context of the changing curriculum. A convenient model to consider is the assessment utility of individual methods (Schuwirth and Van Der Vleuten 2004). While the traditional focus may be on methods with higher reliability, those with higher validity and educational impact are important for “harder to assess” attributes and outcomes such as professionalism, patient safety and self-care. A “composite reliability” approach should be adopted to ensure that learning progress is achieved across all Domains (Moonen-van Loon et al. 2013).

Adopting a new assessment method should be part of a purposeful design. Care should be taken not to simply increase the assessment workload although if this includes previously non-assessed components some flexibility may be required. Assessment methods should be replaced on a “like for like” basis to reduce unintentional duplication or narrowing of overall assessment. For example, a clinical assessment method should be replaced by another. If workplace based assessment (WBA) are added to objective structured clinical examinations (OSCE) it may be advisable to decrease the number of OSCEs and sample items for both from the same blueprint, focusing on methods, individual item quality and assessor training. Further, some assessment changes may require substantial re-design. For example, moving from competencies to EPAs, a contextualized “meta-competency” that is linked to workplace roles and tasks, may require much broader, multi-method assessment over an appropriate time period (ten Cate et al. 2015).

Clarity is required about how each assessment contributes to achieving graduate outcomes. Curriculum change is an opportunity to increase the quality and quantity of feedback provided; the lack of this is a common student complaint (Lefroy et al. 2015), whether the primary driver is “for”, “of” or “as” learning. Students may behave differently for each of these purposes. All assessment can be “for” learning and so central to career development. Progress decisions should consider a wide range of different assessments with different methods and assessors. Consider making some assessments “hurdles” during learning rather than graded assessments at the end of the stage or year. This spreads the assessment workload and ensures that important but “harder to assess” attributes are included.

Other options for change should be explored as there may be more than one approach to addressing identified issues. For example, increasing the number and the quality of assessment items, adding clinical scenarios to written items, increasing the number of clinical assessors and improving assessor training in current methods may improve assessment utility more than changing methods (Schuwirth and Van Der Vleuten 2004; van der Vleuten and Schuwirth 2005). The role of technology in assessment ideally should be driven by educational needs. Technology advances rapidly and provides useful assessment tools and management software. However, technology should be an enabler, should not dictate assessment design principles and requires contingency planning as outages occur occasionally (Sen Gupta et al. 2021)

The importance of individual learning styles is controversial, but learners may adapt their learning approaches in response to assessment practices (Newble and Jaeger1983; Wilkinson et al. 2007). For example, a decision to increase the number of MCQ assessments for clinical rotations may encourage rote learning from textbooks rather than taking opportunities to engage in clinical practice. There may be some generational influences with students and faculty usually a generation apart, so there may be differences in preferences and understanding (Reyes et al. 2020). Given the diversity of students joining health professions education, consideration should be given to the potential for selected assessment tools unfairly disadvantage particular groups of students (de Lisa and Lindenthal 2012; Tai et al. 2023). Current medical school entrants may be more interested in “co-creation” of curriculum content, delivery and assessment (Plochoki 2019; Twenge 2009). This interest and energy, if captured, may be an asset for change processes.

Agreed national standards and graduate outcomes should guide the scope of assessment

Curriculum changes should adhere to clearly defined national standards for the relevant jurisdiction and profession. For medicine, these are focused on graduate outcomes, usually based on input from regulators, funders, the profession and the broader community, although overseen by the regulator. Programmes may have additional graduate outcomes for their local context, branding purposes, such as research proficiency, work-readiness or scope of practice. These outcomes should be assessed but not at the expense of the national graduate outcomes. Future-orientation may be important and should be considered, e.g. impact of climate change, planetary health and evolving roles of graduates. Students are interested in these topics and may contribute strongly to discussions if invited, but they are difficult to assess. Another area where the assessment programme can “drift” away from the revised curriculum relates to learning outcomes for each year/phase/stage. These should be calibrated carefully and mapped to graduate outcomes, particularly if they represent progression decision points. This contributes to the consistency of assessment practices throughout the whole programme. Formal standard setting methods should be applied as appropriate to each assessment method (Norcini, 2005). Although imperfect, these methods may improve the precision of assessments, usually combining pre-hoc consensus judgments with post-hoc psychometrics, resulting in fairer decisions, particularly if stronger psychometric measures are adopted, such as Item Response Theory (IRT) (Schauber and Hecht 2020).

Progress decision-making processes should be evidence-based, clear and consistent

The management of candidates at the borderline between “safe” and “unsafe” to proceed is becoming increasingly important and yet may be highly sensitive to curriculum and assessment change. Two complementary issues must be considered when decision-making assessment data are changed – can the borderline be defined more precisely and can we improve how to manage candidates at, near or below borderline scores?

There are many sources of potential error in any score and this matters more at the borderline where decisions are made about learning progress. The goal of assessment is to identify and manage those sources of error so that candidate scores are true measures of ability with as few false positives and negatives as possible. Minimization of error may be summarized as “reducing the grey zone” between a clear pass and a clear fail (Schauber and Hecht 2020). The traditional approach is to award a pass if combining scores across methods reaches a defined numerical score, either criterion or norm-referenced, with some measure of precision expressed as either p-values or confidence intervals. In HPE the context is more complex than in most higher education degree programmes because many methods and assessors are used and assessment data include both quantitative and qualitative information which cannot be combined to produce a single numerical score. Further, the stakes in HPE are higher for candidates (careers), patients (safety) and employers (quality of care), so minimizing error and improving precision are paramount. There is increasing emphasis on assessment item quality, blueprinting, sampling and standard setting. Combining numerical scores should adhere to classical measurement theory – ideally all scores are “interval” measurements (Foulkes et al. 1994). Including qualitative information, including narrative and reflective components is essential but makes more complex the task of reaching an overall progress decision (Charon 2001; Wald et al. 2012). Ideally, all information is considered, based on consensus judgements by experienced assessors, with discrepancies discussed by a panel. Any weighting of methods or items should reflect what is necessary to make diagnoses or guide management, guided by graduate outcomes. Clarity is required for how all assessment data are managed. The use of the Standard Error of Measurement may improve predictability of borderline scores (Hays et al. 2008). Rules of progression should be consistent with overall university regulations to reduce successful appeals (Hays et al. 2015). Decision-making processes should be available to all staff and students. Even relatively minor curriculum or assessment changes can inadvertently alter decision-making algorithms, potentially reducing alignment and producing leniency for weaker candidates.

The second issue concerns managing candidates with borderline scores. Most HPE learners are academically strong and commence learning in highly competitive HPE programmes with an expectation that they will succeed. This places greater responsibility on learning institutions to identify learners not progressing as expected and offer them increased learning support. Evidence is mounting that borderline candidates are more likely to fail future assessments and are often identified early in HPE programmes (Cleland et al. 2008). Providing increased learning and pastoral support is now an expectation. In HPE poor academic progress is usually related to non-academic pressures that may be amenable to intervention (Hays et al. 2011). Increasing access is a current “hot topic” for students previously excluded from entry due to rigid selection and workplace ability factors. There is pressure to adopt “universal design for learning” (UDL) approaches that were originally developed for special education. The laudable aim is to improve inclusivity by increasing flexibility in teaching and assessment methods (Rao et al. 2014), but its application in HPE higher education remains uncertain.

Quality assurance

Two quality assurance processes are available to ensure that medical programmes meet the standards laid down for both higher education and the broader community that graduates will serve. Most HPE programmes are delivered within accredited higher education institutions (usually universities) which must maintain the quality of all education programmes through internal quality assurance processes overseen by senior faculty. The second quality assurance process, common to most professional programmes, is external to the university, managed by health professional regulation agencies and including broader professional, regulatory, employer and funder perspectives on contemporary roles of graduates. Both processes involve periodic reviews by a panel selected carefully to represent stakeholders and provide relevant expertise.

Internal QA

Internal QA is under the control of the relevant institution. The review panel includes representatives from other education programmes within the university and some external representatives of similar programmes in other universities, the relevant health profession and local healthcare providers and employers. The primary focus is scrutiny of policies, systems and procedures, showcasing best practices, identification of areas for improvement, and dissemination of lessons learned. This is achieved mostly by desk audit of documentation, with some interviews with faculty and students. Assessment is part of that focus, but there are often substantial differences between health professional and other programmes with respect to methods, standard setting and rules of progression. Internal processes must take account of external accreditation requirements, particularly those relating to future healthcare and community roles of graduates but focus on more generic higher education standards of curriculum design. Internal QA reviews are usually less intensive than external accreditation processes. Reports should be submitted as an appendix to support external accreditation review submissions, as this provides evidence of compliance with higher education standards, allowing accreditation processes to focus on compliance with HPE standards. An understated but substantial benefit of dual internal and external QA processes is that they inform each other. Further, the self-reflection component of external QA may encourage stronger internal QA.

Internal QA for postgraduate educational programmes is similar in principle although different in context. These programmes are often not delivered by universities so “institution” merges with “profession”. These are smaller entities which may not have the resources for comprehensive educational expertise, although the need for internal QA is perhaps even higher than for university-delivered programmes. The same principles should be applied.

External QA (accreditation)

Accreditation processes are under the control of the regulator for the relevant jurisdiction. The panels comprise entirely external members from other HPE programmes, the regulator and trained representatives of the community, the wider profession and recent graduates. The primary focus is on meeting defined national standards that include issues such as governance, sustainability of resources and engagement with the health care system. Curriculum, teaching and learning and assessment standards are explored carefully from the perspective of how well graduates achieve graduate outcomes and are ready to enter the profession. Assessment practices differ from those of the wider higher education sector; this may be an area of overlap and a degree of tension between universities and regulators. Accreditation standards for assessment are intentionally broad, encouraging providers to innovate so long as graduate outcomes are achieved and evidence-based current assessment developments are reflected. Current emphases include authenticity, WBA, applied knowledge, professionalism, and assessment for learning with regular, constructive and meaningful feedback. Constructive alignment must be demonstrated in curriculum and assessment mapping.

While the expression of medical education and assessment may vary between institutions within accreditation jurisdictions, variations may be greater across national boundaries, particularly amidst the resourcing realities in “global south” countries where accreditation processes may differ from those in “global south” countries (Nadarajah et al. 2023). Institutions may have greater autonomy and may choose to produce either a workforce to address local needs or graduates that will migrate to more attractive or safer countries. This does not mean that less resourced nations necessarily have lower quality programmes, as there are many examples of locally developed innovative and effective programmes (Antunes dos Santos and Nunes 2019). “Transplanting” programmes from well to less resourced nations does not necessarily ensure quality as curriculum delivery relies on the local healthcare system, which may be quite different to that of the “parent” programme. Even though most of the global health professional workforce is trained in global south countries, less is known about this diversity in those contexts because fewer than 20% of the publications in medical education come from those countries (Wondimagegn et al. 2021; Amaral and Norcini. 2019).

Conclusion

Curriculum review is relatively frequent, whether unplanned or planned, to manage the emergence of new knowledge and skills, developments in teaching and learning methods or new roles in the healthcare system. Changes may be driven by either internal (faculty or students) or external (employer or regulator) pressures. Reviewing assessment practices may be overlooked by debate on content or delivery, not recognising the centrality of assessment in guiding learning. There is potential for weakening constructive alignment and producing unintended consequences for learning, undermining or negating the impact of important, even well-designed curriculum changes. Changes to any component of the curriculum ecosystem - graduate outcomes, content, instructional design or assessment – should trigger an automatic review of the whole nexus to maintain constructive alignment. Powerful external drivers of curriculum change include national examinations, programme accreditation and healthcare system expectations. Internal drivers are also important, such as adoption of new learning technologies and learning preferences of students and faculty. Each assessment programme sits within its own context which may influence the outcomes of curriculum and assessment review. Achieving optimal and sustainable outcomes requires strong governance and support, stakeholder engagement, curriculum and assessment expertise and internal quality assurance processes. This consensus paper provides guidance on managing assessment during curriculum change.

Glossary

Curriculum ecosystem: Describes the complex inter-relationship of learning content, delivery and assessment that exists in the dynamic context of both higher education and healthcare systems.

Disclosure statement

The authors have no declarations of interest to report.

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.

Notes on contributors

Richard B. Hays

The authors are an international group recruited through AMEE networks to provide expertise, contribution from educators and learners, and representation of geographic and health systems diversity.

Richard B Hays, James Cook University, Townsville, Australia.

Tim Wilkinson

Tim Wilkinson, University of Otago, Christchurch, New Zealand.

Lionel Green-Thompson

Lionel Green-Thompson, University of Capetown, South Africa.

Peter McCrorie

Peter McCrorie, University of Cyprus, Larnaca, Cyprus.

Valdes Bollela

Valdes Bollela, Universidade Cidade de Sao Paulo, Brazil.

Vishna Devi Nadarajah

Vishna Devi Nadarajah, Newcastle University, Johor, Malaysia.

M. Brownell Anderson

M. Brownell Anderson, Universidade do Minho, Braga, Portugal.

John Norcini

John Norcini, FAIMER, Philadelphia, USA.

Dujeepa D. Samarasekera

Dujeepa D. Samarasekera, University of Singapore, Singapore.

Katharine Boursicot

Katharine Boursicot, Health Professional Assessment Consultancy, Singapore.

Bunmi S Malau-Aduli

Bunmi S Malau-Aduli, University of Newcastle, Newcastle, Australia.

Madalina Elena Mandache

Madalina Elena Mandache, University of Medicine and Pharmacy of Craiova, Romania.

Azhar Adam Nadkar

Azhar Adam Nadkar, Stellenbosch University, Capetown, South Africa.