R.I.M.E. and reason: multi-station OSCE enhancement to neutralize grade inflation

ABSTRACT

To offset grade inflation, many clerkships combine faculty evaluations with objective assessments including the Medical Examiners Subject Examination (NBME-SE) or Objective Structured Clinical Examination (OSCE), however, standardized methods are not established. Following a curriculum transition removing faculty clinical evaluations from summative grading, final clerkship designations of fail (F), pass (P), and pass-with-distinction (PD) were determined by combined NBME-SE and OSCE performance, with overall PD for the clerkship requiring meeting this threshold in both. At the time, 90% of students achieved PD on the Internal Medicine (IM) OSCE resulting in overall clerkship grades primarily determined by the NBME-SE. The clerkship sought to enhance the OSCE to provide a more thorough objective clinical skills assessment, offset grade inflation, and reduce the NBME-SE primary determination of the final clerkship grade. The single-station 43-point OSCE was enhanced to a three-station 75-point OSCE using the Reporter-Interpreter-Manager-Educator (RIME) framework to align patient encounters with targeted assessments of progressive skills and competencies related to the clerkship rotation. Student performances were evaluated pre- and post-OSCE enhancement. Student surveys provided feedback about the clinical realism of the OSCE and the difficulty. Pre-intervention OSCE scores were more tightly clustered (SD = 5.65%) around a high average performance with scores being highly negatively skewed. Post-intervention OSCE scores were more dispersed (SD = 6.88%) around a lower average with scores being far less skewed resulting in an approximately normal distribution. This lowered the total number of students achieving PD on the OSCE and PD in the clerkship, thus reducing the relative weight of the NMBE-SE in the overall clerkship grade. Student response was positive, indicating the examination was fair and reflective of their clinical experiences. Through structured development, OSCE assessment can provide a realistic and objective measurement of clinical performance as part of the summative evaluation of students.

KEYWORDS:

• OSCE
• clerkship
• assessment
• grade inflation
• RIME
• internal medicine

Introduction

Grade inflation is a common problem within core medical school clerkships [1–3] with an increasing trend of students receiving the highest available grade [4]. As many as 55% of Internal Medicine (IM) clerkship directors have reported grade inflation was a substantial issue at their school with 78% reporting it as a significant problem, and 38% feeling students have passed, who should have failed [2]. Much of the discussion surrounding grade inflation focuses on variations in clinical evaluations by faculty with limited variation attributed to the student [5]. Factors cited as impacting faculty evaluations include unclear student-faculty expectations, lack of sufficient exposure to the trainee, or fear of reprisal for poor evaluations [1,2].

Some clerkships have demonstrated success in creating more objective clinical evaluation tools through the integration of familiar frameworks, such as Reporter-Interpreter-Manager-Educator (RIME) [6] to guide faculty toward a more objective assessment of student competencies [7]. Others, such as Ryan, et. Al, have used an adaptive O-RIM (Observer, Reporter, Interpreter, Manager) model however challenges with using the framework can be seen in the need for ongoing faculty training or situations where students may be placed in a more observational role [8]. However, discussion of the consistent application of the RIME framework as an overall evaluation tool is ongoing, both in terms of defining the individual components [9] and its overall use as a summative evaluation tool rather than an observational framework for formative feedback regarding patterns, trends, and progress [10].

With the many challenges in standardizing faculty clinical evaluations, many programs rely on additional components to determine a student’s overall grade. A review of current trends summarized in a report by the Alliance For Academic Internal Medicine suggests that while over 90% of programs continue to use faculty evaluations as part of their assessments, many programs use a combination of assessment tools as part of their overall summative evaluations with 89% including performance on standardized examinations. Of programs surveyed, 61% indicated intentional limiting of the weight for standardized exams in grade outcomes with another 8% planning to implement this practice in the next year. Objective Structured Clinical Examination (OSCE) utilization is increasing with 46% of IM clerkships reporting using an OSCE or simulated patient encounter as a part of the clerkship grading process [11].

A longstanding method for evaluation, the OSCE can provide benefits to both students and faculty. OSCE use has been shown to be a more reliable assessment of a student’s clinical performance when compared to faculty evaluations [12]. One study found that clinical evaluation and OSCE scores were significantly and positively associated with NBME-SE scores and suggested that the OSCE may assess a broader range of clinical performance domains [13]. Further, student perceptions of an OSCE indicate a high level of fairness, support for OSCE as a main method of assessment, and support for the ability of an OSCE to mirror clinical scenarios [14]. In contrast to clinical evaluations, the OSCE may also provide improved methods for directed feedback as it can be separated from evaluator-student relationships that can develop in the clinical setting [15]. While OSCE is an increasingly utilized method for student performance evaluation, the optimal protocol for testing and evaluation is not clearly established however trends in OSCE use and development can suggest methods to improve their use in assessment. This can include increased reliability between OSCE and clinical performance as the number of stations increases [16]. Other programs have shown success in their use of expansive standardized patient encounters by implementing a step-wise evaluation process for clinical reasoning performance. Similar to integration of clinical evaluations, this was noted specifically where RIME was already established as a conceptual framework for student evaluation and feedback [17].

In the academic year (AY) 2019, The University of Kansas School of Medicine (KUSOM) curriculum reform reached the third-year clerkships introducing three distinct differences in summative evaluations. These included (1) all faculty clinical evaluations would transition from summative to formative pass/fail, (2) a requirement for an objective assessment of the student’s clinical knowledge outside of the patient care environment and (3) a student’s final clerkship grade would be a combination of said required objective assessment and the National Board of Medical Examiners Subject Examination (NBME-SE) score; the pre-reform final clerkship four-tiered grading system of Fail, Satisfactory, High Satisfactory, Superior would be replaced with three tiers – Fail (F), Pass (P), and Pass-with-Distinction (PD). For the Internal Medicine clerkship, the achievement of pass with distinction (PD) as an overall clerkship designation required meeting PD threshold in both the OSCE and NBME-SE. During this academic year AY20–21, the IM OSCE in was comprised of a standardized patient (SP) encounter evaluated the student’s ability to obtain a history, complete a physical examination, and provide an oral presentation of their findings along with differential diagnosis and confirmatory diagnostic recommendations. The OSCE was graded by a faculty using a yes-no checklist for 43 points.

Following the implementation of the curriculum reform, grade inflation concerns arose after AY20–21 OSCE data review demonstrated a highly negatively skewed distribution with a large majority of students scoring higher than 90% resulting in PD for the OSCE but only approximately 35% of students tri-campus achieved PD for the NBME-SE. This resulted in the final IM clerkship grade being primarily determined by a student’s NBME-SE score, with 35% of tri-campus students achieving PD in the clerkship.

The aim of our project was to develop an enhanced multi-station OSCE with the integration of the familiar stepwise RIME framework to establish a wider grade distribution curve reflecting a more comprehensive assessment of clinical skills vital to the mastery of the clerkship curriculum including clinical reasoning, data interpretation, prioritization, medical decision making, and implementation of care plans. In addition to increasing the clinical difficulty of the OSCE, additional goals including providing a more broad assessment across multiple clinical scenarios and by increasing overall points, help reduce the NBME-SE’s dominating effect on the final clerkship grade.

Materials & methods

OSCE concept and RIME framework

The RIME format was chosen for OSCE integration because it has been shown to be effective as a commonly used evaluation tool across all clerkships [8] and had historic use within our clerkship as feedback for student progression during their clinical experiences. The adaptability of the RIME format has also been demonstrated through EPA mapping [18] which provided the framework for creating patient encounters, each focusing primarily on assessing the associated clinical skills of the RIME schema (Figure 1). Educator was excluded in the context of our third-year curriculum, which is in line with other studies where students were mostly identified as Interpreters and did not include evaluation of the Educator level [8]. For each station, a separate organ system was chosen, ensuring that a broad range of diagnoses could be explored, while also protecting students with deficiencies in one area from being scored down across all three encounters. Following this, each station was assigned a method of information delivery and a method for evaluation. While the degree of difficulty of each station was varied, all stations were given the same point designation of 25 points, increasing the OSCE scoring to 75 points.

Figure 1. Overall concept of OSCE patient encounters adapted from Meyers et al.

Reporter station

In the context of Meyer’s article [18], the pre-intervention IM clerkship OSCE was evaluated and classified as a Reporter level. As most of the framework for this encounter already existed, it largely remained in the familiar two-phase format. The first phase focused on the SP encounter, centered around a new patient visit which was deliberately chosen as the lack of familiarity with the patient would promote thoroughness in obtaining a history and performing a physical examination. After completing the patient encounter, the student would move to the second phase, a new room where the student presented findings to a standardized patient in the role of a faculty physician. Formal grading of the presentation was completed by faculty from a video recording of the encounter after it was completed.

Interpreter station

The Interpreter station focused on a follow-up patient visit which was also divided into two phases. In the first phase, the student reviewed a recent progress note from the patient’s prior visit and then entered the room to interview an SP about their current symptoms. Following this, the students moved to a second phase outside of the SP to document their encounter. There, they were provided with additional objective data including lab work and other testing and asked to create a brief progress note that specifically required synthesis and creation of a prioritized problem list including all subjective and objective information about the patient. Data was provided to the learner after the SP portion of the encounter to avoid the need to have the SP prepared to answer multiple potentially in-depth questions related to the diagnoses or medical histories. Evaluation of the note was completed by faculty after the encounter with a provided checklist to review elements of the electronic student note.

Manager station

For the Manager station, a video handover was created that placed the student having just arrived at a room where a response had been called to help stabilize an acutely decompensated patient. The video handover was scripted in the traditional SBAR format [19] which is familiar to learners at our institution and has been adapted by others for use in providing handoffs and urgent or emergent situations [20]. The handover provided the clinical context of the scenario, ending with a prompt that directed the learner to review a series of multiple-choice questions (MCQ) that included data interpretation and management recommendations for the acute concern, questions related to ongoing hospitalization after stabilization, and finally questions about discharge planning. For this station, the encounter remained at a single physical location as both the video handoff and questions could be accomplished on the same device. Question lock-out functions did not let students return to previous answers to allow for the case to unfold.

OSCE implementation & RIME overlap

The proposed OSCE changes were confirmed with clerkship directors from all campuses to ensure the ability of each to provide on-site testing with capacity for the OSCE enhancement related to rooms, SP availability, and technology to deliver video content. Each campus confirmed that testable content was incorporated, but not explicitly revealed, into clerkship educational sessions. This ensured students were taught correct standards of evaluation and treatment and ensure all students were exposed to the same content regardless of clinical variability in both variation in patient populations or preceptor practice in the clinical environment. An expectation guide for the OSCE was also created to inform students of the general structure and requirements of the OSCE, as well as specific expectations for each station.

While core content for each encounter centered around a single RIME designation, some overlap was essential to ensure the cases felt comparable to the students’ clinical experiences. For instance, if attempting to have the Reporter station focus only on the associated EPAs for that group, this would not require a student to interpret their findings into a differential diagnosis or provide recommendations for diagnostic steps. This left the encounter feeling incomplete. While this overlap is important to evaluating across the RIME spectrum, extra steps were taken to reduce unnecessary information provided by the SP unrelated to their encounter and ensure the faculty assessment tools for each station focused on the primary RIME component.

Before final implementation, an OSCE pilot was completed using three fourth-year medical students who had completed the residency match process. Based on feedback, slight modifications to the OSCE were made. The OSCE was implemented at all three rotation sites at the start of the 2021–2022 academic year. No other significant changes to the IM clerkship rotation, such as clinical rounding structure or duration of the rotation were changed.

Evaluator reliability across campuses

As the curriculum transition placed an increased emphasis on summative grading for the OSCE, accurate and consistent scoring across the multiple-station tri-campus evaluation was important. An OSCE grading committee was created consisting of clerkship directors and/or skills lab physicians on all campuses.

Additional modifications were made in line with suggestions by Wilby et al., who noted more specific, objective, and depersonalized feedback and lower rates of hedging for grades by their OSCE evaluation faculty who did not have clinical relationships with students, suggesting the impact of developing working relationships between evaluator and student in the clinical setting over time [15]. As a result, the Reporter station transitioned from the historical use of in-room clinical rounding faculty as the assessor to utilizing standardized patients in the role of a an attending physician to whom the students would present. This helped ensure the student was not presenting to their clinical faculty or those they may have already had a working relationship such as mentor, coach, or small group facilitator. Final assessment was instead completed indirectly by video review.

To further increase evaluator reliability, evaluations for the Reporter and Interpreter stations were blinded and double-graded. Whenever possible, both faculty completing the video assessment would not have clinical experience or other working relationship with the student being evaluated. In a situation where this was not possible, at least one of the faculty cross-graders would need to be unfamiliar with the student. Any ongoing discrepancies in grading were reviewed as a committee. To further increase objectively, discussions of only utilizing audio to complete the assessment checklist were discussed but ultimately full review including video was required as some students used gesturing or hand movements to describe aspects of the patient presentation.

As suggested by Wilby, a master document was created to outline all acceptable/unacceptable wording and phrasing related to presentation or documentation checklist items to ensure consistency in scoring across all sites and student blocks. This was done in the initial post-enhancement academic year through intercampus cross-grading to ensure reliability and standardization across all sites. This method was not required for the Manager station as it was multiple choice responses. As a backup, all SP encounters were filmed for review only in case of recording error.

Student grades and feedback

The tentative cutoffs for F, P, and PD, were established by looking at the prior academic year’s percentage of PD on NBME-SE. In consideration for the required elements for both PD and failure, we considered Ryan’s study that identified the Reporter level as required to pass, the removal of Educator in evaluation, and the consideration that most common designation across all clerkships was Interpreter. [8] From this, we determined that in the event of OSCE failure, determination for remediation would only be based on the collective score from the Reporter and Interpreter stations as the Manager station was felt to be a higher-level and not necessarily reflective of the core competencies required to pass the clerkship. Any student who scored below the remediation level of Reporter and Interpreter stations had their entire OSCE regraded in a blinded fashion for the third time. Because the Reporter and Interpreter reflected core competencies of the clerkship, a conjunctive scoring method was adapted and a minimum passing score for remediation was established. Further, it was determined that passing of these stations was required in order to be eligible for PD on the overall OSCE, ensuring this grade could not be obtained solely from the successful completion of the Manager station. Any student below two or fewer points of pass with distinction also have their entire OSCE regraded in a blinded fashion for the third time to ensure evaluations were consistent with standards across all campuses.

The data collected for analyses included raw scores on the OSCE and on the NBME-SE for both AY20–21 and AY21–22, as PD ranges are required in both sections to obtain a PD for the course. The formal grade cutoffs for F, P, and PD were not determined until completion of block two of the academic year. This was done in recognition that grade determination could not be solely based on block 1 students as they were limited in clinical and clerkship experience. The Office of Medical Education of our institution provided a projection on expected OSCE PD, NBME-SE PD and overall grade PD based off the prior year and expected increased score trends throughout the year. Our overall target final grade PD is approximately 25%- 30% of students and this resulted in a projected 45% OSCE PD, with an anticipated 27% overall clerkship PD.

Student feedback was obtained through a five-question Likert-item survey regarding the OSCE’s portrayal of realistic patient encounters through the lens of the RIME framework including reporting a patient’s history and physical exam, interpreting new information and developing a problem list, and providing medical management and medical decision-making based on emerging information. Additional questions evaluated the overall time provided to complete the OSCE and the overall difficulty. Surveys were anonymous, made available to students on paper after completion of the OSCE.

Results

The distributions of student OSCE scores pre-intervention (AY 20–21; n = 201) and post-intervention (AY 20–21; n = 200) were compared (Figure 2). In the pre-intervention assessment, scores were more tightly clustered (SD = 5.65%) around a markedly high average performance (M = 94.25%), with scores being highly negatively skewed (−1.68). A large majority (86%) of students scored higher than 90%, the PD threshold at the time, with some below-average scoring students even earning this distinction. Students could only score one standard deviation above average before earning the maximum score possible. In contrast, post-intervention scores were more dispersed (SD = 6.88%) around a lower average score (M = 81.00%), with scores being far less skewed (−0.39), resulting in an approximately normal distribution. Fewer students earned PD (46.80%), set at 62 out of 75 points or 82.67%, comparatively requiring at least slightly above average performance. A larger range of above average post-intervention scores were available, with scores as high as three standard deviations above average being possible before earning the maximum score.

Figure 2. Frequency & Pearson Type IV distributions of percent correct earned on OSCE by academic year.

The pass threshold was also examined relative to student performance, with both the pre- and post-intervention versions of the OSCE having a pass threshold set at 70%. To fail the OSCE, pre-intervention students needed to earn a score approximately four standard deviations below the mean, making such an event extremely unlikely. By contrast, post-intervention students needed score approximately two standard deviations below the mean to fail, which assisted in the identification of students who required remediation or further support.

Post-intervention surveys completed by 145 students provided qualitative feedback regarding the realism of the OSCE stations (Figure 3). Predominantly positive, average student ratings of 4.4 (Reporter), 4.4 (Interpreter), and 4.2 (Manager), respectively, 4 being somewhat agree and 5 being strongly agree. Feedback regarding time allotment was only slightly positive, with an average ranking across the campuses of 3.5 out of 5, 3 being neutral and 4 being somewhat agree. Finally, feedback suggested the OSCE was perceived as slightly difficult, with an average student rating of 3.5 out of 5, 3 being neutral and 4 being somewhat difficult. Student open-ended responses suggested more time was needed for the patient encounter.

Figure 3. Percent response selected & average response by survey question.

Discussion

The OSCE enhancement, with the addition of two additional stations, was overall successful in increasing the grade distribution curve to better distinguish student performance. Placement of the threshold for the highest score of ‘Pass-with-Distinction’ was more ideal in aligning with above-average performance, though this threshold was still within 1 standard deviation from the mean. Placement of the fail threshold could be revisited, potentially to further help identify at-risk students. While it would have been interesting to directly compare performance on the Reporter one pre- and post-intervention, the checklist of items was shifted post-intervention to allow more points for reporting as compared to interpreting making a score comparison not feasible. We achieved the goal of having the weight of the OSCE and the NBME-SE ‘better aligned’ in determining the overall grade of the clerkship, which was determined by conjunctive scoring (students needed a PD on the OSCE and NBME-SE to achieve PD for the clerkship). However, this analysis did not factor in the impact of the NBME-SE which further reduces the number of students who achieve pass with distinction for the composite clerkship grade.

Student feedback was affirming regarding creating standardized scenarios that reflected their clinical experiences. We feel that with the combination of the OSCE and NBME-SE scores, we can better determine the clinical aptitude of students rotating through our clerkship, which has also had an impact on our ability to provide stronger summative letters of support for PD students in their residency applications. In subsequent academic years, there have not been significant changes to the OSCE or scoring system. Updates to the document outlining all acceptable/unacceptable is reviewed on a case-by-case basis by the grading committee if new wording or phrases arises. In response to student feedback about time constraints, an additional five minutes was added to the patient encounter for the Reporter station. Overall, ongoing evaluation will be required to ensure consistency in performance in future academic years.

There are limitations to this study. This was a single-institution study focusing on the effects of an intervention of a single AY. While the OSCE has been shown to be a good indicator of clinical performance, it may not provide direct comparisons to clinical expertise and the standardized patient environment itself has limitations, namely in time provided for clinical encounters. Additional limitations of this project include that it was at a single academic institution and that students were only given one station for each Reporter, Interpreter, or Manager category for evaluation and the Management station students were required to select management strategies using multiple choice questions which does not reflect medical decision making in actual patient care. An option for future assessment would be to increase the number of stations for each category of assessment and to utilize open-ended questions or a verbal discourse with an evaluator, though notably this would increase challenges in both resource allocation and reliable scoring depending on availability of testing software and evaluator training. Finally, additional factors that may have impacted their grades, namely the level of preparation prior to the OSCE were not compared.

Ethical approval

This study was formally submitted at our institution’s IRB for review and was deemed a quality improvement project due to the required nature for all students’ participation in the OSCE experience and therefore IRB approval was waived 4 September 2021.

Data

No outside data sources. All data was obtained by and interpreted by the authors.

Acknowledgments

The authors would like to thank the University of Kansas Health System, Department of Internal Medicine, standardized patients, piloting students, and many faculty who were essential to the development and implementation of the OSCE across multiple campuses. The authors would like to thank Scott Owings and Ali Jahansooz who provided input into the early phases of OSCE development to align clinical content, assessment goals, and OSCE structure along with Cayla Teal who helped develop review and establish post-intervention grade thresholds following score review. The authors would like to thank Ali Jahansooz, Cynthia Kibet, and Brad Daily who continue to serve on the tri-campus grading committee. The authors would lastly like to give a very special thank you for the many people in our tri-campus Clinical Skills teams including David Becker, Julie Mack, Ralph Course, and Carey Rawson who were essential initial OSCE development, piloting, technological innovation and support, and standardized patient development, who along with Sarah Talley, Brad Daily and Mary Vernon continued implementation and development of the OSCE across multiple campuses.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This project was funded by a one-time Medical Education Mentored Grant through The Academy of Medical Educators at the University of Kansas School of Medicine in 2021.