ABSTRACT
Examinations are essential in assessing student learning in medical education. Ensuring the quality of exam questions is a highly challenging yet necessary task to assure that assessments are equitable, reliable, and aptly gauge student learning. The aim of this study was to investigate whether the incorporation of student feedback can enhance the quality of exam questions in the Renal and Urinary System course, offered to second-year medical students. Using a single-arm between-person survey-based design, we conducted an a priori power analysis to establish the sample size. The exam comprised 100 multiple-choice questions written by a panel of 31 instructors. A total of 125 medical students took the exam in 2021. Following the exam, student feedback was collected, resulting in the revision of 12 questions by two subject experts. In the following year, the revised questions were administered to a new cohort of 125 second-year medical students. We used Fisher’s z-transformation to test the significance of differences in point-biserial correlations between the 2021 and 2022 cohorts. The results reveal that 66% of the revised exam questions exhibited significantly higher point-biserial correlations. This demonstrates the positive impact of involving students in the exam revision process. Their feedback enhances question clarity, relevance, alignment with learning objectives, and overall quality. In conclusion, student participation in exam evaluation and revision can improve the quality of exam questions. This approach capitalises on students experiences and feedback and complements the traditional approaches to ensure the quality of exam questions, benefiting both the institution and its learners.
Introduction
Exams play a crucial role in evaluating student learning in medical education [Citation1]. Numerous studies have emphasised the importance of exams in monitoring student progress and ensuring they acquire the necessary knowledge and skills to succeed in the field [Citation2]. Consequently, exam questions must undergo regular revision to align with learning objectives, course content, and licencing requirements. Additionally, ensuring the quality of exam questions is essential to establish that assessments are fair, reliable, and accurately measure student learning.
Traditionally, various strategies have been employed to ensure the quality of exam questions including peer review [Citation3], pilot testing of exam questions [Citation4], the use of standardised criteria, such as those provided by the Association of American Medical Colleges (AAMC) [Citation5], and psychometric analysis [Citation6].
Complementary to traditional approaches, an innovative approach involves allowing students to actively participate in this process by providing feedback and suggesting alternative exam questions to the instructor, who then chooses and edits them to create questions for future exams. Students’ feedback can enhance the development and revision of exam questions including:
Improved quality: Student feedback can provide valuable information on the clarity and relevance of exam questions, which can be used to improve their quality [Citation7].
Promoting student engagement: Students’ involvement in the process of revising and developing exam questions is an empowering experience for students that can promote a sense of inclusion and personal investment in the assessment [Citation4,Citation8].
Better alignment with learning objectives: Student feedback can provide insight into how well exam questions align with course learning objectives, allowing for necessary revisions to be made.
Increased validity of exam questions: student feedback can help to establish whether the exams reflect the students’ learning progress and to identify the areas of need of improvement [Citation6].
Considering the advantages associated with integrating student feedback into the development and refinement of in-house exam questions, this study aimed to investigate whether incorporating students’ feedback into the process of creating and revising exam questions for the final exam in the Renal and Urinary System course (RUOS), which is taken by second-year medical students, will enhance the quality of these exam questions.
Materials and Methods
Design, Power Analysis and Sample
We conducted a single-arm between-person survey-based design. An a-priori power analysis conducted at a significance level of 0.05 with a desired statistical power of 0.95, revealed that a total sample size of N = 111 participants would be required to explore point-biserial correlations effectively, assuming a small to medium effect size (Cohen’s d = 0.3). However, our study was conducted using a single-arm between-person survey-based design with a sample of 125.
Material
The exam consists of 100 multiple-choice instructor-written questions created by a panel of 31 instructors who taught various sessions of RUOS course. Each question includes five response options, with one correct response. These instructors have undergone training in developing questions in the style of the National Board of Medical Examiners (NBME). The clinical and basic sciences course directors have reviewed these questions to ensure their alignment with the USMLE clinical vignette style and with specific session objectives. Additionally, the assessment team has meticulously reviewed and vetted the questions to identify and rectify grammatical errors, spelling mistakes, and to ensure the proper use of abbreviations and numerical units.
Phase 1: Student Post-Exam Feedback
In 2021, all students who completed the exam had two opportunities to provide feedback on exam questions. The first opportunity arose during the exam itself, which was administered using an educational software platform ExamSoft (Dallas, TX) that enables the creation, secure delivery of exams, and collection of student feedback on exam questions. Additionally, students were given the chance to provide written feedback during an exam review session using a feedback form.
Content Analysis of Students’ Feedback
After the exam, student feedback on the exam questions was collected. Following a calibration phase, two subject experts independently conducted a thorough review of the feedback. In the next step, the expert panel agreed upon several common concerns and themes among the responses:
The question contained information not covered in class.
The question vignette was unclear or confusing.
The question vignette did not offer sufficient information to determine the correct response.
The question had multiple correct answers.
The question answer choices lacked clarity.
The experts also evaluated the psychometric characteristics of the exam questions, such as difficulty (using question difficulty index) and discrimination (using point-biserial correlations). However, it was decided to prioritise addressing the identified issues with the content validity of the questions over the psychometrics. The panel of subject experts opted to amend 12 questions and decided that the remaining 88 exam questions did not require any revision.
Phase 2
The revised exam questions were administered to the next cohort of medical students in the following year, 2022, with a total of 125 participants.
Results
displays the point-biserial correlations of the exam questions in 2021 and 2022. Using Fisher’s z-transformation, we tested whether the differences between point-biserial correlations are significant. The results show that the point-biserial correlations of 8 exam questions (66% of the total revised exam questions) significantly improved, while three exam questions exhibited lower point-biserial correlations after the revision. In addition, the average point-biserial correlation of the revised exam questions increase from rpb = .13 in 2021 to rpb = .22* while the average point-biserial correlation of the remaining 88 exam questions did not change significantly (from rpb = .27** in 2021 to rpb = .28** in 2022).
Figure 1. Point-biserial correlations of exam questions for the renal and urinary organ system course.
Discussion
The results of this study provide empirical support that involving students in the process of exam development, evaluations, and revisions can significantly improve the quality of exam questions. Students can provide a unique perspective and insight to identify areas where questions may be unclear or problematic, leading to more accurate and effective assessments. Their input can enhance the relevance and alignment of exam questions with the learning objectives but also fosters a sense of ownership and engagement in their education. Additionally, students’ involvement in this process contributes to a more comprehensive and well-rounded assessment strategy that ultimately benefits the educational institution and its learners.
While we recognise the importance of addressing potential biases in student feedback, such as response bias or reluctance to provide critical feedback, we acknowledge that our study did not explicitly address these aspects. Nevertheless, ensuring anonymity in the feedback process can encourage honest and candid responses from students. Additional limitations of this study pertain small sample size and limited generalisability as the study focuses specifically on the Renal and Urinary System course and, hence, the findings may not be generalisable to other subject matters or institutions with different exam development, delivery, and evaluation procedure and policies. Therefore, replication of this study across other subjects using anonymous student feedback and a larger reviewer panel is warranted.
Ethics Declarations
This study was approved by the institutional review board of Oakland University William Beaumont School of Medicine, (approval number IRB 2022-390).
Disclosure Statement
No potential conflict of interest was reported by the author(s).
References
- Perleth M, Picker C. High-stakes exams in medical education: a systematic review. J Med Educ. 2011;8(2):94–3. doi: 10.5116/ijme.5b2e.aa44
- Levine MB, Redick RJ. Impact of epidemiological and statistical skills training in undergraduates. Med Educ. 2012;45(11):1138–1144. doi: 10.1111/j.1365-2923.2011.03951.x)
- Gorjidooz J, Taleghani F, Montazeri A, et al. The effect of relevance of exam questions on student performance. Health Qual Life Outcomes. 2015;13(1):83. doi: 10.1186/s12955-015-0270-4
- Tan LY, Tan BH, Teo KH. Creative art and medical student development: a qualitative study. Med Educ. 2017;51(2):174–183. doi: 10.1111/medu.13131
- Norcini JJ, Shea JA, Petrusa ER, et al. Characteristics of examination items that affect the validity of the examination score. Acad Med. 2007;82(12):1138–1144. doi: 10.1097/ACM.0b013e31815d17c7
- Mody S, Johnson J, Bostrom A. The effect of exam question clarity on student performance. Med Educ. 2017;51(5):516–524. doi: 10.1111/medu.13225
- Lo C, Durning S, Tan J. Teaching during consultation: factors affecting the resident-fellow teaching interaction. Med Educ. 2015;49(7):717–725. doi: 10.1111/medu.12721
- Boud D, Keogh R, Walker D, et al. High-stakes testing in medical education: impact on student learning and patient care. J Med Educ. 2007;4(1):51–57. doi: 10.1080/10401330709336718