Quality assurance of item writing: During the introduction of multiple choice questions in medicine for high stakes examinations

Abstract

Background: One Norwegian medical school introduced A-type MCQs (best one of five) to replace more traditional assessment formats (e.g. essays) in an undergraduate medical curriculum. Quality assurance criteria were introduced to measure the success of the intervention.

Method: Data collection from the first four year-end examinations included item analysis, frequency of item writing flaws (IWF) and proportion of items testing at a higher cognitive level (K2). All examinations were reviewed before after delivery and no items were removed.

Results: Overall pass rates were similar to previous cohorts examined with traditional assessment formats. Across 389 items, the proportion of items with ≥5% of candidates marking two or more functioning distracters was ≥47.5%. Removal of items with high p-values (≥85%), this item distracter proportion became >75%. With each successive year in the curriculum the proportion of K2 items used rose steadily to almost 50%. 31/389 (7%) items had IWFs. 65% items had a discriminatory power, ≥0.15.

Conclusions: Five item quality criteria are recommended: (1) adherence to an in-house style, (2) item proportion testing at K2 level, (3) functioning distracter proportion, (4) overall discrimination ratio and (5) IWF frequency.

Additional information

Notes on contributors

James Ware

JAMES WARE was formerly an academic surgeon, now whole time working in the field of medical education.

Torstein Vik

TORSTEIN VIK, as former Vice Dean for Medical Education was responsible for the introduction of new examination formats at Norwegian University of Science and Technology (NTNU), Trondheim, Norway.