Feedback, fairness, and validity: effects of disclosing and reusing multiple-choice questions in medical schools

Figures & data

Figure 1. Overview of origin and types of items. We analysed all exams conducted between the decision to disclose multiple-choice questions in future examinations in fall 2017 and fall 2019. Fall 2017 was the first exam disclosed to examinees but without reuse of previously disclosed items, thus constituting our baseline for comparison with the exams from spring 2018 to fall 2019, which included reused, previously disclosed items in growing proportions.

Figure 2. Item psychometrics over terms. (a) Proportion of reused, disclosed items across terms. Note that fall 2017 served as baseline without any disclosed items. (b) Mean difficulty and discrimination coefficients, across all item groups, per term.

Figure 3. Boxplot diagram of difficulty and discrimination coefficients between item groups. Boxplot whiskers mark the 10^th and 90^th percentiles of the data. (a) Average difficulty coefficients per item group, across terms (including baseline). (b) Average discrimination coefficients per item group, across terms (including baseline).