It takes only 100 true–false items to test medical students: true or false?

Abstract

Multiple true/false (MTF) questions are often used to test medical students. Statistical methods can give an indication of how many of these questions are needed for a reliable test, but it would be useful in addition to have a graphical indication of reliability. Therefore, in an attempt to estimate the smallest number of items needed for reliable testing, the marks of students were plotted throughout an examination. A total of 211 medical students were given 60 pathology questions comprising 300 true/false items. The cumulative percentage mark throughout the 300 items was calculated and graphed for five students each from the top, middle and bottom total scores. For the other 196 students, percentage marks were calculated at 100 and 300 items. The reduction in reliability induced by the 300 to 100 item reduction was calculated by the Spearman–Brown formula. The cumulative percentage graphs showed that, after early fluctuations in each student's mark, the total mark stabilized after 100 items. The mark fluctuated slightly either up or down after 100 marks, but in 96.7% of students it differed by fewer than 10 percentage points between 100 and 300 items. The reliability coefficient was reduced from 0.94 in the 300-item test to 0.85 in the 100-item test. In conclusion, student marks appear to stabilize after 100 true/false items. If the level of difficulty of an examination remains constant, and items are of high discriminatory value, 100 true/false items appear to be sufficient to assess medical students in the MTF format.