Of Small Beauties and Large Beasts: The Quality of Distractors on Multiple-Choice Tests Is More Important Than Their Quantity

References

• Baghaei, F., & Amrahi, N. (2011). The effects of the number of options on the psychometric characteristics of multiple choice items. Psychological Test and Assessment Modeling, 53(2), 192–211.
   Google Scholar
• Brozo, W. G., Schmelzer, R. V., & Spires, H. A. (1984). A study of testwiseness clues in college and university teacher-made tests with implications for academic assistance centers (Technical Report 84-01). Atlanta, GA: Georgia State University, College Reading and Learning Assistance.
   Google Scholar
• Budescu, D. V., & Nevo, B. (1985). Optimal number of options: An investigation of the assumption of proportionality. Journal of Educational Measurement, 22(3), 183–196. doi:10.1111/jedm.1985.22.issue-3
   Web of Science ®Google Scholar
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
   Google Scholar
• Costin, F. (1970). The optimal number of alternatives in multiple-choice achievement tests: Some empirical evidence for a mathematical proof. Educational and Psychological Measurement, 30(2), 353–358. doi:10.1177/001316447003000217
   Web of Science ®Google Scholar
• Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. doi:10.1007/BF02310555
   Web of Science ®Google Scholar
• DiBattista, D., & Kurzawa, L. (2011). Examination of the quality of multiple-choice items on classroom tests. The Canadian Journal for the Scholarship of Teaching and Learning, 2(2), Article 4. doi:10.5206/cjsotl-rcacea
   Google Scholar
• Diedenhofen, B., & Musch, J. (2015). cocor: A comprehensive solution for the statistical comparison of correlations. PLoS ONE, 10(4), e0121945. doi:10.1371/journal.pone.0121945
   PubMed Web of Science ®Google Scholar
• Diedenhofen, B., & Musch, J. (2016). cocron: A web interface and r package for the statistical comparison of Cronbach’s alpha coefficients. International Journal of Internet Science, 11, 51–60.
   Web of Science ®Google Scholar
• Downing, S. M. (1992). True-false, alternate-choice, and multiple-choice items. Educational Measurement: Issues and Practice, 11(3), 27–30. doi:10.1111/j.1745-3992.1992.tb00248.x
   Google Scholar
• Ebel, R. L. (1968). Blind guessing on objective achievement tests. Journal of Educational Measurement, 5(4), 321–325. doi:10.1111/jedm.1968.5.issue-4
   Web of Science ®Google Scholar
• Ebel, R. L. (1969). Expected reliability as a function of choices per item. Educational and Psychological Measurement, 29(3), 565–570. doi:10.1177/001316446902900302
   Web of Science ®Google Scholar
• Edwards, B. D., Arthur, W., & Bruce, L. L. (2012). The 3-option format for knowledge and ability multiple-choice tests: A case for why it should be more commonly used in personnel testing. International Journal of Selection and Assessment, 20(1), 65–81. doi:10.1111/j.1468-2389.2012.00580.x
   Web of Science ®Google Scholar
• Farhady, H., & Shakery, S. (2000). Number of options and economy of multiple-choice tests. Roshd Foreign Language Teaching Journal, 14, 132–140.
   Google Scholar
• Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G* Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160. doi:10.3758/BRM.41.4.1149
   PubMed Web of Science ®Google Scholar
• Feldt, L. S., Woodruff, D. J., & Salih, F. A. (1987). Statistical inference for coefficient alpha. Applied Psychological Measurement, 11(1), 93–103. doi:10.1177/014662168701100107
   Web of Science ®Google Scholar
• Fisher, R. A. (1925). Statistical methods for research workers. Edinburgh, Scotland: Oliver and Boyd.
   Google Scholar
• Green, K., Sax, G., & Michael, W. B. (1982). Validity and reliability of tests having differing numbers of options for students of differing levels of ability. Educational and Psychological Measurement, 42(1), 239–245. doi:10.1177/0013164482421030
   Web of Science ®Google Scholar
• Haladyna, T. M. (2004). Developing and validating multiple-choice test items (3rd ed.). New York, NY: Routledge.
   Google Scholar
• Haladyna, T. M., & Downing, S. M. (1989). Validity of a taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 2(1), 51–78. doi:10.1207/s15324818ame0201_4
   Google Scholar
• Haladyna, T. M., & Downing, S. M. (1993). How many options is enough for a multiple-choice test item. Educational and Psychological Measurement, 53(4), 999–1010. doi:10.1177/0013164493053004013
   Web of Science ®Google Scholar
• Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–334. doi:10.1207/S15324818AME1503_5
   Web of Science ®Google Scholar
• Hossiep, R., & Schulte, M. (2008). Bochumer Wissenstest (BOWIT) manual. Göttingen, Germany: Hogrefe.
   Google Scholar
• Kuder, G. F., & Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2(3), 151–160. doi:10.1007/BF02288391
   Google Scholar
• Lee, H., & Winke, P. (2013). The differences among three-, four-, and five-option-item formats in the context of a high-stakes English-language listening test. Language Testing, 30(1), 99–123. doi:10.1177/0265532212451235
   Web of Science ®Google Scholar
• Leiner, D. J. (2012, March 6). SoSci panel: The noncommercial online access panel. Poster presented at the GOR 2012, Mannheim, Germany. Retrieved from https://www.soscisurvey.de/panel/download/SoSciPanel.GOR2012.pdf.
   Google Scholar
• Lord, F. M. (1977). Optimal number of choices per item—A comparison of four approaches. Journal of Educational Measurement, 14(1), 33–38. doi:10.1111/j.1745-3984.1977.tb00026.x
   Web of Science ®Google Scholar
• Owen, S. V., & Froman, R. D. (1987). What’s wrong with 3-option multiple choice items? Educational and Psychological Measurement, 47(2), 513–522. doi:10.1177/0013164487472027
   Web of Science ®Google Scholar
• Ratcliff, R. (1993). Methods for dealing with reaction time outliers. Psychological Bulletin, 114(3), 510–532. doi:10.1037/0033-2909.114.3.510
   PubMed Web of Science ®Google Scholar
• Rodriguez, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3–13. doi:10.1111/j.1745-3992.2005.00006.x
   Google Scholar
• Straton, R. G., & Catts, R. M. (1980). A comparison of 2-choice, 3-choice and 4-choice item tests given a fixed total number of choices. Educational and Psychological Measurement, 40(2), 357–365. doi:10.1177/001316448004000212
   Web of Science ®Google Scholar
• Tarrant, M., & Ware, J. (2008). Impact of item-writing flaws in multiple-choice questions on student achievement in high-stakes nursing assessments. Medical Education, 42(2), 198–206. doi:10.1111/j.1365-2923.2007.02957.x
   PubMed Web of Science ®Google Scholar
• Tarrant, M., & Ware, J. (2010). A comparison of the psychometric properties of three- and four-option multiple-choice questions in nursing assessments. Nurse Education Today, 30(6), 539–543. doi:10.1016/j.nedt.2009.11.002
   PubMed Web of Science ®Google Scholar
• Tarrant, M., Ware, J., & Mohammed, A. M. (2009). An assessment of functioning and non-functioning distractors in multiple-choice questions: A descriptive analysis. BMC Medical Education, 9(1), 1–8. doi:10.1186/1472-6920-9-40
   PubMed Web of Science ®Google Scholar
• Thanyapa, I., & Currie, M. (2014). The number of options in multiple choice items in language tests: Does it make any difference? Evidence from Thailand. Language Testing in Asia, 4(1), 1–21. doi:10.1186/s40468-014-0008-7
   Google Scholar
• Trepte, S., & Verbeet, M. (Eds.). (2010). Allgemeinbildung in Deutschland. Erkenntnisse aus dem SPIEGEL-Studentenpisa-Test. Wiesbaden, Germany: VS Verlag für Sozialwissenschaften.
   Google Scholar
• Trevisan, M. S., Sax, G., & Michael, W. B. (1991). The effects of the number of options per item and student ability on test validity and reliability. Educational and Psychological Measurement, 51(4), 829–837. doi:10.1177/001316449105100404
   Web of Science ®Google Scholar
• Tversky, A. (1964). On the optimal number of alternatives at a choice point. Journal of Mathematical Psychology, 1(2), 386–391. doi:10.1016/0022-2496(64)90010-0
   Web of Science ®Google Scholar
• Zoanetti, N., Beaves, M., Griffin, P., & Wallace, E. M. (2013). Fixed or mixed: A comparison of three, four and mixed-option multiple-choice tests in a fetal surveillance education program. BMC Medical Education, 13(1), 1–11. doi:10.1186/1472-6920-13-35
   PubMed Web of Science ®Google Scholar