New Multiple-Choice Measures of Historical Thinking: An Investigation of Cognitive Validity

REFERENCES

• Afflerbach, P., & Johnston, P. (1984). On the use of verbal reports in reading research. Journal of Literacy Research, 16, 307–322. doi:10.1080/10862968409547524
   Web of Science ®Google Scholar
• Anastasi, A. (1986). Evolving concepts of test validation. Annual Review of Psychology, 37, 1–16. doi:10.1146/annurev.ps.37.020186.000245
   Web of Science ®Google Scholar
• Anderson, N. J., Bachman, L., Perkins, K., & Cohen, A. (1991). An exploratory study into the construct validity of a reading comprehension test: Triangulation of data sources. Language Testing, 8, 41–66. doi:10.1177/026553229100800104
   Google Scholar
• Ayala, C. C., Shavelson, R. J., Yin, Y., & Schultz, S. E. (2002). Reasoning dimensions underlying science achievement: The case of performance assessment. Educational Assessment, 8, 101–121. doi:10.1207/S15326977EA0802_02
   Google Scholar
• Ayala, C. C., Yin, Y., Shavelson, R. J., & Vanides, J. (2002, April). Investigating the cognitive validity of science performance assessment with think alouds: Technical aspects. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA.
   Google Scholar
• Baron, C. (2012). Understanding historical thinking at historic sites. Journal of Educational Psychology, 104, 833–847. doi:10.1037/a0027476
   Web of Science ®Google Scholar
• Baxter, G. P., & Glaser, R. (1998). Investigating the cognitive complexity of science assessments. Educational Measurement: Issues and Practice, 17(3), 37–45. doi:10.1111/j.1745-3992.1998.tb00627.x
   Google Scholar
• Black, P., & Wiliam, D. (1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80, 139–148. doi:10.1177/003172171009200119
   Web of Science ®Google Scholar
• Bruner, J. S. (2009). The process of education. Cambridge, MA: Harvard University Press. (Original work published 1960)
   Google Scholar
• California Department of Education. (1998). History–social science content standards for California public schools. Sacramento, CA: Author.
   Google Scholar
• California Department of Education. (2009, March). Released test questions: US history grade 11. Retrieved from http://www.cde.ca.gov/ta/tg/sr/documents/cstrtqhssmar18.pdf
   Google Scholar
• Cronbach, L. J. (1988). Five perspectives on validity argument. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 3–17). Hillsdale, NJ: Erlbaum.
   Google Scholar
• Ercikan, K., & Seixas, P. (Eds.). (2015). New directions in assessing historical thinking. New York, NY: Routledge. doi:10.4324/9781315779539
   Google Scholar
• Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
   Google Scholar
• Farr, R., Pritchard, R., & Smitten, B. (1990). A description of what happens when an examinee takes a multiple-choice reading comprehension test. Journal of Educational Measurement, 27, 209–226. doi:10.1111/j.1745-3984.1990.tb00744.x
   Web of Science ®Google Scholar
• Fitzpatrick, R., & Morrison, E. J. (1971). Performance and product evaluation. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 237–270). Washington, DC: American Council on Education.
   Google Scholar
• Frederiksen, N. (1984). The real test bias: Influences of testing on teaching and learning. American Psychologist, 39, 193–202. doi:10.1037/0003-066X.39.3.193
   Web of Science ®Google Scholar
• Freedle, R., & Kostin, I. (1994). Can multiple-choice reading tests be construct-valid? A reply to Katz, Lautenschlager, Blackburn, and Harris. Psychological Science, 5, 107–110. doi:10.1111/j.1467-9280.1994.tb00640.x
   Web of Science ®Google Scholar
• Gordon, E. W. (2013). A public policy statement. The Gordon Commission on the Future of Assessment in Education. Retrieved from http://gordoncommission.org/rsc/pdfs/gordon_commission_public_policy_report.pdf
   Google Scholar
• Haladyna, T. M. (2004). Developing and validating multiple-choice test items (3rd ed.). London, UK: Erlbaum.
   Google Scholar
• Hamilton, L. S., Nussbaum, E. M., & Snow, R. E. (1997). Interview procedures for validating science assessments. Applied Measurement in Education, 10, 181–200. doi:10.1207/s15324818ame1002_5
   Web of Science ®Google Scholar
• Holt, T. (1990). Thinking historically: Narrative, imagination, and understanding. New York, NY: College Entrance Examination Board.
   Google Scholar
• Horn, C. (2006). The technical realities of measuring history. In S. G. Grant (Ed.), Measuring history: Cases of state-level testing across the United States (pp. 9–27). Greenwich, CT: Information Age.
   Google Scholar
• Immerwahr, D. (2008). The fact/narrative distinction and student examinations in history. The History Teacher, 41, 199–205.
   Google Scholar
• Kaliski, P., Smith, K., & Huff, K. (2015). The importance of construct validity evidence in history assessment: What is often overlooked or misunderstood? In K. Ercikan & P. Seixas (Eds.), New directions in assessing historical thinking (pp. 195–205). New York, NY: Routledge.
   Google Scholar
• Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Westport, CT: Praeger.
   Google Scholar
• Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1–73. doi:10.1111/jedm.12000
   Web of Science ®Google Scholar
• Katz, I. R., Bennett, R. E., & Berger, A. E. (2000). Effects of response format on difficulty of SAT-mathematics items: It’s not the strategy. Journal of Educational Measurement, 37, 39–57. doi:10.1111/j.1745-3984.2000.tb01075.x
   Web of Science ®Google Scholar
• Koretz, D. (2008). Measuring up. Boston, MA: Harvard University Press.
   Google Scholar
• Larkin, J. H., & Rainard, B. (1984). A research methodology for studying how people think. Journal of Research in Science Teaching, 21, 235–254. doi:10.1002/tea.3660210302
   Web of Science ®Google Scholar
• Leighton, J. P., Rogers, W., & Maguire, T. O. (1999). Assessment of student problem-solving on ill-defined tasks. The Alberta Journal of Educational Research, 45, 409–427.
   Web of Science ®Google Scholar
• Lévesque, S. (2008). Thinking historically: Educating students for the twenty-first century. Toronto, Canada: University of Toronto Press.
   Google Scholar
• Li, M., Ruiz-Primo, M., & Shavelson, R. (2006). Towards a science achievment framework: The case of TIMSS 1999. In S. Howie & T. Plomp (Eds.), Contexts of learning mathematics and science: Lessons learned from TIMSS (pp. 291–311). London, UK: Routledge.
   Google Scholar
• Linn, R. L., Baker, E. L., & Dunbar, S. B. (1991). Complex, performance-based assessment: Expectations and validation criteria. Educational Researcher, 20(8), 15–21. doi:10.3102/0013189X020008015
   Google Scholar
• Linn, R. L., & Herman, J. L. (1997). Standards-led assessment: Technical and policy issues in measuring school and student progress (CSE Technical Report 426). Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles.
   Google Scholar
• Madaus, G., West, M., Harmon, M., Lomax, R., & Viator, K. (1992). The influence of testing on teaching math and science in grades 4–12. Boston, MA: Boston College, Center for the Study of Testing, Evaluation, and Educational Policy.
   Google Scholar
• Martin, D., Maldonado, S. I., Schneider, J., & Smith, M. (2011). A report on the state of history education: State policies and national programs. National History Education Clearinghouse. Retrieved from http://teachinghistory.org/system/files/teachinghistory_special_report_2011.pdf
   Google Scholar
• Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York, NY: American Council on Education.
   Google Scholar
• National Assessment Governing Board. (2011). U.S. history framework for the 2010 National Assessment of Educational Progress. U.S. Department of Education. Retrieved from http://www.nagb.org/publications/frameworks/historyframework.pdf
   Google Scholar
• National Center for Education Statistics. (n.d.). NAEP questions tool. Retrieved August 12, 2013, from http://nces.ed.gov/nationsreportcard/itmrlsx/search.aspx?subject=history
   Google Scholar
• National Governors Association Center for Best Practices, & Council of Chief State School Officers. (2010). Common core state standards for English language arts & literacy in history/social studies, science, and technical subjects. Retrieved from http://www.corestandards.org/assets/CCSSI_ELA%20Standards.pdf
   Google Scholar
• Nisbett, R., & Wilson, T. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231–259. doi:10.1037//0033-295x.84.3.231
   Web of Science ®Google Scholar
• Nokes, J. D. (2013). Building students’ historical literacies: Learning to read and reason with historical evidence. New York, NY: Routledge.
   Google Scholar
• Paxton, R. J. (1999). A deafening silence: History textbooks and the students who read them. Review of Educational Research, 69, 315–339. doi:10.3102/00346543069003315
   Web of Science ®Google Scholar
• Pellegrino, J. W., Chudowsky, N., & Glaser, R. (Eds.). (2001). Knowing what students know. Washington, DC: National Academy Press. doi:10.17226/10019
   Google Scholar
• Popham, W. J. (2003). Living (or dying) with your NCLB tests. School Administrator, 60(11), 10–14.
   Google Scholar
• Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Erlbaum.
   Google Scholar
• Reich, G. A. (2009). Testing historical knowledge: Standards, multiple-choice questions and student reasoning. Theory & Research in Social Education, 37, 325–360. doi:10.1080/00933104.2009.10473401
   Google Scholar
• Reich, G. A. (2013). Imperfect models, imperfect conclusions: An exploratory study of multiple-choice tests and historical knowledge. Journal of Social Studies Research, 37, 3–16. doi:10.1016/j.jssr.2012.12.004
   Google Scholar
• Reisman, A. (2011). The “document-based lesson”: Bringing disciplinary literacy into high school history classrooms with adolescent struggling readers. Journal of Curriculum Studies, 44, 233–264. doi:10.1080/00220272.2011.591436
   Web of Science ®Google Scholar
• Reisman, A. (2012). Reading like a historian: A document-based history curriculum intervention in urban high schools. Cognition and Instruction, 30, 86–112. doi:10.1080/07370008.2011.634081
   Web of Science ®Google Scholar
• Rouet, J. F., Favart, M., Britt, M. A., & Perfetti, C. A. (1997). Studying and using multiple documents in history: Effects of discipline expertise. Cognition and Instruction, 15, 85–106. doi:10.1207/s1532690xci1501_3
   Web of Science ®Google Scholar
• Seixas, P. (2017). A model of historical thinking. Educational Philosophy and Theory, 49, 593–605. doi:10.1080/00131857.2015.1101363
   Web of Science ®Google Scholar
• Seixas, P., Gibson, L., & Ercikan, K. (2015). A design process for assessing historical thinking: The case of a one-hour test. In K. Ercikan & P. Seixas (Eds.), New directions in assessing historical thinking (pp. 102–116). New York, NY: Routledge.
   Google Scholar
• Seixas, P., Morton, T., Colyer, J., & Fornazzari, S. (2013). The big six: Historical thinking concepts. Toronto, Canada: Nelson Education.
   Google Scholar
• Shepard, L. A. (1993). Evaluating test validity. Review of Research in Education, 19, 405–450. doi:10.3102/0091732X019001405
   Web of Science ®Google Scholar
• Shepard, L. A. (2000). The role of assessment in a learning culture. Educational Researcher, 29(7), 4–14. doi:10.3102/0013189X029007004
   Google Scholar
• Smith, M. (2017). Cognitive validity: Can multiple-choice items tap historical thinking? American Educational Research Journal. Advance online publication. doi:10.3102/0002831217717949
   Web of Science ®Google Scholar
• Stahl, S. A., Hynd, C. R., Britton, B. K., McNish, M. M., & Bosquet, D. (1996). What happens when students read multiple source documents in history? Reading Research Quarterly, 31, 430–456. doi:10.1598/RRQ.31.4.5
   Web of Science ®Google Scholar
• Stiggins, R. (2002). Assessment crisis: The absence of assessment for learning. Phi Delta Kappan, 83, 758–765. doi:10.1177/003172170208301010
   Web of Science ®Google Scholar
• Taylor, K. L., & Dionne, J. P. (2000). Accessing problem-solving strategy knowledge: The complementary use of concurrent verbal protocols and retrospective debriefing. Journal of Educational Psychology, 92, 413–425. doi:10.1037/0022-0663.92.3.413
   Web of Science ®Google Scholar
• Van Someren, M., Barnard, Y., & Sandberg, J. (1994). The think aloud method: A practical guide to modeling cogntive processes. London, UK: Academic Press.
   Google Scholar
• VanSledright, B. (2014). Assessing historical thinking and understanding: Innovative designs for new standards. New York, NY: Taylor & Francis.
   Google Scholar
• VanSledright, B. A. (2004). What does it mean to think historically… and how do you teach it? Social Education, 68, 230–233.
   Google Scholar
• Wainer, H. (2011). Uneducated guesses: Using evidence to uncover misguided education policies. Princeton, NJ: Princeton University Press.
   Google Scholar
• White, P. (1980). Limitations on verbal reports of internal events: A refutation of Nisbett and Wilson and of Bem. Psychological Review, 87, 105–112. doi:10.1037/0033-295X.87.1.105
   Web of Science ®Google Scholar
• Wineburg, S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–87. doi:10.1037/0022-0663.83.1.73
   Web of Science ®Google Scholar
• Wineburg, S. (1998). Reading Abraham Lincoln: An expert/expert study in the interpretation of historical texts. Cognitive Science, 22, 319–346. doi:10.1016/S0364-0213(99)80043-3
   Web of Science ®Google Scholar
• Wineburg, S. (2001). Historical thinking and other unnatural acts: Charting the future of teaching the past. Philadelphia, PA: Temple University Press.
   Google Scholar
• Wineburg, S. (2004). Crazy for history. Journal of American History, 90, 1401–1414. doi:10.2307/3660360
   Web of Science ®Google Scholar
• Wineburg, S., Martin, D., & Monte-Sano, C. (2012). Reading like a historian: Teaching literacy in middle and high school classrooms. New York, NY: Teachers College Press.
   Google Scholar