Change in Engagement During Test Events: An Argument for Weighted Scoring?

References

• Baumeister, R. F., Vohs, K. D., & Tice, D. M. (2007). The strength model of self-control. Current Directions in Psychological Science, 16(6), 351–355. doi:10.1111/j.1467-8721.2007.00534.x
   Web of Science ®Google Scholar
• Bertrams, A., Baumeister, R. F., & Englert, C. (2016). Higher self-control capacity predicts lower anxiety-impaired cognition during math examinations. Frontiers in Psychology, 7, 485. doi:10.3389/fpsyg.2016.00485
   PubMed Web of Science ®Google Scholar
• Boe, E. E., May, H., & Boruch, R. (2002). Student task persistence in the third international mathematics and science study: A major source of achievement differences at the national, classroom, and student levels. CRESP Research Report. https://repository.upenn.edu/gse_pubs/412
   Google Scholar
• Borgonovi, F., & Biecek, P. (2016). An international comparison of students’ ability to endure fatigue and maintain motivation during a low-stakes test. Learning and Individual Differences, 49, 128–137. doi:10.1016/j.lindif.2016.06.001
   Web of Science ®Google Scholar
• Cronbach, L. J. (1960). Essentials of psychological testing (2nd ed.). New York, NY: Harper & Row.
   Google Scholar
• Davis, J., & Ferdous, A. (2005). Using item difficulty and item position to measure test fatigue [paper presentation]. American Educational Research Association 2005 Annual Meeting, Montreal, Quebec, Canada.
   Google Scholar
• Debeer, D., & Janssen, R. (2013). Modeling item‐position effects within an IRT framework. Journal of Educational Measurement, 50(2), 164–185. doi:10.1111/jedm.12009
   Web of Science ®Google Scholar
• Eklöf, H. (2010). Skill and will: Test-taking motivation and assessment quality. Assessment in Education Principles, Policy & Practice, 17(4), 345–356. doi:10.1080/0969594X.2010.516569
   Google Scholar
• Finney, S. J., Perkins, B. A., & Satkus, P. (2020). Examining the simultaneous change in emotions during a test: Relations with expended effort and test performance. International Journal of Testing, 20(4), 274–298. doi:10.1080/15305058.2020.1786834
   Web of Science ®Google Scholar
• Goetz, T., Preckel, F., Pekrun, R., & Hall, N. C. (2007). Emotional experiences during test taking: Does cognitive ability make a difference? Learning and Individual Differences, 17(1), 3–16. doi:10.1016/j.lindif.2006.12.002
   Web of Science ®Google Scholar
• Haladyna, T. M., & Downing, S. M. (2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement Issues & Practice, 23(1), 17–27. doi:10.1111/j.1745-3992.2004.tb00149.x
   Google Scholar
• Harmes, J. C., & Wise, S. L. (2016). Assessing engagement during the online assessment of real-world skills. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on technology tools for real-world skill development (pp. 804–823). Hershey, PA: IGI Global. doi:10.4018/978-1-4666-9441-5.ch031
   Google Scholar
• Hartig, J., & Buchholz, J. (2012). A multilevel item response model for item position effects and individual persistence. Psychological Test and Assessment Modeling, 54(4), 418–431. http://journaldatabase.info/articles/multilevel_item_response_model_for.html
   Google Scholar
• Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. doi:10.1111/jedm.12000
   Web of Science ®Google Scholar
• Kingsbury, G. G., & Zara, A. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2(4), 359–375. doi:10.1207/s15324818ame0204_6
   Google Scholar
• Leary, L. F., & Dorans, N. J. (1985). Implications for altering the context in which test items appear: A historical perspective on an immediate concern. Review of Educational Research, 55(3), 387–413. doi:10.3102/00346543055003387
   Web of Science ®Google Scholar
• Lee, Y., & Jia, Y. (2014). Using response time to investigate students’ test-taking behaviors in a NAEP computer-based study. Large-Scale Assessments in Education, 2(1), 1–24. doi:10.1186/s40536-014-0008-1
   Google Scholar
• Lindner, M. A., Lüdtke, O., & Nagy, G. (2019). The onset of rapid-guessing behavior over the course of testing time: A matter of motivation and cognitive resources. Frontiers in Psychology, 10, 1–14. doi:10.3389/fpsyg.2019.01533
   PubMed Web of Science ®Google Scholar
• Lindner, C., Nagy, G., & Retelsdorf, J. (2018). The need for self-control in achievement tests: Changes in students’ state self-control capacity and effort investment. Social Psychology of Education, 21(5), 1113–1131. doi:10.1007/s11218-018-9455-9
   Web of Science ®Google Scholar
• List, M. K., Robitzsch, A., Lüdtke, O., Köller, O., & Nagy, G. (2017). Performance decline in low-stakes educational assessments: Different mixture modeling approaches. Large-Scale Assessments in Education, 5(1), 1–25. doi:10.1186/s40536.017.0049.3
   Google Scholar
• Meijer, R. R. (2003). Diagnosing item score patterns on a test using item response theory-based person-fit statistics. Psychological Methods, 8(1), 72–87. doi:10.1037/1082-989X.8.1.72
   PubMed Web of Science ®Google Scholar
• Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5–11. doi:10.2307/1175249
   Google Scholar
• Myers, A. J., & Finney, S. J. (2021). Change in self-reported motivation before to after test completion: Relation with performance. The Journal of Experimental Education, 89(1), 74–94. doi:10.1080/00220973.2019.1680942
   Web of Science ®Google Scholar
• Nagy, G., Nagengast, B., Becker, M., Rose, N., & Frey, A. (2018). Item position effects in a reading comprehension test: An IRT study of individual differences and individual correlates. Psychological Test and Assessment Modeling, 60(2), 165–187. https://www.psychologie-aktuell.com/fileadmin/download/ptam/2-2018_20180627/03_PTAM-2-2018_Nagy_v2.pdf
   Google Scholar
• Nagy, G., Ulitzsch, E., & Lindner, M. A. (2022). The role of rapid guessing and test‐taking persistence in modelling test‐taking engagement. Journal of Computer Assisted Learning, 39(3), 751–766. doi:10.1111/jcal.12719
   Web of Science ®Google Scholar
• Pastor, D. A., Ong, T. Q., & Strickman, S. N. (2019). Patterns of solution behavior across items in low-stakes assessments. Educational Assessment, 24(3), 189–212. doi:10.1080/10627197.2019.1615373
   Web of Science ®Google Scholar
• Penk, C., & Richter, D. (2017). Change in test-taking motivation and its relationship to test performance in low-stakes assessments. Educational Assessment, Evaluation and Accountability, 29(1), 55–79. doi:10.1007/s11092-016-9248-7
   Web of Science ®Google Scholar
• Schnipke, D. L. (1995). Assessing speededness in computer-based tests using item response times ( Unpublished doctoral dissertation). Johns Hopkins University.
   Google Scholar
• Setzer, J. C., Wise, S. L., van den Heuvel, J. R., & Ling, G. (2013). An investigation of examinee test-taking effort on a large-scale assessment. Applied Measurement in Education, 26(1), 34–49. doi:10.1080/08957347.2013.739453
   Web of Science ®Google Scholar
• Sireci, S. G. (2020). Standardization and understandardization in educational assessment. Educational Measurement Issues & Practice, 39(3), 100–105. doi:10.1111/emip.12377
   Google Scholar
• Weirich, S., Hecht, M., Penk, C., Roppelt, A., & Böhme, K. (2017). Item position effects are moderated by changes in test-taking effort. Applied Psychological Measurement, 41(2), 115–129. doi:10.1177/0146621616676791
   PubMed Web of Science ®Google Scholar
• Weiss, D. J., & Kingsbury, G. G. (1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21(4), 361–375. doi:10.1111/j.1745-3984.1984.tb01040.x
   Web of Science ®Google Scholar
• Wise, S. L. (2017). Rapid-guessing behavior: Its identification, interpretations, and implications. Educational Measurement Issues & Practice, 36(4), 52–61. doi:10.1111/emip.12165
   Google Scholar
• Wise, S. L. (2021). Six insights regarding test-taking disengagement. Educational Research & Evaluation, 26(5–6), 328–338. doi:10.1080/13803611.2021.1963942
   Web of Science ®Google Scholar
• Wise, S. L., & DeMars, C. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43(1), 19–38. doi:10.1111/j.1745-3984.2006.00002.x
   Web of Science ®Google Scholar
• Wise, S. L., & Kingsbury, G. G. (2022). Performance decline as an indicator of generalized test-taking disengagement. Applied Measurement in Education, 35(4), 272–286. doi:10.1080/08957347.2022.2155651
   Web of Science ®Google Scholar
• Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183. doi:10.1207/s15324818ame1802_2
   Web of Science ®Google Scholar
• Wise, S. L., & Kuhfeld, M. R. (2021a). A method for identifying partial test-taking engagement. Applied Measurement in Education, 34(2), 150–161. doi:10.1080/08957347.2021.1890745
   Web of Science ®Google Scholar
• Wise, S. L., & Kuhfeld, M. R. (2021b). Using retest data to evaluate and improve effort-moderated scoring. Journal of Educational Measurement, 58(1), 130–149. doi:10.1111/jedm.12275
   Web of Science ®Google Scholar
• Wise, S. L., & Ma, L. (2012, April). Setting response time thresholds for a CAT item pool: The normative threshold method [paper presentation]. National Council on Measurement in Education 2012 Annual Meeting, Vancouver, Canada.
   Google Scholar
• Wise, S. L., Pastor, D. A., & Kong, X. J. (2009). Correlates of rapid-guessing behavior in low-stakes testing: Implications for test development and measurement practice. Applied Measurement in Education, 22(2), 185–205. doi:10.1080/08957340902754650
   Web of Science ®Google Scholar