The (Non)Impact of Differential Test Taker Engagement on Aggregated Scores

References

• Adams, R. J., Wilson, M., & Wang, W. C. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21(1), 1–23. doi:10.1177/0146621697211001
   Web of Science ®Google Scholar
• Boe, E. E., May, H., & Boruch, R. F. (2002). Student task persistence in the Third International Mathematics and Science Study: A major source of achievement differences at the national, classroom, and student levels (Research Report No. 2002-TIMSS1). Philadelphia, PA: University of Pennsylvania, Center for Research in Evaluation in Social Policy.
   Google Scholar
• Borghans, L., & Schils, T. (2012). The leaning tower of Pisa: Decomposing achievement test scores into cognitive and noncognitive components. Unpublished manuscript, School of Business and Economics, Maastricht University, The Netherlands.
   Google Scholar
• Butler, J., & Adams, R. J. (2007). The impact of differential investment of student effort on the outcomes of international studies. Journal of Applied Measurement, 8, 279–304.
   PubMedGoogle Scholar
• Cosgrove, J., & Cartwright, F. (2014). Changes in achievement on PISA: The case of Ireland and implications for international assessment practice. Large-Scale Assessments in Education, 2, 2.
   Google Scholar
• Cronbach, L. J. (1960). Essentials of psychological testing (2nd ed.). New York, NY: Harper & Row.
   Google Scholar
• Debeer, D., Buchholz, J., Hartig, J., & Janssen, R. (2014). Student, school, and country differences in sustained test-taking effort in the 2009 PISA reading assessment. Journal of Educational and Behavioral Statistics, 39(6), 502–523. doi:10.3102/1076998614558485
   Web of Science ®Google Scholar
• Eivers, E. (2010). PISA: Issues in implementation and interpretation. The Irish Journal of Education/Iris Eireannach an Oideachais, 38, 94–118.
   Google Scholar
• Elköf, H., Pavešič, B. J., & Grønmo, L. S. (2014). A cross-national comparison of reported effort and mathematics performance in TIMSS advanced. Applied Measurement in Education, 27, 31–45. doi:10.1080/08957347.2013.853070
   Web of Science ®Google Scholar
• Goldhammer, F., Martens, T., Christoph, G., & Lüdtke, O. (2016). Test-taking engagement in PIAAC (OECD Education Working Papers, No. 133). Paris, France: OECD Publishing.
   Google Scholar
• Haladyna, T. M., & Downing, S. M. (2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23(1), 17–27. doi:10.1111/j.1745-3992.2004.tb00149.x
   Google Scholar
• Hopfenbeck, T. N., Lenkeit, J., El Masri, Y., Cantrell, K., Ryan, J., & Baird, J. A. (2017). Lessons learned from PISA: A systematic review of peer-reviewed articles on the Programme for International Student Assessment. Scandinavian Journal of Educational Research, 62, 1–21. doi:10.1080/00313831.2016.1258726
   Web of Science ®Google Scholar
• Jagacinski, C. M., & Nicholls, J. G. (1990). Reducing effort to protect perceived ability: “They’d do it but I wouldn’t”. Journal of Educational Psychology, 82(1), 15–21. doi:10.1037//0022-0663.82.1.15
   Web of Science ®Google Scholar
• Messick, S. (1984). The psychology of educational measurement. Journal of Educational Measurement, 21(3), 215–237. doi:10.1111/j.1745-3984.1984.tb01030.x
   Web of Science ®Google Scholar
• OECD (2016). PISA-based test for schools technical report. Retrieved from http://www.oecd.org/pisa/aboutpisa/PfS_TechReport_CRC_final.pdf
   Google Scholar
• Penk, C., & Schipolowski, S. (2015). Is it all about value? Bringing back the expectancy component to the assessment of test-taking motivation. Learning and Individual Differences, 42, 27–35. doi:10.1016/j.lindif.2015.08.002
   Web of Science ®Google Scholar
• Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
   Google Scholar
• Schnipke, D. L. (1995). Assessing speededness in computer-based tests using item response times (Unpublished doctoral dissertation). Baltimore, MD: Johns Hopkins University.
   Google Scholar
• Thompson, T., Davidson, J. A., & Barber, J. G. (1995). Self-worth protection in achievement motivation: Performance effects and attributional behavior. Journal of Educational Psychology, 87(4), 598–610. doi:10.1037/0022-0663.87.4.598
   Web of Science ®Google Scholar
• Thorndike, E. L. (1904). An introduction to the theory of mental and social measurements. Oxford, UK: Science Press.
   Google Scholar
• Wigfield, A., & Eccles, J. S. (2000). Expectancy–value theory of achievement motivation. Contemporary Educational Psychology, 25(1), 68–81. doi:10.1006/ceps.1999.1015
   PubMed Web of Science ®Google Scholar
• Wise, S. L. (2015). Effort analysis: Individual score validation of achievement test data. Applied Measurement in Education, 28(3), 237–252. doi:10.1080/08957347.2015.1042155
   Web of Science ®Google Scholar
• Wise, S. L. (2017). Rapid-guessing behavior: Its identification, interpretations, and implications. Educational Measurement: Issues and Practice, 36(4), 52–61. doi:10.1111/emip.12165
   Web of Science ®Google Scholar
• Wise, S. L., & DeMars, C. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1–17. doi:10.1207/s15326977ea1001_1
   Google Scholar
• Wise, S. L., & DeMars, C. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43(1), 19–38. doi:10.1111/j.1745-3984.2006.00002.x
   Web of Science ®Google Scholar
• Wise, S. L., & DeMars, C. E. (2010). Examinee non-effort and the validity of program assessment results. Educational Assessment, 15(1), 27–41. doi:10.1080/10627191003673216
   Google Scholar
• Wise, S. L., & Gao, L. (2017). A general approach to measuring test-taking effort on computer-based tests. Applied Measurement in Education, 30(4), 343–354. doi:10.1080/08957347.2017.1353992
   Web of Science ®Google Scholar
• Wise, S. L., & Kingsbury, G. G. (2016). Modeling student test-taking motivation in the context of an adaptive achievement test. Journal of Educational Measurement, 53(1), 86–105. doi:10.1111/jedm.12102
   Web of Science ®Google Scholar
• Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183. doi:10.1207/s15324818ame1802_2
   Web of Science ®Google Scholar
• Wise, S. L. & Kuhfeld, M. R. (in press). A cessation of measurement: Identifying test taker disengagement using response time. In M. J. Margolis & R. A. Feinberg (Eds.), Integrating timing considerations to improve testing practices. New York, NY: Routledge.
   Google Scholar
• Wise, S. L., & Ma, L. (2012, April). Setting response time thresholds for a CAT item pool: The normative threshold method. Paper presented at the annual meeting of the National Council on Measurement in Education, Vancouver, Canada.
   Google Scholar
• Zamarro, G., Hitt, C., & Mendez, I. (2016). When students don’t care: Reexamining international differences in achievement and noncognitive skills. Unpublished manuscript, Department of Education Reform, University of Arkansas.
   Google Scholar