Using Response Times for Joint Modeling of Response and Omission Behavior

References

• Alvarez, I., Niemi, J., & Simpson, M. (2014). Bayesian inference for a covariance matrix. arXiv preprint arXiv:1408.4050.
   Google Scholar
• Barnard, J., McCulloch, R., & Meng, X.-L. (2000). Modeling covariance matrices in terms of standard deviations and correlations, with application to shrinkage. Statistica Sinica, 10, 1281–1311.
   Web of Science ®Google Scholar
• Bolsinova, M., Tijmstra, J., & Molenaar, D. (2016). Response moderation models for conditional dependence between response time and response accuracy. British Journal of Mathematical and Statistical Psychology, 69(1), 62. doi:10.1111/bmsp.12076
   PubMed Web of Science ®Google Scholar
• Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., … Riddell, A. (2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76(1), 1–32. doi:10.18637/jss.v076.i01
   PubMed Web of Science ®Google Scholar
• Choe, E. M., Zhang, J., & Chang, H.-H. (2018). Sequential detection of compromised items using response times in computerized adaptive testing. Psychometrika, 83(3), 650–673. doi:10.1007/s11336-017-9596-3
   PubMed Web of Science ®Google Scholar
• Culbertson, M. J. (April 2011). Is it wrong? Handling missing responses in IRT. Paper presented at the Annual Meeting of the National Council on Measurement in Education, New Orleans, LA. Retrieved from http://hdl.handle.net/2142/18819
   Google Scholar
• Ayala, R. J., Plake, B. S., & Impara, J. C. (2001). The impact of omitted responses on the accuracy of ability estimation in item response theory. Journal of Educational Measurement, 38(3), 213–234. doi:10.1111/j.1745-3984.2001.tb01124.x
   Web of Science ®Google Scholar
• Finch, H. (2008). Estimation of item response theory parameters in the presence of missing data. Journal of Educational Measurement, 45(3), 225–245. doi:10.1111/j.1745-3984.2008.00062.x
   Web of Science ®Google Scholar
• Fox, J.-P. (2010). Bayesian item response modeling: Theory and applications. New York: Springer Science & Business Media.
   Google Scholar
• Fox, J.-P., Klein Entink, R., & van der Linden, W. J. (2007). Modeling of responses and response times with the package cirt. Journal of Statistical Software, 20(7), 1–14. doi:10.18637/jss.v020.i07
   Web of Science ®Google Scholar
• Fox, J.-P., & Marianti, S. (2016). Joint modeling of ability and differential speed using responses and response times. Multivariate Behavioral Research, 51(4), 1–14. doi:10.1080/00273171.2016.1171128
   PubMed Web of Science ®Google Scholar
• Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2014). Bayesian data analysis, Vol. 2. Boca Raton, FL: CRC Press.
   Google Scholar
• Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel hierarchical models, Vol. 1. New York, NY, USA: Cambridge University Press.
   Google Scholar
• Gelman, A., & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–472. doi:10.1214/ss\%2F1177011136
   Google Scholar
• Gelman, A., & Shirley, K. (2011). Inference from simulations and monitoring convergence. In S. Brooks, A. Gelman, G. Jones, & X. L. Meng (Eds.), Handbook of Markov Chain Monte Carlo (pp. 163–174). Boca Raton, FL: Chapman Hall.
   Google Scholar
• Goldhammer, F. (2015). Measuring ability, speed, or both? Challenges, psychometric solutions, and what can be gained from experimental control. Measurement: Interdisciplinary Research and Perspectives, 13(3-4), 133–164. doi:10.1080/15366367.2015.1100020
   PubMed Web of Science ®Google Scholar
• Gonzalez, E., & Rutkowski, L. (2010). Principles of multiple matrix booklet designs and parameter recovery in large-scale assessments. IEA-ETS Research Institute Monograph, 3, 125–156.
   Google Scholar
• Guo, J., Gabry, J., & Goodrich, B. (2018). Rjags: R interface to Stan. R package version 2.18.2. Retrieved from https://CRAN.R-project.org/package=rstan
   Google Scholar
• Hoffman, M. D., & Gelman, A. (2014). The no-u-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(1), 1593–1623.
   Google Scholar
• Holman, R., & Glas, C. A. W. (2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology, 58(1), 1–17. doi:10.1111/j.2044-8317.2005.tb00312.x
   PubMed Web of Science ®Google Scholar
• Klein-Entink, R. H., Fox, J.-P., & van der Linden, W. J. (2009b). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74(1), 21–48. doi:10.1007/s11336-008-9075-y
   PubMed Web of Science ®Google Scholar
• Klein-Entink, R. H., Linden, W. J., & Fox, J.-P. (2009a). A Box-Cox normal model for response times. British Journal of Mathematical and Statistical Psychology, 62(3), 621–640. doi:10.1348/000711008X374126
   PubMed Web of Science ®Google Scholar
• Klein-Entink, R. H., Kuhn, J.-T., Hornke, L. F., & Fox, J.-P. (2009c). Evaluating cognitive theory: A joint modeling approach using responses and response times. Psychological Methods, 14(1), 54–75. doi:10.1037/a0014877
   PubMed Web of Science ®Google Scholar
• Köhler, C., Pohl, S., & Carstensen, C. H. (2015a). Investigating mechanisms for missing responses in competence tests. Psychological Test and Assessment Modeling, 57(4), 499–522.
   Google Scholar
• Köhler, C., Pohl, S., & Carstensen, C. H. (2015). Taking the missing propensity into account when estimating competence scores: Evaluation of item response theory models for nonignorable omissions. Educational and Psychological Measurement, 75(5), 850–874. doi:0.1177/0013164414561785 doi:10.1177/0013164414561785
   PubMed Web of Science ®Google Scholar
• Köhler, C., Pohl, S., & Carstensen, C. H. (2017). Dealing with item nonresponse in large-scale cognitive assessments: The impact of missing data methods on estimated explanatory relationships. Journal of Educational Measurement, 54(4), 397–419. doi:10.1111/jedm.12154
   Web of Science ®Google Scholar
• Lee, Y.-H., & Jia, Y. (2014). Using response time to investigate students’ test-taking behaviors in a NAEP computer-based study. Large-Scale Assessments in Education, 2(1), 8. doi:10.1186/s40536-014-0008-1
   Google Scholar
• Lewandowski, D., Kurowicka, D., & Joe, H. (2009). Generating random correlation matrices based on vines and extended onion method. Journal of Multivariate Analysis, 100(9), 1989–2001. doi:10.1016/j.jmva.2009.04.008
   Web of Science ®Google Scholar
• Lord, F. M. (1974). Estimation of latent ability and item parameters when there are omitted responses. Psychometrika, 39(2), 247–264. doi:10.1007/BF02291471
   Web of Science ®Google Scholar
• Lord, F. M. (1983). Maximum likelihood estimation of item response parameters when some responses are omitted. Psychometrika, 48(3), 477–482. doi:10.1007/BF02293689
   Web of Science ®Google Scholar
• Maris, G., & van der Maas, H. (2012). Speed-accuracy response models: Scoring rules based on response time and accuracy. Psychometrika, 77(4), 615–633. doi:10.1007/s11336-012-9288-y
   Web of Science ®Google Scholar
• Mislevy, R. J., & Wu, P.-K. (1996). Missing responses and IRT ability estimation: Omits, choice, time limits, and adaptive testing. ETS Research Report Series, 1996(2), i–36. doi:10.1002/j.2333-8504.1996.tb01708.x
   Google Scholar
• Molenaar, D., Oberski, D., Vermunt, J., & De Boeck, P. (2016). Hidden Markov item response theory models for responses and response times. Multivariate Behavioral Research, 51(5), 606–626. doi:10.1080/00273171.2016.1192983
   PubMed Web of Science ®Google Scholar
• Molenaar, D., Tuerlinckx, F., & van der Maas, H. L. (2015). A generalized linear factor model approach to the hierarchical framework for responses and response times. British Journal of Mathematical and Statistical Psychology, 68(2), 197–219. doi:10.1111/bmsp.12042
   PubMed Web of Science ®Google Scholar
• Muthén, L. K., & Muthén, B. O. (2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling, 9(4), 599–620. doi:10.1207/S15328007SEM0904_8
   Web of Science ®Google Scholar
• National Center for Education Statistics (2009, May 13). NAEP technical documentation. Retrieved from https://nces.ed.gov/nationsreportcard/tdw/
   Google Scholar
• Neal, R. M. (2011). MCMC using Hamiltonian dynamics. In S. Brooks, A. Gelman, G. Jones, & X. L. Meng, (Eds.), Handbook of Markov Chain Monte Carlo (pp. 113–162). Boca Raton, FL: CRC Press.
   Google Scholar
• OECD (2013). Technical report of the survey of adult skills (PIACC). Technical report, Organisation for Economic Co-operation and Development.
   Google Scholar
• OECD (2017). PISA 2015 technical report. Technical report, Organisation for Economic Co-operation and Development.
   Google Scholar
• O’Muircheartaigh, C., & Moustaki, I. (1999). Symmetric pattern models: A latent variable approach to item non-response in attitude scales. Journal of the Royal Statistical Society: Series A (Statistics in Society), 162(2), 177–194. doi:10.1111/1467-985X.00129
   Web of Science ®Google Scholar
• Petscher, Y., Mitchell, A. M., & Foorman, B. R. (2015). Improving the reliability of student scores from speeded assessments: An illustration of conditional item response theory using a computer-administered measure of vocabulary. Reading and Writing, 28(1), 31–56. doi:10.1007/s11145-014-9518-z
   PubMed Web of Science ®Google Scholar
• Pohl, S., & Becker, B. (2018). Performance of missing data approaches under nonignorable missing data conditions. Manuscript submitted for publication.
   Google Scholar
• Pohl, S., & Carstensen, C. H. (2012). NEPS technical report – Scaling the data of the competence tests. Bamberg, Germany: Otto-Friedrich- Universität, Nationales Bildungspanel.
   Google Scholar
• Pohl, S., Gräfe, L., & Rose, N. (2014). Dealing with omitted and not-reached items in competence tests: Evaluating approaches accounting for missing responses in item response theory models. Educational and Psychological Measurement, 74(3), 423–452. doi:10.1177/0013164413504926
   Web of Science ®Google Scholar
• Pohl, S., Ulitzsch, E., & von Davier, M. (2019). Using response times to models not-reached items due to time limits. Psychometrika, 1–29. doi:10.1007/s11336-019-09669-2
   PubMed Web of Science ®Google Scholar
• Pokropek, A. (2016). Grade of membership response time model for detecting guessing behaviors. Journal of Educational and Behavioral Statistics, 41(3), 300–325. doi:10.3102/1076998616636618
   Web of Science ®Google Scholar
• Qian, H., Staniewska, D., Reckase, M., & Woo, A. (2016). Using response time to detect item preknowledge in computer-based licensure examinations. Educational Measurement: Issues and Practice, 35(1), 38–47. doi:10.1111/emip.12102
   Web of Science ®Google Scholar
• R Development Core Team (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. http://www.R-project.org
   Google Scholar
• Rose, N. (2013). Item Nonresponses in Educational and Psychological Measurement. PhD thesis, Friedrich-Schiller-Universität Jena. https://d-nb.info/1036873145/34
   Google Scholar
• Rose, N., von Davier, M., & Nagengast, B. (2017). Modeling omitted and not-reached items in IRT models. Psychometrika, 82(3), 795–819. doi:10.1007/s11336-016-9544-7
   Web of Science ®Google Scholar
• Rose, N., von Davier, M., & Xu, X. (2010). Modeling nonignorable missing data with item response theory (IRT). ETS Research Report Series, 2010(1), i–53. doi:10.1002/j.2333-8504.2010.tb02218.x
   Google Scholar
• Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592. doi:10.1093/biomet/63.3.581
   Web of Science ®Google Scholar
• Schuurman, N., Grasman, R., & Hamaker, E. (2016). A comparison of inverse-wishart prior specifications for covariance matrices in multilevel autoregressive models. Multivariate Behavioral Research, 51(2-3), 185–206. doi:10.1080/00273171.2015.1065398
   PubMed Web of Science ®Google Scholar
• Stan Development Team. (2017). Stan modeling language: User’s guide and reference manual. Version 2.17.0.
   Google Scholar
• Ulitzsch, E., von Davier, M., & Pohl, S. (2018). A multi-process item response model for not-reached items due to time limits and quitting. Manuscript submitted for publication.
   Google Scholar
• van der Linden, W. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204. doi:10.3102/10769986031002181
   Web of Science ®Google Scholar
• van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308. doi:10.1007/s11336-006-1478-z
   Web of Science ®Google Scholar
• van der Linden, W. J. (2009). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46(3), 247–272. doi:10.1111/j.1745-3984.2009.00080.x
   Web of Science ®Google Scholar
• van der Linden, W. J., & Guo, F. (2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73(3), 365–384. doi:10.1007/s11336-007-9046-8
   Web of Science ®Google Scholar
• van der Linden, W. J., Klein-Entink, R. H., & Fox, J.-P. (2010). IRT parameter estimation with response times as collateral information. Applied Psychological Measurement, 34(5), 327–347. doi:10.1177/0146621609349800
   Web of Science ®Google Scholar
• van Erp, S., Mulder, J., & Oberski, D. L. (2017). Prior sensitivity analysis in default Bayesian structural equation modeling. Psychological Methods, 23(2), 363–388. doi:10.1037/met0000162
   PubMed Web of Science ®Google Scholar
• Venables, W. N., & Ripley, B. D. (2002). Modern applied statistics with S (4th ed.). ISBN 0-387-95457-0. New York, NY: Springer. Retrieved from http://www.stats.ox.ac.uk/pub/MASS4
   Google Scholar
• von Davier, M. (2005). A general diagnostic model applied to language testing data. ETS Research Report Series, 2005(2), i–35. doi:10.1002/j.2333-8504.2005.tb01993.x
   Google Scholar
• Wang, C., & Xu, G. (2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68(3), 456–477. doi:10.1111/bmsp.12054
   PubMed Web of Science ®Google Scholar
• Weeks, J. P., von Davier, M., & Yamamoto, K. (2016). Using response time data to inform the coding of omitted responses. Psychological Test and Assessment Modeling, 58(4), 671–701.
   Google Scholar
• Wise, S. L. (2017). Rapid-guessing behavior: Its identification, interpretation, and implications. Educational Measurement: Issues and Practice, 36(4), 52–61. doi:10.1111/emip.12165
   Web of Science ®Google Scholar
• Wise, S. L., & DeMars, C. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43(1), 19–38. doi:10.1111/j.1745-3984.2006.00002.x
   Web of Science ®Google Scholar
• Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183. doi:10.1207/s15324818ame1802_2
   Web of Science ®Google Scholar
• Xu, D., Chatterjee, A., & Daniels, M. (2016). A note on posterior predictive checks to assess model fit for incomplete data. Statistics in Medicine, 35(27), 5029–5039. doi:10.1002/sim.7040
   PubMed Web of Science ®Google Scholar
• Yamamoto, K., Khorramdel, L., & von Davier, M. (2013). Scaling PIAAC cognitive data. In Technical report of the survey of adult skills (PIAAC). Retrieved from http://www.oecd.org/skills/piaac/_TechnicalReport_17OCT13.pdf
   Google Scholar
• Yan, D., Mislevy, R. J., & Almond, R. G. (2003). Design and analysis in a cognitive assessment. ETS Research Report Series, 2003(2), i–47. doi:10.1002/j.2333-8504.2003.tb01924.x
   Google Scholar
• Zhan, P., Jiao, H., Wang, W.-C., & Man, K. (2018). A multidimensional hierarchical framework for modeling speed and ability in computer-based multidimensional tests. arXiv preprint arXiv:1807.04003.
   Google Scholar