Measuring Ability, Speed, or Both? Challenges, Psychometric Solutions, and What Can Be Gained From Experimental Control

REFERENCES

• Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412. doi:10.1016/j.jml.2007.12.005
   Web of Science ®Google Scholar
• Bassili, J. N. (1995). Response latency and the accessibility of voting intentions: What contributes to accessibility and how it affects vote choice. Personality and Social Psychology Bulletin, 21(7), 686–695. doi:10.1177/0146167295217003
   Web of Science ®Google Scholar
• Bassili, J. N. (1996). The how and why of response latency measurement in telephone surveys. In N. Schwarz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 319–346). San Francisco, CA: Jossey-Bass.
   Google Scholar
• Bassili, J. N., & Fletcher, J. F. (1991). Response-time measurement in survey-research—a method for Cati and a new look at nonattitudes. Public Opinion Quarterly, 55(3), 331–346. doi:10.1086/269265
   Web of Science ®Google Scholar
• Batchelder, W., & Riefer, D. (1999). Theoretical and empirical review of multinomial process tree modeling. Psychonomic Bulletin & Review, 6(1), 57–86. doi:10.3758/BF03210812
   PubMed Web of Science ®Google Scholar
• Baxter, B. (1941). An experimental analysis of the contributions of speed and level in an intelligence test. Journal of Educational Psychology, 32(4), 285–296. doi:10.1037/h0061115
   Google Scholar
• Breznitz, Z. (1987). Increasing first graders’ reading accuracy and comprehension by accelerating their reading rates. Journal of Educational Psychology, 79(3), 236–242. doi:10.1037/0022-0663.79.3.236
   Web of Science ®Google Scholar
• Breznitz, Z., & Berman, L. (2003). The underlying factors of word reading rate. Educational Psychology Review, 15(3), 247–265. doi:10.1023/A:1024696101081
   Web of Science ®Google Scholar
• Carroll, J. B. (1993). Human cognitive abilities. A survey of factor-analytical studies. New York, NY: Cambridge University Press.
   Google Scholar
• Chuderski, A. (2013). When are fluid intelligence and working memory isomorphic and when are they not? Intelligence, 41(4), 244–262. doi:10.1016/j.intell.2013.04.003
   Web of Science ®Google Scholar
• Cronbach, L. J., & Warrington, W. G. (1951). Time-limit tests: Estimating their reliability and degree of speeding. Psychometrika, 16(2), 167–188. doi:10.1007/BF02289113
   PubMedGoogle Scholar
• Davison, M. L., Semmes, R., Huang, L., & Close, C. N. (2012). On the reliability and validity of a numerical reasoning speed dimension derived from response times collected in computerized testing. Educational and Psychological Measurement, 72(2), 245–263. doi:10.1177/0013164411408412
   Web of Science ®Google Scholar
• De Ayala, R. J. (2009). The theory and practice of item response theory. New York, NY: Guilford Press.
   Google Scholar
• De Boeck, P., Bakker, M., Zwitser, R., Nivard, M., Hofman, A., Tuerlinckx, F., & Partchev, I. (2011). The estimation of item response models with the lmer function from the lme4 package in R. Journal of Statistical Software, 39(12), 1–28. doi:10.18637/jss.v039.i12
   PubMed Web of Science ®Google Scholar
• De Boeck, P., & Partchev, I. (2012). IRTrees: Tree-based item response models of the GLMM family. Journal of Statistical Software, 48, 1–28. doi:10.18637/jss.v048.c01
   Web of Science ®Google Scholar
• Dennis, I., & Evans, J. S. B. T. (1996). The speed–error trade-off problem in psychometric testing. British Journal of Psychology, 87(1), 105–129. doi:10.1111/j.2044-8295.1996.tb02579.x
   Web of Science ®Google Scholar
• Doran, H., Bates, D., Bliese, P., & Dowling, M. (2007). Estimating the multilevel Rasch model: With the lme4 package. Journal of Statistical Software, 20(2), 1–18. doi:10.18637/jss.v020.i02
   Web of Science ®Google Scholar
• Drechsler, R., Rizzo, P., & Steinhausen, H.-C. (2010). The impact of instruction and response cost on the modulation of response-style in children with ADHD. Behavioral and Brain Functions, 6(31). doi:10.1186/1744-9081-6-31
   PubMed Web of Science ®Google Scholar
• Eisenberg, P., & Wesman, A. G. (1941). Consistency in response and logical interpretation of psychoneurotic inventory items. Journal of Educational Psychology, 32(5), 321–338. doi:10.1037/h0060946
   Google Scholar
• Ferrando, P. J., & Lorenzo-Seva, U. (2007). An item response theory model for incorporating response time data in binary personality items. Applied Psychological Measurement, 31(6), 525–543. doi:10.1177/0146621606295197
   Web of Science ®Google Scholar
• Furneaux, W. D. (1961). Intellectual abilities and problem solving behavior. In H. J. Eysenck (Ed.), Handbook of abnormal psychology (pp. 167–192). New York, NY: Basic Books.
   Google Scholar
• Glas, C. A. W., & Pimentel, J. L. (2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 68(6), 907–922. doi:10.1177/0013164408315262
   Web of Science ®Google Scholar
• Goldhammer, F., & Kroehne, U. (2014). Controlling individuals’ time spent on task in speeded performance measures: Experimental time limits, posterior time limits, and response time modeling. Applied Psychological Measurement, 38(4), 255–267. doi:10.1177/0146621613517164
   Web of Science ®Google Scholar
• Goldhammer, F., Kroehne, U., & Hahnel, C. (2014, July). Timed administration of items increases convergent validity: Examples from word recognition and sentence verification. Paper presented at the 9th conference of the International Test Commission (ITC), San Sebastian, Spain.
   Google Scholar
• Goldhammer, F., Moosbrugger, H., & Krawietz, S. A. (2009). FACT-2—the Frankfurt adaptive concentration test: Convergent validity with self-reported cognitive failures. European Journal of Psychological Assessment, 25(2), 73–82. doi:10.1027/1015-5759.25.2.73
   Web of Science ®Google Scholar
• Goldhammer, F., Naumann, J., & Greiff, S. (2015). More is not always better: The relation between item response and item response time in Raven’s matrices. Journal of Intelligence, 3(1), 21–40. doi:10.3390/jintelligence3010021
   Google Scholar
• Goldhammer, F., Naumann, J., Stelter, A., Tóth, K., Rölke, H., & Klieme, E. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106(3), 608–626. doi:10.1037/a0034716
   Web of Science ®Google Scholar
• Gulliksen, H. (1950). Theory of mental tests. New York, NY: Wiley.
   Google Scholar
• Hacker, M., Goldhammer, F., & Kroehne, U. (2015, July). Controlling time-related individual differences in test-taking behavior by presenting time information. Paper presented at the 13th European Conference on Psychological Assessment, Zurich, Switzerland.
   Google Scholar
• Heitz, R. P. (2014). The speed-accuracy tradeoff: History, physiology, methodology, and behavior. Frontiers in Neuroscience, 8 (1–19). doi:10.3389/fnins.2014.00150
   Web of Science ®Google Scholar
• Jentzsch, I., & Leuthold, H. (2006). Control over speeded actions: A common processing locus for micro- and macro-trade-offs? Quarterly Journal of Experimental Psychology, 59(8), 1329–1337. doi:10.1080/17470210600674394
   Web of Science ®Google Scholar
• Kaernbach, C. (1991). Simple adaptive testing with the weighted up-down method. Perception & Psychophysics, 49(3), 227–229. doi:10.3758/BF03214307
   PubMed Web of Science ®Google Scholar
• Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. doi:10.1111/jedm.12000
   Web of Science ®Google Scholar
• Kelley, T. (1927). Interpretation of educational measurements. Yonkers, NY: World Book.
   Google Scholar
• Kendall, L. M. (1964). The effects of varying time limits on test validity. Educational and Psychological Measurement, 24(4), 789–800. doi:10.1177/001316446402400406
   Web of Science ®Google Scholar
• Klein Entink, R. H., Fox, J.-P., & van der Linden, W. J. (2009). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74(1), 21–48. doi:10.1007/s11336-008-9075-y
   PubMed Web of Science ®Google Scholar
• Klein Entink, R. H., Kuhn, J.-T., Hornke, L. F., & Fox, J.-P. (2009). Evaluating cognitive theory: A joint modeling approach using responses and response times. Psychological Methods, 14(1), 54–75. doi:10.1037/a0014877
   PubMed Web of Science ®Google Scholar
• Lee, Y.-H., & Chen, H. (2011). A review of recent response-time analyses in educational testing. Psychological Test and Assessment Modeling, 53(3), 359–379.
   Google Scholar
• Lee, Y.-H., & Jia, Y. (2014). Using response time to investigate students’ test-taking behaviors in a NAEP computer-based study. Large-scale Assessments in Education, 2(1), 1–24. doi:10.1186/s40536-014-0008-1
   Google Scholar
• Lien, M.-C., Ruthruff, E., Remington, R. W., & Johnston, J. C. (2005). On the limits of advance preparation for a task switch: Do people prepare all the task some of the time or some of the task all the time? Journal of Experimental Psychology: Human Perception and Performance, 31(2), 299–315. doi:10.1037/0096-1523.31.2.299
   PubMed Web of Science ®Google Scholar
• Lienert, G. A., & Ebel, O. (1960). Ein Index zur empirischen Bestimmung der Niveau-Eigenschaften eines psychologischen Tests [An index to empirically compute the level properties of a psychological test]. Metrika, 3(1), 117–127. doi:10.1007/BF02613444
   Google Scholar
• Loeys, T., Rosseel, Y., & Baten, K. (2011). A joint modeling approach for reaction time and accuracy in psycholinguistic experiments. Psychometrika, 76(3), 487–503. doi:10.1007/s11336-011-9211-y
   Web of Science ®Google Scholar
• Lohman, D. F. (1979). Spatial ability: Individual differences in speed and level (Technical report no. 9). Stanford, CA: Stanford University, Aptitude Research Project, School of Education.
   Google Scholar
• Lohman, D. F. (1986). The effect of speed-accuracy tradeoff on sex differences in mental rotation. Perception & Psychophysics, 39(6), 427–436. doi:10.3758/BF03207071
   PubMed Web of Science ®Google Scholar
• Lohman, D. F. (1989). Individual differences in errors and latencies on cognitive tasks. Learning and Individual Differences, 1(2), 179–202. doi:10.1016/1041-6080(89)90002-2
   Google Scholar
• Lu, Y., & Sireci, S. G. (2007). Validity issues in test speededness. Educational Measurement: Issues and Practice, 26(4), 29–37. doi:10.1111/j.1745-3992.2007.00106.x
   Google Scholar
• Luce, R. D. (1986). Response times: Their roles in inferring elementary mental organization. Oxford, UK: Oxford University Press.
   Google Scholar
• Maris, G., & van der Maas, H. (2012). Speed-accuracy response models: Scoring rules based on response time and accuracy. Psychometrika, 77(4), 615–633. doi:10.1007/s11336-012-9288-y
   Web of Science ®Google Scholar
• Messick, S. (1994). The matter of style: Manifestations of personality in cognition, learning, and teaching. Educational Psychologist, 29(3), 121–136. doi:10.1207/s15326985ep2903_2
   Web of Science ®Google Scholar
• Molenaar, D., Tuerlinckx, F., & van der Maas, H. L. J. (2015). A bivariate generalized linear item response theory modeling framework to the analysis of responses and response times. Multivariate Behavioral Research, 50(1), 56–74. doi:10.1080/00273171.2014.962684
   PubMed Web of Science ®Google Scholar
• Moosbrugger, H., & Goldhammer, F. (2007). FAKT-II. Frankfurter Adaptiver Konzentrationsleistungs-Test II. Computerprogramm. Grundlegend neu bearbeitete und neu normierte 2. Auflage des FAKT von Moosbrugger und Heyden (1997) [FAKT-II. Frankfurt adaptive concentration-test. Second, completely revised and renormed edition of the FAKT by Moosbrugger and Heyden (1997)]. Bern, Switzerland: Huber.
   Google Scholar
• Naumann, J., & Goldhammer, F. (2015). The time on task effect in digital reading is moderated by persons’ skills and tasks’ demands. Manuscript submitted for publication.
   Google Scholar
• Nietfeld, J., & Bosma, A. (2003). Examining the self-regulation of impulsive and reflective response styles on academic tasks. Journal of Research in Personality, 37(3), 118–140. doi:10.1016/s0092-6566(02)00564-0
   Web of Science ®Google Scholar
• Onwuegbuzie, A. J., & Seaman, M. A. (1995). The effect of time constraints and statistics test anxiety on test performance in a statistics course. Journal of Experimental Education, 63(2), 115–124. doi:10.1080/00220973.1995.9943816
   Web of Science ®Google Scholar
• Partchev, I., & De Boeck, P. (2012). Can fast and slow intelligence be differentiated? Intelligence, 40(1), 23–32. doi:10.1016/j.intell.2011.11.002
   Web of Science ®Google Scholar
• Partchev, I., De Boeck, P., & Steyer, R. (2013). How much power and speed is measured in this test? Assessment, 20(2), 242–252. doi:10.1177/1073191111411658
   PubMed Web of Science ®Google Scholar
• Preckel, F., Wermer, C., & Spinath, F. M. (2011). The interrelationship between speeded and unspeeded divergent thinking and reasoning, and the role of mental speed. Intelligence, 39(5), 378–388. doi:10.1016/j.intell.2011.06.007
   Web of Science ®Google Scholar
• Ranger, J., & Kuhn, J.-T. (2012). Improving item response theory model calibration by considering response times in psychological tests. Applied Psychological Measurement, 36(3), 214–231. doi:10.1177/0146621612439796
   Web of Science ®Google Scholar
• Ranger, J., Kuhn, J.-T., & Gaviria, J.-L. (2015). A race model for responses and response times in tests. Psychometrika 80(3), 791–810. doi:10.1007/s11336-014-9427-8
   Google Scholar
• Ranger, J., & Ortner, T. (2012). The case of dependency of responses and response times: A modeling approach based on standard latent trait models. Psychological Test and Assessment Modeling, 54(2), 128–148.
   Google Scholar
• Ratcliff, R., & Smith, P. L. (2004). A comparison of sequential sampling models for two-choice reaction time. Psychological Review, 111(2), 333–367. doi:10.1037/0033-295X.111.2.333
   PubMed Web of Science ®Google Scholar
• Reed, A. V. (1973). Speed-accuracy trade-off in recognition memory. Science, 181(4099), 574–576. doi:10.1126/science.181.4099.574
   PubMed Web of Science ®Google Scholar
• Rindler, S. E. (1979). Pitfalls in assessing test speededness. Journal of Educational Measurement, 16(4), 261–270. doi:10.1111/j.1745-3984.1979.tb00107.x
   Web of Science ®Google Scholar
• Roskam, E. E. (1987). Toward a psychometric theory of intelligence. In E. E. Roskam & R. Suck (Eds.), Progress in mathematical psychology, 1 (pp. 151–174). New York, NY: Elsevier Science.
   Google Scholar
• Roskam, E. E. (1997). Models for speed and time-limit tests. In W. J. van der Linden, & R. Hambleton (Eds.), Handbook of modern item response theory (pp. 87–208). New York, NY: Springer.
   Google Scholar
• Rouder, J. N., Province, J. M., Morey, R. D., Gomez, P., & Heathcote, A. (2015). The lognormal race: A cognitive-process model of choice and latency with desirable psychometric properties. Psychometrika, 80(2), 491–513. doi:10.1007/s11336-013-9396-3
   Google Scholar
• Salomon, G., & Globerson, T. (1987). Skill may not be enough: The role of mindfulness in learning and transfer. International Journal of Educational Research, 11, 623–637. doi:10.1016/0883-0355(87)90006-1
   Google Scholar
• Schnipke, D. L., & Scrams, D. J. (1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34(3), 213–232. doi:10.1111/j.1745-3984.1997.tb00516.x
   Web of Science ®Google Scholar
• Schouten, J. F., & Bekker, J. A. M. (1967). Reaction time and accuracy. Acta Psychologica, 27, 143–153. doi:10.1016/0001-6918(67)90054-6
   PubMed Web of Science ®Google Scholar
• Semmes, R., Davison, M. L., & Close, C. (2011). Modeling individual differences in numerical reasoning speed as a random effect of response time limits. Applied Psychological Measurement, 35(6), 433–446. doi:10.1177/0146621611407305
   Web of Science ®Google Scholar
• Sorensen, L. J., & Woltz, D. J. (2015). Transforming response time and errors to address tradeoffs in complex measures of processing speed. Learning and Individual Differences, 40, 73–83. doi:10.1016/j.lindif.2015.04.002
   Web of Science ®Google Scholar
• Stafford, R. E. (1971). The speededness quotient: A new descriptive statistic for tests. Journal of Educational Measurement, 8(4), 275–277. doi:10.1111/jedm.1971.8.issue-4
   Web of Science ®Google Scholar
• Thorndike, E. L., Bregman, E. O., Cobb, M. V., & Woodyard, E. (1926). The measurement of intelligence. New York, NY: Teachers College Bureau of Publications.
   Google Scholar
• Thurstone, L. L. (1937). Ability, motivation, and speed. Psychometrika, 2(4), 249–254. doi:10.1007/BF02287896
   Google Scholar
• Tuerlinckx, F., & De Boeck, P. (2005). Two interpretations of the discrimination parameter. Psychometrika, 70(4), 629–650. doi:10.1007/s11336-000-0810-3
   Web of Science ®Google Scholar
• Van Breukelen, G. J. P. (2005). Psychometric modeling of response speed and accuracy with mixed and conditional regression. Psychometrika, 70(2), 359–376. doi:10.1007/s11336-003-1078-0
   Web of Science ®Google Scholar
• Van Breukelen, G. J. P., & Roskam, E. E. C. I. (1991). A Rasch model for the speed-accuracy tradeoff in time limited tests. In J.-P. Doignon & J.-C. Falmagne (Eds.), Mathematical psychology (pp. 251–271). New York, NY: Springer.
   Google Scholar
• Van der Linden, W. J. (2005). Linear models for optimal test design. New York, NY: Springer.
   Google Scholar
• Van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308. doi:10.1007/s11336-006-1478-z
   Web of Science ®Google Scholar
• Van der Linden, W. J. (2009a). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46(3), 247–272. doi:10.1111/j.1745-3984.2009.00080.x
   Web of Science ®Google Scholar
• Van der Linden, W. J. (2009b). Predictive control of speededness in adaptive testing. Applied Psychological Measurement, 33(1), 25–41. doi:10.1177/0146621607314042
   Web of Science ®Google Scholar
• Van der Linden, W. J., & Glas, C. A. W. (2010). Statistical tests of conditional independence between responses and/or response times on test items. Psychometrika, 75(1), 120–139. doi:10.1007/s11336-009-9129-9
   Web of Science ®Google Scholar
• Van der Linden, W. J., Scrams, D. J., & Schnipke, D. L. (1999). Using response-time constraints to control for differential speededness in computerized adaptive testing. Applied Psychological Measurement, 23(3), 195–210. doi:10.1177/01466219922031329
   Web of Science ®Google Scholar
• Van der Maas, H. L. J., Molenaar, D., Maris, G., Kievit, R. A., & Borsboom, D. (2011). Cognitive psychology meets psychometric theory: On the relation between process models for decision making and latent variable models for individual differences. Psychological Review, 118(2), 339–356. doi:10.1037/a0022749
   PubMed Web of Science ®Google Scholar
• Veenman, M. V. J., & Beishuizen, J. J. (2004). Intellectual and metacognitive skills of novices while studying texts under conditions of text difficulty and time constraint. Learning and Instruction, 14(6), 621–640. doi:10.1016/j.learninstruc.2004.09.004
   Web of Science ®Google Scholar
• Verhelst, N. D., Verstralen, H. H. F. M., & Jansen, M. G. (1997). A logistic model for timelimit tests. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 169–185). New York, NY: Springer.
   Google Scholar
• Vernon, P. A., Nador, S., & Kantor, L. (1985). Reaction times and speed-of-processing: Their relationship to timed and untimed measures of intelligence. Intelligence, 9(4), 357–374. doi:10.1016/0160-2896(85)90020-0
   Web of Science ®Google Scholar
• Wainer, H., Dorans, N. J., Green, B. F., Steinberg, L., Flaugher, R., Mislevy, R. J., & Thissen, D. (1990). Computerized adaptive testing: A primer. Hillsdale, NJ: Lawrence Erlbaum.
   Google Scholar
• Walczyk, J. J., & Griffith-Ross, D. A. (2006). Time restriction and the linkage between subcomponent efficiency and algebraic inequality success. Journal of Educational Psychology, 98(3), 617–627. doi:10.1037/0022-0663.98.3.617
   Web of Science ®Google Scholar
• Walczyk, J. J., Kelly, K. E., Meche, S. D., & Braud, H. (1999). Time limitations enhance reading comprehension. Contemporary Educational Psychology, 24(2), 156–165. doi:10.1006/ceps.1998.0992
   PubMed Web of Science ®Google Scholar
• Wang, T., & Hanson, B. A. (2005). Development and calibration of an item response model that incorporates response time. Applied Psychological Measurement, 29(5), 323–339. doi:10.1177/0146621605275984
   Web of Science ®Google Scholar
• White, P. O. (1973). Individual differences in speed, accuracy and persistence: A mathematical model for problem solving. In H. J. Eysenck (Ed.), The measurement of intelligence. Lancaster, UK: Medical and Technical Publishing.
   Google Scholar
• Wickelgren, W. A. (1977). Speed-accuracy tradeoff and information processing dynamics. Acta Psychologica, 41, 67–85. doi:10.1016/0001-6918(77)90012-9
   Web of Science ®Google Scholar
• Wilhelm, O., & Schulze, R. (2002). The relation of speeded and unspeeded reasoning with mental speed. Intelligence, 30(6), 537–554. doi:10.1016/S0160-2896(02)00086-7
   Web of Science ®Google Scholar
• Wise, S. L., & DeMars, C. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43(1), 19–38. doi:10.1111/j.1745-3984.2006.00002.x
   Web of Science ®Google Scholar
• Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183. doi:10.1207/s15324818ame1802_2
   Web of Science ®Google Scholar
• Wright, D. E., & Dennis, I. (1999). Exploiting the speed-accuracy trade-off. In P. L. Ackerman, P. C. Kyllonen, & R. D. Roberts (Eds.), Learning and individual differences: Process, trait, and content determinants (pp. 231–248). Washington, DC: American Psychological Association.
   Google Scholar
• Yamamoto, K., & Everson, H. (1997). Modeling the effects of test length and test time on parameter estimation using the HYBRID model. In J. Rost & R. Langeheine (Eds.), Applications of latent trait and latent class models in the social sciences (pp. 89–98). Münster, Germany: Waxman.
   Google Scholar
• Zhang, J., & Rowe, J. B. (2014). Dissociable mechanisms of speed-accuracy tradeoff during visual perceptual learning are revealed by a hierarchical drift-diffusion model. Frontiers in Neuroscience, 8, 69. doi:10.3389/fnins.2014.00069
   PubMed Web of Science ®Google Scholar