Item Difficulty and Interviewer Knowledge Effects on the Accuracy and Consistency of Examinee Response Processes in Verbal Reports

REFERENCES

• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (AERA, APA, NCME) . 1999 . Standards for educational and psychological testing , Washington , DC : Author .
   Google Scholar
• American Institutes for Research (AIR). (2009). Cognitive lab testing. http://www.air.org/topics/topic_cognitive_lab_testing.aspx (http://http://www.air.org/topics/topic_cognitive_lab_testing.aspx)
   Google Scholar
• Beilock , S. L. and Carr , T. H. 2001 . On the fragility of skilled performance: What governs choking under pressure? . Journal of Experimental Psychology: General , 130 : 701 – 725 .
   PubMed Web of Science ®Google Scholar
• Beilock , S. L. , Kulp , C. A. , Holt , L. E. and Carr , T. H. 2004 . More on the fragility of performance: Choking under pressure in mathematical problem solving . Journal of Experimental Psychology: General , 133 : 584 – 600 .
   PubMed Web of Science ®Google Scholar
• Boekaerts , M. and Corno , L. 2005 . Self-regulation in the classroom: A perspective on assessment and intervention . Applied Psychology: An International Review , 54 : 199 – 231 .
   Web of Science ®Google Scholar
• Butler , R. and Neuman , O. 1995 . Effects of task and ego achievement goals on help-seeking behaviors and attitudes . Journal of Educational Psychology , 87 : 261 – 271 .
   Web of Science ®Google Scholar
• Cain , K. M. and Dweck , C. S. 1995 . The relation between motivational patterns and achievement cognitions through the elementary school years . Merrill-Palmer Quarterly , 41 : 25 – 52 .
   Web of Science ®Google Scholar
• Cohen , D. J. and Snowden , J. L. 2008 . The relations between document familiarity, frequency, and prevalence and document literacy performance among adult readers . Reading Research Quarterly , 43 ( 1 ) : 9 – 26 .
   Google Scholar
• Covington , M. V. 1992 . Making the grade , Cambridge , England : Cambridge University Press .
   Google Scholar
• Darling-Hammond , L. 1996 . What matters most: A competent teacher for every child . Phi Delta Kappan , 77 : 193 – 201 .
   Google Scholar
• Desimone , L. M. and Le Floch , K. C. 2004 . Are we asking the right questions? Using cognitive interviews to improve surveys in education research . Educational Evaluation and Policy Analysis , 26 : 1 – 22 .
   Web of Science ®Google Scholar
• Dibner , A. S. 1956 . Cue-counting: A measure of anxiety in interviews . Journal of Consulting Psychology , 20 : 475 – 478 .
   PubMedGoogle Scholar
• Ercikan , K. , Arim , R. , Law , D. , Domene , J. , Gagnon , F. and Lacroix , S. 2010 . Application of think aloud protocols for examining and confirming sources of differential item functioning identified by expert review . Educational Measurement: Issues and Practice , 29 : 24 – 35 .
   Google Scholar
• Ericsson , K. A. 2006 . “ Protocol analysis and expert thought: Concurrent verbalizations of thinking during experts' performance on representative tasks ” . In The Cambridge handbook of expertise and expert performance , Edited by: Ericsson , K. A. , Charness , N. , Feltovich , P. J. and Hoffman , R. R. 223 – 241 . Cambridge , UK : Cambridge University Press .
   Google Scholar
• Ericsson , K. A. and Simon , H. A. 1993 . Protocol analysis: Verbal reports as data , Cambridge , MA : The MIT Press .
   Google Scholar
• Ferrara , S. and DeMauro , G. E. 2006 . “ Standardized assessment of individual achievement in K–12 ” . In Educational measurement , 4th , Edited by: Brennan , R. L. 579 – 621 . Westport , CT : National Council on Measurement in Education and American Council on Education .
   Google Scholar
• Goldhaber , D. D. and Brewer , D. J. 2000 . Does teacher certification matter? High school teacher certification status and student achievement . Educational Evaluation and Policy Analysis , 22 : 129 – 145 .
   Web of Science ®Google Scholar
• Gorin , J. S. 2006 . Test design with cognition in mind . Educational Measurement: Issues and Practice , 25 ( 4 ) : 21 – 35 .
   Google Scholar
• Kane , M. T. 2006 . “ Validation ” . In Educational measurement , 4th , Edited by: Brennan , R. L. 17 – 64 . Westport , CT : National Council on Measurement in Education and American Council on Education .
   Google Scholar
• Krause , M. S. and Pilisuk , M. 1961 . Anxiety in verbal behavior: A validation study . Journal of Consulting Psychology , 25 : 414 – 419 .
   PubMedGoogle Scholar
• Leighton , J. P. 2004 . Avoiding misconception, misuse, and missed opportunities: The collection of verbal reports in educational achievement testing . Educational Measurement: Issues and Practice , 23 : 6 – 15 .
   Google Scholar
• Leighton , J. P. , Cui , Y. and Cor , M. K. 2009 . Testing expert-based and student-based cognitive models: An application of the attribute hierarchy method and hierarchical consistency index . Applied Measurement in Education , 22 : 229 – 254 .
   Web of Science ®Google Scholar
• Lewis , B. and Linder , D. 1997 . Thinking about choking? Attentional processes and paradoxical performance . Personality and Social Psychology Bulletin , 23 : 937 – 944 .
   PubMed Web of Science ®Google Scholar
• Mahl , G. F. 1956 . Disturbances and silences in the patient's speech in psychotherapy . Journal of Applied Social Psychology, , 53 : 1 – 15 .
   Google Scholar
• Mahl , G. F. 1987 . “ Everyday disturbances of speech ” . In Language in psychotherapy: Strategies of discovery , Edited by: Russell , R. L. 213 – 269 . New York , NY : Plenum .
   Google Scholar
• Midgley , C. and Urdan , T. 2001 . Academic self-handicapping and performance goals: A further examination . Contemporary Educational Psychology , 26 : 61 – 75 .
   PubMed Web of Science ®Google Scholar
• Monk , D. H. and King , J. 1994 . “ Multi-level teacher resource effects on pupil performance in secondary mathematics and science: The role of teacher subject matter preparation ” . In Contemporary policy issues: Choices and consequences in education , Edited by: Ehrenberg , R. G. 29 – 58 . Ithaca , NY : ILR Press .
   Google Scholar
• Norris , S. P. 1988 . Controlling for background beliefs when developing multiple-choice critical thinking tests . Educational Measurement , 7 : 5 – 11 .
   Google Scholar
• Norris , S. P. 1990 . Effect of eliciting verbal reports of thinking on critical thinking performance . Journal of Educational Measurement , 27 : 41 – 58 .
   Web of Science ®Google Scholar
• Norris , S. P. 1991 . “ Informal reasoning assessment: Using verbal reports of thinking to improve multiple-choice test validity ” . In Informal reasoning and education , Edited by: Voss , J. F. , Perkins , D. N. and Segal , J. W. 451 – 472 . Hillsdale , NJ : Erlbaum .
   Google Scholar
• O'Neil , H. F. and Abedi , J. December 1996 . Reliability and Validity of a State Metacognitive Inventory: Potential for Alternative Assessment , December , Los Angeles , CA : CSE Technical Report 469. National Center for Research on Evaluation Standards and Student Testing (CRESST). University of California .
   Google Scholar
• Pintrich , P. R. 2000 . Multiple goals, multiple pathways: The role of goal orientation in learning and achievement . Journal of Educational Psychology , 92 : 544 – 555 .
   Web of Science ®Google Scholar
• Ryan , A. M. and Pintrich , P. R. 1997 . Should I ask for help? The role of motivation and attitudes in adolescents' help seeking in math class . Journal of Educational Psychology , 89 : 329 – 341 .
   Web of Science ®Google Scholar
• Ryan , K. E. and Ryan , A. M. 2005 . Psychological processes underlying stereotype threat and standardized math test performance . Educational Psychologist , 40 : 53 – 63 .
   Web of Science ®Google Scholar
• Sawyer , T. P. Jr. and Hollis-Sawyer , L. A. 2005 . Predicting stereotype threat, test anxiety, and cognitive ability test performance: An examination of three models . International Journal of Testing , 5 : 225 – 246 .
   Google Scholar
• Shrout , P. E. and Fleiss , J. L. 1979 . Intraclass correlations: Uses in assessing rater reliability . Psychological Bulletin , 2 : 420 – 428 .
   Web of Science ®Google Scholar
• Snow , R. E. and Lohman , D. F. 1989 . “ Implications of cognitive psychology for educational measurement ” . In Educational measurement , 3rd , Edited by: Linn , R. L. 263 – 331 . New York , NY : American Council on Education, Macmillan .
   Google Scholar
• Spielberger , C. D. , Gonzalez , H. P. , Taylor , C. J. , Anton , E. D. , Algaze , B. , Ross , G. R. and Westberry , L. G. 1980 . Manual for the test anxiety inventory (“Test Attitude Inventory”) , Redwood City , CA : Consulting Psychologists Press .
   Google Scholar
• Sundre , D. L. and Kitsantas , A. 2004 . An exploration of the psychology of the examinee: Can examinee self-regulation and test-taking motivation predict consequential and non-consequential test performance? . Contemporary Educational Psychology , 29 : 6 – 26 .
   Web of Science ®Google Scholar
• Turner , J. C. , Midgley , C. , Meyer , D. K. , Gheen , M. , Anderman , E. M. , Kang , Y. and Patrick , H. 2002 . The classroom environment and students' reports of avoidance strategies in mathematics . Journal of Educational Psychology , 94 : 88 – 106 .
   Web of Science ®Google Scholar
• Wang , X. 2011 . Evaluating four coding schemes for categorizing cognitive models in verbal reports for educational measurement studies. Unpublished manuscript , Leighton, J. P.
   Google Scholar
• Willis , G. B. 2005 . Cognitive interviewing: A tool for improving questionnaire design , Thousand Oaks , CA : Sage Publications .
   Google Scholar
• Wilson , T. D. 1994 . The proper protocol: Validity and completeness of verbal reports . Psychological Science , 5 : 249 – 252 .
   Web of Science ®Google Scholar
• Wolf , L. F. and Smith , J. K. 1995 . The consequence of consequence: Motivation, anxiety, and test performance . Applied Measurement in Education , 8 : 227 – 242 .
   Web of Science ®Google Scholar
• Wolf , L. F. , Smith , J. K. and Birnbaum , M. E. 1995 . Consequence of performance, test motivation and mentally taxing items . Applied Measurement in Education , 8 : 341 – 352 .
   Web of Science ®Google Scholar
• Zucker, S., Sassman, C., & Case, B. J. (February, 2004). Cognitive Labs. Technical Report. Pearson Education. http://pearsonassess.com/NR/rdonlyres/E5CD33E6-D234-46F3-885A-9358575372FB/0/CognitiveLabs_Final.pdf (http://http://pearsonassess.com/NR/rdonlyres/E5CD33E6-D234-46F3-885A-9358575372FB/0/CognitiveLabs_Final.pdf)
   Google Scholar