Cognitive Processing of Self-Report Items in Educational Research: Do They Think What We Mean?

REFERENCES

• Ainley , M. and Hidi , S. 2002 . “ Dynamic measures for studying interest and learning ” . In Advances in motivation and achievement: New directions in measures and methods , Edited by: Pintrich , P. R. and Maehr , M. L. Vol. 12 , 43 – 76 . Amsterdam : Elsevier .
   Google Scholar
• Alwin , D. F. 2001 . Research on survey quality . Sociological Methods and Research , 20 : 1 – 39 .
   Google Scholar
• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education . 1985 . Standards for educational and psychological testing , Washington , DC : American Psychological Association .
   Google Scholar
• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education . 1999 . Standards for educational and psychological testing , Washington , DC : American Psychological Association .
   Google Scholar
• Ames , C. 1992 . Classrooms: Goals, structures, and student motivation . Journal of Educational Psychology. , 84 : 261 – 271 .
   Web of Science ®Google Scholar
• Arthur , M. W. , Hawkins , J. D. , Pollard , J. A. , Catalano , R. F. and Baglioni , A. J. 2002 . Measuring risk and protective factors for substance abuse, delinquency, and other adolescent problem behaviors: The Communities That Care Youth Survey . Evaluation Review. , 26 : 575 – 601 .
   PubMed Web of Science ®Google Scholar
• Bandura , A. 1997 . Self-efficacy: The exercise of control , New York : W. H. Freeman .
   Google Scholar
• Barone , D. F. , Maddux , J. E. and Snyder , C. R. 1997 . Social cognitive psychology: History and current domains , New York : Plenum .
   Google Scholar
• Bjorner , J. B. , Kosinski , M. and Ware , J. E. Jr. 2003 . Calibration of an item pool for assessing the burden of headaches: An application of item response theory to the Headache Impact Test (HIT™) . Quality of Life Research , 12 : 913 – 933 .
   PubMed Web of Science ®Google Scholar
• Blazevski , J. , De Groot , E. , Friedel , J. and Musu , L. Lessons learned from cognitive pretesting: Students' sense of efficacy in mathematics . Paper presented at the annual meeting of the American Educational Research Association . San Francisco , CA . April .
   Google Scholar
• Boekaerts , M. 2002 . “ The on-line motivation questionnaire: A self-report instrument to assess students' context sensitivity ” . In Advances in motivation and achievement: New directions in measures and methods , Edited by: Pintrich , P. R. and Maehr , M. L. Vol. 12 , 77 – 120 . Amsterdam : Elsevier .
   Google Scholar
• Bong , M. and Skaalvik , E. M. 2003 . Academic self-concept and self-efficacy: How different are they really? . Educational Psychology Review , 15 : 1 – 40 .
   Web of Science ®Google Scholar
• Camara , W. J. and Lane , S. 2006 . A historical perspective and current views on the standards for educational and psychological testing . Educational Measurement: Issues and Practice , 25 ( 3 ) : 35 – 41 .
   Google Scholar
• Cronbach , L. J. 1971 . “ Test validation ” . In Educational measurement, , 2nd ed. , Edited by: Thorndike , R. L. Washington , DC : American Council on Education .
   Google Scholar
• DeMaio , T. J. and Rothgeb , J. M. 1996 . “ Cognitive interviewing techniques: In the lab and in the field ” . In Answering questions: Methodology for cognitive and communicative processes in survey research , Edited by: Schwarz , N. and Sudman , S. 177 – 196 . San Francisco : Jossey-Bass .
   Google Scholar
• Desimone , L. M. and Le Floch , K. C. 2004 . Are we asking the right questions: Using cognitive interviews to improve surveys in education research . Educational Evaluation and Policy Analysis. , 26 : 1 – 22 .
   Web of Science ®Google Scholar
• Eccles , J. S. , Wigfield , A. and Schiefele , U. 1998 . “ Motivation to succeed ” . In Handbook of child psychology, , 5th ed. , Edited by: Damon , W. and Eisenberg , N. Vol. 3 , 1017 – 1095 . New York : Wiley . Social, emotional, and personality development, Vol. Ed.
   Google Scholar
• Elliot , A. J. 2005 . “ A conceptual history of the achievement goal construct ” . In Handbook of competence and motivation , Edited by: Elliot , A. J. and Dweck , C. S. 52 – 72 . New York : Guilford .
   Google Scholar
• Foddy , W. 1998 . An empirical evaluation of in-depth probes used to pretest survey questions . Sociological Methods and Research , 27 : 103 – 133 .
   Web of Science ®Google Scholar
• Forsyth , B. H. and Lessler , J. T. 1991 . “ Cognitive laboratory methods: A taxonomy ” . In Measurement errors in surveys , Edited by: Biemer , P. P. , Groves , R. M. , Lyberg , L. E. , Mathiowietz , N. A. and Sudman , S. 393 – 418 . New York : Wiley .
   Google Scholar
• Fraser , B. 1997 . “ Classroom environments ” . In Psychology and educational practice , Edited by: Walberg , H. J. H. and Geneva , D. 323 – 341 . Berkeley , CA : McCutchan .
   Google Scholar
• Gilbert , M. C. and Kempler , T. M. Lessons learned from cognitive pretesting: Students' Perceptions of their teachers' use of “real world” connections . Paper presented at the annual meeting of the American Educational Research Association . San Francisco , CA . April .
   Google Scholar
• Green , J. L. , Camilli , G. and Elmore , P. B. 2006 . Handbook of complementary methods in education research , Mahwah , NJ : Lawrence Erlbaum Associates .
   Google Scholar
• Greeno , J. G. 2006 . “ Theoretical and practical advances through research on learning ” . In Handbook of complementary methods in education research , Edited by: Green , J. L. , Camilli , G. and Elmore , P. B. 795 – 822 . Mahwah , NJ : Lawrence Erlbaum Associates .
   Google Scholar
• Hassin , R. , Uleman , J. S. and Bargh , J. A. 2005 . The new unconscious , New York : Oxford University Press .
   Google Scholar
• Hastie , R. 1987 . “ Information processing theory for survey researchers ” . In Social information processing and survey methodology , Edited by: Hippler , H. J. , Schwartz , N. and Sudman , S. 42 – 70 . New York : Springer-Verlag .
   Google Scholar
• Hofer , B. and Pintrich , P. R. , eds. 2002 . Personal epistemology: The psychology of beliefs about knowledge and knowing , Mahwah , NJ : Lawrence Erlbaum Associates .
   Google Scholar
• Jabine , T. B. , Straf , M. L. , Tanur , J. M. and Tourangeau , R. 1984 . Cognitive aspects of survey methodology: Building a bridge between disciplines , Washington , DC : National Academy Press .
   Google Scholar
• Järvelä , S. , Salonon , P. and Lepola , J. 2002 . “ Dynamic assessment as a key to understanding student motivation in a classroom context ” . In Advances in motivation and achievement: New directions in measures and methods , Edited by: Pintrich , P. R. and Maehr , M. L. Vol. 12 , 207 – 240 . Amsterdam : Elsevier .
   Google Scholar
• Jobe , J. B. and Mingay , D. J. 1989 . Cognitive research improves questionnaires . American Journal of Public Health , 79 : 1053 – 1055 .
   PubMed Web of Science ®Google Scholar
• Kane , M. 1992 . An argument-based approach to validity . Psychological Bulletin. , 112 : 527 – 535 .
   Web of Science ®Google Scholar
• Kane , M. 2001 . Current concerns in validity theory . Journal of Educational Measurement , 38 : 319 – 342 .
   Web of Science ®Google Scholar
• Kaplan , A. , Middleton , M. J. , Urdan , T. and Midgley , C. 2002 . “ Achievement goals and goal structures ” . In Goals, goal structures and patterns of adaptive learning , Edited by: Midgley , C. 21 – 53 . Mahwah , NJ : Lawrence Erlbaum Associates .
   Google Scholar
• Karabenick , S. A. Chair . Cognitive validity and the construct validation of motivation-related assessments: what, why, and how . Symposium presented at the annual meeting of the American Educational Research Association . San Francisco , CA . April .
   Google Scholar
• Karabenick , S. A. and Woolley , M. E. The role of cognitive validity in the construct validation of motivation-related assessment . Paper presented at the annual meeting of the American Educational Research Association . San Francisco , CA . April .
   Google Scholar
• Kelly , K. L. , Bonney , C. R. and Ammon , B. V. Cognitive pretesting: A revealing look at students' understanding about classroom mastery goal structure . Paper presented at the annual meeting of the American Educational Research Association . San Francisco , CA . April .
   Google Scholar
• Meece , J. L. , Anderman , E. M. and Anderman , L. H. 2006 . “ Classroom goal structure, student motivation, and academic achievement ” . In Annual review of psychology , Edited by: Fiske , S. T. , Kazdin , A. E. and Schachter , D. L. Vol. 57 , 487 – 503 . Palo Alto , CA : Annual Reviews .
   Google Scholar
• Messick , S. 1989 . “ Validity ” . In Education measurement, , 3rd ed. , Edited by: Linn , R. L. 13 – 103 . New York : Macmillan .
   Google Scholar
• Messick , S. 1995 . Validity of psychological assessment: Validation of inferences from persons' responses and performance as scientific inquiry into scoring meaning . American Psychologist , 9 : 741 – 749 .
   Web of Science ®Google Scholar
• Messick , S. Standards-based score interpretation: Establishing valid grounds for valid inferences . (Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments) . Washington , DC : U.S. Government Printing Office .
   Google Scholar
• Messick , S. 1996b . “ Validity of performance assessment ” . In Technical issues in large-scale performance assessment , Edited by: Phillips , G. 1 – 18 . Washington , DC : National Center for Educational Statistics .
   Google Scholar
• Midgley , C. , Maehr , M. L. , Hicks , L. , Roeser , R. , Urdan , T. Anderman , E. 1996 . The Patterns of Adaptive Learning Scales (PALS) , Ann Arbor : The University of Michigan .
   Google Scholar
• Midgley , C. , Maehr , M. L. , Hruda , L. , Anderman , E. , Anderman , L. Freeman , K. 2000 . Manual for the Patterns of Adaptive Learning Scales (PALS) , Ann Arbor : University of Michigan .
   Google Scholar
• Murphy , P. K. and Alexander , P. A. 2000 . A motivated exploration of motivation terminology . Contemporary Educational Psychology , 25 : 3 – 53 .
   PubMed Web of Science ®Google Scholar
• National Center for Education Statistics (NCES) . 2003 . NAEP background questionnaires http://www.nces.ed.gov/nationsreportcard/bgquest.aspRetrieved February 8, 2005, from
   Google Scholar
• National Council of Teachers of Mathematics . 1989 . Curriculum and evaluation standards for school mathematics , Reston , VA : Author .
   Google Scholar
• National Council of Teachers of Mathematics . 2000 . Principles and standards for school mathematics , Reston , VA : Author .
   Google Scholar
• Nolen , S. B. 2003 . Learning environment, motivation, and achievement in high school science . Journal of Research in Science Teaching. , 40 : 347 – 368 .
   Web of Science ®Google Scholar
• Patrick , H. 2004 . “ Re-examining classroom mastery goal structure ” . In Advances in motivation. Volume 13: Motivating students, improving schools: The legacy of Carol Midgley , Edited by: Pintrich , P. R. and Maehr , M. L. 233 – 263 . Amsterdam : Elsevier-JAI .
   Google Scholar
• Patrick , H. Teacher practices that communicate mastery goal structure: Students' views . Paper presented at the annual meeting of the American Educational Research Association . San Francisco , CA .
   Google Scholar
• Pellegrino , J. W. , Chudowsky , N. and Glaser , R. , eds. 2001 . Knowing what students know , Washington , DC : National Academy Press .
   Google Scholar
• Pintrich , P. R. 2003 . A motivational science perspective on the role of student motivation in learning contexts . Journal of Educational Psychology. , 95 : 667 – 686 .
   Web of Science ®Google Scholar
• Pintrich , P. R. and De Groot , E. V. 1990 . Motivational and self-regulated learning components of classroom academic performance . Journal of Educational Psychology , 82 : 33 – 40 .
   Web of Science ®Google Scholar
• Pintrich , P. R. and Maehr , M. L. , eds. 2002 . Advances in motivation and achievement: New directions in measures and methods (Vol. 12) , Amsterdam : JAI-Elsevier .
   Google Scholar
• Race , K. H. and Powell , K. R. 2000 . Assessing student perceptions of classroom methods and activities in the context of an outcomes-based evaluation . Evaluation Review , 24 : 635 – 646 .
   Web of Science ®Google Scholar
• Ruiz-Prima , M. A. , Schultz , S. E. , Ki , M. and Shavelson , R. 2001 . On the validity of cognitive interpretations of scores from alternative concept-mapping techniques . Educational Assessment , 7 : 99 – 141 .
   Google Scholar
• Ryan , A. M. and Patrick , H. 2001 . The classroom social environment and changes in adolescents' motivation and engagement during middle school . American Educational Research Journal. , 38 : 437 – 460 .
   Web of Science ®Google Scholar
• Schwarz , N. 1999 . Self-reports . American Psychologist , 54 : 93 – 105 .
   Web of Science ®Google Scholar
• Shrout , P. E. and Fleiss , J. L. 1979 . Intraclass correlations: Uses in assessing rater reliability . Psychological Bulletin. , 86 : 420 – 428 .
   PubMed Web of Science ®Google Scholar
• Singer , D. G. and Revenson , T. A. 1997 . A Piaget primer: How a child thinks , New York : Plume .
   Google Scholar
• Sirken , M. G. , Herrmann , D. J. , Schechter , S. , Schwarz , N. , Tanur , J. M. and Tourangeau , R. , eds. 1999 . Cognition and survey research , New York : John Wiley & Sons .
   Google Scholar
• Stipek , D. J. , Salmon , J. M. , Givvin , K. B. , Kazemi , E. , Saxe , G. and MacGyvers , V. L. 1998 . The value (and convergence) of practices suggested by motivation research and promoted by mathematics education reformers . Journal for Research in Mathematics Education. , 29 : 465 – 488 .
   Web of Science ®Google Scholar
• Sudman , S. , Bradburn , N. and Schwarz , N. 1996 . Thinking about answers: The application of cognitive processes to survey methodology , San Francisco : Jossey-Bass .
   Google Scholar
• Thelk , A. D. and Hoole , E. R. 2006 . What are you thinking? Postsecondary student think-alouds of scientific and quantitative reasoning items . The Journal of General Education , 55 : 17 – 39 .
   Google Scholar
• Third International Mathematics and Science Study (TIMSS) . 1995 . http://www.timss.bc.edu/timss1995.html Retrieved July 1, 2007, from
   Google Scholar
• Tourangeau , R. 2003 . Cognitive aspects of survey measurement and mismeasurement . International Journal of Public Opinion Research. , 15 : 3 – 7 .
   Web of Science ®Google Scholar
• Tourangeau , R. and Rasinski , K. A. 1988 . Underlying context effects in attitude measurement . Psychological Bulletin. , 103 : 299 – 314 .
   Web of Science ®Google Scholar
• Tourangeau , R. , Rips , L. J. and Rasinski , K. 2000 . The psychology of survey response , Cambridge , , UK : Cambridge University Press .
   Google Scholar
• Trochim , M. K. 2006 . Online research methods knowledge base http://www.socialresearchmethods.net/kb/measval.htm Retrieved on November 1, 2006, from
   Google Scholar
• Urdan , T. 2004 . Using multiple methods to assess students' perceptions of classroom goal structures . European Psychologist , 9 : 222 – 231 .
   Web of Science ®Google Scholar
• Urdan , T. C. , Kneisel , L. and Mason , V. 1999 . “ Interpreting messages about motivation in the classroom ” . In Advances in motivation and achievement: The role of context , Edited by: Urdan , T. C. Vol. 11 , 123 – 158 . Stamford , CT : JAI Press .
   Google Scholar
• Urdan , T. and Mestas , M. 2006 . The goals behind performance goals . Journal of Educational Psychology. , 98 : 354 – 365 .
   Web of Science ®Google Scholar
• Willis , G. B. 2005 . Cognitive interviewing: A tool for improving questionnaire design , Thousand Oaks , CA : Sage .
   Google Scholar
• Wilson , T. D. 2002 . Strangers to ourselves: Discovering the adaptive unconscious , Cambridge , MA : Harvard University Press .
   Google Scholar
• Winne , P. , Jamieson-Noel , D. and Muis , K. 2002 . “ Methodological issues and advances in researching tactics, strategies, and self-regulated learning ” . In Advances in motivation and achievement: New directions in measures and methods , Edited by: Pintrich , P. R. and Maehr , M. L. Vol. 12 , 121 – 156 . Amsterdam : JAI-Elsevier .
   Google Scholar
• Winne , P. and Perry , N. E. 2000 . “ Measuring self-regulated learning ” . In Handbook of self-regulation , Edited by: Boekaerts , M. , Pintrich , P. and Zeidner , M. 531 – 566 . San Diego , CA : Academic .
   Google Scholar
• Woolley , M. E. , Bowen , G. L. and Bowen , N. K. 2004 . Cognitive pretesting and the developmental validity of child self-report instruments: Theory and applications . Research on Social Work Practice. , 14 : 191 – 200 .
   PubMed Web of Science ®Google Scholar
• Woolley , M. E. , Bowen , G. L. and Bowen , N. K. 2006 . The development and evaluation of procedures to assess child self-report item validity . Educational and Psychological Measurement. , 66 : 687 – 700 .
   PubMed Web of Science ®Google Scholar
• Yu , S. L. and Wolters , C. A. 2002 . “ Issues in the assessment of motivation in students from ethnic minority populations ” . In Advances in motivation and achievement: New directions in measures and methods , Edited by: Pintrich , P. R. and Maehr , M. L. Vol. 12 , 349 – 380 . Amsterdam : JAI-Elsevier .
   Google Scholar