Cognitive Processing of Self-Report Items in Educational Research: Do They Think What We Mean?

Abstract

Techniques emerging from the considerable research on cognitive aspects of survey methodology include various forms of probing and cognitive interviewing. These techniques are used to examine whether respondents' interpretations of self-report items are consistent with researchers' assumptions and intended meanings given the constructs the items are designed to measure. However, although informal procedures are common, such developments have not been systematically applied in educational research. We describe how information derived from the systematic application of cognitive pretesting can contribute to determining the validity—designated cognitive validity—of self-report items. Examples are presented from prominent motivation-related instruments that assess real-world instructional practices, mastery classroom goal structure, and student self-efficacy. The implications and pragmatics of adopting this approach are discussed.

ACKNOWLEDGMENTS

This article is based on work supported by the National Science Foundation under Grant No. 0335369. Any opinions, findings, and conclusions or recommendations expressed in this material are ours and do not necessarily reflect the views of the National Science Foundation.

Appreciation is extended to Richard Newman for his comments on an earlier draft of the article, to AnneMarie Conley for her contributions to the research reported here, and to Pam Macinnis-Weir for logistical support. Details and examples regarding the implementation of procedures suggested here are available upon request.

Notes

¹Response options may include multiple-choice alternatives (e.g., a list of who helps students the most with their homework) but are frequently Likert scales or 5- and 7-point continua anchored with such phrases as not at all true of me and very true of me (and sometimes at the midpoint). Conceptually, the information-processing approach would apply irrespective of the response options, although types of options may place different burdens on the cognitive processes involved. With continua, when studying younger populations, instructions and examples are usually employed to ensure that the options are understood. Pretesting procedures can assess the success of those procedures as well.