Rorschach Score Validation as a Model for 21st-Century Personality Assessment

Abstract

Recent conceptual and methodological innovations have led to new strategies for documenting the construct validity of test scores, including performance-based test scores. These strategies have the potential to generate more definitive evidence regarding the validity of scores derived from the Rorschach Inkblot Method (RIM) and help resolve some long-standing controversies regarding the clinical utility of the Rorschach. After discussing the unique challenges in studying the Rorschach and why research in this area is important given current trends in scientific and applied psychology, I offer 3 overarching principles to maximize the construct validity of RIM scores, arguing that (a) the method that provides RIM validation measures plays a key role in generating outcome predictions; (b) RIM variables should be linked with findings from neighboring subfields; and (c) rigorous RIM score validation includes both process-focused and outcome-focused assessments. I describe a 4-step strategy for optimal RIM score derivation (formulating hypotheses, delineating process links, generating outcome predictions, and establishing limiting conditions); and a 4-component template for RIM score validation (establishing basic psychometrics, documenting outcome-focused validity, assessing process-focused validity, and integrating outcome- and process-focused validity data). The proposed framework not only has the potential to enhance the validity and utility of the RIM, but might ultimately enable the RIM to become a model of test score validation for 21st-century personality assessment.

Notes

Despite the decades-long controversy surrounding the RIM, the measure continues to be discussed in graduate-level psychological assessment courses (Childs & Eyde, 2002), and practitioners report that they use the RIM regularly when conducting clinical and forensic assessments (Camara, Nathan, & Puente, 2000; Hilsenroth & Stricker, 2004).

In addition to documenting test score validity and reliability, RIM researchers have emphasized the importance of obtaining adequate norms—including culture-specific norms—for use in RIM research and practice. Much of this discussion has focused on the need for obtaining normative RIM data from large, representative clinical and community samples whose key demographic features are well-documented (see Shaffer, Erdberg, & Meyer, 2007). Given the potential utility of the RIM in basic personality research, similar arguments can be made for obtaining adequate normative data from elementary school, high school, and college students.

Although cognitive, social, and clinical psychologists continue to distinguish implicit from explicit processes, in recent years evidence has accumulated confirming that the implicit–explicit distinction is not sharp, but a matter of degree (see Bargh & Morsella, 2008). Some psychological processes occur largely outside awareness, others occur more deliberately and consciously, and many processes fall somewhere in the middle (and are even capable of shifting across the implicit–explicit continuum over time; see Erdelyi, 2004). With respect to the RIM, Meyer (1996a) suggested that the implicit–explicit distinction can be usefully conceptualized in terms of the degree of conscious penetration associated with any given psychological process (and associated construct).

The only noteworthy procedural difference between retest reliability assessment for individual and protocol-level RIM scores occurs when reliability is inadequate. When this occurs for a complex RIM score, the retest reliabilities of that score's components should be scrutinized to ascertain which component(s) undermined the retest reliability of the ratio or derivation.

The most recent edition of the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council of Research on Education, 1999) describes validity as “the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests” (p. 9). The 1999 Standards still enumerates distinct types of validity evidence, but argues that distinctions among various types of validity evidence are less sharp than earlier frameworks had suggested, and that multiple forms of converging evidence should be used to establish the validity of test scores.

Approaching this issue from a different perspective, Meyer (1996a) discussed the importance of optimizing the accuracy of outcome judgments for criterion variables in RIM construct validity studies. He suggested two complementary strategies: (a) using a variant of Longitudinal Expert Evaluation of All Data (LEAD) procedures when judgment-dependent outcome criteria are used in RIM investigations; and (b) aggregating judgments across diverse raters to moderate bias and error variance in judgments (see Meyer, 1996a, pp. 611–614 for a detailed discussion of these procedures).

In addition to employing a PF approach, these ROD–IDI comparison studies utilized a process dissociation strategy wherein naturally occurring influences on test scores (e.g., mood state) are deliberately manipulated to illuminate underlying response processes (Bornstein, 2002, 2009). This requires understanding as fully as possible the phenomenology of self-report and performance-based tests. As Bornstein (2009) noted, at least three psychological processes occur as people respond to self-report test items. First, testees engage in introspection, turning their attention inward to determine if the statement captures some aspect of their feelings, thoughts, motives, or behaviors. Second, a retrospective memory search occurs, as testees attempt to retrieve instances wherein they experienced or exhibited the responses described in the test item. Finally, testees might engage in deliberate self-presentation, deciding whether, given the context and setting in which they are being evaluated, it is better to answer honestly or modify their response to depict themselves in a particular way. A very different set of psychological processes occur as people complete the RIM; here the fundamental challenge is to create meaning in a stimulus that can be interpreted in multiple ways. To do this, testees must direct their attention outward (rather than inward), and focus on the stimulus (not the self); they then attribute meaning to the stimulus based on properties of the inkblot, the associations primed by these stimulus properties, and their particular way of organizing information.