ABSTRACT
The purpose of this paper is to elaborate and comment on some themes related to construct validity, construct choice, and construct respecification in the Campbellian validity system. It is argued that Reichardt’s [(2008, November). An alternative to the Campbellian conceptualization of validity. (Paper presentation). Evaluation 2008 Conference, Denver, CO, United States] assertion that construct validity and external validity are equivalent in the Campbellian system is not correct. The illustrated restricted kind of construct respecification given by Shadish, Cook, and Campbell [(2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin] can be interpreted as part of causal explanation. Their illustrations of general respecifications, on the other hand, should not be considered construct respecifications, but be regarded instead as tentative basic explanations of study results. Choice of research problem or hypothesis – as well as of the included constructs – should be based on practical or theoretical needs in the research field.
The concept of construct validity has been extensively discussed in methodological literature (e.g., Bagozzi, Citation2007; Campbell & Stanley, Citation1963; Cole & Maxwell, Citation2003; Cook & Campbell, Citation1979; Cronbach, Citation1982; Goodwin, Citation2008; Kruglanski & Kroy, Citation1976; MacCallum & Austin, Citation2000; Mark, Citation1986; Messick, Citation1989; Reichardt, Citation2002, Citation2006, Citation2008; Shadish et al., Citation2002). The literature includes philosophical, historical, and methodological trends in the definition and application of the concept. The concept has been used in causal and descriptive research as well as in professional work.
The present paper concerns construct validity in quantitative causal research where constructs are used for representing cause, outcome, persons (units, in general), and setting in the research problem or hypothesis. Since the Campbellian validity system is clearly the most dominant validity system for such research in education, psychology, and related disciplines, and since construct validity is a central part of the system, the account below refers to the latest revision of the system by Shadish et al. (Citation2002), which is abbreviated as SCC. The purpose of this paper is to elaborate and comment on some themes related to construct validity, construct choice, and construct respecification in SCC. More precisely, Reichardt’s (Citation2008) arguments for that construct validity is equivalent to external validity in the Campbellian system are evaluated, and – in addition – some treatments of construct choice, construct validity, and construct respecification in SCC are discussed.
The following account is divided into four sections. The first section contains a sketch of the Campbellian system, the second and third sections contain discussions related to the themes mentioned above, whereas the last section includes final remarks. The term “causal relationship” is used synonymously with “causal effect” or “effect”, and the term “sampling particular” is often replaced by “construct indicator” or “indicator”. The focus is on conceptual matters, omitting statistical and technical details.
Sketch of the Campbellian system
The 2002 system includes four validities, which are defined in SCC as follows: (a) Statistical conclusion validity concerns the truth of “two related statistical inferences that affect the covariation component of causal inferences: (1) whether the presumed cause and effect covary and (2) how strongly they covary” (p. 42). (b) Internal validity is the truth of “inferences about whether observed covariation between A and B reflects a causal relationship from A to B in the form in which the variables were manipulated or measured” (p. 53). (c) Construct validity is “the degree to which inferences are warranted from the observed persons, settings, and cause and effect operations included in a study to the constructs that these instances might represent” (p. 38). (d) External validity is defined as “the validity of inferences about whether the causal relationship holds over variation in persons, settings, treatment variables, and measurement variables” (p. 38).
The following points are of particular importance for the present paper:
The problem of construct validity is a representation problem, i.e., the validity is adequate to the extent the indicator subclass is representative of the chosen construct class. This definition of construct validity holds for all four elements – cause, outcome, persons, and setting – in a causal research problem or hypothesis as well as in conclusions about their solutions.
SCC define causal description as “identifying that a causal relationship exists between A and B”, and causal explanation as “explaining how A causes B” (SCC, p. 505). A and B stand for constructs in a given causal study (SCC, p. 94). The full explanation involves decomposing the cause into its causally effective parts, decomposing the effect into its causally affected parts, and identifying the mediating process through which the effective causal parts influence the causally affected outcome parts (SCC, p. 10).
Construct respecification of constructs will in many cases be relevant “perhaps because the planned operations were not implemented as intended or because evidence suggests that constructs other than the intended ones may better represent what the study actually did” (SCC, p. 82). Hence, this definition consists of two alternatives of construct respecification.
Is the Campbellian construct validity equivalent to external validity in SCC?
Reichardt (Citation2008) argues that the Campbellian construct validity depends on the causal relationship under study, and that construct and external validity in the system are equivalent. The arguments, if correct, would imply fundamental errors in the Campbellian system. It is therefore necessary to evaluate thoroughly his arguments.
Based on SCC’s assertion (p. 67) that “construct choice depends on the context in which the construct is to be used”, Reichardt’s central arguments in the 2008 paper are as follows (where Shadish et al. (Citation2002) are abbreviated as SCC):
In the context of SCC’s validity typology, the relevant inference is a causal inference such as an inference of internal validity. As noted above, an internal validity inference is the inference that a causal relationship exists for the sampling particulars in the given study. Therefore, to say that the sampling particulars represent a higher-order construct is to say that the same causal inference would hold if the higher-order construct were substituted in place of the sampling particulars in that inference. In turn, to say that the same causal inference holds for a higher-order construct as for the sampling particulars is to say that the causal relationship holds over variations in either persons, treatments, observations, or settings. (pp. 15–16)
But this is exactly how SCC define external validity which is “the validity of inferences about whether the cause–effect relationship holds over variation in persons, setting, treatment variables, and measurement variables” (see Table 1). Hence, construct validity is equivalent to external validity as defined by SCC.” (p. 18)
Reichardt illustrates his arguments by the following:
Consider a study to assess the effect of A (as manipulated) on B (as measured), which was conducted using a sample of sophomores who happen to be taken an introductory psychology course at a given university. In this context, what higher-order construct represents the individuals in the study? Do the individuals in the study only represent themselves, or do they represent a higher-order construct such as a broader subset of sophomores at the given university, sophomores at universities across the country, young adults, adults, humans, or some other of the myriad possibilities? The answer is that, in the given context, the sampling particulars represent a given higher-order construct if and only if the causal relationship between A and B that holds true for the sampling particulars also holds true for the higher—order construct. (p. 16)
Equivalence between the Campbellian construct and external validity means, for each level of validity, that construct validity implies external validity, and that external validity implies construct validity. Hence, equivalence requires that both implications are correct. For example, high construct validity must imply high external validity, and high external validity must imply high construct validity. The correctness of the two latter implications is evaluated below, using the Campbellian system as a frame and Reichardt’s sophomore example for illustration, where the effect of A on B is supposed to be 30.
High construct validity in connection with the sophomore students means that the observed students can be considered highly representative of the actual person construct class. Consequently, the causal effect of 30 of the treatment at the indicator level will hold for this person construct subclass as well as for the remaining (unobserved) persons in that person construct class, i.e., over this variation in persons. In line with SCC (p. 91), high construct validity implies therefore high external validity in this case. However, the Campbellian external validity encompasses also attempts to generalize to non-representative subgroups within this person construct class as well as to person groups outside this class, and the causal relationship for such groups may be quite different than 30. External validity will then be unsatisfactory. The first implication above, i.e., high construct validity implies high external validity, is therefore incorrect. As for the second implication above, i.e., high external validity implies high construct validity, suppose that the causal relationship of 30 for the study group holds also for a quite different group. Generalization of the causal relationship across these two groups will then have high external validity according to SCC, but this validity does not imply anything about construct validity within either group. Thus, also the second implication is incorrect. A similar reasoning about the two implications can be given for treatment, outcome, and setting, as well. For instance, if two treatments have the same impact on the same outcome, the respective external validity will be high, but this high external validity contains no information about construct validity in connection with the two treatments.
The preceding arguments are not limited to the examples used but holds in general. The conclusion is therefore that both implications are incorrect, and that the Campbellian construct and external validity are not equivalent. Note that only the first implication is dealt with by Reichardt, whereas equivalence requires both implications.
In sum, Reichardt’s critical arguments about the Campbellian construct validity are incorrect. The Campbellian construct and external validity are not equivalent, and the question of construct validity is not the same as the question of external validity, contrary to what is maintained by Reichardt (p. 18). Hence, his arguments do not point to any logical errors in the Campbellian system.
Construct respecification
To clarify the concept of construct respecification it will be appropriate to elaborate and analyse the following quotation from SCC’s (pp. 20–21) description of Fortin and Kirouac’s (Citation1976) randomized experiment:
The treatment was a brief educational course administered by several nurses, who gave a tour of their hospital and covered some basic facts about surgery with individuals who were to have elective abdominal or thoracic surgery 15–20 days later in a single Montreal hospital. Ten specific outcome measures were used after the surgery, such as an activities of living scale and a count of the analgesics used to control pain. Now compare this study with its likely target constructs – whether patient education (the target cause) promotes physical recovery (the target effect) among surgical patients (the target population of units) in hospitals (the target universe of settings) … . Empirical results often force researchers to change their initial understanding of what the domain under study is. Sometimes the reconceptualization leads to a more restricted inference about what has been studied. Thus, the planned causal agent in the Fortin and Kirouac (Citation1976) study – patient education – might need to be respecified as informational patient education if the information component proved to be causally related to recovery from surgery but the tour of the hospital did not. Conversely, data can sometimes lead researchers to think in terms of target constructs and categories that are more general than those with which they began a research program. Thus, the creative analyst of patient education studies might surmise that the treatment is a subclass of interventions that function by increasing “perceived control” or that recovery from surgery can be treated as a subclass of “personal coping”. Subsequent readers of the study can even add their own interpretations, perhaps claiming that perceived control is really a special case of the even more general self-efficacy construct.
Let us focus on SCC’s illustration of the two kinds of construct respecification (or “reconceptualization” as used in the quotation), i.e., one kind resulting in a more restricted construct and another kind leading to a more general construct. Observe that since the given causal research problem includes four target constructs – representing cause, outcome, persons (or units, in general), and setting – a respecification of one or more constructs leads to a new problem. Note further that since both kinds of respecification are incompletely described, some elaborations are given below. Finally, it is presupposed that the treatment consists of only the two mentioned components (tour and information).
As for the restricted kind of respecification, since only the information component has a causal effect on recovery (say 10 points), the total effect of patient education on recovery will be the same (10 points). Hence, in this case, causal description is that patient education has a 10-point effect on recovery, whereas the respective causal explanation is that this total effect of 10 points is due only to the information-component effect of 10 points.
For illustration, the pattern of the component effects might be varied in several ways. One alternative might be the opposite of the pattern in the quotation, i.e., only the tour component has an effect on recovery. Another alternative is that the two-component effects are equal. Consequently, the causal explanation in the first case is that the tour component is the only causally effective part, and that the two components are equally effective parts in the second case.
Hence, the account above shows that the construct choice for the restricted kind of respecification in the quotation is in line with the second alternative of SCC’s (p. 82) definition of respecification, where the respecified construct – informational patient education – is a function of the pattern of the two-component effects in the causal explanations. Observe also that construct validity will not be strengthened by this respecification, i.e., construct validity concerning the treatment will be the same for the original and respecified constructs. The respecification has therefore resulted in a more precise, nuanced representation of what the study did, not in a strengthened construct validity.
As for SCC’s illustrations of the second kind of construct reprecification, the proposed construct classes include the original target construct classes as subclasses. However, though the general constructs are broader than their original target constructs, the former constructs cannot be considered more valid or to better represent the study result. Both alternatives in SCC’s (p. 82) definition should therefore be counted as irrelevant for their illustrations of the second kind, and the original constructs should not be replaced by the proposed constructs. Their illustrations should not be counted as examples of construct respecification, but rather as possible basic theoretical explanations of the study results.
The preceding account shows therefore that SCC’s illustrated kind of restricted respecification can be considered in line with the second alternative in their definition of respecification (p. 82), whereas their illustrated second kind should not be counted as construct respecification, but as tentative, valuable basic explanations of the study results.
SCC’s illustration of the restricted kind concerns the treatment in the Fortin and Kirouac’s patient study, but an analogous analysis of this kind could be given for the outcome in the study. Suppose, for example, that the outcome consists of two components, namely life quality and pain reduction, and that only the former component is affected by the treatment. In this case, the planned outcome construct – recovery – may be respecified as life-quality recovery. Hence, whereas causal description concerns – as before – the total effect of the treatment on the outcome, the respecified construct will now be based on the pattern of the treatment’s impacts on the two outcome components in the causal explanation.
The illustrations above of restricted respecification for treatment and outcome are in line with SCC’s (p. 82) second alternative in their definition of construct respecification, and the illustrations are given in terms of causal explanation, which are limited to the relation between cause and outcome. Thus, analogous illustrations of restricted respecification do not apply to persons and setting.
Suppose now, for clarification, that the planned operations for the four target constructs in the research problem stated by SCC have not been achieved as intended, i.e., in line with the first alternative in their definition of respecification (p. 82).
As for the target causal construct, since the treatment consists of the two mentioned components, i.e., information and tour, an unsuccessful implementation of the planned causal operations will correspond to an unsuccessful implementation of one component or both components. Such an unsuccessful case may lead to a respecified construct which better represents the sampling particulars than the original construct, and thus leads to better construct validity. For example, suppose the information component has been perfectly implemented while the tour component has been neglected by the investigator. In this case, the planned construct – patient education – might be replaced by a respecified construct, namely informational patient education. The latter more restrictive construct will fit the sampling particulars better than the planned construct and will therefore strengthen construct validity.
The present case is similar to SCC’s restricted kind of respecification in the quotation. However, note that the respecification in the present case is based on the resulting sampling particulars, while the SCC’s illustration of the restrictive kind depends on the pattern of the component causal results in the causal explanation.
The analysis above is related to the causal construct, but a corresponding account of construct respecification can be given for outcome, persons, and setting. In each case, an unsuccessful implementation may lead to a respecified construct, which better represents the respective sampling particulars than the planned construct, and therefore strengthens construct validity.
Final remarks
The following points contain a summary of some important themes in the two preceding sections as well as a brief account of new themes. The sequence of the points does not correspond to degree of importance.
| 1 | Reichardt’s (Citation2008) arguments that construct validity is equivalent to external validity in the Campbellian validity system are not correct. It follows that his arguments do not point at any logical errors in the system. | ||||
Reichardt’s criterion for construct choice is based on study results, since the causal relationship that holds true for sampling particulars also holds true – is preserved – at the construct level. This criterion – appropriately termed the preservation criterion – implies that the same indicator across studies will lead to a large variation of constructs, due to a variation of causal results. Such a wide construct variation cannot be accepted, and this preservation criterion should therefore be considered of questionable utility in causal research.
| 2 | The study’s research problem or hypothesis and their constructs of cause, outcome, persons, and setting should be chosen in line with practical or theoretical needs in the research field. Hence, the four constructs depend on such needs, and they are here termed relevant constructs. | ||||
The selection of the construct is therefore limited by such needs in the research field, which implies that some freedom in construct choice is lost. This dependency of construct choice on needs may lead to the following difficulty: Suppose that an indicator in a study has been selected from another construct than the planned/relevant one, and that a construct respecification is actual. However, if the other construct is clearly less relevant than the planned one, respecification leads to good construct validity and poor construct relevance. Hence, we have a kind of conflict between validity and relevance, i.e., whether not to respecify, which means poor validity and good relevance, or to respecify, which implies good validity and poor relevance. How to handle such a case is unclear and is not discussed by SCC. One possible solution of this conflict might, of course, be to repeat the study, but now with satisfactory implementation, which would make construct respecification unnecessary, and would therefore lead to adequate construct validity and adequate construct relevance. Finally, note that since the conflict concerns increase or reduction of construct validity, the first alternative of the SCC’s (p. 82) definition of construct respecification is presupposed here.
| 3 | The study’s construct choice cannot be directly influenced by the study’s causal results in the Campbellian system, but the results can have an indirect effect on the choice. This case is illustrated by SCC’s (p. 82) account of Feldman’s (Citation1968) cooperation study meant by them as an example of construct respecification. The research problem in the study was whether compatriots receive greater cooperation than foreigners for various experimental manipulations. Five measures of the cooperation construct were used. The data analysis showed that the causal relationship for two measures was quite different than for the three others, and as concluded by SCC: “This forced Feldman to specify two kinds of cooperation (low-cost favours versus forgoing one’s own financial advantage)”. | ||||
Note that the two different relationships tell us only that two constructs are needed, not which constructs are involved, which will require additional data. This difference in causal effects will therefore give us some indirect information about the constructs. Finally, since the construct respecification, in this case, is to divide the original cooperation construct into two subconstructs, the second alternative of SCC’s (p. 82) definition is relevant here.
Disclosure statement
No potential conflict of interest was reported by the author(s).
References
- Bagozzi, R. P. (2007). On the meaning of formative measurement and how it differs from reflective measurement. Comment on Howell, Breivik, and Wilcox (2007). Psychological Methods, 12(2), 229–237. https://doi.org/10.1037/1082-989X.12.2.229
- Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Rand-McNally.
- Cole, D. A., & Maxwell, S. E. (2003). Testing mediational models with longitudinal data: Questions and tips in the use of structural equation modeling. Journal of Abnormal Psychology, 112(4), 558–577. https://doi.org/10.1037/0021-843X.112.4.558
- Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Rand-McNally.
- Cronbach, L. J. (1982). Designing evaluations of educational and social programs. Jossey-Bass.
- Feldman, R. (1968). Response to compatriot and foreigner who seek assistance. Journal of Personality and Social Psychology, 10(3), 202–214. https://doi.org/10.1037/h0026567
- Fortin, F., & Kirouac, S. (1976). A randomized controlled trial of preoperative patient education. International Journal of Nursing Studies, 13(1), 11–24. https://doi.org/10.1016/0020-7489(76)90025-0
- Goodwin, G. J. (2008). Research in psychology: Methods and design. Wiley.
- Kruglanski, A. W., & Kroy, M. (1976). Outcome validity in experimental research: A reconceptualization. Journal of Representative Research in Social Psychology, 7, 168–178.
- MacCallum, R. C., & Austin, J. T. (2000). Applications of structural equation modeling in psychological research. Annual Review of Psychology, 51(1), 201–226. https://doi.org/10.1146/annurev.psych.51.1.201
- Mark, M. M. (1986). Validity typologies and the logic and practice of quasi-experimentation. In W. M. K. Trochim (Ed.), Advances in quasi-experimental design and analysis (pp. 47–66). Jossey-Bass.
- Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed, pp. 13–103). Macmillan.
- Reichardt, C. S. (2002). Review of: Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA.: Houghton Mifflin. Social Service Review, 76(3), 510–514. https://doi.org/10.1086/345281
- Reichardt, C. S. (2006). The principle of parallelism in the design of studies to estimate treatment effects. Psychological Methods, 11(1), 1–18. https://doi.org/10.1037/1082-989X.11.1.1
- Reichardt, C. S. (2008, November). An alternative to the Campbellian conceptualization of validity. (Paper presentation). Evaluation 2008 Conference, Denver, CO, United States.
- Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.