A Cross-Classified CFA-MTMM Model for Structurally Different and Nonindependent Interchangeable Methods

ABSTRACT

Multirater (multimethod, multisource) studies are increasingly applied in psychology. Eid and colleagues (2008) proposed a multilevel confirmatory factor model for multitrait-multimethod (MTMM) data combining structurally different and multiple independent interchangeable methods (raters). In many studies, however, different interchangeable raters (e.g., peers, subordinates) are asked to rate different targets (students, supervisors), leading to violations of the independence assumption and to cross-classified data structures. In the present work, we extend the ML-CFA-MTMM model by Eid and colleagues (2008) to cross-classified multirater designs. The new C4 model (Cross-Classified CTC[M-1] Combination of Methods) accounts for nonindependent interchangeable raters and enables researchers to explicitly model the interaction between targets and raters as a latent variable. Using a real data application, it is shown how credibility intervals of model parameters and different variance components can be obtained using Bayesian estimation techniques.

Keywords:

• Bayesian analysis
• cross-classification
• MTMM modeling
• structurally different and interchangeable methods

Appendix A: Formal definition of the C4 model

The formal definition of the C4 model is based on the following random experiment and the following assumptions.

Step 1: The random experiment

The random experiment of MTMM measurement designs with structurally different and nonindependent interchangeable methods (raters) is given by the Cartesian crossproduct × of the following sets Ω(·): (A1) Equation (A1(A1) ) states that first a target t is randomly chosen out of a set of possible targets Ω_T. Then, a rater r is randomly selected out of a common set of possible raters Ω_R and the rating ω_ijk on item i = {1,…, i,…, I}, construct j = {1,…, j,…, J}, and method k = {1,…, k,…, K} is observed. One crucial assumption regarding the random experiment is that the selection of targets is independent from the selection of raters and vice versa.

The possible outcomes are mapped into the set of possible targets p_T: Ω → Ω_T, into the set of possible raters p_R: Ω → Ω_R, and into the set of real numbers . The variables (p_T, p_R, Y_rtijk, and Y_tijk) are random variables on the probability space; p_T is called target variable and p_R is called rater variable.

Step 2: Definition of true scores and error variables

According to this sampling experiment, it is possible to define the latent variables in the C4 model as random variables. In particular, the true scores pertaining to the structurally different methods are defined as conditional expectations of the observed variables given the target variable: (A2) For reasons of simplicity, we choose the first method (k = 1, the structurally different method) as reference method. In a similar logic, the true scores pertaining to the set of nonindependent interchangeable methods can be defined in terms of conditional expectations: (A3) According to Definition (A3(A3) ), the true scores of the set of nonindependent interchangeable methods (k = 2, nonreference method) are defined as conditional expectations of the observed variables Y_rtij2 given the target variable p_T and the rater variable p_R.

The measurement error variables are then defined as residuals with respect to their corresponding true scores: (A4) (A5) Due to this definition, the measurement error variables are necessarily uncorrelated with the true scores and have an expectation of zero.

Step 3: Decomposition of the true score of the nonindependent interchangeable methods

Next, the true scores of the interchangeable methods τ_rtij2 are further decomposed as follows: (A6) (A7) Equations (A6(A6) ) and (A7(A7) ) state that the true scores pertaining to the set of nonindependent interchangeable methods can be decomposed into a true target T_tij2 ≔ E(Y_rtij2|p_T) effect, a true rater R_rij2 ≔ E(Y_rtij2|p_R) effect, and a true target-rater-interaction Int_rtij2 ≔ E(Y_rtij2|p_T, p_R) − E(Y_rtij2|p_T) − E(Y_rtij2|p_R) variable. The true target effect is defined as the conditional expectation of the observed variables given the target variable [i.e., E(Y_rtij2|p_T)]. The true rater effect is defined as the conditional expectation of the observed variables given the rater variable [i.e., E(Y_rtij2|p_R)], and the true target-rater-interaction effect is defined as part of the true scores τ_rtij2 that is not due to the true target T_tij2 and true rater R_rij2 effect.

Step 4: Definition of common method effects

According to the previous steps, the measurement equation of the observed variables can be represented as follows: (A8) (A9) In Equation (A8(A8) ), the true score of the reference method τ_tij1 has been replaced by an indicator-specific latent trait T_tij1 variable. Common method effects can be defined on the target level by predicting the true score variables of the nonreference method (dependent variable, T_tij2) by the true score variables measured by the reference method (independent variable, T_tij1). The latent linear regression can be expressed as follows: (A10) The common method effect is then defined as a latent residual variable with regard to the true target variable as measured by the reference method (here, T_tij1): (A11) The common method variables capture the part of the true target variable as measured by the nonreference method that cannot be explained by the true target variable as measured by the reference method. Due to the definition of the common method variables as latent residuals, they are necessarily uncorrelated with the latent trait variables (T_tij1) as well as functions of T_tij1 and have an expectation (mean) of zero.

Resubstituting Equations (A10(A10) ) and (A11(A11) ) into Equation (A9(A9) ), yields (A12) (A13)

Step 5: Additional assumptions

In the last step, additional assumptions are imposed in order to identify and estimate the model parameters. The following assumptions must be imposed in order to identify and estimate the C4 model: (A14) (A15) (A16) (A17) Equation (A14(A14) ) implies that the interaction effects (Int_rtij2) are homogeneous across different items of the same trait method unit (TMU). As a consequence of this assumption, latent interaction factors (Int_rtj2) can be defined. This Assumption (A14(A14) ) is necessary for separating measurement error influences from true interaction effects. Note that the factor loading λ^Int_ij2 does not vary across clusters (i.e., no random factor loadings). The Assumptions (A15(A15) ) to (A17(A17) ) imply that the latent measurement error variables are mutually uncorrelated with each other, which is a common and widely accepted assumption in cross-sectional latent variable models.

Additional assumptions that are useful and recommended for parsimony reasons, but which are not necessary for model identification purposes, are (A18) (A19) The Assumptions (A18(A18) ) to (A19(A19) ) allow specifying unidimensional latent method factors (CM_tj2 and R_rj2) instead of indicator-specific latent method factors. In most empirical applications, method effects will be highly correlated across different indicators pertaining to the same TMU.

Finally, the measurement equation of the least restrictive variant of the C4 model with two methods (one structurally different method and one set of nonindependent interchangeable methods) is given by (A20) (A21)

Appendix B: Prior settings

We used informative priors with regard to the intercepts and factor loadings for multiple reasons. First, due to the centering it could be assumed that the intercepts will be estimated close to zero. Second, the factor loadings pertaining to the same TMU should be close to 1 because previous studies indicated that these items are rather homogeneous. Third, the trait factor loadings pertaining to the nonreference method should be positive but smaller than the loading parameters of the same TMU. This can be derived from previous applications of the CTC(M-1) model (Koch, 2013; Koch et al., 2014; Nussbeck et al., 2006), revealing low convergence between self-reports and other reports. According to our prior specification λ^T_ij2 ∼ N(.2, .2), the 95% of the values would be within the interval of -0.69 and 1.09. We used noninformative priors for the variances and covariances of the latent variables as commonly recommended in the literature.

Notes

Throughout the present work, we use the term “bias” as deviations from conditional expectations (e.g., true or trait scores). That is, the term “bias” should not be misinterpreted as a kind of “false” evaluation.