A Cross-Classified CFA-MTMM Model for Structurally Different and Nonindependent Interchangeable Methods

References

• Asparouhov, T., & Muthén, B. (2012). General random effect latent variable modeling: Random subjects, items, contexts, and parameters. Retrieved from http://www.statmodel.com/download/NCMErevision2.pdf
   Google Scholar
• Bauer, D., Howard, A. L., Baldasaro, R. E., Curran, P. J., Hussong, A. M., Chassin, L., & Zucker, R. A. (2013). A trifactor model for integrating ratings across multiple informants. Psychological Methods, 18(4), 475–493. doi: 10.1037/a0032475
   PubMed Web of Science ®Google Scholar
• Beretvas, S. N. (2011). Cross-classified and multiple membership models. In J. Hox & R. J. Kyle (Eds.), Handbook of advanced multilevel analysis (pp. 313–334). New York, NY: Routledge Taylor & Francis Group.
   Google Scholar
• Bull, H. D., Schultze, M., Geiser, C., & Scheithauer, H. (2013). The role of cognitive empathy in the association between social intelligence and relational aggression: A MTMM analysis of informant agreement in the assessment of empathy deficits in relationally aggressive adolescents. (Manuscript submitted for publication.)
   Google Scholar
• Bull, H. D., Schultze, M., & Scheithauer, H. (2009). School-based prevention of bullying and relational aggression: The fairplayer.manual. European Journal of Developmental Science, 3(3), 312–317. doi: 10.1007/s11121-009-0128-y
   Google Scholar
• Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. doi: 10.1037/h0046016
   PubMed Web of Science ®Google Scholar
• Carretero-Dios, H., Eid, M., & Ruch, W. (2011). Analyzing multitrait-mulitmethod data with multilevel confirmatory factor analysis: An application to the validation of the state-trait cheerfulness inventory. Journal of Research in Personality, 45, 153–164. doi:10.1016/j.jrp.2010.12.007
   Web of Science ®Google Scholar
• Castro-Schilo, L., Widaman, K. F., & Grimm, K. J. (2013). Neglect the structure of multitrait-multimethod data at your peril: Implications for associations with external variables. Structural Equation Modeling: A Multidisciplinary Journal, 20(2), 181–207. doi:10.1080/10705511.2013.769385
   Web of Science ®Google Scholar
• Cho, S.-J., Partchev, I., & De Boeck, P. (2012). Parameter estimation of multiple item response profile model. British Journal of Mathematical and Statistical Psychology, 65, 438–466. doi:10.1111/j.2044-8317.2011.02036.x
   PubMed Web of Science ®Google Scholar
• Cho, S.-J., & Rabe-Hesketh, S. (2011). Alternating imputation posterior estimation of models with crossed random effects. Computational Statistics & Data Analysis, 55, 12–25. doi:10.1016/j.csda.2010.04.015
   Web of Science ®Google Scholar
• Conway, J. M., & Huffcutt, A. I. (1997). Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331–360. doi:10.1207/s15327043hup1004\_2
   Web of Science ®Google Scholar
• Curtis, S. M. (2012). mcmcplots: Create plots from MCMC output [Computer software manual]. Retrieved from http://CRAN.R-project.org/package=mcmcplots
   Google Scholar
• Danay, E., & Ziegler, M. (2011). Is there really a single factor of personality? A multirater approach to the apex of personality. Journal of Research in Personality, 45, 560–567. doi:10.1016/j.jrp.2011.07.003
   Web of Science ®Google Scholar
• Dumenci, L. (2000). Multitrait-multimethod analysis. In S. D. Brown & H. E. A. Tinsley (Eds.), Handbook of applied multivariate statistics and mathematerial modeling (pp. 583–611). San Diego, CA: Academic Press.
   Google Scholar
• Eid, M. (2000). A multitrait-multimethod model with minimal assumptions. Psychometrika, 65, 241–261. doi:10.1007/BF02294377
   Web of Science ®Google Scholar
• Eid, M., & Diener, E. (2006). Introduction: The need for multimethod measurement in psychology. In M. Eid & E. Diener (Eds.), Handbook of psychological measurement: A multimethod perspective (pp. 3–8). Washington, DC: American Psychological Association.
   Google Scholar
• Eid, M., Lischetzke, T., & Nussbeck, F. (2006). Structural equation models for multitrait-multimethod data. In M. Eid & E. Diener (Eds.), Handbook of multimethod measurement in psychology (pp. 289–299). Washington, DC: American Psychological Association.
   Google Scholar
• Eid, M., Lischetzke, T., Nussbeck, F. W., & Trierweiler, L. I. (2003). Separating trait effects from trait-specific method effects in multitrait-multimethod models: A multiple-indicator CT-C(M-1) model. Psychological Methods, 8, 38–60. doi:10.1037/1082-989X.8.1.38
   PubMed Web of Science ®Google Scholar
• Eid, M., Nussbeck, F. W., Geiser, C., Cole, D. A., Gollwitzer, M., & Lischetzke, T. (2008). Structural equation modeling of multitrait-multimethod data: Different models for different types of methods. Psychological Methods, 13, 230–253. doi:10.1037/a0013219
   PubMed Web of Science ®Google Scholar
• Fielding, A., & Goldstein, H. (2006). Cross-classified and multiple membership structures in multilevel models: An introduction and review (Tech. Rep.). Research Report RR791. London, UK: Department for Education and Skills.
   Google Scholar
• Gaugler, T., & Akritas, M. G. (2011). Testing for interaction in two-way random and mixed effects models: The fully nonparametric approach. Biometrics, 67, 1314–1320. doi:10.1111/j.1541-0420.2011.01579.x
   PubMed Web of Science ®Google Scholar
• Geiser, C., Eid, M., West, S. G., Lischetzke, T., & Nussbeck, F. W. (2012). A comparison of method effects in two confirmatory factor models for structurally different methods. Structural Equation Modeling, 19(3), 409–436. doi:10.1080/10705511.2012.687658
   Web of Science ®Google Scholar
• Geiser, C., Eid, M., & Nussbeck, F. W. (2008). On the meaning of the latent variables in the CT-C(M-1) model: A comment on Maydeu-Olivares and Coffman (2006). Psychological Methods, 13, 49–57. doi:10.1037/1082-989X.13.1.49
   PubMed Web of Science ®Google Scholar
• Geiser, C., Koch, T., & Eid, M. (2014). Data-generating mechanisms versus constructively defined latent variables in multitrait-multimethod analysis: A comment on Castro-Schilo, Widaman, and Grimm (2013). Structural Equation Modeling: A Multidisciplinary Journal, 21(4), 509–523. doi:10.1080/10705511.2014.919816
   PubMed Web of Science ®Google Scholar
• Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2004). Bayesian data analysis (2nd ed.). New York, NY: Chapman & Hall.
   Google Scholar
• Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. New York, NY: Cambridge University Press.
   Google Scholar
• Gelman, A., & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457–472. doi:10.1214/ss/1177011136
   Google Scholar
• Ghorpade, J. (2000). Managing five paradoxes of 360-degree feedback. The Academy of Management Executive, 14, 140–150. doi:10.5465/AME.2000.2909846
   Web of Science ®Google Scholar
• Goldstein, H. (1994). Multilevel cross-classified models. Sociological Methods & Research, 22, 364–375. doi:10.1177/0049124194022003005
   Web of Science ®Google Scholar
• Goldstein, H. (2011). Cross-classified data structures. In W. A. Shewhart & S. S. Wilks (Eds.), Multilevel statistical models (pp. 243–254). Chichester, UK: John Wiley & Sons. doi:10.1002/9780470973394.ch12
   Google Scholar
• Hox, J., & Maas, C. (2006). Multilevel models for multimethod measures. In M. Eid & E. Diener (Eds.), Handbook of psychological measurement: A multimethod perspective (pp. 269–282). Washington, DC: American Psychological Association.
   Google Scholar
• Hoyt, W. T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it? Psychological Methods, 5, 64–86. doi:10.1037/1082-989X.5.1.64
   PubMed Web of Science ®Google Scholar
• Jeon, M., & Rabe-Hesketh, S. (2012). Profile-likelihood approach for estimating generalized linear mixed models with factor structures. Journal of Educational and Behavioral Statistics, 37, 518–542. doi:10.3102/1076998611417628
   Web of Science ®Google Scholar
• Jeon, M., & Rijmen, F. (2014). Recent developments in maximum likelihood estimation of MTMM models for categorical data. Frontiers in Psychology, 5(269), 1–7. doi:10.3389/fpsyg.2014.00269
   Web of Science ®Google Scholar
• Julian, M. W. (2001). The consequences of ignoring multilevel data structures in nonhierarchical covariance modeling. Structural Equation Modeling, 8(3), 325–352. doi:10.1207/S15328007SEM0803\_1
   Web of Science ®Google Scholar
• Kelava, A. (2009). Multikollinearität in nicht-linearen latenten Strukturgleichungsmodellen [Multicollinearity in non-linear structural equation models]. (Doctoral dissertation). Goethe University, Frankfurt, Germany. Retrieved from http://publikationen.ub.uni-frankfurt.de/frontdoor/index/index/docId/6267
   Google Scholar
• Kenny, D. A. (1976). An empirical application of confirmatory factor analysis to the multitrait-multimethod matrix. Journal of Experimental Social Psychology, 12(3), 247–252. doi:10.1016/0022-1031(76)90055-X
   Web of Science ®Google Scholar
• Kenny, D. A. (1995). The multitrait-multimethod matrix: Design, analysis, and conceptual issues. In P. Shrout & S. T. Fiske (Eds.), Personality research, methods, and theory: A festschrift honoring Donald W. Fiske (pp. 111–124). Hillsdale, NJ: Lawrence Erlbaum.
   Google Scholar
• Kenny, D. A., & Kashy, D. A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin 112(1), 165–172. doi: 10.1037/0033-2909.112.1.165
   Web of Science ®Google Scholar
• Koch, T. (2013). Multilevel structural equation modeling of multitrait-multimethod-multioccasion data. (Doctoral dissertation). Freie Universität Berlin, Germany. Retrieved from http://www.diss.fu-berlin.de/diss/servlets/MCRFileNodeServlet/FUDISS_derivate_000000013668/Thesis_Koch_gedreht.pdf
   Google Scholar
• Koch, T., Eid, M., & Lochner, K. (in press). Multitrait-multimethod-analysis: The psychometric foundation of CFA-MTMM models. In P. Irwing, T. Booth, & D. Hughes (Eds.), The Wiley handbook of psychometric testing. London, UK: John Wiley & Sons.
   Google Scholar
• Koch, T., Schultze, M., Eid, M., & Geiser, C. (2014). A longitudinal multileve CFA-MTMM model for interchangeable and structurally different methods. Frontiers in Psychology, 5(311), 1–9. doi:10.3389/fpsyg.2014.00311
   Web of Science ®Google Scholar
• Kunter, M., Schümer, G., Artelt, C., Baumert, J., Klieme, E., Neubrand, M., ... Weiß, M. (2003). Pisa 2000 - Dokumentation der Erhebungsinstrumente [PISA 2000: Documentation of the measures].
   Google Scholar
• Luo, W., & Kwok, O.-M. (2009). The impacts of ignoring a crossed factor in analyzing cross-classified data. Multivariate Behavioral Research, 44, 182–212. doi: 10.1080/00273170902794214
   Google Scholar
• Maas, C. J., Lensvelt-Mulders, G. J., & Hox, J. J. (2009). A multilevel multitrait-multimethod analysis. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 5(3), 72–77. doi: http://dx.doi.org/10.1027/1614-2241.5.3.72
   Web of Science ®Google Scholar
• Mahlke, J., Schultze, M., Koch, T., Eid, M., Eckert, R., & Brodbeck, F. (2015). A multilevel CFA–MTMM approach for multisource feedback instruments: Presentation and application of a new statistical model. Structural Equation Modeling: A Multidisciplinary Journal. doi:10.1080/10705511.2014.990153
   Web of Science ®Google Scholar
• Marsh, H. W., & Grayson, D. (1995). Latent variable models of multitrait-multimethod data. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (p. 177–198). Thousand Oaks, CA: Sage Publications.
   Google Scholar
• Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52, 1187–1197. doi:10.1037/0003-066X.52.11.1187
   Web of Science ®Google Scholar
• Marsh, H. W., Wen, Z., & Hau, K.-T. (2004). Structural equation models of latent interactions: Evaluation of alternative estimation strategies and indicator construction. Psychological Methods, 9, 275–300. doi:10.1037/1082-989X.9.3.275
   PubMed Web of Science ®Google Scholar
• Meiser, T., & Steinwascher, M. A. (2014). Different kinds of interchangeable methods in multitrait-multimethod analysis: A note on the multilevel CFA-MTMM model by Koch et al. (2014). Frontiers in Psychology, 5(615), 1–3. doi:10.3389/fpsyg.2014.00615
   PubMed Web of Science ®Google Scholar
• Moosbrugger, H., Schermelleh-Engel, K., & Klein, A. (1997). Methodological problems of estimating latent interaction effects. Methods of psychological research online, 2, 95–111.
   Google Scholar
• Muthén, B., & Asparouhov, T. (2012). Bayesian structural equation modeling: A more flexible representation of substantive theory. Psychological Methods, 17, 313–335. doi: http://dx.doi.org/10.1037/a0026802
   Google Scholar
• Nussbeck, F. W., Eid, M., & Lischetzke, T. (2006). Analysing multitrait-multimethod data with structural equation models for ordinal variables applying the WLSMV estimator: What sample size is needed for valid results? British Journal of Mathematical and Statistical Psychology, 59, 195–213. doi:10.1348/000711005X67490
   PubMed Web of Science ®Google Scholar
• Pham, G., Koch, T., Helmke, A., Schrader, F.-W., Helmke, T., & Eid, M. (2012). Do teachers know how their teaching is perceived by their pupils? Procedia - Social and Behavioral Sciences 46, 3368–3374. doi:10.1016/j.sbspro.2012.06.068
   Google Scholar
• Plummer, M. (2003). Jags: A program for analysis of Bayesian graphical models using gibbs sampling [Computer software manual]. Retrieved from http://citeseer.ist.psu.edu/plummer03jags.html
   Google Scholar
• Pohl, S., & Steyer, R. (2010). Modeling common traits and method effects in multitrait-multimethod analysis. Multivariate Behavioral Research 45(1), 45–72. doi: 10.1080/00273170903504729
   PubMed Web of Science ®Google Scholar
• Pohl, S., Steyer, R., & Kraus, K. (2008). Modelling method effects as individual causal effects. Journal of the Royal Statistical Society: Series A (Statistics in Society), 171(1), 41–63. doi:10.1111/j.1467-985X.2007.00517.x
   Google Scholar
• Praetorius, A.-K., Berner, V.-D., Zeinz, H., Scheunpflug, A., & Dresel, M. (2013). Judgment confidence and judgment accuracy of teachers in judging self-concepts of students. The Journal of Educational Research, 106, 64–76. doi:10.1080/00220671.2012.667010
   Web of Science ®Google Scholar
• R Core Team. (2014). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved from http://www.R-project.org/
   Google Scholar
• Rabe-Hesketh, S., Skrondal, A., & Pickles, A. (2004, October). GLLAMM manual [Computer software manual]. Working Paper 160. Berkeley, CA: U.C. Berkeley Division of Biostatistics Working Paper Series. Retrieved from http://biostats.bepress.com/ucbbiostat/paper160
   Google Scholar
• Rabe-Hesketh, S., Skrondal, A., & Pickles, A. (2005). Maximum likelihood estimation of limited and discrete dependent variable models with nested random effects. Journal of Econometrics, 128, 301–323. doi:10.1016/j.jeconom.2004.08.017
   Web of Science ®Google Scholar
• Raudenbush, S. W. (1993). A crossed random effects model for unbalanced data with applications in cross-sectional and longitudinal research. Journal of Educational and Behavioral Statistics, 18, 321–349. doi:10.3102/10769986018004321
   Web of Science ®Google Scholar
• Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models (2nd Ed.). Thousand Oaks, CA: Sage.
   Google Scholar
• Rijmen, F., Tuerlinckx, F., De Boeck, P., & Kuppens, P. (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8, 185–205. doi:10.1037/1082-989X.8.2.185
   PubMed Web of Science ®Google Scholar
• Sahai, H., & Ojeda, M. M. (2005). Two-way crossed classification with interaction. Vol. II: Unbalanced data theory, methods, applications, and data analysis. New York, NY: Springer. doi:10.1007/978-0-8176-8168-5\_4
   Google Scholar
• Schöne, C., Dickhäuser, O., Spinath, B., & Stiensmeier-Pelster, J. (2002). Skalen zur Erfassung des schulischen Selbstkonzepts (SESSKO) [Scales for the measurement of academic self-concept.]. Göttingen, Germany: Hofgrefe.
   Google Scholar
• Schultze, M. (2012). Evaluating what the crowd says. A longitudinal structural equation model for exchangeable and structurally different methods for evaluating interventions. Diploma thesis, Freie Universität, Berlin, Germany.
   Google Scholar
• Schultze, M., Koch, T., & Eid, M. (2015). The effects of nonindependent rater sets in multilevel–multitrait–multimethod models. Structural Equation Modeling: A Multidisciplinary Journal, 22, 439–448. doi:10.1080/10705511.2014.937675
   Web of Science ®Google Scholar
• Schumacker, R. E., & Marcoulides, G. A. (1998). Interaction and nonlinear effects in structural equation modeling. Mahwah, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Searle, S. R. (2006). Linear models for unbalanced data (Wiley series in probability and statistics). Hoboken, NJ: John Wiley & Sons.
   Google Scholar
• Searle, S. R., Casella, G., & McCulloch, C. E. (2009). Variance components (Wiley series in probability and statistics). (Vol. 391). Hoboken, NJ: John Wiley & Sons.
   Google Scholar
• Shrout, P. E. (1995). Measuring the degree of consensus in personality judgments. In P. E. Shrout & S. T. Fiske (Eds.), Personality research, methods and theory: A festschrift honoring Donald W. Fiske (pp. 79–92). Hillsdale, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Van Der Linde, A. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64, 583–639. doi:10.1111/1467-9868.00353
   Web of Science ®Google Scholar
• Steyer, R. (1989). Models of classical psychometric test theory as stochastic measurement models: Representation, uniqueness, meaningfulness, identifiability, and testability. Methodika, 3, 25–60.
   Google Scholar
• Steyer, R., & Eid, M. (2001). Messen und Testen [Measurement and testing.] (2nd Ed.). Heidelberg, Germany: Springer.
   Google Scholar
• Su, Y.-S., & Yajima, M. (2015). R2jags: A package for running jags from r [Computer software manual]. Retrieved from http://CRAN.R-project.org/package=R2jags (R package version 0.05-01)
   Google Scholar
• Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9(1), 1–26. doi:10.1177/014662168500900101
   Web of Science ®Google Scholar
• Yammarino, F. J. (2003). Modern data analytic techniques for multisource feedback. Organizational Research Methods, 6, 6–14. doi:10.1177/1094428102239423
   Web of Science ®Google Scholar
• Yammarino, F. J., & Atwater, L. E. (1997). Do managers see themselves as others see them? Implications of self-other rating agreement for human resources management. Organizational Dynamics, 25, 35–44. doi:10.1016/S0090-2616(97)90035-8
   Web of Science ®Google Scholar
• Ziegler, A., Dresel, M., Schober, B., & Stöger, H. (2008). Motivationstestbatterie für Schülerinnen und Schüler der Jahrgangsstufen 5-10 (MTB 5-10) [Motivational test battery for students pertaining to grade 5-10.]. Unpublished instrument. Ulm, Germany: Universität.
   Google Scholar
• Zimmermann, D. W. (1975). Probability spaces, hilbert spaces, and the axioms of test theory. Psychometrika, 40, 395–412. doi: 10.1007/BF02291765
   Web of Science ®Google Scholar