A Multilevel Factor Analysis of Third-Party Evaluations of Noncognitive Constructs Used in Admissions Decision Making

References

• Al Ansari, A., Al Khalifa, K., Al Azzawi, M., Al Amer, R., Al Sharqi, D., Al-Mansoor, A., & Munshi, F. M. (2015). Cross-cultural challenges for assessing medical professionalism among clerkship physicians in a Middle Eastern country (Bahrain): Feasibility and psychometric properties of multisource feedback. Advances in Medical Education and Practice, 6, 509–515. doi:10.2147/AMEP.S86068
   PubMed Web of Science ®Google Scholar
• Asparouhov, T., & Muthén, B. (2012). Multiple group multilevel analysis (Mplus web notes: No. 16). Retrieved from http://www.statmodel.com/examples/webnotes/webnote16.pdf
   Google Scholar
• Cheung, M. W. L., & Au, K. (2005). Applications of multilevel structural equation modeling to cross-cultural research. Structural Equation Modeling, 12, 598–619. doi:10.1207/s15328007sem1204_5
   Web of Science ®Google Scholar
• Cronbach, L. J. (1976). Research on classrooms and schools: Formulation of questions, design and analysis. Stanford, CA: Stanford Evaluation Consortium, Stanford University.
   Google Scholar
• Croon, M. A., & Van Veldhoven, M. J. P. M. (2007). Predicting group-level outcome variables from variables measured at the individual level: A latent variable multilevel model. Psychological Methods, 12, 45–57. doi:10.1037/1082-989X.12.1.45
   PubMed Web of Science ®Google Scholar
• Dansereau, F., Cho, J., & Yammarino, F. J. (2006). Avoiding the fallacy of the wrong level. Group & Organization Management, 31, 536–577. doi:10.1177/1059601106291131
   Web of Science ®Google Scholar
• Funder, D. C. (1995). On the accuracy of personality judgment: A realistic approach. Psychological Review, 102, 652–670. doi:10.1037/0033-295X.102.4.652
   PubMed Web of Science ®Google Scholar
• Funder, D. C. (2012). Accurate personality judgment. Current Directions in Psychological Science, 21(3), 177–182. doi:10.1177/0963721412445309
   Web of Science ®Google Scholar
• Goldstein, H., Bonnet, G., & Rocher, T. (2007). Multilevel structural equation models for the analysis of comparative data on educational performance. Journal of Educational and Behavioral Statistics, 32, 252–286. doi:10.3102/1076998606298042
   Web of Science ®Google Scholar
• Gross, J., Lakey, B., Orehek, E., & Edinger, K. (2009). Person perception in the college classroom: Accounting for taste in students’ evaluations of teaching effectiveness. Journal of Applied Social Psychology, 37, 1609–1638. doi:10.1111/j.1559-1816.2009.00497.x
   Google Scholar
• Hopkins, D. J., & King, G. (2010). A method of automated nonparametric content analysis for social science. American Journal of Political Science, 54, 229–247. doi:10.1111/ajps.2010.54.issue-1
   Web of Science ®Google Scholar
• Horn, J. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179–185. doi:10.1007/BF02289447
   PubMed Web of Science ®Google Scholar
• Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. doi:10.1080/10705519909540118
   Web of Science ®Google Scholar
• Jöreskog, K. G. (1999). How large can a standardized coefficient be? Retrieved from http://www.ssicentral.com/lisrel/techdocs/HowLargeCanaStandardizedCoefficientbe.pdf
   Google Scholar
• Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1–73. doi:10.1111/jedm.2013.50.issue-1
   Web of Science ®Google Scholar
• Kim, S., & Kyllonen, P. C. (2006). Rasch rating scale modeling of data from the Standardized Letter of Recommendation (ETS Research Report RR-06-33). Princeton, NJ: Educational Testing Service.
   Google Scholar
• Klieger, D., Ezzo, C., Bochenek, J. L., Cline, F. A., & Li, C. (2015). The predictive validity of non-cognitive skills for graduate and professional student success: Some initial findings. In K. Marino (Ed.), Validity and fairness issues in assessing higher education students. Chicago, IL: Symposium conducted at the annual meeting of the American Educational Research Association.
   Google Scholar
• Klockars, A. J., & Yamagishi, M. (1988). The influence of labels and positions in rating scales. Journal of Educational Measurement, 25(2), 85–96.
   Web of Science ®Google Scholar
• Kyllonen, P. C. (2008). The research behind the ETS Personal Potential Index (PPI): A background paper from ETS. Princeton, NJ: Educational Testing Service. Retrieved from http://www.ets.org/Media/Products/PPI/10411_PPI_bkgrd_report_RD4.pdf
   Google Scholar
• Liu, O. L., Minsky, J., Ling, G., & Kyllonen, P. (2009). Using the standardized letters of recommendation in selection results from a multidimensional Rasch model. Educational and Psychological Measurement, 69, 475–492. doi:10.1177/0013164408322031
   Web of Science ®Google Scholar
• McCaffrey, D. F., Yuan, K., Savitsky, T. D., Lockwood, J. R., & Edelen, M. O. (2014). Uncovering multivariate structure in classroom observations in the presence of rater errors. Educational Measurement: Issues and Practice, 34, 34–46. doi:10.1111/emip.12061
   Web of Science ®Google Scholar
• Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5–11. doi:10.3102/0013189X018002005
   Google Scholar
• Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performance as scientific inquiry into scoring meaning. American Psychologist, 9, 741–749. doi:10.1037/0003-066X.50.9.741
   Web of Science ®Google Scholar
• Muthén, B. O. (1991). Multilevel factor analysis of class and student achievement components. Journal of Educational Measurement, 28, 338–354. doi:10.1111/j.1745-3984.1991.tb00363.x
   Web of Science ®Google Scholar
• Muthén, B. O. (1994). Multilevel covariance structure analysis. Sociological Methods and Research, 22, 376–398. doi:10.1177/0049124194022003006
   Web of Science ®Google Scholar
• Muthén, B. O., Du, T., S., H. C., & Spisic, D. (1997). Robust inferences using weighted least squares and quadraic estimating equations in latent variable modeling with categorical and continuous outcomes ( Unpublished manuscript). Graduate School of Education and Information Studies, University of California Los Angeles, Los Angeles, CA. Retrieved from http://www.gseis.ucla.edu/faculty/muthen/articles/Article_075.pdf
   Google Scholar
• Muthén, L. K., & Muthén, B. O. (2007). Mplus users guide (5th ed.). Los Angeles, CA: Muthén & Muthén.
   Google Scholar
• Oliveri, M. E., & Ezzo, C. (2014). The role of noncognitive measures in higher education admissions. Journal of the World Universities Forum, 6(4), 55–65.
   Google Scholar
• Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
   Google Scholar
• Sanderson, P. M., & Fisher, C. (1994). Exploratory sequential data analysis: Foundations. Human-Computer Interaction, 9(3–4), 251–317.
   Web of Science ®Google Scholar
• Schweig, J. D. (2014). Multilevel factor analysis and student ratings of instructional practice. UCLA: Education 0249. Retrieved from http://escholarship.org/uc/item/9t03w2hc
   Google Scholar
• Sirotnik, K. A. (1980). Psychometric implications of the unit-of-analysis problem (with examples from the measurement of organizational climate). Journal of Educational Measurement, 17(4), 245–282. doi:10.1111/jedm.1980.17.issue-4
   Web of Science ®Google Scholar
• Stapleton, L. M., Yang, J. S., & Hancock, G. R. (2016). Construct meaning in multilevel settings. Journal of Educational and Behavioral Statistics, 41(5), 481–520. doi:10.3102/1076998616646200
   Web of Science ®Google Scholar
• Tucker, L. R., & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10. doi:10.1007/BF02291170
   Web of Science ®Google Scholar
• Van Barneveld, C. (2005). The dependability of medical students’ performance ratings as documented on in-training evaluations. Academic Medicine, 80(3), 309–312. doi:10.1097/00001888-200503000-00023
   PubMed Web of Science ®Google Scholar
• Whetzel, D. L., & McDaniel, M. A. (2009). Situational judgment tests: An overview of current research. Human Resource Management Review, 19, 188–202. doi:10.1016/j.hrmr.2009.03.007
   Web of Science ®Google Scholar
• Zumbo, B. D., & Forer, B. (2011). Testing and measurement from a multilevel view: Psychometrics and validation. In J. A. Boivard, K. F. Geisinger, & C. W. Buckendahl (Eds.), High stakes testing in education—Science and practice in K–12 settings (pp. 177–190). Washington, DC: American Psychological Association Press.
   Google Scholar