A Multilevel Factor Analysis of Third-Party Evaluations of Noncognitive Constructs Used in Admissions Decision Making

ABSTRACT

The assessment of noncognitive traits is challenging due to possible response biases, “subjectivity” and “faking.” Standardized third-party evaluations where an external evaluator rates an applicant on their strengths and weaknesses on various noncognitive traits are a promising alternative. However, accurate score-based inferences from third-party evaluations requires disentangling score variance due to raters versus applicants by utilizing a multilevel factor analysis (MFA). To date, MFA is highly underutilized in the measurement field. In this study, we apply the MFA to analyze third-party evaluations using data from the Personal Potential Index (PPI). The PPI is a third-party measure used to evaluate graduate school applicants noncognitive traits to help inform admissions decisions. We analyzed 12,693 ratings of 6,249 applicants divided into two randomly selected subgroups. We conducted multilevel exploratory factor analysis with one subgroup and tested the hypothesized structure with the other subgroup. This work illustrates the advantages and challenges of using MFA approach to support the meaningful and valid interpretation of scores from third-party evaluations.

Notes

¹ The software syntax used to estimate the models is available from the author on request. Also see Asparouhov and Muthén (2012) for further details on conducting MFA in Mplus.

² For Horn’s parallel analysis we generated 100,000 simulated random samples of 3,123 vectors of 24 independent normal random variables. For each sample, we calculated the correlation matrix and the eigenvalues of the matrix and ordered the eigenvalues by size (i.e., largest, second largest, etc.), and found the 95th percentile for each size. We compared the ordered eigenvalues for the estimated between-applicant correlation matrix to the 95th percentiles from the simulation. The number of observed eigenvalues that were larger than the simulated values suggests the number of factors to retain in the model. The first three eigenvalues for the data were 18.57, 1.75, 0.83, and the 95th percentiles of the first three eigenvalues from the simulation were 1.18, 1.15, and 1.13. Consequently, the Horn’s parallel analysis results suggest including two factors at the between level.