Teaching Principal Components Using Correlations

ABSTRACT

Introducing principal components (PCs) to students is difficult. First, the matrix algebra and mathematical maximization lemmas are daunting, especially for students in the social and behavioral sciences. Second, the standard motivation involving variance maximization subject to unit length constraint does not directly connect to the “variance explained” interpretation. Third, the unit length and uncorrelatedness constraints of the standard motivation do not allow re-scaling or oblique rotations, which are common in practice. Instead, we propose to motivate the subject in terms of optimizing (weighted) average proportions of variance explained in the original variables; this approach may be more intuitive, and hence easier to understand because it links directly to the familiar “R-squared” statistic. It also removes the need for unit length and uncorrelatedness constraints, provides a direct interpretation of “variance explained,” and provides a direct answer to the question of whether to use covariance-based or correlation-based PCs. Furthermore, the presentation can be made without matrix algebra or optimization proofs. Modern tools from data science, including heat maps and text mining, provide further help in the interpretation and application of PCs; examples are given. Together, these techniques may be used to revise currently used methods for teaching and learning PCs in the behavioral sciences.

KEYWORDS:

• Factor analysis
• heat map
• optimality
• rotation
• variance explained

Article information

Conflict of interest disclosures: Each author signed a form for the disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.

Ethical principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Funding: This work was not supported by external grants or funding.

Role of the Funders/Sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgments: The authors would like to thank the editor, associate editor, and reviewers for their comments on prior versions of this manuscript. The ideas and opinions expressed herein are those of the authors alone, and endorsement by the authors' institutions is not intended and should not be inferred.

Notes

1 These eigenvalues and eigenvectors appear in the classic spectral decomposition of S as S = E Λ E’, where the columns of E are the eigenvectors corresponding to the ordered eigenvalues, which are the diagonal elements of the diagonal matrix Λ. Further re-expression as S = BB’, where B = E Λ^1/2 motivates the “PC Method” for estimating loadings in FA models, and can be used to define the PCs themselves.

2 The R-squared statistic is also equal to the squared correlation: R²(Y|X₁,… ,X_k) = {r(Y, L_x)}².

3 If the criterion to maximize is chosen to be the total of the R² statistics rather than the average, then the maximized value is just the eigenvalue itself in the case of correlation-based PCA.

4 A non-singular linear transformation of the variables PC = [PC₁,… ,PC_q] is given by L = PC M, where M is an invertible (non-singular) q×q matrix.

5 If the students have seen FA already, it may also be noted here that the first PC is clearly different from the common factor. Even if two standardized variables are uncorrelated, thus unexplained by a common factor, they are somewhat strongly related to the first PC, with average R² equal to 0.5.

6 It might be noted here, for students who have had exposure to FA, that the R² statistics cannot be obtained by summing the squared loadings when the rotation is oblique.

7 At this point, a “heads up” may be given to the students that underlying dimensionality will be dealt with more formally when FA is presented. Also, students might be told that when FA is discussed, the rotated coefficients will be used differently: they will define effects of latent traits on the observed variables, rather than linear combinations of observed variables.