Effective Dimensionality: A Tutorial

Abstract

The topic of this tutorial is the effective dimensionality (ED) of a dataset, that is, the equivalent number of orthogonal dimensions that would produce the same overall pattern of covariation. The ED quantifies the total dimensionality of a set of variables, with no assumptions about their underlying structure. The ED of a dataset has important implications for the “curse of dimensionality”; it can be used to inform decisions about data analysis and answer meaningful empirical questions. The tutorial offers an accessible introduction to ED, distinguishes it from the related but distinct concept of intrinsic dimensionality, critically reviews various ED estimators, and gives indications for practical use with examples from personality research. An R function is provided to implement the techniques described in the tutorial.

Keywords:

• Correlation
• curse of dimensionality
• effective dimensionality
• entropy
• intrinsic dimensionality