The Treatment of Failures in Validation Research

Abstract

When validity coefficients are computed on samples that are restricted directly or indirectly on a predictor, standard correction formulas may be used to estimate the validity expected in a less restricted group, such as applicants. In some validation research, criterion distributions may also be restricted because the lowest performers leave or fail before final criterion scores are determined. When validation samples are restricted by direct selection on both predictor and criterion variables, validity coefficients may be greatly under- estimated. Standard formula corrections are not appropriate when there is direct bidimensional truncation. In circumstances in which it may reasonably be assumed that the low performers combined with the successful performers are normally distributed, criterion scores may be estimated for the presumed failures. These estimated scores allow for the dichotomous pass-fail infor- mation, which ignores individual differences between successes, to be com- bined with the continuous final criterion scores that are unavailable for failures. Formulas were developed for estimating the mean criterion score for a lower segment of the criterion distribution. Using these formulas on theoretical and observed bivariate data, estimated criterion scores were pooled with observed scores, and adjusted correlations were computed. The adjusted correlations were compared to the uncorrected correlations observed on the doubly restricted validation sample. In every case, the adjusted correlations were better estimates of the underlying true relationship than were the doubly restricted values. Finally, to obtain unrestricted population estimates, the correlations that were computed using estimated criterion scores were then corrected for range restriction on the predictor using standard formulas.