The Other Side of Method Bias: The Perils of Distinct Source Research Designs

Abstract

Common source bias has been the focus of much attention. To minimize the problem, researchers have sometimes been advised to take measurements of predictors from one observer and measurements of outcomes from another observer or to use separate occasions of measurement. We propose that these efforts to eliminate biases due to common source variance create serious problems. To demonstrate the problems of using what we term the “distinct sources” measurement design, we provide an integrative review of the literature regarding both contamination and deficiency of measures. Building on this theme, the article uses simulated data to demonstrate how using data from distinct observers or occasions of measurement can distort estimates of predictor importance at least as much as common source variance. Alternative multisource designs are advocated and examined for tractability by simulating various numbers of observations and sources in the research design.

Notes

¹The use of the term stability coefficients is derived from Cureton (1971) and refers to the portion of a measure of a trait measurement that is not associated with the particular time or occasion of measurement. Although Cureton was using the term exclusively to indicate the stability of intraindividual differences over time, we use it here to refer to both this stability of intraindividual differences and the stability of a trait's manifestation across multiple observers.

²The STATA and LISREL code used in these simulations is available from John Kammeyer-Mueller upon request.