Planned Missing Data Designs for Research in Cognitive Development

Abstract

Data collection can be the most time- and cost-intensive part of developmental research. This article describes some long-proposed but little-used research designs that have the potential to maximize data quality (reliability and validity) while minimizing research cost. In planned missing data designs, missing data are used strategically to improve the validity of data collection in one of two ways. Multiform designs allow one to increase the number of measures assessed on each participant without increasing each participant's burden. Two-method measurement designs allow one to reap the benefits of a cost-intensive gold-standard measure, using a larger sample size made possible by a rougher, cheaper measure. We explain each method using examples relevant to cognitive development research. With the use of analysis methods that produce unbiased results, planned missing data designs are an efficient way to manage cost, improve data quality, and reduce participant fatigue and practice effects.

ACKNOWLEDGMENTS

This work was supported by a Banting postdoctoral fellowship from the Social Sciences and Humanities Research Council of Canada to M. Rhemtulla and a National Science Foundation grant (NSF0066969; T. D. Little & W. Wu, co-principal investigators).

We thank members of the University of Kansas Center for Research Methods and Data Analysis (Todd Little, Director) for feedback on a previous draft of this article.

Notes

¹Researchers are encouraged to examine and report their unplanned missingness, whether or not a planned missing design is used. Many software packages contain tools to help visualize and quantify the amount of missing data. With planned missing designs, it may be helpful to use a single-imputation technique such as EM (“expectation-maximization”) imputation to create a complete data set, and then re-delete unplanned missing values before carrying out analyses to characterize the remaining missingness (e.g., tests of MCAR missingness, sensitivity analyses for MNAR missingness; see Enders, 2010). A single EM imputation should not be used to carry out the substantive analyses of interest.

²If the full sample has a complex structure, including stratified sampling characteristics, nesting, and/or subgroups with small N, it will be important to ensure that the subsample administered the expensive measure is broad enough to capture this diversity. It may be necessary to choose a larger random sample for the expensive measure and/or assign participants with particular characteristics (e.g., minorities) to receive both measures. If the latter approach is taken, the variables used to select participants must be included in the missing data estimation procedure.

³We assume that the degree of bias in each of these measures is roughly known from prior literature, including explicit validation attempts.

⁴Each measure should be composed of at least two variables (as shown in Figure 2). We used indicators that represent the two halves of each scale for simplicity, but such parceling is not necessary. For example, each scale item could be included as a separate indicator of attention. For more information about parceling options, see Little, Rhemtulla, Gibson, & Schoemann (in press).