Subsampling Callbacks to Improve Survey Efficiency

Abstract

The National Comorbidity Survey (NCS), a 1990–1992 nationwide face-to-face survey of the U.S. population age 15–54 regarding the prevalence of psychiatric comorbidity, required, as with most large-scale surveys, multiple callbacks to maximize response rates. As a consequence, it appeared that a substantial proportion of the data collection cost came from the many callbacks required to obtain a small proportion of interviews with difficult-to-reach respondents. Thus we were led to explore whether the efficiency of data collection could be increased by subsampling a random proportion α of the originally sampled units from the mth callback attempt forward, with case weights used to remove bias. Similar considerations led the designers of the American Community Survey (ACS) to subsample households in the face-to-face portion of a three-mode postal, telephone, and face-to-face survey. In this article we determine the expected cost of a given interview under two strategies—subsampling versus no subsampling—as a function of the probability of obtaining an interview at the kth callback, the cost of the kth callback, and the cost of an interview or cost of a refusal if obtained at the kth callback. We then determine the sample size required to maintain a constant variance of a mean estimated under the two sampling strategies. We give an efficiency ratio as the ratio of the total costs under the two sampling strategies. This ratio is minimized as a function of m (the callback on which subsampling begins) and α (the proportion of sampling units retained) under the assumed cost and probability-of-interview structure. We then show that subsampling becomes a potentially efficient strategy whenever (a) the per callback or per interview cost is increasing and (b) the probability of a successful interview attempt is decreasing. Although these conditions are routinely met in sample surveys, it appears that they must be met to a substantial degree for useful savings to occur. In the case of the NCS, we found that only trivial savings could be achieved. Although unfortunate in one sense, this finding is useful in proving that a strategy that seemed intuitively reasonable offers only very modest practical benefit. Subsampling in the ACS seems to be more cost-effective; however, our analysis suggests that subsampling should begin at the telephone rather than the face-to-face mode.

Key Words:

• Survey design