Getting the Right Data Up Front: A Key Challenge

ABSTRACT

Rational decisions require transforming data into useful information by appropriate analyses. Such analyses, however, can be only as good as the data upon which they are based. In this article, we urge that careful consideration be given, up front, to procuring the right data and provide some guidelines. We begin by distinguishing briefly between four different approaches to data procurement: designed experiments, census and random sampling studies, observational studies, and so-called systems development studies. We then present and illustrate some key concepts to consider in obtaining the right data and propose a disciplined step-by-step process for data gathering, illustrating this with a study to assess the reliability of a newly designed washing machine. We conclude by commenting on some practical challenges.

KEYWORDS:

• data cleaning
• data gathering
• data procurement
• data quality
• systems development studies

Notes

¹One reason for this seems to be that training in statistics has traditionally focused principally on the analysis of data, with little attention given to their effective procurement. The planning of investigations is addressed in specialized courses on topics such as the design of experiments, sampling studies, and the planning of clinical trials. These, however, are typically advanced courses to which few practitioners are exposed.

²There are some exceptions. For example, in assessing a highly unusual, but very important, event, such as the occurrence of a rare disease or an infrequent, but costly, product failure or safety concern, one typically strives to use the entire population to estimate the occurrence probability and then get detailed information on each occurrence.

³In 2010, the long-form questionnaire was replaced by the American Community Survey, which is conducted every year (http://www.census.gov/acs/www/).

⁴We thank our former colleague, Wayne Nelson, for suggesting this question.

*Retired.