Special Section on Published Articles That Have Influenced Policy

This section of Statistics and Public Policy assembles five previously published articles in statistics that have influenced public policy.¹ The articles span application disciplines from welfare policy to space travel. Each article is introduced by a contemporary statistician who comments on the article's long-range impact on public policy. They are presented in chronological order.

• Palmer, G. L. (1943), “Factors in the Variability of Response in Enumerative Studies,” Journal of the American Statistical Association, 38, 143–152. http://amstat.tandfonline.com/doi/abs/10.1080/01621459.1943.10501790#.VGuY-XDTF_xI

Introduced by Sharon Lohr, Vice President, Westat ([email protected])

Gladys Palmer (1895–1967) wrote numerous highly cited books and papers on labor mobility and social statistics. She was a professor at the University of Pennsylvania from 1931 to 1967 and was a consultant to the U.S. Office of Statistical Standards from 1940 to 1962. Her 1943 paper titled “Factors in the Variability of Response in Enumerative Studies” presented results from one of the earliest systematic studies of errors in responses to surveys. In this study, respondents in 2686 Philadelphia households were asked the same set of questions on two visits, where the second visit took place on average less than 9 days after the first visit. On each visit, responses were recorded for household members on marital status, age, employment status, and education. Palmer reported that even when the same person in the household was interviewed both times, more than 10 percent of the ages reported differed by at least a year. Marital status was more consistent, with only 1.7 percent discrepant results between visit 1 and 2, but more than 20 percent of persons had different reported education levels on the two visits.

Palmer's paper influenced the development of theory of non-sampling errors in surveys (see Deming, 1950, p. 37; Hansen et al., 1953, Chapter 2; Dalenius, 1962, p. 341). She showed that even a census will have errors in response, and she emphasized the importance of assessing the quality and reliability of survey measurements. This important contribution toward the theory of total survey error established that even quantities such as age, which you would expect to be the same for interviews that took place one week apart, could exhibit variability in survey responses.

In this era of “big data,” many people think that the answers to all public policy questions can be found simply by having a powerful enough computer and mining existing data. Palmer's paper reminds us that big data sets are only as good as the individual responses they contain.

• Dalal, S. R., Fowlkes, E. B., and Hoadley, B. (1989), ``Risk Analysis of the Space Shuttle: Pre-Challenger Prediction of Failure,’’ Journal of the American Statistical Association, 84, 945–957. http://amstat.tandfonline.com/doi/abs/10.1080/01621459.1989.10478858#.VGuZwzTF_xI

Introduced by Vijay Nair, Donald A. Darling Professor of Statistics, University of Michigan ([email protected])

We are all familiar with the phrase “A picture is worth a thousand words.” This paper is a powerful illustration of that adage, although it arises unfortunately in the context of a terrible tragedy. But perhaps we should qualify the message: The right picture is worth a thousand words … and it may even save lives!

The paper by Dalal, Fowlkes, and Hoadley is a must-read for anyone who wants to get an appreciation for the power of statistics. Through a retrospective analysis of O-ring failure data from the Challenger tragedy, the authors demonstrate how someone with a basic understanding of statistical concepts could have foreseen potential danger in launching the shuttle at the temperature of 31°F on January 27, 1986. This is definitely not a case of 20/20 hindsight. The message can be gleaned by simply comparing Figure 1a vs Figure 1b: Morton Thiokol engineers had failed to include data for which there were zero “incidents” in past launches. Apparently Figure 1a had played an important role in their conclusion that the historical data did not show a temperature effect. Dalal et al. show that as many as 5 incidents would have been predicted at the launch temperature of 31°F. It's an exemplary demonstration of the key role that statistics can play and should have played in decision making in this critical case.

One can only infer that the Morton Thiokol team did not include anyone with basic statistical expertise. Unfortunately, this is increasingly the case with government contractors as well as government organizations. A group of us have bemoaned the lack of statistical expertise within the Department of Defense as part our service on panels of the National Research Council. The paper by Dalal et al. is one illustration, albeit a very powerful one, of the dangers of relying on advice from nonexperts in making extremely critical decisions.

• Hansen, M. H. and Tepping, B. J. (1990), ``Regression Estimates in Federal Welfare Quality Control Programs,’’ Journal of the American Statistical Association, 85, 856–864. http://amstat.tandfonline.com/doi/abs/10.1080/01621459.1990.10474951#.VGubHTTF_xI

Introduced by Cynthia Z.F. Clark, former Administrator of the USDA's National Agricultural Statistics Service ([email protected])

The 1990 JASA paper by Morris Hansen and Benjamin Tepping summarizes an extensive review by the authors. The review was commissioned by the National Research Council charged to examine whether the federal quality control procedure being used for the Food Stamp (now SNAP) and Aid to Families with Dependent Children programs was statistically valid. The review involved examining both the state and federal quality control procedures (being used for different policy purposes) and their relationships as well as the application of regression estimates to the federal quality control procedures. The application of the regression estimator had been widely challenged.

The main goal of the state quality control program was to guide administrative improvements and program evaluation while the federal quality control program provided estimates of overpayment and underpayment error rates across states. The authors describe the use of a regression estimator to determine these error rates - the procedure used by the Food and Nutrition Service in USDA and the Office of Family Assistance in HHS to estimate these rates. The authors state that the challenges to the regression estimator do not recognize the difference between classical regression analysis and the application of a regression estimator to data from a sample survey as was used to collect the data for the measures.

The error rates have been used over time as the programs have evolved even with accompanying legislative updates and changes. The authors demonstrate that these quality control procedures are based on a solid statistical foundation. As John Neter points out in his comments “the main issues concerning these QC programs are not, in my view, technical statistical issues, but rather involve broader consideration.” The research is evidence of the impact of statistical procedures on program decisions and the allocation of program funds to meet societal needs.

• Jeffrey Grogger and Greg Ridgeway (2006), ``Testing for Racial Profiling in Traffic Stops from Behind a Veil of Darkness,’’ Journal of the American Statistical Association, 101, 878–887. http://amstat.tandfonline.com/doi/abs/10.1198/016214506000000168#.VGublDTF_xI

Introduced by James Lynch, Professor and Chair of the Department of Criminology and Criminal Justice, University of Maryland ([email protected])

The rate of common law crime and particularly violent crime has declined massively in the last 20 years. With this good news, the attention of policy makers in the justice arena has turned to issues of equity and whether the burden of some of the coercive policies that many believe contributed to the crime decline are being borne disproportionately by racial minorities and the poor. Chief among these coercive policies are aggressive patrol practices such as stop and frisk and traffic stops. Advocates of these practices tout their crime reduction effects, while opponents lament the biased manner in which these policies are carried out and the long term negative effects on the legitimacy of the police. In many cases, these debates are conducted with great emotion and conviction but little useful data. More recently, with increased interested in evidence based policy in the Department of Justice, more data are being collected on these issues and more thoughtful empirical analysis is being done. The Police-Public Contact Supplement to the National Crime Victimization Survey (NCVS) is an instance of the former and this excellent article by Jeff Grogger and Greg Ridgeway is an example of the latter.

Addressing the issue of bias in decision making within the justice system is exceedingly complex, since it often requires inferring motive from a relationship between race (or some other status) and coercion while holding all else constant. It is, as Grogger and Ridgeway note, the holding all else constant that is so difficult. This paper makes a major contribution to the debates over racial bias in traffic stops by employing a very clever strategy for establishing the counter factual. The authors are very careful and explicit in enumerating and testing their assumptions and they also test the sensitivity of their analysis to known weakness in the data. This kind of painstaking work not only provides high quality evidence on a specific issue, but it also sends the meta-message that science and not emotion should drive policy.

This analysis provides an additional service to policy making by laying bare the model by which policies have both positive and negative effects. Specifically, this paper tests a model of bias that assumes that profiling is the product of the individual decision-making of the police officer. The analysis finds no support for this assertion. This raises the question of whether the patterns of stops by race are the product of organizational policies and practices that amount to institutional racism and not individual bias. The authors appear to have the data to test this alternative explanation and thereby move both the science and the debate along.

• Ash, A. and Lamperti, J. (2008), “Florida 2006: Can Statistics Tell Us Who Won Congressional District-13?’’ CHANCE, 21, 18–24. http://amstat.tandfonline.com/doi/abs/10.1080/09332480.2008.10722895#.VGucLTTF_xI

Introduced by Linda J. Young, Chief Mathematical Statistician and Director of Research and Development, USDA's National Agricultural Statistics Service ([email protected])

If voters do not believe that election results reflect the intent of the voters, emotions, which may already be high in tightly contested races, can become even more volatile. As a swing state with a large number of electoral votes, the spotlight is often on Florida's elections. Most will recall the “hanging chads” of the 2000 Presidential election that were debated in the U.S. Supreme Court. But, in that same election, Palm Beach County had a much higher than expected number of votes for the third party candidate, and statistical analyses led to questions of whether the “butterfly ballot” used there could have also impacted the election's outcome. In the paper by Ash and Lamperti, undervotes potentially arising from another poorly designed ballot are considered. Could the design have altered the outcome of the 2006 election for District 13's representative to Congress?

Ash and Lamperti raise the interesting question of whether statistical analyses should be used in court with the goal of forcing a new election or, perhaps even more radically, of overturning an election's outcome. Statistical analyses such as theirs contributed to public awareness followed by demands for change. Consequently, Florida has revised its election processes. Paper trails are now required, and election audits are conducted. One can easily argue that more needs to be done to ensure the integrity of Florida's election results. Thoughtful articles, such as this one, provide a foundation for such changes.

Notes

The articles in this issue are limited to selections from journals published by Taylor & Francis.