Will the ASA's Efforts to Improve Statistical Practice be Successful? Some Evidence to the Contrary

Figures & data

Table 1. Citation counts of selected articles and books critical of NHST, 1960–2017, and the percentage of empirical social and management science research employing it, 1960–2007.

		Citation counts
Years	Article/Book	1960–1969	1970–1979	1980–1989	1990–1999	2000–2009	2010–2017	1960–2017	Social Sciences	Management Sciences
1960–1969	Rozeboom (1960)	28	55	71	104	213	283	754	56%	52 %
	Bakan (1966)	27	138	215	219	192	301	1092
	Meehl (1967)	9	95	85	134	220	291	834
	Lykken (1968)	8	137	200	215	226	376	1162
	Total	72
1970–1979	Morrison and Henkel (1970)	—	121	124	167	193	176	781	72%	80%
	Greenwald (1975)	—	54	194	215	193	272	928
	Carver (1978)	—	7	208	267	334	342	1158
	Meehl (1978)	—	9	225	475	517	568	1794
	Total		616
1980–1989	Leamer (1983)	—	—	204	427	903	876	2410	84%	89%
	Oakes (1986)	—	—	18	216	332	243	809
	Berger and Sellke (1987)	—	—	38	230	328	440	1036
	Gigerenzer and Murray (1987)	—	—	14	282	254	166	716
	Gigerenzer et al. (1989)	—	—	5	244	503	430	1182
	Rosnow and Rosenthal (1989)	—	—	2	224	344	393	963
	Total			1603
1990–1999	Cohen (1990)	—	—	—	469	786	890	2145	92%	92%
	Cohen (1994)	—	—	—	543	1590	2070	4203
	Kirk (1996)	—	—	—	69	620	802	1491
	Schmidt (1996)	—	—	—	186	576	447	1209
	Harlow, Mulaik, and Steiger (1997)	—	—	—	39	374	287	700
	Wilkinson et al. (1999)	—	—	—	12	1410	1590	3012
	Total				4737
2000–2009	Nickerson (2000)	—	—	—	—	321	582	903	92%	93%
	Gigerenzer (2004)	—	—	—	—	82	446	528
	Kline (2004)	—	—	—	—	298	734	1032
	Ziliak and McCloskey (2008)	—	—	—	—	75	681	756
	Total					10,884
2010–2017	Wasserstein and Lazar (2016)	—	—	—	—	—	762	762
	Total						14,448	32,360

NOTE: The figures listed in the last two columns show the percentage of empirical work using NHST. They are from Hubbard (2016, p. 233).

Rozeboom, W. W. (1960), “The Fallacy of the Null-Hypothesis Significance Test,” Psychological Bulletin, 57, 416–428.

 PubMed Web of Science ®Google Scholar

Bakan, D. (1966), “The Test of Significance in Psychological Research,” Psychological Bulletin, 77, 423–427.

 Web of Science ®Google Scholar

Meehl, P. E. (1967), “Theory-Testing in Psychology and Physics: A Methodological Paradox,” Philosophy of Science, 34, 103–115.

 Web of Science ®Google Scholar

Lykken, D. T. (1968), “Statistical Significance in Psychological Research,” Psychological Bulletin, 70, 151–159.

 PubMed Web of Science ®Google Scholar

Morrison, D. E., and Henkel, R. E. (eds.) (1970), The Significance Test Controversy: A Reader, Chicago, IL: Aldine.

 Google Scholar

Greenwald, A. G. (1975), “Consequences of Prejudice Against the Null Hypothesis,” Psychological Bulletin, 82, 1–20.

 Web of Science ®Google Scholar

Carver, R. P. (1978), “The Case Against Statistical Significance Testing,” Harvard Educational Review, 48, 378–399.

 Web of Science ®Google Scholar

——— (1978), “Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology,” Journal of Consulting and Clinical Psychology, 46, 806–834.

 Web of Science ®Google Scholar

Leamer, E. E. (1983), “Let's Take the Con Out of Econometrics,” American Economic Review, 73, 31–43.

 Web of Science ®Google Scholar

Oakes, M. (1986), Statistical Inference: A Commentary for the Social and Behavioural Sciences, Chichester, UK: Wiley.

 Google Scholar

Berger, J. O., and Sellke, T. (1987), “Testing a Point Null Hypothesis: The Irreconcilability of p Values and Evidence,” Journal of the American Statistical Association, 82, 112–122.

 Web of Science ®Google Scholar

Gigerenzer, G., and Murray, D. J. (1987), Cognition as Intuitive Statistics, Hillsdale, NJ: Lawrence Erlbaum.

 Google Scholar

Gigerenzer, G., Swijtink, Z., Porter, T., Daston, L., Beatty, J., and Krüger, L. (1989), The Empire of Chance: How Probability Changed Science and Everyday Life, Cambridge, UK: Cambridge University Press.

 Google Scholar

Rosnow, R. L., and Rosenthal, R. (1989), “Statistical Procedures and the Justification of Knowledge in Psychological Science,” American Psychologist, 44, 1276–1284.

 Web of Science ®Google Scholar

Cohen, J. (1990), “Things I Have Learned (So Far),” American Psychologist, 45, 1304–1312.

 Web of Science ®Google Scholar

——— (1994), “The Earth is Round (p <.05),” American Psychologist, 49, 997–1003.

 Web of Science ®Google Scholar

Kirk, R. (1996), “Practical Significance: A Concept Whose Time Has Come,” Educational and Psychological Measurement, 56, 746–759.

 Web of Science ®Google Scholar

Schmidt, F. L. (1996), “Statistical Significance Testing and Cumulative Knowledge in Psychology: Implications for the Training of Researchers,” Psychological Methods, 1, 115–129.

 Web of Science ®Google Scholar

Harlow, L. L., Mulaik, S. A., and Steiger, J. H. (eds.) (1997), What If There Were No Significance Tests? Mahwah, NJ: Lawrence Erlbaum.

 Google Scholar

Nickerson, R. S. (2000), “Null Hypothesis Significance Testing: A Review of an Old and Continuing Controversy,” Psychological Methods, 5, 241–301.

 PubMed Web of Science ®Google Scholar

Gigerenzer, G. (2004), “Mindless Statistics,” Journal of Socio-Economics, 33, 587–606.

 Google Scholar

Kline, R. B. (2004), Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research, Washington, DC: American Psychological Association.

 Google Scholar

Ziliak, S. T., and McCloskey, D. N. (2008), The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives, Ann Arbor, MI: University of Michigan Press.

 Google Scholar

Wasserstein, R. L., and Lazar, N. A. (2016), “The ASA's Statement on p-Values: Context, Process, and Purpose,” The American Statistician, 70, 129–133.

 Web of Science ®Google Scholar

Hubbard, R. (2016), Corrupt Research: The Case for Reconceptualizing Empirical Management and Social Science, Thousand Oaks, CA: SAGE Publications, Inc.

 Google Scholar