102
Views
106
CrossRef citations to date
0
Altmetric
Original Articles

The Use of Statistical Significance Tests in Research

Bootstrap and Other Alternatives

Pages 361-377 | Published online: 15 Apr 2014

Keep up to date with the latest research on this topic with citation updates for this article.

Read on this site (30)

Daniel A. Sass & Michael A. Sanchez. (2023) A Variable Selection Algorithm for Creating Replicable Factor Structures. Multivariate Behavioral Research 58:1, pages 48-70.
Read now
Adam E. Barry, Danny Valdez, Patricia Goodson, Leigh Szucs & Jovanni V. Reyes. (2019) Moving college health research forward: Reconsidering our reliance on statistical significance testing. Journal of American College Health 67:3, pages 181-188.
Read now
Juhong Christie Liu, Kristen St. John & Anna M. Bishop Courtier. (2017) Development and Validation of an Assessment Instrument for Course Experience in a General Education Integrated Science Course. Journal of Geoscience Education 65:4, pages 435-454.
Read now
Corey Peltier. (2017) “What If” analysis: Benefits of utilizing a “What If” analysis in excel. Communications in Statistics - Theory and Methods 46:12, pages 6119-6129.
Read now
Wenhua Lu, Jingang Miao & E. Lisako J. McKyer. (2014) A Primer on Bootstrap Factor Analysis as Applied to Health Studies Research. American Journal of Health Education 45:4, pages 199-204.
Read now
RobinK. Henson, Prathiba Natesan & ErikaD. Axelson. (2014) Comparisons of Improvement-Over-Chance Effect Sizes for Two Groups Under Variance Heterogeneity and Prior Probabilities. The Journal of Experimental Education 82:2, pages 205-228.
Read now
Linda Reichwein Zientek, Z. Ebrar Yetkiner & Bruce Thompson. (2010) Characterizing the Mathematics Anxiety Literature Using Confidence Intervals as a Literature Review Mechanism. The Journal of Educational Research 103:6, pages 424-438.
Read now
Haiyan Bai & Wei Pan. (2008) Resampling methods revisited: advancing the understanding and applications in educational research. International Journal of Research & Method in Education 31:1, pages 45-62.
Read now
Maria Elizabeth Grabe & DanG. Drew. (2007) Crime Cultivation: Comparisons Across Media Genres and Channels. Journal of Broadcasting & Electronic Media 51:1, pages 147-171.
Read now
EricR. Buhi. (2005) The Insignificance of “Significance” Tests: Three Recommendations for Health Education Researchers. American Journal of Health Education 36:2, pages 109-112.
Read now
Jianmin Guan, Ping Xiang & Xiaofen Deng Keating. (2004) Evaluating the Replicability of Sample Results: A Tutorial of Double Cross-Validation Methods. Measurement in Physical Education and Exercise Science 8:4, pages 227-241.
Read now
NancyL. Leech & AnthonyJ. Onwuegbuzie. (2004) A Proposed Fourth Measure of Significance: The Role of Economic Significance in Educational Research. Evaluation & Research in Education 18:3, pages 179-198.
Read now
Xitao Fan. (2001) Statistical Significance and Effect Size in Education Research: Two Sides of a Coin. The Journal of Educational Research 94:5, pages 275-282.
Read now
ThomasR. Knapp & ShlomoS. Sawilowsky. (2001) Constructive Criticisms of Methodological and Editorial Practices. The Journal of Experimental Education 70:1, pages 65-79.
Read now
ThomasA. Devaney. (2001) Statistical Significance, Effect Size, and Replication: What Do the Journals Say?. The Journal of Experimental Education 69:3, pages 310-320.
Read now
KevinM. Kieffer, RobertJ. Reese & Bruce Thompson. (2001) Statistical Techniques Employed in AERJ and JCP Articles from 1988 to 1997: A Methodological Review. The Journal of Experimental Education 69:3, pages 280-309.
Read now
RebeccaP. Ang. (1998) Use of the Jackknife Statistic to Evaluate Result Replicability. The Journal of General Psychology 125:3, pages 218-228.
Read now
Tammi Vacha-Haase & Johanna E. Nilsson. (1998) Statistical Significance Reporting: Current Trends and Uses in MECD. Measurement and Evaluation in Counseling and Development 31:1, pages 46-57.
Read now
Bruce Thompson & PatriciaA. Snyder. (1997) Statistical Significance Testing Practices in The Journal of Experimental Education . The Journal of Experimental Education 66:1, pages 75-83.
Read now
Xitao Fan & Lin Wang. (1996) Comparability of Jackknife and Bootstrap Results: An Investigation for a Case of Canonical Correlation Analysis. The Journal of Experimental Education 64:2, pages 173-189.
Read now
William P. Erchul, C. Gay Covington, Jan N. Hughes & Joel Meyers. (1995) Further Explorations of Request-Centered Relational Communication Within School Consultation. School Psychology Review 24:4, pages 621-632.
Read now
D. M. Gorman. (1995) On the Difference Between Statistical and Practical Significance in School-based Drug Abuse Prevention. Drugs: Education, Prevention and Policy 2:3, pages 275-283.
Read now
Bruce Thompson. (1994) The Revised Program Evaluation Standards and their Correlation with the Evaluation use Literature. The Journal of Experimental Education 63:1, pages 54-81.
Read now
William Asher. (1993) The Role of Statistics in Research. The Journal of Experimental Education 61:4, pages 388-393.
Read now
William D. Schafer. (1993) Interpreting Statistical Significance and Nonsignificance. The Journal of Experimental Education 61:4, pages 383-387.
Read now
Joel R. Levin. (1993) Statistical Significance Testing From Three Perspectives. The Journal of Experimental Education 61:4, pages 378-382.
Read now
Ronald P. Carver. (1993) The Case Against Statistical Significance Testing, Revisited. The Journal of Experimental Education 61:4, pages 287-292.
Read now

Articles from other publishers (76)

Sinem Birant, Mert Veznikli, Yelda Kasimoglu, Mine Koruyucu, Atıf Ahmet Evren & Figen Seymen. (2023) Path Analysis of the Relationships between the Eruption Time of the First Primary Teeth and Various Factors in Twins. Children 10:4, pages 683.
Crossref
Hakan ÇİTE, Sümeyra GÜRBÜZER & Menşure ALKIŞ KÜÇÜKAYDIN. (2022) The Use of Slow Motion and Digital Concept Maps in Primary School: An Evaluation in Terms of Science Attitudes and Metacognitive Awareness. Pamukkale University Journal of Education.
Crossref
Ron Tzur, Heather Lynn Johnson, Alan Davis, Nicola M. Hodkowski, Cody Jorgensen, Bingqian Wei & Anderson Norton. (2022) A stage-sensitive written measure of multiplicative double counting for grades 3-8. Studies in Educational Evaluation 74, pages 101152.
Crossref
Juan - Martell Muñoz, José Fernando Mora Romo, Alejandro Nuñez & Laura Karina Castro Saucedo. (2022) Happiness in University Students in Uaz, México. Revista iberoamericana de psicología 15:1, pages 103-112.
Crossref
Volker Gehrau, Katharina Maubach & Sam FujarskiVolker Gehrau, Katharina Maubach & Sam Fujarski. 2022. Einfache Datenauswertung mit R. Einfache Datenauswertung mit R 271 317 .
Tesfaye Yadete, Kavita Batra, Dale M. Netski, Sabrina Antonio, Michael J. Patros & Johan C. Bester. (2021) Assessing Acceptability of COVID-19 Vaccine Booster Dose among Adult Americans: A Cross-Sectional Study. Vaccines 9:12, pages 1424.
Crossref
Manoj Sharma, Kavita Batra & Ravi Batra. (2021) A Theory-Based Analysis of COVID-19 Vaccine Hesitancy among African Americans in the United States: A Recent Evidence. Healthcare 9:10, pages 1273.
Crossref
Peter Dixon & Scott Glover. (2020) Assessing evidence for replication: A likelihood-based approach. Behavior Research Methods 52:6, pages 2452-2459.
Crossref
Seth R. Koslov, Arjun Mukerji, Katlyn R. Hedgpeth & Jarrod A. Lewis-Peacock. (2019) Cognitive Flexibility Improves Memory for Delayed Intentions. eneuro 6:6, pages ENEURO.0250-19.2019.
Crossref
Marc N. Branch. (2018) The “Reproducibility Crisis:” Might the Methods Used Frequently in Behavior-Analysis Research Help?. Perspectives on Behavior Science 42:1, pages 77-89.
Crossref
Jamie N. Hershaw & Mark L. Ettenhofer. (2018) Insights into cognitive pupillometry: Evaluation of the utility of pupillary metrics for assessing cognitive load in normative and clinical samples. International Journal of Psychophysiology 134, pages 62-78.
Crossref
Min Hu, Xiangpeng Wang, Wenwen Zhang, Xueping Hu & Antao Chen. (2017) Neural interactions mediating conflict control and its training-induced plasticity. NeuroImage 163, pages 390-397.
Crossref
Bonnie M. HaeckerForrest C. LaneLinda R. Zientek. (2017) Evidence-Based Decision-Making. Journal of School Leadership 27:6, pages 860-883.
Crossref
Adam E. Barry, Leigh E. Szucs, Jovanni V. Reyes, Qian Ji, Kelly L. Wilson & Bruce Thompson. (2016) Failure to Report Effect Sizes. Health Education & Behavior 43:5, pages 518-527.
Crossref
Dana L Wolff-Hughes, James J McClain, Kevin W Dodd, David Berrigan & Richard P Troiano. (2016) Number of accelerometer monitoring days needed for stable group-level estimates of activity. Physiological Measurement 37:9, pages 1447-1455.
Crossref
Sujay Dutta & Chris Pullig. (2015) A commentary on reporting effect size and confidence intervals: Response to Palmer and Strelan (2014). Journal of Business Research 68:5, pages 1082-1085.
Crossref
Marc Branch. (2014) Malignant side effects of null-hypothesis significance testing. Theory & Psychology 24:2, pages 256-277.
Crossref
Steve Majerus & Claire Boukebza. (2013) Short-term memory for serial order supports vocabulary development: New evidence from a novel word learning paradigm. Journal of Experimental Child Psychology 116:4, pages 811-828.
Crossref
Jessica Middlemis Maher, Jonathan C. Markey & Diane Ebert-May. (2013) The Other Half of the Story: Effect Size Analysis in Quantitative Research. CBE—Life Sciences Education 12:3, pages 345-351.
Crossref
Willi Hager. (2013) The statistical theories of Fisher and of Neyman and Pearson: A methodological perspective. Theory & Psychology 23:2, pages 251-270.
Crossref
Astrid Fritz, Thomas Scherndl & Anton Kühberger. (2012) A comprehensive review of reporting practices in psychological journals: Are effect sizes really enough?. Theory & Psychology 23:1, pages 98-122.
Crossref
Johanna Leppavirta. (2011) ASSESSING UNDERGRADUATE STUDENTS’ CONCEPTUAL UNDERSTANDING AND CONFIDENCE OF ELECTROMAGNETICS. International Journal of Science and Mathematics Education 10:5, pages 1099-1117.
Crossref
Juan García García, Elena Ortega Campos & Leticia De la Fuente Sánchez. (2013) The Use of the Effect Size in JCR Spanish Journals of Psychology: From Theory to Fact. The Spanish journal of psychology 14:2, pages 1050-1055.
Crossref
Yu‐Hui Fang, Chao‐Min Chiu & Eric T.G. Wang. (2011) Understanding customers' satisfaction and repurchase intentions. Internet Research 21:4, pages 479-503.
Crossref
김민성. (2011) Quantitative Methods in Geography Education Research: Concept and Application of Effect Size. The Journal of The Korean Association of Geographic and Environmental Education 19:2, pages 205-220.
Crossref
Shuyan Sun & Wei Pan. (2011) The Philosophical Foundations of Prescriptive Statements and Statistical Inference. Educational Psychology Review 23:2, pages 207-220.
Crossref
Michael Dickson & Davis Baird. 2011. Philosophy of Statistics. Philosophy of Statistics 199 229 .
Kevin J. Warrian, Luciano L. Lorenzana, Dara Lankaranian, Jyoti Dugar, Sheryl S. Wizov & George L. Spaeth. (2010) The Assessment of Disability Related to Vision Performance-Based Measure in Diabetic Retinopathy. American Journal of Ophthalmology 149:5, pages 852-860.e1.
Crossref
Judith Harrison, Bruce Thompson & Kimberly J. Vannest. (2017) Interpreting the Evidence for Effective Interventions to Increase the Academic Performance of Students With ADHD: Relevance of the Statistical Significance Controversy. Review of Educational Research 79:2, pages 740-775.
Crossref
KEVIN J. WARRIAN, LUCIANO L. LORENZANA, DARA LANKARANIAN, JYOTI DUGAR, SHERYL S. WIZOV & GEORGE L. SPAETH. (2009) ASSESSING AGE-RELATED MACULAR DEGENERATION WITH THE ADREV PERFORMANCE-BASED MEASURE. Retina 29:1, pages 80-90.
Crossref
Matthew T James, Jianguo Zhang, Andrew W Lyon & Brenda R Hemmelgarn. (2008) Derivation and internal validation of an equation for albumin-adjusted calcium. BMC Clinical Pathology 8:1.
Crossref
Amy Fredrickson, Peter J. Snyder, Jennifer Cromer, Elizabeth Thomas, Matthew Lewis & Paul Maruff. (2008) The use of effect sizes to characterize the nature of cognitive change in psychopharmacological studies: an example with scopolamine. Human Psychopharmacology: Clinical and Experimental 23:5, pages 425-436.
Crossref
Gantt P. GallowayEdward G. SingletonM. Douglas Anglin, Richard A. Rawson, Marinelli-Casey Patricia, Balabis Joseph, Bradway Richard, Brown Alison Hamilton, Burke Cynthia, Christian Darrell, Cohen Judith, Cosmineanu Florentina, Dickow Alice, Donaldson Melissa, Frazier Yvonne, Thomas E. Freese, Gallagher Cheryl, Gantt P. Galloway, Gulati Vikas, Herrell James, Horner Kathryn, Huber Alice, Martin Y. Iguchi, Russell H. Lord, Michael J. McCann, Minsky Sam, Morrisey Pat, Obert Jeanne, Pennell Susan, Reiber Chris, Rodrigues Norman, Stalcup Janice, P.H.S. Alex Stalcup, Ewa S. Stamper, Stimson Janice, Manser Sarah Turcotte, Vandersloot Denna, Weiner Ahndrea, Woodward Kathryn & Zweben Joan. (2008) How Long Does Craving Predict Use of Methamphetamine? Assessment of Use One to Seven Weeks after the Assessment of Craving. Substance Abuse: Research and Treatment 1, pages SART.S775.
Crossref
Merideth A. Addicott, Dawn M. Marsh‐Richard, Charles W. Mathias & Donald M. Dougherty. (2007) The Biphasic Effects of Alcohol: Comparisons of Subjective and Objective Measures of Stimulation, Sedation, and Physical Activity. Alcoholism: Clinical and Experimental Research 31:11, pages 1883-1890.
Crossref
Linda Reichwein Zientek & Bruce Thompson. (2007) Applying the bootstrap to the multivariate case: Bootstrap component/factor analysis. Behavior Research Methods 39:2, pages 318-325.
Crossref
Bob Ives. (2007) Graphic Organizers Applied to Secondary Algebra Instruction for Students with Learning Disorders. Learning Disabilities Research & Practice 22:2, pages 110-118.
Crossref
Heibatollah Baghi, Siamak Noorbaloochi & Jean B. Moore. (2007) Statistical and Nonstatistical Significance. Quality Management in Health Care 16:2, pages 104-112.
Crossref
Clint D Kelly. (2006) Replicating Empirical Research In Behavioral Ecology: How And Why It Should Be Done But Rarely Ever Is. The Quarterly Review of Biology 81:3, pages 221-236.
Crossref
Martin J. BergeeJamila L. McWhirter. (2016) Selected Influences on Solo and Small-Ensemble Festival Ratings. Journal of Research in Music Education 53:2, pages 177-190.
Crossref
Todd C. Campbell. (2016) An Introduction to Clinical Significance: An Alternative Index of Intervention Effect for Group Experimental Designs. Journal of Early Intervention 27:3, pages 210-227.
Crossref
Nekane Balluerka, Juana Gómez & Dolores Hidalgo. (2005) The Controversy over Null Hypothesis Significance Testing Revisited. Methodology 1:2, pages 55-70.
Crossref
Crystal Reneé Hill & Bruce Thompson. 2005. Higher Education: Handbook of Theory and Research. Higher Education: Handbook of Theory and Research 175 196 .
Bruce Thompson. (2004) The “significance” crisis in psychology and education. The Journal of Socio-Economics 33:5, pages 607-613.
Crossref
Rubén Ledesma. (2004) AlphaCI: un programa de cálculo de intervalos de confianza para el coeficiente alfa de Cronbach. Psico-USF 9:1, pages 31-37.
Crossref
Jason E. King. (2003) Bootstrapping Confidence Intervals For Robust Measures Of Association. Journal of Modern Applied Statistical Methods 2:2, pages 512-519.
Crossref
Bob Ives. (2016) Effect Size Use in Studies of Learning Disabilities. Journal of Learning Disabilities 36:6, pages 490-504.
Crossref
R. Houtkamp, H. Spekreijse & P. R. Roelfsema. (2003) A gradual spread of attention. Perception & Psychophysics 65:7, pages 1136-1144.
Crossref
Anthony J. Onwuegbuzie & Joel R. Levin. (2003) Without Supporting Statistical Evidence, Where Would Reported Measures of Substantive Importance Lead? To No Good Effect. Journal of Modern Applied Statistical Methods 2:1, pages 133-151.
Crossref
Xitao Fan. (2016) Using Commonly Available Software For Bootstrapping In Both Substantive And Measurement Analyses. Educational and Psychological Measurement 63:1, pages 24-50.
Crossref
Jacob Kraemer Tebes, Joy S. Kaufman & Christian M. Connell. 2003. Encyclopedia of Primary Prevention and Health Promotion. Encyclopedia of Primary Prevention and Health Promotion 42 61 .
John C. Hanes. (2016) A Nonparametric Approach to Program Evaluation: Utilizing Number Needed to Treat, L’Abbé Plots, and Event Rate Curves for Outcome Analysis. American Journal of Evaluation 23:2, pages 165-182.
Crossref
Bruce Thompson. (2016) What Future Quantitative Social Science Research Could Look Like: Confidence Intervals for Effect Sizes. Educational Researcher 31:3, pages 25-32.
Crossref
Frank Baugh. (2016) Correcting Effect Sizes for Score Reliability: A Reminder that Measurement and Substantive Issues are Linked Inextricably. Educational and Psychological Measurement 62:2, pages 254-263.
Crossref
Bruce Thompson. (2011) “Statistical,” “Practical,” and “Clinical”: How Many Kinds of Significance Do Counselors Need to Consider?. Journal of Counseling & Development 80:1, pages 64-71.
Crossref
B. Thomas Gray. (2016) A Factor Analytic Study of the Substance Abuse Subtle Screening Inventory (SASSI). Educational and Psychological Measurement 61:1, pages 102-118.
Crossref
Scott R. Glover & Peter Dixon. (2001) Dynamic illusion effects in a reaching task: Evidence for separate visual representations in the planning and control of reaching.. Journal of Experimental Psychology: Human Perception and Performance 27:3, pages 560-572.
Crossref
Paola Palladino, Paola Poli, Gabriele Masi & Mara Marcheschi. (2000) The Relation Between Metacognition and Depressive Symptoms in Preadolescents With Learning Disabilities: Data in Support of Borkowski's Model. Learning Disabilities Research and Practice 15:3, pages 142-148.
Crossref
Tammi Vacha-HaaseJohanna E. NilssonDavid R. ReetzTeresa S. LanceBruce Thompson. (2016) Reporting Practices and APA Editorial Policies Regarding Statistical Significance and Effect Size. Theory & Psychology 10:3, pages 413-425.
Crossref
Rossana De Beni & Paola Palladino. (2000) Intrusion errors in working memory tasks. Learning and Individual Differences 12:2, pages 131-143.
Crossref
James R. Rodrigue, William F. Kanasky, Shannon I. Jackson & Michael G. Perri. (2000) The Psychosocial Adjustment to Illness Scale—Self Report: Factor structure and item stability.. Psychological Assessment 12:4, pages 409-413.
Crossref
Barbara B. Lockee, John K. Burton & Lawrence H. Cross. (1999) No comparison: Distance education finds a new use for ‘No significant difference’. Educational Technology Research and Development 47:3, pages 33-42.
Crossref
Bruce Thompson. (2016) Statistical Significance Tests, Effect Size Reporting and the Vain Pursuit of Pseudo-Objectivity. Theory & Psychology 9:2, pages 191-196.
Crossref
Bruce Thompson. (2016) If Statistical Significance Tests are Broken/Misused, What Practices Should Supplement or Replace Them?. Theory & Psychology 9:2, pages 165-181.
Crossref
Bruce Thompson. (2017) Improving Research Clarity and Usefulness with Effect Size Indices as Supplements to Statistical Significance Tests. Exceptional Children 65:3, pages 329-337.
Crossref
Michael J. Chen & Xitao Fan. (1998) The relationship between variance components and mean difference effect size. Current Psychology 17:4, pages 301-311.
Crossref
Bruce Thompsons & Patricia A. Snyder. (2011) Statistical Significance and Reliability Analyses in Recent Journal of Counseling & Development Research Articles . Journal of Counseling & Development 76:4, pages 436-441.
Crossref
Rebecca P. Ang. (2016) Use of Double Cross-Validation and Bootstrap Methods to Estimate Replicability of Results of Multiple Regression. Perceptual and Motor Skills 86:3_suppl, pages 1143-1152.
Crossref
Ruma Falk. (2016) Replication-A Step in the Right Direction. Theory & Psychology 8:3, pages 313-321.
Crossref
David Sohn. (2016) Statistical Significance and Replicability. Theory & Psychology 8:3, pages 291-311.
Crossref
Bruce Thompson. (2016) Book Reviews. Educational and Psychological Measurement 58:2, pages 334-346.
Crossref
J. Thomas Kellow. (2016) Beyond Statistical Significant Tests: The Importance of Using Other Estimates of Treatment Effects to Interpret Results Evaluation. American Journal of Evaluation 19:1, pages 123-134.
Crossref
John C. Caruso & Norman Cliff. (2016) Empirical Size, Coverage, and Power of Confidence Intervals for Spearman's Rho. Educational and Psychological Measurement 57:4, pages 637-654.
Crossref
Bruce Thompson. (2016) Rejoinder: Editorial Policies Regarding Statistical Significance Tests: Further Comments. Educational Researcher 26:5, pages 29-32.
Crossref
Bruce Thompson. (2016) Research news and Comment: AERA Editorial Policies Regarding Statistical Significance Testing: Three Suggested Reforms. Educational Researcher 25:2, pages 26-30.
Crossref
Xitao FanWilliam G. Jacoby. (2016) BOOTSREG: An SAS Matrix Language Program for Bootstrapping Linear Regression Models. Educational and Psychological Measurement 55:5, pages 764-768.
Crossref
Bruce Thompson. (2016) Book Reviews. Educational and Psychological Measurement 55:2, pages 340-350.
Crossref

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.