Significance, Effect Sizes, Stepwise Methods, and Other Issues: Strong Arguments Move the Field

References

• Abelson , R. P. 1997 . “ A retrospective on the significance test ban of 1999 (If there were no significance tests, they would be invented) ” . In What if there were no significance tests? , Edited by: Harlow , L. L. , Mulaik , S. A. and Steiger , J. H. 117 – 141 . Mahwah, NJ : Erlbaum .
   Google Scholar
• Aiken , L. S. , West , S. G. , Sechrest , L. , Reno , R. R. , Roediger , H. L. , Scarr , S. , Kazdin , A. E. and Sherman , S. J. 1990 . The training in statistics, methodology, and measurement in psychology . American Psychologist , 45 : 721 – 734 .
   Web of Science ®Google Scholar
• American Psychological Association . 1994 . Publication manual of the American Psychological Association, , 4th ed , Washington, DC : Author .
   Google Scholar
• Anderson , D. R. , Burnham , K. P. and Thompson , W. L. 2000 . Null hypothesis testing: Problems, prevalence, and an alternative . Journal of Wildlife Management , 64 : 912 – 923 .
   Web of Science ®Google Scholar
• Capraro , M. M. , Capraro , R. M. and Henson , R. K. 2001 . Measurement error of scores on the Mathematics Anxiety Rating Scale across studies . Educational and Psychological Measurement , 61 : 373 – 386 .
   Web of Science ®Google Scholar
• Caruso , J. C. 2000 . Reliability generalization of the NEO personality scales . Educational and Psychological Measurement , 60 : 236 – 254 .
   Web of Science ®Google Scholar
• Caruso , J. C. , Witkiewitz , K. , Belcourt-Dittloff , A. and Gottlieb , J. Reliability of scores from the Eysenck Personality Questionnaire: A reliability generalization (RG) study . Educational and Psychological Measurement , 61 (in press)
   Web of Science ®Google Scholar
• Cliff , N. 1987 . Analyzing multivariate data , San Diego, CA : Harcourt Brace Jovanovich .
   Google Scholar
• Cohen , J. 1994 . The earth is round (p< .05) . American Psychologist , 49 : 997 – 1003 .
   Web of Science ®Google Scholar
• Cohen , J. and Cohen , P. 1983 . Applied multiple regression/correlation analysis for the behavioral sciences , Hillsdale, NJ : Erlbaum .
   Google Scholar
• Cortina , J. M. and Dunlap , W. P. 1997 . Logic and purpose of significance testing . Psychological Methods , 2 : 161 – 172 .
   Web of Science ®Google Scholar
• Courville , T. and Thompson , B. 2001 . Use of structure coefficients in published multiple regression articles: β is not enough . Educational and Psychological Measurement , 61 : 229 – 248 .
   Web of Science ®Google Scholar
• Cumming , G. and Finch , S. 2001 . A primer on the understanding, use and calculation of confidence intervals that are based on central and noncentral distributions . Educational and Psychological Measurement , 61 : 532 – 574 .
   Web of Science ®Google Scholar
• Duncan , O. D. 1975 . Introduction to structural equation models , New York : Academic Press .
   Google Scholar
• Fan , X. and Thompson , B. 2001 . Confidence intervals about score reliability coefficients, please: An EPM guidelines editorial . Educational and Psychological Measurement , 61 : 517 – 531 .
   Web of Science ®Google Scholar
• Finch , S. , Cumming , G. and Thomason , N. 2001 . Reporting of statistical inference in the Journal of Applied Psychology: Little evidence of reform . Educational and Psychological Measurement , 61 : 181 – 210 .
   Web of Science ®Google Scholar
• Finch , S. , Thomason , N. and Cumming , G. Past and future APA guidelines for statistical practice . Theory & Practice. , (in press)
   Google Scholar
• Gorsuch , R. L. 1983 . Factor analysis, , 2nd ed , Hillsdale, NJ : Erlbaum .
   Google Scholar
• Hagen , R. L. 1997 . In praise of the null hypothesis statistical test . American Psychologist , 52 : 15 – 24 .
   Web of Science ®Google Scholar
• Harlow , L. L. , Mulaik , S. A. and Steiger , J. H. 1997 . What if there were no significance tests , Mahwah, NJ : Erlbaum .
   Google Scholar
• Helms , J. E. 1999 . Another meta-analysis of the White Racial Identity Attitude Scale's Cronbach alphas: Implications for validity . Measurement and Evaluation in Counseling and Development , 32 : 122 – 137 .
   Web of Science ®Google Scholar
• Horst , P. 1966 . Psychological measurement and prediction , Belmont, CA : Wadsworth .
   Google Scholar
• Huberty , C. J. 1989 . “ Problems with stepwise methods—Better alternatives ” . In Advances in social science methodology , Edited by: Thompson , B. Vol. 1 , 43 – 70 . Greenwich, CT : JAI Press .
   Google Scholar
• Huberty , C. J. 1994 . Applied discriminant analysis , New York : Wiley .
   Google Scholar
• Hyde , J. S. 2001 . Reporting effect sizes: The roles of editors, textbook authors, and publication manuals . Educational and Psychological Measurement , 61 : 225 – 228 .
   Web of Science ®Google Scholar
• International Committee of Medical Journal Editors . 1997 . Uniform requirements for manuscripts submitted to biomedical journals . Journal of the American Medical Association , 277 : 927 – 934 .
   PubMed Web of Science ®Google Scholar
• Kirk , R. E. 2001 . Promoting good statistical practices: Some suggestions . Educational and Psychological Measurement , 61 : 213 – 218 .
   Web of Science ®Google Scholar
• Knapp , T and Sawilowsky , S. 2001 . Constructive criticisms of methodological and editorial practices . The Journal of Experimental Education , 70 : 65 – 79 .
   Web of Science ®Google Scholar
• Lancaster , B. P. 1999 . “ Defining and interpreting suppressor effects: Advantages and limitations ” . In Advances in social science methodology , Edited by: Thompson , B. Vol. 5 , 139 – 148 . Stamford, CT : JAI Press .
   Google Scholar
• Levin , J. R. 1998 . To test or not to test H0? . Educational and Psychological Measurement , 58 : 311 – 331 .
   Web of Science ®Google Scholar
• Levine , M. S. 1977 . Canonical analysis and factor comparison , Beverly Hills : Sage .
   Google Scholar
• Ludbrook , J. and Dudley , H. 1998 . Why permutation tests are superior to t and F tests in medical research . The American Statistician , 52 : 127 – 132 .
   Web of Science ®Google Scholar
• Melton , A. 1962 . Editorial . Journal of Experimental Psychology , 64 : 553 – 557 .
   Google Scholar
• Meredith , W. 1964 . Canonical correlations with fallible data . Psychometrika , 29 : 55 – 65 .
   Web of Science ®Google Scholar
• Nickerson , R. S. 2000 . Null hypothesis significance testing: A review of an old and continuing controversy . Psychological Methods , 5 : 241 – 301 .
   PubMed Web of Science ®Google Scholar
• Pedhazur , E. J. 1982 . Multiple regression in behavioral research: Explanation and prediction, , 2nd ed. , New York : Holt, Rinehart and Winston .
   Google Scholar
• Pedhazur , E. J. and Schmelkin , L. P. 1991 . Measurement, design, and analysis: An integrated approach , Hillsdale, NJ : Erlbaum .
   Google Scholar
• Schmidt , F. L. 1996 . Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers . Psychological Methods , 1 : 115 – 129 .
   Web of Science ®Google Scholar
• Smithson , M. 2001 . Correct confidence intervals for various regression effect sizes and parameters: The importance of noncentral distributions in computing intervals . Educational and Psychological Measurement , 61 : 605 – 632 .
   Web of Science ®Google Scholar
• Snyder , P. 1991 . “ Three reasons why stepwise regression methods should not be used by researchers ” . In Advances in educational research: Substantive findings, methodological developments , Edited by: Thompson , B. Vol. 1 , 99 – 105 . Greenwich, CT : JAI Press .
   Google Scholar
• Thompson , B. 1984 . Canonical correlation analysis: Uses and interpretation , Thousand Oaks, CA : Sage .
   Google Scholar
• Thompson , B. 1989 . Why won't stepwise methods die? . Measurement and Evaluation in Counseling and Development , 21 : 146 – 148 .
   Web of Science ®Google Scholar
• Thompson , B. 1994 . Guidelines for authors . Educational and Psychological Measurement , 54 : 837 – 847 .
   Google Scholar
• Thompson , B. 1995 . Stepwise regression and stepwise discriminant analysis need not apply here: A guidelines editorial . Educational and Psychological Measurement , 55 : 525 – 534 .
   Web of Science ®Google Scholar
• Thompson , B. 1996 . AERA editorial policies regarding statistical significance testing: Three suggested reforms . Educational Researcher , 25 ( 2 ) : 26 – 30 .
   Google Scholar
• Thompson , B. 1997 . The importance of structure coefficients in structural equation modeling confirmatory factor analysis . Educational and Psychological Measurement , 57 : 5 – 19 .
   Web of Science ®Google Scholar
• Thompson , B. 1998a . In praise of brilliance: Where that praise really belongs . American Psychologist , 53 : 799 – 800 .
   Web of Science ®Google Scholar
• Thompson , B. , Harlow , L. , Mulaik , S. and Steiger , J. 1998b . Review of What if there were no significance tests? . Educational and Psychological Measurement , 58 : 332 – 344 .
   Web of Science ®Google Scholar
• Thompson , B. 1998c . Statistical significance and effect size reporting: Portrait of a possible future . Research in the Schools , 5 ( 2 ) : 33 – 38 .
   Google Scholar
• Thompson , B. 1999a . “ Five methodology errors in educational research: A pantheon of statistical significance and other faux pas ” . In Advances in social science methodology , Edited by: Thompson , B. Vol. 5 , 23 – 86 . Stamford, CT : JAI Press .
   Google Scholar
• Thompson , B. 1999b . Journal editorial policies regarding statistical significance tests: Heat is to fire as p is to importance . Educational Psychology Review , 11 : 157 – 169 .
   Web of Science ®Google Scholar
• Thompson , B. 2000 . “ Canonical correlation analysis ” . In Reading and understanding more multivariate statistics , Edited by: Grimm , L. and Yarnold , P. 285 – 316 . Washington, DC : American Psychological Association .
   Google Scholar
• Thompson , B. “Statistical,” “practical,” and “clinical”: How many kinds of significance do counselors need to consider? . Journal of Counseling and Development. , (in press)
   Web of Science ®Google Scholar
• Thompson , B. and Vacha-Haase , T. 2000 . Psychometrics is datametrics: The test is not reliable . Educational and Psychological Measurement , 60 : 174 – 195 .
   Web of Science ®Google Scholar
• Vacha-Haase , T. 1998 . Reliability generalization: Exploring variance in measurement error affecting score reliability across studies . Educational and Psychological Measurement , 58 : 6 – 20 .
   Web of Science ®Google Scholar
• Vacha-Haase , T. 2001 . Statistical significance should not be considered one of life's guarantees: Effect sizes are needed . Educational and Psychological Measurement , 61 : 219 – 224 .
   Web of Science ®Google Scholar
• Vacha-Haase , T. , Kogan , L. , Tani , C. R. and Woodall , R. A. 2001 . Reliability generalization: Exploring reliability coefficients of MMPI clinical scales scores . Educational and Psychological Measurement , 61 : 45 – 59 .
   Web of Science ®Google Scholar
• Vacha-Haase , T. , Kogan , L. R. and Thompson , B. 2000 . Sample compositions and variabilities in published studies versus those in test manuals: Validity of score reliability inductions . Educational and Psychological Measurement , 60 : 509 – 522 .
   Web of Science ®Google Scholar
• Vacha-Haase , T. , Nilsson , J. E. , Reetz , D. R. , Lance , T. S. and Thompson , B. 2000 . Reporting practices and APA editorial policies regarding statistical significance and effect size . Theory & Psychology , 10 : 413 – 425 .
   Web of Science ®Google Scholar
• Viswesvaran , C. and Ones , D. 2000 . Measurement error in “Big Five Factors” personality assessment: Reliability generalization across studies and measures . Educational and Psychological Measurement , 60 : 224 – 235 .
   Web of Science ®Google Scholar
• Whittington , D. 1998 . How well do researchers report their measures? An evaluation of measurement in published educational research . Educational and Psychological Measurement , 58 : 21 – 37 .
   Web of Science ®Google Scholar
• Wilkinson L., & APA Task Force on Statistical Inference . 1999 . Statistical methods in psychology journals: Guidelines and explanations . American Psychologist , 54 : 594 – 604 . http://www.apa.org/journals/amp/amp548594.htmlreprint available through the APA Home Page
   Web of Science ®Google Scholar
• Yin , P. and Fan , X. 2000 . Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies . Educational and Psychological Measurement , 60 : 201 – 223 .
   Web of Science ®Google Scholar