Beyond Cohen's d: Alternative Effect Size Measures for Between-Subject Designs

REFERENCES

• Algina , J. , Keselman , H. J. and Penfield , R. D. 2005 . An alternative to Cohen's standardized mean difference effect size: A robust parameter and confidence interval in the two independent groups case . Psychological Methods , 10 : 317 – 328 . doi: 10.1037/1082-989X.10.3.317
   PubMed Web of Science ®Google Scholar
• Alhija , F. N.-A. and Levy , A. 2009 . Effect size reporting practices in published articles . Educational and Psychological Measurement , 69 : 245 – 265 . doi: 10.1177/0013164408315266
   Web of Science ®Google Scholar
• American Educational Research Association . 2006 . Standards for reporting on empirical social science research in AERA publications . Educational Researcher , 35 : 33 – 40 . doi: 10.3102/0013189X035006033
   Google Scholar
• American Psychological Association . 2010 . Publication manual of the American Psychological Association (6th ed.) , Washington , DC : Author .
   Google Scholar
• Andersen , M. B. , McCullagh , P. and Wilson , G. J. 2007 . But what do the numbers really tell us?: Arbitrary metrics and effect size reporting in sport psychology research . Journal of Sport & Exercise Psychology , 29 : 664 – 672 . Retrieved from http://journals.humankinetics.com/jsep
   PubMed Web of Science ®Google Scholar
• APA Publications and Communications Board Working Group on Journal Articles Reporting Standards . 2008 . Reporting standards for research in psychology: Why do we need them? What might they be? . American Psychologist , 63 : 839 – 851 . doi: 10.1037/0003-066X.63.9.839
   PubMed Web of Science ®Google Scholar
• Bond , C. F. Jr. , Wiitala , W. L. and Richard , F. D. 2003 . Meta-analysis of raw mean differences . Psychological Methods , 8 : 406 – 418 . doi: 10.1037/1082-989X.8.4.406
   PubMed Web of Science ®Google Scholar
• Bonett , D. G. 2008 . Confidence intervals for standardized linear contrasts of means . Psychological Methods , 13 : 99 – 109 . doi: 10.1037/1082-989X.13.2.99
   PubMed Web of Science ®Google Scholar
• Brunner , E. and Puri , M. L. 2001 . Nonparametric methods in factorial designs . Statistical Papers , 42 : 1 – 52 . doi: 10.1007/s003620000039
   Web of Science ®Google Scholar
• Burr , I. W. and Cislak , P. J. 1968 . On a general system of distributions: I. Its curve-shape characteristics: II. The sample median . Journal of American Statistical Association , 63 : 627 – 635 . Retrieved from http://www.amstat.org/publications/jasa.cfm
   Web of Science ®Google Scholar
• Cliff , N. 1993 . Dominance statistics: Ordinal analyses to answer ordinal questions . Psychological Bulletin , 114 : 494 – 509 . doi: 10.1037/0033-2909.114.3.494
   Web of Science ®Google Scholar
• Cliff , N. 1996 . Ordinal methods for behavioral data analysis , Mahwah , NJ : Erlbaum .
   Google Scholar
• Cohen , J. 1969 . Statistic power analysis in the behavioral sciences , New York , NY : Academic Press .
   Google Scholar
• Cohen , J. 1988 . Statistic power analysis in the behavioral sciences , Hillsdale , NJ : Erlbaum .
   Google Scholar
• Cohen , J. 1994 . The earth is round (p < .05) . American Psychologist , 49 : 997 – 1003 . doi: 10.1037/0003-066X.49.12.997
   Web of Science ®Google Scholar
• Cooper , H. , Hedges , L. V. and Valentine , J. C. 2009 . The handbook of research synthesis and meta-analysis (2nd ed.) , New York , NY : Russell Sage Foundation .
   Google Scholar
• Delaney , H. D. and Vargha , A. 2002 . Comparing several robust tests of stochastic equality with ordinally scaled variables and small to moderate sized samples . Psychological Methods , 7 : 485 – 503 . doi: 10.1037/1082–989X.7.4.485
   PubMed Web of Science ®Google Scholar
• Dunleavy , E. M. , Barr , C. D. , Glenn , D. M. and Miller , K. R. 2006 . Effect size reporting in applied psychology: How are we doing? . The Industrial-Organizational Psychologist , 43 : 29 – 37 . Retrieved from http://www.openj-gate.com/browse/Archive.aspx?year=2009&Journal_id=102632
   Google Scholar
• Efron , B. and Tibshirani , R. J. 1993 . An introduction to the bootstrap , New York , NY : Chapman & Hall .
   Google Scholar
• Ellison , B. E. 1964 . Two theorems for inferences about the normal distribution with applications in acceptance sampling . Journal of American Statistical Association , 59 : 89 – 95 . Retrieved from http://www.amstat.org/publications/jasa.cfm
   Web of Science ®Google Scholar
• Fidler , F. , Cumming , G. , Thomason , N. , Pannuzzo , D. , Smith , J. , Fyffe , P. and Schmitt , R. 2005 . Evaluating the effectiveness of editorial policy to improve statistical practice: The case of the Journal of Consulting and Clinical Psychology . Journal of Consulting and Clinical Psychology , 73 : 136 – 143 . doi: 10.1037/0022-006X.73.1.136
   PubMed Web of Science ®Google Scholar
• Glass , G. V. 1976 . Primary, secondary, and meta-analysis research . Educational Researcher , 5 : 3 – 8 . doi: 10.3102/0013189X005010003
   Google Scholar
• Glass , G. V. , McGaw , B. and Smith , M. L. 1981 . Meta-analysis in social research , Newbury Park , CA : Sage .
   Google Scholar
• Grissom , R. J. 1994 . Probability of the superior outcome of one treatment over another . Journal of Applied Psychology , 79 : 314 – 316 . doi: 10.1037/0021-9010.79.2.31
   Web of Science ®Google Scholar
• Grissom , R. J. 2000 . Heterogeneity of variance in clinical data . Journal of Consulting and Clinical Psychology , 68 : 155 – 165 . doi: 10.1037/0022-006X.68.1.155
   PubMed Web of Science ®Google Scholar
• Grissom , R. J. and Kim , J. J. 2001 . Review of assumptions and problems in the appropriate conceptualization of effect size . Psychological Methods , 6 : 135 – 146 . doi: 10.1037/1082-989x.6.2.135
   PubMed Web of Science ®Google Scholar
• Grissom , R. J. and Kim , J. J. 2005 . Effect sizes for research: A broad practical approach , Mahwah , NJ : Erlbaum .
   Google Scholar
• Grissom , R. J. and Kim , J. J. 2012 . Effect sizes for research: Univariate and multivariate applications (2nd ed.) , New York , NY : Routledge .
   Google Scholar
• Harrison , J. , Thompson , B. and Vannest , K. J. 2009 . Interpreting the evidence for effective interventions to increase the academic performance of students with ADHD: Relevance of the statistical significance controversy . Review of Educational Research , 79 : 740 – 775 . doi: 10.3102/0034654309331516
   Web of Science ®Google Scholar
• Hedges , L. V. 1981 . Distributional theory for Glass's estimator of effect size and related estimators . Journal of Educational Statistics , 6 : 107 – 128 . doi: 10.2307/1164588
   Google Scholar
• Hedges , L. V. 1982 . Estimation of effect size from a series of independent experiments . Psychological Bulletin , 92 : 490 – 499 . doi: 10.1037/0033-2909.92.2.490
   Web of Science ®Google Scholar
• Hedges , L. V. and Olkin , I. 1984 . Nonparametric estimators of effect size in meta-analysis . Psychological Bulletin , 96 : 573 – 580 . doi: 10.1037/0033-2909.96.3.573
   Web of Science ®Google Scholar
• Hedges , L. V. and Olkin , I. 1985 . Statistical methods for meta-analysis , Orlando , FL : Academic Press .
   Google Scholar
• Hess , M. R. and Kromrey , J. D. Robust confidence intervals for effect sizes: A comparative study of Cohen's d and Cliff's delta under non-normality and heterogeneous variance . Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA ,
   Google Scholar
• Hess , B. , Olejnik , S. and Huberty , C. J. 2001 . The efficacy of two improvement-over-chance effect sizes for two-group univariate comparisons under variance heterogeneity and nonnormality . Educational and Psychological Measurement , 61 : 909 – 936 . doi: 10.1177/00131640121971572
   Web of Science ®Google Scholar
• Hinkle , D. E. 2003 . Applied Statistics for the Behavioral Sciences , Boston , MA : Houghton Mifflin .
   Google Scholar
• Hogarty , K. Y. and Kromrey , J. D. We've been reporting some effect sizes: Can you guess what they mean? . Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA ,
   Google Scholar
• Huberty , C. J. 2002 . A history of effect size indices . Educational and Psychological Measurement , 62 : 227 – 240 . doi: 10.1177/0013164402062002002
   Web of Science ®Google Scholar
• Huberty , C. J. and Lowman , L. L. 2000 . Group overlap as a basis for effect size . Educational and Psychological Measurement , 60 : 543 – 563 . doi: 10.1177/0013164400604004
   Web of Science ®Google Scholar
• Huynh , C. L. A unified approach to the estimation of effect size in meta-analysis . Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco. (ERIC Document Reproduction Service No. 306 248) ,
   Google Scholar
• Jacobson , N. , Follette , W. C. and Revenstorf , D. 1984 . Psychotherapy outcome research: Methods for reporting variability and evaluating clinical significance . Behavior Therapy , 15 : 336 – 352 . doi: 10.1016/S0005-7894(84)80002-7
   Web of Science ®Google Scholar
• Jacobson , N. and Truax , P. 1991 . Clinical significance: A statistical approach to defining meaningful change in psychotherapy research . Journal of Consulting and Clinical Psychology , 59 : 12 – 19 . doi: 10.1037//0022-006X.59.1.12
   PubMed Web of Science ®Google Scholar
• James , G. S. 1951 . The comparison of several groups of observations when the ratios of the population variances are unknown . Biometrika , 38 : 234 – 329 . Retrieved from http://biomet.oxfordjournals.org
   Web of Science ®Google Scholar
• James , G. S. 1954 . Tests of linear hypothesis in univariate and multivariate analysis when the ratios of the populations variances are unknown . Biometrika , 41 : 19 – 43 . Retrieved from http://biomet.oxfordjournals.org
   Web of Science ®Google Scholar
• Kelley , K. 2005 . The effects of nonnormal distributions on confidence intervals around the standardized mean difference: Bootstrap and parametric confidence intervals . Educational and Psychological Measurement , 65 : 51 – 69 . doi: 10.1177/0013164404264850
   Web of Science ®Google Scholar
• Kelley , K. and Preacher , K. J. 2012 . On effect size . Psychological Methods , 17 : 137 – 152 . doi: 10.1037/a0028086
   PubMed Web of Science ®Google Scholar
• Keselman , H. J. , Algina , J. , Lix , L. M. , Wilcox , R. R. and Deering , K. N. 2008 . A generally robust approach for testing hypotheses and setting confidence intervals for effect sizes . Psychological Methods , 13 : 110 – 129 . doi: 10.1037/1082-989X.13.2.110
   PubMed Web of Science ®Google Scholar
• Keselman , H. J. , Huberty , C. J. , Lix , L. M. , Olejnik , S. , Cribbie , R. A. , Donahue , B. and …Levin , J. R. 1998 . Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses . Review of Educational Research , 68 : 350 – 386 . doi: 10.3102/00346543068003350
   Web of Science ®Google Scholar
• Kirk , R. E. 1996 . Practical significance: A concept whose time has come . Educational and Psychological Measurement , 56 : 746 – 759 . doi: 10.1177/0013164496056005002
   Web of Science ®Google Scholar
• Kirk , R. E. 2007 . Effect magnitude: A different focus . Journal of Statistical Planning and Inference , 137 : 1634 – 1646 . doi: 10.1016/j.jspi.2006.09.011
   Web of Science ®Google Scholar
• Kline , R. B. 2004 . “ Beyond significant testing: Reforming data analysis methods in behavioral research ” . Washington , DC : American Psychological Association .
   Google Scholar
• Kraemer , H. C. and Andrews , G. 1982 . A nonparametric technique for meta-analysis effect size calculation . Psychological Bulletin , 91 : 404 – 412 . doi: 10.1037/0033-2909.91.2.404
   Web of Science ®Google Scholar
• Kraemer , H. C. and Kupfer , D. J. 2006 . Size of treatment effects and their importance to clinical research and practice . Biological Psychiatry , 59 : 990 – 996 . doi: 10.1016/j.biopsych.2005.09.014
   PubMed Web of Science ®Google Scholar
• Kramer , S. and Rosenthal , R. 1999 . “ Effect sizes and significance levels in small-sample research ” . In Statistical strategies for small sample research , Edited by: Hoyle , R. 59 – 79 . Thousand Oaks , CA : Sage .
   Google Scholar
• Kromrey , J. D. and Coughlin , K. B. ROBUST_ES: A SAS macro for computing robust estimates of effect size . Paper presented at the annual meeting of the SouthEast SAS Users Group, Hilton Head, SC , Retrieved from http://analytics.ncsu.edu/sesug/2007/PO19.pdf
   Google Scholar
• Levy , P. 1967 . Substantive significance of significant differences between groups . Psychological Bulletin , 67 : 37 – 40 . doi: 10.1037/h0020415
   PubMed Web of Science ®Google Scholar
• MacGrath , R. E. and Meyer , G. J. 2006 . When effect sizes disagree: The case of r and d . Psychological Methods , 11 : 386 – 401 . doi: 10.1037/1082-989X.11.4.386
   PubMed Web of Science ®Google Scholar
• Matthews , M. S. , Gentry , M. , McCoach , D. B. , Worrell , F. C. , Matthews , D. and Dixon , F. 2008 . Evaluating the state of a field: Effect size reporting in gifted education . The Journal of Experimental Education , 77 : 55 – 65 . doi: 10.3200/JEXE.77.1.55-68
   Web of Science ®Google Scholar
• Maxwell , S. E. , Camp , C. J. and Arvey , R. D. 1981 . Measures of strength of association: A comparative examination . Journal of Applied Psychology , 66 : 525 – 534 . doi: 10.1037/0021-9010.66.5.525
   Web of Science ®Google Scholar
• McGraw , K. O. and Wong , S. P. 1992 . A common language effect size statistic . Psychological Bulletin , 111 : 361 – 365 . doi: 10.1037/0033-2909.111.2.361
   Web of Science ®Google Scholar
• Meline , T. and Wang , B. 2004 . Effect reporting practices in AJSLP and other ASHA journals, 1999–2003 . American Journal of Speech-Language Pathology , 13 : 202 – 207 . Retrieved from http://ajslp.asha.org
   PubMed Web of Science ®Google Scholar
• Meyers , L. S. , Gamst , G. and Guarino , A. J. 2006 . Applied multivariate research: Design and interpretation , Thousand Oaks , CA : Sage .
   Google Scholar
• Murphy , B. P. 1976 . Comparison of some two sample means tests by simulation . Communications in Statistics B: Simulations and Computation , 5 : 23 – 32 . doi: 10.1080/03610917608812004
   Web of Science ®Google Scholar
• Nakagawa , S. and Cuthill , I. C. 2007 . Effect size, confidence interval, and statistical significance: A practical guide for biologists . Biological Reviews , 82 : 591 – 605 . doi: 10.1111/j.1469-185X.2007.00027.x
   PubMed Web of Science ®Google Scholar
• Natesan , P. and Thompson , B. 2007 . Extending improvement-over change I-index effect size simulation studies to cover some small-sample cases . Educational and Psychological Measurement , 67 : 59 – 72 . doi: 10.1177/001364406292028
   Web of Science ®Google Scholar
• Oakes , M. 1986 . Statistical inference: A commentary for the social and behavioral sciences , New York , NY : Wiley .
   Google Scholar
• Odgaard , E. C. and Fowler , R. L. 2010 . Confidence intervals for effect sizes: Compliance and clinical significance in the Journal of Consulting and Clinical Psychology . Journal of Consulting and Clinical Psychology , 78 : 287 – 297 . doi: 10.1037/a0019294
   PubMed Web of Science ®Google Scholar
• Olejnik , S. and Algina , J. 2000 . Measures of effect size for comparative studies: Applications, interpretations, and limitations . Contemporary Educational Psychology , 25 : 241 – 286 . doi: 10.1006/ceps.2000.1040
   PubMed Web of Science ®Google Scholar
• Osborne , J. W. 2008 . Sweating the small stuff in educational psychology: How effect size and power reporting failed to change from 1969 to 1999, and what that means for the future of changing practices . Educational Psychology , 28 : 151 – 160 . doi: 10.1080/01443410701491718
   Web of Science ®Google Scholar
• Peng , C. -Y. J. , Chen , L.-T. , Chiang , H.-M. and Chiang , Y.-C. 2013 . The impact of APA and AERA guidelines an effect size reporting . Educational Psychology Review , 25 : 157 – 209 . doi: 10.1007/s10648-013-9218-2
   Web of Science ®Google Scholar
• Pratt , J. W. 1964 . Obustness [sic] of some procedures for the two-sample location problem . Journal of American Statistical Association , 59 : 665 – 680 . Retrieved from http://www.amstat.org/publications/jasa.cfm
   Web of Science ®Google Scholar
• Pratt , J. W. and Gibbons , J. D. 1981 . Concepts of nonparametric theory , New York , NY : Springer-Verlag .
   Google Scholar
• Rosenthal , R. 1994 . “ Parametric measures of effect size ” . In The handbook of research synthesis , Edited by: Cooper , H. and Hedges , L. V. 231 – 244 . New York , NY : Russell Sage Foundation .
   Google Scholar
• Ruscio , J. 2008 . A probability-based measure of effect size: Robustness to base rates and other factors . Psychological Methods , 13 : 19 – 30 . doi: 10.1037/1082-989X.13.1.19
   PubMed Web of Science ®Google Scholar
• Sarhan , A. S. and Greenberg , B. G. 1962 . Contributions to order statistics , New York , NY : Wiley .
   Google Scholar
• Smith , M. L. and Glass , G. V. 1977 . Meta-analysis of psychotherapy outcome studies . American Psychologist , 32 : 752 – 760 . doi: 10.1037/0003-066X.32.9.752
   PubMed Web of Science ®Google Scholar
• Smith , M. L. and Honoré , H. H. 2008 . Effect size reporting in current health education literautre . American Journal of Health Studies , 23 : 130 – 135 . Retrieved from http://www.va-ajhs.com
   Google Scholar
• Snyder , P. A. and Thompson , B. 1998 . Use of tests of statistical significance and other analytic choices in a school psychology journal: Review of practices and suggested alternatives . School Psychology Quarterly , 13 : 335 – 348 . doi: 10.1037/h0088990
   Web of Science ®Google Scholar
• Somers , R. H. 1968 . An approach to the multivariate analysis of ordinal data . American Sociological Review , : 971 – 977 . doi: 10.2307/2092687
   Web of Science ®Google Scholar
• Staudte , R. G. and Sheather , S. J. 1990 . Robust estimation and testing , New York , NY : Wiley .
   Google Scholar
• Steiger , J. H. and Fouladi , R. T. 1997 . “ Noncentrality interval estimation and the evaluation of statistical models ” . In What if there were no significance tests? , Edited by: Harlow , L. , Mulaik , S. and Steiger , J. H. 221 – 257 . Hillsdale , NJ : Erlbaum .
   Google Scholar
• Sun , S. Y. , Pan , W. and Wang , L. L. 2010 . A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology . Journal of Educational Psychology , 102 : 989 – 1004 . doi: 10.1037/a0019507
   Web of Science ®Google Scholar
• Tabachnick , B. G. and Fidell , L. S. 2007 . Using multivariate statistics , Boston , MA : Pearson/Allyn & Bacon .
   Google Scholar
• Thompson , B. 1988 . A note on statistical significance testing . Measurement and Evaluation in Counseling and Development , 20 : 146 – 148 .
   Web of Science ®Google Scholar
• Thompson , B. 1989 . Statistical significance, result importance, and result generalizability: Three noteworthy but somewhat different issues . Measurement and Evaluation in Counseling and Development , 22 : 2 – 5 .
   Web of Science ®Google Scholar
• Thompson , B. 1993 . Statistical significance testing in contemporary practice (theme issue) . The Journal of Experimental Education , 61 : 285 – 393 . (Guest Ed.)
   Web of Science ®Google Scholar
• Thompson , B. 1994 . Guidelines for authors . Educational and Psychological Measurement , 54 : 837 – 847 .
   Google Scholar
• Thompson , B. 2002 . What future quantitative social science research could look like: Confidence intervals for effect sizes . Educational Researcher , 31 : 25 – 32 . doi: 10.3102/0013189X031003025
   Google Scholar
• Thompson , B. 2008 . Foundations of behavioral statistics: An insight-based approach , New York , NY : Guilford Press .
   Google Scholar
• Thompson , B. and Snyder , P. A. 1997 . Statistical significance testing practices . The Journal of Experimental Education , 66 : 75 – 83 . doi: 10.1080/00220979709601396
   Web of Science ®Google Scholar
• Vacha-Haase , T. , Nilsson , J. E. , Reetz , D. R. , Lance , T. S. and Thompson , B. 2000 . Reporting practices and APA editorial policies regarding statistical significance and effect size . Theory and Psychology , 10 : 413 – 425 . doi: 10.1177/0959354300103006
   Web of Science ®Google Scholar
• Vargha , A. and Delaney , H. D. 1998 . The Kruskal-Wallis test and stochastic homogeneity . Journal of Educational and Behavioral Statistics , 23 : 170 – 192 . doi: 10.2307/1165320
   Web of Science ®Google Scholar
• Vargha , A. and Delaney , H. D. 2000 . A critique and improvement of the CL common language effect size statistics of McGraw and Wong . Journal of Educational and Behavioral Statistics , 25 : 101 – 132 . doi: 10.2307/1165329
   Web of Science ®Google Scholar
• Welch , B. L. 1951 . On the comparison of several mean values: An alternative approach . Biometrika , 38 : 330 – 336 .
   Web of Science ®Google Scholar
• Wilcox , R. R. 2005 . Introduction to robust estimation and hypothesis testing (2nd ed.) , San Diego , CA : Elsevier Academic Press .
   Google Scholar
• Wilkinson , L. and the APA Task Force on Statistical Inference . 1999 . Statistical methods in psychology journals: Guidelines and explanations . American Psychologist , 54 : 594 – 604 . doi: 10.1037/0003-066X.54.8.594
   Web of Science ®Google Scholar
• Yin , P. and Fan , X. 2001 . Estimating R2 shrinkage in multiple regression: A comparison of different analytical methods . The Journal of Experimental Education , 69 : 203 – 224 . doi: 10.1080/00220970109600656
   Web of Science ®Google Scholar
• Yuen , K. K. 1974 . The two-sample trimmed t for unequal population variances . Biometrika , 61 : 165 – 170 . doi: 10.2307/2334299
   Web of Science ®Google Scholar
• Yuen , K. K. and Dixon , W. J. 1973 . The approximate behavior and performance of the two-sample trimmed t . Biometrika , 60 : 369 – 374 . doi: 10.2307/2334550
   Web of Science ®Google Scholar
• Zhang , Z. and Schoeps , N. 1997 . On robust estimation of effect size under semiparametric models . Psychometrika , 62 : 201 – 214 . doi: 10.1007/BF02295275
   Web of Science ®Google Scholar
• Zientek , L. R. , Capraro , M. M. and Capraro , R. M. 2008 . Reporting practices in quantitative teacher education research: one look at the evidence cited in the AERA Panel Report . Educational Researcher , 37 : 208 – 216 . doi: 10.3102/0013189X08319762
   Web of Science ®Google Scholar
• Zimmerman , D. M. and Zumbo , B. D. 1993 . Rank transformations and the power of the Student t test and Welch t’ test for non-normal populations with unequal variances . Canadian Journal of Experimental Psychology , 47 : 523 – 539 . doi: 10.1037/h0078850
   Web of Science ®Google Scholar