Detecting bias in meta-analyses of distance education research: big pictures we can rely on

References

• Abrami, P. C., & Bernard, R. M. (2006). Research on distance education: In defense of field experiments. Distance Education, 27, 5–26. doi:10.1080/0158791060065311610.1080/01587910600653116
   Google Scholar
• *Allen, M., Mabry, E., Mattrey, M., Bourhis, J., Titsworth, S., & Burrell, N. (2004). Evaluating the effectiveness of distance learning: A comparison using meta-analysis. Journal of Communication, 54, 402–420. doi:10.1111/j.1460-2466.2004.tb02636.x10.1111/jcom.2004.54.issue-3
   Web of Science ®Google Scholar
• Bernard, R. M. (2010a, May). The past, present and future: Meta-analysis of online and distance education research. Invited symposium presentation at the American Educational Research Association, Denver, CO.
   Google Scholar
• Bernard, R. M. (2010b, October). The state of research in online and blended learning and how we can move forward (in the short-term and the long-term). Invited presentation at the 16th annual Sloan-C International Conference on Online Learning, Orlando, FL.
   Google Scholar
• Bernard, R. M. (2014). Things I have learned about meta-analysis since 1990: Reducing bias in search of “the big picture”. Canadian Journal of Learning and Technology, 40. Retrieved from http://www.cjlt.ca/index.php/cjlt/issue/current
   Google Scholar
• Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R., Surkes, M., & Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research, 79, 1243–1289. doi:10.3102/003465430933384410.3102/0034654309333844
   Web of Science ®Google Scholar
• *Bernard, R. M., Abrami, P. C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P. A., Fiset, M., & Huang, B. (2004). How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74, 379–439. doi:10.3102/0034654307400337910.3102/00346543074003379
   Web of Science ®Google Scholar
• *Bernard, R. M., Borokhovski, E., Schmid, R. F., Tamim, R. M., & Abrami, P. C. (2014). A meta-analysis of blended learning and technology use in higher education: From the general to the applied. Journal of Computing in Higher Education, 26, 87–122. doi:10.1007/s12528-013-9077-310.1007/s12528-013-9077-3
   Web of Science ®Google Scholar
• Bethel, E. C., & Bernard, R. M. (2010). Developments and trends in synthesizing diverse forms of evidence: Beyond comparisons between distance education and classroom instruction. Distance Education, 31, 231–256. doi:10.1080/01587919.2010.51395010.1080/01587919.2010.513950
   Web of Science ®Google Scholar
• Borokhovski, E., Tamim, R. M., Bernard, R. M., Abrami, P. C., & Sokolovskaya, A. (2012). Are contextual and designed student–student interaction treatments equally effective in distance education? Distance Education, 33, 311–329. doi:10.1080/01587919.2012.72316210.1080/01587919.2012.723162
   Web of Science ®Google Scholar
• Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. (2005). Comprehensive meta-analysis (Version 2) [Computer software]. Englewood, NJ: Biostat.
   Google Scholar
• Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). Introduction to meta-analysis. Chichester: Wiley.
   Google Scholar
• Campbell, D. T., & Stanley, J. (1963). Experimental and quasi-experimental designs for research. Chicago, IL: Rand-McNally.
   Google Scholar
• *Cavanaugh, C. S. (2001). The effectiveness of interactive distance education technologies in K-12 learning: A meta-analysis. International Journal of Educational Telecommunications, 7, 73–88. Retrieved from http://www.editlib.org/p/8461/
   Google Scholar
• *Cavanaugh, C. S., Gilan, K. J., Kromrey, J., Hess, M., & Blomeyer, R. (2004). The effects of distance education on K-12 student outcomes. Learning Point Associates/North Central Regional Educational Laboratory (NCREL), 39 Pages. Retrieved from http://files.eric.ed.gov/fulltext/ED489533.pdf
   Google Scholar
• Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53, 445–459. doi:10.3102/0034654305300444510.3102/00346543053004445
   Web of Science ®Google Scholar
• Clark, R. E. (1994). Media will never influence learning. Educational Technology Research and Development, 42, 21–29. doi:10.1007/BF0229908810.1007/BF02299088
   Web of Science ®Google Scholar
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
   Google Scholar
• Cook, D. A. (2009). The failure of e-learning research to inform educational practice, and what we can do about it. Medical Teacher, 31, 158–162. doi:10.1080/01421590802691393
   PubMed Web of Science ®Google Scholar
• *Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. M. (2008). Internet-based learning in the health professions: A meta-analysis. Journal of the American Medical Association, 300, 1181–1196. doi:10.1001/jama.2009.69
   Web of Science ®Google Scholar
• Cooper, H. M. (2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Thousand Oaks, CA: Sage.
   Google Scholar
• Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7–29. doi:10.1177/095679761350496610.1177/0956797613504966
   PubMed Web of Science ®Google Scholar
• Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463. doi:10.1111/j.0006-341X.2000.00455.x10.1111/j.0006-341X.2000.00455.x
   PubMed Web of Science ®Google Scholar
• Eysenck, H. J. (1978). An exercise in mega-silliness. American Psychologist, 33, 517. doi:10.1037/0003-066X.33.5.51710.1037/0003-066X.33.5.517.a
   Web of Science ®Google Scholar
• Eysenck, H. J. (1984). Meta-analysis: An abuse of research integration. The Journal of Special Education, 18, 41–59. doi:10.1177/00224669840180010610.1177/002246698401800106
   Web of Science ®Google Scholar
• Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8. doi:10.3102/0013189X00501000310.3102/0013189X005010003
   Google Scholar
• Glass, G. V. (1978). Integrating findings: The meta-analysis of research. In L. Shulman (Ed.), Review of research in education (Vol. 5, pp. 351–379). Itasca, IL: Peacock.
   Google Scholar
• Hammerstrøm, K., Wade, A., & Jørgensen, A. M. K. (2010). Searching for studies: A guide to information retrieval for Campbell Systematic Reviews (Supplement 1). Oslo: The Campbell Collaboration. doi:10.4073/csrs.2010.1. Retrieved from http://www.campbellcollaboration.org/resources/research/new_information_retrieval_guide.php
   Google Scholar
• Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York, NY: Routledge.
   Google Scholar
• Hedges, L. V., & Olkin, I. (1980). Vote counting methods in research synthesis. Psychological Bulletin, 88, 359–369.10.1037/0033-2909.88.2.359
   Web of Science ®Google Scholar
• Jackson, G. B. (1980). Methods for integrative reviews. Review of Educational Research, 50, 438–460. doi:10.3102/0034654305000343810.3102/00346543050003438
   Web of Science ®Google Scholar
• *Jahng, N., Krug, D., & Zhang, Z. (2007). Student achievement in online education compared to face-to-face education. European Journal of Open, Distance and E-Learning, 27. Retrieved from http://www.eurodl.org/
   Google Scholar
• Lefebvre, C., Manheimer, E., & Glanville, J. (2011). Chapter 6: Searching for studies. In J. P. T. Higgins & S. Green (Eds.), Cochrane handbook for systematic reviews of interventions (Version 5.1.0) [updated March 2011]. The Cochrane Collaboration. Retrieved from http://www.cochrane-handbook.org
   Google Scholar
• Lipsey, M. W. (2003). Those confounded moderators in meta-analysis: Good, bad, and ugly. The ANNALS of the American Academy of Political and Social Science, 587, 69–81. doi:10.1177/000271620225079110.1177/0002716202250791
   Web of Science ®Google Scholar
• Lou, Y., Bernard, R. M., & Abrami, P. C. (2006). Media and pedagogy in undergraduate distance education: A theory-based meta-analysis of empirical literature. Educational Technology Research and Development, 54, 141–176. doi:10.1007/s11423-006-8252-x10.1007/s11423-006-8252-x
   Web of Science ®Google Scholar
• *Machtmes, K., & Asher, J. W. (2000). A meta-analysis of the effectiveness of telecourses in distance education. American Journal of Distance Education, 14, 27–46. doi:10.1080/0892364000952704310.1080/08923640009527043
   Google Scholar
• *Means, B., Toyoma, Y., Murphy, R., & Bakia, M. (2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record, 115, 1–47. Retrieved from http://www.tcrecord.org
   Web of Science ®Google Scholar
• Pigott, T. D. (2013). Advances in meta-analysis. New York, NY: Springer.
   Google Scholar
• Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2005). Publication bias in meta-analysis—Prevention, assessment and adjustments. Chichester: Wiley.
   Google Scholar
• Russell, T. L. (2001). The no significant difference phenomenon: A comparative research annotated bibliography on technology for distance education (5th ed.). Montgomery, AL: The International Distance Education Certification.
   Google Scholar
• Schmid, R. F., Bernard, R. M., Borokhovski, E., Tamim, R. M., Abrami, P. C., Surkes, M. A., Wade, C. A., & Woods, J. (2014). The effects of technology use in postsecondary education: A meta-analysis of classroom applications. Computers & Education, 72, 271–291. doi:10.1016/j.compedu.2013.11.002
   Web of Science ®Google Scholar
• Shachar, M. (2002). Differences between traditional and distance learning academic outcomes: A meta-analytic approach (Unpublished doctoral dissertation). Truro University International, Los Angeles, CA.
   Google Scholar
• *Shachar, M., & Neumann, Y. (2003). Differences between traditional and distance education academic performances: A meta-analytic approach. International Review of Research in Open and Distance Education, 4. Retrieved from http://www.irrodl.org/content/v4.2/shacharneumann.html
   Google Scholar
• Simonson, M., Schlosser, C., & Orellana, A. (2011). Distance education research: A review of the literature. Journal of Computing in Higher Education, 23, 124–142. doi:10.1007/s12528-011-9045-810.1007/s12528-011-9045-8
   Web of Science ®Google Scholar
• *Sitzmann, T., Kraiger, K., Stewart, D., & Wisher, R. (2006). The comparative effectiveness of web-based and classroom instruction: A meta-analysis. Personnel Psychology, 59, 623–664. doi:10.1111/j.1744-6570.2006.00049.x10.1111/peps.2006.59.issue-3
   Web of Science ®Google Scholar
• Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research, 81, 4–28. doi:10.3102/003465431039336110.3102/0034654310393361
   Web of Science ®Google Scholar
• US Department of Education. (2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, DC: Author. Retrieved from http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf
   Google Scholar
• *Williams, S. L. (2006). The effectiveness of distance education in allied health science programs: A meta-analysis of outcomes. American Journal of Distance Education, 20, 127–141. Retrieved from http://www.10.1207/s15389286ajde2003_210.1207/s15389286ajde2003_2
   Google Scholar
• *Zhao, Y., Lei, J., Yan, B., Lai, C., & Tan, H. S. (2005). What makes the difference? A practical analysis of research on the effectiveness of distance education. Teachers College Record, 107, 1836–1884. Retrieved from http://www.tcrecord.org10.1111/tcre.2005.107.issue-8
   Web of Science ®Google Scholar