Detecting bias in meta-analyses of distance education research: big pictures we can rely on

Abstract

This article has two interrelated purposes. The first is to explain how various forms of bias, if introduced during any stage of a meta-analysis, can provide the consumer with a misimpression of the state of a research literature. Five of the most important bias-producing aspects of a meta-analysis are presented and discussed. Second, armed with this information, we examine 15 meta-analyses of the literatures of distance education (DE), online learning (OL), and blended learning (BL), conducted from 2000 to 2014, with the intention of assessing potential sources of bias in each. All of these meta-analyses address the question: “How do students taking courses through DE, OL, and BL compare to students engaged in pure classroom instruction in terms of learning achievement outcomes?” We argue that questions asked by primary researchers must change to reflect issues that will drive improvements in designing and implementing DE, OL, and BL courses.

Keywords:

• distance education
• blended and online learning
• meta-analysis
• quality of research synthesis

Acknowledgements

This study was supported in part by funds to Bernard, Abrami, and Schmid from the Social Sciences and Humanities Research Council of Canada.

Notes

1. For r, d and g effect size types, the superscript ⁺ denotes the average.

2. A Type I error occurs when the null hypothesis is rejected when it should be accepted. A Type II error occurs when the null hypothesis is accepted when it should be rejected.