Research limitations: the need for honesty and common sense

A quick look through the articles in this issue offers a handy instant view of the focus of current research into learning with technologies. Educational researchers are overwhelmingly keen on using technology to flip learning, to bring context into the classroom through Virtual and Augmented Reality, and to use enquiry or problem-based scenarios and game-based activities for collaborative and personal learning engagement. The concerns of teachers continue to include the development of critical thinking, the impact or not of learning preferences, cognitive load, peer assessment and what constitutes learner presence.

At a time when the Pandora’s Box of the world wide web is shaken by contrasting notions and implications of data privacy, it is not surprising to see Information Security, Data Management and Governance, and Data-Enabled Institutional Culture as three of the top ten IT issues for 2018 raised in Educause Review (Grajek, 2018). Leading educational researchers are aware that Higher Education is being radically reshaped by digital technology, and this journal can provide evidence of debates within Higher Education around the student’s and the organisation’s experience of digital enhancements for learning. This issue includes two calls for papers for future Special Issues relating to learning analytics and the use of “big data”, and the application of crowd-sourcing within education.

But as we press forward with research-led and practitioner-led exploratory and explanatory studies in this field, we should perhaps take a moment to consider how potentially misleading such studies may easily be. Otherwise, we may fall into the trap that Higher Education teachers constantly warn their students about: that of accepting data, especially self-report but also other quantitative data, at face value. While it is unusual for this journal to publish small studies founded on self-report data alone, particularly where a new application or course design has been tested on only one learner cohort, it is not difficult to find reviews of literature in papers which cite such work uncritically. A recent study of this behaviour in the tourism research field (Yüksel, 2017) found considerable response bias of varying kinds in the papers reviewed, but also found that such papers could find further airing through citation. It is incumbent upon journals such as this to emphasise the need for cautious statements where data may include bias, and for papers always to include a statement of limitations, whether related to research design, methodology, findings or the conclusions drawn. Such limitations may be included in the methodology section or the conclusions, but without them we cannot be sure that authors have genuinely had regard to potential areas of exclusion or bias which affect the results they report. It is unfortunately often the case that it falls to peer reviewers to point out the need for limitations to be included.

Limitations generally fall into some common categories, and in a sense we can make a checklist for authors here. Price and Murnan (2004) gave an excellent and detailed summary of possible research limitations in their editorial for the American Journal of Health Education. They discussed limitations affecting internal and external validity; internal validity relating to limitations of the study design and its internal integrity, external validity relating to the outward generalisability of the reported results. The construction of questions within a survey instrument, the layout and type of questions in surveys, the level of understanding of the respondents, an over-dependence on convenience sampling and general access difficulties, confounding variables, statistical inadequacies, low response rates, absence of or impossibility of control groups, these all offer potential limitations in educational research. But perhaps the most common issue for papers submitted to this journal is the “Hawthorne effect”, the notion that an innovation or intervention may produce positive results not related to the innovation but to the presence of an innovation and that the respondents are given the focus of attention in a research study. All such limitations and more should of course be wisely considered prior to the execution of research, but at very least must form part of the presentation of a study for publication.

This is not just about being self-critical or particularly humble in presenting our research. Identifying limitations, and explaining to the reader what impact these limitations have on the study results, not only demonstrates rigour but also gives the authors a chance to identify clear directions for future research. In fact this section or paragraph can be one of the most exciting parts to read in a paper, as we get a sense of the challenges faced in the study and the forward thinking about prospective improvements for further research which can be a rich seam of opportunity for a range of authors. As they mine this seam for research opportunities, we should also remember that we are discussing learning here. Despite the plethora of experimental and quasi-experimental designs we see, the learner often has only one “go” at learning at a time. If we start to experiment with live learner cohorts we can let them down. Researchers in the field of digital and interactive learning should perhaps take to heart some lessons from early medical research and experiment first on themselves. Learning communities of teachers, designers and trainers can espouse beta designs readily, and can pilot or pioneer new designs before they lead to serious limitations of applicability and generalizability.

The papers in this issue demonstrate clear and thoughtful understanding of research limitations, an attribute and research practice which we, as editors, strongly recommend to potential contributors to the journal.