What do we know about blended learning to inform police education? A rapid evidence assessment

ABSTRACT

This paper presents the findings of a rapid review of evidence on what works in blended learning in adult education which was undertaken to support the evidence informed introduction of blended learning in police education nationally across all police forces in England and Wales. Following a systematic search, screening, and quality assessment process, 42 studies, including 10 systematic reviews, were included in the final synthesis. Findings suggest that blended learning is at least as good as, if not more effective than, only face-to-face, or only online learning methods. The research evidence highlights the importance of bespoke design to suit learner needs and the type of content to be delivered, as well as the importance of providing appropriate support to both instructors and learners to engage with blended methods. Based on the findings, a toolkit is provided to guide police educators in designing and delivering curricula using blended methods.

KEYWORDS:

• Police education
• training
• blended learning
• evidence synthesis
• course design

Introduction

Recent developments in police recruitment for England and Wales (E&W) have compelled professional development units in police forces to rethink their recruit and in-service officer training capacity. The National Uplift Programme which envisages the recruitment of 20,000 additional officers over three years (2020 to 2023) combined with the roll-out of a graduate entry Police Educational Qualifications Framework (PEQF) in 2018 have demanded radical changes in police training. The move from focusing on traditional ‘training’ that emphasised military style drill, the memorisation of police powers and procedures, and the learning from field tutors on the job (Wood & Tong, 2009) to highlighting the need for ‘education’ which enables officers to face complex challenges of contemporary policing, has been slow but sure in coming. The focus on education implies developing officers’ critical thinking and problem solving skills, as well as integrating theory with practice, which is in common with some US approaches to training and education of police officers (Ramshaw & Soppitt, 2018). These series of changes to police training and education place greater reliance on online training methods as they are considered more resource effective and cost efficient (Cordner & Shain, 2011). Furthermore, these developments have provided the impetus for adopting more blended learning (BL) methods to cope with capacity issues, and as the next step in improving and expanding existing learning programmes in police organisations. The onset of lockdown restrictions following the pandemic in March 2020 compelled most police training to be moved online, almost overnight. Current research indicates that police training units and academies face many challenges in adopting these alternative modes of delivery (Belur et al., 2021; White et al., 2022). Recent years have been a time for great change in police forces as they have seen the opportunities and possibilities afforded by remote learning and there is greater appetite internationally to introduce new virtual and BL methods to police education.

Although police professional development units have been keen to introduce changes to revamp and modernise training content and methodologies, such efforts have often not been accompanied by an in-depth understanding of adult learning principles or assessment of learner needs (Donavant, 2009). Resulting efforts have been haphazard and ill thought-through, thus producing ‘change for the sake of change’ because of the gap between research scholarship and practice (Donavant, 2009). Attempts to evaluate whether online education works in a police professional development context have indicated that whilst some learners prefer online methods, others show a preference for traditional teaching; and effectiveness of training methods depends on learner type and their previous educational levels (Donavant, 2009). More recent efforts to evaluate specific police training delivered online have either found little significant difference between outcomes for online training as compared to classroom training (Anderle, 2018) or focused more on effectiveness of the content rather than the delivery method of the training (for e.g. Drew et al., 2021). However, there has been little or no systematic attempt to identify whether BL is in fact effective in enhancing police learning as compared to traditional learning methods, and what, if any, factors contribute to the success of BL for professional development.

The National Learning and Development programme in E&W to introduce BL is premised on the fundamental assumption that BL is as good as, or an improvement over, traditional training for achieving learning outcomes. It is also presumed to be more cost and resource effective. Furthermore, fiscal tightening means training is likely to be subject to more evaluation and cost benefit analysis than ever before (Cordner & Shain, 2011). For these two reasons, and in pursuance of an evidence-based approach, this rapid evidence assessment (REA) was undertaken to examine the existing evidence and explore whether BL would indeed be an effective approach for police education.

The paper is organised as follows: we begin by defining some of the key terms used. We then present the methods underpinning the REA, including research aims; search, screening, and coding strategies; the inclusion criteria; and quality assessment. We then present our findings on 1) comparing BL with other types of learning methods; and 2) synthesising elements of BL important for the learner and the instructor, and identifying promising aspects of effective course design and delivery. This is followed by a discussion section where we develop a toolkit to guide police educators and trainers in designing BL programmes for police officers based on our findings. Finally, we identify the limitations and conclusions of our study.

Defining key terms

A plethora of definitions for the terms virtual and BL exist. The global outbreak of COVID19 prompted many educational institutions and training providers to move teaching and learning to online platforms. However, this approach is not new and, for some time, has been used in various contexts including for police training in England and Wales. From the 1960s, digital technology emerged as ‘Computer Assisted Learning’ and ‘Computer Aided Instruction’ as ways to support more individualised learning. These approaches did not necessarily require the use of the internet, unlike more recently when broader terms such as ‘web-based,’ ‘online,’ ‘e-learning’ encompass any learning with at least some mediation by digital technology, and often assume use of the internet. There are many – somewhat ambiguous – terms associated with teaching and learning that is accessed and supported by the internet. Additionally, most traditional training programmes in professional or practice contexts or higher education today use some mix of face-to-face interaction and online elements. These are therefore, largely blended by default. In other words, the result of convenience or necessity, rather than from a carefully considered strategic vision for enhancing overall learning. There is, however, a growing body of literature focused on developing BL to support adult education and professional training programmes (c.f. Galvis, 2018; Mubayrik, 2018; Philipsen et al., 2019).

After considering these various interpretations of what virtual and blended learning mean, we think it essential to clarify some of the key terms used as follows -

Traditional face-to-face learning often envisages learning taking place in a classroom in a face-to-face situation, where content and learning material is disseminated by an instructor to the students. This allows for interaction between teacher and students, and between students amongst themselves. The term ‘virtual learning’ is an ambiguous one and is alternatively called, ‘online learning,’ ‘e-learning’, ‘digital learning’, or ‘distance learning’ to indicate that the tutor and learners are not physically in the same location. Virtual learning requires the learner to access content uploaded to a web-based platform sometimes referred to as a Virtual Learning Environment. It may include elements to enable students to ‘interact intuitively with the environment or objects within it’ and which has some elements of ‘actually being there or a feeling of presence’ (Wilson, 1997, pp. 1057–58), including video conferencing functionality which could be conceptualised as ‘virtual face-to-face.’ Thus, there are two aspects to virtual learning:

• Online synchronous – whereby learning takes place with the learner(s)/tutor or learner(s)/peer(s) are available at the same time and communicating via a digital forum. This might be facilitated by live video conferencing for a ‘virtual face-to-face’ experience and tools such as live chat functionality.

• Online asynchronous – whereby learning takes place at a time chosen by the learner who accesses materials and activities that have been pre-loaded on a platform or virtual learning environment (VLE). Interaction, if any, between learner(s) and tutors or with peers is not done in real time.

Amongst the many definitions for ‘blended learning’ (BL), we have adopted Staker and Horn’s (2012, p. 3) version of BL which refers to ‘a formal education programme in which a student learns at least in part through online delivery of content and instruction with some element of student control over time, place, path and/or pace; and at least in part at a supervised brick-and-mortar location away from home.’ Thus, for the purposes of this REA, BL incorporates, as part of a deliberate and considered strategy, both online and traditional in–person learning methods.

Accordingly, the REA is focused on comparing whether BL is more effective as compared to only face-to-face or only online learning.

Methods

A rapid review of existing evidence refers to a ‘timely rigorous review of the literature in order to make evidence-based recommendations’ and is increasingly becoming an acceptable alternative to systematic reviews, especially in time and resource restricted settings (Varker et al., 2015, p. 1199). It applies the same rigour in searching, locating, and synthesising literature as a systematic review, but is narrower in scope and breadth. Key for any REA are four pre-defined steps: (a) identifying a research question, (b) defining structured search and selection criteria and data sources, (c) setting and executing diligent filtering stages and ensuring inter-rater reliability, and (d) synthesising and analysing the findings (Berry et al., 2011). We outline steps taken to fulfil these stages.

Research aim

Research questions guiding this review were determined in consultation with the working group of police practitioners leading the national programme for introducing BL in police forces in E&W. Working group members were keen to understand whether BL would be as effective, if not more than, traditional learning methods. Thus, the key research question identified was: How does BL compare with traditional face-to-face learning and online learning? Further, practitioners were also interested in identifying best practice to make BL as effective as possible and to guide training and education departments in adopting it. Thus, a subsidiary research question emerged: What factors contribute to the success of BL?

The review aims to combine the findings of the two research questions to develop a toolkit to guide police educators to design and provide evidence informed BL programmes to both recruit and in-service police officers.

Search terms

The primary research question was deconstructed to define key and search terms. Scoping searches were used to further refine the relevant search terms that were considered suitable to identify studies related to the effectiveness of BL. These search terms were discussed with subject matter experts and underwent multiple reviews to refine the results. Three categories of search terms were used to cover the type of learning method, the learner, and the outcome measured.

Terms related to the type of training provided, including ‘virtual’, ‘hybrid’, ‘blended’, ‘e-learning’, ‘digital’ and ‘remote’ with ‘learning’, ‘education’, and ‘training’

AND

Terms related to the learner including terms such as ‘adult’, ‘professional’, or ‘student’

AND

Terms related to the outcome, including ‘satisfaction’, ‘behavior change’, ‘attitude change’, ‘skills enhancement’, or ‘upskill*’¹

To answer the subsidiary research question, we thematically analysed and synthesised the studies that met our inclusion criteria, so as to identify best practice and provide evidence informed recommendations for in-house police training and learning programmes.

Inclusion and exclusion criteria

Several inclusion and exclusion criteria were developed based on considerations of practical relevance to England and Wales (and by extension to policing in other similar contexts) and availability of resources. The results were screened against the following pre-set inclusion criteria:

• Time restrictions: Only studies published in or since 2016 were included to ensure practical relevance to BL as the use of digital technology is changing rapidly.

• Language restrictions: Only studies available in English were included.

• Geographic restrictions: Only studies conducted in contexts similar to the UK were included as they were considered relevant to education in this context. Thus, studies conducted in the USA, Canada, UK, Europe, Australia, and New Zealand were considered relevant to the review.

Additional selection criteria were focused on reporting empirical evidence that spoke to our research questions. They were:

• Studies must report empirical evaluations.

• Studies could adopt either qualitative, quantitative, or mixed methodologies.

• Randomised controlled trials (RCTs), quasi-experimental, or before-and-after research designs were acceptable.

• Studies should compare two or more types of learning methods or formats of courses using the same learning method.

• Studies must report a measured outcome, including student satisfaction, student engagement, or knowledge gain.

• Studies must be focused on adult learning.

Studies were excluded only if they were considered unsuitable for, or irrelevant to, police education. Thus, studies solely focused on measuring the impact either of specific technological tools or software, or a learning method specific to a particular theoretical or technical subject, and those using social media as a teaching tool, were excluded.

The search process

Searches were carried out across five databases, namely Scopus, Web of Science, the British Education Index, ERIC (Education Research Information Center), and ACM Digital Library. In addition, the British Library EThOS and Advance Higher Education databases were searched for grey literature.² Furthermore, nine studies that met our inclusion criteria from a previous systematic map (Dryer-Beers et al., 2020) were included. The sifting process was managed using the EPPI Reviewer 4 software.³

Three researchers screened the 4511 studies returned in the initial searches, on title and abstract. At the outset, inter-rater reliability (IRR) was measured where three coders screened the same randomly selected 100 studies. Agreement between the three coders was found to be around 70% in this early stage, which was considered less than acceptable. Disagreements were resolved through discussion. At this point the code book was refined. Coders met regularly to clarify doubts and improve shared understanding of the inclusion criteria. Following the IRR test coding, equal numbers of studies were randomly allocated to the three researchers for screening on title and abstract. The final list of studies that met the inclusion criteria were equally distributed in the randomly allocated lots screened by the three researchers. This indicated acceptable IRR at the screening stage.

Following the first round of screening on title and abstract 769 studies met the inclusion criteria, The scope of the review was further refined to only include studies that compared two or more types of learning methods or compared different course designs using the same learning method. Studies were again randomly reallocated to the three coders and the refined inclusion criteria was applied. A total of 115 studies were found eligible for coding on full text. Two researchers then screened the full texts and ultimately 91 studies⁴ (of which 10 studies were systematic reviews or meta-analyses) were deemed relevant. Any disagreements or doubts were resolved through regular discussions throughout the process.

Quality assessment

All 81 studies reporting primary evaluations were quality assessed by two researchers. The first 10 studies were jointly assessed by the researchers to establish shared understanding of the tool and assessment criteria. A bespoke quality assessment (QA) tool developed earlier for a similar study (see, Belur et al., 2020) was used. Studies were given quality ratings based on three QA tools⁵ to rate mixed methods, qualitative and quantitative studies and were rated as weak, moderate, or strong according to the scoring system laid down by each tool. The tools were combined to create a bespoke instrument using MS Excel with standardised data input and logic rules to automate the scoring process.⁶ Thereupon, eight studies got a strong score, 23 received moderate, and 50 studies got weak scores. As a result, the REA reports findings from a total of 41 studies⁷ – including 31 studies scoring strong or moderate and 10 systematic reviews or meta-analyses (Figure 1 below).

Figure 1. PRISMA diagramme of the sifting process.

Synthesis approach

Since the included studies covered a thematically and methodologically highly diverse spectrum, they were considered too heterogeneous in terms of outcomes measured to conduct a meta-analysis. Instead, the review followed the analysis methods discussed by Thomas and Harden (2008) whereby findings were synthesised in three stages: firstly, selected studies were coded and relevant information extracted, then studies were clustered in descriptive themes, and finally, analytical categories were constructed based on emerging themes (Hoon, 2013; Thomas & Harden, 2008).

Studies were synthesised narratively to answer both research questions. The first research question was subdivided into three parts – comparing BL with traditional face-to-face learning; BL with virtual learning; and comparing virtual and traditional face-to-face learning methods. In answering the second research question recurring themes related to improving engagement, student satisfaction, and learning outcomes were identified as important. The step of creating distinct analytical categories and ‘going beyond’ the content of the original studies (Britten et al., 2002) was at times rather difficult, as many of the themes/outcomes were highly interconnected and influenced more than one of the intended outcomes of improving engagement, satisfaction, and learning. (Thomas & Harden, 2008). For example, social presence of the instructor may be important for student engagement and may be equally important for achieving learning outcomes.

These thematic findings were combined to highlight some guiding principles underlying BL design factors to potentially enhance learning outcomes. Thus, we focused on identifying personal factors that related to the learner and instructor, and pedagogic factors that related to the design of the learning programmes. Five themes identified in the synthesis reported below were reconfigured in the discussion section with the intention of informing how BL could enhance police education.

Results

Of the 41 studies included in the REA, eight studies (including two systematic reviews) compared BL to face-to-face learning; one study compared BL to virtual methods; 10 studies (including two systematic reviews) compared virtual learning to face-to-face learning; five studies (including two systematic reviews) compared BL with both virtual and face-to-face learning. The remaining studies focused on different aspects of a single method of learning – mainly different types of digital delivery (9 studies including one systematic review) as well as blended or flipped learning⁸ (three studies including one systematic review).

Of all included studies 20 studies (50%) were located in the USA, three studies (~7%) were in the UK, twelve studies (28.5%) were from European countries and six studies (~14%) from other countries (Australia, Canada, and one systematic review from China⁹).

A breakdown of the research designs used by the studies indicated that 10 of the 41 studies (23%) were systematic reviews or meta-analyses; 18 studies (42%) used a quasi-experimental design; eight studies (19%) were RCTs; five studies (11%) used a before-and-after design. Almost all the studies were focused on university courses (with a few associated with a professional degree) with only one exception of a course that was industry based (see, Beinicke & Bipp, 2018).

RQ 1: How does blended learning compare with traditional face-to-face learning and online learning?

The effectiveness of BL as compared to other standalone learning methods was measured at various levels in increasing order of impact – learner satisfaction, learner engagement, knowledge gain, and behaviour change (Kirkpatrick & Kirkpatrick, 2006). We found that most studies reported on one or more of the first three levels of learning. It is presumed that higher learner satisfaction will lead to greater engagement with both, the learning process and taught materials, thereby leading to better learning outcomes for students.

Although this might be pedagogically sound and make logical sense, our study did not find that this link was automatic or necessarily positively associated. Some studies reported higher student satisfaction with a teaching method, but it was not necessarily associated with higher learning as measured by exam grades (e.g. Ebner & Gegenfurtner, 2019). Similarly, some studies indicated that although students did not feel confident in their subjective measurement of success at the end of the online course, assessment of learning over a period of 8 to 10 weeks indicated it was just as effective as that of students in face-to-face classrooms (e.g. Beinicke & Bipp, 2018).

BL compared with traditional face-to-face learning

Eight studies compared BL with traditional face-to-face learning methods. The evidence regarding this was mixed – with four studies, (including three systematic reviews) finding BL to be more effective than traditional face-to-face methods (Webster et al., 2020; Liu et al., 2016; Vo et al., 2017; Bolsen et al., 2016)) for knowledge gain; one study finding it marginally less effective than face-to-face teaching (Monk et al., 2020); two studies finding that students performed equally as well as traditional learning (Littenberg-Tobias & Reich, 2020; Weightman et al., 2017); and one study reporting that BL was better than face-to-face learning in terms of long term memory retention but less so as compared to digital learning alone (Michael & Michael, 2019).

Additionally, the number of successful completions in the BL format was higher than the traditional format (Webster et al., 2020) mainly because it allowed students to learn at their own pace. Further, BL was found more advantageous for female students and led to much improved scores for STEM disciplines over their male counterparts and the non-STEM subjects (Webster et al., 2020; Liu et al., 2016). In contrast, Monk et al. (2020) found that male students tended to do better in BL settings as compared to face-to-face settings. The explanation for this apparent contradiction can be located in the description or type of blended setting since the BL course where females did better than male learners was described as being collaborative and encouraged group working which is said to suit the temperament of females more (Webster et al., 2020). On the other hand, the blended part of the course that Monk et al. (2020) evaluated which differed from the fully face-to-face format, consisted of asynchronous individual learning online and not collaborative tasks. Thus, the design and structure of the course plays a very important role in how the course is perceived and the impact it has on learners. Other reasons for why BL might have an advantage include the ability for students to review material available online flexibly and multiple times (Liu et al., 2016); or because it optimally blends both kinds of learning modalities to suit the requirements of different students (Michael & Michael, 2019).

Contrarily, Monk et al. (2020) found that the average marks for the same course in a BL class were lower compared to its traditional counterpart, but not significantly so. However, students preferred the face-to-face format compared to the blended one mainly because they perceived classrooms to be better for asking questions and receiving instant responses from the instructors, as well as, being a less distracting environment than being online (Monk et al., 2020).

BL compared with online learning

Only one study compared BL directly with online learning (Philipsen et al., 2019) but four others (including a systematic review and a meta-analysis) included a comparison of BL with online as well as traditional classroom teaching – in essence, comparing all three types of learning formats.

Two studies found no difference between BL and online learning in terms of learning outcomes (Philipsen et al., 2019; Weightman et al., 2017), and three studies identified the superiority of one form of BL – flipped instruction (which they considered to be blended even when delivered purely online or only face-to-face) over either of the standalone formats (Liu et al., 2016; Bolsen et al., 2016; Michael & Michael, 2019)

The only study that compared blended courses with online flipped learning concluded that students who were previously used to attending face-to-face sessions were less accepting of the online aspect of the course, but there was overall no difference in the learning outcomes between the two types of delivery (Philipsen et al., 2019). The study concluded that regardless of the medium – online or face-to-face, – it was the flipped aspect of learning that was most effective.

Similarly, Weightman et al. (2017) found no statistically significant difference in terms of student learning outcomes between all three formats of learning. Furthermore, 14 of 19 studies included in the systematic review reported no student preference for a particular format and the remaining studies reported advantages and disadvantages of various formats without students expressing a preferred option. The authors conclude that consequently, the choice of format is totally dependent on educators and based on personal preference or contextual conditions.

Liu et al. (2016) on the other hand found that BL resulted in better outcomes than either online or face-to-face learning methods by themselves. They suggest one of the reasons might be because BL students are more engaged and less likely to experience isolation or reduced interest in the topic. Bolsen et al. (2016) found that the BL courses resulted in the best outcomes for students in terms of course content and knowledge as compared to the other two formats. However, their findings suggest that although overall BL and online learning were superior to traditional face-to-face learning, there was a greater drop-out rate in the online only learning format. Michael and Michael (2019) found that each learning method had a different impact on learner memory, with online learning being most suited for long term memory.

Face-to-face compared with online learning

A total of ten studies (including one systematic review and one meta-analysis) compared traditional face-to-face learning with online learning and five studies (including one systematic review and one meta-analysis) compared face-to-face with online as well as some form of BL.

Although several studies found that there was little difference between learning outcomes for the face-to-face and online modalities (Lucero et al., 2017; Van Der Beek et al., 2020; Stöhr et al., 2016; Butz & Stupnisky, 2016; Soper, 2017; Ebner & Gegenfurtner, 2019; Gegenfurtner & Ebner, 2019; Weightman et al., 2017), the findings were a bit more nuanced when the subject matter, the type of learner, and student satisfaction or preferences were considered.

Three studies claimed that the learning outcomes were better for students in face-to-face contexts as compared to fully online courses (Stephan et al., 2019; Callister & Love, 2016; Mitra & Beenen, 2019). Additionally, some studies reported students preferred face-to-face teaching over the online format (Stephan et al., 2019; Ebner & Gegenfurtner, 2019; Gegenfurtner & Ebner, 2019).

Although only two studies claimed that online learning led to better learning outcomes for students as compared to traditional face-to-face teaching (Beinicke & Bipp, 2018; Bolsen et al., 2016), some studies reported advantages associated with online learning, in that it was more cost and time effective (Soper, 2017) and could, in some cases, use off the shelf online materials (Bolsen et al., 2016). It needs to be acknowledged that although moving learning online might be cost effective for the educational institution in the long run, costs to students for procuring appropriate computer equipment and internet connection need to be accounted for. Online learning might also involve hidden costs to students if they do not have appropriate study spaces. However, other studies seemed to suggest that online training can only be effective if properly resourced and supported (Stephan et al., 2019; Bolsen et al., 2016; Butz & Stupnisky, 2016; Stöhr et al., 2016), and incorporated all the elements of good face-to-face teaching (Lockman & Schirmer, 2020; Van Der Beek et al., 2020).

Whether instructors should choose face-to-face or online methods ought to depend on the kind of learner and the type of content being taught. Digital methods were found to be more suitable for students with high self-efficacy, autonomy and self-regulation (Beinicke & Bipp, 2018; Bolsen et al., 2016; Lucero et al., 2017; Mitra & Beenen, 2019; Stöhr et al., 2016). Furthermore, online or virtual learning methods were better suited to deliver knowledge that was to be retained over long term or for what they term as factual or declarative knowledge (Beinicke & Bipp, 2018; Michael & Michael, 2019). Alternatively, face-to-face methods were especially suited for students with low self-efficacy to begin with (Lucero et al., 2017) and for topics that required greater interaction with the instructor (Callister & Love, 2016), as well as for teaching procedural knowledge or application of theory (Beinicke & Bipp, 2018).

The evidence indicates that there is no unequivocal superiority of one learning method over the others in terms of knowledge acquisition. It leads us to concur with Lockman and Schirmer (2020) who found that learning outcomes in all three methods are roughly similar, therefore, the focus should be on identifying whether specific practices or elements are more productive. Similarly, Beinicke and Bipp (2018, p. 524) conclude that, ‘it is not the training setting or delivery media per se that leads to higher levels of trainings success in the long run but in order to be effective, trainings and their theoretical framework should be designed taking especially the type of learning content … [and we add – type of learner] into account’. Thus, the discussion section will focus on identifying those aspects of all three types of learning methods that might possibly be useful for, and relevant to, designing a BL approach for police education.

RQ 2: What factors contribute to the success of blended learning?

The evidence indicates that the effectiveness of all learning methods,, including blended learning, is dependent on good course design which caters to different types of learners, and when appropriate support is provided to tutors. We therefore highlight some implications for the design of BL for police forces that relate to meeting the needs of diverse learners, ways to support and upskill instructors, and course design considerations before, during, and after delivery.

Learners

Learner motivation is important and affects learners’ preferences for traditional, online, or blended options. Studies concluded that learners with high confidence in their own ability to manage their learning, and those who have an orientation towards reflection and experimentation, may choose and perform better in fully online learning environments. Mitra and Beenen (2019) found, for example, that learners with more intrinsic motivation had a higher level of interest in the content of a course, than the more performance-oriented learners who chose a traditional face-to-face mode of learning. Both extrinsic and intrinsic motivations play a role in shaping learning. For example, learners with a performance/goal orientation are motivated by the desire to get good grades and to compare well to their peers. Performance can be more obvious in the face-to-face environment, which might explain the preference for this mode for learners with this orientation. Learners who have an orientation towards mastery (i.e. mastering the subject) are more concerned with doing the best they can. Being less concerned with performance in relation to others may explain why they are happier with an online learning environment where they have more obvious responsibility for their own learning. As Mitra and Beenen (2019) conclude, taking different types of motivation into account in course design is important. Their results also suggest that incorporating strategies for increasing intrinsic motivation, and mastery, rather than performance orientation (i.e. getting learners to think less about how well they are doing and more about the learning itself) will help learners benefit more from a virtual learning environment.

Linked to motivation are learner self-efficacy and self-regulation. Self-efficacy, which indicates judgements about one’s own abilities to complete a task and to what level of competence, correlates strongly with learning (see, Beinicke & Bipp, 2018 for a discussion of relevant literature). Learners with high self-efficacy and the ability to self-regulate their learning cope well with the independence and flexibility of virtual learning, particularly with asynchronous tasks. In fact, some of the studies suggest that these learners may choose a fully online learning option because they already have confidence in their own abilities to manage their learning (e.g. Lucero et al., 2017; Mitra & Beenen, 2019). There are also examples of these students feeling more anxious or angry in the online environment as it takes more effort and time to do the work (Stephan et al, 2019). It is crucially important to support learners to develop self-regulation (Beinicke & Bipp, 2018). Where scaffolding and support for learners to develop these skills is not included, there is an inbuilt risk that the more independent learners do better, and less confident learners do worse within the BL environment (see, Stöhr et al., 2016).

When designing BL courses, learning and development practitioners therefore need to consider their learners’ motivations, levels of self-efficacy, and ability to regulate their own learning. It is important also to note that where learners have moved from face-to-face to online, there is less acceptance of this format (Phillips & O’flaherty, 2019). Thus, there is some merit in either setting expectations or introducing virtual elements of a BL course early on to gain acceptance for this mode of learning. Recent findings seem to suggest that when confronted with the inevitability of virtual learning as in the context of the pandemic, students are more amenable to virtual as compared to face-to-face learning (e.g. Mali & Lim, 2021).

There are many types of support that instructors or tutors can provide for BL. For example, the evidence shows that regular responses and messages from tutors build the confidence of learners who prefer tutor – learner interaction. A clear, published timetable for activities, with deadlines, plus websites or virtual environments that are easy to use and understand, along with a predictable pattern to the design of the programme, all help learners manage their learning (see, Webster et al., 2020). Clear timescales for activities act as formative assessment opportunities, which can have a positive impact on final assessment; however, whilst given timescales might avoid learner procrastination, they can also reduce learner control (Ee et al., 2018). Explaining the reason for a type of activity, and why an activity may be offered in a virtual rather than face-to-face form, also helps learners understand what type of engagement is expected of them (see, Betihavas et al.’s (2016) discussion of preparation for the flipped classroom). Further, Monk et al. (2020) report that learners felt the need for more self-regulation and time management strategies on a course where virtual activities were purely asynchronous online activities with no opportunities for collaboration. They concluded that although a blended approach is as successful as face-to-face, this only works if the online activities are well designed and relevant.

Although learners may start with different levels of self-efficacy, a good course will increase all learners’ ability to regulate their own learning, so that they are better prepared for a professional life where ongoing learning is required. The evidence shows that a judicious blend of face-to-face and virtual can help achieve this. For example, Webster et al. (2020) report that better outcomes are achieved when a mix of pre-recorded or pre-written materials (which students can access independently and at their own pace) are combined with opportunities for peer learning and interaction with tutors in a face-to-face environment.

Opportunities to build relationships and develop social dimensions of learning is also an important factor. Incorporating virtual informal spaces where students can socialise and collaborate for group activities also support learning, particularly as they highlight the value of peer learning (Butz & Stupnisky, 2016). The presence of the tutor online supports the development of a sense of community amongst learners (Cho & Tobias, 2016).

Instructors

As with learners, instructors have different preferences and skills (Michael & Michael, 2019), and this preference may well influence the efficacy of a particular mode of teaching and learning. However, given that there is evidence of the value of a blended approach for learners through activities such as flipped lectures (Goedhart et al., 2019), supporting instructors or tutors to develop their skills and open their minds up to non-traditional methods can have long term benefits for learners. Where a decision has been made to move to BL, then supporting tutors to develop skills for online teaching, as Phillips and O’flaherty (2019) recommend, and helping instructors develop a more learner-focused, mentoring type approach would be beneficial. Tutors may also find that moving to activities such as a flipped classroom, takes some of the pressure off, allowing them to be more responsive to learners (Goedhart et al., 2019). Webster et al. (2020) also note that the inclusion of the face-to-face interaction with the tutor and group work can be important as part of the blended approach but require the tutors to set up in-class activities to facilitate interaction between learners amongst themselves and with instructors. They must not only mediate and provide feedback on student contributions where necessary, but also be prepared to relinquish control over the activities. Admittedly, relinquishing control might be more difficult for culturally resistant police trainers, nonetheless, if changes introduced via blended learning are to be effective, it would require a change in mindset and an open approach on the part of instructors.

Course design – preparatory stage

In thinking about course design, police educators need to consider ways to prepare learners for the blended experience. In contexts where learners are used to and expect face-to-face learning, making explicit the expectations and requirements of a blended environment is important. Learners need to spend their time on the learning and not on trying to make sense of a virtual platform or struggling with a piece of software. As with all course design, constructive alignment – where learning outcomes, learning activities, and assessment align (Biggs, 2003) – should govern design. Learners need to understand the links between the activities they are engaged in and the learning outcomes and assessment for the course. According to Cho and Tobias (2016), merely including discussion spaces in a BL environment and hoping this will achieve learning is not sufficient if it is not clear how the discussion links to the learning outcomes. For example, discussion makes greater sense when learners are being asked to ‘co-create’ and collaborate as a specific learning outcome, whilst also receiving appropriate and timely feedback from the instructors.

Managing expectations early on so that learners are not disappointed is also important for their motivation and ongoing engagement. This is especially true for a flipped classroom model, which requires learners to be primed in advance since it involves pre-class activities completed at the learner’s own pace to engender deeper learning. The combination of self-regulated activities completed prior to in-class activities and usually involving peer learning can help improve the quality of study (Goedhart et al., 2019).

IT preparedness was identified as one of the most important issues for both instructors and learners by several studies which considered access to, and skills in using the technology, as being crucial to the success of BL (see, Mitra & Beenen, 2019; Van Der Beek et al., 2020). Evidence indicates learners will struggle if they are not supported to learn how to use the software and virtual learning platforms that are required for a particular course. Familiarity, and early introduction to virtual elements on a course are important (Phillips & O’flaherty, 2019). Similarly, tutors need support and training to understand the functionalities of software and platforms to be able to design courses more effectively. Preparation for specific pedagogical models, such as the flipped classroom, can be more resource intensive, with tutors having to prepare pre-class materials, and facilitate in-class interactive activities. However, since resources can be re-used, this initial time investment may need to be repeated only periodically when materials need updating (Betihavas et al., 2016).

Course design – during delivery stage

The evidence also illustrates the importance of considering the implementation phase of a learning programme to increase student engagement, satisfaction, and learning.

The timing and mode of tutor interactions with learners, particularly in the virtual environment, both asynchronously and synchronously, are important. Learners need regular messaging and input from tutors, whether this be to help them understand an activity, prompting them to participate (see, Abbott & Provident, 2016), or to comment on their learning. The studies illustrate that instructors do not necessarily have to provide a lot of input into students’ discussions to ensure learning, but learners need to feel that the instructor is present and engaged (see for e.g. Cho & Tobias, 2016). Building a sense of an online community and generating opportunities for student interaction is key to engagement (see, Callister & Love, 2016).

The studies show peer engagement can be achieved, for example, with the use of games either as a type of warm-up exercise for learning, or as a more focused learning activity (see, Fu et al., 2016; Karay et al., 2020). Similarly mixing virtual practice opportunities, such as simulations (see, Mattila et al., 2020), or virtual internships (Theelen et al., 2020), with other learning in a BL environment can improve both confidence and preparedness for the real professional environment. The simulation of the authentic professional environment where learners can see consequences for actions, provides learners with a more realistic sense of what they will face in their professional roles (Theelen et al., 2020). Choice of activity and how it is managed is key to course design and needs to link to the type of content being learned.

Van Der Beek et al. (2020) demonstrate how moving from course content focused on declarative knowledge (i.e. factual content) to procedural knowledge (i.e. how to implement that knowledge) is important in courses designed for professional education. Beinicke and Bipp (2018) suggest that learning procedural knowledge (i.e. applying declarative knowledge), for a professional setting, will require trying out realistic scenarios, being able to interact in real time with colleagues and tutors, making mistakes and trying again, all of which is arguably easier within a face-to-face environment. An example of this is illustrated in Callister and Love’s (2016) study where learners developed negotiation skills more effectively in the face-to-face environment. Undeniably, there are a number of areas of police training, for example, self-defence skills or use of force training, which would necessarily require a substantial element of face-to-face learning.

Course design – follow up and evaluation stage

The success of a learning programme depends primarily on meeting the learning outcomes, which might be greater if the learner is both engaged and satisfied with the learning programme. However, engagement and satisfaction are not only instrumental to learning, but can be ends in themselves as a measure of the success of a course. Furthermore, although engagement, satisfaction, and learning are linked, they are not necessarily mutually dependent. It is possible that a learner is not fully satisfied, i.e. has not enjoyed all aspects of the learning experience or the mode of learning, but has learned a great deal and succeeded in the final assessments (see, Betihavas et al., 2016 for an example of this in relation to the flipped classroom model). There is a balance therefore to be struck between trying to improve engagement, and feelings of satisfaction, but with the ultimate priority of inculcating the necessary knowledge, understanding, and skills. Feelings of satisfaction might be negatively influenced by the increased workload from groupwork and virtual learning activities that require self-regulation, but still have positive overall learning outcomes (Betihavas et al., 2016). Conversely, Lee et al. (2016) reports higher satisfaction amongst learners on a course who were given the opportunity to choose how to learn and decide in what order to complete activities. This did not necessarily lead to better learning outcomes but did increase satisfaction with the learning experience.

Furthermore, the way assessments are designed can also lead to improving learning outcomes. This refers to both formative and summative assessments. For example, Karay et al.’s (2020) study reports that when learners can be encouraged to see formative testing as a learning experience, and if these tests are designed to be done at the learner’s convenience, they have a greater impact on learning. On the other hand, Ee et al. (2018) suggest that giving learners a flexible window within which to complete assessments, but with a deadline, helps to keep learners on track, whilst developing self-regulation, skills that would be immensely relevant to police officers in managing their routine work commitments on the job in the long run.

Discussion

The evidence indicates that all three types of learning approaches – face-to-face, virtual, and BL are similarly effective in achieving knowledge acquisition. But studies also show that a blended approach can lead to improved outcomes for learners (e.g. Beinicke & Bipp, 2018; Bolsen et al., 2016; Liu et al., 2016; Vo et al., 2017; Webster et al., 2020), as the BL environment can provide the optimum conditions for the different types of learning required. The move towards improving education and training as part of the police professionalisation agenda envisages officers being qualified at a graduate level in England and Wales. Consequently, police education units are looking farther afield for good practice in higher education as well as other professions. Thus, the findings from this REA, which rely mainly on evidence from the higher education sector, are relevant for adopting blended approaches to the education of recruits and in-service officers.

A concerted national effort in England and Wales to support and encourage a BL approach towards recruit and in-service police officers’ education, as part of the wider professionalization agenda, predates the pandemic. It was intended to address two important issues – (1) the delivery of more effective training which is also cost and resource efficient, and (2) coping with the enhanced training requirements following on from the introduction of the PEQF and the National Uplift Programme to recruit more officers. The impact of the lock down restrictions during the pandemic in 2020 made it necessary for organisations to move all or most of their training online and demonstrated that it is possible to do so, thus fuelling a greater appetite for adopting BL methods moving forward.

A note of caution must be introduced, if police organisations adopt BL by default, which means randomly assigning parts of the curriculum to virtual or face-to-face sessions more out of necessity (as in the case of the pandemic) and/ or convenience (saving resources, time and costs), the results may not be significant (Donavant, 2009). The true impact of BL, as the evidence presented above shows, emerges from good design and delivery. However, merely stating this does not provide any direct guidance on how course designers might begin making decisions about making BL more effective.

The findings of the REA indicate that four factors ought to be considered before decisions are made about which methods of learning would be most appropriate to deliver what type of content and for which type of learner. They depend on the answers to a series of questions

• What is the type of knowledge being delivered?

• What is the purpose of the learning?

• Does the content require memorisation, assimilation, or application? and finally,

• What are the needs of the learners?

Thus, decisions have to be made about whether the content is declarative (i.e. facts and concepts which need to be remembered and understood) or procedural (i.e. application of theory which need analysis, evaluation and problem solving skills) and whether the learner has self-efficacy (can learn autonomously) or requires a great deal of support from the instructor (needs interaction to learn). Based on the finding from the wider evidence we have come up with a simple initial set of questions that might help police educators to make preliminary decisions about how a blended approach might work. This simple tool (which is not exhaustive) considers the content of the course, the purpose of the knowledge transfer and what tools are required to deliver the highest learning impact, whilst considering the type of learner involved.

We hasten to add that this tool comes with a caveat that it is purely a starting point for decision making for in-house police education units. Taking appropriate design decisions would additionally involve wider discussion and collaboration between subject matter experts, pedagogy experts, technical designers, and the learners themselves (Figure 2).

Figure 2. Toolkit to guide BL course design for police educators.

In making decisions about the course content, most police education units in E&W are familiar with Bloom’s Taxonomy,¹⁰ where the lower levels of the pyramid (remember, understand) could roughly equate to declarative knowledge and the higher levels (evaluate, create) refer to procedural knowledge aspects of the course. Interim levels (apply and analyse) span both kinds of knowledge. Similarly, it would be useful if training and education units invested in enquiring into the type of learner, their motivation, and their needs prior to designing a course. Additionally, the design of a course is more nuanced than simply putting a swathe of learning material online for students to consume at their own pace followed by face-to-face sessions to discuss cases or conduct reflective practice exercises. It requires appropriate scaffolding at each level of the learning process so that a truly blended approach will support online learning with the help of focused tutorials and recorded face-to-face sessions. Relevant learning material could be made available online to students to go over as many times as they require to consolidate learning. Providing appropriate instructor support at each level is key to the success of a BL approach.

A visualisation of the resulting architecture of a BL programme by design that incorporates Bloom’s taxonomy and the findings from the literature are distilled very simplistically in Figure 3 below as a first step towards designing BL programmes such that the sum is greater than the individual learning approaches so that synergies produce better outcomes.

Figure 3. Architecture of BL approach using Bloom’s Taxonomy

There are six important points to note. Firstly, it is possible that students might prefer face-to-face over BL and this would affect their satisfaction with BL methods and their evaluation of the BL programme. However, evidence indicates that there is more learning gain over a longer period within a BL environment, perhaps because students can revisit online materials numerous times, if necessary, to consolidate learning over time. Therefore, course designers need to think about how and when they measure learning and give feedback to learners.

Secondly, given not all learners are equally equipped to cope with the self-regulation demands of the virtual elements of a course, there can be greater variation in individual learning outcomes for learners without appropriate support.

Thirdly, although moving substantial sections of the training on to virtual platforms might be cost, time, and resource effective in the long run, it requires adequate resourcing up front to design a course. This includes investment in the necessary technology, and the provision of necessary support for individual instructors and learners.

Fourthly, it is important for education units to ensure that they provide adequate opportunities for officers (especially recruit officers) to socialise and be exposed to the positive aspects of police culture and working practices to ensure learners do not suffer the same degree of isolation from other learners as in a purely online format.

Fifthly, the design of assessments needs to align, as stated above, with the learning outcomes and the learning activities. Thus, assessment design in the BL environment needs to consider how co-creation and collaborative activities are reflected in assessments of learning.

Finally, it is important that police education units have a well-developed evaluation plan in place to assess the impact of BL and to ensure that they are collecting feedback from tutors and learners to inform the next round of learning activity.

Limitations

Although the search terms uncovered a substantial number of studies, there were few high-quality studies to make up the evidence base on the topic. We suggest two explanations for why this might be so – firstly, the poor-quality rating to many studies that met our inclusion criteria (50 out of 82) is an artefact of the Quality Assessment Tool we were using – which may not have been suitable for assessing, especially the quantitative studies, in the domain of educational research. Alternatively, there might be a dearth of methodologically sound studies because of practical and ethical issues with conducting experimental or quasi experimental studies in the field of education more broadly. Perhaps researchers are thus limited in their ability to adopt a rigorous methodology for the purposes of building an evidence base. Furthermore, it is possible that qualitative studies that did not score highly when quality assessed, might have had some important insights into student experience or teacher perspectives, which had to unfortunately be excluded.

Secondly, there might have been other relevant studies uncovered, had we completed forward and backward searches of the bibliography of included studies, however, due to time and resource constraints this could not be done.

Finally, the REA was specifically aimed at informing the BL programme for police organisations and therefore only what we considered relevant evidence has been analysed in detail and this has been a subjective decision.

Conclusion

In conclusion, the evidence indicates that by simply adopting a BL approach to replace traditional teaching methods will not guarantee better learning outcomes or greater satisfaction for the officer learners. Instead, the adoption of BL offers the opportunity for improved learning and a better experience for learners, provided it is adapted to suit individual learner requirements and the purpose of the learning. There is a need for education units in police forces to understand the principles of good BL course design. This requires them to: understand the needs of their learners up front; gauge their level of self-efficacy prior to commencing the training programme; set their expectations about the modalities of the learning programme early on; equip them with adequate tools and skills to work in a BL environment; pay careful attention to the design of the course and the assessment of outcomes; and finally, provide adequate opportunities for peer interaction and appropriate feedback from instructors throughout the learning process.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

Project funded by the National Police Chiefs’ Council, UK [National Police Chief’s Council Police Uplift Programme project 26a];

Notes on contributors

Jyoti Belur

Jyoti Belur qualified in Economics at the University of Mumbai (India) where she worked as a lecturer before joining the Indian Police Service and serving as a senior police officer. She has postgraduate degrees in Police Management from Osmania University (India) and in Human Rights from the University of Essex (UK) and is a Fellow of the Higher Education Academy (FHEA). Her PhD thesis was completed at the London School of Economics (UK). Now an Associate Professor, she has undertaken research for the UK Home Office, College of Policing, ESRC and the Metropolitan Police Service. Jyoti is the programme convenor for the BSc in Professional Policing. She currently also serves in an advisory capacity on the Her Majesty’s Inspectorate of Constabulary, Fire and Rescue Services (HMICFRS) and is a member of the London Policing Ethics Panel. Her research interests include policing, police training and education, systematic reviews and evaluations, and violence against women and children.

Helen Glasspoole-Bird

Helen Glasspoole-Bird is a research associate at the University of York where she is part of a team undertaking a review of evidence on primary science teaching in England. Her background is in Psychology (BSc, Liverpool) and primary teaching (PGCE, Cambridge). Since 2002, Helen has worked in Higher Education, first as a lecturer on undergraduate and postgraduate programmes for Initial Teacher Education and Education Studies and more recently in research posts. Leading on from an MA in Education (Oxford Brookes), Helen’s PhD (UCL, IOE) focuses on anticipatory and emerging professional identities of teachers, nurses and police officers to analyse the ways in which individuals manage anxiety associated with their development as frontline professionals. Her research interests focus on: formation of self and identity; transitions as liminal spaces; qualitative research methods; and reflection and reflexivity in education.

Clare Bentall

Clare Bentall is a Lecturer in Education at UCL Institute of Education, where she has experience of designing and running online MA programmes. She has over 20 years’ experience in training other educators, particularly within higher and professional education and is a Senior Fellow of the Higher Education Academy (SFHEA). She is also Associate Director of the Development Education Research Centre, and editor the International Journal for Development Education and Global Learning. Clare also does a range of consultancy work within clinical education, facilitating workshops for clinical educators and advising on the development of national curricula for various aspects of healthcare education.

Julian Laufs

Julian Laufs is a currently a PhD candidate at the UCL Jill Dando Institute of Security and Crime Science where he focuses on the prevention of future crime in smart cities. He also researches the usefulness and cost of smart crime prevention technologies as well their impact on police demand dynamics. His other research interests include innovation in policing and national security.

Notes

1. ? and * are Boolean operators to cover all spellings and endings for the chosen terms.

2. Grey literature is assumed to be not formally published, has limited distribution, and is not easily accessible but inclusion of which helps guard against publication bias (Hopewell et al., 2005)

3. The EPPI Reviewer software is a specialised software for managing systematic reviews and other types of syntheses (see – https://eppi.ioe.ac.uk/CMS/Default.aspx?alias=eppi.ioe.ac.uk/cms/er4&)

4. Contact lead author for full list of studies that were quality assessed.

5. The Mixed Methods Appraisal Tool (Hong et al., 2018) for appraising mixed-methods studies; the Critical Appraisal Skills Programme Qualitative Checklist (CASP, 2017) for appraising qualitative studies, and the Effective Public Health Practice Project Quality Assessment Tool for appraising quantitative studies (EPHPP, 2017). The bespoke instrument is available from the authors on request.

6. A different scoring system to that recommended by authors was implemented for use with both the CASP and MMAT tools. For details see, (Belur et al., 2020).

7. See, Appendix A for full list of scored studies.

8. Flipped learning reverses the traditional teaching model of lecture in class followed by assignments to be completed afterwards. It requires proactive engagement from both, the learners to familiarise themselves with the taught material online in a self-directed manner, and the instructor, to design learning activities to engage the students and consolidate learning in the classroom (Brewer & Movahedazarhouligh, 2018).

9. This study was an outlier, since the systematic review included studies conducted from countries of interest.

10. For details on Blooms Taxonomy see https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy/

Appendix A

List of included studies (n = 41)

Table

No.	Author	Year	Description of course/courses being evaluated	Type of study	QA score
(1)	Abbott & Provident	2016	Different types of digital delivery	Before and after	Medium
(2)	Alessio et al	2018	Different types of digital delivery	RCT	Strong
(3)	Asiri et al.	2019	Different types of digital delivery	RCT	Medium
(4)	Beinicke & Bipp	2018	Virtual vs. face-to-face	RCT	Strong
(5)	Benson et al. a	2019	Academic integrity in e- learning module	Before and after	Medium
(6)	Betihavas et al.	2016	Flipped classroom	Systematic review	NA
(7)	Bolsen et al.	2016	Virtual vs face-to-face vs blended	Quasi-experimental	Medium
(8)	Brasier et al.	2019	Different types of digital delivery	RCT	Medium
(9)	Butz & Stupnisky	2016	Virtual vs. face-to-face	Quasi-experimental	Strong
(10)	Callister & Love	2016	Virtual vs. face-to-face	Quasi-experimental	Medium
(11)	Chen et al. a	2018	Various learning tools for computer studies	Meta-analysis	NA
(12)	Chen Fei et al.	2017	Flipped classroom	Systematic review	NA
(13)	Cho & Tobias	2016	Different types of digital delivery	Quasi-experimental	Medium
(14)	de Hei et al. a	2020	Intercultural competence in classrooms	Before and after	Medium
(15)	Dousay & Trujillo	2019	Different types of digital delivery	Quasi-experimental	Strong
(16)	Ebner & Gegenfurtner b	2019	Virtual vs. face-to-face	Meta-analysis	NA
(17)	Ee et al.	2018	Different types of blended delivery	Quasi-experimental	Medium
(18)	Fu et al.	2016	Game based learning	Systematic review	NA
(19)	Gegenfurtner & Ebnerb	2019	Virtual vs. face-to-face	Systematic review	NA
(20)	Goedhart et al.	2019	Flipped classroom	Before and after	Medium
(21)	Kapp et al	2020	Different types of digital delivery	Quasi-experimental	Strong
(22)	Karay et al.	2020	Mobile testing for formative assessments	RCT	Medium
(23)	Lee et al.	2016	Different types of digital delivery	RCT	Medium
(24)	Liu et al.	2016	Virtual vs face-to-face vs blended	Meta-analysis	NA
(25)	Littenberg-Tobias & Reich	2020	Blended vs. face-to-face	Quasi-experimental	Medium
(26)	Lockman & Schirmer	2020	Different types of digital delivery	Systematic review	NA
(27)	Lucero et al.	2017	Virtual vs face-to-face	Quasi-experimental	Medium
(28)	Mattila et al.	2020	Online simulation tool	Before and after	Strong
(29)	Michael & Michael	2019	Virtual vs face-to-face vs blended (flipped)	Quasi-experimental	Medium
(30)	Mitra & Beenen	2019	Virtual vs. face-to-face	Quasi-experimental	Strong
(31)	Monk et al.	2020	Blended vs. face-to-face	Quasi-experimental	Strong
(32)	Park & Kim a	2016	Virtual tutee system for academic learning	Quasi-experimental	Medium
(33)	Phillips & O’flaherty	2019	Virtual vs blended	Quasi-experimental	Medium
(34)	Soper	2017	Virtual vs. face-to-face vs. self-study	RCT	Medium
(35)	Stephan et al.	2019	Virtual vs. face-to-face	Quasi-experimental	Medium
(36)	Stöhr et al.	2016	Virtual vs. face-to-face	Quasi-experimental	Medium
(37)	Theelen et al	2020	Virtual internships	Before and after	Medium
(38)	Van Der Beek et al.	2020	Virtual vs. face-to-face	RCT	Medium
(39)	Vo et al.	2017	Blended vs. face-to-face	Systematic review	NA
(40)	Webster et al.	2020	Blended vs. face-to-face	Quasi-experimental	Medium
(41)	Weightman et al.	2017	Virtual vs face-to-face vs blended	Systematic review	NA

^a Studies were coded but findings were not relevant to the research questions.

^b Studies were basically reporting findings based on the same data.