Teachers’ experiences of monitoring their students in online higher education: recommendations for course design and opportunities for learning analytics

ABSTRACT

Owing to the worldwide pandemic, use of technology and online education has increased. Studies into teachers' experiences in Higher Education indicated that teachers find it hard to monitor their students' progress during online education. Adequate teacher monitoring is essential, since it allows teachers to adapt their teaching strategies to student needs. Therefore, it was investigated what monitoring strategies teachers employed in online education and what challenges they experienced. Interviews were held with 10 teachers. The results showed that teachers primarily monitored students during online guided sessions. A difficulty was the lack of non-verbal and other observational cues that are normally available. To deal with this challenge, teachers created explicit monitoring opportunities by adjusting their course design. In the discussion, recommendations are given to stimulate teacher monitoring, and the potential role of automated analyses (learning analytics) to aid teacher monitoring is discussed.

KEYWORDS:

• Blended learning
• online education
• teacher monitoring
• course design
• learning analytics

Introduction

Technology has evolved into an integral part of how we organise our educational systems, acting as a tool to support or enhance learning and teaching processes (Mayer, 2019). Owing to the worldwide pandemic, technology has taken on an even bigger role, as many teachers in Higher Education (HE) have had to shift to providing their courses in a fully online fashion (UN, 2020). While the pandemic is (hopefully) a temporary crisis, it has urged and stimulated developments in HE concerning use of technology that are likely to last to some degree, even when the pandemic is over. It is therefore worthwhile to consider what lessons we can learn from this crisis and what (online) education will look like in the future.

In this article, we focus specifically on the pedagogical model known as the flipped classroom (Staker & Horn, 2012) in HE, which is the combination of preparatory self-study activities and teacher-guided face-to-face interactive sessions. To ensure effective support, teachers need to monitor their students closely to get a good understanding of what support students need (Lajoie, 2005). From the earliest studies appearing about the pandemic, one of the challenges teachers report is that it is hard for them to monitor their students, harder than it was in the physical context (Van der Spoel et al., 2020). This raises the question how teachers monitor their students in the face-to-face context, why monitoring is more difficult in the online context, and how teachers have coped with these difficulties.

In this study we delve into these questions further. The transition to online education offers not only the opportunity to examine how teachers adapt their practice to the new situation, but also to shed new light on how teachers’ practice was shaped in the face-to-face context. The first goal of this study is to understand in more detail the monitoring strategies teachers use in HE. The second goal is to infer best practices from teachers on how to ensure proper monitoring and support of their students in the context of online education. The third goal is to look ahead and to formulate directions for future research on how teachers can be supported in the monitoring process.

Theoretical background

The role of the teacher in the flipped classroom

Blended learning (BL) is the term for instructional models in which face-to-face education is combined with online educational activities (Staker & Horn, 2012). A specific implementation of BL is what is called the flipped classroom (Staker & Horn, 2012, p. 10). In this model, online self-study activities are combined with face-to-face teacher-guided activities. The self-study activities are realised through technological tools, for example by having students watch instructional videos. The teacher-guided sessions take place in a physical/face-to-face context. Thus, education is ‘flipped’ in the sense that knowledge transfer performed in traditional lectures is replaced by self-study, thereby freeing time for the teacher and students to focus on higher-order processing activities in the teacher-guided sessions.

Teachers' ability to provide support to students during guided sessions is essential for student learning (Gillies et al., 2008; Hattie & Timperley, 2007). Increasingly more attention is given in the literature to how teachers make decisions about the support they offer to students (Loibl et al., 2020). Offering support broadly consists of two steps (Lajoie, 2005). First, teachers engage in monitoring to estimate how their students are doing. This phase is called teacher monitoring, and is defined in this article as teachers’ assessment of students for the purpose of making informed decisions about what support to provide to students (Lajoie, 2005). Teachers may use different sources of information, or cues (Loibl et al., 2020), to make inferences about how their students are doing. The second step is that teachers decide what support students need at a particular moment, for example in terms of assigning particular tasks, providing specific instructions or choosing collaboration partners for students. Effective support is given when the support matches the need of the student (Lajoie, 2005). It is therefore essential that teacher monitoring of their students results in an accurate depiction of students’ current level, collaboration skills, or any other aspect of student behaviour on which the teacher wants to provide support.

Given the importance of teacher monitoring, it is surprising that so few studies have investigated teacher monitoring practices in the flipped classroom. Reviews do show that the flipped classroom indeed enables teachers to spend more time interacting with students (Al-Samarraie et al., 2020; Lundin et al., 2018), that the flipped classroom requires other skills from teachers including offering personalised support during sessions (Brewer & Movahedazarhouligh, 2018), and that the flipped classroom leads to higher learning outcomes as long as the face-to-face time between teachers and students is not reduced (Van Alten et al., 2019). Thus, the teacher–student interaction is an essential element of the flipped classroom. However, to the best of my knowledge there are no detailed accounts of how teachers monitor their students and how they use insights from monitoring to provide personalised attention to students in the flipped classroom.

In theoretical terms, teachers have opportunities to monitor their students in the flipped classroom via two pathways (Kaliisa et al., 2020). First, teachers may monitor the preparatory self-study activities. Because those activities often take place online, information is available to teachers about those activities, for example how many students have watched a web lecture or which parts of a video are watched more often than others (Rubio-Fernández et al., 2019; Van Leeuwen, 2019). This information could enable decisions such as how to structure the guided sessions and which students to set up for collaboration (Van Leeuwen, 2019).

The second pathway for monitoring concerns the guided sessions. During those sessions, teachers may obtain information about their students by observing or interacting with students (Hargreaves, 2005; Lundin et al., 2018; Van Geel et al., 2018). This information can be used for on-the-spot decisions such as what specific instructions to provide to students, and whether to end or initiate new activities. Figure 1 displays the two monitoring pathways.

Figure 1. Overview of two monitoring pathways.

Monitoring in times of the pandemic

Because of the pandemic, teachers in HE have had to shift their courses to the fully online context, including those courses that make use of a flipped classroom design. In the research community, there is large interest in teachers’ experiences with transitioning to online education to understand how teachers have coped with this change (e.g. Ferdig et al., 2020; Marek et al., 2021; Van der Spoel et al., 2020). One of the recurring findings that teachers report is that it was hard for them to monitor their students. Van der Spoel et al. (2020) for example reported that teachers experienced ‘very strongly’ that it was more difficult ‘to monitor students’ well-being and learning processes, causing students to disappear off the radar’ (p. 629). Although there has thus been an increase in attention for teacher monitoring owing to the shift to online education, there have not yet been detailed investigations of how teachers monitored their students during fully online education and how it relates to the two pathways described above. Therefore, in this article an attempt is try to make sense of teachers’ experience with online education in terms of the theory described above, namely the distinction between monitoring and supporting, and the two pathways available for monitoring.

From that theoretical perspective, a possible explanation for why teachers find it more difficult to monitor their students in online education could be that these difficulties arise because of a difference in the availability of cues to base monitoring on. Teacher monitoring of their students might also be less accurate because face-to-face guided sessions are replaced by online guided sessions in which interaction tends to be more delayed and more superficial (Gloria & Uttal, 2020; Van der Spoel et al., 2020). It could also be that teachers have shifted their monitoring strategies in terms of the two pathways available to them. Teachers may rely more on pathway 1 of monitoring the preparatory activities to adapt to the challenges of online teaching.

So, an important question is how monitoring students in the fully online context differs from monitoring students in a face-to-face context. Given that the pandemic is not over yet and that the role of technology is likely to stay at an increased level after the pandemic, it is worthwhile to understand what monitoring challenges teachers have experienced and how they have dealt with these challenges. By answering these questions, best practices can be formulated for teachers in how to shape their monitoring process, and monitoring challenges can be identified that need further attention or support in future education.

The present study

To summarise, the flipped classroom model consists of the combination of self-study activities and teacher-guided sessions. Teacher–student interaction during the guided sessions is an integral element in stimulating student learning, and it is therefore essential that teachers accurately monitor students’ needs for support. In Higher Education, there are typically less student–teacher contact hours than for example in primary or secondary education. Making good use of those hours is therefore important, and teacher monitoring of students is an essential component of that.

Models of teacher decision making describe that teachers can make use of different monitoring strategies and different informational cues. As education has shifted to a fully online context, monitoring has become a more complex process and the question is why this is the case. The goal of this article is to shed more light on this question by investigating what cues teachers use in online education in terms of the two monitoring pathways available to them, what challenges teachers experienced concerning monitoring, and how these compare to their experiences in face-to-face education. Therefore, the following research questions were formulated. Concerning online education in which self-study activities and guided sessions are employed:

1. Which monitoring strategies do teachers use in terms of the two monitoring pathways and what specific cues do they use?

2. What monitoring challenges do teachers experience when they compare their online experiences with the face-to-face setting?

Answering these questions is relevant for three reasons. First, it would provide more insight into the monitoring process that teachers employ in HE. Second, best practices can be distilled for teachers on how to effectively monitor their students in online education. Finally, it would allow us to formulate directions for future research based on the unresolved monitoring challenges that teachers report.

Methods

Design and participants

Sample sizes in qualitative research should be large enough to achieve saturation, which means that the collected data provide all relevant insights and further data collection will not yield new insights (Onwuegbuzie & Leech, 2007; Saunders et al., 2018). At the same time, the sample should be small enough to allow for a thorough analysis. Following these guidelines, we selected 10 Higher Education teachers to participate in this study (Onwuegbuzie & Leech, 2007).

Teachers were selected who had experience with face-to-face education as well as experience with online education during the pandemic starting in 2020. Interviewees were selected by first emailing department leaders to ask which teachers they thought were suitable candidates; i.e. who they knew had relevant experience and the relevant skills to reflect on their experiences. Those teachers were asked to participate and recruitment was stopped once 10 interviewees were found. The sample consisted of six female and four male teachers. Their average age was 40.5 years (SD = 6.9). On average, they had 10.7 years of teaching experience (SD = 6.8). The teachers represented several faculties within the University: five from Social Sciences, three from Medicine, one from Science and one from Law. All participants signed informed consent forms for their answers to be used for scientific purposes. They did not receive any compensation for participating. The study was approved by the ethics committee under file number 21–0307.

Interview instrument and procedure

The interview session was structured along the themes posed in the research questions and consisted of two main parts. In part 1, the focus was on RQ1 – What aspects do teachers monitor in terms of the two monitoring pathways? Teachers were first asked about the importance they attached to monitoring in their teaching practice and how in their practice they balanced monitoring in between and during sessions.

They were then asked, both for during sessions and self-study activities, what typical study activities they employed in their course design and how they monitored them. They were prompted to explain what aspects they monitor during those activities and what specific information (cues) they used to monitor those aspects. For example, if a teacher indicated they monitor students’ content understanding, the teacher was asked from what specific cues this could be inferred. During this process, teachers were naturally inclined to compare their monitoring strategies to those employed in the face-to-face context. They were free to elaborate on this aspect, as it partly answered research question 2. In general, though, this first part of the interview was relatively structured.

Part 2 of the interviews focused on RQ2 – challenges related to monitoring. Teachers were asked (even if they had already made some remarks about this) how their monitoring strategies differed from the face-to-face context, what information about their students they would like to have most that they currently did not (if they could have access to any information they wanted), what the biggest monitoring challenge was, and what lessons they had learnt from the transition to online education. This second part of the interview was thus semi-structured. The main topics were addressed for every participant, but time was available to ask follow-up questions and for the teachers to raise additional topics.

All in all, the interviews took between 60 and 75 minutes. Owing to Covid restrictions, the interviews were conducted online. Participants received an information and consent letter a week prior to the interview. At the start of the interview, the purpose of the interview was again briefly explained. The interviews were audio-recorded so that the interviewer could focus on the conversation and transcribe it at a later point. To increase validity of the interviews, a member check was performed. Transcripts of the interviews were sent to the participants, who had the opportunity to make changes in case of incorrect or incomplete wording. After analyses of the results (see below), a member debriefing also took place. A report with a summary of the findings was provided to all participants to provide them insight into the interpretation of the findings before work on the current article started.

Analyses research question 1 – systematic approach

For research question 1, a structured approach was used in which the participants’ answers were transferred to a table to denote the aspects teachers monitor in blended courses and which specific cues they use for those aspects (following Miles & Huberman, 1994). One table was created for pathway 1 (monitoring outside of guided sessions) and one for pathway 2 (monitoring within guided sessions). For every aspect that the teacher indicated monitoring in their teaching practice, a new data row was created in the table listing the aspect the teacher monitors and what specific cues the teacher used. A teacher could name multiple cues for one aspect.

After all answers were listed for all participants, the table was ordered according to monitoring aspects. All named cues belonging to that aspect were summarised. The result was thus a table with one column containing the aspects teacher monitor, and in the second column all the information or cues teachers mentioned to use for each of these aspects.

Analyses research question 2 – grounded theory approach

For research question 2, an explorative, narrative approach was used. All the comments teachers made in all their answers were coded for aspects that made monitoring easier or more challenging, also in comparison to the physical context. Following a grounded theory approach, coding occurred in three steps (Boeije, 2010; Locke, 2002).

First, all comments went through open coding by assigning all relevant utterances a descriptive code. To stay close to the data, Boeije (2020) recommended performing this phase in collaboration to ensure reliability. Therefore, open coding was performed together with a research assistant. Discussion occurred about interpretation of the interview transcripts, and when views differed, agreement was reached after discussion. In total, 487 utterances were coded. Data saturation occurred at the seventh transcript. From that point on, the already created open codes were sufficient to code the remaining part of the data.

After open coding, the author and a research assistant together categorised the codes into overarching categories, also called axial coding. This was done by first collecting the codes that denoted the exact same theme (when two open codes were voiced slightly differently) and then putting the codes together that were thematically related. The first round of discussing the open codes resulted in 29 axial codes.

In the final phase of selective coding, the core challenges and themes associated with monitoring were selected by relating the axial codes to each other and trying to find thematical relations between them. At this point we used Figure 1 (that denotes the blended learning design and the two pathways for monitoring) to further synthesise the codes into higher-level categories. Guiding questions in this process were for example How are the various codes related? Under which circumstances does monitoring occur or not? What is most important for understanding the teachers’ experiences? (Boeije, 2010, p. 116). This process resulted in seven main themes.

Results

Research question 1: monitoring strategies

Balance between monitoring in between and during sessions

Teachers were first asked about the two monitoring pathways and how teachers balance between the two in online education. Of the 10 teachers, five indicated that they primarily monitor their students during guided sessions (pathway 2), two indicated they primarily monitor the preparatory activities (pathway 1), two indicated it was divided evenly over the two pathways, and one teacher indicated it differs according to the course she teaches.

Monitoring cues

Table 1 displays the results for monitoring pathway 1, and Table 2 for pathway 2. The tables display which aspect teachers monitor, how many teachers mentioned each aspect and what cues teachers used to assess each specific aspect. The number of mentioned aspects and cues is bigger for pathway 2 than for pathway 1, which is in line with 5 out of 10 teachers indicating pathway 2 as their primary source for monitoring.

Table 1. Teacher monitoring for pathway 1 – monitoring preparatory activities and in between sessions.

Aspect of student activity teachers monitor	Number of teachers that mentioned this aspect	Total mentions	Monitoring cues (frequency of mentioning)
Content understanding	10	20	Reading student work (13), What score students obtain on a quiz (4)
Participation	5	6	Did student watch web lecture (2), Did student submit assignment (2), When has student submitted the assignment (1)
Language use	3	3	Reading student work (3)
Motivation	1	2	Does student ask question via email (1) or discussion board (1)
Intelligence	2	2	Reading student work (2)
Regulation	1	1	When has student submitted the assignment (1)
Self-efficacy	1	1	Type of question student asks in between sessions (1)
Well-being	1	1	Messages from students in between sessions (1)
Total		36

Table 2. Teacher monitoring for pathway 2 – concerning guided sessions.

Aspect of student activity teachers monitor	Number of teachers that mentioned this aspect	Total mentions	Monitoring cues
Content understanding	9	34	Observe/listen to student discussions (12), Ask student questions in conversations/discussions (10), The type of question students ask (3), Ask student to contact the teacher in case of questions (3), Have students create a padlet (2), Use poll or quiz (2), Ask student to share screen (1)
Participation	8	27	Whether students actively participate in activity (5), What student is doing – for example using telephone, eating (3), The type of contributions students make in the activity (3), Whether students answer or pass on a question (2), Whether students ask questions (2), Ask students whether they are present (2), Whether camera is turned on (2), Non-verbal behaviour via camera (1), Students’ status displayed in their profile (1), Whether students arrive on time (1), Whether students engage in discussion with each other (1), Non-verbal signs of engagement (1), Whether student prepared the activity (1)
Interest	6	16	Whether students discuss with each other (4), Whether students ask questions (3), Ask students about their interest (3), Whether student participates in the activity (3), Students explicitly indicate their interest (1), Use a poll (1), Whether students are prepared (1).
Motivation	7	11	Whether students actively participate (3), Non-verbal signals via camera (2), Does student ask questions during meeting (2), Does student stay after the meeting to ask questions (1), Ask students about their motivation (1), enthusiasm in tone of voice (1), Do students engage in discussion with each other (1)
Student well-being	3	6	Ask students how they are doing (2), Ask via a poll (1) or by displaying an emoticon (2), Students’ home environment (via camera) (1)
Regulation	5	5	Students explicitly mention difficulties (2), Ask students about regulation strategies (1), the type of questions students ask (1), Whether students submitted the assignment (1)
Organizational matters	3	3	Students ask questions (1), Ask students about the course setup (1), Students explicitly mention difficulties (1)
Self-efficacy	2	2	Students explicitly mention difficulties (2)
Intelligence	1	2	The type of answers students give (1) and the type of questions they ask (1)
Total		106

Looking at Table 1, the primary aspect that teachers monitor concerning the preparatory activities, and which was mentioned by all teachers, is students’ content understanding. Teachers do so primarily by reading the work that students submit on the preparatory activities. Many of the teachers for example use a program called FeedbackFruits (2021), which allows teachers to upload instructional videos with embedded questions for students to answer. Those answers can be read by the teacher in preparation for the guided session.

Another frequently mentioned aspect that teachers monitor is student participation, which is monitored by checking whether assignments were completed or submitted or not or by checking whether an activity such as watching a video was completed. The other aspects were only mentioned by one or a couple of teachers.

Looking at Table 2, we again see students’ content understanding and student participation as the two most often-mentioned aspects that teachers monitor. For content understanding, monitoring occurs by listening to student discussions in small groups and by the teacher asking student questions during content discussions. Thus, interaction that occurs during the online guided sessions is an essential part of monitoring. For student participation, teachers mention not only whether students are active, but also a number of ‘visual cues’ that teachers obtain via students’ cameras.

Two other aspects that are mentioned by more than half of the teachers are student interest and student motivation. The type of cues that are used for these aspects overlap, and they overlap with the cues for student participation as well. This shows that one and the same behaviour can serve as a cue for multiple aspects at the same time.

Another finding that becomes apparent from the cues listed in Table 2 is that teachers not only observe student behaviour, but also that teachers actively seek the input from students to obtain information. For almost all aspects, there is a cue included related to the teacher outright asking the student about that aspect.

Research question 2: monitoring challenges

Teachers were also asked about the challenges associated with monitoring. Figure 2 displays the seven core themes that were found in the interviews, placed onto the core elements of a blended learning design. Table 3 displays the frequency of occurrence of each of the themes. Themes 1 through 5 relate to monitoring pathway 2 (monitoring during guided sessions). Theme 6 related to monitoring pathway 1 (monitoring the preparatory activities). Theme 7 is related to the overarching level of online education.

Figure 2. Overview of the seven core themes that were found in the interviews.

Table 3. Frequency of occurrence for each theme.

Theme	Number of utterances	Percentage of utterances
T1	25	5.1
T2	123	25.3
T3	84	17.3
T4	40	8.2
T5	133	27.3
T6	27	5.5
T7	55	11.3
Total	487	100.0

Light grey boxes indicate themes are closely related to each other. Themes 1 and 2 are related because they concern the types of cues that are available to teachers in face-to-face and online meetings. Themes 3 and 4 are both about the consequences of the difficulties of monitoring. Theme 5 is connected with an arrow to Themes 1 through 4 because it represents the core strategy that teachers used to deal with the challenges of online monitoring.

Theme 1: the power of face-to-face observation

When teachers reflected on how they monitored their students in online meetings, they inevitably made the comparison to how they do so in face-to-face meetings. The first theme therefore concerns the unique opportunities that face-to-face meetings offer for monitoring. Online monitoring by no means comes close to what a teacher is able to discern about students from walking around in a classroom. There are three main features that lead to such high-quality monitoring.

The first is the teacher’s ability to walk around and to use all their senses: teachers can see all their students at once, can hear students talking and can hear multiple groups at once, and there is almost a magical ability to see how someone is doing just by seeing that person in a classroom. P1 remarks: ‘During the guided sessions I get a feeling for how students are doing’. The teacher can also see the students’ desk and what they are doing, i.e. are they working on the assignment, taking notes, paying attention?

The second feature of face-to-face meetings is the opportunity for smooth two-way interaction. Talking and listening to students is a major source of monitoring and occurs much more smoothly than it does online. Also, being on campus offer the possibility for teachers and students to walk up to each other to discuss one-on-one.

The third feature of face-to-face meetings has to do with a physical dimension that transcends the two aspects above. There is an added dimension to being together in the same room; a sense of togetherness that stimulates both students and teacher that adds to the teacher’s ability to monitor what is going on. P8 puts it as: ‘I get more energy from it [face-to-face meetings], it is more personal, and it surpasses the formal learning goals’.

Theme 2: grasping in the dark in online sessions

The second theme is the direct counterpart for theme 1, and concerns the difficulties associated with monitoring students during online sessions. This theme occurred in about 25% of all coded utterances and was therefore one of the most discussed themes.

The first difficulty with online monitoring is the absence of cues that enables high-quality monitoring. Students are visible to teachers as separate small screens, and students are usually asked to turn their microphone off when they are not speaking. Teachers can only to a small degree make use of non-verbal signals, are often unable to monitor facial expressions and cannot see what students are really doing. Compared to the richness and level of detail that teachers usually observe, monitoring students online is experienced as a poor alternative.

The second difficulty is that interaction in online sessions is of a different nature than in face-to-face meetings. Plenary conversations are more formal, and students tend to be more hesitant to speak up. Because of having to turn microphones on and off and having to regulate turn taking, the pace of conversation is slower. All of these factors result in less in-depth discussions, which constitute a major source of monitoring for teachers. There is also a new dimension that needs to be navigated, namely that you are entering each other’s homes through the webcam. Making remarks about a poster or pet in the background was mentioned as a way to start small talk, but not all students are comfortable with that.

To summarise, online monitoring takes up more time and results in less breadth and quality of monitoring than face-to-face meetings. P5 remarks that there is a serious danger of students becoming ‘small icons on your screen instead of human beings’.

Theme 3: shifting roles and responsibilities

Theme 3 deals with the consequences of the lack of monitoring cues for how the roles and responsibilities are divided in online education. This theme was also in the top three of most mentioned themes. For the teachers, the first shift is that they felt responsible for monitoring students’ well-being more closely because of the pandemic. Time taken up by talking about the pandemic took away time that would otherwise be spent on monitoring and supporting content understanding. The decreased ability to monitor students had a number of detrimental consequences. For example, teachers reported that, because of the uncertainty of the situation, they acted less proactively in addressing students, they provided less elaborate explanations because they got less direct feedback from students, and teachers acted more on their own ideas of what they thought could be useful instead of on student input. In short, because of the lack of monitoring cues there was less personalised teaching that is at the heart of quality education.

Teachers also indicated they tended to rely more on the students’ own responsibility to monitor themselves. P3 for example states that she urged her students to contact her when something was the matter, either during or after the online meeting. However, as discussed above, teachers simultaneously experienced that students are often hesitant to ask for help. Especially the students who are in the ‘average level’ and who could truly benefit from teacher support tended to disappear from the teacher’s radar. P6 explains: ‘students at the lower end who get into trouble will ask you for help. But the average student who thinks he is doing well but who could improve does not ask me any questions. It is very hard to reach those students’.

Theme 4: general teaching challenges

Besides the challenges outlined in theme 3 that are connected to the online context, there are also challenges that teachers already dealt with in face-to-face meetings, which were enlarged in the online setting. The most prominent issue is balancing between monitoring at the group and individual student level: it is a challenge for teachers to obtain an accurate idea of how every individual student is doing and to provide every student with individual attention or feedback. This challenge is enlarged in the online setting. Concerning monitoring, having a group in front of you in a classroom allows you to observe the whole group at once, whereas in the online context it is very difficult to look at the camera screens of many students at the same time. Concerning teaching, it is harder to address individual students because you cannot simply walk up to them. In the online context you need to approach individual students outside of the guided session. P2 explains: ‘I only explain something [during the guided session] when I think it is relevant for the whole group. So I treat them more like a group, which I do not want’.

Theme 5: creating monitoring opportunities through course design

Theme 5 deals with the relation between monitoring and course design and is the most frequently occurring theme in the interviews. The teachers soon found out during the transition to online education that they would not be able to monitor their students well unless they explicitly created monitoring opportunities. The majority of teachers indicated that small-scale, interactive work forms during the guided sessions would be their best opportunity for monitoring, as it would allow them to observe student discussions in depth. Also, in small groups it is easier to observe students’ facial expressions and to have smoother conversations, as everyone can leave their microphone on. In the MS Teams environment that was used by the teachers, small groups were created by using the break-out groups facility, which lets students meet in several sub-channels within the plenary meeting. Teachers were able to move between these sub-channels and thus to listen in and interact with each group to some extent. Teachers also created explicit ways to gather information. They for example assigned one student the chairing role per group, whose task it was to report back to the teacher what was discussed. Some teachers also had students work on a concrete product such as a shared text document, so that the teacher could monitor that product instead of the group’s discussion.

Outside of the collaborative assignments, teachers created new ways of monitoring as well. Teachers for example made use of short polls during plenary discussions or asked students to post an emoticon in the chat to indicate how they were doing. Some teachers created an additional 10-minute meeting prior to the official session in which students could come in for informal talk. Some teachers sent messages via the Teams chat functionality to individual students. One teacher even phoned all students to check how they were doing. A danger of using all these explicit methods according to P10 was that if you keep asking students to fill in polls, the effectiveness of that method decreases. P4 put it as: ‘I was searching for a balance not to do too much and not to monitor them too closely’.

Theme 6: the (limited) value of information from preparatory activities

While the majority of monitoring happens during the guided sessions, the teachers did also use information from the preparatory activities. Theme 6 concerns reasons for why teachers did or did not monitor the preparatory activities closely.

Different types of preparatory activities deliver different types of information to teachers, and this was related to whether or not the teachers found the information worth monitoring. For example, simple frequencies of how many students watched instructional videos was deemed less informative than being able to read students’ answers on a quiz or a written assignment. Teachers found the latter type of information quite valuable, especially because every student is forced to enter a reply, in contract to plenary discussions during guided sessions in which some students stay silent. The preparatory activities therefore provide a broader picture of how students are doing.

The preparatory activities also serve as a check for the monitoring that occurs during the guided sessions. P2: ‘sometimes a student does not have an active attitude during the guided session, but does do very well on the preparatory assignment. Then I know that my judgement was not correct’. Conversely, the work that a student hands in for the preparatory activity can also be a reason to monitor that student more closely during the guided session. P3: ‘when a student scores bad on a quiz, I will keep in mind to monitor that student more closely in the subsequent session’. So, the two monitoring pathways can go hand in hand.

Even though teachers see the potential of monitoring the preparatory activities, they often do not have time to actually do so owing to high workload. Also, teachers often need to check each and every individual answer. There are no aggregated data such as common errors or common questions that the teachers can easily distil.

Theme 7: advantages of technology and future outlook

The last theme concerns what teachers have learnt from the transition to online education. At first teachers tried to translate their face-to-face practices to the online context. After gaining more experience, the teachers feel they have gotten better at adapting the design of their courses. They also started to see more advantages of the online guided sessions. Teachers indicated they had become more creative in using tools or widgets for (collaborative) activities, and they appreciated the efficiency and availability of location-independent meetings. Even though there are many complexities and issues to deal with, there is also a hopeful and creative outlook towards the future. In particular, teachers reflected on the possibilities of merging the best of both worlds. Some teachers already had ideas about ‘hybrid teaching’, in which students and teacher would be together in a classroom, but work on shared products via their laptops. That way, the teacher would have all monitoring cues available while also making use of the advantages of technology. P8: ‘The disadvantage of not being able to observe students’ attitude, activity, what they are doing … that would for a large part disappear’.

Discussion

In this study it was investigated how teachers in HE monitored their students when they transitioned to online education, and which challenged teachers experienced. By framing teachers’ monitoring processes in terms of two available pathways, namely monitoring of preparatory activities and monitoring of guided sessions, it was investigated what specific sources or cues teachers use to monitor their students. Shedding light on monitoring processes is important not only from a theoretical perspective, but also to formulate practical recommendations for teachers on how to monitor their students, as well as avenues for further research to address teachers’ challenges concerning monitoring.

Research question 1: monitoring strategies

Most teachers indicated that, similar to the face-to-face setting, they primarily monitored their students during the guided sessions. The aspects of student behaviour that teachers looked for most are content understanding, participation, motivation and interest. One of the principles underlying the flipped classroom model is that teacher–student interaction in the guided session is central to stimulating student learning (Staker & Horn, 2012; Van Alten et al., 2019). Based on the findings in this study, it seems that in the online flipped classroom the guided sessions still take on a central role, even when monitoring students was more difficult than in the face-to-face context. The cues that teachers used in the online guided most were observing student answers to questions and how students interact with each other. Teachers not only observe what students do, but often outright ask questions as a strategy for obtaining information. They indicated they did this more often than they usually did in the face-to-face guided session, as a means to compensate for the lack of available cues. To a lesser extent, teachers also monitored the preparatory activities to monitor content understanding and participation. The most often used cues were reading students’ submitted work and reviewing scores on quizzes. There definitely seems to be an opportunity here that is currently not being used to its full potential. While teachers select preparatory activities to prepare students for the guided sessions, the teachers make little use of the data resulting from those activities to prepare themselves. Given the demanding nature of monitoring guided sessions, one could argue that having information about students’ content understanding before the session starts would be extremely valuable. The teachers that were interviewed in this study indicated that they had too little time to monitor the preparatory activities or that the information was not valuable enough to monitor. Instead, they primarily relied on the monitoring in the guided sessions. Studying in more detail how the preparatory activities, the guided sessions and informational cues to the teacher could tie in together is an important direction for future research (Kaliisa et al., 2020).

Research question 2: monitoring challenges

The second research question concerned which monitoring challenges teachers experienced compared to the face-to-face setting and how they dealt with those challenges. Seven core themes were distilled from the interviews. Recapping the theoretical framework of this study, teachers’ scaffolding of student learning broadly consists of two steps: monitoring (via specific cues) and supporting (Lajoie, 2005; Loibl et al., 2020). Within the step of monitoring, two pathways can be discerned for the flipped classroom (Kaliisa et al., 2020), namely monitoring preparatory activities (pathway 1) and monitoring students during guided sessions (pathway 2). Theme 6 directly relates to monitoring pathway 1, and themes 1, 2 and 5 relate to monitoring pathway 2. Themes 3 and 4 relate to the subsequent phase of supporting; these themes concern the consequences of challenges that occur in the phase of monitoring. Finally, theme 7 concerns the role of technology in teachers’ practice.

As monitoring primarily occurred during guided sessions, the themes for research question 2 to a large degree centre around the difference in availability of cues in guided sessions in the face-to-face versus the online context. As teachers reflected on their experiences in the online guided sessions, they realised just how important the physical element of face-to-face meetings is. The physical component does not only ensure smooth interaction and the ability to observe students, but also creates the opportunity for an atmosphere of trust that is needed for a motivating study environment in which students do not feel reluctant to approach the teacher. The online environment offers much fewer cues for monitoring students, and monitoring becomes harder as group size increases. In the Introduction the study by Van der Spoel et al. (2020) was discussed, which found that teachers had monitoring difficulties in online education. The findings from this study confirm the expectation that this difficulty is caused by the extreme difference in availability of monitoring cues (Loibl et al., 2020), and this study has provided more details on what cues are involved. As a result of the constraints of the online context and teachers’ decreased ability to monitor students, the roles and responsibilities in online education change. Teachers feel less able to provide personalised teaching and rely more on students’ self-monitoring skills to seek support. The importance of students’ self-regulation is similar to that reported in other types of online education, such as MOOCs (Hew & Cheung, 2014). The finding that teachers were less able to personalise their teaching is a sign that adequate monitoring is indeed essential to the teacher’s profession (Lajoie, 2005). It is therefore important that teachers find ways to deal with the challenges they experience, especially if the share of online education increases. Below, it is discussed what solutions teachers used and which challenges remain to be addressed in future research.

Limitations

The most prominent limitation of the present study is its sample size. Effort was made to secure a sample that would allow for data saturation, yet remain small enough to allow for thorough analysis. Despite these efforts, the sample does contain participants that fit a variety of different categories, for example gender, study programme, teaching experience, attitude towards teaching in general and online/blended teaching in particular. This variety means that generalisability of the outcomes only partially applies (Guest et al., 2006). The results and suggested implications of this study should therefore be seen in light of this limitation.

Another potential limitation is that the results are based on teachers’ self-reports about their monitoring strategies. It has been argued that interviews are less objective than for example observing what teachers actually do. However, monitoring strategies are difficult if not impossible to observe, and interviews were therefore deemed methodologically appropriate.

Best practices for enhancing monitoring

The most occurring theme in the interviews was how course design can be leveraged as a tool to create explicit monitoring opportunities. For preparatory activities, content quizzes are one of the most adequate means to find out how students are doing. The two monitoring pathways can go hand in hand, because the insights from the preparatory activities can be used to closer monitor specific students during the guided sessions.

For the guided sessions, teachers mentioned a considerable number of monitoring strategies. Figure 3 visualises the recommendations for the structure of an online guided session. Left and right of the main sessions are the recommended walk-in moments for small-talk or discussing organisational matters concerning the course. In the main session, it is recommended to start off with a plenary opening in which attention is given to students’ well-being, for example by using polls. Ensuring that every student feels seen and heard will create a motivational and open atmosphere in which students feel less hesitant to speak up. This is especially important since the teacher cannot observe everything and thus needs to partly rely on students’ own responsibility to let the teacher know when support is needed. The bulk of the guided session is recommended to consist of small group activities. Listening to student discussions, according to the interviewed teachers, is one of the central ways to discover students’ content understanding. To enable monitoring of multiple groups at once, teachers can assign one student per group to be the chair that reports back to the teacher what was discussed. Teachers can also ask students to work on a product such as a written text or mind map that the teacher can later check for content understanding. In the plenary closing of the session, teachers can collect input from the small group to discuss interesting ideas or issues that are still unclear. Polls can be used again here to check students’ progress.

Figure 3. Recommendations for the structure of a guided session.

Directions for future research

While adequate course design helped the interviewed teachers to monitor their students, several challenges remain. Below three directions for future research are outlined that tackle these challenges.

The first direction for future research concerns the monitoring pathway of the preparatory activities. The interviewed teachers found this a valuable source of information, but often lacked the time to go through all of the students’ assignments. Automated analyses of students’ preparatory activities (i.e. learning analytics) may play a larger role in informing teachers of students’ content understanding. Information about frequency of individual activities may not be valuable enough because it is not actionable for teachers (Van Leeuwen, 2019; Wise & Shaffer, 2015). Instead, automated content analysis of students’ written texts could be beneficial for teachers by indicating which concepts need further attention in the guided session. While text analysis is a large field, the recent review by Ferreira-Mello and colleagues (Ferreira-Mello et al., 2019) lists no applications of this technique yet in the flipped classroom.

Concerning monitoring during guided sessions, teachers indicated they use discussions between students as a primary source to monitor students’ content understanding. However, online environments only allow the teacher to interact with one group at a time, and the teacher needs to decide blindly which group to spend time on with the risk of disrupting instead of stimulating the group process. A solution could be an additional interface that would allow the teacher an overview of all groups at once. Such an interface could make use of automated analyses of students’ facial expressions or analyses of the conversation to inform the teacher which group could benefit from the teacher’s presence (Zhou et al., 2021).

The interviewed teachers were optimistic about the future and they had learnt valuable lessons from the transition to online education. Teachers mentioned the potential of the hybrid classroom that combines the best of face-to-face and online education. Microphones, computers and flexible seating could allow students to collaborate with technological tools while they are co-located in the same room. For teachers, such classrooms would offer a rich set of cues, but also the creative use of technological tools for learning activities. Some universities have indeed begun experimenting with hybrid classrooms (Hod & Katz, 2020; Kaplan, 2021).

Conclusion

To conclude, in this study it was investigated how teachers monitor their students in the flipped classroom, and how they continued to do so during the transition to fully online education. Although teacher monitoring and support of their students is essential, there has been no detailed investigation yet of teachers’ strategies for monitoring in the flipped classroom. In the present study, the model of teacher decision making was connected to two potential monitoring pathways, and it was investigated what aspects teachers monitor and what cues they use to do so. The conclusion is that teachers mostly monitored students during online guided sessions, and that they had to change their course design to deal with the lack of availability of cues in the online context to explicitly create monitoring opportunities. Practical recommendations were given how teachers can create these opportunities.

Although several challenges still remain, the outlook for the future is positive. Teachers have learned valuable lessons for integrating technology into their practice, and there are promising opportunities for technology to play a larger role in supporting both student learning and teacher monitoring thereof.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This study was supported by a grant from the Higher Education Research focus area at Utrecht University.

Notes on contributors

Anouschka van Leeuwen

Anouschka van Leeuwen is Assistant Professor at Utrecht University. Her research interests include learning analytics, blended learning and collaborative learning.