Leveraging Artificial Intelligence to Predict Young Learner Online Learning Engagement

ABSTRACT

The recent surge of online language learning services in the past decade has benefitted second language learners. However, there is a lack of understanding of whether learners, especially young learners, are engaged in online learning, and how educators can enhance the engagement of the online learning experience. This study examines an artificial intelligence (AI)- powered automated system that uses voice and facial recognition to track both teacher and learner speech, facial expressions, and interactions in real-time in a one-to-one 25-minute online English class. Each learner completed a learner engagement survey within 72 hours of the online class. Results demonstrated that young learners were highly engaged during this one-to-one online learning setting (mean = 4.5, out of 5). Learners’ frontal face exposure (indicating their attentiveness during class) and English proficiency levels are significant and positive predictors of learner engagement. Teachers’ total length of speech and instructional time tended toward significance in predicting learner engagement. Educational implications are discussed.

KEYWORDS:

• Artificial Intelligence (AI)
• learner engagement
• online tutoring
• data science applications in education
• English as a Foreign Language (EFL)

The recent surge of online language tutoring services in the past decade has benefitted second language learners (Meticulous Research, 2020). There are two types of online tutoring services: asynchronous and synchronous (Khan, 2006). In asynchronous tutoring, learners study on their own schedule and access learning materials online. In synchronous tutoring, instruction is live with the tutor and the learner being simultaneously present. Synchronous tutoring has two modes: one-to-one and one-to-many.

Artificial intelligence (AI) based automated measures, such as facial and voice recognition, can be an efficient tool to measure learning and teaching and provide immediate and actionable information to educational stakeholders, especially in one-to-one settings (Dewan, Lin, Wen, Murshed, & Uddin, 2018). However, little research has been done to understand how AI-extracted learning and teaching indicators predict learner engagement. This study furthers the emerging research in the educational technology field by leveraging AI data from both learners and teachers to better understand young learners’ online learning engagement in the one-to-one online tutoring setting.

Theoretical background

Definition and importance of learner engagement

Engagement has been called the “holy grail of learning” (Sinatra, Heddy, & Lombardi, 2015, p. 1), and is defined as “the amount of physical and psychological energy that the learner devotes to the academic experience” (Astin, 1997, p. 297). It is associated with improved outcomes, such as critical thinking (Elyas & Al-Bogami, 2019), academic achievement (Reeve & Tseng, 2011), and persistence in learning (Kuh, Cruce, Shoup, Kinzie, & Gonyea, 2008). Learner engagement is also a key factor in second language learning, framed as a “drive that is capable of both generating and sustaining long-term behavior” (Dorneyi & Ryan, 2015, p. 99).

The type of engagement learners experience varies with age. Elementary school learners are particularly swayed by classroom experiences, whereas older learners indicate the setting of long-term goals increases their engagement (Leona et al., 2021). Determining learner engagement and enhancing it is an important factor for long-term language acquisition. Student engagement changes in response to lesson activities and classroom interactions, as well as the learners’ motivational state (Oga-Baldwin & Nakata, 2020).

Measuring young learner engagement

Measuring young learner engagement, can be difficult (Henrie, Halverson, & Graham, 2015). Observational evaluation, where an observer’s subjective opinion on the learner’s engagement, is commonly used (Dewan, Murshed, & Lin, 2019). These evaluations contain checklists of actions meant to indicate engagement such as “sitting quietly” and “no tardy cards.” However, these learning indicators have not been shown to measure engagement but rule-following (Whitehill, Serpell, Lin, Foster, & Movellan, 2014). Therefore, social researchers recommend conducting surveys with children directly such as self-report surveys (Fredricks & McColskey, 2012).

A 2017 study (Oga-Baldwin & Nakata, 2017) measured children’s engagement in a 5th grade English as a Foreign Language (EFL) class in Japan using self-report surveys. Researchers found that learner engagement during in-person class was on average 3.68 out of 5 points. Additionally, the researchers gathered independent rankings by external observers classifying student engagement as low, moderate, or high. The students’ self-reported data broadly matched with the external rankings for engagement. Further research has shown the impact of classroom activities on learners decreases as they grow older (Nikolov, 2002). For young learners, it is essential to discover what types of interactions increase engagement.

AI-based methods to measure engagement

Engineers and researchers have used AI to draw insight into teaching and learning indicators. Computer vision enables computers to extract useful information from digital images, videos, and other visual inputs (IBM, n.d.-a). Voice recognition is the capability of a program to identify an individual user’s voice (IBM, n.d.-b). These AI-based methods mirror external observations to classify learning engagement as an emerging area of research (Sharma, Giannakos, & Dillenbourg, 2020). Automated methods are mostly being used to study adult online learning (Whitehill et al., 2014) and rarely being examined for young learners.

Computer vision analyzes cues from the face, body posture, and hand gestures to estimate learner engagement (Sharma et al., 2020). Video stream data is captured using a webcam or similar device. Typically, the engagement detection systems are trained using large datasets that are human labeled to track and classify features including facial expressions and eye movement (Friesen & Ekman, 1978). Facial recognition methods have only recently begun to be deployed for engagement detection in learning but show promise (Dewan et al., 2019). Krithika and Lakshmi Priya (2016) found eye and head movement is an important indicator to detect adult learners’ emotion and engagement during online instruction. Ramakrishnan, Ottmar, LoCasale-Crouch, and Whitehill (2019) trained a smiling facial recognition model specific to young learners to analyze the classroom climate. Their results showed that learners’ and teacher’s smiles accurately indicated the dynamics of a positive classroom climate. Another study analyzed video data and found that teachers use smiling to guide, confirm, and correct student responses. Smiling is “an available resource for teachers to construct pedagogically relevant actions” (Jakonen & Evnitskaya, 2020, p. 29).

Researchers analyzed teacher and learner audio files in an online context and found that teacher talk, an important teaching indicator, is related to student engagement (Hirschel, 2018). The correct amount of teacher talk is difficult to pinpoint and varies widely depending upon variables including learner age and second language proficiency. Paul (2003) recommended raising teacher awareness of teacher talk and focusing on expanding the comprehensible input. Walsh and Li (2013) showed that teacher talk directly shapes learning opportunities in language classrooms suggesting that increased wait-time, extended learner turns, and increased planning time contributes to learner speaking engagement in the classroom. Whereas teacher talk interruptions were shown to negatively impact learner participation (Yataganbaba & Yıldırım, 2016).

Research questions

To lay the groundwork to leverage AI-based measures generated, this study investigates the following research questions:

1. What is the engagement level reported by young learners studying English in a one-to-one online tutoring platform?

2. Can AI-measured learning and teaching indicators predict self-reported learner engagement?

Methods

Participants

Between November and December 2020, 159 participants were sampled from a student database of an online education technology company, VIPKid (https://www.vipkid.com/en-us/). VIPKid provides elective informal after-school online tutoring programs. Participants in this study enrolled in the Major Course (MC), VIPKid’s flagship English curriculum line. Instruction is provided via a synchronous one-to-one, online class format taught remotely by teachers. A lesson comprises a pre-class video, a one-to-one 25-minute class, and post-class enrichment activities. Learners and teachers access the classes by logging into VIPKid’s proprietary platform. Teachers follow the interactive slide decks for the one-to-one class.

Among all the learner participants, there were 79 males (51.6%) and 74 females (48.4%). We followed the recommendations of Bell (2007) that survey should be completed with children aged 7 or above. Therefore, our participants ages ranged from 7 to 14 years, with the average age being 9.1 years. Learners were enrolled in MC Levels 2 through 6 with higher MC levels indicating a higher level of English proficiency determined by an entrance placement test. Level 3 had the most learners with 58 learners (37.9%) and Level 5 had the fewest with 13 learners (8.5%).

Materials and measures

Engagement survey

Our study used an updated engagement survey instrument from Oga-Baldwin and Nakata (2017). The questionnaire from 2017 was translated from English into Chinese by two researchers (Table 2). The overall engagement is the average score of all eight items with the Cronbach’s alpha reliability of 0.90.

The survey was piloted twice. First, with eight Chinese learners aged 7–10 to ensure that it was clear and understandable. Second, using two different versions of the five-point smiley face Likert scale (SFL) developed by Hall, Hume, and Tazzyman (2016), one with endpoints anchored at a sad face to a happy face and the other with variations of only happy faces. The piloting revealed that learners were able to understand the translated language. The happy-face-only variation received the greatest scale usage in learner responses, which is consistent with findings in Whitehill et al. (2014). Based on the pilot results, the survey was finalized with the translated Chinese language and the scale as shown in Figure 1.

Figure 1. Five-point smiley face Likert scale used in the learner survey.

AI-measured learning and teaching indicators

AI-measured learning and teaching indicators were generated through voice recognition and computer vision technology for both the learner and teacher. These variables include commonly used and discussed AI measures on learner image, teacher behaviors, and instructional time. The descriptions of variables are listed in Table 1.

Table 1. Artificial Intelligence (AI) -Measured Learning and Teaching Indicator.

Variables	Description
Voice recognition
T first speech time (min)	The timestamp of teachers’ first speech, indicating how soon they started to speak during the class.
T speech total length (min)	Teachers’ total length of speech indicating how much they spoke during class
Computer vision
Number of L frontal face	The total count of learners’ frontal face exposures captured during class averaged to every minute, indicating how attentive they were. Note that the learners’ frontal face exposure is captured every second.
Number of L smiling face	The total count of learners’ smiling faces captured on average during class averaged to every minute. Note that the learner’s smiling face is captured every second.
Number of T smiling face	The total count of teachers’ smiling faces captured during class averaged to every minute. Note that the teacher’s smiling face is captured every second.
Instructional time (min)	Length of class time when both the teacher and learner were present

Note. L=Learner; T=Teacher

Table 2. Descriptive Statistics on Student Self-Reported Engagement Scores.

Item	Mean	Min	Max	SD
In today’s class, I paid attention.	4.4	2	5	0.7
In today’s class, I spoke a lot of English.	4.4	2	5	0.8
In today’s class, I participated in the activities.	4.4	1	5	0.9
In today’s class, I continued working until the end of class.	4.6	2	5	0.7
In today’s class, I felt interested.	4.5	1	5	0.8
I enjoyed today’s class.	4.7	1	5	0.7
In today’s class, I felt good.	4.6	1	5	0.7
In today’s class, I enjoyed learning new things.	4.6	2	5	0.7
Overall Engagement	4.5	2.1	5	0.6

Note. Sample size = 153

To generate voice recognition variables, the AI system extracted the audio information from the video playback. The teacher and the learner each have an audio file. Next, voice activity detection (VAD), an algorithm that detects an audio signal and segments it into portions that contain speech or silent parts (Ivry, Berdugo, & Cohen, 2019) was run for each speaker’s audio file using the threshold classification method. The threshold classification method classifies data according to a preset threshold. Those within the threshold range belong to one category (e.g. “voiced”) while the rest go to the other (e.g. “unvoiced”). This method was used to identify the start time and end time of the voiced and unvoiced segments in the audio (Song, Wang, & Feng-Juan Guo, 2009). Two VAD models, for learners and teachers respectively, were trained. The models used a large set of transcribed data that removed meaningless vocal interferences for better accuracy in calculating these speech variables (Bhatia, Davidson, Kalidindi, Mukherjee, & Peters, 2006). The VAD models indicate when teachers and learners begin speaking during class and calculate the total length of speech for each speaker. See Figure 2 for a visual representation of the VAD audio signal segmentation. Portions that have detected speech are bordered in black rectangles.

Figure 2. Visual representation of VAD audio signal segmentation.

To generate the computer vision variables, the AI system first extracted video information from the video recording. For every 1-second video frame, the face detection algorithm RetinaFace (Deng et al., 2019) was used to identify whether there is a face in the video frame. Next, using the facial angle model “Fsa-net” (Yang, Chen, Lin, & Chuang, 2019), the yaw angle of the learner’s head posture was calculated for every frame with a recognized face. The learner’s head pose information can be used to gauge attention (Stiefelhagen & Zhu, 2002). The head yaw angle is used to track whether or not a learner is looking at the screen. For our purposes, if the head yaw angle is more than 30° from the computer screen, the image is categorized as non-frontal facing (See Figure 3).

Figure 3. Calculation of Head Yaw Angle for Off- Screen Viewing.

To count the number of smiling faces, the “RetinaFace” set of recognized faces was categorized as either “smiling” or “non-smiling” using a self-developed image classification model trained using the ResNet module (He, Zhang, Ren, & Sun, 2016) in the Azure Machine Learning designer. This image classification model is a supervised learning method (Xie, Girshick, Dollár, Tu, & He, 2017) and was trained using a set of 10,000+ human-labeled images.

Procedure

Upon enrollment in the one-to-one tutoring platform, learners and their guardians signed consent forms to allow the company to collect and analyze voice and video data to report learning progress and monitor class interactions. This study invited “active” learners, learners who had completed at least four classes and taken a class within the two weeks preceding the start, to participate. 295 parents of active learners indicated they would participate and signed a consent form for the study. They were incentivized to complete the survey with a platform-specific currency which could be used as credits to exchange for classes and peripheral products. Among them, 193 learners completed the survey within two weeks after receiving it (survey response rate = 65%), out of whom, 159 participants’ surveys were completed within 72 hours after they completed a one-to-one class. We excluded 34 participants who filled in the survey after 72 hours, as they may inaccurately recall their class over time, and some of them started taking a second class which could confuse their memory of the specific class surveyed.

Data analysis

All analyses were performed using SPSS (2020). First, descriptive analyses were performed to understand the distribution of each variable. Second, a Pearson correlation table was created to examine the preliminary interrelationships among the AI learning and teaching indicators. Third, multiple regression models were used to predict learner self-reported engagement from AI learning and teaching indicators and learner and teacher background characteristics.

Three learners had instructional times under 10 minutes and were removed from the analysis due to reasons such as leaving class early, illness, etc. Three other learners were enrolled in MC levels above 6. These classes are 50-minutes in length. We excluded these learners from the analysis because 25-minute classes and 50-minute classes are not comparable on indicators that are derived from counts of interactions over the class period. Therefore, the final sample size was 153. Because learner participants responded to all survey questions, there is no missing survey data.

Results

Table 2 describes learners’ self-reported overall engagement scores. On a scale of 1 to 5, with 5 being the highest level of engagement, learners reported a high level of engagement as the average score on each item and the total scores were all above 4.

Descriptive statistics for the online learning and teaching indicators and their correlations are presented in Table 3. The results indicated that the learner’s smiling face was positively associated with the learner’s frontal face exposure (r = .28) and the teacher’s smiling face (r = .21).

Table 3. Descriptive Statistics of Online Learning and Teaching variables and their correlation

Variables	1	2	3	4	5	6	Mean	SD
1. T first speech time (minutes)	-						0.07	.12
2. T speech total length (minutes)	.09	-					0.92	0.33
3. # L frontal face	.15	.04	-				0.54	6.78
4. # L smiling face	.11	.15	.28**	-			0.11	0.57
5. # T smiling face	.01	.05	.04	.21**	-		1.55	4.61
6. Instructional Time	.08	.05	.09	.08	.01	-	6.6	0.71

Note. Sample size = 153 # = number of; L=Learner; T=Teacher

**Correlation is significant at the 0.01 level (2-tailed).

We used a multiple regression model to predict learner self-reported engagement from all the AI learning and teaching indicators, learner demographic chracteristics (English proficiency level, gender, age), and instruction time. These independent variables have been found to be predictive of learning outcomes in the literature (Oga-Baldwin & Nakata, 2017). The multiple regression results show that learner frontal face exposure, learner English proficiency level, teacher speech total length, and instructional time are significant (or marginally significant) predictors of learner self-reported engagement (Table 4, Figures 4 to 7).

Table 4. Multiple Regression Predicting The Learner Engagement

Predictors	Coefficient	s.e.	p
Intercept	3.51	0.59	<.001
# L frontal face	0.007	0.003	.03
# L smiling face	-0.006	0.005	.27
# T smiling face	0.004	0.003	.28
T first speech time (min)	-0.01	0.02	.60
T speech total length (min)	-0.65	0.39	.09
English proficiency level	0.15	0.05	.003
L Age	-0.05	0.04	.21
L Gender	0.04	0.10	.72
Instructional time	0.03	0.02	.09

Note. # = number of; L=Learner; T=Teacher

Adjusted R² = .105

Figure 4. Learner Frontal Face Exposure and Learner Self-Reported Engagement Scores.

Figure 5. English Proficiency Level and Learner Self-Reported Engagement Scores.

Figure 6. Teacher Speech Total Length and Learner Self-Reported Engagement Scores.

Figure 7. Instructional Time and Learner Self-Reported Engagement Scores.

Learners who had more frontal face exposure reported significantly higher levels of overall engagement than their peers with less frontal face exposure (β = 0.007, p < .05). Figure 4 also shows the positive association between learner frontal face exposure and self-reported engagement score. As shown in both Table 4 and Figure 5, learners with higher English proficiency reported a higher level of overall engagement than those who reported lower English proficiency levels (β = 0.16, p < .05). Two marginally significant relationships show that the shorter the total length of teacher speech (β = −0.65, p = .09) and the longer the instructional time (β = 0.03, p = .09), the higher the learner self-reported engagement score (Table 4 and Figures 6 and 7).

Discussion

Self-reported online learner engagement among young learners

This study found that our study participants reported a high level of engagement (4.5 out of 5) learning English in a synchronous one-to-one online tutoring platform. This engagement level is higher than what was reported in the traditional EFL classrooms (Oga-Baldwin & Nakata, 2017). In Oga-Baldwin & Nakata’s study, using the same survey and scale as in this study, students in 16 traditional brick-and-mortar fifth-grade Japanese EFL classes reported an average engagement ranging between 3.18–4.14 (mean = 3.68, SD = 0.83). Learners in our study likely reported higher levels of engagement due to the one-to-one nature of the courseware. Most learners in traditional brick-and-mortar classrooms compete for the teacher’s attention. In a one-to-one environment, the learner has the complete attention of their teacher which may relate to a high level of engagement.

Additionally, the course content has been specifically designed to engage the learner’s attention. To maintain high learner engagement, the courseware follows a “best practices” framework synthesized by Taylor and Parsons (2011): (1) interaction, (2) exploration, (3) relevancy, (4) multimedia, (5) instruction, and (6) authentic assessment. Learners had more opportunities to interact with their teachers as compared to traditional classrooms. In terms of “exploration,” the lessons in this program utilize Bloom’s Taxonomy (Krathwohl, 2002). Learners are required to analyze, apply, and create something new (e.g., formulating their own ideas/opinions using the language and content learned in that lesson). Regarding “relevancy,” lessons were organized by themes and topics based on learners’ age, interests, and abilities, increasing the relevance of the content to learners. Finally, the curriculum primarily calls for cross-subject exploration and collaboration, often called “21st Century Skills” in recent literature (Taylor & Parsons, 2011). The curriculum itself may be more engaging for learners than that which they typically encounter in a brick-and-mortar classroom which further explains the learner’s high engagement scores.

AI learning and teaching indicators

Our correlational analysis shows that AI-measured learners’ smiling faces were positively associated with both learner frontal face exposure and the teacher’s smiling faces. Echoing what has been found in classroom setting that learners’ facial feedback (i.e., smiling) is positively correlated attention in classroom conditions (Marsh, Rhoads, & Ryan, 2019), our findings indicate in one-on-one online learning settings, learner’s smiling face is related to frontal face exposure, a measure of attention. Furthermore, previous research emphasized that teacher enthusiasm could enhance a range of positive learner outcomes, including attention, engagement, and recall in a classroom setting (Marsh et al., 2019). Our results suggest the mutual influence of teachers’ smiling faces and learners’ smiling faces in a synchronous one-to-one online language tutoring setting.

Predictors of engagement

Amongst all the AI learning and teaching indicators, learner frontal face exposure was a significant predictor of learner engagement. An evident increment of engagement was found when the count of frontal face exposures within a lesson increased. Our learner frontal face exposure measure indicated whether the learner was directly facing the screen. When the learners had low counts of learner frontal face exposure, they most likely were not attentive and focused on the lesson content. Behaviors like drowsiness and turning the head to talk were identified as low engagement indicators in the context of online learning (Hwang & Yang, 2008).

The teacher’s total length of speech has a tendency toward significance in predicting learners’ engagement. Learner engagement dipped significantly when teacher talk was over half the class time. Consistent with Paul (2003) recommending raising awareness of teacher talk time, our findings emphasize the negative impact of teacher talk reducing young learner’s engagement. The importance in language learning for young learners is to encourage learners to speak more during the lessons.

Instructional time was also found to tend toward significance in predicting learners’ engagement. Teachers who stayed 10 minutes or longer had learners rate their engagement very high (4.8+ out of 5). Although class time is set to be 25 minutes, some teachers voluntarily chose to stay longer. The end of the class period is for rapport-building activities during which teachers and learners engage in conversation related to their personal lives. These brief extension moments allow the teachers to give additional support and attention to the learners. This finding is consistent with previous research on how teachers’ additional support and attention impacted learners (Blazar & Kraft, 2017). In both the previous research and this study, teachers’ additional support for learners was proven to be related to a higher level of learner engagement.

Finally, English proficiency level was a significant predictor of learner engagement. Learners with higher English proficiency reported a higher level of overall engagement than learners with lower English proficiency in all Levels, with an exception between Levels 4 and 5 where a plateau was observed. This makes sense as high-proficiency learners were more able to use the target language to express themselves and interact with the teachers, thus an improved learner engagement is observed. However, language learning literature supports that learner engagement is related to teacher and learner interactions (Davidson, 1999), but not necessarily learners’ language proficiency. More research is needed to examine the impact of learner proficiency on learner engagement.

Practical implications

Online learning platforms can use automated AI measures to predict and monitor learner engagement. One important AI indicator is learner frontal face exposure. If a learner is detected as not facing the screen for a certain amount of time, an automatic reminder can be sent to the learner to pull their attention back. The message can be sent to the teacher so that the teacher could differentiate teaching to get the learner’s attention back.

Similarly, if the system detects teachers are speaking too much, an automated reminder can provide tips to teachers to give learners opportunities to speak up. For example, for low proficiency-level learners, the system suggests using simple activities like “read aloud” to allow learners to participate.

Instructional time is a bit difficult to integrate into practical settings as teachers cannot always voluntarily extend class. However, teachers may benefit from trainings on effective strategies to build teacher learner relationships. If the teachers can show that they care for their learners and are willing to provide extra support, the learners are more likely to be engaged in their remote learning environment.

Limitations and future research

Our study has several limitations that need to be addressed in future research. First, our AI indicators of engagement can be improved. Learner frontal face exposure measured by head orientation has limited precision in estimating a person’s attentional direction; future studies need to utilize eye gaze and body posture data to achieve better precision to measure attentional direction (Shirama, 2012). Future studies also need to include other emotional recognitions such as boredom, confusion, frustration, etc., to boost engagement prediction (Dewan et al., 2019). Recent studies have noted that racial bias exists in facial recognition programs (Raji et al., 2020). The accuracy of the facial recognition model, RetinaFace, used to recognize teacher and learner faces should be examined for any racial biases. Future studies can also benefit from a measure of the quality of teacher speech.

Second, this study has a sample size of 153 participants. Future studies need to replicate the findings of this study using a larger sample.

Third, although self-reported survey data has the advantage of capturing user experience and perceptions, they can be objective and biased (Short et al., 2009). Future research may aim to overcome this limitation by using mixed methods (e.g., structured interviews) to provide a more detailed picture of learner engagement.

Finally, one needs to generalize the findings of this study to other online learning environments (i.e., Massive Open Online Courses) with caution. Additional research is necessary to validate our approach in other online contexts with young learners.

Conclusion

Despite these limitations, this study is the first to report the engagement level among young learners in the one-on-one online language tutoring context, and the first to predict young learners’ self-reported engagement using AI learning and teaching indicators. This study provides a promising contribution to the field of engagement study of young learners using education technology and offers a valid starting point for further research in this area.

Disclosure statement

No potential conflict of interest was reported by the author(s).