Facial expressions capturing emotional engagement while learning with desktop VR: the impact of emotional regulation and personality traits

ABSTRACT

Virtual reality (VR) has been shown to induce excessive affective processing, which in turn impacts the learning process and learning outcomes. Therefore, a better understanding of emotional dynamics and how emotions are instigated while learning with VR is needed. For this purpose, the students learning process with a VR simulation was captured using continuous measurement of facial expressions. In addition, personality traits and self-regulated emotion (SRE) characteristics were examined. The findings suggest that overall, joy was the most predominant emotion expressed by students, while intraindividual components of personality traits and SRE strategies were associated with negative emotional expressions. Specifically, students with high conscientiousness or agreeableness personality traits tended to experience less anger and/or sadness. In addition, the findings demonstrate that the experience of these negative emotions was mediated by the SRE strategies of reappraisal and suppression, implying a mechanism of emotional engagement while learning with VR. This suggests that while individual personality traits are relatively stable and immutable, SRE preferences can be supported as part of the VR instructional design and scaffolding. Future studies should explore how students’ emotion regulation preferences can be supported to benefit their emotional experiences while learning.

KEYWORDS:

• Virtual reality
• personality traits
• emotion self-regulation
• facial expressions analysis
• emotional engagement

Introduction

Virtual reality (VR) is a computer-generated environment to model and simulate three-dimensional (3D) visual experiences and interactions. While learning with VR students can maneuver and manipulate the 3D representations of a physical location or of an occurrence, which facilitates the illusion that this virtual place truly exists (Di Natale et al., 2020; Ghanbarzadeh et al., 2014). VR-based simulations create a psychological sensation and illusion of being within the artificially created world as if it was a real place (Radianti et al., 2020). This evokes the feeling of “being there” or a “sense of presence” (Witmer & Singer, 1998). As such, these unique immersive characteristics of learning with VR can increase learners’ affective arousal and boost their emotional engagement. To this end, the current study examines how and to what extent emotions are being instigated during learning with VR; and what role emotion self-regulation and individual personality components play in emotional dynamics.

Emotional engagement

Engagement is a multi-dimensional construct that integrates three main indicators that relate to behavioral, emotional, and cognitive characteristics of students’ learning (D'Mello et al., 2017; Fredricks et al., 2004; Sinatra et al., 2015; Skinner, 2016). This study focuses only on the emotional dimension of engagement, which mainly emphasizes students’ emotional states during learning (Christenson et al., 2012). Emotional involvement during learning can be positive, which is characterized by emotions such as enjoyment, interest, and pride (Renninger & Bachrach, 2015). Emotional states can also be negative, and students might experience emotions such as boredom, anger, and anxiety (Skinner, 2016; Skinner et al., 2008). Theories of motivation, including the self-determination theory, and the control-value theory (CVT) of achievement emotions (Deci & Ryan, 1985; Pekrun & Linnenbrink-Garcia, 2012) underline the role that both positive and negative emotions have on students’ involvement in learning activities and underscore how affective dynamics sustain or disrupt learners’ engagement to impact learning performance (D’Mello & Graesser, 2012; Gupta et al., 2018; Pekrun & Perry, 2014).

Emotional engagement with VR is an important factor to consider when investigating learning processes with VR. This is important because VR fosters a higher level of a sense of presence than standard media (Makowski et al., 2017), and as some studies suggest, is also able to elicit different emotional experiences (Awada et al., 2021; Felnhofer et al., 2015). For example, studies in psychological therapy show that VR can activate different degrees of fear and thus is being used to treat phobias (Freitas et al., 2021), such as spider phobia (Miloff et al., 2019), claustrophobia (Hudson et al., 2021), social anxiety (Horigome et al., 2020), and aerophobia (Cardoş et al., 2017). Studies have also demonstrated that VR can increase positive affect and decrease negative affect, and as a result has been used in affective computing to improve mental health and emotional well-being (Prabhu et al., 2019; Sun et al., 2022). While the research in the psychology and therapy domains on emotional experiences in VR is growing, to date only a handful of studies in education have examined emotions instigated during learning with VR. The limited research on emotional engagement while learning with VR suggests that VR induces intense basic emotions, such as joy and sadness (Makransky & Lilleholt, 2018; Parong & Mayer, 2021), as well as complex emotions such as awe and gratitude (Collange & Guegan, 2020). A recent study by Dubovi (2022) demonstrated that learning with VR induced predominantly more joy than other basic emotions, however, only the negative emotion of anger was significantly associated with learning achievements. Nevertheless, while more research is needed on learning with VR and the involved emotional experiences, a recent systematic review on emotions in virtual learning (Henritius et al., 2019), concluded that most extant studies rely on post-hoc data and operationalize emotions as traits rather than states, suggesting the need for continuous measurements.

Indeed, emotions are considered reactive by nature, as they occur in response to situations, actions, or events, therefore, during a learning episode, emotions inevitably fluctuate (Ahn & Harley, 2020; Pekrun, 2006). Facial expression analysis has been suggested as efficient in capturing learners’ fast-changing emotional state during learning with VR (Dubovi, 2022). In this way, varying sets of action unit activities can be matched by automated artificial neural networks to different discrete emotions in very short time intervals (Ekman, 1992; Ekman & Rosenberg, 1997). This approach can provide an objective, continuous and detailed view of students’ transitory emotions throughout a learning session (Dindar et al., 2020). Notably, since emotions fluctuate not only as a response to the academic flow of events but may also be triggered by intraindividual personality components and self-regulation tendencies, it is necessary to view emotional engagement through these lenses as well.

Emotion regulation and personality traits

Self-regulated emotion (SRE) refers to the processes of reducing or increasing the intensity or duration of positive and negative emotions during learning to reach optimal levels of functioning (Gross & John, 2003). SRE is part of self-regulated learning processes (SRL), which over the past decades are being conceptualized as multi-dimensional constructs which include not only cognitive and metacognitive components but also emotional aspects of learning (Schunk & Greene, 2017; Zheng et al., 2021; Zimmerman, 2013). Moreover, SRE has been demonstrated as an independent SRL construct that should be investigated and explicitly taught in addition to cognitive and behavioral self-regulated strategies (Ben-Eliyahu, 2019; Ben-Eliyahu & Linnenbrink-Garcia, 2015; Zheng et al., 2021). The current study focuses on reappraisal and suppression SRE strategies in learning (Gross, 1998).

Reappraisal refers to the reframing of one's perception of a situation or the modification of its emotional significance, by altering the thoughts about the situation or about the capacity to fulfill the demands of the situation (Gross, 1998; Gross & John, 2003). Reappraisal is antecedent-focused, meaning it is often used prior to experiencing the emotion to decrease negative emotions. For example, reminding oneself that even though preparation for an exam might be arduous, it can lead to better performance on the exam. Reappraisal has been shown to be positively related with higher positive emotions and adaptive outcomes such as higher learning outcomes (Leroy et al., 2012; Stiller et al., 2019; Strain & D'Mello, 2015), and negatively related to negative emotions and maladaptive outcomes (Gross, 2015).

Research findings suggest that reappraisal is an adaptive form of emotion regulation which is more effective than suppression at decreasing the experience of emotion (Troy et al., 2013). Suppression is response-focused, occurs after the emotional tendencies are triggered, and involves the process of inhibition of ongoing negative or positive emotion-expressive behavior (Gross & John, 2003). Suppression has substantial psychological and cognitive costs, such as reliable decrements in both objective memory and memory confidence ratings (Gross, 2002). Furthermore, suppression decreases emotion expression but not negative emotional experience and as a result, was found positively related to negative emotions and to impaired memory (Gross, 2015; Johns et al., 2008).

Learners employ an emotion regulation strategy depending on situational context-specific factors, emotion regulation goals, and self-efficacy (Gross, 2015). Ben-Eliyahu and Linnenbrink-Garcia (2013) found that self-regulated emotion strategies are differentially employed based on course preferences (favorite and least favorite courses). Additional studies have also suggested that personality traits of the Five-Factor Model (FFM) are linked with emotion regulation strategies (Barańczuk, 2019; Digman, 1990). FFM is the universal most used taxonomy of personality traits in psychology (Extraversion, Agreeableness, Conscientiousness, Neuroticism and Openness to Experience) (Barańczuk, 2019; McCrae, 2020). Berkovich and Eyal (2021) found that a teacher’s personality affects their emotion regulation strategies, pointing toward conscientiousness as a distinct personality trait that differentiates between various used emotion regulation strategies (Berkovich & Eyal, 2021). Extraversion and neuroticism are recognized as the three personality traits most relevant to emotion (see Watson et al., 1988). Extraverts are characterized as being outgoing, sociable, and enthusiastic, and thus are more prone to positive and happy emotional states (Lucas & Fujita, 2000; Reisenzein et al., 2020). In contrast, neurotic individuals experience more negative states such as anxiety and anger (De Raad, 1996; Watson et al., 1988). While a large body of research has examined the relation between personality traits and academic performance, there has been a lack of research on the impact of personality on emotional engagement while learning. Since the mechanism shaping the academic learning outcomes is proposed to be self-regulation (Abe, 2020; Komarraju et al., 2009; Poropat, 2009; Sorić et al., 2017; Vedel, 2014), there remains much to be learned about the role of SRE, personality traits, and emotional engagement. To the author's best knowledge, this study is the first to examine the association of personality traits, SRE, and emotions within the VR learning context.

Research questions

1. What emotions are instigated, and to what extent and in which sequence, during a learning process with a VR simulation as depicted by facial expression analysis?

2. How is emotional engagement while learning with VR impacted by SRE and individual personality traits?

Methods

Participants

Overall, 65 nursing students participated in the study. The final sample size was 61 since four participants were excluded from the data analysis due to facial expression calibration errors. The majority of the participants were female (n = 44) and the mean age of the research sample was 23 (±5.1) years.

The university’s ethics committee approved the study (#0001776-2).

Research design and procedure

This study is part of a more extensive study. Participants’ demographics, personality traits, and SRE strategies were assessed via a questionnaire. Following calibrations required to verify that facial expressions were recorded properly, the learning session with a VR-based simulation was initiated and lasted for a mean time of 41.18 (±13.7) minutes. Facial expressions were recorded and real-time analyzed during the entire learning experience with VR. The data collection was supported by the iMotions 9.0 Biometric Research Platform (https://imotions.com).

The VR-based simulation

The VR simulation was designed using the Unity platform (https://unity.com). VR architecture is based on our previous study (Dubovi et al., 2017). The simulation provides a three-dimensional (3D) simulated world where individuals are self-represented as avatars. VR is a responsive platform in which participants can maneuver the 3D representations of a physical location or of a phenomena, which in turn supports the sensation of being immersed in a real world. There is no single type of VR, rather VR environments can be classified along the immersive continuum they offer. The technological attributes of VR differentiate between low-immersive VR which is often called desktop VR, and high-immersive VR. The desktop low-immersive VR takes the form of a window into a virtual world displayed on a computer monitor, with interactions via a mouse, keyboard, or joystick (Choi et al., 2016). During the learning process with the low-immersive desktop VR, participants are fully aware of the real physical world and therefore it is considered a safe environment in terms of having a low risk for cybersickness and immersion injury (Martirosov et al., 2022). High-immersive VR involves a high degree of interactivity and high-cost peripheral devices. Such devices comprise a head-mounted display in which a high graphical fidelity screen is mounted in front of one’s eyes with separate lenses for each eye and sound delivered through earphones. By blocking out many visual elements of the real-world environment and inducing sensory stimuli that correspond with the virtual environment, high-immersive VR enables the user to more fully immerse in the virtual environment. We chose for the current study the low-immersive VR due to recent studies that show that more immersive technology is related to lower levels of learning, suggesting that high-immersive VR incorporates more presence but less learning (Jensen & Konradsen, 2018; Makransky, Terkildsen, et al., 2019; Ning Woon et al., 2020; Parong & Mayer, 2021). The possible reason for this outcome has been explained by Markransky and colleagues (2019), who demonstrated through EEG that high-immersive VR is associated with a triggering of high cognitive load which might prohibit knowledge acquisition, as compared to desktop low-immersive VR.

The simulation was designed to model a hospital ward with two main spaces (Figure 1). The first space represented the patient room and the second space represented the medicine room with various medical equipment and medications. Clinical scenarios were designed to support procedure training and development of clinical reasoning. Deliberations regarding practice were facilitated by providing students with opportunities to assess the virtual patient for his medical condition, interpret the physician’s instructions, calculate appropriate dosages, administer medication, and monitor the medication’s side-effects. During these steps, the VR technology provided individual feedback on the student’s performance together with video directions.

Figure 1. Screenshots of the VR simulation while a student’s avatar: a. Assessing patient health status; b. Performing hand hygiene procedures; c. Choosing the right medication from the medicine cabinet.

Data collection instruments

Facial expression recognition. Affectiva Affdex algorithm, which was provided by the iMotions 9.0 (https://imotions.com) platform, allowed this study to collect and extract the 7 basic emotions of joy, anger, surprise, contempt, fear, sadness, and disgust (Ekman & Rosenberg, 1997; Kulke et al., 2020). The seven emotional dimensions can also be categorized by their valence (positive or negative) or by the absence of valence, i.e. neutral. Facial expressions analysis was at a 30 Hz frequency and was collected for the overall learning experience (Figure 2). The algorithm uses the Facial Action Coding System, which identifies “action units” and then classifies them to provide the likelihood that each of the 7 emotions will occur. Next, thresholding for time and absolute amplitude probability was carried out, based on the iMotions guidelines (https://imotions.com/blog/facial-expression-analysis/), to extract emotion percentage metrics.

Figure 2. A screenshot of the real-time facial expression data collection while a student is learning with the VR based simulation.

The Big Five Inventory (BFI). The BFI is a widely used measure of the Five-Factor Model (FFM) and yields measures of five dimensions of personality (John et al., 1991; John & Srivastava, 1999). Five personality traits: Conscientiousness (e.g. “I tend to be reliable and perform my work”); Agreeableness (e.g. “I tend to help others”); Neuroticism (e.g. “I tend to worry”); Extraversion (e.g. “I am an enthusiastic person”); and Openness (e.g. “I tend to have active imagination”). Participants had to indicate how much each phrase item pertains to them on a five-point Likert-type scale. The mean rating of all 44 items was calculated for each personality trait. The Cronbach alphas of the variables in this study were: Conscientiousness, α = .82; Agreeableness, α = .70; Neuroticism, α = .73; Extraversion, α = .79; and Openness, α = .75.

Emotion Regulation Questionnaire (ERQ). An adapted version of the ERQ was used to assess the students’ reappraisal and suppression SRE processes (Ben-Eliyahu & Linnenbrink-Garcia, 2013; Gross & John, 2003). Five items tapped into the ability to reframe the situation in order to adjust emotions (i.e. reappraisal; e.g. “I control my emotions by changing the way I think about the situation I’m in”); and four items reported on the inhibition of emotional expression strategy (i.e. suppression; e.g. “I control my emotions by not expressing them”). Items were rated on a seven-point Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree). The reliabilities were within an acceptable range, Reappraisal (α = .79) and Suppression (α = .81).

Data analysis

The data analysis is presented with respect to the two research questions.

Analysis regarding research question 1 concerning the emotions elicited during learning with VR:

A Kruskal–Wallis H test was performed to account for the descriptive statistics of the seven emotions as depicted by facial expression analysis. This test was performed due to the data’s nonnormal distribution. This was followed by a one-way repeated measures ANOVA to follow each emotion's fluctuations during the VR time phases.

Analysis regarding research question 2 concerning the impact of SRE and individual personality traits on emotions while learning with VR:

A bivariate parametric correlation analysis was used to evaluate associations between the participants’ personality traits, self-regulation, and emotional engagement in terms of facial expressions. Next, based on the results of the correlation analyses, only variables which were found to be significantly associated were selected for inclusion in the subsequent two models of hierarchical regression analyses. For a similar approach to regression see (Duffy et al., 2018; Greene et al., 2014). In Model 1 (anger), personality traits (entered first) and SRE strategies (entered second) were included to determine whether these combinations of variables significantly explain anger. In the second model (sadness), personality traits (entered first) and SRE strategies (entered second) were selected to determine whether these combinations of variables significantly explain sadness.

For the mediation analyses, based on the hierarchical multiple regression analysis results, a PROCESS (model 4) was used to assess whether reappraisal strategy mediates the association between conscientiousness, agreeableness, and engagement; and evaluate whether suppression strategy mediates the association between agreeableness and sadness. To evaluate the significance of indirect effects, we used a bootstrap estimation approach (Hayes, Montoya, & Rockwood, 2017).

The data analysis was conducted using SPSS version 27.0 software.

Results

Facial expression analysis was used to collect and extract students’ emotional engagement while learning with VR (Figure 2). The Kruskal–Wallis H test of facial expressions revealed that participants experienced significantly (χ2(6) = 314.710, p < .001) more joy 4.0±3.6% than other emotions such as surprise 1.7±2.6%, fear 0.3±0.9%, contempt 1.5±2.5%, disgust 0.3±0.6%, sadness 0.4±1.2%, and anger 0.8±1.6% (Table 1, Figure 3). A one-way repeated measures ANOVA analysis was conducted to examine the fluctuation of facial expressions throughout the VR learning process which was divided into three stages during the average time spent with VR (41.18 ± 13.7 min): initial stage (first 0–12 min), mid-stage (13–27 min), and final stage (28–42 min). This division is based on previous research by Tonguç and Ozkara (2020), who suggested splitting facial expression changes across three stages of the learning sequence. The results revealed a significant interaction between the type of emotions derived from the facial expressions analysis and the VR simulation timeline stage (F (2, 580) = 15.214, p < 0.001,). Post-hoc analysis indicated significantly more joyful expressions than other emotions in the initial stage of the VR simulation (p < 0.001; Figure 3).

Figure 3. The mean change in students’ emotions as measured by facial expression analysis (time percentage) according to the VR simulation stages (minutes).

Table 1. Bivariate intercorrelations between personality traits, SRE strategies and emotional engagement as measured by facial expressions.

	Variable	Mean (±SD)	1	2	3	4	5	6	7	8	9	10	11	12	13	14
FFM	1. Conscientiousness	3.9(±0.5)	–
	2. Agreeableness	4.1(±0.4)	0.355**	–
	3. Neuroticism	2.4(±0.6)	−0.617***	−0.500**	–
	4. Extraversion	3.3(±0.6)	0.438***	0.241	−0.433***	–
	5. Openness	3.7(±0.5)	0.185	0.238	−0.033	0.152
SRE	6. Reappraisal	5.5(±0.9)	0.368**	0.330**	−0.459***	0.275*	0.167	–
	7. Suppression	3.3(±1.3)	−0.322*	−0.364**	0.234	−0.452***	−0.043	0.012	–
Facial expressions	8. Joy (%)	4.0(±3.6)	0.183	0.033	−0.249	0.207	−0.012	0.210	−0.029	–
1.	9. Surprise (%)	1.7(±2.6)	0.109	0.043	−0.081	0.05	0.214	0.123	0.011	−0.068	–
9.	10. Fear (%)	0.3(±0.9)	−0.046	0.13	0.094	−0.033	0.122	0.006	−0.05	−0.051	0.294*	–
9.	11. Contempt (%)	1.5(±2.5)	0.074	0.066	0.099	−0.008	0.098	0.016	−0.067	0.137	−0.006	0.011	–
9.	12. Disgust (%)	0.3(±0.6)	0.124	0.057	−0.015	−0.114	0.108	0.104	0.036	0.017	0.08	−0.027	0.545***	–
9.	13. Sadness (%)	0.4(±1.2)	−0.043	−0.328*	−0.076	−0.015	−0.025	0.041	0.287*	0.081	0.013	−0.100	−0.095	0.009	–
9.	14. Anger (%)	0.8(±1.6)	−0.290*	−0.276*	0.217	−0.206	−0.303*	−0.301*	−0.031	−0.073	−0.054	−0.085	−0.021	−0.001	0.274*	–
	15. Neutral (%)	90.1(±5.9)	0.289*	0.218	−0.275*	0.114	0.227	0.179	.002	−0.745***	0.082	0.070	−0.330**	−0.135	-.020	−0.390**

Note: Data represents the Pearson r or Spearman r_s. FFM, five factor-model; SRE, self-regulated emotion

*p < .05; **p < .01; ***p <.001

Personality traits and SRE strategies were assessed by self-reported questionnaires. Results demonstrated that the highest scored personal traits were conscientiousness and agreeableness (Table 1; F (4, 240)  = 81.356, p < 0.001), and the most used SRE strategy was reappraisal (Table 1; t  = 10.803, p < 0.001). To account for the impact of personality traits on emotional engagement, a bivariate intercorrelations analysis was performed (Table 1). Results revealed that when it comes to emotional engagement, conscientiousness, agreeableness, and openness, these personality traits were negatively associated with anger (r = −0.290 p<0.05, r = −0.276 p<0.05, r = −0.303, p<0.05; respectively); and agreeableness was negatively associated with sadness (r = −0.328 p<0.05). The analysis also showed that SRE impacts emotional engagement. Particularly, there was a significant negative correlation between reappraisal and anger (r = −0.301, p < 0.05), namely, the use of a reappraisal strategy was associated with fewer anger expressions. In contrast, a suppression strategy was correlated with more sadness expressions (r = 0.287, p < 0.05).

In the context of personality traits and emotion regulation processes, as Table 1 shows, conscientiousness, agreeableness and extraversion were positively correlated with reappraisal (r = 0.368 p<0.01, r = 0.330 p<0.01, r = 0.275, p<0.05; respectively), and negatively with neuroticism (r = −0.459, p<0.001). By contrast, conscientiousness, agreeableness and extraversion personality traits were negatively associated with a suppression strategy of emotion regulation (r = −0.322 p <0.05, r = −0.364 p<0.01, r = −0.452 p<0.001; respectively).

Building upon the bivariate intercorrelation analysis findings (Table 1), only the significant intercorrelations were selected for inclusion in the two models of multivariant hierarchical regression analyses. This was to explain emotional engagement in terms of experienced anger and sadness. The hierarchical regression analysis for anger revealed that the inclusion of personality traits of conscientiousness and agreeableness accounted for 10% of the variance, F (2, 58)  = 3.397, p < .05 (Table 2). Adding the SRE strategy showed that reappraisal explained an additional 15% of the variance (the entire model explained 25%), F(2, 56) = 4.881, p < 0.05 (Table 2). Hierarchical regression analysis for sadness revealed that agreeableness accounted for 9% of the variance, F (2, 58)  =  3.198, p < .05 (Table 2). Adding the SRE strategy of suppression explained an additional 13% of the variance (the entire model explained 22%), F(2, 56) = 2.295, p < 0.5 (Table 2).

Table 2. Summary of two multivariate hierarchical regression analyses to explain emotional engagement in terms of anger and sadness (N = 61).

	Variable		Anger			Sadness
			β	t		β	t
Step 1
	Conscientiousness		−0.90	−2.328*		0.042	0.307
	Agreeableness		−0.276	−2.206*		−0.240	−2.125*
	R^2		0.105			0.092
	F		3.397*			3.198*
Step 2
	Conscientiousness		−0.16	−1.330		0.074	0.506
	Agreeableness		−0.15	−1.624		−0.188	−1.279
	Reappraisal		−0.31	−2.765*		0.03	0.278
	Suppression		−0.03	−0.217		0.286	2.295*
	R^2		0.251			0.217
	F		4.881*			4.371*

*p < .05

The hierarchical multiple regression analysis proposed two mediation effects: the first is the indirect effect of reappraisal on the association of conscientiousness and agreeableness with anger; and the second is the indirect effect of suppression on the association of agreeableness with sadness. Based on the above results, we used Baron and Kenny’s method (Baron & Kenny, 1986) to test the simultaneous direct and indirect effects for each of the two models using the PROCESS (model 4) with the bootstrap estimation approach with 5000 samples (Hayes, 2012, 2017). The first model is shown in Figure 4. The results indicated that the indirect coefficients were significant for reappraisal with both independent variables of conscientiousness (b = −0.1007, 95% CI = −0.2205 to −0.0175; Figure 4) and agreeableness (b = −.0819, 95% CI = −0.1800 to −0.0030; Figure 4). The second model is shown in Figure 5. Notably, also the association of agreeableness with sadness was fully mediated by the indirect effect of suppression strategy (b = −.0895, 95% CI = −0.1983 to −0.0101; Figure 5).

Figure 4. A mediation model for demonstrating the impact of the reappraisal strategy on the association between conscientiousness and agreeableness personality traits and anger.

Figure 5. A mediation model to demonstrate the impact of the suppression maladaptive SRE strategy on the association between the agreeableness personality trait and sadness.

Discussion

The main goal of this study was to examine how emotions are being instigated during a learning process with VR, and are being shaped by intraindividual components of personality tendencies and SRE strategies. The study’s main findings point out that personality does matter when it comes to emotional engagement, and that SRE strategies of reappraisal and suppression play an important mediation role in experiencing less negative emotional engagement. Thus, this study advances our understanding of how personality shapes emotional engagement while learning with VR by the use of emotional SRE strategies.

Overall, the granular, continuous data source of emotional engagement using automatic recognition of students’ facial expressions revealed that joy (i.e. enjoyment or happiness) was the most predominant emotion that students expressed. This research finding is in line with recent studies that showed that joy expressions were triggered with higher intensity by technology-based educational tools, such as games and robot tutors, than by instructional traditional approaches such as lecture (Ninaus et al., 2019; Tonguç & Ozkara, 2020; Velentza et al., 2021). In addition, the current study results showed that the frequency of the joy expression changed across the different VR learning stages. Particularly, the joy expression was more intense during the first 0–12 min of the VR learning session compared to later stages. This rapid and sharp increase in the joy expressions at the beginning of a learning interaction with VR can be explained by the novelty of the VR medium (Jeno et al., 2019). Despite the observation that facial expressions of joy were the most frequent emotion expressed by students, there were no significant associations between joy and the FMM taxonomy-based personality tendencies nor the SRE strategies.

Factors that influence emotional engagement with VR

The facial expressions of anger and sadness, which were expressed at a relatively low frequency with no significant changes across the learning session, were significantly associated with personality dispositions and emotion regulation strategies. These findings suggest that students with a high conscientiousness personality trait tended to experience less anger and participants with a high agreeableness personality trait tended to experience less anger and sadness. Importantly, two mediation effects emerged in the analysis which proposes that the mechanism that shapes the experience of negative emotional engagement of conscientiousness and agreeableness learners were the SRE strategies. Findings demonstrate that learners with high scoring characteristics of conscientiousness and agreeableness seemed to employ significantly more of the adaptive SRE strategy of reappraisal, which in turn resulted in experiencing less anger. In addition, learners with high scoring agreeableness characterized by the traits of compliance and tendermindedness also tended to use less of the maladaptive SRE strategy of suppression, which in turn resulted in less sadness.

These results expand the knowledge on what is known about emotional engagement within the learning context. Until now, this understanding was predominately considered as a by-product of achievement motivations across different types of learning contexts and course preferences (Ben-Eliyahu, 2019; Ben-Eliyahu & Linnenbrink-Garcia, 2013; D’Mello & Graesser, 2012; Pekrun, 2006; Pintrich, 2004). More recently there has been a focus on targeting emotional engagement as an interdependent outcome of personality and interpersonal emotion regulation strategies (Ben-Eliyahu & Zeidner, 2020). While individual personality traits are relatively stable and immutable (Bleidorn et al., 2021; Stieger et al., 2021), previous studies have demonstrated that SRE strategies can be enhanced by counselling and training (Cohen & Mor, 2018; Hurtado-Parrado et al., 2019; Kim et al., 2009; Volkaert et al., 2020). Thus, the current study’s findings suggest that the magnitude of negative emotional experiences can be altered by impacting emotion regulation preferences. Since the CVT theory categorizes negative emotions as experiences which are expected to hinder learning (Camacho-Morles et al., 2021; Pekrun & Perry, 2014), the current study suggests that SRE strategies should be explicitly taught in schools to combat such negative experiences. Future research should investigate how students’ propensity to use the adaptive SRE strategy of reappraisal can be supported and how this can impact students’ tendency to experience negative emotions while learning.

The intraindividual components of personality and the SRE link

The students reported using more reappraisal strategies than suppression strategies, and this was associated with their personality traits. Notably, students who had higher conscientiousness, agreeableness, and extraversion tended to use more reappraisal strategies and less of the maladaptive suppression strategies. These findings are consistent with the Barańczuk (2019) metanalysis which showed that the characteristics of conscientiousness such as being disciplined, diligent, and striving for goals are positively linked to the adaptive reappraisal strategy and negatively to the suppression strategy (Connor-Smith & Flachsbart, 2007). Furthermore, the association of higher agreeableness with a higher use of the reappraisal strategy and a lower maladaptive use of the suppression strategy was demonstrated and theoretically explained previously (John & Srivastava, 1999; Reisenzein et al., 2020). Barańczuk (2019) proposed that the link to greater compliance, empathy, and altruism that agreeable individuals are characterized by, forms a more supportive emotion socialization process that might explain the relationship between agreeableness and emotion regulation strategies.

The findings showed that being more extravert was related to a greater use of adaptive SRE strategies. Previous studies have shown that extroverted individuals are more enthusiastic, sociable and problem solving, which in turn increases the possibility of greater typically adaptive and lower typically maladaptive emotion regulation strategies (Barańczuk, 2019; Connor-Smith & Flachsbart, 2007). Moreover, in contrast to conscientiousness, agreeableness and extraversion, neuroticism was found to be negatively related to reappraisal use. Considering the avoidance, low emotional stability and behavioral inhibition characteristics of neuroticism, it might be supposed that students who declare greater neuroticism may use more strategies that cut them off from the negative experiences through using avoidance rather than adaptive strategies (Barańczuk, 2019; De Raad, 1996).

This study’s finding of an association between personality traits and the use of SRE strategies therefore reveals that interpersonal characteristics have a pivotal role in students’ emotional engagement while learning.

Limitations and future research

There are some limitations of the current study. Previous research has suggested that negative facial expressions are quicker to appear and thus facial expressions are more sensitive to detecting negative emotional valence (van Bommel et al., 2020; Zeinstra et al., 2009). Thus, further studies should also incorporate other emotional engagement measurements to illuminate the role of positive emotions. Moreover, the current analysis focused on the aggregate facial expressions view, further studies should also incorporate the individual perspective of facial expressions coding (Liaw et al., 2021; Sawyer et al., 2017). For example, it is important to understand what exactly, from the individual perspective, triggered a facial expression of anger when interacting with VR, using the bottom-up approach. It is also important to unfold how emotions are manifested during specific events in the simulation, for example when negative feedback is received due to drug dosage miscalculation. Finally, it is important to note that in the context of CVT theory, there is an ongoing debate regarding the effect of activating the negative emotion of anger on learning outcomes (Ahn & Harley, 2020; Pekrun et al., 2009). Therefore, the effect of anger as a precursor of deeper cognitive processing should be further studied.

Conclusion

Using facial expression analysis to detect students’ emotional dynamics moment-by-moment while learning with VR demonstrated that overall, joy was the most predominant emotion expressed by students. However, in the case of the tendency to negative emotions, this study shows that intraindividual components of personality traits and SRE strategies might color the entire gamut of student emotional engagement. While personality traits are relatively stable and immutable, SRE strategies can be altered and supported.

A body of recent psychology research shows that VR can be used for the development of adaptive SRE strategies to modify the emotional response (Colombo et al., 2021; Montana et al., 2020). We therefore suggest that further research should evaluate the utilization of explicit SRE strategy prompts to benefit students’ emotional experiences while learning with VR (Price et al., 2018). Moreover, multimodal software such as for unintrusive facial expression analysis can be used to analyze in real-time emotional experiences to provide personalized automated SRE strategy prompts and to evaluate their impact (Wiedbusch et al., 2021). Since the research in this area is in its infancy, it is important to further develop our understanding of how to design explicit prompts to help learners be more capable of regulating their affective states and increasing learning outcomes.

Acknowledgements

The author would like to thank engineer Sergey Bondarchuck with whom the VR learning environment was designed and programmed, Liat Liberman for assisting with data collection.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported, in part, by the Joy Ventures Academic Grant .

Notes on contributors

Ilana Dubovi

Ilana Dubovi is a learning science researcher at the Nursing Department at Tel Aviv University. Her work spans the fields of educational technology, lifelong learning, cognition and emotion in learning processes, and learning analytics.