Towards multimodal emotion recognition in e-learning environments

Abstract

This paper presents a framework (FILTWAM (Framework for Improving Learning Through Webcams And Microphones)) for real-time emotion recognition in e-learning by using webcams. FILTWAM offers timely and relevant feedback based upon learner's facial expressions and verbalizations. FILTWAM's facial expression software module has been developed and tested in a proof-of-concept study. The main goal of this study was to validate the use of webcam data for a real-time and adequate interpretation of facial expressions into extracted emotional states. The software was calibrated with 10 test persons. They received the same computer-based tasks in which each of them were requested 100 times to mimic specific facial expressions. All sessions were recorded on video. For the validation of the face emotion recognition software, two experts annotated and rated participants’ recorded behaviours. Expert findings were contrasted with the software results and showed an overall value of kappa of 0.77. An overall accuracy of our software based on the requested emotions and the recognized emotions is 72%. Whereas existing software only allows not-real time, discontinuous and obtrusive facial detection, our software allows to continuously and unobtrusively monitor learners' behaviours and converts these behaviours directly into emotional states. This paves the way for enhancing the quality and efficacy of e-learning by including the learner's emotional states.

Keywords:

• e-learning
• human–computer interaction
• multimodal emotion recognition
• real-time face emotion recognition
• webcam

Acknowledgements

We thank our colleagues who participated in the face emotion recognition proof-of-concept study. We also thank the raters for their help in rating and analysing the data sets. We thank Jason Saragih for permission to develop the affective computing software based on his FaceTracker software (Saragih et al., 2010). This research has been sponsored by the Netherlands Laboratory for Lifelong Learning (NELLL) of the Open University of the Netherlands.

Notes on contributors

Kiavash Bahreini is a computer scientist with an interest in affective computing, human–computer interaction, machine learning, real-time applications, communication skills, and e-learning applications; he is a Doctoral researcher in the Centre for Learning Sciences and Technologies (CELSTEC) at the Open University of the Netherlands.

Rob Nadolski is an educational technologist with an interest in enhancing learner support facilities, e-learning applications, complex cognitive skills; he is an Assistant Professor in the Centre for Learning Sciences and Technologies (CELSTEC) at the Open University of the Netherlands.

Wim Westera is an educational media researcher with an interest in serious gaming and simulation; he is a Full Professor in the Centre for Learning Sciences and Technologies (CELSTEC) at the Open University of the Netherlands.