A Multimodal Human-Computer Interaction for Smart Learning System

Abstract

The rise of digitalization and computing devices has transformed the educational landscape, making traditional teaching methods less productive. In this context, early and continuous user interaction is crucial for designing and developing effective learning applications. The field of Human-Computer Interaction (HCI) has seen significant technological growth, enabling educators to provide quality educational services through smart input and output channels. However, to prevent students from discontinuing their studies and help them grow their careers, a multimodal HCI approach is needed. This paper proposes a multimodal deep learning multi-layer Convolutional Neural Network (CNN) to improve the educational experience. Our designed system aims to create a promising solution for improving the educational experience and enabling educators to provide high-quality educational services to students. Our implementation results show promising real-time performances, including a high success rate in a constriction learning concept, a quality interaction experience, and enhanced educational services. We evaluated the accuracy of five multimodal inputs, including Finger Touch (FT), Hands Up (HU), Hands Down (HD), Voice Command (VC), and Click/Typing (CT). The results indicate an average accuracy of 90.8%, 87%, 88.6%, 91.8%, and 87%, respectively, demonstrating the effectiveness of our proposed approach.

Keywords:

• Human-Computer Interaction
• multimodal HCI
• multilayer CNN
• deep learning model
• smart Learning System

View correction statement:

Correction

Ethical approval

We did not use animals and Human participants in the study reported in this work.

Consent form

For this type of study consent for publication is not required.

Informed consent

For this type of study informed consent is not required.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article was originally published with errors, which have now been corrected in the online version. Please see Correction (http://dx.doi.org/10.1080/10447318.2023.2220184)

Additional information

Funding

The authors did not receive support from any organization for the submitted work.

Notes on contributors

Tareq Mahmod Alzubi

Tareq M. Alzubi is an Assistant Professor at Al-Balqa Applied University, Jordan. Received PhD degree in Computer Graphics from Santiago De Compostela University, Spain. Tareq works and researches in multi-disciplinary fields involving Human-Computer Interaction, Computer Vision, and Machine Learning.

Jafar A. Alzubi

Jafar A. Alzubi is an Associate Professor at Al-Balqa Applied University, Jordan. Received PhD degree in Advanced Telecommunications from Swansea University - UK. Jafar works and researches in a multi and interdisciplinary environment involving Machine Learning, Cyber Security, and Networks Security. which resulted in publishing more than sixty articles.

Ashish Singh

Ashish Singh is an Assistant Professor, the School of Computer Engineering, Kalinga Institute of Industrial Technology, Bhubaneswar, Odisha. He has completed his M.Tech and PhD degree from NIT Patna, India. He is working in the field of Healthcare Security, Edge Computing, Cloud Computing, and the Internet of Things.

Omar A. Alzubi

Omar A. Alzubi received his PhD degree in Computer and Network Security from Swansea University, United Kingdom. Currently, he is a professor at Al-Balqa Applied University, Jordan. Professor Alzubi research interests include cyber security, machine learning, and networks. His cumulative research experience resulted in the publishing more than fifty articles.

Murali Subramanian

Murali Subramanian received his PhD from VIT University in 2017. He is presently serving her alma mater as an Associate Professor. He has published around 25 papers in various International Journals. His research interests include Data Aggregation in Wireless Sensor Network, Data analysis, Artificial intelligence & Biologically Inspired Optimization Algorithms.