Enhancing Online Learning: A Multimodal Approach for Cognitive Load Assessment

References

• Al-Nafjan, A., & Aldayel, M. (2022). Predict students’ attention in online learning using EEG data. Sustainability, 14(11), 6553. https://doi.org/10.3390/su14116553
   Web of Science ®Google Scholar
• Andrzejewska, M., & Skawińska, A. (2020). Examining students’ intrinsic cognitive load during program comprehension – an eye tracking approach (pp. 25–30). Artificial Intelligence in Education.
   Google Scholar
• Baltrušaitis, T., Robinson, P., & Morency, L. (2016). OpenFace: An open source facial behavior analysis toolkit. https://doi.org/10.1109/wacv.2016.7477553
   Google Scholar
• Barsoum, E., Zhang, C., Ferrer, C. C., & Zhang, Z. (2016). Training deep networks for facial expression recognition with crowd-sourced label distribution [Paper presentation]. In Proceedings of the 18th ACM international conference on multimodal interaction, Tokyo, Japan. https://doi.org/10.1145/2993148.2993165
   Google Scholar
• Bon, O. L., Neu, D., Berquin, Y., Lanquart, J. P., Hoffmann, R., Mairesse, O., & Armitage, R. (2012). Ultra-Slow delta power in chronic fatigue syndrome. Psychiatry Research, 200(2–3), 742–747. https://doi.org/10.1016/j.psychres.2012.06.027
   PubMed Web of Science ®Google Scholar
• Bounhar, T., Yamak, Z., H. V., & Baudry, D. (2022). A dataset and methodology for self-efficacy feeling prediction during industry 4.0 VR activity [Paper presentation]. In Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Christchurch, New Zealand, pp. 176–182.
   Google Scholar
• Bott, N. T., Madero, E. N., Glenn, J. M., Lange, A. R., Anderson, J. J., Newton, D., Brennan, A., Buffalo, E. A., Rentz, D. M., & Zola, S. M. (2020). Device-embedded cameras for eye tracking- based cognitive assessment: Implications for teleneuropsychology. Telemedicine Journal and e-Health, 26(4), 477–481. https://doi.org/10.1089/tmj.2019.0039
   PubMed Web of Science ®Google Scholar
• Chan, P. K., Zhou, T., Tang, S., & Huang, Z. (2022). Attentional bias to negative emotions in anxious individuals: An eye tracking study [Paper presentation]. In 2022 13th International Conference on E-Business, Management and Economics, Beijing, China.
   Google Scholar
• Chen, C., Pedersen, S., & Murphy, K. L. (2011). Learners’ perceived information overload in online learning via computer-mediated communication. Research in Learning Technology, 19(2), 101–116. https://doi.org/10.3402/rlt.v19i2.10345
   Google Scholar
• Chen, N., Ko, H., Kinshuk, K., & Lin, T. (2005). A model for synchronous learning using the Internet. Innovations in Education and Teaching International, 42(2), 181–194. https://doi.org/10.1080/14703290500062599
   Web of Science ®Google Scholar
• Cooper, G. (1990). Cognitive load theory as an aid for instructional design. Australasian Journal of Educational Technology, 6(2), 108–113. https://doi.org/10.14742/ajet.2322
   Google Scholar
• Debie, E., Rojas, R. F., Fidock, J., Barlow, M., Kasmarik, K., Anavatti, S. G., Garratt, M., & Abbass, H. A. (2021). Multimodal fusion for objective assessment of cognitive workload: A review. IEEE Transactions on Cybernetics, 51(3), 1542–1555. https://doi.org/10.1109/TCYB.2019.2939399
   PubMed Web of Science ®Google Scholar
• Dong, S., Kim, B., & Lee, S. Y. (2016). EEG-based classification of implicit intention during self-relevant sentence reading. IEEE Transactions on Cybernetics, 46(11), 2535–2542. https://doi.org/10.1109/TCYB.2015.2479240
   PubMed Web of Science ®Google Scholar
• Duchowski, A. T., Krejtz, K., Krejtz, I., Biele, C., Niedzielska, A., Kiefer, P., Raubal, M., & Giannopoulos, I. (2018). The index of pupillary activity: Measuring cognitive load vis-à-vis task difficulty with pupil oscillation [Paper presentation]. In Proceedings of the 2018 CHI conference on human factors in computing systems, Montreal, QC, Canada.
   Google Scholar
• Gammulle, H., Denman, S., Sridharan, S., & Fookes, C. (2021). TMMF: temporal multi-modal fusion for single-stage continuous gesture recognition. IEEE Transactions on Image Processing, 30, 7689–7701. https://doi.org/10.1109/TIP.2021.3108349
   Google Scholar
• Gharagozlou, F., Saraji, G. N., Mazloumi, A., Nahvi, A., Nasrabadi, A. M., Foroushani, A. R., Kheradmand, A. A., Ashouri, M., & Samavati, M. (2015). Detecting driver mental fatigue based on EEG alpha power changes during simulated driving. PubMed, 44(12), 1693–1700. https://pubmed.ncbi.nlm.nih.gov/26811821
   Google Scholar
• Goodwin, T. R., & Harabagiu, S. M. (2016). Multi-modal patient cohort identification from EEG report and signal data. PubMed, 2016, 1794–1803. https://pubmed.ncbi.nlm.nih.gov/28269938.
   Google Scholar
• Hart, S. G. (2006). NASA-task load index (NASA-TLX); 20 years later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 50(9), 904–908. https://doi.org/10.1177/154193120605000909
   Google Scholar
• Herbig, N., Pal, S., Krüger, A., & Genabith, J. (2021). Multi-modal estimation of cognitive load in post-editing of machine translation. In Tra&Co Group (Ed.), Translation, interpreting, cognition: The way out of the box (pp. 1–32). Language Science Press. https://doi.org/10.5281/zenodo.4545029
   Google Scholar
• Hussain, M. S., Calvo, R. A., & Chen, F. (2013). Automatic cognitive load detection from face, physiology, task performance and fusion during affective interference. Interacting with Computers, 26(3), 256–268. https://doi.org/10.1093/iwc/iwt032
   Web of Science ®Google Scholar
• Kaplan-Rakowski, R., Lin, L., & Wojdynski, T. (2021). Learning vocabulary using 2D pictures is more effective than using immersive 3D stereoscopic pictures. International Journal of Human–Computer Interaction, 38(4), 299–308. https://doi.org/10.1080/10447318.2021.1938394
   Web of Science ®Google Scholar
• Khedher, a B., Jraidi, I., & Frasson, C. (2019). Predicting learners’ performance using EEG and Eye tracking features. The Thirty-Second International Florida Artificial Intelligence Research Society Conference (FLAIRS-32) (pp. 396–401). Association for the Advancement of Artificial Intelligence.
   Google Scholar
• Kirschner, P. A., Sweller, J., Kirschner, F., & Zambrano, R. J. (2018). From cognitive load theory to collaborative cognitive load theory. International Journal of Computer-Supported Collaborative Learning, 13(2), 213–233. https://doi.org/10.1007/s11412-018-9277-y
   PubMed Web of Science ®Google Scholar
• Lahat, D., Adali, T., & Jutten, C. (2015). Multimodal data fusion: An overview of methods, challenges, and prospects. Proceedings of the IEEE, 103(9), 1449–1477. https://doi.org/10.1109/JPROC.2015.2460697
   Web of Science ®Google Scholar
• Li, G., Huang, S., Xu, W., Jiao, W., Jiang, Y., Gao, Z., & Zhang, J. (2020). The impact of mental fatigue on brain activity: A comparative study both in resting state and task state using EEG. BMC Neuroscience, 21(1), 1–9. https://doi.org/10.1186/s12868-020-00569-1
   Web of Science ®Google Scholar
• Li, Y., Yu, A. W., Meng, T., Caine, B., Ngiam, J., Peng, D., Shen, J., Wu, B., Lu, Y., Zhou, D., Le, Q. V., Yuille, A., & Tan, M. (2022). DeepFusion: LiDAr-camera deep fusion for multi-modal 3D object detection [Paper presentation]. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.
   Google Scholar
• Liu, L., Zhang, L., Shu, Y., Wang, S., Wu, H., & Xv, M. (2023). Evaluation of a collaborative reading annotation system through multimodal data analysis [Paper presentation]. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, Hamburg, Germany.
   Google Scholar
• Liu, Y., Zhan, P., Fu, Y., Chen, Q., Man, K., & Luo, Y. (2023). Using a multi-strategy eye-tracking psychometric model to measure intelligence and identify cognitive strategy in Raven’s advanced progressive matrices. Intelligence, 100, 101782. https://doi.org/10.1016/j.intell.2023.101782
   Google Scholar
• Luo, S., Lan, Y., Peng, D., Li, Z., Zheng, W., & Lu, B. (2022). Multimodal emotion recognition in response to oil paintings [Paper presentation]. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Glasgow, Scotland, United Kingdom.
   Google Scholar
• Maqableh, W., Alzyoud, F. Y., & Zraqou, J. (2023). The use of facial expressions in measuring students’ interaction with distance learning environments during the COVID-19 crisis. Visual Informatics, 7(1), 1–17. https://doi.org/10.1016/j.visinf.2022.10.001
   PubMedGoogle Scholar
• Martinez‐Marquez, D., Pingali, S., Panuwatwanich, K., Stewart, R. A., & Mohamed, S. a M. (2021). Application of eye tracking technology in aviation, maritime, and construction industries: A systematic review. Sensors, 21(13), 4289. https://doi.org/10.3390/s21134289
   PubMed Web of Science ®Google Scholar
• Mazher, M., Aziz, A. A., Malik, A. S., & Amin, H. U. (2017). An EEG-based cognitive load assessment in multimedia learning using feature extraction and partial directed coherence. IEEE Access. 5, 14819–14829. https://doi.org/10.1109/ACCESS.2017.2731784
   Web of Science ®Google Scholar
• Örün, Ö., & Akbulut, Y. (2019). Effect of multitasking, physical environment and electroencephalography use on cognitive load and retention. Computers in Human Behavior, 92, 216–229. https://doi.org/10.1016/j.chb.2018.11.027
   Web of Science ®Google Scholar
• Paas, F., Tuovinen, J. E., Tabbers, H. K., & Van Gerven, P. W. M. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist, 38(1), 63–71. https://doi.org/10.1207/S15326985EP3801_8
   Web of Science ®Google Scholar
• Paas, F., & Van Merriënboer, J. J. G. (1994). Instructional control of cognitive load in the training of complex cognitive tasks. Educational Psychology Review, 6(4), 351–371. https://doi.org/10.1007/BF02213420
   Web of Science ®Google Scholar
• Qiao, P., Zhu, X., Guo, Y., Sun, Y., & Qin, C. (2021). The development and adoption of online learning in pre- and post-COVID-19: Combination of technological system evolution theory and unified theory of acceptance and use of technology. Journal of Risk and Financial Management, 14(4), 162. https://doi.org/10.3390/jrfm14040162
   Web of Science ®Google Scholar
• Sathik, M. M., & Jonathan, S. G. (2013). Effect of facial expressions on student’s comprehension recognition in virtual educational environments. SpringerPlus, 2(1), 1–9. https://doi.org/10.1186/2193-1801-2-455
   PubMedGoogle Scholar
• Skaramagkas, V., Giannakakis, G., Ktistakis, E., Manousos, D., Karatzanis, I., Tachos, N. S., Tripoliti, E. E., Marias, K., Fotiadis, D. I., & Tsiknakis, M. (2023). Review of eye tracking metrics involved in emotional and cognitive processes. IEEE Reviews in Biomedical Engineering, 16, 260–277. https://doi.org/10.1109/RBME.2021.3066072
   PubMedGoogle Scholar
• Souchet, A. D., Philippe, S., Lourdeaux, D., & Leroy, L. (2021). Measuring visual fatigue and cognitive load via eye tracking while learning with virtual reality head-mounted displays: A review. International Journal of Human–Computer Interaction, 38(9), 801–824. https://doi.org/10.1080/10447318.2021.1976509
   Web of Science ®Google Scholar
• Sulaiman, N., Ismail, N., Islam, M. N., Rashid, M., Jadin, M. S., Mustafa, M., & Samsuri, F. (2022). Development of EEG-based system to identify student learning attention ability. Proceedings of the 12th National Technical Seminar on Unmanned System Technology 2020 (pp. 627–639). Springer.
   Google Scholar
• Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. https://doi.org/10.1207/s15516709cog1202_4
   Web of Science ®Google Scholar
• Tasmi, T., & Mohammad Zavid, P. (2021). Cognitive load measurement based on EEG signals. In T. Simon George (Ed.), The science of emotional intelligence (pp. Ch. 7). IntechOpen eBooks. https://doi.org/10.5772/intechopen.96388
   Google Scholar
• Tong, S., & Nie, Y. (2022). Measuring designers’ cognitive load for timely knowledge push via eye tracking. International Journal of Human–Computer Interaction, 39(6), 1230–1243. https://doi.org/10.1080/10447318.2022.2057898
   Web of Science ®Google Scholar
• Uysal, M. P. (2015). Evaluation of learning environments for object-oriented programming: Measuring cognitive load with a novel measurement technique. Interactive Learning Environments, 24(7), 1590–1609. https://doi.org/10.1080/10494820.2015.1041400
   Web of Science ®Google Scholar
• Wang, C., & Guo, J. (2019). A data-driven framework for learners’ cognitive load detection using ECG-PPG physiological feature fusion and XGBoost classification. Procedia Computer Science, 147(c), 338–348. https://doi.org/10.1016/j.procs.2019.01.234
   Google Scholar
• Warrick, A. (2021). Strategies for reducing cognitive overload in the online language learning classroom. International Journal of Second and Foreign Language Education, 1(2), 25–37. https://doi.org/10.33422/ijsfle.v1i2.124
   Google Scholar
• Wilson, J., Nair, S., Scielzo, S., & Larson, E. C. (2021). Objective measures of cognitive load using deep multi-modal learning. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(1), 1–35. https://doi.org/10.1145/3448111
   Google Scholar
• Winn, B., Whitaker, D., Elliott, D. B., & Phillips, N. J. (1994). Factors affecting light-adapted pupil size in normal human subjects. PubMed, 35(3), 1132–1137. https://pubmed.ncbi.nlm.nih.gov/8125724
   Google Scholar
• Yadegaridehkordi, E., Noor, N. F. M., Ayub, M. N. B., Affal, H., & Hussin, N. (2019). Affective computing in education: A systematic review and future research. Computers & Education, 142(c), 103649. https://doi.org/10.1016/j.compedu.2019.103649
   Web of Science ®Google Scholar
• Zhang, B., Chai, C., Yin, Z., & Shi, Y. (2021). Design and implementation of an EEG-based learning- style recognition mechanism. Brain Sciences, 11(5), 613. https://doi.org/10.3390/brainsci11050613
   PubMed Web of Science ®Google Scholar
• Zhang, J., Feng, F., Han, T., Gong, X., & Duan, F. (2022). Detection of autism spectrum disorder using fMRI functional connectivity with feature selection and deep learning. Cognitive Computation, 15(4), 1106–1117. https://doi.org/10.1007/s12559-021-09981-z
   Web of Science ®Google Scholar
• Zhang, Y., Guo, R., Peng, Y., Kong, W., Nie, F., & Lu, B. (2022). An auto-weighting incremental random vector functional link network for EEG-based driving fatigue detection. IEEE Transactions on Instrumentation and Measurement, 71, 1–14. https://doi.org/10.1109/TIM.2022.3216409
   Web of Science ®Google Scholar