Facial emotion detection using thermal and visual images based on deep learning techniques

References

• Elliott EA, Jacobs AM. Facial expressions, emotions, and sign languages. Front Psychol. 2013;4(115):1–4. doi:10.3389/fpsyg.2013.00115.
   PubMedGoogle Scholar
• Mehrabian A. Communication without words. Psychol Today. 1968;2:53–56. doi:10.4324/9781315080918-15.
   Google Scholar
• Kaiser J, Crespo-Llado MM, Turati C, et al. The development of spontaneous facial responses to others’ emotions in infancy: an EMG study. Sci Rep. 2017;7(1):17500. doi:10.1038/s41598-017-17556-y.
   PubMed Web of Science ®Google Scholar
• He Z, Li Z, Yang F, et al. Advances in multimodal emotion recognition based on brain-computer interfaces. Brain Sci. 2020;10(10):687. doi:10.3390/brainsci10100687.
   PubMed Web of Science ®Google Scholar
• Spezialetti M, Placidi G, Rossi S. Emotion recognition for human-robot interaction: recent advances and future perspectives. Front Robot AI. 2020;7:532279. doi:10.3389/frobt.2020.532279.
   PubMed Web of Science ®Google Scholar
• Alzubaidi L, Zhang J, Humaidi AJ, et al. Review of deep learning: concepts, CNN architectures, challenges, applications, future directions. J Big Data. 2021;8(1):53. doi:10.1186/s40537-021-00444-8.
   PubMedGoogle Scholar
• Yamashita R, Nishio M, Togashi K, et al. Convolutional neural networks: an overview and application in radiology. Insights Imaging. 2018;9(4):611–629. doi:10.1007/s13244-018-0639-9.
   PubMed Web of Science ®Google Scholar
• Hammal Z, Couvreur L, Caplier A, et al. Facial expression classification: An approach based on the fusion of facial deformations using the transferable belief model. Int J Approx Reason. 2007;46(3):542–567. doi:10.1016/j.ijar.2007.02.003.
   Web of Science ®Google Scholar
• Kyperountas M, Tefas A, Pitas L. Salient feature and reliable classifier selection for facial expression classification. Pattern Recognit. 2010;43(3):972–986. doi:10.1016/j.patcog.2009.07.007.
   Web of Science ®Google Scholar
• Ojo AK, Idowu TO. Improved model for facial expression classification for fear and sadness using local binary pattern histogram. J Adv Math Comput Sci. 2020;35(5):22–33. doi:10.9734/jamcs/2020/v35i530279.
   Google Scholar
• Lahiri BB, Bagavathiappan S, Jayakumar T, et al. Medical applications of infrared thermography: A review. Infrared Phys Technol. 2012;55(4):221–235. doi:10.1016/j.infrared.2012.03.007.
   PubMed Web of Science ®Google Scholar
• Yoshitomi Y, Miyawaki N, Tomita S, et al. Facial expression recognition using thermal image processing and neural network. Proceedings of the 6th IEEE International Workshop on Robot and Human Communication; RO-MAN’97 SENDAI. 2002;46:542–567. https://doi.org/10.1109/ROMAN.1997.647016.
   Google Scholar
• Goulart C, Valadão C, Delisle-Rodriguez D, et al. Visual and thermal image processing for facial specific landmark detection to infer emotions in a child-robot interaction. Sensors. 2019;19(13):2844. doi:10.3390/s19132844.
   PubMed Web of Science ®Google Scholar
• Prabhakaran AK, Nair JJ, Sarath S. Thermal facial expression recognition using modified ResNet152. In: Thampi SM, Gelenbe E, Atiquzzaman M, Chaudhary V, Li KC, editor. Advances in computing and network communications. Lecture notes in electrical engineering, vol 736. Singapore: Springer; 2021. p. 389–396. doi:10.1007/978-981-33-6987-0_32.
   Google Scholar
• Bhattacharyya A, Chatterjee S, Sen S, et al. A deep learning model for classifying human facial expressions from infrared thermal images. Sci Rep. 2021;11(1):20696. doi:10.1038/s41598-021-99998-z.
   PubMed Web of Science ®Google Scholar
• Siam AI, Soliman NF, Algarni AD, et al. Deploying machine learning techniques for human emotion detection. Comput Intell Neurosci. 2022;2022:1–16. doi:10.1155/2022/8032673.
   Google Scholar
• LeCun Y, Boser B, Denker JS, et al. Backpropagation applied to handwritten zip code recognition. Neural Comput. 1989;1(4):541–551. doi:10.1162/neco.1989.1.4.541.
   Web of Science ®Google Scholar
• Shamsaldin AS, Fattah P, Rashid TA, et al. The study of the convolutional neural networks applications. UKH J Sci Eng. 2019;3(2):32–40. doi:10.25079/ukhjse.v3n2y2019.
   Google Scholar
• Sarvamangala DR, Kulkarni RV. Convolutional neural networks in medical image understanding: a survey. Evol Intell. 2022;15(1):1–22. doi:10.1007/s12065-020-00540-3.
   PubMed Web of Science ®Google Scholar
• Sun L, Jiang Z, Chang Y, et al. Building a patient-specific model using transfer learning for four-dimensional cone beam computed tomography augmentation. Quant Imaging Med Surg. 2021;11(2):540–555. doi:10.21037/qims-20-655.
   PubMed Web of Science ®Google Scholar
• Li X, Shen X, Zhou Y, et al. Classification of breast cancer histopathological images using interleaved DenseNet with SENet (IDSNet). PloS one. 2020;15(5):e0232127. doi:10.1371/journal.pone.0232127.
   PubMed Web of Science ®Google Scholar
• Nirthika R, Manivannan S, Ramanan A, et al. Pooling in convolutional neural networks for medical image analysis: a survey and an empirical study. Neural Comput Appl. 2022;34(7):5321–5347. doi:10.1007/s00521-022-06953-8.
   PubMed Web of Science ®Google Scholar
• He K, Zhang X, Ren S, et al. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016:770–778. https://doi.org/10.1109/CVPR.2016.90.
   Google Scholar
• Yang D, Martinez C, Visuña L, et al. Detection and analysis of COVID-19 in medical images using deep learning techniques. Sci Rep. 2021;11:19638. doi:10.1038/s41598-021-99015-3.
   PubMedGoogle Scholar
• Al-Haija QA, Adebanjo A. Breast cancer diagnosis in histopathological images using ResNet-50 convolutional neural network. IEEE international IOT, Electronics and Mechatronics Conference (IEMTRONICS). 2020: 1–7. doi:10.1109/IEMTRONICS51293.2020.9216455.
   Google Scholar
• Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 3rd International Conference on Learning Representations (ICLR); 2015 May 7-9; San Diego, CA, USA. http://arxiv.org/abs/1409.1556.
   Google Scholar
• Khan MA, Rajinikanth V, Satapathy SC, et al. VGG19 network assisted joint segmentation and classification of lung nodules in CT images. Diagnostics. 2021;11(12):2208. doi:10.3390/diagnostics11122208.
   Web of Science ®Google Scholar
• Garg T, Garg M, Mahela OP, et al. Convolutional neural networks with transfer learning for recognition of COVID-19: A comparative study of different approaches. AI. 2020;1(4):586–606. doi:10.3390/ai1040034.
   Google Scholar
• Yolcu G, Oztel I, Kazan S, et al. Facial expression recognition for monitoring neurological disorders based on convolutional neural network. Multimed Tools Appl. 2019;78(22):31581–31603. doi:10.1007/s11042-019-07959-6.
   PubMed Web of Science ®Google Scholar
• Towardsdatascience [Internet]. Ontario, Canada: Medium; [cited 2023 Jan 27]. Available from: https://towardsdatascience.com/error-analysis-in-neural-networks-6b0785858845.
   Google Scholar
• uWaterloo Voice [Internet]. Ontario, Canada: Medium; [cited 2023 Jan 29]. Available from: https://medium.com/uwaterloo-voice/error-analysis-in-deep-learning-6df3b3d335af.
   Google Scholar
• TensorFlow.org [Internet]. California, United States: Google LLC;[cited 2023 Feb 2]. Available from: https://www.tensorflow.org/api_docs/python/tf/keras/losses.
   Google Scholar
• DigitalOcean [Internet]. New York, USA: Word Press; [cited 2023 Jan 27]. Available from: https://www.digitalocean.com/community/tutorials/loss-functions-in-python.
   Google Scholar
• Towardsdatascience [Internet]. Ontario, Canada: Medium; [cited 2023 Jan 27]. Available from: https://towardsdatascience.com/what-are-rmse-and-mae-e405ce230383#:~:text = Root%20Mean%20Squared%20Error%20(RMSE,deviation%20from%20the%20actual%20values.
   Google Scholar
• Elbarawy YM, Ghali NI, El-Sayed RS. Facial expressions recognition in thermal images based on deep learning techniques. Int J Image Graphics Signal Process (IJIGSP). 2019;11(10):1–7. doi:10.5815/ijigsp.2019.10.01.
   Google Scholar