Facial emotion detection using thermal and visual images based on deep learning techniques

ABSTRACT

The main objectives of this study are (i) to determine the different emotions, such as happy, anger, neutral, and sad, based on the visual and thermal images for facial expression recognition of healthy individuals; (ii) to train the modified pre-trained models, such as DenseNet-121, ResNet-50, and VGG-19 based on transfer learning technique, using the visual and thermal image dataset, and (iii) to compare the three different customize model with that of three pre-trained models in terms of accuracy. The customized CNN models were proposed to classify the four classes of emotions with high efficiency. Among thermal and visual image dataset, the thermal images have produced highest classification accuracy of 95.8% using the Customized Net-3 model and have outperformed the three modified pre-trained models while classifying the four facial expressions. Thus, the proposed customized CNN model is proven effective for facial emotion recognition based on thermal imaging using deep learning techniques.

KEYWORDS:

• Facial expression
• emotion recognition
• thermal imaging
• deep neural network
• convolution neural network
• Facial thermal image
• Visual image
• Pre-trained models

Acknowledgements

The authors would like to convey their heartfelt appreciation to SRM Institute of Science and Technology, Kattankulathur, Chennai, Tamil Nadu, India, for providing the facilities in campus for the data acquisition process.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Richa Rashmi

Richa Rashmi received her Bachelor degree in Biomedical Engineering from Sathyabama University, Chennai, Tamil Nadu, India in the year 2016. She received her Master degree in Biomedical Engineering in the year 2018 from Amrita Vishwa Vidyapeetham (Amrita University), Ettimadai, Coimbatore, Tamil Nadu, India. Currently, she is a Full-Time Research Scholar in the Department of Biomedical Engineering at SRM Institute of Science and Technology, Kattankulathur, Tamil Nadu. Her research interests are in the area of Medical Imaging, Machine Learning, and Deep Learning.

U. Snekhalatha

U. Snekhalatha is currently working as an professor in the Department of Biomedical Engineering, SRMIST, kattankulathur. She has 19 years of teaching experience. She pursued her Doctorate in Biomedical Engineering at SRMIST (2015). Her area of interest includes biomedical signal processing, medical image processing, biomedical instrumentation, machine learning, and deep learning techniques. She published 100 research articles in reputed peer reviewed international journals and international conferences. She has filed 5 Indian patents in which, one patent is granted and four patents are in the published stage. She obtained the best researcher award for publications in Nature indexed journal during Research Day function held on 1st March 2021. She has obtained the best paper award and gold medals for some research paper publications.

Anela L. Salvador

Anela L Salvador is a Program chair, Electronics Engineering department, College of Engineering, Architecture and Fine arts, Batangas State University. Her area of interest are Signal Processing, Energy Harvesting etc

Alex Noel Joseph Raj

Dr. Alex Noel Joseph Raj received his Ph.D. degree in computer vision and real-time implementations from the University of Warwick, Coventry, UK, Master’s degree in Applied Electronics from Anna University and Bachelor’s degree in Electrical Engineering from Madras University. Dr. Raj is currently with the College of Engineering, Shantou University, China in the Department of Electronic Engineering. He has previously worked with Valeport LTD Totnes, UK, and Vellore Institute of Technology. His research interests include deep learning, signal and image processing, and FPGA implementations. He has rich experience in the design and implementation of real-time algorithms on reconfigurable hardware.