A Weighted Deep Ensemble for Indian Sign Language Recognition

Abstract

This work concentrates on developing an Indian sign language (ISL) recognition system using a forearm-worn wearable device to assist hearing-impaired persons. A novel ensemble of convolution neural networks (CNN) is proposed for robust ISL recognition using multi-sensor data. The accuracy for classification of 50 ISL signs improved from 92.5% obtained using a single CNN to 94.2% with 10 ensemble members created using the bagging approach and soft-voting for decision aggregation. Then, the ensemble of CNNs was optimized using weighted voting, where the weights were determined using a differential evolution algorithm. This further improved the classification accuracy to 96.6% with 10 ensemble members.

KEYWORDS:

• Convolution neural networks
• Differential evolution
• Ensemble learning
• Indian sign language
• Wearable sensors
• Weighted voting

DISCLOSURE STATEMENT

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Science and Engineering Research Board, a statutory body of the Department of Science & Technology (DST), Government of India (Grant number ECR/2016/000637).

Notes on contributors

Rinki Gupta

Rinki Gupta received PhD in signal processing from the Centre for Applied Research in Electronics (CARE), Indian Institute of Technology (IIT), Delhi, in 2014. Thereafter, she worked on a project funded by Ministry of Defence at IIT Delhi. She joined Amity School of Engineering and Technology in November 2015. She was also the principal investigator in a project sanctioned by SERB, DST. Her present research interests include time-frequency analysis, speech and audio signal processing and multi-sensor data fusion. Corresponding author. Email: [email protected]

Ananya Shekhar Bhatnagar

Ananya Shekhar Bhatnagar received BTech degree in electronics and communication engineering from Amity School of Engineering and Technology in 2021. She is currently working as a software engineer in Hindustan Computers Limited (HCL), India. Her present research interests include artificial intelligence, machine learning and data science. Email: [email protected]

Ghanapriya Singh

Ghanapriya Singh is working as an assistant professor in Department of Electronics and Communication Engineering at National Institute of Technology, Kurukshetra and on lien at National Institute of Technology, Uttarakhand. She received PhD degree in signal processing from Indian Institute of Technology Delhi (IITD). During her research at IITD, she was working on a project with STMicroelectronics, USA. She is an inventor of 5 US patents (all granted). She is an investigator in the projects funded by MeitY and DST. Her current research interests include context awareness, image processing, speech processing and signal processing for IoT. Email: [email protected]