Multimodal Emotion Recognition for Children with Autism Spectrum Disorder in Social Interaction

Abstract

Autism Spectrum Disorders (ASD) remain a healthcare challenge and gain considerable attention due to the increasing prevalence rates and insupportable burden on families and society. It is noted that the recognition of children’s emotional states plays an important role in the evaluation and intervention process of ASD. In this paper, we aim to address the problem of automatic recognition of the emotional states of ASD children in social interactive scenarios. Since the child can be unconstrained in realistic scenarios, the face occlusion under pose variations and uncertain backgrounds become challenges of this task. To tackle this problem, we employ both facial expressions as well as body poses as cues to recognize the emotional states while most traditional methods only leverage the former. Firstly for the facial information, spatial features are extracted through convolutional neural networks followed by a temporal transformer to extract temporal information. Then for the body pose information, graph convolutional networks combined with the self-attention part are used to represent spatial features and temporal convolutional layers for temporal counterparts. Finally, different multimodal fusion ways are explored to generate final recognition results. We evaluate this method on a challenging database collected by us in real-world child-clinician interactive scenarios and the proposed method achieved significantly better results than baselines using only facial information. Thus it is suggested that there is a potential to assist in clinical practice by providing the recognized emotion as feedback.

Keywords:

• Emotion recognition
• multimodal fusion
• affective computing
• autism

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was as supported by the National Natural Science Foundation of China (Grant No. 61733011); Shanghai Key Clinical Disciplines Project and Guangdong Science and Technology Research Council (Grant No. 2020B1515120064); in part by the National Key Research and Development Program of China (Grant No. 2022YFC3601700); by the National Natural Science Foundation of China (Grant No. 52275013); in part by the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2020B1515120064); in part by the Shenzhen Science and Technology Program (Grant No. JCYJ20210324120214040); and in part by the International Cooperation and Exchange of the National Natural Science Foundation of China (Grant No. 62261160652).

Notes on contributors

Jingjing Liu

Jingjing Liu received a BE degree from the Beijing Institute of Technology in 2017. She is currently a PhD candidate at the Robotics Institute, Shanghai Jiao Tong University. Her current research interests include multimodal human behavior analysis and applications in human-computer interaction and medical diagnosis.

Zhiyong Wang

Zhiyong Wang received the PhD degree in mechanical engineering from the School of Mechanical Engineering, Shanghai Jiao Tong University in 2023. He is currently an assistant Professor of Harbin Institute of Technology Shenzhen, Shenzhen, China. His research interests include human-computer interaction, gaze estimation and their applications in medical rehabilitation.

Wei Nie

Wei Nie received a BE degree from the Harbin Institute of Technology in 2020. He is currently pursuing a PhD degree with the School of Mechanical Engineering and Automation, Harbin Institute of Technology (Shenzhen). His current research interests include facial analysis and applications in autistic screening.

Jia Zeng

Jia Zeng received the BS degree in mechatronic engineering from Central South University, Changsha, China, in 2017. He is currently pursuing the PhD degree in rehabilitation robotics with the Robotics Institute, Shanghai Jiao Tong University. His research interests include bio-signal processing and machine learning algorithm for humanCcomputer interaction.

Bingrui Zhou

Bingrui Zhou is a pediatrician of Childrens hospital of Fudan University, Shanghai, China. She received the PhD degree in pediatrics from Fudan University, Shanghai, in 2017. She is specialized in nutritional diseases, early screening, diagnosis and treatment of developmental behavioral disorders in children.

Jingxin Deng

Jingxin Deng received the Master’s Degree in pediatrics from Children’s Hospital of Fudan University, Shanghai, China, in 2021. She is currently pursuing a PhD in Childrens Hospital of Fudan University, Shanghai, China. Her research interests include early child development and mechanisms of autism spectrum disorders.

Huiping Li

Huiping Li is a pediatrician of Children’s hospital of Fudan University, Shanghai, China. She received the PhD degree in pediatrics from Fudan University, Shanghai, in 2018. Her specialty is the diagnosis and treatment of developmental behavioral disorders in children.

Qiong Xu

Qiong Xu is the associate director of the Child Health Care Dept., Children’s Hospital of Fudan University, Shanghai, China. Her clinical focus is on early detection and intervention for ASD in both community- and hospital-based practices. Her research interests cover ASD and other genetic neurological disorders.

Xiu Xu

Xiu Xu (INSAR Fellow) is a Professor in Pediatrics, the Chief in the Division of Child Health Care, Childrens Hospital of Fudan University. She contributed to an ASD screening program in The Three-level Network of Child service in Shanghai. Her research interests include early diagnosing and intervention for ASD.

Honghai Liu

Honghai Liu (Fellow, IEEE) is a Professor with the State Key Laboratory of Robotics and Systems, Harbin Institute of Technology Shenzhen, China. He is also with the University of Portsmouth, UK. His research interests include biomechatronics, pattern recognition, intelligent video analytics, intelligent robotics, and their practical applications.