The Impact of Combining Kinesthetic and Facial Expression Displays on Emotion Recognition by Users

Abstract

Several studies have investigated the relevance of haptics to convey various types of emotions physically. This article investigates the improvement of the recognition rate of emotions using visuo-haptic feedback compared to facial and haptic expressions alone. Four experiments were conducted in which the recognition rates of emotions using facial, haptic and visuo-haptic expressions were tested. The first experiment evaluates the recognition rate of emotions using facial expressions. The second experiment collects a large corpus of 3D haptic expressions of certain emotions and subsequently identifies the relevant haptic expression for each emotion. The third experiment evaluates the selected haptic expressions through statistical and perceptive tests to retain the ones that result in the most accurate identification of the corresponding emotion. Finally, the fourth experiment studies the effect of visuo–haptic coupling on the recognition of the investigated emotions. Generally, emotions with high amplitudes of pleasure are better recognized in the visual modality. However, emotions with high activation are better recognized in the haptic modality. These results also highlighted the finding that participants are not equally aided by each modality when recognizing emotions efficiently. Beyond the recognition rate, multimodal expressions improved the sensation of presence and expressivity.

Additional information

Notes on contributors

Yoren Gaffary

Yoren Gaffary is a Ph.D. student in Computer Science at the LIMSI-CNRS lab. He received a master’s degree in Information, Learning and Cognition at Paris South University. His Ph.D. thesis concerns affective computing using mediated touch with robotic devices coupled with virtual humans.

Victoria Eyharabide

Victoria Eyharabide is an associate professor at Paris-Sorbonne University. She received a Ph.D. in Computer Science in 2010 from UNICEN University. Her Ph.D. thesis proposes an ontology-based approach to personalize human-computer interactions using context-enriched user profiles. Her research interests are related to the study of emotions in multimodal human–computer interactions and computational models of nonverbal expressions of affects for designing interactive virtual characters.

Jean-Claude Martin

Jean-Claude Martin is a full professor of Computer Science at Paris-Sud University. He conducts his research at LIMSI-CNRS, where he is head of the Cognition Perception Use group. His research consists of designing interactive virtual agents that are inspired by psychological theories of emotions and social behaviors.

Mehdi Ammi

Mehdi Ammi is an associate professor at Paris-Sud University specializing in haptics for virtual reality and teleoperation. In particular, he is interested in all aspects of haptic processes ranging from physiological mechanisms to the search for operational methodologies designed to integrate the haptic modality in different type of applications.