Facial expression recognition based on spatio-temporal interest points for depth sequences

Abstract

Of the numerous facial expression recognition methods previously proposed, most are based on texture frames or sequences. Recently, the development of depth sensors has raised new possibilities of dealing with 3D data. The proposed method extracts histograms of oriented gradient (HOG) and optical flow (HOF) for STIPs directly from depth sequences rather than involving registration/deformation techniques to find correspondence in 3D scan data. Mutual information score (MIS) and weighted matching score (WMS) are, respectively, calculated on the basis of naïve-Bayes mutual information maximization and constrained matching pairs. Finally, the MIS and WMS results are concatenated into feature vectors which are then fed into a support vector machine for facial expression classification. The proposed method is applied to the public BU-4DFE (Binghamton University 4D Facial Expression) database for six different facial expressions: anger, disgust, fear, happy, sad and surprise. Experimental results confirm that the proposed method is simple but effective.

Keywords:

• Depth sequence
• Facial expression recognition
• Spatio-temporal interest points
• Histogram of oriented gradient
• Histogram of optical flow
• Matching pairs

Acknowledgement

The authors would like to thank L. Yin, X. Chen, Y. Sun, T. Worm and M. Reale for providing BU-4DFE dataset and the anonymous reviewers for the valuable and insightful comments on the earlier version of this manuscript.

ORCiD

J.-C. Huang http://orcid.org/0000-0002-0086-5210