Anger or Joy? Emotion Recognition Using Nonlinear Dynamics of Speech

Figures & data

FIGURE 1 Two-dimensional arousal–valence emotion space.

FIGURE 2 PSR (with τ = 1 and d = 3) of four German sentences; (a), (c): Two males expressed two sentences in anger. (b), (d): Two males expressed two sentences in joy. (e), (g): Two females expressed two sentences in anger. (f), (h): Two females expressed two sentences in joy.

FIGURE 3 Attractor forming trajectory of three consecutively following vectors a₁, a₂ and a₃ and geometrical properties of this attractor describing values: angle between consecutive trajectory parts (θ), the distance to the centroid of the attractor (a₁, a₂, and a₃), the length of trajectory leg (t₁ and t₂), and the distance to the identity line (d₁, d₂, and d₃).

FIGURE 4 Process of extracting nonlinear dynamics features (NLDs).

TABLE 1 List of Extracted Features

NLD(1) features: (1440)

Speech signal, voiced speech, unvoiced speech, glottal waveform, pitch contour, and energy contour;

Apply 20 time domain functions to:

Distance to the centroid (), delta , and double-delta ;

Length of trajectory legs (),delta , and double-delta ;

Current angle between consecutive trajectory legs (), delta (), and double-delta ();

Distance from the identity line (), delta (), delta (), and double-delta ();

NLD(2) features: (4800)

Speech signal, voiced speech, unvoiced speech, glottal waveform, pitch contour, and energy contour;

The first 200 DCT coefficients of:

Distance to the centroid (), delta , and double-delta ;

Length of trajectory legs (),delta , and double-delta ;

Current angle between consecutive trajectory legs (), delta (), and double-delta ();

Distance from the identity line (), delta (), delta (), and double-delta ();

Prosodic features: (241)

Apply 20 time domain functions to:

Pitch, delta pitch, double-delta pitch;

Energy, delta energy, double-delta energy;

Ratio between the duration of voiced and unvoiced speech;

ZCR, ZCR delta, and ZCR-double delta;

TEO, TEO delta, and TEO-double delta;

Spectral features: (960)

Apply 20 time domain functions to:

12 MFCCs, their deltas, and their double-deltas;

4 formants, their deltas, and their double-deltas;

Note!

20 time domain functions include:

min, max, range, mean, median, trimmed mean 10%, trimmed mean 25%, 1st, 5th, 10th, 25th, 75th, 90th, 95th, and 99th percentile, interquartile range, mean average deviation, standard deviation, skewness, kurtosis

TABLE 2 Number of Utterances in the Berlin Database

Emotional State	Anger	Boredom	Disgust	Fear	Joy	Neutral	Sadness	Total
female	67	46	35	32	44	40	37	301
male	60	35	11	37	27	39	25	234
total	127	81	46	69	71	79	62	535

FIGURE 5 The training and testing procedure in one trial.

FIGURE 6 Average FDR for the prosodic features, (a) Females, and (b) Males.

FIGURE 7 Average FDR for the spectral features, (a) Females, and (b) Males.

FIGURE 8 Average FDR for the proposed NLD(1) features, (a) Females, and (b) Males.

FIGURE 9 Average FDR for the proposed NLD(2) features, (a) Females, and (b) Males.

FIGURE 10 Average FDR for different types of features, (a) Females and (b) Males.

FIGURE 11 Classification accuracy for different types of features, (a) Females, and (b) Males.

FIGURE 12 Classification accuracy for different types of NLD(2) features, (a) Females, and (b) Males.

FIGURE 13 Classification accuracy for three NLD(2) feature sets, (a) Females, and (b) Males.

TABLE 3 Confusion Matrix for Using Combined Prosodic and Spectral Features (Females)

Emotion	Anger	Boredom	Rate(%)
Anger	59	8	88.06%
Boredom	10	34	77.27%
Precision (%)	85.51%	80.95%

Average recognition rate: 83.78%.

Number of features: 350.

TABLE 4 Confusion Matrix for Using Combined Prosodic and Spectral Features (Males)

Emotion	Anger	Boredom	Rate(%)
Anger	57	3	95%
Boredom	4	23	85.19%
Precision (%)	93.44%	88.46%

Average recognition rate: 91.95%.

Number of features: 250.

TABLE 5 Confusion Matrix for Using Proposed “Combination 2” Features (Females)

Emotion	Anger	Joy	Rate(%)
Anger	67	0	100%
Joy	1	43	97.73%
Precision (%)	98.53%	100%

Average recognition rate: 99.1%.

Number of features: 300.

TABLE 6 Confusion Matrix for Using Proposed NLD(2) Features (Males)

Emotion	Anger	Joy	Rate(%)
Anger	60	0	100%
Joy	1	26	96.3%
Precision (%)	98.36%	100%

Average recognition rate: 98.85%.

Number of features: 100.

TABLE 7 Confusion Matrix of the Method Proposed by Wu, Falk, and Chan (2011; Females and Males)

Emotion	Anger	Boredom	Disgust	Fear	Joy	Neutral	Sadness	Rate(%)
Anger	119	0	1	1	6	0	0	93.7
Boredom	0	78	0	0	0	3	0	96.3
Disgust	0	0	42	1	1	2	0	91.3
Fear	2	0	0	62	3	2	0	89.86
Joy	12	0	1	3	52	3	0	73.24
Neutral	0	3	1	0	0	75	0	94.94
Sadness	0	0	0	0	0	0	62	100
Precision (%)	89.47	96.3	93.33	92.54	83.9	86.21	100

Average recognition rate: 91.59%.

TABLE 8 The Results of Classification After Applying Our Proposed Classifier as an Error Correction Stage for the Method Proposed by Wu, Falk, and Chan (2011; Mixed Females and Males)

Emotion	Anger	Boredom	Disgust	Fear	Joy	Neutral	Sadness	Rate(%)
Anger	125	0	1	1	0	0	0	98.43
Boredom	0	78	0	0	0	3	0	96.3
Disgust	0	0	42	1	1	2	0	91.3
Fear	2	0	0	62	3	2	0	89.86
Joy	2	0	1	3	62	3	0	87.32
Neutral	0	3	1	0	0	75	0	94.94
Sadness	0	0	0	0	0	0	62	100
Precision (%)	96.9	96.3	93.33	92.54	93.94	86.21	100

Average recognition rate: 94.58%.

Wu, S., T. H. Falk, and W. Y. Chan. 2011. Automatic speech emotion recognition using modulation spectral features. Speech Communication 53:768–85.

 Web of Science ®Google Scholar