Structure and weights optimisation of a modified Elman network emotion classifier using hybrid computational intelligence algorithms: a comparative study

Abstract

Artificial neural networks are efficient models in pattern recognition applications, but their performance is dependent on employing suitable structure and connection weights. This study used a hybrid method for obtaining the optimal weight set and architecture of a recurrent neural emotion classifier based on gravitational search algorithm (GSA) and its binary version (BGSA), respectively. By considering the features of speech signal that were related to prosody, voice quality, and spectrum, a rich feature set was constructed. To select more efficient features, a fast feature selection method was employed. The performance of the proposed hybrid GSA-BGSA method was compared with similar hybrid methods based on particle swarm optimisation (PSO) algorithm and its binary version, PSO and discrete firefly algorithm, and hybrid of error back-propagation and genetic algorithm that were used for optimisation. Experimental tests on Berlin emotional database demonstrated the superior performance of the proposed method using a lighter network structure.

Keywords:

• emotion recognition
• speech processing
• modified Elman neural network
• optimisation
• gravitational search algorithm
• firefly algorithm

1. Introduction

Human emotions recognition is one of the interesting applications in recent years that can be performed using speech or (and) image signals. Note that there are two important information sources in the speech signal: (a) an explicit source which contains the linguistic content, and (b) an implicit source which carries the paralinguistic information about the speaker. In the last four decades, several methods focused on developing automatic speech recognition systems to extract linguistic information. Although decoding the paralinguistic information such as emotion needs more research efforts (Jaywant & Pell, 2012; Kamaruddin, Wahab, & Quek, 2012; Mariooryad & Busso, 2014; Origlia, Cutugno, & Galatà, 2014; Schuller, Batliner, Steidl, & Seppi, 2011). The emotion recogniser is an effective tool in human-computer interfacing applications such as lie detecting, developing learning environments, consumer relationships, computer tutorial, call-center, and in-car boards (Ai et al., 2006; Devillers & Vidrascu, 2006; Javidi & Fazlizadeh Roshan, 2013; Khanchandani & Hussain, 2009; Polzehl, Sundaram, Ketabdar, Wagner, & Metze, 2009; Tian et al., 2014).

In recent years, research on emotion recognition from speech focused on extracting reliable informative features, selecting appropriate feature set, and combining powerful classifiers to improve the performance of emotion detection systems in real-life applications (Chen, Mao, Xue, & Cheng, 2012; Fernandez & Picard, 2011; Gharavian, Sheikhan, Nazerieh, & Garoucy, 2012; López-Cózar, Silovsky, & Kroul, 2011; Milton & Tamil Selvi, 2014; Sheikhan, Bejani, & Gharavian, 2013; Vlasenko, Prylipko, Böck, & Wendemuth, 2014).

Several features have been employed for emotion recognition from speech such as the following sets listed below:

• Pitch frequency (F₀), log energy (LE), formant frequencies, and Mel-frequency cepstral coefficients (MFCCs) (Arias, Busso, & Yoma, 2014; Kao & Lee, 2006),

• F₀, LE, formant frequencies, MFCCs, vocal tract cross-section areas (A_k), and speech rate (Gharavian & Sheikhan, 2010; Ververidis & Kotropoulos, 2006),

• Linear prediction coefficients (LPCs) and MFCCs (Pao, Chen, Yeh, & Chang, 2008),

• F₀, LE, MFCCs, and LPCs (Altun & Polat, 2009),

• Zero crossing rate, LE, F₀, and harmonics-to-noise ratio (Gajšek, Struc, & Mihelič, 2010),

• Harmony features based on the psychoacoustic harmony perception known from music theory (Yang & Lugger, 2010),

• Statistics of MFCCs computed over three phoneme types (stressed vowels, unstressed vowels, and consonants) (Bitouk, Verma, & Nenkova, 2010),

• Jitter, shimmer, LPCs, linear prediction cepstral coefficients (LPCCs), MFCCs, derivative of MFCCs (dMFCCs), second derivative of MFCCs (ddMFCCs), log frequency power coefficients, and perceptual linear prediction coefficients (Yeh, Pao, Lin, Tsai, & Chen, 2010), and

• Modulation spectral features using an auditory filter-bank and a modulation filter-bank for speech analysis (Wu, Falk, & Chan, 2011).

Similarly, different classification methods were employed in this field such as k-nearest neighbour (Fersini, Messina, & Archetti, 2012; Pao et al., 2008; Väyrynen, Toivanen, & Seppänen, 2011), decision trees (El Ayadi, Kamel, & Karray, 2011; Mower, Busso, Lee, Narayanan, & Lee, 2011; Rong, Li, & Chen, 2009), Bayesian networks (El Ayadi et al., 2011), optimum path forest (Iliev, Scordilis, Papa, & Falcão, 2010), hidden Markov models (Kockmann, Burget, & Černocky, 2011), Gaussian mixture models (GMMs) (Gharavian, Sheikhan, & Pezhmanpour, 2011), support vector machines (SVMs) (Chandaka, Chatterjee, & Munshi, 2009), artificial neural networks (ANNs) (Ahmed Hendy & Farag, 2013; Caridakis, Karpouzis, & Kollias, 2008; Gharavian et al., 2012; Sheikhan, Safdarkhani, & Gharavian, 2011), and hybrid models (Gharavian, Sheikhan, & Ashoftedel, 2013; López-Cózar et al., 2011).

The present study used a hybrid heuristic method for finding the optimum weight set and architecture of a recurrent neural emotion classifier based on gravitational search algorithm (GSA) and its binary version (BGSA), respectively. This hybrid model is called GSA-BGSA in this paper. By considering prosody-related features (such as pitch-related, formant-related, energy contour-related, and timing features), voice quality features (such as mean and standard deviation of pitch and amplitude perturbation quotients (APQs)), and spectral-based features (such as MFCC-based features) of speech signal, a rich feature set including 164 features was constructed. With the aim of selecting more efficient features and reducing the number of input features to the recurrent neural network (RNN), the sequential forward feature selection (SFFS) was employed as a fast feature selection method. Experimental tests on Berlin emotional database demonstrated the superior performance of proposed hybrid GSA-BGSA method over similar hybrid methods based on particle swarm optimisation (PSO) algorithm and its binary version (BPSO), PSO and discrete firefly algorithm (DFA), and hybrid of error back-propagation (EBP) and genetic algorithm (GA) that were used for such optimisations.

Section 2 of this paper reviews the background and related work on optimising ANNs. Section 3 explains the structure of the RNN model used in this study. The details of hybrid GSA-BGSA algorithm are presented in Section 4. The feature set is introduced in Section 5. The brief review of Berlin emotional speech dataset and the experimental results are provided in Section 6. In this way, the performance of proposed method is compared with similar hybrid methods such as EBP-GA, PSO-BPSO, PSO-DFA, and some other emotion recognition systems implemented in recent years and tested on Berlin emotional database. Section 7 concludes the paper and mentions the future research directions.

2. Related work on optimising ANNs

Note that ANN is a nature-based computing technique that was developed as a parallel-distributed network model based on the biological learning process of human brain. The mostly used training algorithm for ANNs, especially multi-layer perceptron (MLP), is the EBP algorithm which is a gradient-based method. However, some inherent problems exist in the EBP algorithm. One of these problems is trapping in local minima, especially for nonlinearly separable pattern classification problems or complex function approximation problems (Gori & Tesi, 1992).

To overcome the problem of multiple local minima in ANNs, several stochastic methods were proposed such as simulated annealing (SA) (Sexton, Dorsey, & Johnson, 1999) and GA (Sexton, Dorsey, & Johnson, 1998) that can find the globally optimal solution with a certain probability.

On the other hand, deterministic techniques (such as tabu search, branch-and-bound, and generalised cutting plane (PintÈr, 1996) can find guaranteed optimal solutions by spending high computational cost. However, modified versions of deterministic techniques were proposed as relatively fast computational algorithms for ANN optimisation (such as the cutting angle method proposed by Beliakov and Abraham (2002) in which the ANN was initially trained using the cutting angle method and then fine-tuning was performed using gradient descent or other optimisation techniques).

For this purpose, a meta-learning framework, as an evolutionary-based algorithm for ANN optimisation, was proposed in (Beliakov & Abraham, 2002) for training and automatic design of the ANNs (Figure 1). The purpose of the proposed method in this study is similar to the meta-learning framework introduced by Beliakov and Abraham (2002); however, a hybrid swarm intelligence-based method was used instead of an evolutionary-based algorithm. In this way, GA (e.g.) searches in a multi-dimensional space based on its global searching capability and varies the number of hidden layers and hidden neurons through application of the genetic operators and evaluation of different architectures according to a fitness function.

Figure 1. Evolutionary-based meta-learning algorithm for ANN optimisation (Beliakov & Abraham, 2002).

Note that the training performance is sensitive to the choice of algorithm's parameters and initial values of the ANN's weights. In other words, selecting the appropriate network architecture and weight parameters strongly affect the convergent behaviour of the EBP algorithm (Ye, Qiao, Li, & Ruan, 2007).

Other computational intelligence (CI)-based methods were also proposed for ANN optimisation in recent decade (Table 1). The role of these methods can be classified into three groups: (a) selecting the optimised architecture, (b) determining the optimised training parameters and/or weights, and (c) obtaining the optimal structure and training weights. The proposed method in this study obtained the optimal structure and training weights of an RNN.

Table 1. Sample CI-based proposed methods for ANN optimisation in recent years.

Optimised item(s)	Method/algorithm	Optimised ANN model	Description
ANN structure	Central force optimisation (CFO) (Meng, Xin, & Min, 2014)	ANN ensembles	Using distributed CFO to optimise individual components in ANN ensembles
ANN training parameters	PSO (Sheikhan & Hemmati, 2012)	Hopfield	Finding five training parameters of a Hopfield ANN
	GSA (Sheikhan & Sharifi Rad, 2013a)	Fuzzy ARTMAP	Optimising training parameters (i.e. learning, vigilance, and choice parameters)
	Modified GSA (Sheikhan & Ahmadluei, 2013)	ART2	Optimising seven training parameters
ANN weights	Modified GSA (Sheikhan & Jadidi, 2014)	MLP	Weight-optimisation of a 3-layer MLP
	Hybrid of improved opposition-based PSO and EBP (Yaghini, Khoshraftar, & Fallahi, 2013)	MLP	Combining the ability of metaheuristics and greedy gradient-based algorithm for ANN training
	PSO-GSA (Mirjalili, Mohd Hashim, & Moradian Sardroudi, 2012)	MLP	Combining the ability of GSA to search for the global optimum and fast searching speed of PSO in the last iterations for more efficient training of MLP
	Fish swarm (Shen, Guo, Wu, & Wu, 2011)	Radial basis function (RBF)	Optimisation of K-means clustering used in the training process of RBF
	Harmony search (Kulluk, Ozbakir, & Baykasoglu, 2012)	MLP	Using five variants of harmony search algorithm for MLP training
	Differential evolution (DE) (Piotrowski, 2014)	MLP	DE algorithm uses the global and local neighborhood-based mutation operators
	Time-variant multi-objective PSO (Qasem & Shamsuddin, 2011)	RBF	Determining the centres and weights of RBF networks simultaneously
ANN training parameters and/or weights	PSO-EBP (Zhang, Zhang, Lok, & Lyu, 2007)	MLP	ANN training using hybrid of PSO algorithm (that converges rapidly during the initial stages of a global search) and EBP method (that is fast convergent around the global optimum)
ANN structure and weights	Improved PSO and BPSO (Yu, Wang, & Xi, 2008)	MLP	Joint optimisation of structure and weights of a 3-layer MLP using improved PSO and BPSO using a self-adaptive evolution strategy
	Binary-real PSO (Zhao & Qian, 2011)	MLP	Using a cooperative binary-real PSO to tune the structure (number of hidden nodes) and parameters of an ANN
	Hybrid of PSO and covariance matrix adaptation based evolutionary strategy (CMA-ES) (Subrahmanya & Shin, 2010)	Elman-type RNN	Simultaneous structure and parameter training method allowing the imposition of certain stability conditions

The GSA is a heuristic algorithm introduced by Rashedi, Nezamabadi-pour, and Saryazdi (2009) and is based on the gravitational law and laws of motion. GSA has a flexible and well-balanced mechanism to enhance exploration and exploitation abilities. A hybrid GSA-BGSA algorithm was used in this study to optimise the network structure (i.e. the number of hidden layer nodes in a recurrent neural network) using BGSA and the connection weights of this network using GSA. The initial number of hidden nodes of the neural network was considered as 75% of the input features (Salchenberger, Cinar, & Lash, 1992) and varied by iterations to achieve the minimum mean squared error.

The proposed method has similarities to the hybrid methods introduced in (Beliakov & Abraham, 2002) or listed in Table 1, because most of these methods employed heuristic and population-based search algorithms such as GA, SA, PSO, and GSA. Most of these heuristic algorithms had a stochastic behaviour and did search in a parallel fashion with multiple initial points. However, the proposed method is different from the mentioned methods because of using BGSA and GSA in optimising weights and number of hidden nodes, respectively. The same idea was proposed in (Abraham, 2002) for optimising architecture, node transfer function and connection weights using evolutionary search mechanism while training network using EBP or Levenberg-Marquardt (LM) in a parallel mode.

3. RNN model equipped with switches in hidden layer

Feedback connections in the RNNs make them ideal for temporal information processing problems. However, the training of RNNs is much more difficult as compared to the static ANNs. It is essential to use an algorithm for automatic determining of suitable structure and weights of the RNNs. As mentioned earlier, this algorithm should perform two tasks: (a) determining the number of hidden layers and constituent nodes, and (b) determining the connection weight values.

An RNN-based classifier was optimised in this study using a GSA-BGSA hybrid algorithm. In this way, BGSA (Rashedi, Nezamabadi-pour, & Saryazdi, 2010) was employed to determine the structure of the RNN (i.e. the number of hidden nodes) and GSA (Rashedi et al., 2009) was used to adjust the training parameters (including weights, initial inputs of the context nodes, and the self-feedback coefficient).

A modified Elman-type RNN (Lin and Hong, 2011) was used in this study. It is noted that Elman ANN is a partial RNN model (Elman, 1990) consists of four layers: input, hidden, context, and output (Figure 2). The context neurons store the previous output of hidden neurons. The modified Elman network has self-feedback connections with fixed coefficients in the context layer. This self-feedback improves the memorisation ability of network which can enhance the convergence and speed of learning process.

Figure 2. Schematic of a modified Elman-type RNN.

The output of the jth hidden layer neuron is calculated as: (1) $o_j^{\mathrm{H}}\left(k\right)=f\left[\sum _i{w}_{ji}^{\mathrm{HI}}\left(k\right)x_i^{\mathrm{I}}\left(k\right)+\sum _r{w}_{jr}^{\mathrm{HC}}\left(k\right)o_r^{\mathrm{C}}\left(k\right)\right];1\le j,r\le m,$(1) where k represents the kth iteration, $w_{ji}^{\mathrm{HI}}$is the weight between the input and hidden node j, x_i is the output of node i, $w_{jr}^{\mathrm{HC}}$is the weight between the context node r and hidden node j, $o_r^{\mathrm{C}}$ is the output of node r in the context layer, and f [.] is the sigmoid function.

The $o_r^C$is calculated as: (2) $o_r^C\left(k\right)=\alpha o_r^C(k-1)+o_j^H(k-1);0\le \alpha <1,$(2) where α is the self-feedback connecting coefficient. The output of the qth output node is calculated as: (3) $y_q\left(k\right)=g\left[\sum _j{w}_{qj}^{\mathrm{OH}}\left(k\right)o_j^{\mathrm{H}}\left(k\right)\right];1\le q\le n,$(3) where $w_{qj}^{\mathrm{OH}}$is the weight between the hidden node j and output node q, and g[.] is taken as a linear function.

4. GSA-BGSA hybrid for optimising RNN

4.1. Review of GSA and BGSA

Rashedi et al. (2009) introduced an optimisation algorithm based on the law of gravity and mass interactions. In GSA, a set of agents called masses were introduced to find the optimum solution by simulation of Newtonian laws of gravity and motion. The performance of objects were measured by their masses, and all these objects attracted each other by the gravity force, while this force caused a global movement of all objects towards the objects with heavier masses.

The mass of each agent is calculated after computing the current population fitness as (Rashedi et al., 2009): (4) $M_i\left(t\right)=\frac{q_i\left(t\right)}{\sum _{j=1}^N{q}_j\left(t\right)};q_i\left(t\right)=\frac{{\mathrm{fit}}_i\left(t\right)-\mathrm{worst}\left(t\right)}{\mathrm{best}\left(t\right)-\mathrm{worst}\left(t\right)},$(4) where N, M_i(t) and fit_i(t) represent the population size, the mass, and the fitness value of agent i at t, respectively. The best(t) and worst(t) are defined for a minimisation problem as Equations (5) and (6), respectively: (5) $\mathrm{best}\left(t\right)=\underset{j\in \{1,\dots ,N\}}{min}{\mathrm{fit}}_j\left(t\right),$(5) (6) $\mathrm{worst}\left(t\right)=\underset{j\in \{1,\dots ,N\}}{max}{\mathrm{fit}}_j\left(t\right).$(6)

To compute the acceleration of an agent, the total forces from a set of heavier masses applied on it should be considered based on a combination of the law of gravity and the second law of Newton on motion as (Rashedi et al., 2009): (7) $\begin{array}{r}{a}_i^d\left(t\right)=\sum _{j\in \mathrm{kbest},j\ne i}{\mathrm{rand}}_{j}G\left(t\right)\frac{M_j\left(t\right)}{R_{i,j}\left(t\right)+e}\left(x_j^d\right(t)-x_i^d(t\left)\right);d=1,2,\dots ,n,i=1,2,\dots ,N,\end{array}$(7) where $a_i^d$presents the acceleration of agent i in dimension d. rand is a uniform random in the interval [0, 1], e is a small value, n is the dimension of the search space, and R_i,j(t) is the Euclidean distance between two agents, i and j. kbest is the set of first K agents with the best fitness value and biggest mass, which is a function of time, initialised to K₀ at the beginning and decreased with time. Here K₀ is set to N (total number of agents) and decreases linearly to 1. G(t) is a decreasing function of time, which is set to G₀ at the beginning and decreases linearly or exponentially towards zero with lapse of time. The exponential reduction is given as: (8) $G\left(t\right)=G_0\exp \left(-\frac{gt}{t_{max}}\right),$(8) where t_max is the total number of iterations. It is noted that $X_i=(x_i^1,x_i^2,\dots ,x_i^N)$indicates the position of agent i in the search space, which is a candidate solution.

Afterwards, the next velocity of an agent is calculated as a fraction of its current velocity added to its acceleration as: (9) $v_i^d(t+1)={\mathrm{rand}}_i\times v_i^d\left(t\right)+a_i^d\left(t\right),$(9) where $v_i^d$presents the velocity of agent i in dimension d. Then, the position of agent i in dimension d is calculated as: (10) $x_i^d(t+1)=x_i^d\left(t\right)+v_i^d(t+1).$(10)

The steps of the GSA algorithm are as follows:

Step 1: Initialisation of X_i(t); i=1,2, … ,N;

Step 2: Fitness evaluation of agents;

Step 3: Update of G(t), best(t), worst(t), and M_i(t); i=1,2, … ,N;

Step 4: Calculation of acceleration and velocity;

Step 5: Update of agents’ position to obtain X_i(t+1); i=1,2, … ,N;

Step 6: Go to Step 2 and repeat until the stopping condition is met.

BGSA was introduced by Rashedi et al. (2010) to extend GSA for tackling binary problems effectively. In BGSA, the position of agents has two values; 0 or 1, and the velocity of an agent represents the probability that a bit (position) takes on 0 or 1. The velocity updating formula remains unchanged, and position updating formula is redefined as: (11) $x_i^d(t+1)=\left\{\begin{array}{ll}1-x_i^d\left(t\right);& {\mathrm{rand}}_i<\left|\tanh \left(v_i^d\right(t+1\left)\right)\right|\\ x_i^d\left(t\right);& \mathrm{otherwise}\end{array}\right..$(11)

4.2. GSA-BGSA for RNN design

The BGSA searches for the number of hidden nodes (i.e. the context nodes) and GSA works on the optimisation of training parameters for each of the structures (agents) present in the BGSA. In the GSA-BGSA, agents of GSA and BGSA work together and are evaluated simultaneously. Each agent was divided to two sub-agents which were subjected to two independent and consecutive processes. The first one was a regular GSA, that is, the traditional velocity and position update of neural network weights. The second one was BGSA, which allowed the agent to determine the number of nodes in single hidden layer of a recurrent neural network.

Two kinds of agents were defined in the proposed hybrid optimisation algorithm: structure and parameter. The structure agent was a binary string whose entries were 1 or 0 (which showed the existence or no-existence of a hidden node). The length of this string was equal to the maximum expected/allowable number of hidden nodes (Max_H). The parameter agent consisted of Max_H parts where each part included self-feedback coefficient, initial inputs of each context node and all the weight connections based on the order of hidden nodes of the structure agent. Assuming $\ell$, m, and n nodes in the input, hidden (and context), and output layers, the encoding of each part in the parameter agent is shown in Figure 3. The number of elements in the parameter agent is $m+m+m\ell +m^2+mn=2m+m\ell +m^2+mn.$

Figure 3. Encoding scheme of each part in the parameter agent.

The MSE function is generally used in fitness evaluation of agents according to the desired optimisation: (12) $\mathrm{MSE}=\frac{1}{nN}\sum _{t=1}^N{\sum _{i=1}^n\left[y_{ti}\left(k\right)-{\tilde{y}}_{ti}\left(k\right)\right]}^2,$(12) where N is the number of training samples, n is the number of RNN outputs, and $y_{ti}\left(k\right)$ and ${\tilde{y}}_{ti}\left(k\right)$ are the target and actual outputs of the tth sample at the time k, respectively. The used fitness function in this study considered both the RNN's size and its convergence accuracy as: (13) $\mathrm{fitness}=\mathrm{MSE}+\beta \frac{m}{\mathrm{Max}\mathrm{\_}H},$(13) where β is a control coefficient for penalising network size and m is the number of hidden nodes. The hybrid algorithm worked on both error performance and complexity minimisation. The search process of the GSA-BGSA hybrid algorithm for updating the network structure and connection weights is shown in Figure 4.

Figure 4. GSA-BGSA optimisation method for tuning the structure and weights of recurrent neural network.

5. Feature set for emotion recognition from speech

To extract features, the speech signal was firstly sampled at the rate of 16 kHz. Then, it was windowed by a 25-msec Hamming window considering 10-msec frame shift. The simulations were performed using Matlab R2014a software on a PC with an Intel Core i5–4460 @ 3.2 GHz CPU and 8GB RAM.

Speech features for emotion recognition can be classified into four categories: prosody-related features, voice quality features, spectral-based features, and Teager-energy operator (TEO)-based features (El Ayadi et al., 2011). Note that these features were combined to represent the speech signal in the most studies of this field.

The prosody-related features used in this study are listed in Table 2. As seen in Table 2, the prosody-related features were grouped as follows: pitch contour-related features (33 features), formant-related features (21 features), energy contour-related features (59 features), and timing features (11 features). The estimation of pitch frequency was based on iterative adaptive inverse filtering method (Alku, 1992). The formant frequencies and formant bandwidths were estimated using LPC analysis.

Table 2. Prosody-related features used in this study.

Type of feature	Features	Number of features
Pitch contour-related	Max(F0), Min(F0), Avg(F0), R(F0), SD(F0), and IQR(F0)	6
	Max(dF0), Min(dF0), Avg(dF0), R(dF0), and SD(dF0)	5
	75% and 90% of F0 and dF0 contour	4
	Max, Avg, Med, and IQR of durations in flat-minima region of pitch contour	4
	Avg and IQR of F0 in flat-minima region of pitch contour	2
	Max, Avg, Med, and IQR of durations in rising-slope region of pitch contour	4
	Avg and IQR of F0 in rising-slope region of pitch contour	2
	Max, Avg, Med, and IQR of durations in falling-slope region of pitch contour	4
	Avg and IQR of F0 in falling-slope region of pitch contour	2
Formant-related	Avg(F1), Avg(F2), and Avg(F3)	3
	Max(F1), Max(F2), and Max(F3)	3
	Min(F1), Min(F2), and Min(F3)	3
	Med(F1), Med(F2), and Med(F3)	3
	SD(F1), SD(F2), and SD(F3)	3
	IQR(F1), IQR(F2), and IQR(F3)	3
	BW(F1), BW(F2), and BW(F3)	3
Energy contour-related	Max, Min, Avg, R, SD, and IQR of energy (E), dE, log-energy (LE), and dLE contours	24
	30%, 50%, and 90% of E, LE, and dLE	9
	30% and 50% of dE contour	2
	Max, Avg, Med, and IQR of durations in rising-slope region of LE and dLE contours	8
	Avg and IQR of LE and dLE values in rising-slope region of LE and dLE contours	4
	Max, Avg, Med, and IQR of durations in falling-slope region of LE and dLE contours	8
	Avg and IQR of LE and dLE values in falling-slope region of LE and dLE contours	4
Timing	Min, Avg, Med, R, and SD of voiced-segment durations	5
	Min, Avg, Med, R, and SD of unvoiced-segment durations	5
	Speaking rate	1

In Table 2, “maximum”, “minimum”, “average”, “median”, “range”, “standard deviation”, “interquartile range”, and “bandwidth” values are abbreviated as “Max”, “Min”, “Avg”, “Med”, “R”, “SD”, “IQR”, and “BW”, respectively. The first derivative of a feature is shown using prefix “d”; for example, dF₀ represents the first derivative of F₀.

Voice quality is related to the acoustic correlates such as voice level, voice pitch, voice formants, and feature boundaries (El Ayadi et al., 2011, Lugger & Yang, 2007). In this study, jitter-related and shimmer-related features were also used for the description of voice quality. Note that jitter and shimmer are the measures of period-to-period fluctuations in F₀ and amplitude, respectively. These measures are calculated as Equations (14) and (15), respectively: (14) $\mathrm{Jitter}=\frac{\left|T_i-T_{i+1}\right|}{\left(1/N\right)\sum _{i=1}^N{T}_i},$(14) (15) $\mathrm{Shimmer}=\frac{\left|A_i-A_{i+1}\right|}{\left(1/N\right)\sum _{i=1}^N{A}_i},$(15) where T_i and A_i are the pitch period and the peak amplitude value of the ith window, respectively, and N is the total number of voiced frames in the utterance (Li et al., 2007). The mean and SD of pitch perturbation quotient and APQ were used as voice quality-related features in this study (i.e. four features) (Kiliç et al., 2004).

The spectral-based features such as MFCCs can model the varying nature of speech spectra under different emotions. The first and second derivatives of spectral features were also included in the feature set to model the temporal dynamic changes in the speech signal. A 36-dimensional MFCC-based vector consisting of 12 MFCCs and their first and second derivatives were used in this study as the spectral-based features.

TEO-based features are used to reflect the energy of the nonlinear energy flow within the vocal tract for a single resonant frequency. Because of using rich energy-related features in this study (as listed in Table 2), the TEO-based features such as TEO-decomposed frequency modulation variation (TEO-FM-Var), normalised TEO autocorrelation envelope area (TEO-Auto-Env), and TEO-based pitch (TEO-Pitch) (Zhou, Hansen, & Kaiser, 1998) were not considered in the feature set to reduce redundant features.

The SFFS was used as a simple, fast, effective and popular technique in this study (Kudo & Sklansky, 2000). However, the CI-based feature selection algorithms can also be used for this purpose (e.g. those reported by the author in (Sheikhan, 2014; Sheikhan & Mohammadi, 2013; Sheikhan & Sharifi Rad, 2013b). The SFFS algorithm selected 60 more significant features. The features were normalised around their mean and standard deviation as: (16) $f_n=\frac{f-\mu }{\sigma }.$(16)

6. Simulation and experimental results

The popular studio recorded Berlin Emotional Speech Database (EMO-DB) (Burkhardt, Paeschke, Rolfes, Sendlmeier, & Weiss, 2005) was used in this study to test the effectiveness of emotion classification using the proposed method. This database covers anger, boredom, disgust, fear, happiness, sadness, and neutral speaker emotions in which ten professional actors speak ten German sentences. The number of utterances in this dataset (whole set) is about 800 (i.e. 700 + some second version); however, only 494 phrases were marked assignable in listening experiments. This limited set was employed in the benchmark comparisons of similar works, as well (Schuller, Vlasenko, Eyben, Rigoll, & Wendemuth, 2009). The number of used speech data in this study for seven emotions is reported in Table 3. The number of training and test speech data was selected in an experiment as 348 and 146, respectively.

Table 3. Number of EMO-DB speech data used in this study for each emotion.

Emotion	Anger	Boredom	Disgust	Fear	Happiness	Sadness	Neutral
Number of speech data	127	79	38	55	64	53	78

In this study, the neural emotion classifier was implemented using four methods: hybrid of EBP algorithm (for training) and GA (for structure optimisation) called EBP-GA, PSO-BPSO hybrid algorithm, PSO-DFA, and the proposed GSA-BGSA. Because of stochastic search in heuristic algorithms, 10 runs of each algorithm were performed and the best results were reported. Initial value of weights was generated at random in the range of [−1, 1]. Other parameter settings of four mentioned methods were performed as follows:

• EBP-GA hybrid algorithm- Learning rate and momentum coefficient were set to 0.1. The GA parameters were set as: population size: 30, crossover probability: 0.8, mutation probability: 0.1, elitism: 5%, rank-based selection: 0.3, maximum number of hidden nodes: 50, and maximum number of generations: 30.

• PSO-BPSO hybrid algorithm- population size: 30, acceleration constants: 2, and inertia factor: decreasing linearly from 0.9 to 0.2 (Eberhart & Kennedy, 1995, Kennedy & Eberhart, 1997), maximum particle velocity: 4, and maximum number of iterations: 50.

• DFA part of hybrid PSO-DFA algorithm- number of fireflies: 30, light intensity at the source: 1, absorption coefficient: 0.22, size of the random step: 0.27, and maximum number of iterations: 50 (Durkota, 2011).

• GSA-BGSA hybrid algorithm- population size: 30, e: 0.01, G₀: 100, g: 20, and maximum number of iterations: 50.

The recognition rates of the proposed neural classifier when employing four hybrid optimisation algorithms are reported in Table 4. As seen in Table 4, the hybrid GSA-BGSA offered the best average recognition rate among the investigated algorithms. It is important that this performance is achieved using smaller number of hidden nodes.

Table 4. Emotion recognition rate of the proposed RNN-based method optimised by hybrid approaches (the best results over 10 runs).

			Average recognition rate (%)
Method	RNN training algorithm	RNN structure optimisation algorithm	Anger	Boredom	Disgust	Fear	Happiness	Sadness	Neutral	Total emotions	Number of hidden nodes
EBP-GA	EBP	GA	74.0	67.1	65.1	76.7	84.9	69.9	78.1	73.7	38
PSO-BPSO	PSO	BPSO	71.2	75.3	67.8	78.1	82.2	71.9	75.3	74.5	35
PSO-DFA	PSO	DFA	80.1	78.8	71.2	76.7	84.9	75.3	79.5	78.1	37
GSA-BGSA	GSA	BGSA	84.2	80.8	73.3	71.9	87.0	76.7	87.7	80.2	32

To perform the speaker-independent experiments, 10-fold cross-validation was used. This validation used utterances from nine speakers for training the classifier and the utterances of a single speaker for testing it. This type of validation tests the speaker-independent performance of the proposed classifier. The average emotion recognition rate of this speaker-independent experiment is reported in Table 5. As seen in Table 5, the hybrid GSA-BGSA offered the best average recognition rate over total emotions among the investigated algorithms.

Table 5. Average emotion recognition rate of the proposed RNN-based method optimised by hybrid approaches (10-fold cross validation).

Method	RNN training algorithm	RNN structure optimisation algorithm	Recognition rate of total emotions (%)
EBP-GA	EBP	GA	70.4
PSO-BPSO	PSO	BPSO	71.8
PSO-DFA	PSO	DFA	73.2
GSA-BGSA	GSA	BGSA	75.8

The speech data extracted from EMO-DB was unbalanced in this study (as seen in Table 3). From the perspective of classifier training, unbalance in the training data distribution often results in poor performance on the minority class (Tang, Zhang, Chawla, & Krasser, 2009). So, better performance estimation (in terms of average recognition rate on total emotions) was reported in Tables 4 and 5 than the real accuracy rate (because of dataset unbalance). Similarly, unbalance in the test data distribution often results in misleading conclusions with certain metrics (Jeni, Cohn, & De La Torre, 2013).

The performance of the proposed system is compared with some other emotion recognition systems (Table 6). All of these systems used Berlin emotional database; however, different feature sets were employed in these researches. As seen in Table 6, the performance of proposed model is superior to most of the reported systems.

Table 6. Performance comparison of proposed GSA-BGSA-optimised RNN emotion recogniser and some other systems tested on Berlin emotional database.

Input features	Number of features	Type of classifier	Speaker-independent average recognition rate (%)
LPCCs+Formants (Rao & Koolagudi, 2013)	26	GMM	68.0
Utterance-level prosodic features (Bitouk et al., 2010)	24	SVM	68.1^a
Class-level prosodic features (Bitouk et al., 2010)	44	SVM	68.6^a
Utterance-level spectral-based features (Bitouk et al., 2010)	78	SVM	67.0^a
Class-level spectral-based features (Bitouk et al., 2010)	237	SVM	75.9^a
Combined features (Bitouk et al., 2010)	261	SVM	78.2^a
Harmonic and Zipf-based features (Xiao, Dellandrea, Dou, & Chen, 2007)	68	Hierarchical	76.2–79.5^b
Selected prosodic features (Wu, Falk, & Chan, 2009)	45	SVM	80.8
Selected prosodic and spectro-temporal features (Wu et al., 2009)	50	SVM	88.6
Proposed selected prosodic, voice quality, and spectral-based features	60	Optimised RNN by GSA-BGSA	75.8

^aRemoving boredom emotion from the analysis.

^bWhen gender classification was also applied.

7. Conclusion and future work

In this study, the GSA-BGSA hybrid method was proposed to tune simultaneously the structure and weights of an RNN-based classifier. This optimised neural classifier was employed as an emotion classifier using Berlin emotional database as a public available database in this field. The number of prosody-related, voice quality, and spectral-based features in this system was set to 124, 4, and 36, respectively. To reduce the number of input features to the RNN, the SFFS method was employed.

The performance of the proposed hybrid GSA-BGSA-neural model was compared with peer hybrid models such as EBP-GA, PSO-BPSO, and PSO-DFA used for such optimisations. Experimental results demonstrated that the proposed method obtained better average recognition rate using a lighter network structure as compared with other peer investigated methods. The performance of proposed method was compared with similar systems tested on Berlin emotional database and implemented in recent years, as well. The comparisons showed the superior performance of proposed method which achieved average recognition rates up to 80.2% as compared to most of the systems.

The proposed method can be modified in future works by inserting another feature selection unit such as ones used in similar works on emotion recognition from speech: principal component analysis or linear discriminate analysis (Haq, Jackson, & Edge, 2008), fast correlation-based filter (Gharavian et al., 2012, 2013), other versions of sequential forward selection (Batliner et al., 2011; Ververidis & Kotropoulos, 2008), least square bound (Altun & Polat, 2009), mutual information (Altun & Polat, 2009), analysis of variations (Gharavian et al., 2013; Sheikhan et al., 2013), and combination of decision tree method and the random forest ensemble (Mower et al., 2011). In addition, the single neural classifier in this study can be replaced by a multiple-classifier scheme to improve the performance of system such as ones reported in (Albornoz, Milone, & Rufiner, 2011; Milton & Tamil Selvi, 2014).

Disclosure statement

No potential conflict of interest was reported by the authors.