Meltdown/Tantrum Detection System for Individuals with Autism Spectrum Disorder

ABSTRACT

The intensive and explosive behavioral problems associated with Autism Spectrum Disorder (ASD) are treated as ‘meltdown or tantrum,’ and it may lead to hyperactivity, impulsivity, aggression, self-injury, and irritability. The present work aims to propose and implement a noninvasive real-time deep learning based Meltdown/Tantrum Detection System (MTDS) for ASD individuals. The noninvasive physiological signals (such that heart rate, skin temperature, and galvanic skin response) were synthetically recorded with a specially designed hardware prototype. The recorded physiological signals were transmitted to an internet connected server where deep learning algorithms such as CNN, LSTM, and CNN-LSTM based Meltdown/Tantrum Detection System (MTDS) were implemented. The trained deep learning model was capable of detecting abnormal states of meltdown or tantrum through real-time received physiological signals. The proposed MTDS system was trained and tested with deep learning algorithms such as CNN, LSTM and hybrid CNN-LSTM, and it was found that hybrid CNN-LSTM was outperformed with an average training and testing accuracy of 96% with low MAE (0.10 for training and 0.04 for testing). Furthermore, 86% of the ASD caregivers favored the proposed MTDS system.

Introduction

The facial and other emotional changes could be a very vital sign to reflect the emotional changes in typically developing, but individuals with ASD often experienced difficulty to articulate their own mental and emotional states (Happé 2018; Montaque, Dallos, and McKenzie 2018; Ryan 2010). An individual with ASD could exhibit minimum of one challenging behavior such as destruction of property, screaming, shouting, and kicking. This kind of challenging behavior could be disruptive and destructive to ASD individuals themselves or others (McGinnis et al. 2019). The occurrence of predisposing, precipitating and perpetuating events such as sensitivity to sensory stimulation, the frustration of needs, and criticism from parents could lead to meltdown or tantrum (Critchley 2002).

In ASD, the hyperactive state such as meltdown or tantrum was described as escalating episodes of conflicts & distress, in which intense and dangerous explosions of difficult behaviors could occur frequently. ASD meltdowns or tantrums could be one of the most challenging parts of autistic persons and their caregivers (Bessette Gorlin et al. 2016). In comparison to the general population, individuals with ASD had more risk of developing challenging behavior since meltdown or tantrum associated with negative outcome lead to a decreased quality of life (Anzalone et al. 2014; Fitzpatrick et al. 2016). It could be essential to frequently detect or monitor such kind of abnormal behavior accurately in ASD individuals.

Noninvasive physiological signals could be able to provide sensitive and convenient measure of assessing changes in sympathetic arousal associated with emotions, cognition and attention. Some noninvasive physiological changes such as body sweating, abnormal heart rate and rise in body temperature could be noticed during hyperactive, meltdown or tantrum related states (Sigman et al. 2003). Furthermore, an efficient system could be required to represent and extract all the information from varying noninvasive physiological signals. It was very hard to fetch real-time physiological data using wired electronics due to the assorted nature of autistics. Wireless Internet-connected Things (IoT) architecture was utilized for healthcare monitoring by user-friendly clinical applications with fast access mobility and envisioned to the modern smart health care system to collect health related information (Baker et al. 2017; Hu et al. 2017; Luo et al. 2018; Yugha and Chithra 2020). Machine and Deep learning was proposed by several authors to overcome the limitations by considering in depth information from physiological datasets (Alzubi, Nayyar, and Kumar 2018; Cheng et al. 2018; Lecun, Bengio, and Hinton 2015; Tahsien, Karimipour, and Spachos 2020). An efficient internet connected monitoring system named “BioSenHealth 1.0” was implemented and identified with better accuracy at low cost (Nayyar, Puri, and Nguyen 2019).

Based on physiological signals, the noninvasive systems and efficient deep learning algorithms could be able to provide sufficient information about hyperactive emotion characteristic of meltdown or tantrum situations in ASD individuals. The objective of this paper was focused to propose an efficient Meltdown or Tantrum Detection System (MTDS) framework for ASD individuals. The proposed MTDS system is able to fetch physiological signals such as Skin Temperature (ST), Heart Rate (HR) and Galvanic Skin Response (GSR) through the proposed data acquisition system. On basis of acquired signals, deep learning models were trained and tested to make future predictions for the classification of the state of normal behavior or the state of meltdown/tantrum in ASD individuals. This paper was further organized with related literature studies, design/implementation of MTDS, and results/discussions after implementations with concluding remarks.

Related Studies

Earlier researchers were using different invasive tools or methods on the basis of samples of blood, saliva or urine to detect the behavioral changes, but these invasive tools/methods were not suitable to be used as a continuous real-time analyzer and in daily life too (Hufnagel et al. 2017). Relevant information must be extracted from the physiological signal received from the sensors implanted or placed on the skin in order to support a specific healthcare application (Hirstein, Iversen, and Ramachandran 2001). Welch suggested that the changes in physiological signals could be measured by sensor based devices and these changes could be helpful in diagnosis of the subjects (Welch 2012).

Physiological parameters such as Skin Temprature (ST), Heart Rate (HR) and Galvanic Skin Response (GSR) considered as an invaluable data source, which could assist in the detection of disease, rehabilitation and treatment (Lydon, Healy, and Dwyer 2013). Goodwin et al. (Goodwin et al. 2006) observed that the physiology of a person affected with autism could be different in outside view than what was measured inside the body. The cardiac responses of the autistic subject were examined during crying and happy stimulus, and observed variations in cardiac rate with changed stimulus sources (Luo et al. 2018). The patterns of arrhythmic heart rate in ASD individuals were observed during the stressed environment (Acharya et al. 2018; Sasikumar, George Priya Doss, and Adalarasu 2015). In comparison to the typically developed subjects, GSR & HR in autistic subjects resulted in atypical and individualized characteristics (Panju et al. 2015; Sasikumar, George Priya Doss, and Adalarasu 2015).

The wearable devices could observe changes in physiological signals noninvasively, which could help in predicting the anxiety level & episodes of self-injury and increase the self-awareness of their internal emotional state to regulate their emotions. Ko et al. (Koo et al. 2018) suggested that 72% parents of the ASD individuals were willing to monitor changes in physiological signals or behavioral parameters of their children to understand their meltdown/tantrum or other emotions. Some physiological sensor based wearable products available in the market were presented in Table 1. The physiological signals such as HR, GSR, and ST could be considered as ASD footprints during hyperactive, meltdown, tantrum, or other related states.

Table 1. Physiological parameters with wearable devices

Name of Product/Reference	Physiological Parameter(s)
Embrace Empatica (Empatica, 2019; Empatica Inc, 2019)	HR, GSR, ST
Visi Mobile (ViSi Mobile, n.d.; Weenk et al. 2017)	HR, BP, EEG, ST
QardioCore (Is QardioCore Clinically Validated, 2019; Qardio, 2019)	HR, ST
Apple Watch Series 4 (Apple, 2019; Hernando et al. 2018)	HR
Northrup et al. (Northrup et al. 2016)	GSR
Simm et al. (Simm et al. 2016)	HR, ST
MyFeel Wristband (Sentio Solutions, 2019)	HR, GSR, ST
Reveal (Awake Labs, 2019)	HR, GSR, ST

HR – Heart Rate, GSR- Galvanic Skin Response, BP – Blood Pressure,

ST – Skin Temperature, EEG – Electroencephalogram.

The state of meltdown or tantrum could be hard to detect or understand in comparison to facial emotions or other behavior of ASD. Today’s advancement in technology could help improve the diagnosis process of ASD, but such diagnoses were not much effective to detect the meltdown/tantrum and provide/arrange remedy to these dangerous states. The generated data was continuously generating in incremental phase multivariate time series data, which could cover symptoms and scenarios, associated with ASDs. The implementation of traditional detection systems failed to work with such kind of large and continuously varying datasets (Krupa et al. 2016; Litjens et al. 2017). The monitoring of physiological signals through noninvasive methods could play very crucial role to provide certain aid to ASD individuals for the detection or prediction of meltdown/tantrum related states (Cabibihan et al. 2018; Krupa et al. 2016). Without the detailed knowledge regarding meltdown or tantrum related events, it could be complicated to start any treatment or therapy of ASD individuals. An efficient system could be required, which could work with multivariate and diverse physiological data for the detection of meltdown or tantrum state in an ASD individual. Working on earlier computer-based designs for meltdown detection was a tedious task for parents and clinicians.

Design and Implementation of MTDs

Deep learning based meltdown or tantrum detection system (MTDS) for ASD was proposed, which was implemented on noninvasive multivariate physiological data related to Heart Rate (HR), Galvanic Skin Response (GSR) and Skin Temperature (ST). This proposed design was capable of updating the data continuously and could be able to issue an alert if meltdown or tantrum state occurred. An interactive graphical environment of the proposed system was designed, which made it easy to understand and access the meltdown or tantrum related behavior by clinicians as well as parents. The individualized physiological signals were remotely transferred to an MTDS implemented internet connected server. Current individualized monitoring and classification of physiological data was then reflected on the internet-connected graphical monitoring tool, which could be accessed remotely by clinicians and parents at any time & anywhere. The proposed design was capable of detecting a meltdown or tantrum state in ASD individuals that could be further used in the improvement of behavior related issues.

A deep learning based hardware wristband prototype was designed to acquire individualized physiological signals such as HR, ST and GSR, which could be able to transmit data to the internet connected server. The deep learning model was implemented on the acquired individualized physiological signals at the IoT server to detect the meltdown or tantrum state. The unique feature of the proposed design was a remotely accessible and user-friendly graphical interface to access the runtime detection by clinicians and parents. These features were facilitated to provide early remedy and to find the reason behind frequently occurring intense and dangerous events. The block diagram of the overall system was shown in Figure 1.

Figure 1. MTDS block diagram.

Hardware Prototype

A specialized hardware prototype was designed in the form of comfortable wearable wristband to detect ASD physiological signals. HR, ST and GSR could play a very crucial role in detecting the mental state of human beings. So, various sensors such as photoplethysmogram (PPG), LM35 transistor and GSR were implanted on the proposed wristband for the measurement of HR, ST, and GSR. The recorded physiological signals were processed and controlled by a battery-operated Arduino Nano, which was further connected to the IoT server through a Wi-Fi module. Prototype design of the proposed wireless IoT connected hardware wristband is presented in Figure 2.

Figure 2. (a) Specially designed hardware wristband prototype. (b) Specially designed hardware wristband connectivity diagram.

Data Acquisition and Preprocessing

The physiological signals such as HR, ST and GSR were recorded from the wristband to incorporate multimodal physiological signals, and these recorded signals were further transmitted to the internet connected server. For the proper recording of the data, the synthetic sessions were conducted in a total duration of 30 minutes. Null values were removed from the recorded data in the first phase of the preprocessing. After removal of null values, the MinMax scaling technique was applied to shrink raw data in an acceptable range of 0 to 1 without modifying the shape of original data. The MinMax scaling divided the minimum values of the feature and then divided by the difference between the original maximum and minimum values. The MinMax scalar formula was applied to scale each feature (x_i) as

1 $MinMaxScaler\left(x_i\right)=\frac{x_i-minimum\left(x\right)}{maximum\left(x\right)-minimum\left(x\right)}$1

The initial pre-classification system was implemented to identify the relevant classification labels with a low or high state of acquired data using the Isolation Forest outlier detection method. The sensitivity of the outlier system was pre-adjusted at 1% to acquire top 1% outlier from multivariate physiological data (HR, GSR and ST) as a threshold for tantrum or meltdown state.

Hybrid CNN-LSTM

A time series data was a sequence of digital signals that was taken sequentially in time steps to predict real time observations. In the present work, the hybrid combination of one-dimensional CNN and LSTM algorithm was utilized for the analysis of acquired time series data and the performance of CNN-LSTM algorithm was also compared with CNN and LSTM. In the CNN-LSTM model, an initial convolutional layer received one-dimensional signal as the input. Each convolution layer was convoluted with their respective kernel (filter) size as

(2) $x_n=\sum _{K=0}^{N-1}{Y}_k{F}_{n-k}$(2)

where Y is signal, F wis filter, N is the number of signals and x are output vectors, respectively. Then, CNN-LSTM algorithm initially established a shallow CNN to extract the original features of the data. Furthermore, its output became a data pool of small amplitude. The Leaky ReLU function was presented as

(3) $f\left(x\right)=\left\{\begin{array}{c}x,ifx>0\\ 0.01,oterwise\end{array}\right.$(3)

The Softmax function was presented as

(4) $p_j=\frac{e^{x_j}}{{\sum }_1^k{e}^{x_k}}forj=1,2,3\dots ..$(4)

These extracted features were converted into a feature matrix and then input into the LSTM for feature fusion. In the CNN-LSTM hybrid model, local features were extracted from the convolutional layer and the LSTM layer then used the ordering of said features. The standard symbols and equations for LSTM were presented as

(5) $f_t=\sigma \left(X_t\ast V_f+H_{t-1}\ast W_f\right)$(5)

(6) $I_t=\sigma \left(X_t\ast V_i+H_{t-1}\ast W_c\right)$(6)

(7) ${\tilde{C}}_t=tanh\left(X_t\ast V_i+H_{t-1}\ast W_i\right)$(7)

(8) $O_t=\sigma \left(X_t\ast V_o+H_{t-1}\ast W_o\right)$(8)

(9) $C_t=f_t\ast C_{t-1}+I_{\hspace{0.17em}}\ast {\tilde{C}}_t$(9)

(10) $h_t=O_t\ast tanh(C_t)$(10)

where

X_t – input vector,

H_t-1 – previous (cell) output,

C_t-1 – previous (cell) memory,

H_t – present (cell) output,

C_t – present (cell) memory,

H_t+1 – next (cell) output,

C_t+1 – next (cell) memory,

W and V – weight vector,

σ – sigmoid function,

tanh – tangent function,

‘*’ represented the element wise multiplication function,

‘+’ represented the element wise addition function.

The architecture of the utilized hybrid CNN-LSTM model, including details of layers, input elements, and extracted output, is shown in Figure 3.

Figure 3. CNN-LSTM architecture for the proposed system.

Implementation Procedure

The work flow of the deep learning based proposed MTDS is shown in Figure 4. Training and testing flow of the proposed MTDS system was presented in Figure 4(a). The monitoring & validation flow was presented in Figure 4(b). Furthermore, the overall training-to-monitoring procedure was described through four implementation procedural parts such as ‘procedure for data acquisition and pre-classification,’ ‘procedure for training and testing of deep learning models,’ ‘procedure for meltdown or tantrum detection,’ and ‘procedure for analysis of meltdown or tantrum detection.’

Figure 4. Proposed work flow diagram of the proposed MTDS. (a) Proposed work flow of CNN-LSTM training and testing. (b) Proposed Work flow of CNN-LSTM validation.

Figure 4. Continued.

Procedure for Data Acquisition and Preprocessing

The following steps were followed for data acquisition and pre-classification of the proposed system, as presented in Figure 5:

1. Individualized physiological signals could be recorded from the wristband and transmitted to the internet-connected server.

2. Redundant and null values were removed from the collected physiological data and then scaled between 0 and 1 using the MinMax scaling algorithm.

3. Then, a pre-classification system performed the classification of initial data based on the 1% top outlier values of HR, GSR and ST. The recognition signal had reflected the present meltdown or tantrum state of autistic, such as high or low. Here, a high state was treated as ‘1ʹ and a low state as ‘0ʹ.

4. Before training of the deep learning based MTDS system at the server, the data acquisition module started removing redundant data entries for the training of the proposed system.

Figure 5. Procedure for data acquisition and pre-classification.

Procedure for Training and Testing of Deep Learning Model

The following steps were followed for the identification of the best model between CNN, LSTM and hybrid CNN-LSTM algorithm, as presented in Figure 6:

1. Acquired pre-classified and pre-processed data fed into the system for training and testing.

2. Training and testing were conducted through pre-classified and pre-processed data entries, and furthermore, K-fold (K = 10) validation was also performed for the optimum deep learning model in this work.

3. The best outcome of the proposed system with acquired physiological signals was identified by a applying deep learning model for 300 epochs (training iterations).

Figure 6. Procedure for training and testing of the deep learning algorithm.

Procedure for Meltdown or Tantrum Detection and Alerting

The following steps were followed for meltdown or tantrum detection in ASD, as presented in Figure 7:

1. The trained optimum deep learning model was deployed for meltdown or tantrum detection in ASD.

2. Individualized physiological signals were recorded from the wristband and transmitted to an internet-connected server in the form of a stream.

3. The proposed monitoring system was initialized for meltdown or tantrum state detection.

4. Data and monitoring outcomes were continuously stored at the server end. Caregiver or clinician could analyze the state of tantrum or meltdown by accessing the stored data. The system was capable of generating an e-mail alert to the caregiver or clinician to highlight the state of tantrum or meltdown.

Figure 7. Procedure for meltdown or tantrum detection and alerting.

Procedure for Monitoring of Meltdown or Tantrum State

The following steps have been followed for the analysis of the proposed system using synthetically updated data, as presented in Figure 8.

1. Recorded data along with the meltdown or tantrum state fed to the monitoring system.

2. Caregiver or clinician could access recorded data remotely using the proposed graphical user interface.

Figure 8. Procedure for monitoring of meltdown or tantrum state.

Results and Discussion

The results of the proposed MTDS system were presented and discussed in this section. First, the comparative analysis of CNN, LSTM, and CNN-LSTM models in the MTDS system was performed and the best model was chosen; further evaluation of the best chosen model was performed through training & testing analysis on the basis of parameters such as accuracy, MAE, precision, recall and F1-socre. The K-fold (K = 10) cross-validation strategy was employed to evaluate the robustness of the proposed model. Furthermore, the analysis of the proposed MTDS design was performed through the validation of caregiver’s responses and statistical validation of caregiver’s responses.

Comparative Analysis of CNN, LSTM and Hybrid CNN-LSTM

The deep learning algorithms such as CNN, LSTM and hybrid CNN-LSTM were utilized to design an efficient MTDS system for tantrum/meltdown state in individuals with ASD. The performance of MTDS was evaluated for 300 epochs, and comparative analysis was conducted within utilized deep learning related algorithms such as CNN, LSTM, and CNN-LSTM. Performance analysis was conducted on the basis of training accuracy (as presented in Table 2 and Figure 9) and testing accuracy (as presented in Table 3 & Figure 10) to identify a better algorithm for the proposed MTDS system. It was observed that the hybrid CNN-LSTM algorithm performed better than CNN and LSTM for detection of the meltdown and tantrum states through the MTDS system.

Table 2. Training accuracy of CNN-LSTM vs. LSTM vs. CNN

Number of Epochs	Training Accuracy
	CNN-LSTM	LSTM	CNN
1	72	64	55
10	95	75	59
25	96	83	62
50	97	87	68
75	98	89	73
100	98	92	75
125	98	91	82
150	98	93	86
175	98	92	86
200	98	92	86
225	98	92	86
250	98	92	86
275	98	92	86
300	98	92	86

Table 3. Testing accuracy of CNN-LSTM vs. LSTM vs. CNN

Number of Epochs	Testing Accuracy
	CNN-LSTM	LSTM	CNN
1	86	59	51
10	96	71	53
25	98	73	59
50	98	84	62
75	98	87	66
100	99	89	72
125	99	91	73
150	99	93	72
175	99	92	79
200	99	92	81
225	99	92	81
250	99	92	83
275	99	92	81
300	99	92	81

Figure 9. Training accuracy of CNN-LSTM vs. LSTM vs. CNN.

Figure 10. Testing accuracy of CNN-LSTM vs. LSTM vs. CNN.

Training and Testing Analysis of CNN-LSTM Based MTDS

Training and testing performance of the CNN-LSTM based MTDS with 90% training data & 10% testing data was analyzed based on parameters, viz., accuracy & MAE, as presented in Figure 11 and Figure 12, respectively. Furthermore, more specific results of the MTDS for meltdown or tantrum detection in ASD with acquired 10% of the testing dataset was performed based on basic parameters, viz., precision, recall and F1-score, as presented in Figure 13–15.

Figure 11. MTDS training and testing accuracy.

Figure 12. MTDS training and testing MAE.

Figure 13. MTDS recall.

Figure 14. MTDS precision.

Figure 15. MTDS F1-score.

K-Fold Cross-Validation of CNN-LSTM Based MTDS

The performance of the trained and tested CNN-LSTM based MTDS was validated with the K-fold (K = 10) method for cross validation. The average K-fold validation results for parameters such as accuracy, precision, recall and F1 score and the confusion matrix for predictive active & predictive non-active state were presented in Table 4 and Figure 16.

Table 4. Average of K-fold cross-validation of MTDS

Confusion Matrix (in number)			Analysis Parameters (in %age)
Meltdown/Tantrum State	True Active	TrueNon-Active	Accuracy	Precision	Recall	F1-Score
Predicted Active	1669	21	96.04	98.76	96.70	97.72
Predictive Non-Active	57	223

Figure 16. Average of K-fold cross-validation of MTDS.

Validation Based on Caregiver’s Responses

The caregivers of both ASD and TD group individuals participated to analyze the proposed system with different levels of knowledge regarding autism, such as Highly Knowledgeable (HK), Average Knowledgeable (AK), and Low Knowledgeable (LK). Here, the roles of caregivers are only to understand the proposed framework and give their verbal reviews on the proposed framework. There is no other involvement of human or animal in this work. Also, during this procedure, none of the caregiver or participant had received any psychoactive medication. The participant and caregiver had the right to withdraw from the study at any time. No name is disclosed throughout this study. The responses of different knowledge levels of caregivers about the proposed system were evaluated to maintain the unbiased validation. The performance of the proposed system was analyzed on a scale of 0 to 10 for the four parameters, viz., user interface, monitoring system, detection system, and alerting system. Average ratings for defined parameters of the proposed system are presented in Table 5 and Figure 17.

Table 5. Caregiver’s responses for the proposed system

Caregiver Details			Rating Parameters (Rating scale between 0–10)
No.	Level	Group	User Interface	Monitoring System	Detection System	Alerting System	Overall
1	MK	ASD	6	8	7	7	7
2	MK	ASD	7	8	5	7	6.75
3	LK	ASD	8	6	7	8	7.25
4	HK	ASD	8	6	6	8	7
5	LK	ASD	5	4	4	5	4.5
6	LK	ASD	8	8	7	6	7.25
7	HK	ASD	7	7	8	6	7
8	LK	ASD	7	6	6	7	6.5
9	MK	ASD	9	8	8	9	8.5
10	MK	ASD	4	5	3	3	3.75
11	MK	TD	7	8	6	8	7.25
12	HK	TD	9	8	7	7	7.75
13	MK	TD	7	6	4	7	6
14	MK	TD	6	7	8	6	6.75
15	LK	TD	8	8	7	6	7.25
Above Average (i.e. 5) Percentage of Satisfaction Level							86.6%

* HK – Highly Knowledgeable regarding ASD, MK – Average Knowledgeable regarding ASD, and LK – Less Knowledgeable regarding ASD

Figure 17. Responses of related caregivers for defined parameters and their overall mean.

Statistical Analysis of Caregiver’s Responses

The statistical analysis was conducted to identify the difference in caregiver’s responses for ASD and TD group. As presented in Table 6, the mean scores related to caregivers of the ASD group were found to be 6.55, which was statistically similar to the mean score of the TD caregivers group where the mean score was found to be 6.25 with two-tailed unequal variance t-test statistics −0.85 at p < .05. Both groups of ASD and TD caregivers rated the overall system as statistically similar.

Table 6. Statistical analysis based on caregiver’s responses

Domain	Caregiver Group	N	Mean	SD	t-Stat	Sig. (2- tailed)
Overall System	ASD	10	6.55	1.39	−0.85	0.411 (p < .05)
	TD	5	7	0.66

Discussion

The training and testing analysis of the hybrid CNN-LSTM was outperformed with an average training and testing accuracy of 96% with low MAE (0.10 for training and 0.04 for testing). The precision, recall, and F1-score values for the proposed CNN-LSTM model resulted in 0.98, 0.95, and 0.97, respectively. The outcome accuracy of hybrid CNN-LSTM was quite high with least losses. Further analysis of caregiver responses of ASD and TD groups was conducted based on parameters such as user Interface, monitoring system, detection system and alerting system and resulted in 86% above-average (greater than 5) favored responses on the rating scale of 0–10. The conducted statistical analysis had identified no difference in the caregiver responses of both ASD and TD groups of individuals about the proposed MTDS. The proposed MTDS was also compared with the earlier available related systems based on the attained accuracy, as presented in Table 7. It was observed from the comparative analysis that the proposed CNN-LSTM based MTDS system resulted in higher accuracy in comparison to the available technologies.

Table 7. Comparative analysis of the proposed system and available technologies/systems

Sr. No.	System	Emotions	Parameter	Algorithm	Accuracy (%age)
1	Proposed MTDS	Meltdown/ Tantrum State Active Or Neutral	HR, GSR, ST	CNN-LSTM	98%
2	Wavelet Transform based Emotion Recoginition (B. Cheng and Liu 2008)	Joy, Anger, Sadness, Pleasure	EMG	BPNN and LMA	75%
3	EEG based Emotion Recognition (Lin et al. 2010)	Joy, Anger, Sadness, Pleasure	EEG	MLP, SVM	82.29%
4	Automatic Emotional State Identification System (Agrafioti, Hatzinakos, and Anderson 2012)	Valence, Arousal	ECG	LDA	82.5%
5	Negative Emotions States Recognition System (Monajati et al. 2012)	Negative, Neutral	GSR, HR, RSP	FART	94%
6	Human Emotion Recognition System (Guendil et al. 2015)	Joy, Anger, Sadness, Pleasure	EMG, RSP, ECG, GSR	WTF, SVM	95%
7	Automatic Driving Stress Detection System (Lan et al. 2016)	Positive, Negative	EEG	SVM, ELM	73.1%
8	Affective Emotional State Recognition Method (Yu et al. 2016)	Arousal, Valence	EEG	LSTM	74.12%
9	Affective State Mapping Schemes (Samara, Menezes, and Galway 2016)	Arousal, Valence	EEG	SVM	79.83%
10	EMG and RSP Emotion Recoginition System (Mirmohamadsadeghi, Yazdani, and Vesin 2016)	Arousal, Valence	EMG, RSP	SVM	74%
11	LSTM – Emotion Recoginition System (Alhagry, Fahmy, and El-Khoribi 2017)	Arousal, Valence, Liking	EEG	LSTM	87.99%
12	Automated Emotional State Detection System (Acharya et al. 2018)	Seizers	ECG	CNN	94.9%
13	Automated Cardiac Arythemia Prediction System (Oh, Ng, Tan, & Acharya, 2018)	Cardiac Arrhythmia	ECG	CNN-LSTM	98.1%
14	Personalized Emotion Detection Method (Zhao et al., 2019)	Arousal, Valence	EEG, ECG, GSR	HGLA	77.5%

EMG – Electromyography, ECG – Electrocardiogram, GSR – Galvanic Skin Response, HR – Heart Rate, RSP – Respiratory, ST – Skin Temperature, EEG – Electroencephalogram, FART – Fuzzy Adaptive Resonance Theory, WTF – Wavelet Transform Features, HGLA – Hyper Graph Learning Algorithm, BPNN – Back Propagation Neural Network, MLP – Multi Layered Perceptron, SVM – Support Vector Machine, ELM – Extreme Learning Machine, LDA – Linear Discriminant Analysis, CNN – Convolutional Neural Network, LSTM – Long Short Term Memory, and LMA – Levenberg Marquardt Algorithm

Conclusion

The detection of meltdown or tantrum in ASD could be complicated due to the complex nature of neurological developmental disorders. Predetermined and expert knowledge was required to successfully detect the state of meltdown or tantrum, since facial emotions or behavior only were not able to identify the state of meltdown or tantrum in ASD. To overcome this problem, a CNN-LSTM based MTDS system was designed and tested for the collected physiological signals. The graphical interface tool was also designed for easy access of this design for clinicians and parents, which could easily reflect the real-time data of the physiological signals and the classification of the state of meltdown/tantrum as active or inactive. It was observed from the experimental results that the proposed deep learning based MTDS design achieved an 10-fold accuracy of 98% with 0.1 MAE. The caregivers were also excited with the most important feature of the system that proposed that the system could perform equally irrespective of ASD and TD groups. It was concluded that the use of noninvasive, real time and intelligent approach for meltdown or tantrum detection could help reduce the drastic behavior of ASD with safe and economic participation. In the clinical perspective, the effective real-time identification of meltdown or tantrum state could help in the improvement of related behavioral issues in autistic subjects.

Disclosure statement

No potential conflict of interest was reported by the author(s).