Use of machine learning methods in prediction of short-term outcome in autism spectrum disorders

ABSTRACT

OBJECTIVE

Studies show partial improvements in some core symptoms of Autism Spectrum Disorders (ASD) in time. However, the predictive factors (e.g. pretreatment IQ, comorbid psychiatric disorders, adaptive, and language skills, etc.) for a better the outcome was not studied with machine learning methods. We aimed to examine the predictors of outcome with machine learning methods, which are novel computational methods including statistical estimation, information theories and mathematical learning automatically discovering useful patterns in large amounts of data.

METHOD

The study the group comprised 433 children (mean age: 72.3 ± 45.9 months) with ASD diagnosis. The ASD symptoms were assessed by the Autism Behavior Checklist, Aberrant Behavior Checklist, Clinical Global Impression scales at baseline (T0) and 12th (T1), 24th (T2), and 36th (T3) months. We tested the performance of for machine learning algorithms (Naive Bayes, Generalized Linear Model, Logistic Regression, Decision Tree) on our data, including the 254 items in the baseline forms. Patients with ≤2 CGI points in ASD symptoms at in 36 months were accepted as the group who has “better outcome” as the prediction class.

RESULTS

The significant proportion of the cases showed significant improvement in ASD symptoms (39.7% in T1, 60.7% in T2; 77.8% in T3). Our machine learning model in T3 showed that diagnosis group affected the prognosis. In the autism group, older father and mother age; in PDD-NOS group, MR comorbidity, less birth weight and older age at diagnosis have a worse outcome. In Asperger’s Disorder age at diagnosis, age at first evaluation and developmental cornerstones has affected prognosis.

CONCLUSION

In accordance with other studies we found early age diagnosis, early start rehabilitation, the severity of ASD symptoms at baseline assessment predicted outcome. Also, we found comorbid psychiatric diagnoses are affecting the outcome of ASD symptoms in clinical observation. The machine learning models reveal several others are more significant (e.g. parental age, birth weight, sociodemographic variables, etc.) in terms of prognostic information and also planning treatment of children with ASD.

KEYWORDS:

• Autism
• machine learning
• prognosis
• predictive measures

Introduction

Autism Spectrum Disorder (ASD) refers to a group of complex neurodevelopmental conditions that occurs within early years of life [1]. ASD characterized with deficits in social interaction, repetitive or stereotypic behaviours, sensory abnormalities and verbal and non-verbal communication [2]. In the treatment of ASD, special education [3], behavioural therapy [4] and psychopharmacological therapies for comorbid psychiatric disorders and irritability [5] are indicated. Recent studies showed improvement in core symptoms of ASD but different results among individual participants had reported affected by various child and family-specific factors, such as pretreatment IQ, comorbid psychiatric disorders, adaptive, and language skills [6]. A major group of research has focused on heritable and genetic causes of ASD [7], which shows to have a highly heritable component and these research point out etiological heterogeneity. Also, research has shown that in children with ASD the rate of de novo genetic variants, that means mutations probably resulted in environmental stressors [8]. As a result, recent studies showed prenatal maternal environment [9,10], parental age [11], early behavioural intervention [12] increasingly being recognized as important for the developmental outcome.

Machine learning methods are novel computational methods include mathematical learning, statistical estimation, and information theories automatically discovering useful patterns in large amounts of data. This method has the advantage of accurate and reliable prediction using data with very large numbers of variables and causal inference within non-experimental data sets [13]. Recent studies in psychiatry are showed these methods been successfully used in diagnostics of ASD [14], classification by altered event-related potentials in Attention Deficit Hyperactivity Disorder [15], and classification by free speech analysis in Schizophrenia [16]. Bishop et al. [17] used machine learning methods to examine adult ASD patients lifetime health problems, and these methods precisely predicted health problems, including cardiovascular, urinary, respiratory systems.

We examined predictors of outcome after 3-year clinical observation, special education and psychiatric treatment in the clinical-based group. To our knowledge in literature, no previous study has examined the psychiatric, developmental and sociodemographic aspects of effecting prognosis on children with ASD using machine learning methods. We used our follow up data for the ASD group and tried to predict short-time outcomes of ASD and find clinical and individual factors affecting improvement in core symptoms of autism.

Methods

The study was carried in Ondokuz Mayis University Samsun, in the years, 2013–2016 with a naturalistic design. All participants in the clinical group had been previously diagnosed according to with the criteria described in the Diagnostic and Statistical Manual of Mental Disorders-IV TR by child psychiatry specialists and also comorbid psychiatric diagnosis is assessed and treatment planned by child psychiatry specialists. Caregivers filled sociodemographic and rehabilitation information form. ASD symptoms were assessed by child psychiatrist using the Autism Behavior Checklist, Aberrant Behavior Checklist, Clinical Global Impression scales at baseline (T0) and 12 (T1), 24 (T2), 36 (T3) months.

Sociodemographic data form

A sociodemographic form was completed at T0 by researchers with information related to the child, father, mother. Questions form were related to the mother and father’s age, the number of siblings, the economic status of the family (low 1500TL, middle 1500–5000TL and high-income 5000TL), the patient’s and the family’s level of education (educated years), medical history (chronic medical diseases), and self-reported history of the psychiatric diseases in first and second degree relatives.

Clinical information form

Child and Adolescent Psychiatry residents completed rehabilitation hours and medical treatments at T0, T1, T2, and T3 in this data form.

Autism Behavior Checklist

Autistic type behaviour problems were measured by the clinician using the Autism Behavior Checklist (ABC) a questionnaire pertaining to the core symptoms of autism [18]. High Test scores indicate different degrees of autistic behaviour such as socially non-responsive behaviour, stereotypic behaviour, perceptual oddities, and echolalic speech. The maximum possible score is 158. The Turkish language version of the ABC has been validated, a cut-off point of 39 was determined and the alpha coefficient and split half reliability for the ABC total score were 0.92 [19]. Child and Adolescent psychiatry specialists completed the Aberrant Behavior Checklist scale at T0, T1, T2, and T3.

Aberrant Behavior Checklist

Rated on a 4-point Likert scale, 58-item informant-based measure of problem behaviours of people with autism or other developmental disabilities [20]. There are five subscales: Social Withdrawal (16 items), Hyperactivity (16 items), Irritability (15 items), Stereotypic Behavior (7 items), and Inappropriate Speech (4 items). Alpha coefficients have ranged from 0.77 to 0.95 across subscales [21]. The Turkish translation and adaptation of the Aberrant Behavior Checklist was conducted on adolescent individuals with intellectual disability [22,23]. Adaptation study of Aberrant Behavior Checklist for preadolescent children was performed later; to do this, three items were rewritten to be congruent with this age range [24]. Internal consistencies for all Aberrant Behavior Checklist subscales were found to be moderate to high. Cronbach’s alpha values were as follows: (I) Irritability, Agitation, Crying: 0.90; (II) Lethargy, Social Withdrawal: 0.81; (III) Stereotypic Behavior: 0.83; (IV) Hyperactivity, Non-compliance: 0.89; (V) Inappropriate Speech: 0.68. Child and Adolescent psychiatry specialists completed the Aberrant Behavior Checklist scale at T0, T1, T2 and T3.

Clinical global impression

The Clinical Global Impression Scale (CGI) was developed by Guy et al. to evaluate the course of all psychiatric disorders at all ages. The CGI is a scale 7-point Likert type scoring and has three dimensions to evaluate the treatment responses of individuals with psychiatric disorders and is completed by the physician during the interview. In this study, the subscale of Clinical Global Impression Scale-Severity was used to evaluate patients. Child and Adolescent psychiatry specialists completed CGI scale at T0, T1, T2, T3.

Machine learning methods

We pre-processed the data in two steps. Firstly, we have cleaned data by using imputation for missing values. We used column-wise delete (if more than 30% data is missing), the last-observation-carried-forward method for CGI points for imputation and for other missing values imputed by using k-nearest neighbourhood method. Secondly, we have applied dimension reduction via principal component analysis. We have converted CGI points into a discrete classification problem in order to apply most of the classification algorithms. Variables which had high rates of correlation (if more than 60%) with CGI points did not use for outcome overfitting. We used last-observation-carried forward method for lost patients in follow-up.

We tested the performance of for machine learning algorithms (Naive Bayes, Generalized Linear Model, Logistic Regression, Decision Tree) on our data, using the 254 items in the sociodemographic and clinical data form, Autistic Behavior Checklist, Aberrant Behavior Checklist as features, and the CGI based (Patients with ≤2 CGI points in ASD symptoms at in 36 months were accepted as the group who has “better outcome”) as the prediction class. We used tenfold cross-validation for train and test sets, where the data split is repeated 10 times with 90% training to 10% test set by changing the test set each time. At the end, the success rates and error rates are calculated by the average of each repetition.

We created prediction models using four machine learning classifiers independently with tenfold cross-validation (Table 1). This process was repeated using all sub-sets and means acquired by this process calculated for Receiver Operating Characteristic curves with Area Under Curve. In order to prevent overfitting issues, algorithms modified with their major settings and left the minor tuning unchanged default values. Modelling was performed with RapidMiner Studio software [25]. After results of cross-validation, we chose automatically optimized decision tree method for the highest AUC (0.707) value (Figure 1).

Figure 1. Decision tree method for prediction of short-term outcome.

Table 1. Results of cross-validation for each classifier.

Classifier	AUC	Accuracy (%)	Sensitivity (%)	Specificity (%)
Naive Bayes	0.696	50.0	95.2	19.4
Generalized linear model	0.668	69.2	80.5	54.8
Logictic regression	0.665	67.3	85.7	54.8
Decision tree	0.707	71.2	81.1	61.3

Ethics committee approval

Approval for the study was granted by the Clinical Research Ethics Committee of Ondokuz University with approval no. B.30.2.ODM.0.20.08/1806, dated 07/2018.

Results

The study group comprised 433 children with ASD diagnosis. 79.4% were male and, 42.2% were diagnosed as Autistic Disorder, 48.5% were diagnosed with Pervasive Developmental Disorder-NOS and %9.2 were Asperger Disorder at time T0. At T0, participants mean age was 72.3 ± 45.9 months and %24.4 of them were younger than 36 months. The comorbid diagnosis was Attention Deficit and Hyperactivity Disorder (n:265, 61.2%), Obsessive Compulsive Disorder (n:105, 24.2%) and Bipolar Disorder (n:32, 7.4%). Other medical diagnoses (epilepsy, cerebral palsy, genetic disorders etc.) are presented of 22.3% of the group.

265 of 433 (61.2%) children were in follow-up more than 12 months. At T2, 219 (50.5%), at T3; 185 (42.7%) children were in follow up. Table 2 shows sociodemographic and clinical variables and weights for machine learning model and Figure 1 shows decision tree diagram for prediction “better outcome” (AUC: 0.707, sensitivity: 81.1%, specificity: 61.3%).

Table 2. Weights of sociodemographic and clinical variables of the study group.

Study parameter	Mean ± SD	Weights
Age (months)	72.3 ± 45.9		0.018
Gender	79.4% Male		0.092
Birth weight (gr)	3216.1 ± 757.2		0.363
Age at autism diagnosis (months)	36.8 ± 21.4		0.204
Developmental Cornerstones (months)	Walking	15.9 ± 8.1	0.361
	First word	20.2 ± 11.1
	Two words	37.9 ± 13.3
Mother age at birth (years)	29.7 ± 4.8		0.325
Father age at birth (years)	32.8 ± 5.5		0.309
Mother education (years)	12.0 ± 3.7		0.301
Father education (years)	12.5 ± 3.5		0.159
Age at rehabilitation started (months)	39.8 ± 18.8		0.128
DSM-IV diagnosis	Autistic disorder	42.2%	1.000
	Pervasive Developmental Disorder-NOS	48.5%
	Asperger disorder	9.2%
Comorbid Diagnosis	ADHD	61.2%	0.361
	MR	27.4%
	OCD	24.2%
	Bipolar disorder	7.3%
	Other neurological diagnosis	22.3%
Autism CGI score at T0	3.9 ± 1.0		0.101

Abbreviations: CGI: clinical global impression; ADHD: attention deficit and hyperactivity disorder; MR: mental retardation; NOS: not other specified.

Discussion

Our aim was to analyse several sociodemographic and clinical factors with regard to ASD prognosis. We evaluated the predictors of “better outcome” in 36-month follow-up among 433 patients. We found improvement in symptoms was substantial: 39.7% at 12 months, 60.7% at 24 months and 77.8% at 36 months. Diagnostic groups as per DSM-IV-TR affected prognoses. Autism group, father and mother age are the most powerful factors for prognosis. In PDD-NOS group, MR comorbidity, birth weight, and age at diagnosis have effected a better outcome. Asperger’s Disorder age at diagnosis, age at first evaluation and developmental cornerstone (speak two words) has effected a better outcome group.

In our analysis of predictors, we compared better and poor outcomes regarding the frequency of several clinical factors. We used four machine learning methods for predict outcome and us choose a decision tree for data visualization, as it has the highest AUC and accuracy belong to this methods. The decision tree is a non-parametric supervised learning method used for classification and regression, and widely used to visually and explicitly represent prediction and decision making in clinical research [25]. This method classifies patient groups into branch-like segments that construct an inverted tree with a root node and leaf nodes. Our best model showed (81% sensitivity, 61% specificity) as a decision tree (Table 1), diagnosis groups effects prognosis. These results are preliminary but in ASD like complex neurodevelopmental disorders, this method may help clinicians to identify subgroups of patients. This approach may provide different treatment strategies to achieve the optimal medical outcome.

We found different factors had different weights on predicting the outcome (Table 2). ASD severity and sub-groups of diagnosis is affected prognosis. DSM-IV subgroups discussed and gathered in one large diagnosis, Autism Spectrum Disorders DSM-5 but still different prognostic outcomes showed between literature [26]. Also, comorbid psychiatric and neurological diagnosis is effected prognosis. We found high rates of ADHD in our study group. ADHD has been shown to occur at higher rates in children with ASD than their typically developing peers [27] and prognostic relevance of social dysfunctioning associated with ADHD [28]. Mental retardation comorbidity effected prognosis, which low cognitive, delayed developmental skills are known to have negative effects on the outcome [29]. Other neurological conditions (epilepsy, cerebral palsy etc.) are common in autism [30,31] and we found they have effects on the important determinant of short-term outcome.

Mother and father age at birth had effected the outcome in our study. In literature advancing paternal age associated with autism which explained by epigenetic alterations associated with aging [32]. Also advancing parental age was more strongly associated with autism with intellectual disabilities [11]. Maternal age is associated with younger age of first evaluation and reported as cofactor of maternal education and other sociodemographic characteristics [34]. We found the birth weight is also an important determinant for the outcome. Literature shows, birth weight of <2500 g were increased risk for autism, although the magnitude of risk from these factors varied according to autism with comorbidities [35]. Low birth weight also associated with a high risk of developing psychiatric symptoms and disorders by adolescence [36] and also emotional, behavioural, social and also academic problems at later times [37].

Autism research requires a more detailed characterization of core ASD symptoms (repetitive behaviours, social deficit) for clinical reasons and neurobiological reasons. Researchers will need more data to identify this symptom, that can help understand the disorder. Also, the prognosis is so heterogeneity that we need reliable and efficient, dimensional measures to help our clinical decisions. Our results provide preliminary support to the decision tree method as an informative method that can assist clinical decision in predicting treatment response in Autism Spectrum Disorders, though further work is required to provide enhanced levels of classification and prediction performance.

Limitations

First of all, there are limitations to the decision tree method. The main disadvantage is that it can be subject to overfitting. There are many factors affecting the outcome of autism, overfitting occurs particularly when using small data sets. We used cross-fold validation to avoid this problem, it is unlikely that this represents overfitting and overestimation of the AUC. Still, this problem can limit the generalizability and robustness of the resultant models. Our data were retrospective and we used short-term variables and also clinician-rated improvement in autism, that may cause overlook of factors that effect core symptoms in treatment. Complex cases with neurological comorbidities did not investigate, that lower generalizability of this study. Also genetic etiologies (i.e. Fragile X, Di-George Syndrome) did not investigated, may affected the outcome [33,34].

Disclosure statement

No potential conflict of interest was reported by the authors.

ORCID

Mirac Baris Usta http://orcid.org/0000-0002-1573-3165

Koray Karabekiroglu http://orcid.org/0000-0001-5439-9940

Berkan Sahin http://orcid.org/0000-0003-4699-3418

Muazzez Aydin http://orcid.org/0000-0002-3099-7337

Armagan Aral http://orcid.org/0000-0002-8069-7586

Emre Ürer http://orcid.org/0000-0002-0562-837X

Additional information

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.