Social cognition in adults with autism spectrum disorders: Validation of the Edinburgh Social Cognition Test (ESCoT)

Abstract

Objective

Many existing tests of social cognition are not appropriate for clinical use, due to their length, complexity or uncertainty in what they are assessing. The Edinburgh Social Cognition Test (ESCoT) is a new test of social cognition that assesses affective and cognitive Theory of Mind as well as inter- and intrapersonal understanding of social norms using animated interactions.

Method

To support the development of the ESCoT as a clinical tool, we derived cut-off scores from a neurotypical population (n = 236) and sought to validate the ESCoT in a sample of Autism Spectrum Disorder (ASD; n = 19) adults and neurotypical controls (NC; n = 38) matched on age and education. The ESCoT was administered alongside established tests and questionnaire measures of ASD, empathy, systemizing traits and intelligence.

Results

Performance on the subtests of the ESCoT and ESCoT total scores correlated with performance on traditional tests, demonstrating convergent validity. ASD adults performed poorer on all measures of social cognition. Unlike the ESCoT, performance on the established tests was predicted by verbal comprehension abilities. Using a ROC curve analysis, we showed that the ESCoT was more effective than existing tests at differentiating ASD adults from NC. Furthermore, a total of 42.11% of ASD adults were impaired on the ESCoT compared to 0% of NC adults.

Conclusions

Overall these results demonstrate that the ESCoT is a useful test for clinical assessment and can aid in the detection of potential difficulties in ToM and social norm understanding.

Keywords:

• Autism Spectrum Disorder
• Edinburgh Social Cognition Test (ESCoT)
• social cognition
• theory of mind
• understanding social norms

Individuals with Autism Spectrum Disorder (ASD) experience difficulties in social functioning as a core feature of ASD (American Psychiatric Association [APA], 2013). The central processes thought to be necessary for effective social functioning are referred to as social cognition (Adolphs, 2009; Baez et al., 2016; Baez et al., 2012; Henry et al., 2015; Van Overwalle, 2009). In everyday social interactions, we use social cognitive abilities such as theory of mind (ToM; i.e., the ability to recognize other people’s mental states to understand and predict their behavior) and the understanding of social norms (Baez et al., 2012, 2013) to interact and respond appropriately to others.

ToM is an important social cognitive ability, and ToM difficulties can cause significant social deficits which profoundly limit functional capacity to engage in meaningful interpersonal relationships and quality of life (Henry et al., 2015). ToM is a multi-dimensional concept, with processes differing based on cognitive or affective judgments (Shamay-Tsoory et al., 2010). Cognitive ToM is defined as the ability to make inferences about the intentions and beliefs of another individual. Affective ToM refers to the ability to make inferences about what another individual is feeling (Kalbe et al., 2010; Sebastian et al., 2011; Shamay-Tsoory et al., 2010). Considerable research has shown that ASD adults have difficulties with both aspects of ToM (Baron-Cohen et al., 2001; Mathersul et al., 2013; Murray et al., 2017).

The specific ToM difficulties that ASD adults experience have been studied extensively over the last several decades (Murray et al., 2017). However, the literature consists of many incongruent findings (Baron-Cohen et al., 2001; Couture et al., 2010; Roeyers et al., 2001); potentially as a consequence of the tests used to assess ToM. Early studies examining ToM in adults used false-belief tests designed for children and found that ASD adults performed as well as neurotypical controls (NC) (Happé, 1994; Jolliffe & Baron-Cohen, 1999; White et al., 2009) but still show marked problems in social interactions in everyday life (Dziobek et al., 2006; Palmen et al., 2012). To overcome this limitation in sensitivity, researchers developed more advanced tests and demonstrated difficulties in adults with ASD on the Reading the Mind in the Eyes (RME) (Baron-Cohen, Wheelwright, Hill, et al., 2001), the Awkward Moments Test (Heavey et al., 2000), the Movie for the Assessment of Social Cognition (MASC) (Dziobek et al., 2006), Reading the Mind in Films Test (RMF) (Golan et al., 2006), The Awareness of Social Inference Test (TASIT) (Mathersul et al., 2013) and the Strange Stories Test (Murray et al., 2017).

Yet, these advanced tests are not without their limitations. For instance, ASD adults can pass forced-choice social cognition tests such as the RME (Baez et al., 2012; Izuma et al., 2011; Klin, 2000; Schilbach et al., 2012), but have difficulties on tests which require spontaneous attributions of mental states (Senju et al., 2009). Moreover, ToM tests using written stories do not capture contextually specific ToM abilities used in everyday social interactions (Frith, 2004; Klin, 2000; Lugnegård et al., 2013). Individuals can pass the Strange Stories test but still exhibit difficulties in real-world social interactions (Scheeren et al., 2013). The abstract nature of tests which use forced-choice answers and lack context limits their ecological validity because the relationship to real-world functioning is unclear (Mathersul et al., 2013). To address this limitation, McDonald et al. (2003) developed the TASIT, which uses short-clips of social interactions. However, it is a lengthy test with an administration time of 60–75 minutes (Mathersul et al., 2013; McDonald et al., 2003), which limits the TASIT’s application in time-sensitive clinical environments. The MASC (Dziobek et al., 2006) is similar to the TASIT; yet, it is a verbal interaction between characters that is dubbed into English. As such, important contextual information relating to the interactions may have been lost in translation, limiting its use in English speaking populations.

Other limitations include current tests of social cognition potentially being related to intellectual abilities. Verbal comprehension significantly correlates with or predicts performance on the RME (Baker et al., 2014), Strange Stories test (Kaland et al., 2002), RMF (Golan et al., 2006) and the TASIT (McDonald et al., 2003). Perceptual reasoning also significantly correlates with performance on the RME (Baker et al., 2014). Such findings may limit the interpretation from these tests.

While most existing tests have focused on assessing ToM, social cognition consists of several different abilities that are simultaneously required during social interactions. Other social cognitive abilities have received less attention in the literature. An individual’s interpersonal (how another person should behave) and intrapersonal (how they themselves should be behave) understanding of the social norms that govern their behavior are important social cognitive abilities. Violating a social norm can be detrimental to existing relationships or opportunities to form social relationships. Inter- and intrapersonal understanding of social norms have been examined separately in ASD adults, showing mixed findings (Baez et al., 2012; Gleichgerrcht et al., 2013; Lehnhardt et al., 2011; Thiébaut et al., 2016; Zalla et al., 2009). To our knowledge, there is currently no clinical test of inter- and intrapersonal understanding of social norms within the same test. Similarly, these abilities have never been examined alongside ToM abilities within the same test. Researchers typically investigate social norm understanding and ToM using different tests. For example, Baez et al. (2012) found that ASD adults were not impaired on affective ToM measured by the RME or intrapersonal understanding of social norms assessed by the Social Norms Questionnaire (Rankin, 2008). This makes direct comparisons problematic, since these different tests may vary in difficulty.

Aims of the current study

The Edinburgh Test of Social Cognition (ESCoT) (Baksh et al., 2018) is a recently developed test of social cognition which assesses cognitive and affective ToM and inter- and intrapersonal understanding of social norms within the same test. In NC adults between the ages of 18–85, it was demonstrated that poorer performance on inter- and intrapersonal understanding of social norms and ESCoT total scores was predicted by the presence of more autism-like traits. Increasing age predicted poorer performance on interpersonal understanding of social norms, cognitive and affective ToM, as well as ESCoT total scores. In addition, female participants were better at inferring what another person was feeling. Finally, performance on the ESCoT was not predicted by verbal comprehension or perceptual reasoning abilities unlike established tests (Baksh et al., 2018). The aim of the present study was to validate the ESCoT in a sample of ASD adults. While new tests have been published such as Strange Stories Film Task (Murray et al., 2017) and Story-based Empathy Task (Dodich et al., 2015), these tests only assess ToM. Moreover, none of the tests assess within-subjects’ social cognitive abilities in different contexts, as well as assessing four social cognitive abilities within the same test.

Aims

1. To examine the convergent validity of the ESCoT against established tests of social cognition. We predicted that better performance on the ESCoT would correlate with better performance on the traditional tests of social cognition.

2. To compare ASD adults and NC adults on the ESCoT and established tests of social cognition. Based on previous findings in the literature, we predicted that ASD adults would be impaired on cognitive ToM; affective ToM and interpersonal understanding of social norms, but not intrapersonal understanding of social norms compared to NC adults.

3. To evaluate the psychometric properties of the ESCoT and compare these to traditional social cognition tests by examining the influence of intelligence, ASD traits, empathy and systemizing traits on performance.

Method

Edinburgh Social Cognition Test (ESCoT)

The ESCoT consists of eleven dynamic, cartoon-style social interactions (each approximately 30 seconds long): one practice interaction, five interactions involving social norm violations and five interactions without social norm violations. Participants watched the animated interaction on a computer screen and a static storyboard depicting a summarized version of the interaction was presented at the end. The storyboard remained on the screen during the subsequent questions for each interaction. Please see Figure 1 for an example interaction.

Figure 1. An example interaction from the Edinburgh Social Cognition Test (ESCoT).

Participants were asked five questions after viewing each animation relating to: (1) general story comprehension; (2) cognitive ToM; (3) affective ToM; (4) interpersonal understanding of social norms; and (5) intrapersonal understanding of social norms. See Table 1 and 2 for further details.

Table 1. Description of the questions from the Edinburgh Social Cognition Test (ESCoT).

ESCoT question	Social cognitive ability	Purpose
Can you tell me what's happening in this story, starting with the first picture and finishing with the last picture?		Allow participants to describe what occurred in the social interaction.
What is X thinking?	Cognitive ToM	Assess participants’ ability to make inferences about another person’s cognitive state.
How does X feel at the end of the animation?	Affective ToM	Assess the ability to make inferences about the affective states of the character in the interaction.
Did X behave as other people should behave?	Interpersonal understanding of social norms	Assess participants’ understanding of the subtle societal rules that govern social behavior and whether the character in the interaction was adhering to these rules.
Would you have acted the same as X in the animation?	Intrapersonal understanding of social norms	Assess how the participants themselves would have acted in the interaction.

Table 2. Example scoring of the Edinburgh Social Cognition Test (ESCoT) taken from Scenario 1.

ESCoT subcomponent	Points	Scoring instructions
Cognitive ToM (max = 3)	3	A social answer that recognises that the elderly lady required assistance and provides a contextual reason of why she needed assistance. Mention of affective state limits mark to 2 points.
	2	A social answer that recognises that the elderly lady required assistance. No more than 2 points can be gained if a contextual reason is not given (prompt).
	1	A non-social answer that recognises that the elderly lady required assistance. No more than 1 point can be gained if there is no mention of the other person from the interaction in the response, even with a contextual reason (prompt).
	0	Don't know/irrelevant answer.
Affective ToM (max = 3)	3	An answer that demonstrates a higher order emotional understanding, with a contextual reason (prompt).
	2	An answer that demonstrates a lower order emotional understanding, with a contextual reason (prompt).
	1	An answer that demonstrates a higher or lower order emotional understanding with no contextual reason (prompt).
	0	Don't know/irrelevant answer.
Interpersonal understanding of social norms (max = 3)	3	A social answer that exhibits an understanding of the relevant social norm highlighting that the man acted in a socially unacceptable manner, and provides a contextual explanation of why the lady needed help.
	2	A social answer that exhibits an understanding of the relevant social norm highlighting that the man acted in a socially unacceptable manner. No more than 2 points can be gained if a contextual explanation is not given (prompt).
	1	A non-social answer that exhibits an understanding of the relevant social norm highlighting that the man acted in a socially unacceptable manner (prompt) OR states that the man acted appropriately.
	0	Don't know/irrelevant answer.
Intrapersonal understanding of social norms (max = 3)	0	Participants receive 0 points for answering the YES/NO question (‘Would you have acted the same as the person in the animation?’) incorrectly.
	1	Participants receive 1 point for answering the YES/NO question (‘Would you have acted the same as the person in the animation?’) correctly.
	1 or 2	An additional 1 point is awarded for answers which highlight a personal attribute of the participant (e.g. I’m a nice person) while answers which highlight the context of the animation receive an additional 2 points.

The responses for cognitive ToM, affective ToM and interpersonal understanding of social norms were scored similarly. Scores were based on the quality of the answer with maximum points awarded for successfully extracting and integrating the relevant information and articulating it in a contextually specific manner. Importantly, response length was not related to quality; participants could score maximum points with a minimal response. For the intrapersonal understanding of social norms, responses that considered the social nuances of the interaction were scored more highly than responses that highlighted personal attributes of the participant.

To allow participants to give their optimal interpretation of each interaction and capture the quality of their response, each participant was prompted once with the question, “Can you tell me more about what you mean by that?” or “Can you explain that in a little bit more detail?” Participants were prompted if they gave a limited response or their response lacked important information from the interaction. The general comprehension question was not scored as misinterpretations of the social interaction would become evident when participants answered the subsequent social cognition test questions. Participants were asked the social cognition test questions even if they misinterpreted the social interaction. Each question was awarded a maximum of 3 points, with a maximum score of 30 points for each social cognitive ability. The maximum total score was 120 points and the ESCoT took approximately 20–25 minutes to complete.

Participants

Nineteen adults (12 males, 7 females) aged 19–66 years (M = 38.47, SD = 15.63) with a diagnosis of Asperger’s Syndrome (AS) or High-Functioning Autism (HFA) according to established DSM-IV criteria (American Psychiatric Association [APA], 2000) were recruited from charities, support groups and from a research database (cf. according to DSM-V, they would all be diagnosed as Autism Spectrum Disorder). Participants confirmed their clinical diagnosis of ASD via official diagnosis letters which they received from their clinician. A comparison group of thirty-eight NC adults (23 males, 15 females) aged 19–67 years (M = 37.50, SD = 17.75) were recruited using online advertisement and through a research volunteer panel.

We used previously acquired data from 236 NC adults between the ages of 18 and 85 (some were included in this study as controls) to establish normative data and derive ESCoT cut-off scores for the subtests and total scores based on the lowest 5^th percentile. These individuals were recruited through online advertisement and through a research volunteer panel. This included 147 younger adults (67 males, M age = 23.39, SD = 4.11, range = 18–35, M education = 16.90, SD = 2.20), 30 middle-aged adults (15 males, M = 50.60, SD = 5.77, range = 45–60, M education = 15.53, SD = 2.86) and 59 older adults (23 males, M = 72.44, SD = 6.05, range = 65–85, M education = 14.58, SD = 2.88). None of the NC adults had any self-reported history of neurological or psychiatric disorders based on the exclusion criteria from the Wechsler Adult Intelligence Scale (WAIS-III) (Wechsler, 1997). Informed consent was obtained from all individuals and the study was approved by the local ethics committee.

The participants’ demographic information, ASD screening questionnaires and IQ scores are reported in Table 3.

Table 3. Demographics information of participants: Mean (SD).

	Max score	ASD n = 19	ASD range	NC n = 38	NC range	Sig*	ηp² (d)
Age		38.47 (15.63)	19–66	37.50 (14.75)	19–67	ASD = NC	0.001 (0.09)
Sex (M:F)		12:7	–	23:15	–	–	–
Years of full-time education		15.21 (2.53)	11–20	16.16 (2.54)	11–21	ASD = NC	0.03 (0.36)
AQ	50	34.63 (7.43)	14–45	16.34 (5.07)	6–25	ASD < NC	0.69 (2.95)
EQ	80	14.21 (8.55)	4–34	29.92 (7.20)	7–42	ASD < NC	0.49 (1.97)
SQ	150	48.11 (19.88)	19–100	45.32 (12.71)	22–70	ASD = NC	0.01 (0.22)
VCI	160	95.50 (15.62)	68–130	105.02 (9.97)^a	76–120	ASD < NC	0.19 (0.96)
PRI	135	99.06 (21.13)	64–131	107.94 (13.07)^a	81–134	ASD = NC	0.10 (0.68)

ASD, Autism Spectrum Disorder; AQ, Autism Spectrum Quotient; EQ, Empathy Quotient; SQ, Systemizing Quotient; VCI, Verbal Comprehension Index; PRI, Perceptual Reasoning Index.

* Analyses were conducted using parametric and non-parametric tests where appropriate. All p < .05.

^a NC n = 37.

Measures

Wechsler Abbreviated Scale of Intelligence, Second Edition (WASI-II) (Wechsler, 2011): The WASI-II was administered as a measure of verbal comprehension and perceptual reasoning. Participants completed four subtests: Vocabulary; Similarities; Block Design; and Matrix Reasoning. The Vocabulary and Similarities subsets provided a Verbal Comprehension Index (VCI) and Block Design and Matrix Reasoning provided a Perceptual Reasoning Index (PRI) (McCrimmon & Smith, 2013; Wechsler, 2011).

Autism Spectrum Quotient (AQ) (Baron-Cohen et al., 2001): This self-report questionnaire assesses whether individuals with a normal IQ possess traits related to the autism spectrum (maximum score = 50).

The Empathy Quotient (EQ) (Baron-Cohen & Wheelwright, 2004): The EQ measures the ability to identify and understand the thoughts and feelings of others and to respond to these with appropriate emotions (maximum score = 80).

The Systemizing Quotient (SQ) (Wheelwright et al., 2006): The SQ assesses the drive to analyze or construct systems such as mechanical systems (maximum score = 150).

The Reading the Mind in the Eyes (RME) (Baron-Cohen et al., 2001): Participants were presented with photographs of the eye region of human faces and selected a response from four adjectives which best described what the individual was thinking or feeling. Prior to performing the test, participants were provided with a glossary to clarify what each adjective meant, in case they were unsure or unfamiliar with the word. Participants were given unlimited time to respond (maximum score = 36). Higher scores indicated better performance.

The Reading the Mind in Films (RMF) (Golan et al., 2006): Participants viewed short scenes of varying durations involving social interactions from feature films and selected a response from four adjectives that best described what the protagonist was thinking or feeling at the end of the scene. Again, participants were provided with a glossary of the adjectives for clarification and responded verbally. There was no time limit for responses and participants could score a maximum of 22 with higher scores indicating better performance.

The Social Norms Questionnaire (SNQ) (Rankin, 2008): The SNQ was developed to screen patients for potential behavior changes and examines how well participants understand the social standards that govern their behavior in UK mainstream culture. Participants were given a list of behaviors (e.g. tell a stranger you don’t like their hairstyle?) and asked to indicate whether or not the behaviors were socially acceptable to perform in the presence of a stranger or acquaintance, not a close friend or family member (maximum score = 22). Higher scores indicated better performance. The SNQ also calculates the types of errors made by participants. An over-adherence error occurs when the statement is socially acceptable, but the participant disagrees with it. A rule-break error is when responses violate a social norm.

Procedure

Participants completed the measures in a single session, taking approximately two hours to complete. The ASD questionnaires were completed online. The order of the tasks was the same for each participant and regular breaks were provided.

Statistical analysis

Inter-rater reliability was assessed using intraclass correlations (ICC) for a random 10% of the sample using raters blind to the diagnosis of the groups. The analysed ESCoT data were age-adjusted based on the regression analyses published in Baksh et al. (2018; see supplementary materials). Parametric and non-parametric analyses were conducted based on initial exploratory analyses (Shapiro-Wilk test, p > 0.05). Correlational analyses were conducted on all participants using Spearman’s rho correlational analyses to examine the relationship between the ESCoT and the established social cognition tests. To examine overall differences on the ESCoT subtests (cognitive ToM, affective ToM, inter- and intrapersonal understanding of social norms) and SNQ subtests (over-adherence and rule break), a Friedman Test was used. If this yielded a significant difference, follow-up analyses were conducted using independent samples and paired-samples t-tests for cognitive and affective ToM while Mann-Whitney U and Wilcoxon signed-rank tests were performed for inter- and intrapersonal understanding of social norms and the SNQ subtests. An independent sample t-test was performed to examine mean group differences on the RME. Mann-Whitney U tests were performed on ESCoT total scores, RMF and SNQ total scores. The alpha values were set at p < 0.05 and the Holm correction to adjust for multiple comparisons was applied. Effect sizes using both partial eta squared (ηp²) and Cohen’s d (Cohen, 1988, 1992) were calculated.

The relationship between performance on all social cognition tests and the ASD screening questionnaires (AQ, EQ and SQ) were examined using an exploratory regression analysis. In the first stage, the background predictors (age, sex, years of education) which significantly correlated with the outcome variables (ESCoT total scores and established social cognition tests) at a pre-specified significance level of p < 0.20 were entered into the analysis (Altman, 1991) using the enter method. We chose a significance level of p < 0.20 over more traditional levels such as p < 0.05 since p < 0.05 can fail in identifying variables known to be important to the outcome variable and simulation studies have shown that a cut-off of p < 0.20 yields better outcomes (Bursac et al., 2008; Lee, 2014). VCI scores were included in the first stage of the regression analysis if VCI scores correlated with the outcome variables (ESCoT total score, RME, RMF and SNQ total scores) at p < 0.20. In the second stage, AQ, EQ and SQ scores were entered using the stepwise method (entry criterion p < 0.05, removal criterion p > 0.10). Finally, we conducted Area Under The Curve (AUC) and Receiver Operator Characteristic (ROC) curve analysis to examine the discriminative abilities of the ESCoT and established tests.

Results

Cut-off scores to detect abnormal performance on the ESCoT

Raw score age adjustments were applied (Baksh et al., 2018; see supplementary materials). ESCoT cut-off scores for the subtests and total scores were based on the lowest 5^th percentile. Using the cut-offs, 6.77% of our 236 NC were impaired on the cognitive ToM subtest, 4.66% on affective ToM, 6.77% on interpersonal understanding of social norms and 6.77% on intrapersonal understanding of social norms. 5.50% of NC adults were impaired on ESCoT total score. Table 4 shows the cut-offs for each subtest and ESCoT total scores.

Table 4. Cut-off scores for the Edinburgh Social Cognition Test (ESCoT) subtests and total score based on the 5^th percentile.

	Maximum score	Cut-off score
Cognitive ToM	30	17 or less
Affective ToM	30	19 or less
Interpersonal understanding of social norms	30	18 or less
Intrapersonal understanding of social norms	30	22 or less
ESCoT total score	120	83 or less

Inter-rater reliability

Excellent inter-rater reliability (Cicchetti, 1994) was observed for all subtests of the ESCoT: cognitive ToM (ICC = 0.80), affective ToM (ICC = 0.97), interpersonal understanding of social norms (ICC = 0.96), intrapersonal understanding of social norms (ICC = 0.98) and the ESCoT total score (ICC = 0.97).

As Table 5 shows the cognitive ToM subtest of the ESCoT significantly correlated with the RME, RMF and SNQ. The affective ToM subtest positively correlated with the RMF. Interpersonal understanding of social norms significantly correlated with the RME and RMF. Intrapersonal understanding of social norms significantly correlated with the RME.

Table 5. Correlations between the tests of social cognition.

	ESCoT total score	Cognitive ToM	Affective ToM	Interpersonal norms	Intrapersonal norms	RME	RMF
Cognitive ToM	0.64***
Affective ToM	0.58***	0.33*
Interpersonal norms	0.86***	0.39**	0.28*
Intrapersonal norms	0.47***	0.002	−0.08	0.55***
RME	0.48**	0.36**	0.25	0.38**	0.27*
RMF	0.42**	0.39**	0.36**	0.30*	0.10	0.62***
SNQ	0.34*	0.33*	0.24	0.19	0.16	0.33*	0.39**

* p < 0.05

** p < 0.01

*** p < 0.001.

ToM; Theory of Mind, ESCoT; Edinburgh Social Cognition Test, RME, Reading the Mind in the Eyes, RMF, Reading the Mind in Films, SNQ, Social Norms Questionnaire. Analyses were conducted using non-parametric tests.

ESCoT total scores significantly positively correlated with the RME, RMF and the SNQ. The RME test significantly correlated with the RMF and SNQ. Performance on the RMF positively correlated with performance on the SNQ.

ESCoT subtest correlations

Cognitive ToM significantly correlated with affective ToM and interpersonal understanding of social norms. While affective ToM positively correlated with interpersonal understanding of social norms, performance on interpersonal understanding of social norms correlated with performance on the intrapersonal understanding of social norms. Intrapersonal understanding of social norms did not significantly correlate with performance on cognitive ToM or affective ToM. All subtests significantly correlated with ESCoT total score.

Group comparisons between ASD adults and NC on ESCoT and established social cognition tests

A non-parametric Friedman test showed a statistically significant difference between the subtests of the ESCoT (χ² (3) = 74.91, p < 0.001) for NC and ASD. Post-hoc analysis with Holm correction for multiple comparisons demonstrated performance was poorer on the cognitive than the affective ToM (t(56) = −7.17, p < 0.001, ηp² = 0.20, d = 1.04). As shown in Table 6, ASD adults scored significantly lower than NC adults, on cognitive ToM (t(23.26) = −4.40, p < 0.001, ηp² = 0.45, d = 1.82) and affective ToM (t(23.76) = −3.70, p < 0.01, ηp² = 0.37, d = 1.52).

Table 6. Results by group for the Edinburgh Social Cognition Test (ESCoT) subtests: Mean (SD).

	Max score	ASD n = 19	ASD range	NC n = 38	NC range
Cognitive ToM	30	18.63 (3.68)	12–27	22.61 (1.97)	19–26
Affective ToM	30	22.16 (4.30)	14–28	26.08 (2.40)	21–30
Interpersonal norms	30	19.26 (3.94)	10–25	23.45 (2.81)	18–28
Intrapersonal norms	30	24.89 (3.02)	19–30	26.74 (2.09)	21–30

ToM; Theory of Mind. All scores were age corrected.

All participants performed poorer on inter- compared to intrapersonal understanding of social norms (Z = −6.31, p < 0.001, ηp² = 0.70, d = 3.05). Furthermore a significant difference was found between groups, with the ASD group performing poorer than NC, on the interpersonal understanding of social norms (U = 140.50, p < 0.001, ηp² = 0.25, d = 1.15) and intrapersonal understanding of social norms (U = 226.50 p < 0.05, ηp² = 0.09, d = 0.64).

Table 7 shows the group comparisons on the established social cognition tests and ESCoT total scores. Overall, performance on the ESCoT was significantly poorer for the ASD group than the NC group. Moreover, scores were significantly lower for the ASD group on the RME and RMF compared to NC. ASD adults performed poorer than NC on the SNQ total scores. There were no statistically significant differences between ASD adults and NC for the SNQ subtests (χ²(1) = 0 .49, p > 0.05).

Table 7. Results by group for the tests of social cognition: Mean (SD).

	Max score	ASD n = 19	ASD range	NC n = 38	NC range	Statistic (df)	p-value	ηp² (d)
ESCoT total score^a	120	84.95 (10.11)	64–96	98.87 (4.92)	90–107	U = 64.50	p < 0.001	0.44 (1.79)
RME	36	23.26 (3.84)	17–29	27.47 (4.03)	17–34	t(55) = −3.77	p < 0.001	0.21 (1.02)
RMF	22	10.26 (2.54)	6–15	14.03 (2.17)^b	8–18	U = 94.00	p < 0.001	0.36 (1.50)
SNQ total score	22	17.37 (2.22)	12–21	18.95 (1.84)	14–22	U = 208.50	p < 0.01	0.12 (0.74)

ESCoT = Edinburgh Social Cognition Test, RME = Reading the Mind in the Eyes, RMF = Reading the Mind in Films, SNQ = Social Norms Questionnaire. Analyses were conducted using parametric and non-parametric tests where appropriate.

^a Age-corrected.

^b NC n = 37.

Post-hoc power analysis

While we reported large effect sizes (smallest Cohen’s d = 0.64), we were nevertheless interested in confirming that our analyses were not underpowered due to our sample size. Post hoc power analyses found power for the group comparisons to be excellent for all group comparisons (all power analyses between 0.94 and 1.00, alpha level = 0.05) with the exception of intrapersonal understanding of social norms (power = 0.59) and the SNQ (power = 0.71).

Relationship between social cognition tests, ASD screening questionnaires and IQ

The variables that correlated with ESCoT total scores at p < 0.20 were age (r_s (57) = −0.19, p < 0.20), years of full-time education (r_s (57) = 0.32, p < 0.20) and VCI scores (r_s (55) = 0.39, p < 0.20). These were included in the regression analysis using the enter method in the first stage. None of the predictor variables correlated significantly with one another at p < 0.01 after years of full-time education was accounted for; therefore, the effect of suppressor variables was not examined. In the stepwise regression analysis for ESCoT total score, years of full-time education (β = 0.84, p < 0.05) and AQ scores (β = −0.62, p < 0.01) were retained in the model and accounted for a significant proportion of variance in ESCoT total scores (R² = 0.61, F(4, 50) = 19.86, p < 0.001).

Only VCI significantly correlated with RME scores at p < 0.20 (r_s (55) = 0.30, p < 0.20). This was included in the regression analysis using the enter method in the first stage. In the final model, VCI scores (β = 0.09, p < 0.05) and EQ scores (β = 0.14, p < 0.01) accounted for a significant proportion of variance in RME performance (R² = 0.19, F(2, 52) = 6.20, p < 0.01).

For the RMF, participants’ sex (1 = male, 2 = female, r_s (56) = 0.25, p < 0.20) and VCI scores (r_s (54) = 0.45, p < 0.01) were included in the first stage of the analysis. In the final model, VCI scores (β = 0.10, p < 0.001) and AQ scores (β = −0.11, p < 0.01) were retained and accounted for a significant proportion of variance in RMF scores (R² = 0.42, F(3, 50) = 12.15, p < 0.001).

Participants’ VCI scores were included in the first stage of the regression analysis for SNQ scores (r_s (55) = 0.34, p < 0.20). In the final model, VCI scores (β = 0.06, p < 0.01), EQ scores (β = 0.06, p < 0.01) and SQ scores (β = 0.03, p < 0.05) accounted for a significant proportion of variance in SNQ scores (R² = 0.32, F(3, 51) = 7.87, p < 0.001).

AUC and ROC analysis for the social cognition tests

The AUC values and 95% confidence intervals for the tests were: 0.91 (0.83–0.98) for the ESCoT total score, 0.77 (0.65–0.89) for the RME, 0.87 (0.77–0.97) for the RMF and 0.71 (0.57–0.86) for the SNQ total score. These results demonstrate that of the social cognition tests, the ESCoT is the most effective at distinguishing between the two groups.

Figure 2 shows a plot of the diagnostic value of the ESCoT compared to established tests of social cognition (RME, RMF and SNQ). Overall, the ESCoT showed the highest accuracy of the four tests.

Figure 2. ROC curves for the ESCoT, RME, RMF and SNQ. The closer the curve comes to the reference line, the less accurate the test.

Number of ASD and NC adults impaired on the ESCoT

Based on our cut-off scores, 36.84% of ASD adults were impaired on the cognitive ToM subtest compared to 0% of NC adults, and 26.31% were impaired on affective ToM compared to 0% of NC. On interpersonal understanding of social norms, 36.84% of the ASD group were impaired compared to 7.89% of the NC group. A total of 15.79% of adults with ASD were impaired on intrapersonal understanding of social norms compared to 5.26% of the NC adults. Finally, 42.11% of ASD adults were impaired on overall ESCoT scores compared to 0% of the NC adults.

Discussion

The ESCoT is a new test of social cognition that assesses cognitive ToM, affective ToM and inter- and intrapersonal understanding of social norms within the same test. We investigated the convergent validity of the ESCoT against traditional tests and compared performance of ASD adults to NC adults on these tests. Moreover, we examined the influence of intelligence and ASD diagnosis on the ESCoT and traditional tests of social cognition. The ESCoT showed good convergent validity with traditional social cognition tests. ASD adults performed poorer on all subtests of the ESCoT and traditional tests compared with NC. Regression results showed that better overall performance on the ESCoT was predicted by more years of education and lower AQ scores. ESCoT total score was not predicted by VCI or PRI; this contrasts with performance on the traditional social cognition tests included in this study. Higher VCI scores predicted better performance on the RME, higher VCI and EQ scores predicted better performance on the RMF while higher VCI, EQ and SQ scores predicted better performance on the SNQ. The ESCoT showed very good discriminative value, followed by the RMF, the RME and the SNQ. Overall, the ESCoT showed the highest accuracy compared to the RME and SNQ. A total of 42.11% of ASD adults were impaired on the ESCoT total score compared to 0% NC.

Significant associations between the ESCoT and traditional social cognition tests provide evidence of convergent validity for the ESCoT as a test of social cognition (see supplementary materials for scatterplots and correlations divided by group). The cognitive ToM subtest of the ESCoT positively correlated with the RME. There is currently debate relating to what the RME assesses and some authors have argued that the RME is a test of emotion recognition (Oakley et al., 2016). While it could be argued that the RME is an affective ToM measure (Duval et al., 2011), our findings suggest that it relates to cognitive components of ToM. Indeed, the RME asks participants to infer what the person is thinking or feeling. Intrapersonal understanding of social norms did not correlate with cognitive and affective ToM of the ESCoT but was positively correlated with the RME. Furthermore, the RME correlated with our measure of empathy. It may be that the RME is related to several aspects of social cognition. The positive correlations between ESCoT subtests (particularly cognitive and affective ToM) suggests that performance on one ability is associated with performance on other abilities, but still show differentiation in performance in that they are predicted by different but overlapping constructs. Performance on cognitive ToM was predicted by age, but performance on affective ToM was predicted by age and sex (Baksh et al., 2018). These findings show that while cognitive and affective ToM are distinct, they do overlap (Kalbe et al., 2010; Sebastian et al., 2011; Shamay-Tsoory et al., 2010), and both should be considered when assessing ToM.

Poorer performance of ASD adults compared to NC on cognitive ToM, affective ToM and interpersonal understanding of social norms supports previous findings (Baron-Cohen, Wheelwright, Hill, et al., 2001; Castelli et al., 2002; Golan et al., 2006; Murray et al., 2017; Thiébaut et al., 2016; Zalla et al., 2009). Although the ESCoT showed good sensitivity on all subtests, the finding that ASD adults showed impaired intrapersonal understanding of social norm is somewhat in contrast to our prediction and previous findings of intact performance (Baez et al., 2012; Gleichgerrcht et al., 2013). Our findings show that ASD adults have difficulties with how they should behave in social interactions. However, it is possible that the ASD group had difficulties in processing the wider context of the interaction, due to difficulties in weak central coherence (the inability to understand context) (Frith, 1989, 2003). In this study, individuals with ASD generally knew when a social norm had been violated and responded appropriately to the yes/no aspect of the question. Yet, in general, participants with ASD gave egocentric responses (e.g., I’m a nice person) regarding why they would/would not have behaved as the character in the animation, instead of referencing the wider context of the interaction (e.g., they needed help, helping others is the right thing to do) to explain why they would have behaved in a particular way.

A major advantage of the ESCoT over existing tests is the magnitude of the reported effects. Our effects sizes for the group differences on cognitive and affective ToM is greater than those on the RME and RMF. Increased ecological validity may explain the greater effect sizes found for the ESCoT compared to the RME, which lacks important contextual information, and the RMF, which uses pre-existing stimuli that are overdramatized (Murray et al., 2017). With the ESCoT, we have shown that contextually driven tests more clearly demonstrate group differences between ASD and NC. Performance on the ESCoT may be more representative of the everyday difficulties faced by ASD adults compared to NC. There are several other advantages of the ESCoT over existing tests in the social cognition literature. Firstly, unlike tests such as the TASIT, the ESCoT is a short and detailed test of social cognition with self-contained interactions. The ESCoT also provides researchers and clinicians with two subtests of ToM and social norm understanding.

All participants performed better on affective ToM, compared to cognitive ToM, which is similar to previous findings (Bottiroli et al., 2016; Shamay-Tsoory et al., 2007). Participants were better on intrapersonal understanding of social norms compared to interpersonal understanding. While other studies may find differential performance on social cognitive abilities using different tests, these tests are not matched for difficulty. Matching social cognition components for equivalent difficulty is challenging to achieve. Future studies in individuals with ASD could examine ways of controlling for level of difficulty between subtests. Similar to previous findings (Murray et al., 2017), we found that poorer performance on the ESCoT in the ASD group could not be explained by general cognitive abilities while performance on the traditional tests was predicted by VCI and PRI, similar to earlier work (Baker et al., 2014; Golan et al., 2006; Kaland et al., 2002; McDonald et al., 2003). This is an advantage for the ESCoT as it can be used in clinical populations who may show some impairment but still have intact verbal abilities. Moreover, our NC group did not perform at ceiling on the test. The variability of performance in the control group is maybe useful for detecting individual differences in nonclinical populations. Further usefulness of the ESCoT is its superior diagnostic value, compared to other social cognition tests, with an AUC value of 0.91.

A limitation of this study is the small sample size for the ASD group. However, the large effects sizes (e.g., Cohens d = 1.79) and post hoc power analyses for the ESCoT indicate that, even with a small sample size of ASD adults, we were able to detect meaningful group differences. Further is the lack of formal diagnostic information relating to symptom level data regarding our ASD participants. Moreover, there were too few females to examine whether there were sex differences on the ESCoT. ASD is a heterogeneous condition (Ghaziuddin & Mountain-Kimchi, 2004), and given our wide age range and the changes in diagnostic criteria for ASD over the past 60 years, heterogeneity is more likely in our sample. It is also unknown whether our participants’ diagnoses were made close to the time of study inclusion. Recent research suggests that ASD symptoms improve in adulthood (Woodman et al., 2015), although other studies suggest that core symptoms of ASD tend to persist and there are few changes in adults over the lifetime (Beadle-Brown et al., 2006; Matson & Horovitz, 2010). Future work should examine the relationship between ASD diagnosis, age of diagnosis, ASD symptoms and ESCoT performance. In our findings, we found that a total of 42.11% of ASD adults were impaired on the ESCoT total score (but still substantially higher than the 0% of NC adults). Overall, ASD individuals performed poorer than NC, but individually, there was heterogeneity with the ASD group on the ESCoT. Further work is needed to identify how scores on the ESCoT vary with ASD symptomology. While the groups were not matched on verbal comprehension scores, in our NC group, verbal comprehension was not a predictor of ESCoT performance. Moreover, ASD participants were within the normal IQ range (full-scale IQ > 70) therefore, future studies would benefit from including lower functioning individuals to examine if the ESCoT may be useful in this sample of individuals. Finally, the smaller group difference in intrapersonal understanding of social norms compared to the other ESCoT subtests and the limited correlations with other tests perhaps indicates that this subtest may not be performing as well as cognitive ToM, affective ToM and interpersonal understanding of social norms. However, few studies have examined intrapersonal understanding of social norms in social cognition research. Therefore, future research should validate this subtest against behaviour to understand this social cognitive ability and how it relates to other domains.

Conclusion

We found that adults with ASD perform poorer than NC on cognitive ToM, affective ToM and inter- and intrapersonal understanding of social norms. These impairments may be responsible for the difficulties frequently observed in social interactions. The convergent validity between the ESCoT and established tests of social cognition show that the ESCoT is a sensitive test of social cognition in ASD adults. We showed that the ESCoT can detect large effects with a restricted sample size and shows better diagnostic accuracy than established tests. Many of the current tests of social cognition have limited use in clinical settings (Dodich et al., 2015) but we have demonstrated that the ESCoT may be a useful test to assess patients and aid in the detection of potential difficulties in ToM and social norm understanding.

Acknowledgements

The authors thank Molly Nelson and Helen Stocks for conducting the inter-rater reliability.

Disclosure statement

The authors have no conflict of interest to report.

Data availability statement

The data that support the findings of this study are available on request from the corresponding author, RAB, upon reasonable request.

Additional information

Funding

The work was supported by a PhD studentship from The University of Edinburgh Centre for Cognitive Ageing and Cognitive Epidemiology, part of the cross council Lifelong Health and Wellbeing Initiative (MR/L501530/1). Funding from the Biotechnology and Biological Sciences Research Council (BBSRC) and Medical Research Council (MRC) is gratefully acknowledged.