Gaze and Motor Traces of Language Processing: Evidence from Autism Spectrum Disorders in Comparison to Typical Controls

ABSTRACT

We investigated what strategies underlie figurative language processing in two groups of participants distinguished by the presence of a developmental deficit, highly-verbal participants with autism, and control participants without autism in two age ranges each. Individuals with autism spectrum disorder are characterised by impaired social interaction and communication. Even at the high end of the spectrum, where structural language is adequate, difficulties in comprehending non-literal aspects of language are widely attested. The exact causes of these problems are, however, still open to debate. In an interactive sentence-picture matching task participants selected the most suitable image representation of a non-literal figurative expression that matched the target meaning, while their eye-movements and hand movements were being tracked. Our results suggest that individuals with ASD have different processing patterns than typically developing peers when interpreting figurative language, even when they provide the correct answers. Both children with and without autism, and participants with autism display greater uncertainty and competition between alternatives when providing the answer, often reflected in also considering the literal interpretation of the expression against its target figurative meaning. We provide evidence that expression transparency and decomposability play a central role in figurative language processing across all groups.

KEYWORDS:

• Language processing
• figurative language
• eye tracking
• mouse tracking
• autism

Introduction

Language comprehension and processing problems have been routinely found in individuals on the autism spectrum (Arunachalam & Luyster, 2016; Eigsti, de Marchena, Schuh, & Kelley, 2011; Kjelgaard & Tager-Flusberg, 2001; Leyfer, Tager-Flusberg, Dowd, Tomblin, & Folstein, 2008; Loukusa & Moilanen, 2009; Saalasti et al., 2008; Tager-Flusberg, 2006; Vulchanova, Saldaña, Chahboun, & Vulchanov, 2015). Autism is a neurodevelopmental deficit of complex aetiology characterized by social and communicative impairment accompanied by stereotyped and repetitive patterns of behaviour (DSM-V, 2013). While many individuals on the spectrum may, in addition, present with language delay and language impairment, individuals on the high end of the spectrum are characterized by adequate structural language skills—defined as phonological, core grammar, and vocabulary skills. Yet problems with the processing and understanding of figurative language, such as idioms, metaphors, jokes, and indirect requests, have been systematically observed in individuals on the spectrum, even when structural language is preserved (Landa, 2000; Tager-Flusberg, 2007; Volden, Coolican, Garon, White, & Bryson, 2009; Volden & Phillips, 2010; Vulchanova, Talcott, Vulchanov, & Stankova, 2012a; Vulchanova, Talcott, Vulchanov, Stankova, & Eshuis, 2012b; Vulchanova et al., 2015).

While some uses of language require literal interpretation, based on composing the meanings of constituent words (e.g., as in “red apple”, which is both an apple and something red), figurative language is non-compositional, in that meaning cannot be derived by computing the meanings of individual constituents (e.g., the idiom “pull up your socks” does not literally require the hearer to act on their socks). For this reason, figurative language may seem to require additional operations—such as, for example, inferencing based on context (both linguistic and perceptual), perceiving the intended meaning of the message (speaker’s intentions)—and it may be more demanding, as revealed in comprehensive research on that topic over recent decades (Cacciari & Tabossi, 1988; Gibbs, 1992; Levorato & Cacciari, 2002; Titone & Connine, 1999). Given that, in comparison to literal language, non-literal, figurative language may require additional or parallel steps in its processing, research in this domain may offer the opportunity to identify what these steps are and whether they involve literal, non-figurative interpretations at all, not only in developmental deficits, but in typical populations as well.

Two types of data can inform investigation of on-line language processing behaviour: participants’ gaze data and hand movements when responding to auditory or written language stimuli. Hand movements have been described as high-fidelity, real-time motor traces of the mind (Freeman, Dale, & Farmer, 2011), while participants’ gaze behaviour reflects language-mediated eye movements and can be used as a proxy for language processing, as routinely documented in, for example, the Visual World Paradigm (Allopenna, Magnuson, & Tanenhaus, 1998; Altmann & Kamide, 2007; Huettig, Rommers, & Meyer, 2011). Furthermore, eye-movement paradigms can reveal the on-line integration of different sources of information (e.g., phonological, semantic, and visual information) during spoken language processing (Huettig & Altmann, 2005; Huettig & McQueen, 2007; Huettig, Quinlan, McDonald, & Altmann, 2006; Spivey, 2007), while manual action reveals the real-time dynamics of decision-making and the fine-grained effects of resolving conflict and competition (Barca, Benedetti, & Pezzulo, 2016; Barca & Pezzulo, 2012, 2015; Flumini, Barca, Borghi, & Pezzulo, 2014; Freeman, 2018; Freeman & Ambady, 2010; Freeman, Stolier, Ingbretsen, & Hehman, 2014; Lepora & Pezzulo, 2015; Quétard et al., 2015; Song & Nakayama, 2009). The main purpose of the current study was to investigate the on-line processing of a selection of figurative expressions (idioms and novel metaphors) on the part of highly verbal participants with autism in comparison to neuro-typical controls using eye tracking in combination with hand-movement data. Measures of visual attention and motor movement while making a response on the target figurative meaning of expressions in the stimuli, and in the context of visually provided information, were expected to provide detailed data on the actual process of considering (or not considering) alternative (e.g., literal) responses.

Figurative language processing

Figurative or non-literal language is a pervasive phenomenon in every-day human communication. It covers a wide range of expressions, such as idioms, metaphors, irony and jokes, hyperbole, indirect requests, as well as other stereotyped expressions, such as cliches. A recent study investigating the incidence of non-literal expressions in e-mails written by young people found that 94.30% of the e-mails included at least one non-literal statement, and participants used on average 2.90 non-literal expressions per e-mail (Whalen, Pexman, & Gill, 2009). Figurative language is commonly characterized as non-transparent in various ways: the comprehender has to go beyond the literal meanings of the constituent words in a figurative expression in order to recover the speaker’s intended meaning. Indeed, there is often a gap, sometimes even a conflict, between the compositional meaning of a given expression (the constructed/literal meaning) and its intended interpretation in context (the intended meaning). In literal utterances, constructed meaning and intended meaning typically coincide. In contrast, for figurative language the intended meaning, depending on the type of expression (metaphor, idiom, hyperbole), can diverge to different degrees from the constructed, literal one. For instance, in metaphor, which is by far one of the most prevalent forms of figurative expression, the hearer needs to establish a meaningful link between two otherwise unrelated concepts. Thus, the expression “music is medicine” can only be parsed through extracting and mapping the relevant features of the metaphor vehicle, medicine (e.g., healing) to features of the topic, music.

While in some cases, the gap between the constructed, literal meaning and the intended, speaker’s meaning can be bridged by interpretation functions or pragmatic inferencing (e.g., in metaphors by using analogy between the two conceptual fields), there are non-literal expressions whose meaning cannot be recovered via the application of interpretive functions to compositional or literal meaning. For example, the figurative meanings of idioms, such as “hit the sack” (go to sleep) or “kick the bucket” (die) cannot be recovered based solely on the meanings of the verbs and their complements. The comprehender must, instead, rely on knowledge or familiarity with the expression, as well as on the context in which the expression is embedded, to construct the appropriate figurative interpretation. A notable property of these expressions is that they have a literal and a figurative meaning, which often seem to be unrelated to one another: “kick the bucket” has both the figurative meaning “to die” and the compositional meaning “to strike a bucket with one’s foot”—these meanings are semantically unrelated. In some cases the relation between figurative and compositional, literal meanings may be more or less direct or transparent, as, for example, in “pop the question” (propose marriage). This raises a series of questions concerning the specific role of compositional meaning in the derivation of figurative interpretations, in particular for expressions such as idioms, where the relationship between the compositional and the figurative meanings may be indirect, opaque, or absent.

Most accounts of figurative language identify this tension between compositional, literal, and figurative meaning as essential to understanding the exact cognitive and neural mechanisms underlying figurative language comprehension. In the past few decades, two types of accounts have dominated the theoretical landscape: (a) indirect access theories (known as the “standard model”), which suggest that the process of interpretation starts with the literal meaning, which needs to be rejected in order to recover the target, figurative interpretation, largely inferentially and only at a second step (Eco, 1986; Levinson, 1983); (b) direct access theories, which assume that the context can, at least in some cases, support the direct construction of figurative meaning, thus by-passing the literal interpretation (Gibbs, 1990; but see Noveck, Bianco, & Castry, 2001 for a re-assessment of evidence). Importantly, these two types of accounts need not be completely opposed, since the construction of figurative interpretations may or may not be mediated by access or computation of compositional meaning. This mediation may depend on a number of factors, such as the decomposability of the figurative expression, the properties of the lexical items in the expression (frequency, collocation with other words), familiarity of the expression, and the presence—or not—of (supportive) context. It is these factors that ought to be in focus, both theoretically and empirically.

Several accounts of idiom comprehension and representation posit that idioms are stored as large “chunks”, akin to single words, and are retrieved as whole units during processing (e.g., Swinney & Cutler’s 1979 lexical representation hypothesis). Interestingly, the processing of idioms that would appear non-decomposable might still be affected by the semantic properties of their component words, such as aspectual features of the head verb. Thus, “kick” in “kick the bucket” is incompatible with the progressive form or adverbials suggesting a protracted event due to its punctual feature (Glucksberg, 1991; Hamblin & Gibbs, 1999). Thus, the distinction between decomposable and non-decomposable idioms might not be as rigid as historically characterized.

To address the question of whether idioms are stored or processed on-line, two main types of processing models have been proposed: non-compositional models, in which idioms are stored and accessed as multi-word “chunks”, and compositional models, which focus on the possibility that individual expression constituents affect the interpretation or usage of the idiom on-line. Both types of approaches aim to address a central question in idiom research: whether retrieval of the constituents’ literal meanings is obligatory, or whether comprehenders are able to bypass this step and directly retrieve the idiomatic meaning, in particular when a supportive context is available. Evidence compatible with both of these perspectives on idiom processing—the compositional (Cacciari & Glucksberg, 1991; Cacciari & Tabossi, 1988; Hamblin & Gibbs, 1999) and non-compositional (Gibbs, 1990, 1994; Swinney & Cutler, 1979)—has been provided in research. Notably, these approaches do not always consider idiom decomposability in the processing model. For example, under either model, “to pop the question” and “to kick the bucket” are processed in largely similar ways, despite the fact that the former is decomposable and the latter is not. However, several experiments have shown that decomposable idioms are processed more easily than non-decomposable idioms via earlier activations for the figurative meanings of decomposable idioms (Caillies & Butcher, 2007; Caillies & Declercq, 2011), shorter reading times for decomposable idioms (Gibbs, Nayak, & & Cutting, 1989), or facilitated processing for the literal and figurative meanings of decomposable idioms (Titone & Connine, 1999). Decomposability may thus confer advantages at many levels of processing. A potential way to account for all of the above stated findings might emerge from Titone and Connine’s (1999) hybrid model (HM) of idiom processing, where the idiom’s decomposability plays a crucial role. Under the HM, idiom comprehension follows two simultaneous, parallel routes: (a) direct access of the idiomatic meaning, and (b) compositional analysis of the idiom into individual words. Decomposability can be described as the extent to which individual words in the idiom can affect its figurative interpretation. It can also be conceived as degree of semantic relatedness between the literal and the figurative meanings: the closer the literal and the figurative interpretations are to each other, the more decomposable the idiom appears to be (Milburn, 2018). Given the range of possible models of idiom processing that have been proposed to account for how language users construct their meaning, it is important to design tasks where factors such as decomposability (transparency) and familiarity are carefully controlled and where figurative expression processing can be captured by measures reflecting the specific stages in constructing the non-literal/figurative interpretation and, specifically, whether and under what conditions accessing the literal meaning is a first stage in idiom processing (see Vulchanova, Milburn, Vulchanov, & Baggio, 2019, for a critical discussion, and Gibbs & Colston, 2012, for a detailed historical overview of figurative language).

Figurative language processing in autism

A series of recent studies, which included highly verbal individuals with autism, carefully matched to controls on both age and structural language and non-verbal intelligence measures, shows evidence of significant differences between the participants with autism and controls in both reaction latencies and response accuracy on tasks involving figurative (non-literal) expressions (Chahboun, Vulchanov, Saldaña, Eshuis, & Vulchanova, 2016, 2017; Vulchanova & Vulchanov, 2018). These results are indicative of problems in the on-line processing and off-line comprehension of a variety of figurative expressions, ranging from highly decomposable/transparent to non-transparent expressions. However, with some notable exceptions (Gold & Faust, 2010, 2012; Saban-Bezalel et al., 2017), little is known about what participants engage in during processing these expressions and what is causing the significant delay in comparison to controls. Studies reflecting the on-line processing of such expressions in that population are missing, and the exact source of these problems is still subject to speculation.

The current study aimed at testing the extent to which participants with autism behave similarly to individuals without an autism diagnosis when responding on-line to a figurative language task. Given that individuals with autism are often reported to interpret language literally, our main goal in this combined eye-tracking and mouse-tracking study was to establish whether potential problems in converging on the target interpretation of the figurative expression and problems in inhibiting the non-target literal interpretation can provide a possible account of the well-attested difficulties in figurative language comprehension in that population. Furthermore, by including control participants, we aimed to establish whether both groups relied on similar steps and strategies when encountering non-literal language.

Behavioural and imaging studies have suggested that the difficulty in processing figurative expressions is proportional to their complexity and have uncovered several factors contributing to this complexity. These include the type of expression, the degree of decomposability/transparency, linguistic structure, source domain of knowledge, and novelty/conventionality. These factors impact on the way speakers comprehend figurative expressions, and different types of expressions may recruit different strategies and resources. These strategies may not be evident from simply studying response accuracy. For instance, in a reading and inference study, Micai, Joseph, Vulchanova, and Saldaña (2016) document that while the participants with autism performed similarly to their typically developing matched peers, they displayed different gaze behaviours (longer fixation duration on target words). Furthermore, recent research has established a discrepancy between performance and neuro-physiological measures in participants with developmental deficits and in ageing individuals. Those studies reveal that even when participants with autism or dyslexia perform similarly to neuro-typical controls, they tend to recruit compensatory mechanisms and strategies to cope with task complexity. Thus, Eden et al. (2004) document increased brain activation in left-hemispheric regions in adult dyslexic readers following intervention, accompanied by compensatory activity in the right perisylvian cortex. A study of language comprehension and brain function in individuals with an optimal outcome from autism has found compensatory neural activation in numerous left- and right-lateralized regions in the optimal outcome group against no significant group differences on accuracy in the language comprehension task (Eigsti, Stevens, Schultz, & Fein, 2016).

Eye-tracking studies of language processing in autism

Eye tracking has been used primarily to study patterns of gaze behaviour in the context of the social and attention deficits in autism (cf. Boraston & Blakemore, 2007 for a review). This research has shown that participants with autism attend to the world in a qualitatively different way in that they do not attend to the social and linguistic cues evident on the speaker’s face (Norbury, 2017). A handful of studies have looked at language processing through the lens of gaze in that population (Norbury, 2017). In an eye-tracking study, Brock, Norbury, Einav, and Nation (2008) investigated whether individuals with autism process words in context. The study employed the visual world (VW) paradigm where images corresponding to target words appeared together with images of phonological competitors and distractors on the screen. The results indicate that adolescents with autism used lexical knowledge of the main verb of the sentence to predict the object of that verb and oriented to its image on the screen, and both participants with autism and controls displayed similar effects of context on eye movements. This study also found that the effect of sentence context on participants’ predictive gaze behaviour was modulated by the participant’s language skills. This study is the first one to address sentence processing in autism and the impact of verb lexical knowledge and the context the main verb provides for processing predictions. Unfortunately, in its design, the study used semantically restricted verbs, whose object arguments were highly predictable given the visual display context (e.g., “fasten”, “stroke”, “feed”), thus allowing for limited competition for the object position. Zhou, Zhan, and Ma (2018) used a similar Visual World paradigm design with a much younger group of participants with autism: 5-year-old Mandarin-speaking pre-school children with autism were matched on age with a control group, and on language skills with a control group of 4-year-old children. This study did not find significant differences between participants with autism and their age-matched controls in the time window following verb onset, except for a significantly larger difference between the verb-biased condition and the verb-neutral condition observed in the typical controls, but not in the children with autism. Furthermore, significant differences in proportion of looks were found between the two 5-year-old groups, but not in the time window following object noun onset between the children with autism and the verbal-skills-matched 4-year-olds. In the group of children with autism, the small difference observed in predictive behaviour between the condition where the verb meaning biased towards a specific object and the neutral condition may suggest decreased semantic sensitivity to verbs in children with autism, while the difference in proportion of looks in the noun window between the children with autism and age-matched controls against the absence of such a difference between the children with autism and the 4-year old controls may be indicative of a developmental delay in the acquisition of verb lexical semantics in autism.

Chita-Tegmark, Arunachalam, Nelson, and Tager-Flusberg (2015) studied word comprehension in children at high risk for autism spectrum disorder (ASD). In a cross-sectional design, including 18-, 24- and 36-month-old children at high and low risk, they established significant differences in word accuracy measured as looks to the target picture on the screen only in the 36-month-old group, with significantly lower accuracy for the high-risk children in comparison to controls, but no differences for the younger groups. The difference in the 36-month-old children was observed both for words expected to be acquired early and for words expected to be acquired late. Interestingly, despite the difference on accuracy, no similar difference between the older and other two groups was observed in speed of orienting to the target image.

A study of relevance to the current experimental design addressed communicative behaviour in participants with autism. The main goal of that study was to investigate the simultaneous processing of iconic gesture and auditory language (Silverman, Bennetto, Campana, & Tanenhaus, 2010). Participants watched videos of a person describing a shape, in one condition only using speech, and in another condition where gesture–speech combinations were used. They had to select a shape out of four present on screen in these two conditions. That study found that iconic gestures facilitated comprehension only in typical participants, as a results of fast integration of auditory and visual information, but not in participants with autism, where gestures hindered the processing in that group. In contrast to controls, participants with autism identified the target more slowly when gesture was combined with speech compared to when speech was used alone. This finding suggests problems even in high-functioning individuals with autism with the processing of information in multiple modalities and multi-modal information integration, which are evident when both visual (gesture) and auditory (language) information have to be processed.

Eraslan, Yaneva, Yesilada, and Harper (2019) is, to the best of our knowledge, the only study to employ eye tracking collecting a number of measures of the type we report in the current one. The main aim of that study was to investigate how individuals with high-functioning autism process text and information displayed on web-pages and what strategies they use when searching for information, in comparison to controls. Participants were shown screen shots of 6 web-pages and asked to respond to two questions by finding information on the screen without being able to scroll/use a mouse or keypad. Participants’ gaze behaviour was tracked while answering the questions. This study did not document significant differences between the two groups on success in finding the correct information as reflected in accuracy on responses. However, in the participants with autism significant differences were found on all on-line measures, including greater distraction by irrelevant elements, gaze scan path length, significantly more transitions between elements on the web-page, and significantly shorter mean fixations. These effects were modulated by web-page type and origin. Even though this study addressed overall information search strategies, not language processing per se, it provides evidence of a quantitative difference in strategy use in the two participant groups.

The current study

The current study is, to the best of our knowledge, the first to investigate figurative language processing on-line by simultaneous use of eye tracking and mouse tracking in autism. It is also the first to use these two methodologies for the purposes of studying figurative language comprehension in children (Erb, 2018) and contributes to the still largely under-researched domain of studying language processing in autism as revealed through patterns of gaze behaviour. The study is based on combined eye- and hand-motion-pattern data from on-line decisions on figurative language in visual context with two groups of participants—highly verbal individuals with autism and matched controls—in two age bands: children (10–12 years of age) and young adults (16–22 years). We used a picture–sentence matching task and employed both eye-tracking and mouse-tracking methodology, which are well suited to testing the current hypotheses. For example, in tasks where participants have to select a target item from two different choices, curvature in mouse trajectories can be interpreted as the effect of uncertainty and the competition between partially activated representations (Barca & Pezzulo, 2012; Bruhn, Huette, & Spivey, 2014; McKinstry, Dale, & Spivey, 2008; Spivey, Grosjean, & Knoblich, 2005; Spivey, Dale, Knoblich, & Grosjean, 2010). In addition to eye- and hand-movement data, we also collected reaction times and response accuracy data reported in Chahboun et al. (2016). The analyses of accuracy and reaction times in that study revealed that the participants with autism performed at a lower level than their typically developing peers. Moreover, the modality in which the stimuli were presented was an important variable in task performance for the more transparent expressions. The individuals with autism displayed higher error rates and greater reaction latencies in the auditory modality than in the visual stimulus presentation modality, implying more difficulty. However, reaction latencies and performance accuracy do not provide information on what the processing of the stimuli involves.

Recording and analysing both mouse movements and eye movements allows multiple levels of observation on the time course of how participants’ choices develop and potentially the stages in constructing the target interpretation of the expression and the extent to which non-target, literal interpretations are considered in the process. Relative to the hand, the eye moves quickly and gives a direct insight into how participants guide overt visual attention to the stimuli. For instance, gaze fixations are used to gather information pertinent to the figurative language task and can illustrate how a participant may be comparing different stimulus options. However, eye fixations on a target can also be indicative of visual salience or interest and do not, by themselves, fully reveal task-relevant underlying processes. Hand movements are slower and require more energy from the participant than do eye movements. When the participant starts moving the mouse to the final target, this can indicate the point when a final choice has been made. Participants may also move the mouse as part of their decision process, pointing at the stimulus they are currently evaluating and moving between options.

Hypotheses

We expected to find a relatively high correlation between hand movements and eye movements in both groups of participants and both age ranges. More specifically, we expected to find convergence in gaze and hand visits to images of interest (target vs. literal image). However, we also expected that the coupling between gaze and manual motion would differ at different time points in the processing, and that gaze data and hand-motion data would index different time-points in the processing of the stimuli. It is estimated that it takes approximately 200 ms to programme and launch an eye movement (Hallett, 1986; Silverman et al., 2010), while initiating hand movements takes longer. Effects of ambiguity persist longer in mouse movements, whereby competition resolution becomes evident in eye movements around 200–300 ms after word offset, while it may continue until 500 ms in hand-movement data (Magnuson, 2005). It has also been suggested that eye tracking is more sensitive to pre-attentive processes before initiation of hand movement (Freeman, 2018; Quétard et al., 2015). For this reason, we expected earlier stages in the process of responding to the task to be more closely reflected in the gaze patterns (fixation duration on images of interest), while later stages and total on-line behaviour may be more evident in the hand movement measures (distance travelled with the mouse/path length, peak velocity).

Given that figurative language skills take time to develop (Nippold, 2006; Vulchanova, Vulchanov, & Stankova, 2011), we further expected to find systematic differences between the children and the adult groups on all measures. Consistent with evidence of atypical recruitment of cognitive resources and compensatory strategies (Eden et al., 2004; Eigsti et al., 2016; Micai et al., 2016), our hypothesis was that individuals with ASD would have different processing patterns from those of controls in both eye and mouse behaviour. Based on the reaction time and accuracy analyses results reported in Chahboun et al. (2016), we expected the gaze data of participants with ASD to display a higher proportion of time (longer gaze duration) spent on the target image. Given the few study-based (Hobson, 2012; Mitchell, Saltmarsh, & Russell, 1997), but overall, mostly anecdotal, reports of overly literal interpretation in autism, we further expected that participants with autism would spend more time inspecting the images corresponding to literal interpretations, as revealed in both gaze (longer duration and more gaze visits) and hand behaviour (longer distance travelled with the mouse), even when providing correct target responses. In addition, based on theoretical accounts of the observed difficulties in understanding figurative language in autism in terms of inability to suppress irrelevant non-target information (Ozonoff, Pennington, & Rogers, 1991; Murphy, Foxe, Peters, & Molholm, 2014; Vulchanova et al., 2015), we expected that participants with ASD would exhibit greater competition between alternatives and would hesitate more between target non-literal and non-target literal meaning before making a decision on the response (measured by mouse visits to target image, mouse distance travelled, and mouse peak velocity). Such data would directly address the question of what strategies and steps participants are taking when processing figurative expressions and potentially inform about the source of the well-attested difficulty in processing figurative language in autism (Chahboun et al., 2016, 2017; Vulchanova et al., 2015). Finally, we expected that degree of decomposability of the expression would impact differentially on their processing in both groups of participants, with more decomposable expressions (e.g., novel metaphors and biological idioms) eliciting fewer and shorter gaze fixations and mouse visits.

Method

Participants

All participants were native speakers of Spanish. In total, 45 participants with ASD were recruited, as well as a control group of 39. Two age groups were considered in this study. The first group, and their corresponding controls, included children between the ages of 10 and 12 years. The choice of this age range was based on research findings regarding typical development, suggesting that children’s skills in figurative language comprehension stabilize at around this age (Kempler, Van Lancker, Marchman, & Bates, 1999; Vulchanova, Saldaña et al., 2015; Vulchanova, Vulchanov et al., 2011). The second group consisted of young adults between the ages of 26 and 22 (see Table 1). Again, the choice of this second age range was because figurative language knowledge continues to develop and consolidates in late adolescence and early adulthood (Nippold, 1998, 2006). In accordance with Helsinki Declaration principles, written consent was acquired from participants and from their legal guardians (usually the parents) before research began. The study was approved by the regional Biomedical Research Ethics Committee. The autism diagnosis was confirmed by licensed professionals according to the Autism Diagnostic Observation Schedule (ADOS) (Lord et al., 2000) (children, $\overline{x}$  = 10,8; young adults, $\overline{x}$ = 12,3).

Table 1. Mean and standard deviation of background measures for each age and group.

	Children				Adults
	ASD		Control		ASD		Control
	N	M (SD)	N	M (SD)	N	M (SD)	N	M (SD)
N	25		19		20		20
Age		11.3 (0.96)		11.9 (0.89)		18.1 (1.65)		18.3 (1.92)
Gender	21		17		17		14
FSIQ		110.71 (14.58)		105.76 (11.56)		108.3 (13.39)		118.34 (8.46)
VCI		118.7 (17.47)		107.4 (15.03)		122.62 (16.71)		117.63 (12.97)

Note: ASD = autism spectrum disorder; FSIQ = intelligence quotient, Full Scale score; VCI = verbal comprehension index.

In addition to age, the individuals with ASD and their typically developing peers were matched on gender, IQ, and verbal comprehension based on the Wechsler scale (Wechsler, 2005, 2012)—WISC-IV for the younger and WAIS-IV for the older participants (see Table 1). The Wilcoxon Test formed the basis for the matching of the groups, and the smallest p value was p = .222, suggesting that there were no significant intergroup differences (see Table 1).

Attrition from gaze data loss

In this study, a Tobii eye tracker T120 with a frequency of 120 Hz was used. The quality of eye-tracking data varies, and usable data can be difficult to obtain from children, typically or atypically developing alike. Therefore, only those participants who had over 50% valid trials and over 50% of their eye gaze recorded were included. This left us with 43 ASD participants (children, n = 23, young adults, n = 20), and 35 control group participants (children n = 17, young adults n = 18).

Apparatus and stimuli

Two types of figurative language—idioms and novel metaphors—were included in the study. Their degree of familiarity and frequency was determined in a pilot study with 50 adult speakers of Spanish. Participants in this pilot study were presented with 124 idioms and instructive expressions (proverbs) and 20 metaphors and were asked: (1) “Do you know this expression?” (2) “If yes, do you know what it means?” (3) “Do you use this expression yourself?” Participants were told that there were no wrong answers, and that they should answer truthfully. Thereafter, numerical values were assigned to the answers (1 for “yes” and 0 for “no”), and the scores from the three questions were averaged. This allowed the figurative expressions to be ranked, in order to determine inclusion in the present study. Those who received a score of 0.80 or more were selected and classified into three main categories: (a) biological idioms; (b) cultural idioms; and (c) instructive expressions (proverbs), as used in Vulchanova et al. (2011). For the first two categories, to account for the grounding of language in human experience and practice, we followed the typology adopted in Penttila, Nenonen, and Niemi (1998), in turn inspired by Searle’s idea of deep background (the human biological nature) vs. local background (local cultural practices). From these categories, the stimuli were selected by two independent expert linguists with a 91% interrater agreement. Biologically based idioms are derived from bodily experience in the physical environment and are therefore often reasonably decomposable and available to compositional interpretation. Examples of this are; “estar con el agua hasta el cuello” (lit. “to be with water up to the neck”: “to be drowning in work”); or “venir como anillo al dedo” (lit. “to come as ring to the finger”: “a perfect fit”). Culturally based idioms, on the other hand, are more idiosyncratic and often vary from one society to another. This makes them more opaque and largely non-decomposable, as may be exemplified with “estar como una cabra” (lit. “to be like a goat”: “to act in a crazy way/to have a screw loose”).

Our third category were instructive expressions (i.e., proverbs), the meaning of which is mostly open to on-line computation, whereby the expression itself provides a context that may facilitate meaning construal, e.g., through inferencing—e.g., “Las mentiras tienen las patas cortas” (lit. “Lies have short legs”: “If you lie, you will be easily detected”). This final category was included for comparison and on evidence from earlier findings (Chahboun, Vulchanov, Saldaña, Eshuis, & Vulchanova, 2017) as cases where more contextual information is included in the expression itself. For a discussion of proverbs and approaches to their nature see Chahboun et al. (2016). We also included novel metaphors as an additional type of non-literal expression. Novel metaphors are considered fairly transparent expressions and are more accessible to interpretation (Chahboun et al., 2017; Kasirer & Mashal, 2016; Mashal & Kasirer, 2014). For this category, those metaphors that earned a rating below .20 on the familiarity and frequency scale in the pilot study were selected for inclusion (e.g., “Estás flotando en el aire”: You are floating on air). The reason for including metaphors was to expose participants to all degrees of transparency within figurative language, from opaque expressions (cultural idioms) to reasonably transparent (biological) to the very transparent (novel metaphors). The latter category has received various interpretations in research, and there is no consensus on their degree of transparency, mostly from the point of view of processing (Giora, 1997, 2003; Gold & Faust, 2010). However, our current research with individuals with autism in comparison to controls has consistently provided evidence of the relative ease of processing novel metaphors, specifically in comparison to other types of figurative language (Chahboun et al., 2016, 2017; Vulchanova & Vulchanov, 2018).

The study was designed as a sentence–picture matching task. In total, 38 expressions were presented visually as text, and 38 were presented auditorily, making 76 in all. Among these, there were 20 biological idioms, 20 cultural idioms, 16 instructive expressions, and 20 metaphors. The reason for including an auditory and a written modality of presentation was to investigate the effect of modality, since this has been found to affect processing in earlier research (Chahboun et al., 2016, 2017). Both expression type, presentation modality, and position on screen for visual stimuli were randomized and counterbalanced between groups and participants. For each expression, the participant was presented with four possible visual representations, the first corresponding to the figurative target meaning of the expression, the second to the literal meaning, the third image to an alternative, but non-target figurative meaning, with the fourth image included as a distractor. For the biological idiom “consultarlo con la almohada” (lit. “to consult with the pillow”; fig. “to sleep on it”), the target image (top left-hand corner) was accompanied by a literal image (top right-hand corner), and two distractors (bottom panel) (cf. Figure 1, and Appendix 1 in Supplementary Material for examples of each category of stimulus). An appropriate context accompanied each expression, presented either auditorily or textually, as the experimental block demanded. It deserves mention that the current design is not a classical Visual World Paradigm design whereby, typically, a single word from the auditory language stimulus corresponds to a visually presented object and participants’ responses are measured as proportion of fixations on the target image around the onset of the target word. Since idioms are multi-chunk expressions, there is no specific point in the auditory unfolding of the expression when the proportion of fixations can be measured. Moreover, for the same reason, our visual stimuli corresponded to the interpretation of the expression as a whole, because a single word and image cannot adequately reflect the meaning of the idiom as a whole. In addition, we did not follow a typical mouse-tracking forced-choice paradigm where only two alternatives are present. The latter would have carried a serious risk of chance responses not present in the current design. For the gaze data, we measured total fixation duration in the four quadrants of the screen (AOIs) corresponding to each type of image. We address the specific features of the design and their implications in the discussion.

Figure 1. Images for the idiom “consultarlo con la almohada” (literally “to consult it with the pillow”, figuratively “to sleep on it”).

The experiment was built and run in Matlab (R2014b) (Mathworks, Natick, MA, USA). Eye movements were monitored at 120 Hz, and stimuli weredisplayed at 1280 × 1024 and 60 Hz by a Tobii T120 (Tobii AB, Danderyd, Sweden). Participants were positioned roughly 60 cm from the screen where the screen subtended roughly 31° × 25°. They were encouraged to minimize their movements as much as possible so as to allow the eye tracker to be properly calibrated, which was done by the participant looking at 5 on-screen calibration points. Recalibration was done if necessary. Mouse-tracking data were recorded using Mouse Tracker (Freeman & Ambady, 2010), combined with Matlab (Mathworks, Natick, MA, USA).

Procedure

Our task material consisted of visual stimuli in the form of images and sentences, which reflected literal meanings and idiomatic meanings. The modality of presentation varied between audio and text, and the category of expression varied between biological and cultural idioms, instructive expressions and metaphors. The presentation of the stimulus proceeded as follows: First, depending on the experimental block, a fixation point, “+”, appeared in the centre of the screen, remaining there for 500 ms. Thereafter, either the screen went blank while an expression was heard through the loud-speakers, or the expression was presented as text on the computer screen. After this, the fixation point reappeared for another 500 ms and was then followed by four images, one in each quadrant of the screen. These reflected the figurative (target) meaning, the literal meaning, figurative (non-target) meaning, and a distractor image, respectively. The task of the participant was to select with a mouse click the image that they considered the best visual representation of the expression, for which there was no time limit. Participants were instructed to move the mouse to its start position for each new stimulus. The position of the images was counterbalanced between participants, expression modality, and order of phrase presentation, although each participant went through both experimental blocks. In addition to gaze, mouse-tracking data were collected to determine the ease of processing.

Data analyses

We collected off-line data and reaction times reported in Chahboun et al. (2016), where main effects of age and group were found in both accuracy and reaction times, as shown in Table 2.

Table 2. Means and standard deviations of the off-line data.

	Children		Adults
	ASD M (SD)	Control M (SD)	ASD M (SD)	Control M (SD)
Reaction times (ms)	7713.78 (6764.01)	5706.92 (4468.66)	5192.64 (3584.8)	4491.85 (3079.08)
Error	0.23 (0.419)	0.18 (0.385)	0.21 (0.409)	0.09 (0.292)

Note: Reaction times = times participants spent in selecting the target image. Mean error = accuratcy of participants’ responses (wrong answers were rated 0, and correct answers were rated 1).

The overall model reported in Chahboun et al. (2016) revealed differences in accuracy for age (children/young adults), χ²(1, 23) = 5.73, p = .0016, and group (control/HFA), χ²(1, 23) = 11.21, p < .001, with more errors by both child participants with and without autism and by young adult participants with autism. In addition, a main effect of type of expression was observed as well, χ²(1, 21) = 8.49, p = .036. Multiple Comparisons of Means with Tukey contrasts revealed that this effect was due to marginally significant differences between instructive expressions and biological idioms (p = .07) and between instructive expressions and novel metaphors (p = .06), with more errors in the instructive expressions category.

We performed separate (individual) analyses of variance (ANOVA) for gaze and hand movement data.

Separate analyses for eye-gaze and hand movements

Eye-gaze data were obtained in a Matlab format and afterwards transformed into comma separated value files (.csv). The output of the mouse-tracking data was originally in a Jon Freeman mouse-tracker format, which was also transformed into.csv files for analysis. Both data were analysed with SPSS (SPSS Statistics 22; IBM, 2013) using repeated measures analysis of variance (ANOVA), using a 2 × 2×2 × 4 design (ASD vs. TD, Presentation Type: Audio v. Text Presentation × Age Group: Young adults vs. Children × Expression Type: Biological, Cultural, Instructive or Metaphor). Significant ANOVA results were followed up with pairwise multiple comparisons t tests with Bonferroni correction. Mouse tracking and eye gaze were examined in separate analyses. Individual trials were screened for extremely fast or slow reaction times (RT). We used a criterion based on the variance of an individual’s Z distribution of RTs. The Z value of the log transform of RT was calculated for each participant, Z_p, and for each item per age and group, Z_i. Only if the sum of the squares of the Z values was smaller or equal to 8, i.e., $Z_p^2+Z_i^2\le 8$, the trial was included.

Results

Eye gaze

Eye-movement data were analysed by first median smoothing (kernel window of 5) spatial data and then labelling when gaze was in one of the four quadrants on screen. These labelled segments of quadrant dwell time were counted to estimate number of visits and averaged for a measure of summed fixation duration.

Dwell time on the target image—fixation durations

A repeated measures ANOVA showed a main effect of group, F(1, 66) = 4.907, p = .03, with longer average fixation durations on the target images in the ASD group, a main effect of age, F(1, 66) = 9.209, p = .001, with longer fixation durations on the target images in children, and a main effect of expression type, F(3, 64) = 19,11, p < .001, with shorter fixation durations on the novel metaphor category.

Dwell time on images reflecting literal meanings

Time spent on the literal images was of specific interest for the study. We found a main effect of group, F(1, 66) = 9.730, p = .003, with longer fixation durations on the images reflecting literal meanings in the ASD group (cf. Figure 2), and a main effect of expression type, F(3, 64) = 26.04, p < .001, with shorter fixation durations on the novel metaphor category.

Figure 2. Number of mouse visits to the target image for each age group and each expression type. Young adults = participants aged 16–22 years; children = participants aged 10–12 years. Expression type: bio = biological, cul = cultural, ins = instructive, novel met = metaphor.

Number of eye-gaze visits to figurative target images

We also considered eye-gaze visits to the figurative target images. We found a main effect of expression type, F(3, 64) = 3.111, p = .032, with fewer eye-gaze visits to the target image in the novel metaphor category compared to all idiom categories.

Number of eye-gaze visits to images reflecting literal meaning

A repeated measures ANOVA showed a main effect of group, F(1, 66) = 8.467, p = .016, with more eye-gaze visits to the images reflecting the literal meanings of the figurative expressions in the ASD group, a main effect of type of expression, F(3, 64) = 10.242, p < .001, with fewer eye-gaze visits on the novel metaphor category and a main effect of modality, F(1, 66) = 6.441, p = .014, with more visits in the visual modality.

Mouse tracking

We report three measures derived from the mouse-tracking data: the number of visits to the pictures presented on the computer screen, the distance travelled with the mouse, and mouse velocity.

Number of mouse visits to figurative target images

Regarding the number of mouse visits to the figurative target images, a repeated measures ANOVA showed a main effect of group, F(1, 66) = 4.907, p = .03, with more mouse visits in the ASD group, a main effect of age, F(1, 66) = 11.104, p = .001, with a larger number of visits in children, and a main effect of type of expression, F(3, 64) = 2.862, p = .044. In addition, a significant interaction was found between age and type of expression, F(3, 64) = 2.788, p = .048 (cf. Figure 2). A pairwise comparison with Bonferroni corrections showed clear differences between children and young adults regarding biological idioms, F(1, 66) = 8.753, p = .004, cultural idioms, F(1, 66) = 9.556, p = .003, and instructive expressions, F(1, 66) = 6.970, p = .010, but no differences in the novel metaphors category, F(1, 66) = 2.371, p = .128.

Number of mouse visits to images reflecting literal meaning

Mouse visits to images reflecting literal meaning were of specific interest for the current study. A repeated measures ANOVA showed a main effect of age, F(1, 66) = 13.456, p < .001, with a higher number of visits in the children’s group. In addition, a significant interaction was found between age and type of expression, F(3, 64) = 2.815, p = .046 (cf. Figure 3). A pairwise comparison with Bonferroni corrections showed clear differences between children and young adults regarding biological idioms, F(1, 66) = 15.219, p < .001, cultural idioms, F(1, 66) = 13.372, p = .001, and instructive expressions, F(1, 66) = 11.688, p = .001, but no differences in the novel metaphors category, F(1, 66) = 2.482, p = .120.

Figure 3. Number of mouse visits to images reflecting literal meanings for each age group of participants and each expression type. Young adults = participants aged 16–22 years; children = participants aged 10–12 years. Expression type: bio = biological, cul = cultural, ins = instructive, novel met = metaphor.

Distance travelled with the mouse

A repeated measures ANOVA showed a main effect of group, F(1, 66) = 4.201, p = .044, with a longer distance for individuals with ASD, and a main effect of age, F(1, 66) = 15.81, p < .001, with a longer mouse distance travelled in children (2–3 times that of adults) (cf. Figure 4(a)). In addition, an interaction was found between expression type and age, F(3, 64) = 2.943, p = .04 (cf. Figure 4(b)). A pairwise comparison with Bonferroni correction on different types of figurative expressions showed that the interaction was driven by differences in the children’s group between novel metaphors, on the one hand, and all categories of idioms, on the other: biological idioms, F(1, 66) = 16.04, p = .011, cultural idioms, F(1, 66) = 14.01, p = .04, and instructive idioms, F(1, 66) = 12.365, p = .027.

Figure 4. (a) Mean mouse distance travelled for each age and group. “Control” = neuro-typical participants; “ASD” = participants with an autism diagnosis; young adults = participants aged 16–22 years; children = participants aged 10–12 years. (b) Mean mouse distance travelled for each age for each type of figurative expression. Young adults = participants aged 16–22 years; children = participants aged 10–12 years. Expression type: bio = biological, cul = cultural, ins = instructive, novel met = metaphor.

Peak mouse velocity

Repeated measures ANOVA showed a main effect of age on peak mouse velocity, F(1, 66) = 4.141, p = .046, with a higher peak velocity in children. Furthermore, we found a significant interaction between age and type of expression, F(3, 64) = 3.003, p = .037 (cf. Figure 5). The interaction was due to a significant difference between children and young adults in cultural idioms, F(1, 66) = 5.228, p = .025, and instructive idioms, F(1, 66) = 10.980, p = .001.

Figure 5. Mean mouse peak velocity for each age for each type of figurative expression. Young adults = participants aged 16–22 years; children = participants aged 10–12 years. Expression type: bio = biological, cul = cultural, ins = instructive, novel met = metaphor.

Furthermore, a significant interaction was found between group and type of expression, F(3, 64) = 4.151, p = .009 (cf. Figure 6). This interaction was due to ASD participants’ differences in peak velocities for biological idioms and cultural idioms, F(1, 66) = 5.228, p = .025, and instructive idioms, F(1, 66) = 10.98, p = .001.

Figure 6. Mean mouse peak velocity for each group for each type of figurative expression. Control = neuro-typical participants; ASD = participants with an autism diagnosis. Expression type: bio = biological, cul = cultural, ins = instructive, novel met = metaphor.

The time courses of mouse velocity for different age groups are shown in Figure 7. For participants who have already responded, the velocity after registering the response was assumed to be equal to 0. For each participant and type of expression, an average velocity was calculated for each time point where mouse position was recorded. These averaged velocities were then averaged over time bins of 200 ms.

Figure 7. Time courses of the mouse velocity for different age groups for each type of expression. Young adults = participants aged 16–22 years (7a); children = participants aged 10–12 years (7b); control = participants without autism (7c); ASD = participants with an autism diagnosis (7d). Expression type: bio = biological, cul = cultural, ins = instructive, novel met = metaphor.

Visual inspection of the plots in Figure 7 reveals that children as a group start moving the mouse more quickly than do adults and keep up higher mouse velocities for longer—that is, they display a flatter curve. This is consistent with the main effect of age in the statistical analysis of peak velocity reported above. Also consistent with the mouse peak velocity analysis, higher peaks in velocity were observed for young adults regarding transparent expressions (biological idioms and novel metaphors) around 2000 ms, while lesser variation in peak velocity between different idioms was observed in the children’s groups. In addition, in the children’s groups, novel metaphors elicited less velocity over time in comparison to idioms (cf. Figure 7(a and b)).

A comparison via visual inspection of the plots in Figure 7 of typically developing individuals (control group) and individuals with ASD shows that individuals with ASD start moving the mouse faster than controls and keep up higher mouse velocities for longer. The control participants demonstrate overall higher peaks in velocity for the most transparent expressions (biological idioms and novel metaphors), and around 2000 ms, but not in later time bins (cf. Figure 7(c and d)).

Finally, for velocity over time, both children and individuals with ASD seem to have fewer peaks in velocity and over a longer time window. This pattern may indicate that the process of responding to the task seems to be more gradual and sustained over time in child participants and participants with autism than in young adult and control groups.

Inspection of the peak velocity plots for each age and participant group shown in Figure 8 provides important observational data. Young adult control participants display higher peaks for the two decomposable and transparent groups of expressions (metaphors and biological idioms), but these peaks obtain around the 2000-ms time window with fast deceleration until 4000 ms, suggesting that a decision has been made. In the child control group, however, novel metaphors appear to be resolved earlier than do biological idioms, and the two show opposite trends (no peak for metaphor vs. high peak for biological idioms) in the time window between 2000 and 4000 ms. The young adult participants with autism display sustained high velocity for novel metaphors even after the 4000-ms window, while peaks in velocity can be observed in the biological idiom category around 4000 ms and later around 6000 ms, indicative of a sustained process of making a decision. Child participants with autism display sustained high velocity over time mostly in the biological idiom category, and as late as until the 8000-ms time window. These observational findings suggest that for both groups of children, biological idioms stand out as a category eliciting sustained high hand motor activity before decision for much longer than for adults.

Figure 8. Time courses of the mouse velocity for each age group and for each participant group for each type of expression. Young adults = participants aged 16–22 years (8a,b), children = participants aged 10–12 years (8c,d); control = participants without autism (8a,c); ASD = participants with an autism diagnosis (8b,d). Expression type: bio = biological, cul = cultural, ins = instructive, novel met = metaphor.

The analyses did not reveal any main effects of presentation modality. Only minor three-way interactions were found, and these are not discussed in the current paper in view of the main research focus of the report.

In addition, a principal component regression (PCR) was performed on the combined gaze and hand data, which corroborated the ANOVA results, thus adding robustness to the separate analyses.

Discussion

The current study aimed at investigating the underlying mechanisms of figurative language processing in individuals with autism compared to their typically developing age- IQ- and verbal comprehension-matched peers. We analysed both eye- and hand-movement data only on a subset of responses where participants had given the target-correct response. It should be mentioned here that despite a significant difference in accuracy between participants with autism and their matched controls, the proportion of correct responses was very high overall (above 80% even for participants with autism, as reported in Chahboun et al., 2016). The combination of gaze and hand-movement data analyses allowed for tapping the strategies that underlie on-line language processing and the extent to which gaze and manual action behaviour overlap and when they diverge. Our main interest focused on the subtle on-line stages in processing up to making a response, and whether participants considered the literal interpretation of the figurative expression as a first or even parallel stage when providing the target response. Consistent with current models of figurative language processing, we were also interested in the extent to which degree of decomposability of the expression affected processing strategies (Vulchanova et al., 2019). The results confirmed our main hypotheses.

We hypothesized that the gaze and hand-movement data will be largely convergent, but would also diverge, depending on the time-line of responding to the task. We found that gaze and manual action converge for target figurative images, with this category eliciting longer fixations and more mouse visits. They diverge, however, concerning literal images, the category that was of specific interest for the study. Here, we observed exclusively gaze behaviour signatures: longer fixation duration and more gaze visits, but no evidence of reconsidering the literal interpretation at later stages of decision on the task. These results are consistent with the idea that at initial stages at figurative language processing language users are actually considering the literal interpretation (gaze), while the later stages (motor action) they converge on the target meaning as the most likely one in the given context.

Our second main hypothesis concerned the viability of literal interpretations, specifically in autism. This hypothesis was confirmed. Regarding eye-gaze results, the participants with autism displayed longer fixation durations than their typically developing peers on both target figurative images (∼400 ms increase) and images reflecting literal meanings (∼200 ms increase). These results could be indicative of either an overall slowed processing speed or longer deliberation. The same was observed in children, but only for target images, as TD children did not fixate on literal images as long as the children with autism did. Together these findings suggest that considering literal interpretations is a potential step in figurative language processing, specifically in participants with autism. Further evidence of the trend to attend to literal interpretations was shown in the gaze visit data, where again a main effect of group was observed. Participants with autism were more likely than controls to re-visit the image on screen corresponding to the literal interpretation of the expression. These results are consistent with earlier findings in our research where the participants with autism converged on more literal responses than controls (Chahboun et al., 2016, 2017; Vulchanova & Vulchanov, 2018). They are also consistent with the results reported in Chouinard and Cummine (2016) where high-functioning participants with autism demonstrated greater difficulty than did controls in inhibiting the literal (unintended) meaning of metaphors, despite being successful in providing overall correct responses. These findings thus provide novel experimental evidence and support earlier research documenting a problem in autism related to stronger activation of literal interpretations and, in all likelihood, inability to suppress them after the initial activation stage (Gold & Faust, 2010; Hobson, 2012; Melogno, Pinto, & Orsolini, 2017; Vulchanova et al., 2012a).

The analyses also provided evidence of the effect of expression type. Decomposability (transparency) effects were observed in both gaze duration and the eye-gaze visits, as well as in mouse peak velocity data. Novel metaphors elicited the shortest fixation durations and fewer visits in all four participant groups (main effect of expression type), and earlier peaks in mouse velocity, with fast deceleration after this, demonstrated in the observational data, especially in the two control groups and the young adult participants with autism, thus confirming the importance of transparency in figurative language processing in both typical and atypical development. This suggests that participants can process novel metaphors more readily and with less effort. This finding is consistent with previous findings reported in Chahboun et al. (2016, 2017), Vulchanova and Vulchanov (2018), and Chouinard and Cummine (2016) but contradicts other results suggesting greater difficulty for novel metaphors in comparison with conventional metaphors (Gold & Faust, 2010, 2012; Gold, Faust, & Goldstein, 2010). The account offered by the latter authors is that, unlike literal language, in novel metaphors comprehension involves creation of new associations that might violate the rules underlying literal comprehension, rendering this type of expression more challenging. It should be observed, however, that in the current study we compared novel metaphors to three categories of idioms, but we did not make a direct comparison with other types of metaphor or non-figurative language. Thus, the current findings should be interpreted in the light of comparison between novel metaphors, on the one hand, and idioms, on the other.

Interestingly, the two decomposable/transparent categories—biological idioms and novel metaphors—exhibit diverging processing trends in our data. While novel metaphors elicit faster and less effortful responses, biological idioms, in comparison, appear to present a challenge seen especially in the mouse peak velocity data and especially in both child groups. This calls for an account that invokes both the underlying cognitive processes involved and the developmental perspective. While metaphors require an explicit association between two inherently unrelated concepts and their figurative intended meaning can be constructed on-line by using the mechanism of analogy, idioms depend mostly on accessing the stored target figurative meaning, even for the most decomposable, transparent types. We find further support for this difference in that all participant groups appear to understand novel metaphors faster and with less effort, while biological idioms appear more challenging, especially for the younger participants. From a developmental perspective, it can be assumed that the two adult groups have already acquired the idioms in our stimuli and rely on direct access to stored figurative meanings, while children with and without autism are more likely to also consider the “lingering” literal meanings of the constituent words in this most transparent category, and for this reason they take longer to inhibit the non-target literal interpretation.

The effect of expression type was observed again in the interaction between expression type and age. Thus, for children, we observed shorter mouse distances specifically in the novel metaphor category. Given that novel metaphors are the most transparent type of expression, this result was anticipated, suggesting that even young children can access the meaning without the need of previous familiarity. The interaction between expression type and age was driven mainly by the difference found in children, and, specifically, in responses to idioms, but not to metaphors (cf. Figure 6). This reflects the fact that children can handle more transparent non-literal expressions, such as metaphors, but they are still learning idioms and their interpretations. This is further confirmed by the fact that, in all types of expression, adults display shorter mouse distances, indicative of greater confidence in responding and better overall skills in figurative language competence.

The analyses of the mouse-tracking data provided further support to our hypotheses. The distance travelled with the mouse revealed a main effect of age and group, with longer mouse distances for children and participants with autism, suggesting more difficulties in processing figurative expressions.

One possible reason for the main effect of group found in the mouse distance travelled may be the well-attested motor skills problems in individuals with autism. However, the findings in that domain are still limited and controversial, and often asymmetries are found between gross- and fine-motor skills in that group (Ozonoff et al., 2008), as well as differences depending on whether standardized tests are used or not. A recent study of manual dexterity in autism (Whyatt & Craig, 2013) reports specific problems in the domain of perception–action coupling, primarily in prospectively controlling hand movement to accurately navigate the pen between the boundaries of a drawing. Since the current task did not require this type of fine motor control, we do not expect that this might have caused the observed group differences. Furthermore, the interaction effects were primarily driven by the children’s data, suggesting that (fine) motor control develops with age, also in participants with autism. This assumption is further strengthened by the mouse peak velocity data, where a main effect only of age, but not of group, was observed. Also, recent evidence in our research demonstrates that high-functioning individuals with autism are similarly accurate as controls in responding by mouse clicks to expressions corresponding to 2D objects positioned on a spatial axial grid (Bochynska, Vulchanova, Vulchanov & Landau, under revision).

One of our hypotheses concerned establishing age differences in our sample. All the ANOVA analyses give evidence of a main effect of age, suggesting that, overall, idioms were more difficult for children than for young adults. Expression type effects and interactions were often observed only for children, but not for young adults. This indicates that children are developing figurative language competence. Yet no age or group differences were found in the novel metaphor category, indicating again the effect of expression transparency, also for participants with autism.

The effect of age is evident in both gaze and manual behaviour during task performance. Child participants displayed higher gaze duration, more mouse visits to target images, and more mouse visits to literal images.

The mouse peak velocity analyses provide further evidence of a main effect of age. Thus, more vector changes were observed in younger children, even when they provide the correct answer. This result suggests that children take longer and are less confident when providing an answer, specifically in the domain of figurative language processing, as already displayed in longer distances travelled with the mouse. The peak velocity data also provide indication of specific difficulties in the children’s groups in the domain of cultural idioms and instructive idioms specifically, as shown through the significant interaction between age and expression type. This result is not unexpected, given that these two categories of figurative language require longer exposure to language and follow a linear curve of development (Vulchanova et al., 2011). This finds further support in the young adult data, where lesser peak velocity was observed in these categories, indicative of better competence in this domain. The ANOVA analyses did not reveal differences in peak mouse velocity between the two age groups for biological idioms and novel metaphors, again suggesting the central role of expression decomposability and transparency in figurative language processing. These findings may further indicate that young adults most probably directly access the meanings of the non-decomposable categories (cultural and instructive idioms), by-passing a decomposition stage, as a result of familiarity and longer exposure, while being less decisive on the two decomposable categories (biological idioms and novel metaphors). This is not the case with the children, who are still learning idioms and do not seem to resort to direct access (Figure 5). These results provide additional support to the hybrid model (Titone & Connine, 1999) of idiom processing, not only for different categories of figurative expressions, but also across the life-span.

The mouse peak velocity data provided unexpected, but important evidence through the interaction between group and type of expression. There was evidence of greater peak velocity only for the participants with autism and only in the decomposable idiom category (biological idioms). Greater peak velocity indicates less certainty in providing a response and further suggests greater competition between alternative choices when assessing these alternatives. Biological idioms are more decomposable and transparent in comparison to both cultural idioms and instructive idioms. They are based on human experience and interaction with the environment and are, as such, more likely to invoke sensorimotor or psycho-physical representations of reality. For this reason, the tension between literal and non-literal interpretation may be greater in that category. Thus, decomposability, added to the inherent embodied nature of the expression and link to experience, may make the literal interpretation a viable option, in turn causing more uncertainty in selecting the target figurative interpretation. This finding is further supported by observational data from the mouse peak velocity plots showing greater and more sustained peaks in velocity in the two child groups, but not in the adult groups. This is novel evidence in support of the idea that decomposability does not always confer advantages in figurative language processing, and its impact on comprehension depends on other factors, such as, for example, degree of semantic relatedness between the literal and the figurative meaning, as well as the frequency of constituent words (Milburn, 2018; Vulchanova et al., 2019).

We hypothesized that differences would be found in our data regarding differential strategies employed by the different groups of participants. The analysis shows that even when correct responses are provided, there are often differences in the strategies used by different groups of participants and different ages. Thus, both children and participants with autism display greater uncertainty when providing an answer, as shown in both mouse and gaze visits and longer gaze duration for visual stimuli, and greater competition when assessing alternatives. This competition may reside in the trend observed in these two groups to assess specifically literal against target interpretations. The overall implication of this finding is that literal interpretations are often activated when processing figurative language, yet the degree of activation may depend on figurative language competence, age, and type of expression. These results are thus consistent with recent theoretical accounts along the lines of Carston’s (2010) “lingering of the literal” idea.

The current study has provided deeper insight into figurative language processing in children compared to young adults and in highly verbal participants with autism compared to typical controls. The analyses of variance of gaze and mouse-movement data suggest that individuals with ASD have different processing patterns than typically developing peers when interpreting figurative language. This is evidenced in longer mouse distance travelled, longer fixation durations on target and literal images, and more mouse and gaze visits to images of interest. Furthermore, both children with and without autism and participants with autism display greater uncertainty and greater competition between alternatives when providing the answer. This competition may often be caused by also considering the literal interpretation of the expression against its target figurative meaning. Finally, we provide evidence that expression transparency plays a central role in figurative language processing across all participant groups; however, its impact may differ depending on expression type, with diverging tendencies for novel metaphors and biological idioms accountable by the differences otherwise between these two types of figurative expression. Our study also provides evidence of the developmental trajectory of figurative language processing. The differences between the children’s groups and the young adult groups suggest that literal representations of figurative language are most probably more active earlier in developing figurative language competence, while later, and as a result of figurative language experience, young adults seem to rely more on direct access of the target figurative meaning and automatized processing (as seen in less visits to images and peak velocity data).

Some remarks on the methodological implications of the current design are in order. The design we used diverges both from classical Visual World (VW) paradigms and most forced-choice mouse-movement paradigms. Our choice was justified by the phenomenon we intended to investigate: non-literal figurative language. A forced-choice paradigm would have been too easy for participants and prone to chance responses. We are fully aware that the choice of design and the type of task might have influenced participants’ responses and results. However, the mouse-tracking data suggest that participants, when uncertain, were mostly considering a choice between the target and non-target literal interpretations, and rarely one between the other two options (non-target figurative or foil), despite randomization. This comes to suggest that the type of design might not have introduced additional confounds in the results. Further research should explicitly compare forced-choice to VW paradigms for use in mouse tracking.

The current study is the first to report both gaze and hand-movement data in figurative language processing, and it is among the few eye-tracking studies of language processing in autism. It also contributes to the still under-researched field of mouse-tracking language studies in children. We have established a clear trend for literal interpretations in both younger groups (children) and the participants with autism. Despite some earlier findings in research, this trend has largely been reported anecdotally. We propose to interpret this finding rather as suggestive of inability to suppress the available literal interpretation, and not of a tendency to provide literal responses, since our analyses were based exclusively on correct responses only. Further evidence of this account can be found in the pattern of processing biological idioms, which elicited larger peaks in mouse velocity in comparison to novel metaphors and the other idiom categories. Since this trend was observed also in younger participants, our findings may be taken as indicative of a developmental delay in the participants with autism, also evident in some of the processing differences between the children and the adults with autism. These findings can be implemented in future successful interventions for autism (e.g., in line with those proposed in research by Melogno et al., 2017). Given the difference we found between novel metaphors and biological idioms, new research is needed to address the processing of decomposable figurative expressions, where carefully controlled stimuli are compared experimentally. This research should aim at establishing the conditions under which decomposability enhances comprehension against conditions when it does not in different populations of language users.

Disclosure statement

No potential conflict of interest was reported by the authors.

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This work was supported by the ERCIM ‘Alain Bensoussan’ Fellowship Programme; European Union 7th Framework Programme for research, technological development and demonstration [Grant Number n° 316748].