Semantic word integration in children with cochlear implants: electrophysiological evidence

ABSTRACT

Differential auditory experiences of children with hearing-loss who receive cochlear implants (CIs) may influence the integration of lexical and conceptual information. Here we measured event-related potentials during a word-picture priming task in CI-using children (n = 29, mean age = 81 months) and typically-hearing children (n = 19, mean age = 75 months) while they viewed audiovisual-word primes and picture targets that were semantically congruent or incongruent. In both groups, semantic relatedness modulated ERP amplitude 300–500 ms after picture onset, signifying an N400 semantic effect. Critically, the CI-using children’s responses to unrelated pairs were significantly more negative than hearing children’s responses. Group differences were mirrored in an earlier 150–275 ms time window associated with a P2 response. The present findings suggest attentional and/or strategic differences impact semantic processing and contribute to the N400 differences observed between groups.

KEYWORDS:

• Children
• cochlear implants
• semantics
• event-related potential
• N400

Introduction

Congenital deafness can lead to significant language delays in children acquiring spoken language. The cascading effects of impoverished linguistic knowledge impact a wide range of behaviours and limit educational outcomes (Calderon & Greenberg, 2005; Perfetti & Sandak, 2000; Pisoni & Geers, 2000). Deaf children who receive a cochlear implant early in life and engage in intensive oral/aural therapy often make great strides in spoken language acquisition. However, even under optimal conditions and the best efforts of clinicians, there is a great deal of variability in language outcomes (Nelson & Crumpton, 2015; Nittrouer et al., 2014; Spencer et al., 2003; Tobey et al., 2012). Studies of vocabulary knowledge in children with cochlear implants paint a complicated picture. While studies have reported that young CI recipients show rapid emergence of first words and substantial amount of vocabulary acquisition at the end of the first year (Ertmer & Inniger, 2009; Faes et al., 2017; Fagan, 2015; Koşaner et al., 2013), there are indications that early rapid vocabulary acquisition does not persist past the first year of CI use (Jung et al., 2020; Koşaner et al., 2013). The composition profiles of vocabulary items learned may also differ from typically hearing children (Jung et al., 2020). Many young CI recipients never acquire vocabulary sizes commensurate with their peers with normal hearing (NH; Dettman et al., 2016; Hayes et al., 2009; Rinaldi et al., 2013) and meta-analysis demonstrates lower expressive and receptive vocabulary knowledge in children with cochlear implants compared to typically hearing children (Lund, 2016).

Children who develop large vocabularies in preschool tend to have better language, reading, and cognitive outcomes than children with smaller vocabularies (Marchman & Fernald, 2008). High-level vocabulary comprehension is a critical attribute in successful reading and is a stable predictor across grade levels for normally hearing English monolinguals (Espin & Deno, 1995; Espin & Foegen, 1996; Yovanoff et al., 2005) and deaf individuals (Cates et al., in press; Choi, 2013; Garrison et al., 1997; Sarchet et al., 2014; Spencer & Marschark, 2010). Understanding the determinants and utilisation of vocabulary knowledge in deaf children is a pressing issue with tangible educational benefits.

Semantic processing and vocabulary development

Vocabulary development is dependent upon a child’s ability to associate lexical labels with conceptual semantic information. For example, a child must be able to associate their experienced-based conceptual knowledge of a cookie with the lexical label of “cookie”. Afterwards, the lexical label of “cookie” can then be used to access corresponding conceptual semantics about cookies. Indeed, lexical knowledge begets further semantic associations; this reciprocal relationship between words and meaning underlies linguistic and intellectual growth. Fundamentally, the ability to acquire vocabulary is dependent upon several component skills which include (a) an ability to perceive and encode lexical labels, (b) ordered semantic conceptual knowledge, and (c) a means to associate and integrate lexical and semantic knowledge. The efficient utilisation of vocabulary requires an ability to relate lexical labels to corresponding semantic representations, which may include the inhibition of competing associations. The fidelity of lexical representations has direct consequences for word recognition. The breadth, depth, and speed of retrieval of word meanings is particularly important in determining the extent to which an individual achieves a good understanding of language (Perfetti, 2007). Recent evidence from Amenta et al. (2021) elucidates the idea that early adult CI users can demonstrate qualitative differences in their word recognition abilities when compared to typical hearing controls. These CI users were slower to comprehend low frequency words, possibly reflecting that these words have less consolidated lexical representations and would thus be more difficult to retrieve (Amenta et al., 2021). Along with this work on adult CI users, there is evidence that children with CIs face challenges in several aspects of these component processes that support vocabulary development and usage.

Regarding the perception and encoding of lexical information, CI-using children may struggle to perceive auditorily-presented lexical labels. Auditory input from a CI is more difficult to comprehend than input from natural hearing (Moore & Shannon, 2009). A spoken lexical label that is difficult to perceive is likely to be harder to phonologically encode, which is an imperative step in early word learning (Houston et al., 2005) and subsequent word utilisation (Perfetti, 2007).

With respect to the quality of conceptual semantic representations, several studies of deaf adults and children have reported differences in the utilisation of semantic knowledge in word association and analogy tasks (Marschark et al., 2004). Green and Shepherd (1975) studied knowledge of antonymous pairs of adjectives, such as good–bad and slow–fast, in both deaf and hearing children. They found a more restrictive semantic system in the deaf children. More recently, Ormel et al. (2010) reported data from a pictorial task requiring exemplar categorisation and a word-picture task assessing superordinate-level semantics where hearing children outperformed the deaf children. They concluded that semantic-categorical knowledge of deaf children appears to be less precise or less finely differentiated than the semantic-categorical knowledge of hearing children.

Finally, there is evidence that associative processes to combine lexical and conceptual semantic information can be influenced by auditory experience, even at the earliest level of word learning. A study by Houston et al. (2012) tested hearing and deaf children aged 12–40 months on their ability to learn novel word and object pairings. Hearing children demonstrated longer looking times to target objects than distractor objects beginning at 18 months, indicative of their early ability to associate lexical labels with novel objects. Performance for deaf children with CI was correlated with auditory experience, such that those children who had more hearing before implantation or had begun using their implants by 14 months of age performed better than deaf children who had less auditory stimulation early in life. Performance on this word learning task was also correlated with later measures of vocabulary, suggesting that early auditory experience may have an ongoing influence on semantic development (Houston et al., 2012).

The highlighted research suggests that early auditory experience can influence capacities that are crucial to word learning and semantic processing. Perception and phonological encoding of spoken lexical labels may be compromised, and deaf children may show more heterogeneous and less-well defined semantic-conceptual knowledge. However, less is known about the associative processes that relate lexical information with conceptual semantic information, and how this is influenced by auditory deprivation and constrains the development of vocabulary knowledge,

Though behavioural studies of semantic development have revealed discrepancies between CI and hearing children, there is a gap in our understanding of the neural processes underlying these group differences. If behavioural data describe the end result of a cognitive process, neural measures such as electroencephalography (EEG) can help shed light on the timing and mechanisms of the cognitive process itself. Here we make use of a word-picture semantic priming paradigm to gain further insights into components of lexical processing and semantic word integration in children with cochlear implants.

The N400 as a neural marker of semantic integration

The N400 ERP component is a well-established neural marker of semantic processing of linguistic information (Kutas & Federmeier, 2011; Swaab et al., 2012). This is a component that has a negative deflection between 200 and 500 ms after stimulus onset. Its scalp distribution is maximally amplified at centroparietal electrode sites, but has been shown to elicit frontocentral effects in response to pictorial stimuli (Barrett & Rugg, 1990). The modulation of the N400 component observed in congruent and incongruent contextual environments is referred to as the N400 effect. While it is widely recognised that the N400 is modulated by semantic factors, the time course and the mechanisms that give rise to the effect is actively debated (Kutas & Federmeier, 2011; Lau et al., 2008; Nieuwland, 2009). With respect to time course, there are questions as to whether ERP components that appear prior to the traditional N400 window reflect predictive linguistic processes that influence the sensory analysis of stimuli. It is also possible that these early components may reflect initial stages of word recognition before word meaning is integrated with sentence context. (Nieuwland, 2009). Analysis of the N400 effect has been further dichotmized. Under an integration account, the effect reflects a combinatorial process in which the ease or difficulty of integrating a current stimulus to a prior context modulates the magnitude of the effect. Alternatively, a lexical access view holds that the N400 effect reflects facilitated activation of features of the long-term memory representation that is associated with a lexical item (Lau et al., 2008). An alternative characterisation is offered by Kutas and Federmeier (2011), such that the measured N400 phenomena reflects a moment in time that captures the intersection of a feedforward flow of stimulus-driven activity with a state of the distributed, dynamically active neural landscape that is semantic memory. Task-induced strategic factors and dispositional states of participants might bias operations across this divide. Finally, while our understanding of the precursors and mechanism of the N400 effect are evolving, it should be further noted that current hypotheses of causal mechanisms underlying N400 effects are largely predicated on findings from adult language processing and the degree to which these alternative hypotheses are applicable to paediatric and special populations remain understudied.

N400 in development

N400 effects have been consistently elicited in typically developing verbal children as young as 19 months (Friedrich & Friederici, 2006; Morgan et al., 2020). Larger N400 responses were found for pairs of pictures with pseudo-word labels compared to pictures with semantically matching word labels, and the presence of an N400 at 19 months was predictive of children’s later language abilities. A similar study by Rämä et al. (2013) showed that N400 effects consistently occurred in 24-month-olds, and in 18-month-olds with high vocabularies. These effects were in response to spoken words pairs that were either semantically congruent or incongruent, and demonstrate an early rapid semantic development in response to spoken language in hearing children (Rämä et al., 2013). Vocabulary size as a proxy for semantic development is often associated with the presence of an N400 response in studies with younger populations, such as before age 2 years. Additionally, the maturation of a child’s semantic processing system can influence the shape of the N400 component, such that the latency and range of the N400 decreases over development (Lindau et al., 2017).

The relationship between vocabulary and N400 effect size appears to be time-sensitive. The factors that drive these developmental correlations are not well understood. A study by Henderson et al. (2011) tested children ages 8–10 years on measures of vocabulary knowledge, listening comprehension, word recognition, and non-word decoding. ERPs were recorded while the children saw semantically congruent or incongruent pairs of pictures and words that were presented synchronously. The ERP results demonstrate a widespread N400 effect that was sensitive to semantic incongruency, consistent with previous findings. A centroparietal N200 response was also found to be sensitive to semantic incongruency, though the authors posit that this response may represent the beginning of the N400 in their study. Critically, magnitude of the N400 effect was moderately correlated with listening comprehension but not with vocabulary knowledge at this age. In light of these findings, the authors suggest that the N400 effect reflects integrative processes that support comprehension as opposed to lexical access processes that would be more sensitive to one’s vocabulary knowledge (Henderson et al., 2011).

N400 in cochlear implant users

Several studies have compared N400 responses between hearing adults and adults who use CIs. Most of these studies focus on older adults who have acquired their deafness post-lingually, which makes for a qualitatively different auditory and linguistic experience from congenitally deaf children. Hahne et al. (2012) used an auditory sentence comprehension task to test for N400 responses to semantic violations in hearing adults and adults with CIs. The N400 responses of CI users persisted beyond the time window of the N400 responses of hearing adults, though patterns of modulation in response to semantic violations were similar to those of the control group. Though the authors do not draw a specific conclusion as to why the breadth of N400 responses was longer for CI users than controls, the results are discussed in a larger context of how degraded auditory input through a CI may slow down speech comprehension processes (Hahne et al., 2012). Other studies with adult CI users have shown that N400 latency is delayed in CI users relative to controls (Finke et al., 2016), and that the N400 is less likely to be present in older adult CI users in an auditory Stroop task (Henkin et al., 2015). These studies draw similar conclusions that the difficulty of speech perception relates to differences in N400 responses between groups.

There is limited work on electrophysiological responses of semantic processing in children with CIs (Bell et al., 2019; Kallioinen et al., 2016; Vavatzanidis et al., 2018). Vavatzanidis et al. (2018) used a longitudinal approach to understand semantic development after cochlear implantation in 32 congenitally deaf children ages 21–65 months. EEG was recorded during exposure to pictures and matched or mismatched spoken words at three time points after CI activation (12, 18, and 24 months post-activation). Though most children were able to demonstrate semantic processing through an N400 effect after only 12 months of CI use, a subset of children did not show an N400 effect at any point after activation. The presence of an N400 effect was strongly associated with performance on language assessments administered 24 months post-activation, such that higher language comprehension and vocabulary scores were associated with greater neural evidence of robust semantic relationships between words and pictures reflected by the N400 effect (Vavatzanidis et al., 2018). Without a hearing comparison group, however, it was difficult for the authors to determine if this variability in semantic development is within the range of typical development or caused by unique factors associated with early auditory deprivation or hearing through a CI.

Kallioinen et al. (2016) used an auditory word and picture semantic priming task to elicit N400 responses from 30 deaf and hard of hearing children aged 5–7 years, 15 of which had at least one CI. The authors predicted that larger N400 mismatch effects would occur in response to between-category semantically mismatched items (e.g. “wolf” and followed by a picture of a car) compared to within-category semantically mismatched items (e.g. “wolf” followed by a picture of a bear). They also predicted larger mismatch effects for hearing children compared to deaf children, under the assumption that semantic discrimination would be easier for groups with better hearing. Unexpectedly, the children with CI demonstrated larger effects in response to between-category semantically mismatched items than hearing controls and children with hearing aids. Behavioural results did not suggest that the CI-using children had better semantic discrimination; additionally, the timing and magnitude differences observed between groups led the authors to tentatively conclude that children with CI may have less precision in semantic processing, or a stronger reliance on predictive processing; this conclusion, however, was not explicitly related to the N400 (Kallioinen et al., 2016).

Bell et al. (2019) recorded ERPs from 12 CI-using children ages 6–9 on a spoken word-picture paradigm. Trials in this task consisted of spoken words followed by pictures that were semantically congruent or incongruent, with an even distribution of congruent and incongruent trials. In both hearing and CI-using children, the incongruent trials elicited more negative N400 responses. The authors found no differences in patterns of responses between CI and comparison groups, despite their predictions that semantic integration would be more difficult for the CI-using children. Behavioural measures, however, indicate that CI-using children performed significantly worse than hearing children on tasks related to spoken language comprehension. The authors’ interpretation of these results is that though spoken language difficulties occur for children with CI, these difficulties are not represented at a neural level by the N400.

In summary, studies of N400 effects in CI users present a complicated picture. Studies of older CI users typically report N400 differences owing to constraints on speech perception. Studies in pediatric CI users show similar (Bell et al., 2019) or exaggerated effects of semantic incongruity (Kallioinen et al., 2016), the latter indexing semantic representational differences and/or arising from strategic processing differences. Importantly, N400 effects appear to be correlated with later language and vocabulary development (Vavatzanidis et al., 2018).

Present study

The present study investigates school-aged children’s N400-responses in a passive word-picture paradigm. This paradigm was developed in order to assess semantic processing of words and pictures in children who use either English or American Sign Language (ASL), with the ultimate goal of understanding how semantic systems develop across different language modalities.¹ Here we present data using English audio-visual primes. We conjecture that successful lexical access of a verbal prime will activate congruent semantic conceptual knowledge and potentially associated lexical forms within the lexicon. The subsequent recognition of target pictures, which we take to be a depictive proxy of conceptual semantic knowledge, will either yield a matching confirmation or disconfirmation between activated semantic knowledge.

A noteworthy aspect of the present experiment is the use audio-visual primes. The decision to use audio-visual primes reflected a desire to provide all participants a naturalistic stimulus that allows participants to make use of available processing capabilities to recognise and encode the prime words. Evidence from adult studies suggests that the presence of visual cues during speech perception improves comprehension, though this facilitation is not consistently shown in children and may differ for children with delayed access to auditory input (Bergeson et al., 2010; Knowland et al., 2014).

Subjects in the present study were children, ages 2½ to 10 years old, with cochlear implants who were tested on the Spoken English-version of the word-picture paradigm. An age-matched comparison group of normal-hearing children were tested on the same paradigm. Subjects saw videos of a speaker saying the names of common nouns as primes, followed by pictures of the nouns as targets. The target picture either matched the word spoken in the prime video (congruent condition) or did not match the word spoken in the prime video (incongruent condition). In order to keep the task simple for the youngest age group, children were not required to make any responses and were only instructed to pay attention to the pairs of videos and pictures. Intermittent short videos of Pokémon figures were included to keep the task engaging. Therefore, no behavioural responses such as reaction times were analysed.

Considering the prior literature, we entertained three hypotheses. First, if perception of speech sounds in CI-using children is degraded compared to children with intact hearing, there may be less complete processing of the audio-visual primes. These shallow inputs may limit the efficient elaboration of word meaning (Perfetti, 2007). In adults, decreased awareness of target stimuli, for example under condition of attentional selection or masking, have shown to attenuate early N400 effects (Holcomb & Grainger, 2009; McCarthy & Nobre, 1993). Under this account we speculate that there will be less overall modulation of the N400 effect with similar patterns manifested across both congruent and incongruent conditions.

Alternatively, if the use of naturalistic audio-visual primes place children with CIs and their hearing peers on more equal footing for the stimulus perception, differences may arise due to less precise or less finely differentiated semantic-categorical knowledge, leading to increased N400 effects for the CI group (Kallioinen et al., 2016).

Finally, to the extent that there are strategic or attentional differences between CI-using children and typically hearing children we may expect differential N400 effects, as attention generally increases both early and late electrophysiological signals (Deacon & Shelley-Tremblay, 2000; Hillyard, 1985; Mangun & Hillyard, 1991). Furthermore, strategic processes such as active prediction have been shown to modulate early components associated with lexical-semantic integration (Brothers et al., 2015). Electrophysiological responses may provide insight into the temporal properties of these effects.

Methods

Subjects

Children with CI

A group of 29 children with cochlear implants participated in the study after their caregivers provided written consent. This sample consisted of 18 boys and 11 girls, with a mean age of 81.10 months (SD = 20.28; range 30–122). Mean age of first implantation was 27.29 months (SD = 20.62; range 6–79). The average Time-in-Sound (TIS) for the children in this sample was 55.64 months (SD = 20.08; range 5–43). Attention Deficit Disorder (ADD) was reported by parents for two of these children, and an additional two children were born with normal hearing and diagnosed with hearing loss after their first year. One child was also diagnosed with Pendred syndrome, which is a genetic disorder that can cause deafness and thyroid problems. Seven of these children had prior or concurrent exposure to ASL at the time of their participation in this study. Table 1 presents characteristics and demographics of the CI-using subjects in this study (Table 1).

Table 1. Cochlear implant-using (CI) subject characteristics.

Subject	Age	Gender	Age at first implant	CIs	TIS
1	30	F	10	bi	20
2	56	F	13	bi	43
3	57	F	40*	uni	17
4	58	M	6	bi	52
5	58	F	53	bi	5
6	61	M	12	bi	49
7	67	M	13	bi	54
8	67	F	13	bi	54
9	72	F	50*	bi	22
10	74	F	7	bi	67
11	74	M	13	uni	61
12	77	M	21	uni	56
13	77	M	14	bi	63
14	78	M	11	bi	67
15	82	M	10	bi	72
16	82	M	36	bi	46
17	86	M	18	bi	68
18	86	M	15	uni	71
19	87	F	31	bi	56
20	87	M	13	bi	74
21	89	M	9	bi	80
22	89	F	40	bi	49
23	100	M	38	bi	62
24	100	M	14	bi	86
25	102	M	15	bi	87
26	106	F	71*	uni	35
27	106	M	79**	uni	27
28	122	F	42	bi	80
29	122	M	67	bi	55

Notes: TIS = Time in Sound, defined as number of months since first CI activation.

*Subject had progressive hearing loss and/or was born with normal hearing.

**Subject had hearing aids prior to cochlear implantation.

Comparison group

A comparison group of children with normal hearing, as reported by their parents, were recruited for the study. This group consisted of 19 children (9 boys and 10 girls) with an average chronological age of 74.58 months (SD = 29.04; range 31–128).

Stimuli

Primes and targets were created from 34 noun concepts currently being used in an ASL-English preschool programme and had readily identifiable single lexical ASL and English forms. The words were a subset of preschool age-appropriate words from the McArthur Communicative Development Inventory for Words and Sentences (Fenson et al., 2007).

Each noun concept was represented in both picture and spoken word form. Pictures were taken from Google Images and were evaluated in-house. Ten children ages 2–10 years participated in a behavioural study by naming the pictures in order to choose the stimuli that would be best representative of the concepts. Only pictures that were suitable for preschool population and received above 90% consensus during evaluation were used. Spoken word stimuli were edited video recordings of a native English female speaker pronouncing each word. Videos were filmed and recorded using a SONY HXR-NX5U camera and microphone against a green screen, which was replaced with a uniform grey background (Final Cut Pro Version 10.4, 2017).

Congruent trials were classified as trials in which the spoken word prime and picture target matched in their represented concepts (e.g. spoken word “cookie” followed by a picture of a cookie). Incongruent trials consisted of a spoken word prime and picture target that did not have matching concepts (e.g. spoken word “table” followed by a picture of a squirrel). Concept pairs in incongruent trials were tested for categorical semantic relatedness by using the WuPalmer algorithm for determining semantic similarity in WordNet, which considers the depth of the two synsets in the WordNet taxonomies, along with the depth of the Least Common Subsumer (Wu & Palmer, 1994). In this measure, pairs of incongruent concepts receive a score between 0 and 1 as a reflection of their semantic relatedness, with highly related concepts receiving scores close to 1. Semantic relatedness scores for the present incongruent stimuli were distributed equally with no clear outliers (M = 0.42; range 0.24–0.74). Pictures and words were each presented twice during the experiment, occurring once in a congruent stimuli trial and once in an incongruent stimuli trial. Of 68 total trials, half of the trials consisted of congruent pairs. The presentation of items in congruent and incongruent conditions were counterbalanced across two versions of the paradigm. For example, subjects who saw the first version of the paradigm initially saw the concept of “cookie” in a congruent condition, whereas subjects who saw the second version initially saw this concept in an incongruent condition.

Experimental procedure

Subjects were seated in a sound-attenuated, dimly lit room facing a centrally positioned AUVIO 05A13 loudspeaker and an LCD monitor. Stimuli were presented using Presentation software (version 20.0, Neurobehavioral Systems, Inc., 2018). They were informed of the general nature of the experiment in that they would be presented with pairs of stimuli, some of which would be congruent and others would be incongruent. A researcher sat beside each participant to encourage vigilance and reduce occasional fidgeting. Other than attending to stimuli, subjects were not given an explicit task to perform during the EEG portion of the experiment.

As shown in Figure 1, spoken word video clips were presented for 900–1930 ms (average: 1158 ms) and picture stimuli were presented for 1000 ms. Sound level was set to 65 dB(A) for children with CIs and 60 dB(A) for typically hearing controls. The interstimulus interval (ISI) was 400 ms between the off-set of the audio-visual word prime and the picture target. The intertrial interval (ITI) between the pairs was 1300 ms. Each prime-target pair was only presented once in a pseudo-randomized order during the experiment. Intermittent trials (n = 10) of short animated Pokémon cartoons were included (average: 4500 ms). These trials were included as motivators for use with our current population. Additionally, occasional trials (n = 9) of a silent static image of the speaker were included to gauge the ERP responses to physical form of the speaker.² Two versions of the experiment were created in order to counter-balance the order of the presentation of prime-target pairs. The EEG recording lasted about 10 min.

Figure 1. Single trial design. Subjects were presented with an audiovisual clip of a woman saying a word for 900-1930ms, followed by an ISI of 400 ms, a picture that lasted 1000 ms and an ITI of 1300 ms. The pictures either semantically matched the spoken word (congruent condition) or did not match the spoken word (incongruent condition).

In addition to collecting EEG data in response to the word-picture paradigm, hearing and CI-using subjects completed the Expressive and Receptive One Word Picture Vocabulary Tests (EOWPVT and ROWPVT, respectively) to gauge their expressive and receptive vocabulary.³ ROWPVT scores were expected to be of greatest relevance to the outcomes of the present paradigm, as the word-picture task measures language comprehension as opposed to production. Therefore, only ROWPVT results are reported below. Although many subjects completed these measures after the EEG session, some of the students’ vocabularies scores were obtained at an earlier date through school-administered testing. Receptive vocabulary scores are missing for 5 hearing subjects due to scheduling issues caused by school closures related to the COVID-19 pandemic.

EEG recording and analysis

Continuous EEG data was collected from 22 electrode sites, using the standard 10/20 system with the Biosemi Active Two System (Biosemi B. V., Amsterdam, Netherlands). The signal was sampled online at 512 Hz and electrode offsets were kept below 20kΩ. The signal was referenced online to the Common Mode Sense (CMS) active electrode, which is placed in the centre of all measuring electrodes and subtracted from the signal later.

The EEG signal was pre-processed using EEGLAB (Delorme & Makeig, 2004) and ERPLAB (Lopez-Calderon & Luck, 2014) toolboxes in MATLAB (The Math-Works, n.d.). The EEG signal was filtered offline using a bandpass filter of 0.1–30 Hz, and was referenced offline to an average of the two mastoids.

The first step of artefact rejection was performed using EEGLAB’s infomax algorithm for independent component analysis (ICA), through which blink and horizontal eye movement components were manually identified and removed from the data. Between one and two eye blink components were removed for all subjects. Electrode sites that were located over the CI processors were eliminated from current analysis. These included sites P7/8, P3/4 and T7/8. These sites typically contained CI artefact due to the fact that they were located at the sites of the CI processor and we were often unable to establish a good connection between electrode and scalp. The remaining nine electrodes were classified into three regions for subsequent analyses: Frontal (comprised of channels Fz, F3, F4, F7, and F8), Central (Cz, C3, and C4), and Parietal (Pz). The continuous signal was then segmented into 1000 ms epochs from −200 ms before to 800 ms after stimulus onset. The second step of artefact rejection involved a voltage threshold of ±115 microvolts on channels of interest; all trials containing voltages over 115 microvolts were rejected. Remaining trials were visually inspected individually and deleted if any artefact remained. On average, each subject retained 90.1% of their trials after this process (range 63.2% to 100%).

ERPs were calculated for incongruent and congruent trials separately for each subject, with the ERP time-locked to the onset of the picture. Grand averages for both groups were produced, the waveforms of which are shown in Figure 2. All statistical analyses were conducted in R (R Core Team, 2018), using an alpha level of .05 for gauging statistical significance.

Figure 2. Grand-averaged event-related potentials (ERPs) for congruent and incongruent picture targets in each group. The average of these nine channels in the frontocentral region of interest were used in all statistical analyses.

Given the age ranges of subject tested, and in lieu of the reduced numbers of hearing controls tested, we opted to use a median split of the data to examine the factor of Chronological Age in subsequent analyses. This results in Younger CI-using children (N = 14, mean age = 64.7 months, SD = 12.9 months); Older CI-using children (N = 15, mean age = 96.4 months, SD = 13.3 months); Younger Hearing children (N = 10, mean age = 52.1 months, SD = 13.0 months); and Older Hearing Children (N = 9, mean age = 99.6 months, SD = 21.8 months).

Results

Receptive vocabulary

Percentile scores of the ROWPVT were calculated based on a child’s performance relative to other children of the same age. The mean percentile for the CI-using children was 50.62 (SD = 25.40) and the mean percentile for the hearing children was 75.29 (SD = 22.33). A t-test of these group differences revealed that the hearing group outperformed the CI group on this measure of receptive vocabulary, t = 3.10, p= .0018.

EEG data

Visual inspection of the data revealed a characteristic initial positivity (P1), followed by a negative going component (N1) beginning at approximately 100 ms post target. This was followed by a positive going component in the P2 window peaking approximately at 220 ms and subsequently a large and extended negative going component approximately 300-600 ms post target (see Figure 2).

P2 response window

Visual inspection of the waveforms shown in Figure 2 prompted an exploratory analysis of group differences in a time window associated with the P2 component. Group differences in mean amplitude and peak latency within this window, along with the primary analysis of group differences in N400 mean amplitude, are reported in Table 2.

Table 2. Group comparisons of electrophysiological responses.

Measure	CI (n = 29) M (SD)	HG (n = 19) M (SD)	t	p
P2 Amplitude (Incongruent)	−4.97 (8.97)	−2.68 (9.14)	−2.569	.010
P2 Amplitude (Congruent)	−1.15 (7.97)	−1.12 (9.02)	−0.0354	.971
N400 Amplitude (Incongruent)	−16.92 (10.24)	−14.01 (10.14)	−2.893	.004
N400 Amplitude (Congruent)	−11.37 (9.63)	−11.14 (9.63)	−0.238	.812

Notes: Measures reflect mean amplitude, recorded at the following electrodes: Fz, F3, F4, F7, F8, Cz, C3, C4, and Pz. CI = cochlear implant-using group; HG = hearing control group.

P2 mean amplitudes and peak latency measures were measured from 150 to 275 ms post-picture onset (see Figure 3). Differences in P2 effect (incongruent – congruent mean amplitude) were assessed in fronto-central sites using three-way ANOVA with factors for Group (Hearing or CI), Chronological Age (defined by a median split), and Region (frontal, central, and parietal). These results are reported in Table 3. A main effect of Group was found, such that CI-using children demonstrated a greater difference in responses between trial conditions; F(1,420) = 11.63, p = .001. No main effect was found for Region, such that P2 effect size did not differ between frontal, central, and parietal sites; F(1,420) = 0.38, p = .744. There was no main effect of Age. However, the interaction between Group and Chronological Age showed a trend towards significance F(2,420) = 3.66, p = .056. No other interactions were significant. An exploration of the Group * Chronological Age interaction revealed a significance difference in the P2 effect size, with older CI-using children showing a larger P2 effect than older hearing controls; F(1,223) = 15.64, p < .001. There was no difference in comparison of the younger participants; F(1,205) = 0.872, p = .351.

Figure 3. Grand-averaged response of CI group (n = 29) in channel Cz demonstrates windows of interest for early sensory (P2-like) component (150-275 ms post-picture onset) and N400 effect (300–500 ms post-picture onset). The red waveform represents responses to Incongruent pictures and the black waveform represents responses to Congruent pictures. The dotted line is the difference wave (Incongruent – Congruent).

Table 3. Summary of ANOVA results for P2 effect size, P2 latency, and N400 effect size.

	F df	F	MSE	p
P2 Effect Size
Group	1, 420	11.63	527.99	.001
Age	1, 420	0.38	17.19	.539
Region	2, 420	0.30	13.42	.744
Group*Age	1, 420	3.66	166.05	.056
Group*Region	2, 420	0.035	1.60	.965
Age*Region	2, 420	2.67	121.02	.071
Group*Age*Region	2, 420	0.58	26.46	.559
P2 Latency
Group	1, 840	1.80	2053.91	.180
Age	1, 840	61.72	70,504.55	.000
Region	2, 840	13.11	14,976.32	.000
Congruency	1, 840	1.45	1653.67	.229
Group*Age	1, 840	4.83	5517.84	.028
Group*Region	2, 840	2.65	3031.31	.071
Age*Region	2, 840	0.87	992.20	.420
Group*Congruency	1, 840	1.15	1312.91	.284
Age*Congruency	1, 840	3.39	3871.71	.066
Region*Congruency	2, 840	0.24	537.90	.790
Group*Age*Region	2, 840	1.34	3068.58	.262
Group*Age*Congruency	1, 840	0.031	35.79	.860
Group*Region*Congruency	2, 840	0.28	628.88	.759
Age*Region*Congruency	2, 840	0.054	122.81	.948
Group*Age*Region*Congruency	2, 840	0.040	90.55	.961
N400 Effect Size
Group	1, 420	10.20	721.61	.002
Age	1, 420	22.95	1623.11	.000
Region	2, 420	0.78	55.35	.458
Group*Age	1, 420	0.019	1.37	.890
Group*Region	2, 420	0.042	2.97	.959
Age*Region	2, 420	0.49	34.65	.613
Group*Age*Region	2, 420	0.206	14.59	.814

Notes: Effect sizes were calculated by finding the difference between Incongruent Mean Amplitude and Congruent Mean Amplitude. P2 effects were measured from 150 to 275 ms post-picture onset at the following electrodes: Fz, F3, F4, F7, F8, Cz, C3, C4, and Pz. N400 effects were measured from 300 to 500 ms post-picture onset at the above electrodes. (n = 48); df = degrees of freedom, MSE = mean squared error.

A similar analysis that tested for differences in P2 mean amplitude was used to assess differences in P2 peak latency, though the latency analysis included an additional factor of Congruency (Congruent or Incongruent). A significant main effect of Chronological Age demonstrated that younger subjects’ P2 responses had longer latencies compared to older subjects; F(1,840) = 61.72, p < .001. We also found a main effect of Region, where peak latencies in the parietal region occurred more slowly than those in frontal and central regions; F(2,840) = 13.11, p < .001. The results did not provide evidence in favour of a main effect of Group or Congruency, suggesting that latency across groups and different trial conditions did not significantly differ. Results of this analysis are represented in Table 3.

N400 response window

Mean amplitudes for each subject was calculated from 300 to 500 ms post-picture onset at each electrode site. N400 mismatch effect sizes were calculated for each subject by subtracting their Congruent mean amplitudes from Incongruent mean amplitudes at each electrode site. These values were entered in as the dependent variable into a three-way ANOVA with the factors of Group (CI or Hearing control), Region (Frontal, Central, and Parietal) and Age (defined by median split). With this, we found a main effect for Group, suggesting that N400 mismatch effect sizes were greater for CI-using children than hearing controls, F(1,420) = 10.20, p = .002. There was also a significant main effect of Chronological Age reflecting that mismatch effects in the younger half of subjects were greater than the mismatch effects of older subjects, F(1,420) = 22.95, p < .001. These data are reported in Table 3.

To further explore the N400 effect, we conducted a 2-way ANOVA with the factors of Group and Congruency to test for differences in mean amplitude within this window. A main effect of Congruency was found, such that both groups had more negative amplitudes to incongruent pictures than congruent pictures, F(1,814) = 88.872, p < .001. There was also a significant interaction of Congruency and Group, indicating that there was a different pattern of N400 responses between groups across conditions, F(1,814) = 7.556, p = .00611. Independent samples t tests indicate a significant between groups difference in 300–500 ms mean amplitude to incongruent pictures only, t(430) = 2.893, p = .004. There was no difference in the groups’ N400 response to congruent pictures, t(430) = 0.238, p = .812. Results of this two-way ANOVA are listed in Table 4, and group by condition differences in amplitude are shown in Figure 4.

Figure 4. Mean amplitude of responses (measured in microvolts) 300–500 ms post picture-onset by condition and group. CI = Cochlear Implant, HG = Hearing. A significant main effect of Condition was found (p < .001), along with a significant interaction of Condition * Group (p < .01). Incongruent picture responses were significantly more negative for CI children than hearing children (p < .01).

Table 4. Summary of ANOVA results for Mean Amplitude from 300 to 500 ms post-picture onset (n = 48); df = degrees of freedom, MSE = mean squared error.

	F df	F	MSE	p
Condition	1, 814	88.872	4358	.000
Group*Condition	1, 814	7.556	371	.00611

Notes: Mean amplitude was recorded at the following electrodes: Fz, F3, F4, F7, F8, Cz, C3, C4, and Pz.

Prompted by an interest in understanding the distribution of these effects across Groups, we tested for differences in incongruent N400 mean amplitude across three regions. These regions were Frontal (comprised of channels Fz, F3, F4, F7, and F8), Central (Cz, C3, and C4), and Parietal (Pz). Results indicate a significant main effect of Region, F(2,812) = 79.028, p < .001; there was also a significant Group * Region interaction, F(2,814) = 4.262, p < .0144. Comparisons of mean amplitude by Region indicate that both groups show widespread negativity in Frontal and Central regions; there was also a trend such that the children with CIs showed enhanced N400 effects in parietal regions relative to the hearing children, though a t-test of mean responses in this region did not yield a significant difference (t = −1.72, p = .0929; see Figure 5).

Figure 5. Mean amplitude of responses (measured in microvolts) 300–500 ms post Incongruent picture-onset by Region. CI = Cochlear Implant, HG = Hearing. Neither group showed a significant difference in mean amplitude of responses between Central and Frontal regions. There was no difference in N400 mean amplitude between Parietal regions across groups, p = .0929.

Correlations with behavioural indices

To better understand the factors that influence the effects occurring in the P2 and N400 windows, we examined correlations between the magnitude of the effects (as defined by Incongruent minus Congruent trial amplitudes) with chronological age and receptive vocabulary. We further examined the influence of Age of CI activation and Time in Sound in children with CI. These data are reported in Table 5 and Figure 6. We observed a significant negative association between age Chronological Age and effects in the P2 window (R = −0.312, p < .001) for CI-using children but no relationship for controls (R = .048, p = .535). There were no significant correlations between magnitude of effects in P2 window and vocabulary as measured with the ROWPVT measure for either subject group. Turning to CI-using children, we assessed the relationship between age of CI activation and magnitude of effects in P2 window; no significant associations were observed. Finally, we examined Time in Sound, a measure of experience with the device. We observed a significant negative correlation (R = −0.26, p < .001) and further, this correlation survives when partialing independent effect of chronological age (R = −0.136, p = .038).

Figure 6. Association of P2 Effect and Age by Group. Each point represents the average P2 effect size per subject across all 9 electrodes. We observed a significant negative correlation between P2 effect magnitude and chronological age in the CI group only, (R= −0.312, p < .001). This correlation was not detected for the HG group (R = 0.048, p = .535).

Table 5. Pearson correlation coefficients for neural responses and behavioural indices.

	CI (n = 29)		HG (n = 19)
	P2 effect	p	P2 effect	p
Chronological age (months)	−0.312	.000	0.048	.535
Receptive One-Word Picture Vocabulary^a	0.033	0.593	0.153	.088
Age of activation (months)	−0.044	0.479	N/A
Time in Sound (months)^2	−0.136	.038	N/A
	N400 effect	p	N400 effect	p
Chronological age (months)	−0.006	.925	0.124	.106
Receptive One-Word Picture Vocabulary^1	−0.045	.472	0.028	.752
Age of activation (months)	−0.007	.913	N/A
Time in Sound (months)^b	−0.011	.86	N/A

Notes: Pearson correlation coefficients (R) for groups’ behavioural indices and P2/N400 effects (calculated by the difference in magnitude of the Incongruent and Congruent picture response). P2 and N400 effects are measured from electrodes Fz, F3, F4, F7, F8, Cz, C3, C4, and Pz. CI = cochlear implant-using group; HG = hearing control group.

^a n = 14 for HG on ROWPVT.

^b Partial correlation controls for variability in Chronological age.

Regression analyses with N400 effects showed no relationships with factors of Chronological Age nor vocabulary for both groups (all p’s > .1), Further, for CI-using children the factors of Age of Activation and TIS failed to reveal any significant correlations (all p’s > .1).

Discussion

The findings of this study show that children with CIs and their hearing counterparts evidence an N400 or mismatch effect, whereby we observe more negative amplitudes in the N400 time widow for incongruent relative to congruent prime-target trials. These effects were widespread through frontal and central sites for both hearing and CI groups and extended more parietally in CI-using children. Additionally, it was revealed through an exploratory analysis that this mismatch effect appeared earlier in a time window typically associated with the P2 response (150–275 ms) and persisted in both younger and older children with CIs relative to hearing controls. Early differences in response to congruent and incongruent stimuli for CI children have been shown in previous studies (Bell et al., 2019; Kallioinen et al., 2016). The N400 effects in the present study are similar to those of Bell et al. (2019) and Kallioinen et al. (2016) in that we found that both hearing and CI-using children demonstrate a mismatch effect to semantically incongruent pairs of words and pictures. Bell and colleagues, however, did not find that the two groups differed in their neural patterns of responses to incongruent stimuli; in other words, the two groups appear to process mismatched stimuli in a similar fashion. The present finding that CI mismatch effects are enhanced for the CI group is more similar to the results of Kallioinen et al. (2016), where CI subjects had larger mismatch effects than controls in reponse to between-semantic category mismatches, similar to those used in the present study. It should be noted that there are several methodological and subject characteristics differences from these two previous studies and the present work, making it difficult to draw direct comparisons between existing literature and our findings. Importantly, previous studies have not utilised audiovisual word stimuli in the way that the present study does. More research is needed to fully understand what role the addition of visual information plays during semantic integration in picture-word priming paradigms. We also note that the interstimulus intervals across these studies was highly variable. Bell et al. (2019) reported that pictures appeared immediately after spoken word offset (no ISI) whereas Kallioinen et al. (2016) state that pictures followed spoken words after 2.3 s. The present study has an ISI in between these of 400 ms, with an intertrial interval of 1300 ms. Such differences in ISI have been known to affect contributions of automatic and strategic processing in the service of lexical access. It is also possible that the present study’s relatively large sample of CI-using children allowed us to detect subtle effects that studies with smaller samples were unable to detect.

We hypothesised that CI-using children who might struggle to perceived the prime stimuli may show an overall attenuation of the N400 effect. This hypothesis is not supported by the current data, which showed that CI-using children exhibited an increase mean amplitude of the N400 effects for semantic incongruity. The observed selective modulation of the N400 suggests that the use of CIs in congenitally deaf children permits access to spoken audio-visual primes that is sufficient to invoke mismatched responses.

We further hypothesised that if CI-using children had less differentiated semantic conceptual knowledge this might pose additional challenges for contextual integration between prime and target resulting in larger N400 effects. This hypothesis was partially supported as our data showed greater N400 effects for semantic mismatches. However, this account does not provide a ready explanation for why semantically congruent trials exhibited an N400 effect on par with the typically hearing controls.

Finally, we conjectured that attentional or strategic differences might differentiate N400 effects in CI users from typically hearing children. The presence of early effects in a P2 window may be an indication of such differences in our population. Early attention effects have been shown to modulate P2 responses in both the auditory and visual domains in children (Jonkman, 2006; Sanders et al., 2006). In the linguistics domain, Neville et al. (1993) reported a P250 labelled component that was sensitive to sentence-level semantic congruity that preceded N400 effects and was larger in younger subjects. A number of recent studies have reported early mismatch effects in lexical-semantic contexts in what we have termed here the P2 window (150–275 ms). The explanations for these early mismatch effects range from relatively passive bottom-up partial activation of orthographic or visual featural properties (Holcomb & Grainger, 2006; Wang et al., 2004), context-based pre-activation of lexical forms (Kim & Lai, 2012), to strategic active prediction of expected forms (Brothers et al., 2015). Other have reported these early effects as reflecting the beginning period of the N400 effect (Coulson et al., 2005; Henderson et al., 2011). While the present experiment was not designed to adjudicate between these competing accounts, we explore evidence that these early P2 window responses may index attentional effects.

As discussed below, the differential engagement of attentional processes, whether implicit or explicit, may provide an account of the specific increased N400 effect to semantic mismatches and not overall differences in N400 in congruent trials.

A conventional hypothesis holds that a poorer comprehender may allocate more attentional resources to lower-level processes than a good comprehender (Hunt et al., 1975; Perfetti & Lesgold, 1977). It has been well-established that the allocation of attentional resources has implications for patterns of activation in lexical semantic decision tasks. There is an existing behavioural literature which suggests that automatic and strategic attentional effects in lexical-semantic processing tasks may give rise to differential facilitation and interference effects. In a two-process theory (Posner, 1982; Posner & Snyder, 1975), semantic facilitation in lexical decision experiments has been attributed to an automatic priming mechanism, while a combination of automatic and conscious attention processes give rise to processing costs which manifest as interference effects (Posner, 1982; but see also Becker, 1982 for a different account). Borrowing from this account, if the CI-using children are making use of more attentional processing, they may exhibit facilitation on par with typically hearing children (who exhibit more automatic processing) yet show increased interference in cases of incongruent stimuli. A related account is offered in Neville et al. (1993) in their developmental study of N400 effects in sentence processing. They speculate that early (i.e. 300–500 msec.) and late (i.e. 500–800 msec.) contextual priming effects may reflect separate sources, related to attention/novelty and semantic integration respectively. The former is more characteristic in developmentally younger participants in their studies. The difference in engagement or attentional resources may reflect strategic differences in our populations. Kallioinen et al. (2016) present a similar logic with respect to group differences in attentional processes in order to account for their pattern of findings, in which CI-users show an exaggerated negativity on incongruent trials relative to hearing controls. These researchers speculate that given specific task demands, CI subjects may engage in a more active strategy than controls, despite the costliness associated with using this strategy. It remains unclear in Kallioinen et al. (2016) and in the present study whether strategic processes reflect explicit or implicit operations. Consideration of demographic variables adds further insights into these effects.

In the present data we further explored the effects of demographic variables in our cohorts. In our data, early mismatch effects measured in the P2 window were sensitive to chronological age, in CI-using children but not in controls. It is interesting to note that the direction of the effect indicates that older CI-using children are showing an increased mismatch effect in the P2 window. Developmental work examining N400 effects generally show that these effects decrease with age. As suggested by Neville et al. (1993), this is interpreted as a decreasing reliance on context for word recognition and decreased effort required to integrate words into context. The present study shows the presence of mismatch effects in a window associated with the P2 in our CI population and increased in mismatch effect as a function of Chronological Age and Time in Sound. This may indicate that CI-using children become more adept with the use of a CI such that there is an increased reliance upon contextualisation to process to word forms. Recent behavioural evidence from Holt et al. (2021) supports the idea that school-aged children with hearing loss who make use of hearing aids or CI can use context to predict the end of sentences as quickly and accurately as hearing controls (Holt et al., 2021). This dependence upon context may include both reliance on audio-visual cues to comprehend word primes but also reliance on semantic integration to confirm initial processing. We conjecture that in cases where contextual integration between a lexical form and a semantic concept is confirmatory (in cases of congruent prime-target trials), co-activation of a lexical word form and a semantic representation (even if impoverished) is sufficient to yield a match. However, in cases of disconfirmation (such as incongruent prime-target trials), the rectification of competing lexical and semantic-conceptual activations may require additional active inhibition, and thus would incur greater processing costs. In the present experiment we hold that these processing costs may, in part, be reflected in the greater N400 effects for incongruent semantic trials.

Finally, we note that independent measures of receptive vocabulary did not interact with mismatch effects. This finding appears partially consistent with findings of Henderson et al. (2011) who failed to find relation between the magnitude of N400 effects and vocabulary knowledge in 8–10 years old.

Future studies are warranted to better identify the nature of the influence that attention and strategy may have on CI-using children’s semantic processing. A behavioural requirement to overtly respond to matching and mismatching trials may increase sensitivity and insure a more uniform attentional engagement with the task. A longer paradigm that permits trial by trial analysis may permit insights into strategic differences across the course of the experiment. Moreover, the age-related effects demonstrated in our findings should be further explored in future studies that make use of a more nuanced approach than a median-split of age, as used presently. Finally, it should be noted that there is much heterogeneity in the present study’s sample and in the wider population of CI-using children, with respect to factors such as age at implantation, Time in Sound, and hearing loss aetiology. The present results may not generalise to all CI using children, and further work is needed to understand what influence these demographic factors may have on neural processes underlying semantic integration.

The present study provides overall evidence that deaf children with cochlear implants demonstrate semantic integration effects that are observed in typically hearing cohorts. However, the exaggerated semantic incongruency effects raises the possibility that these children may utilise different strategic and attentional processes during a lexical priming task. Notably, these effects begin in an early window associated with the P2 and attention, and these early effects are largest in older CI-using children. More work will be needed to understand how differences in early audition may causally affect lexical-semantic processing and how the patterns of responses seen in paediatric and special populations map onto current controversies regarding neural mechanisms underlying the N400 effects. A further understanding of these mechanisms and processes may lead to more targeted language interventions for this population.

Acknowledgements

We are grateful to Kayla Vodacek, Tarah Schively, and Journie Dickerson for their assistance in creating stimuli and testing participants. We would like to thank the children and families and staff who participated in this research and especially the CCHAT Center, Sacramento, CA, USA; Weingarten Children’s Center, Redwood City, CA, USA; and the Hearing Speech and Deafness Center, Seattle, WA, USA.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy and ethical restrictions.

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This research was supported by National Institute on Deafness and Other Communication Disorders [grant number NIDCD R01DC014767] awarded to David P. Corina, PhD.

Notes

1 The present paper only discusses responses to the English word forms and their associated pictures.

2 These data are not reported here.

3 Three of the CI participants completed comparable receptive vocabulary questionnaires (OWLS-2, CASL-2) that were administered by their schools.