Processing of indexical information requires time: Evidence from change deafness

Abstract

Studies of change detection have increased our understanding of attention, perception, and memory. In two innovative experiments we showed that the change detection phenomenon can be used to examine other areas of cognition—specifically, the processing of linguistic and indexical information in spoken words. One hypothesis suggests that cognitive resources must be used to process indexical information, whereas an alternative suggests that it is processed more slowly than linguistic information. Participants performed a lexical decision task and were asked whether the voice presenting the stimuli changed. Nonwords varying in their likeness to real words were used in the lexical decision task to encourage participants to vary the amount of cognitive resources/processing time. More cognitive resources/processing time are required to make a lexical decision with word-like nonwords. Participants who heard word-like nonwords were more likely to detect the change when it occurred (Experiment 1) and were more confident that the voice was the same when it did not change (Experiment 2). These results suggest that indexical information is processed more slowly than linguistic information and demonstrate how change detection can provide insight to other areas of cognition.

Keywords:

• Change deafness
• Indexical information
• Lexical information

Studies of change detection, change blindness, and change deafness—the (in)ability of participants to detect visual or auditory changes in the environment—have increased our understanding of attention-, perception-, and memory-related processes. In the present experiments, we demonstrate how the change detection phenomenon can be used to examine other cognitive processes—namely, the processing of linguistic and indexical information in spoken words—by combining a change-detection task with a lexical decision task (a commonly used task in psycholinguistics). Linguistic properties convey phonological, semantic, and syntactic information about a spoken word, whereas indexical properties convey information about the age, gender, and so on of the speaker (Abercrombie, 1967). The speech signal simultaneously conveys both pieces of information to the listener, posing an interesting problem: How is each dimension of the signal processed?

One hypothesis states that attention is required to process each dimension of the speech signal (Mullennix & Pisoni, 1990). In this perspective, attention is viewed as a limited resource. Allocating a large amount of this resource to process one stimulus dimension leaves less of this resource to process another stimulus dimension, resulting in poorer performance in a task involving the stimulus dimension that has been allocated fewer resources. To test this hypothesis, Mullennix and Pisoni used a modified speeded-classification task in which participants categorized a spoken word in terms of the indexical information (gender of the speaker) or the linguistic information (initial consonant of the word). In both classification tasks, listeners experienced interference from the unattended dimension, suggesting that the processing of linguistic and indexical information required shared cognitive resources.

An alternative hypothesis proposed by McLennan and Luce (2005)—based on adaptive resonance theory (Grossberg, 1986)—suggests that indexical information is processed more slowly than linguistic information. In this framework, acoustic–phonetic input activates “chunks” in memory, or learned sets of associated features that vary in size (e.g., an individual feature, a phoneme, a word, etc.). The interaction of top-down information in memory with the bottom-up sensory information creates a resonant state, resulting in the recognition of that input.

Features and chunks that occur frequently establish resonant states more quickly than less frequently occurring features and chunks. McLennan and Luce (2005) asserted that because the abstract features found in linguistic information occur more frequently than the more specific features found in indexical information, resonance states form more quickly for linguistic than for indexical information. McLennan and Luce found support for the different time courses of processing linguistic and indexical information in three long-term repetition-priming experiments.

Relevant to the present study, McLennan and Luce (2005) encouraged participants to quickly process the stimuli in a lexical decision task by using nonwords with low phonotactic probability as foils. Nonwords with low phonotactic probability are rated less like real words in English (Vitevitch, Luce, Charles-Luce, & Kemmerer, 1997), enabling one to make a rapid “lexical decision”. With less time spent processing the stimuli, resonant states are formed only to the linguistic information in the input. To encourage participants to spend more time processing the stimuli in the lexical decision task, McLennan and Luce used word-like nonwords as foils. The greater similarity of such nonwords to real words requires more time to make a “lexical decision”. With more time spent processing the stimuli, resonant states are formed to the indexical information in the input.

In the present experiments we exploited the change-deafness phenomenon to further examine the processing of linguistic and indexical information. Similar to the change-blindness phenomenon (Simons & Levin, 1998), participants fail to detect (what appear to be) obvious changes to the speaker during an experiment (Vitevitch, 2003). It is important to clarify that the present study is not designed to test how or why change-deafness (or change detection in general) occurs; for such studies see Gregg and Samuel (2008), and Pavani and Turatto (2008). Rather, we are using the change-deafness paradigm to examine whether the processing of indexical information requires attentional resources (Mullennix & Pisoni, 1990) or additional processing time (McLennan & Luce, 2005). These two hypotheses make different predictions regarding performance in the change detection task.

In this experiment, participants engaged in a lexical decision task; halfway through the experiment a different speaker began presenting the stimuli. At the end of the experiment, participants were asked whether the same or two different speakers produced the stimuli. As in McLennan and Luce (2005), participants heard one of two types of nonwords in the lexical decision task: word-like or less word-like nonwords. If processing indexical information requires attentional resources (Mullennix & Pisoni, 1990), then, in the case of word-like nonwords, the lexical decision will require more processing resources than when making a lexical decision in the presence of less word-like nonwords. With more resources allocated to the lexical decision task in the word-like nonword condition, fewer resources will be available to process indexical information. Therefore, participants hearing word-like nonwords will be less likely to detect the change in the speaker than participants hearing less word-like nonwords.

Alternatively, if indexical information is processed more slowly than linguistic information (McLennan & Luce, 2005), then participants in the word-like nonword condition will spend more time processing the stimuli, allowing resonant states to form to the indexical information in the input. In this case, participants hearing word-like nonwords will be more likely to detect the change in the speaker than participants hearing less word-like nonwords (who spend less time processing the stimuli to make the lexical decision).

EXPERIMENT 1

Preliminary screening of talkers

Before describing the methods and results of the change deafness experiment, we describe the methods and results of a preliminary experiment used to screen the stimuli used in Experiments 1 and 2. Because the change deafness paradigm relies on the ability of listeners to distinguish between two voices, we first ascertained whether listeners could actually discriminate between the two voices used in the experiments. To test the ability of listeners to accurately distinguish between the two voices, a group of listeners participated in an AX-task in which the same word (or nonword) was presented with each instance separated by 50 ms of silence (e.g., “cat”—“cat”). Listeners were asked to indicate whether the voice producing the words (or nonwords) was the same speaker or two different speakers.

Method

Participants

Twenty participants enrolled in psychology courses at the University of Kansas took part in the preliminary screening of the stimuli for credit toward a course requirement. All participants reported that they were native-English speakers and had no history of speech/hearing disorders. No participant took part in more than one experiment reported here.

Materials

The monosyllabic stimuli examined in the prescreening AX-task and used in Experiments 1 and 2 consisted of 48 words that were highly familiar (based on ratings on a 7-point scale) and occurred relatively often in the language, 48 word-like nonwords, and 48 less word-like nonwords. The last phoneme of the 48 real words was changed to create the word-like nonwords. The less word-like nonwords were items with low phonotactic probability randomly selected from Vitevitch and Luce (1999; Appendix A). All of the stimuli used in the present experiments and summary information about those stimuli are provided here in Appendices A, B, and C of the current paper.

Two male speakers recorded all of the stimuli using a high-quality microphone at a normal speaking rate in an IAC sound-attenuated booth. The stimuli were recorded digitally using a Marantz PMD671 solid-state recorder at a sampling rate of 44.1 kHz. The sound files were edited using Sound Edit 16 (Macromedia, Inc.). The amplitude of the sound files was adjusted with the normalize function to amplify the words to their maximum value without clipping or distorting the sound and without changing the pitch of the words.

Speaker A had a fundamental frequency (F₀) of 130.03 Hz, and Speaker B had an F₀ of 125.40 Hz. The fundamental frequencies were calculated by finding F₀ from the vowel portion of 5 randomly selected stimuli from each type of stimulus (5 word, 5 word-like nonword, 5 less word-like nonword; the same items were selected for each speaker) for a total of 15 fundamental frequencies, which were then combined to obtain a mean F₀.

Four lists of 144 items were created. Each list contained all of the words and all of the nonwords (48 stimuli × 3 types of items = 144). The lists differed in that half of the words (and nonwords) were in the same voice, and the other half were in different voices. Same voice trials contained two different utterances of the stimulus by the same speaker. The items that were in the same/different voices were counterbalanced across the four lists, as were the voices that were the same.

Procedure

Participants in the preliminary screening task were seated in front of an iMac running PsyScope, which was used to randomize and present stimuli and to record responses. Participants heard a word or a nonword twice in a row (with an interstimulus interval, ISI, of 50 ms) over headphones and indicated with a button press whether the two items were said by the same or by different speakers.

Results and discussion

Accuracy rates indicating whether the same or two different speakers presented the stimuli were quite high. For the real words, participants correctly indicated that different speakers presented the words at a rate of 94.27%. For the word-like nonwords, participants correctly indicated that different speakers presented the items at a rate of 93.75%. For the less word-like nonwords, participants correctly indicated that different speakers presented the items at a rate of 93.75% (identical to the word-like nonwords).1 These results suggest that listeners could readily distinguish between the two voices, even though the fundamental frequency of the two speakers did not differ drastically. Furthermore, these results suggest that it is unlikely that any acoustic or phonetic differences that exist among the stimuli will differentially influence participants' ability to attend to or discriminate among the voices in the experiments that follow.

CHANGE DEAFNESS TASK

Method

Participants

Forty-four participants enrolled in psychology courses at the University of Kansas took part in the change deafness experiment for credit toward a course requirement. All participants reported that they were native-English speakers and had no history of speech/hearing disorders. Half of the participants (n = 22) were randomly assigned to the word-like nonword condition, and the remaining participants (n = 22) were randomly assigned to the less word-like nonword condition.

Materials

Four counterbalanced lists of 96 stimuli were constructed for the lexical decision task. Each list contained the same 48 real words, but two lists contained the word-like nonwords, and two lists contained the less word-like nonwords. The two lists containing the same type of nonwords differed in the order of talker. The words and nonwords that appeared in the first (and second) half of one list appeared in the first (and second) half of the remaining lists. The order of the items in the first (and second) half of each list appeared in a different random order for each participant.

The sound files used in Experiments 1 and 2 consisted of the utterance that was subjectively judged by the second author to be the “better” of the two utterances used in the preliminary screening experiment (based on amount of noise in the sound file, overall intelligibility, etc.). For the sound files used in this experiment that contained the words, Speaker A had a mean duration of 565 ms (SD = 91), and Speaker B had a mean duration of 594 ms (SD = 96); this difference was not statistically significant, t(94) = 1.52, p > .05. For the sound files used in this experiment that contained the word-like nonwords, Speaker A had a mean duration of 577 ms (SD = 101), and Speaker B had a mean duration of 616 ms (SD = 116); this difference was not statistically significant, t(94) = 1.75, p > .05. For the sound files used in this experiment that contained the less word-like nonwords, Speaker A had a mean duration of 635 ms (SD = 126), and Speaker B had a mean duration of 662 ms (SD = 131); this difference was not statistically significant, t(94) = 1.02, p > .05.

Procedure

The same equipment as that used in the preliminary screening was used in the lexical decision task. Participants heard a stimulus over headphones and indicated with a button press whether they heard a word or a nonword. Trials began with “Ready” appearing on the screen for 500 ms, followed by presentation of the stimulus. After the participant responded, the next trial began. Halfway through the experiment (Trial 49) the voice of the speaker changed. Upon completion of the lexical decision task participants were asked: (a) Did you notice anything unusual about the experiment? (b) Was the voice in the first half of the experiment the same that in the second half?

Results and discussion

Although detection of the change in speaker is the primary dependent measure in this experiment, we first wished to verify that manipulating the word-likeness of the nonwords in the lexical decision task influenced the amount of available cognitive resources or the amount of time spent processing the words. Therefore, we begin by reporting the reaction times to the words in the two nonword conditions.

One group of participants heard the words and the word-like nonwords, and another group of participants heard the same words and the less word-like nonwords. Participants hearing word-like nonwords correctly responded to the words more slowly (M = 1,029.95, SD = 180.51) than participants hearing less word-like nonwords (M = 947.71, SD = 121.67); t(42) = 1.77, p < .05, suggesting that participants in the word-like nonword condition spent more time processing the words than participants in the less word-like nonword condition.

Although participants hearing word-like nonwords spent more time processing the real words than the participants hearing less word-like nonwords, accuracy in their lexical decisions to the real words was equivalent, t(42) = 1.39, p = .1717 (word-like nonwords, M = 92.49%, SD = 4.2; less word-like nonwords, M = 90.63%, SD = 4.7), further suggesting that the manipulation of the nonwords in the lexical decision task was successful in influencing the amount of time spent processing the words, but did not adversely affect performance in other ways.2

Most importantly, participants who performed the lexical decision task with word-like nonwords were significantly more likely to detect the change in speaker (86%; 19 of 22 participants either stated that the voices were different in response to Question 1 or said “No” to Question 2) than those who performed the lexical decision task with less word-like nonwords (63%; 14 of 22 participants either stated that the voices were different in response to Question 1 or said “No” to Question 2; χ² = 3.93, p < .05). These findings suggest that the processing of indexical information may not be dependent on the allocation of processing resources (i.e., Mullennix & Pisoni, 1990). Instead, more time is required to process indexical information than linguistic information (McLennan & Luce, 2005).

EXPERIMENT 2

To further examine how indexical information is processed, we again asked participants to perform a lexical decision task with nonwords that were either word-like or less word-like. Instead of asking participants whether the talker changed during the experiment, we asked participants to rate their level of confidence on a scale from 1 to 10 that the voice did not change during the experiment. Importantly, and in contrast to Experiment 1, all of the participants heard the same voice throughout the entire experiment (half heard Speaker A, and half heard Speaker B). This variation in the change detection paradigm enabled us to examine how indexical information is processed using a different dependent measure.

If processing linguistic and indexical information requires processing resources, (Mullennix & Pisoni, 1990), then participants hearing word-like nonwords should be less confident that the voice did not change during the experiment (because few resources are left to process indexical information), whereas participants hearing less word-like nonwords should be highly confident that the voice did not change. Alternatively, if indexical information is processed more slowly than linguistic information (McLennan & Luce, 2005), then participants hearing word-like nonwords (i.e., more time processing the stimuli) should be highly confident that the voice did not change during the experiment, whereas participants hearing less word-like nonwords (i.e., less time processing the stimuli) should be less confident that the voice did not change.

Method

Participants

Thirty-six participants from the same population as that in Experiment 1 participated in this experiment. Half of the participants (n = 18) were randomly assigned to the word-like nonword condition, and the remaining participants (n = 18) were randomly assigned to the less word-like nonword condition.

Materials

The stimuli in Experiment 1 were used in this experiment.

Procedure

The equipment used in Experiment 1 was used in this experiment. In this experiment, only lists of words and nonwords that had the same speaker were used (i.e., the voices did not change). Also, participants in this experiment were asked to rate on a 10-point scale: How confident are you (1 = least sure, 10 = most sure) that the voice did not change during the experiment?

Results and discussion

Four participants were excluded from the analyses because their accuracy rate to the nonwords was below 75%, suggesting that they had a tendency to identify all of the stimuli as real words and perhaps were not properly engaged in the lexical decision task (3 participants in the word-like nonword condition, and 1 participant in the less word-like nonword condition). One additional participant (from the word-like nonword condition) was removed from the analysis for giving a confidence rating that was more than 2 standard deviations above the mean (note that the exclusion of this individual works against the predicted result).

To verify that manipulating the similarity of the nonwords to real words in the lexical decision task influenced the amount of time spent processing the words, we report the reaction times to the words in the two nonword conditions. Participants hearing word-like nonwords correctly responded to the words more slowly (M = 1,170.14, SD = 173.43) than participants hearing less word-like nonwords (M = 1,027.25, SD = 155.11); t(29) = 5.61, p < .05, suggesting that participants in the word-like nonword condition spent more time processing the words than participants in the less word-like nonword condition.

Although participants spent more time processing the real words in the word-like nonword condition than the less word-like nonword condition, participants were equally accurate in their lexical decisions to the real words, t(29) = 0.5369, p = .5948. In the word-like nonword condition, participants correctly identified the real words 92.97% of the time (SD = 3.71), and in the less word-like nonword condition, participants correctly identified the real words 93.75% of the time (SD = 5.02), further suggesting that the manipulation of the nonwords in the lexical decision task was successful in influencing the amount of time spent processing the words, but did not adversely affect performance in other ways.3

Given the difference in the amount of time processing the words in the lexical decision task, we predicted that listeners would be more confident that the voice remained the same in the word-like nonword condition than in the less word-like nonword condition. As predicted, participants hearing word-like nonwords (M = 7.50; SD = 2.03) were more confident that the speaker in the experiment did not change than the participants hearing less word-like nonwords (M = 6.12, SD = 2.20), t(29) = 1.80, p < .05.

The results of this experiment are consistent with those of Experiment 1. As in Experiment 1, the manipulation of the nonwords led participants to spend different amounts of time processing the words. Participants hearing word-like nonwords were more confident that the voice did not change during the experiment than were participants hearing less word-like nonwords. This significantly higher level of confidence suggests that indexical information is processed more slowly than linguistic information, as suggested by McLennan and Luce (2005).

GENERAL DISCUSSION

The change detection phenomenon has provided much insight regarding attention, perception, and memory.4 The present experiments demonstrate how this phenomenon can also be used to examine the cognitive mechanisms involved in the processing of linguistic and indexical information in spoken words. Specifically, Mullennix and Pisoni (1990) argued that limited cognitive resources must be allocated to process one type of information, resulting in detrimental performance in the processing of the other type of information, whereas McLennan and Luce (2005) argued that indexical information is processed more slowly than linguistic information. The results of Experiments 1 and 2 are consistent with the hypothesis proposed by McLennan and Luce: Indexical information is processed more slowly than linguistic information.

Although we conclude that indexical information is processed more slowly than linguistic information, McLennan and Luce (2005) discussed the possibility that depth of processing (e.g., Lockhart, Craik, & Jacoby, 1976) might also be involved in the processing of indexical information. In contrast, we do not believe that depth of processing provides a viable alternative account of how indexical information is processed, nor do we believe that it adequately accounts for the present results. Our conviction in this matter comes from the critique of the depth (or levels) of processing account by Eysenck (1978; and others), who argued, among other things, that the concept of “depth” is vague, ill-defined, and circular in nature, and that the approach offers only a description of the data, not an explanation of the data. Alternatively, the temporal framework proposed by McLennan and Luce (2005) is based on a computational model (i.e., adaptive resonance theory; Grossberg, 1986), which offers a testable and falsifiable mechanistic account. Indeed, adaptive resonance theory is well supported by data from computer simulations and experimental tests of those computational predictions (for a brief review, see Carpenter & Grossberg, 2003). The results of the present studies, therefore, provide converging evidence for the framework proposed by McLennan and Luce (2005).

Combining the change-deafness and lexical decision tasks, as in the present experiments, might also provide important insights about the cognitive mechanisms that are impaired in phonagnosia (Van Lancker, Cummings, Kreiman, & Dobkin, 1988), a neurological condition characterized by an impaired ability to recognize and discriminate voices. If patients with phonagnosia have intact linguistic processing and only show impairments in processing indexical information, then patients with phonagnosia should be influenced by the manipulation of the nonwords in the lexical decision task, as was observed in the present experiment. However, patients with phonagnosia would not differentially detect the changes in speaker (if at all) in the two nonword conditions, as was observed in the present experiment. We further believe that the change detection phenomenon—when combined with more commonly used tasks—can provide important insights into other areas of cognitive processing in addition to attention, perception, memory, and, as demonstrated in the present experiments, spoken language processing.

Acknowledgments

This research was supported by a grant from the National Institutes of Health (NIH) to the University of Kansas through the Schiefelbusch Institute for Life Span Studies (NIDCD R01 DC 006472). The experiments in this report partially fulfilled the requirements for a Master's degree in Psychology awarded to A.D. We thank the members of the committee (Andrea Greenhoot and Joan Sereno) for their comments and suggestions.

Notes

¹ For the sake of completeness we report that for the real words, participants correctly indicated that the speaker was the same at a rate of 98.77%. For the word-like nonwords, participants correctly indicated that the speaker was the same at a rate of 97.44%. For the less word-like nonwords, participants correctly indicated that the speaker was the same at a rate of 97.50%.

² Only for completeness do we report the speed and accuracy with which the nonwords in each condition were responded to. Word-like nonwords were correctly responded to with a mean = 1,247.36 ms (SD  = 384.65) and a mean accuracy = 81.2% (SD = 6.3). The less word-like nonwords were correctly responded to with a mean = 1,086.78 ms (SD = 389.73) and a mean accuracy = 90.4% (SD = 5.1).

³ Only for completeness do we report the speed and accuracy with which the nonwords in each condition were responded to. Word-like nonwords were correctly responded to with a mean = 1,328.95 ms (SD = 704.11) and a mean accuracy = 81.6% (SD = 10.2). The less word-like nonwords were correctly responded to with a mean = 1,089.29 ms (SD = 388.25) and a mean accuracy = 92.3% (SD = 7.8).

⁴ For an interesting application of the change detection paradigm to objectively evaluate the prominence of various elements of package labels, see Bix, Kosugi, Bello, Sundar, and Becker (2010).

APPENDIX A

Stimuli used in the experiments

Words		Word-like nonwords		Less word-like nonwords
badge	leaf	b@f	lit	CeJ	SYm
bag	lick	b@l	lIg	CeZ	SYp
cab	limb	k@k	lIS	D^d	SYv
cape	mad	kel	m@l	D^g	T^C
cheek	mat	Cil	m@b	D^S	T^S
chick	mice	CId	mYg	DiS	T^z
cuff	neck	k^l	nEn	DYb	Teg
deck	noon	dEs	num	DYp	TeT
dice	pace	dYp	pek	DYz	TeZ
dock	pad	dab	p@f	geZ	Tez
fig	pen	fIk	pEd	giD	TRT
goal	phone	gon	fod	giJ	TRz
gum	pin	g^k	pIr	giT	w^S
hair	rain	hek	rem	nRg	wYD
ham	reef	h@n	rin	nRz	y^C
haze	ring	h@f	rIl	nYD	y^S
hill	rope	hIn	rok	S^C	y^z
hood	sail	hUn	s@b	Seg	yiD
jack	seal	J@n	sib	SeJ	yig
kid	tack	kIz	t@s	SeT	yiS
knob	tool	nap	tuk	SRg	ziC
lab	vine	l@J	vYk	SRz	ziD
lake	wick	lel	wId	SYb	ziJ
lash	wine	l@G	wYk	SYD	ziS
Note: The last phoneme of each real word was changed to form a word-like nonword. The less word-like nonwords were randomly selected from the items with low phonotactic probability in Vitevitch and Luce (1999; Appendix A). All nonwords are phonologically transcribed using a convention based on standard keyboard characters (see Vitevitch & Luce, 2004).

APPENDIX B

Characteristics of the words

Familiarity	6.91 (0.183)
Frequency of occurrence	26.75 (24.84)
Neighbourhood density	19.27 (5.31)
Neighbourhood frequency	240.64 (320.05)
Sum of the phones	0.1599 (0.0421)
Sum of the biphones	0.0080 (0.0045)
Note: Mean values are shown, with standard deviations in parentheses.

APPENDIX C

Characteristics of the nonwords

	Word-like nonwords	Less word-like nonwords
Sum of the phones	0.1623 (0.0403)	0.0565 (0.0121)
Sum of the biphones	0.0083 (0.00534)	0.0009 (0.00006)
Note: Mean values are shown, with standard deviations in parentheses. Not surprisingly, the differences in the sum of the phones and the sum of the biphones between the word-like nonwords and less word-like nonwords are statistically significant. For the sum of the phones, t(94) = 17.71, p < .0001. For the sum of the biphones, t(94) = 9.47, p < .0001. The phone and biphone values were obtained from the website described in Vitevitch and Luce (2004).