Cochlear Implant Users Experience the Sound-To-Music Effect

ABSTRACT

The speech-to-song illusion is a robust effect where repeated speech induces the perception of singing; this effect has been extended to repeated excerpts of environmental sounds (sound-to-music effect). Here, we asked whether repetition could elicit musical percepts in cochlear implant (CI) users, who experience challenges with perceiving music due to both physiological and device limitations. Thirty adult CI users and thirty age-matched controls with normal hearing (NH) completed two repetition experiments for speech and nonspeech sounds (water droplets). We hypothesized that CI users would experience the sound-to-music effect from temporal/rhythmic cues alone, but to a lesser magnitude compared to NH controls, given the limited access to spectral information CI users receive from their implants. We found that CI users did experience the sound-to-music effect but to a lesser degree compared to NH participants. Musicality ratings were not associated with musical training or frequency resolution, and among CI users, clinical variables like duration of hearing loss also did not influence ratings. Cochlear implants provide a strong clinical model for disentangling the effects of spectral and temporal information in an acoustic signal; our results suggest that temporal cues are sufficient to perceive the sound-to-music effect when spectral resolution is limited. Additionally, incorporating short repetitions into music specially designed for CI users may provide a promising way for them to experience music.

KEYWORDS:

• Sound-to-music effect
• speech-to-song illusion
• cochlear implants
• music

Introduction

From a young age, speech and song are easy to distinguish from one another. However, the speech-to-song illusion illustrates that this seemingly categorical distinction can also be manipulated (Vanden Bosch der Nederlanden et al., 2022). This illusion is a unique perceptual transformation where a spoken phrase is first experienced as it was originally intended (as speech), and with repetition, listeners experience the same recording instead as if it had been sung (Deutsch et al., 2011). The speech-to-song effect was originally reported with the phrase “sometimes behave so strangely” (Figure 1(a); Deutsch, 1995, 2003), and it has since been replicated with other English phrases (Rowland et al., 2019; Tierney et al., 2013, 2018a, 2018b), in several other languages (Falk & Rathcke, 2010; Falk et al., 2014; Groenveld et al., 2020; Rathcke et al., 2021), and even in languages unfamiliar to the listener (Jaisin et al., 2016; Margulis et al., 2015). Because the speech-to-song illusion is widely experienced across many cultures and languages, it can provide a unique window into understanding how auditory and cognitive systems generate music from non-musical inputs.

Figure 1. An example of how cochlear implants process speech. We processed an original stimulus file from Deutsch 2003 (A) through a cochlear implant simulation (Litvak et al., 2007) and plotted the respective waveforms (B) and spectrograms of component frequencies over time (C) within the range of CIs. An enhanced view of the word “so” (green) illustrates the similarity between acoustic (D) and CI processed (E) samples for high frequency consonants like/s/while low frequency vowels like/o/lack the regularly spaced bands corresponding to formants and have a coarse representation of pitch. Without clear spectral resolution of harmonics or timbre, music perception through a cochlear implant is impeded and overall enjoyment tends to be low. Audio files used in this figure may be freely downloaded from our OSF repository (https://osf.io/dujt3/). Original file – clip_Figure1.wav; CI processed file – clipVocoded_Figure1.wav.

The speech-to-song illusion is not dependent on musical training (Vanden Bosch Der Nederlanden et al., 2015), and perceivers have highly stable musicality responses (Tierney et al., 2018a) that are difficult to unhear once perception of the illusion occurs (Groenveld et al., 2020). Prior work has also found that acoustic properties of speech such as pitch contour can be manipulated to make a speech stimulus sound more or less musical with repetition (Falk et al., 2014; Tierney et al., 2018a). We refer to musicality ratings as the perceived song-like or music-like quality of repeating sounds (e.g., as in Simchy-Gross & Margulis, 2018; Tierney et al., 2021). In the present study, we questioned whether repetition could elicit musical percepts in a group of individuals known to experience difficulties with music perception and appreciation in their daily lives – cochlear implant users (Drennan et al., 2015; Gfeller et al., 2012; Kang et al., 2009).

A cochlear implant (CI) is a neuroprosthetic that enables a sense of hearing for individuals with significant sensorineural hearing loss. These devices transduce sounds into a series of electrical pulses that are sent by radio frequency to an internal receiver and are ultimately transmitted to an electrode array that is surgically implanted in the cochlea. There, electrical current is applied according to the tonotopic mapping of the cochlea (i.e., from high to low frequencies, spiraling inward; Figure 1(c)) to trigger action potentials in otherwise latent auditory nerve fibers. The place of stimulation along the cochlea conveys a rudimentary sense of pitch-one that is both sparse and frequency-shifted in comparison to normal hearing.

Several factors limit the spectral resolution of sounds heard with a CI, the effects of which are detrimental for music perception. One factor is the interaction between neighboring electrodes that results from the spread of electrical excitation within the fluid-filled cochlea – commonly referred to as channel interaction. In typical acoustic hearing, thousands of auditory hair cells mechanically transduce sound waves in a precise tonotopic mapping. CIs must bypass these damaged cells to directly stimulate primary auditory neurons. As a result, current spreads through the surrounding fluid. This can effectively reduce a 22-electrode array to only a fraction of channels that are actually functional (Berg et al., 2019, 2021; Friesen et al., 2001; Gifford et al., 2022). Thus, the nature of the electrode-neural interface remains a limiting factor for the number of useable channels. For low frequencies, this means that a range of over a hundred Hertz is mapped to a single electrode and up to 1,000 Hertz for the highest-frequency electrodes (Ali et al., 2015). Not surprisingly, this affects frequency discrimination and music perception for CI users who require, on average, three semitones (Drennan et al., 2015; Fujita & Ito, 1999; Kang et al., 2009) or sometimes as much as a full octave to discriminate different musical pitches (Gfeller et al., 2002). In contrast, listeners with typical acoustic hearing can distinguish frequency differences well below one semitone. Because musical melodies typically contain smaller intervals than many CI listeners can differentiate between, even simple, monophonic tunes can be difficult for CI users to identify (Drennan et al., 2015).

Another factor that affects music perception with a CI is how sounds are encoded by implant processors. All commercially available implants prioritize temporal information in low modulation frequencies (i.e., below 300 Hz) that correspond to the slow changes in the temporal envelope, or the overall outline of a sound wave (Figure 1(b)). That is, current CI systems extract the frequency-specific temporal envelope corresponding to each electrode’s bandpass filter. This approach works well for conveying speech, though it discards faster modulations corresponding to temporal fine structure. Important musical qualities like pitch and timbre rely on temporal fine structure, which are poorly conveyed through an implant. Even experimental CI technologies attempting to preserve temporal fine structure (Riss et al., 2014) still prove challenging due to physiological limitations (Oxenham et al., 2004). In sum, the sound envelope is most important for speech perception and fine structure for pitch perception (Smith et al., 2002), and CIs more successfully preserve the former.

A CI can afford remarkable speech perception abilities in quiet based primarily on temporal encoding of speech envelopes; however, the addition of even moderate background noise like neighboring talkers can drastically reduce understanding (Dunn et al., 2020; Gifford et al., 2018). For the same reason, two musical sounds with the same envelope (i.e., matching attack, duration, and decay) can also be rendered indistinguishable through CI processing despite having quite different temporal fine structures that correspond to say, a snare drum and a flute. Unsurprisingly, many CI users report reduced satisfaction with music listening following CI activation (Riley et al., 2018). Furthermore, rhythm – encoded by the temporal envelope of an acoustic signal rather than temporal fine structure – is more easily perceived by CI users. For this reason, CI users generally perform similar to normal hearing (NH) controls on rhythm tasks (Brockmeier et al., 2011; Kim et al., 2010; Limb & Roy, 2014) and prefer music with salient rhythms (Buyens et al., 2014, 2018).

To illustrate the differences between acoustic hearing and cochlear implant processing in the context of the speech-to-song effect, we plotted the waveforms of a normal, unaltered spoken phrase from (Deutsch et al., 2011) (Figure 1(a)) and the same stimulus filtered through a cochlear implant processing algorithm (Litvak et al., 2007); both sound files can be downloaded from our Open Science Framework (OSF) repository (https://osf.io/dujt3/). The waveforms of the unfiltered stimulus versus the filtered stimulus are quite similar (Figure 1(b)), illustrating the high temporal fidelity of cochlear implants. In contrast, the low spectral resolution of CIs becomes more apparent when considering the frequency composition of the spectrogram (Figure 1(c)). In particular, the visualization of the word “so” illustrates the initial broad-spectrum frequencies of a consonant followed then by horizontal bands of formants corresponding to a resonant vowel (Figure 1(d), acoustic). While the fundamental frequency of the vowel is evident in a dark band on the CI-processed spectrogram (Figure 1(e), electric), the formants are not. In effect, CI processing makes vowels appear more like consonants in a spectrogram and similarly obscures musical elements like harmonics and timbre.

The underlying acoustics that differ between speech and song are subtle (Vanden Bosch der Nederlanden et al., 2022) and may not be perceptible after CI implantation. Though bottom-up, peripheral encoding with CIs is limited, successful music and song perception may still be possible for CI users via top-down perceptual mechanisms such as repetition. This is similar to how top-down mechanisms such as semantic content can facilitate speech recognition (e.g., Moberly & Reed, 2019). Assessing the sound-to-music effect in CI users is critical for understanding what types of cues (e.g., temporal, spectral, repetition-based) contribute to music perception, and if the auditory system in individuals with sensorineural hearing loss is able to make sense of a sound-to-music effect (i.e., generating musicality from repetition of different types of acoustic input).

A consideration for whether CI users could experience a perceptual transformation to song comes in part from prior studies identifying a broader “sound-to-music effect” for non-speech sounds. A wide range of sounds seem more musical upon repetition, including randomly generated and complex tones (Margulis & Simchy-Gross, 2016), a range of environmental sounds (Simchy-Gross & Margulis, 2018; Tierney et al., 2018b) including water droplets (Rowland et al., 2019), and percussive cross-sticks aligned with speech rhythms (Rowland et al., 2019). It seems to be a broader property of the auditory system that once a stimulus is looped over and over, without being altered or harmonically transposed during the intervening repetitions (Deutsch et al., 2011), many stimuli start to be perceived as more musical. And, though both melody and rhythm play a role in the effect (Falk et al., 2014), the studies mentioned above suggest that rhythm alone may be sufficient to elicit repetition-induced musicality.

We hypothesized that CI users would experience the sound-to-music effect based primarily upon the well-preserved temporal cues of cochlear implants that convey rhythm. However, we also hypothesized they would experience the effect at a lower magnitude compared to NH controls, given controls’ greater access to melodic cues also contributing to the effect.

Methods

Participants

Thirty CI users and thirty age-matched listeners with normal hearing (NH) were included in this study (see Table 1 for participant demographics). This study was approved by the Vanderbilt University Institutional Review Board (IRB). CI users were recruited from the Vanderbilt Cochlear Implant Lab database and the Vanderbilt Bill Wilkerson center, while NH control participants were recruited from the community. Seven CI users and eight NH controls reported formal musical training (more details in Music experience survey section).

Table 1. Participant demographic data.

	CI users	NH controls
N	30	30
Sex n (% female)	20 (67%)	24 (80%)
Mean Age in years	56.5 ± 10.8	50.8 ± 10.4
Number of CIs	one: n = 22 two: n = 8	—
Residual acoustic hearing	18 (30%)	—
Duration of Hearing Loss	14.7 years ±13.54	—
Implant Manufacturer	AB − 53% Cochlear − 43% MED-EL − 3%	—
Musical Training	n = 7	n = 8

Participants from both groups were included in the study if they self-reported 1) no major neurological or psychiatric conditions and 2) English as a first language. Because two CI users self-reported neurological histories (n = 1 stroke 8 years prior, n = 1 medicated epilepsy), we conducted our main analyses both with and without these participants in the models. Additional inclusion criteria included a hearing screening for NH controls requiring pure tone thresholds <20 dB HL in both ears for octave frequencies 250–8000 Hz. CI users had to have postlingual onset of moderate-to-profound hearing loss. An additional two CI users and three NH controls completed the study but were excluded from all analyses due to technical difficulties (n = 3), prelingual deafness (n = 1 CI user), and an abnormal hearing screening (n = 1 control).

For all CI users, moderate-to-profound hearing loss had occurred after 2 years of age, and, for the 27 CI users for whom this data was available, the mean duration of hearing loss was 14.7 years (SD = 13.54, range = 1–47 years). Here, we used the duration of significant hearing loss (i.e., when participants reported difficulty communicating over the phone, or sudden/profound loss) which we attained through a mixture of self-report and clinical records. Twenty-two individuals had unilateral CIs and eight had bilateral CIs. For 13 individuals who had residual hearing in the non-implanted ear (contralateral to the CI), we removed any hearing aids and occluded this ear during testing with a foam earplug so that the experiments were completed with CI-only listening. Six CI users, however, had residual acoustic hearing in the implanted ear(s) yielding a form of “hybrid hearing” that involves combined electric-acoustic stimulation, or EAS. These six CI users did have access to some – typically low frequency – acoustic cues during experimental testing.

Frequency Discrimination Task

To account for the role of frequency perception in the speech-to-song illusion, we designed a 3-interval, 2-alternative forced-choice (2AFC) frequency discrimination task. In each trial, participants were presented with three intervals: one reference interval of a 440 Hz pure tone (300-ms duration, 0.5 interstimulus interval) and two other intervals (one of which was a different frequency). Participants indicated which of the remaining two intervals sounded different from the others by selecting the corresponding interval number (“2” or “3”) on a touchscreen. The experiment began with a large target increment of 200 Hz (e.g., 640 Hz on the first trial) and narrowed via an adaptive staircase procedure (2-down, 1-up). There were 12 total reversals. Steps were initially 50% for the first four reversals, 20% for the next four reversals, and 5% for the last four reversals. Threshold was calculated as the geometric mean of the last eight reversals. The task was programmed in MATLAB 2018a and took approximately 4 minutes to complete.

To avoid the undue influence of outliers in subsequent analyses, we winsorized frequency discrimination thresholds separately for each group for values below the 5 or above the 95%ile cutoffs using the winsor function in MATLAB.

Standardized Frequency Discrimination Task for CI Users: UWCAMP

For a clinical measure of frequency discrimination, CI users also completed a standardized task: the pitch discrimination subset of the University of Washington’s Clinical Assessment of Music Perception (UWCAMP) (Kang et al., 2009). In this 2AFC task, participants identify the complex tone that is higher in pitch. The task is administered in three interleaved adaptive tracks (1-up and 1-down tracking procedure) for three base frequencies (262, 330, 391 Hz), corresponding to the notes C4, E4, and G4. This task returns a threshold at 75% accuracy on a psychometric function. For consistency, we also winsorized these thresholds.

Main Repetition Experiments: Stimuli and Task

We tested 14 speech stimuli (English phrases and sentences) and 14 environmental (water) stimuli developed by Rowland et al. (2019). Speech stimuli ranged from 0.739 to 2.592 seconds long and water stimuli ranged from 0.700 to 2.870 seconds long. Sound files and deidentified data are available on Open Science Framework (https://osf.io/dujt3/).

Using similar experimental methods to (Rowland et al., 2019) (Figure 2), we asked participants to listen to a single presentation of an audio clip, after which they made a “pre-repetition” perceptual rating in response to the question “How much does this sound like speech or singing” (speech condition) or “How much does this sound like music” (water condition). Participants responded by clicking along a horizontal line with the endpoints “exactly like speech/exactly like singing” (speech condition) or “not at all like music/exactly like music” (water condition); this scale represented a 0–1 continuum. The stimulus was then looped 16 times while participants continuously responded to the same question on the same scale; these continuous responses were collected following the protocol of (Rowland et al., 2019) but were not analyzed in the present study. After the 16 loops, participants provided a second “post-repetition” rating. Four practice trials for each condition confirmed participant understanding prior to the start of each experiment. We administered the water condition followed by the speech condition for all participants with the exception of one CI user who, due to technical difficulties, only completed the speech condition. As in (Rowland et al., 2019), our analyses focus on the discrete pre- and post-repetition responses.

Figure 2. Schematic of the trial structure for the speech condition. Figure reproduced with permission from (Rowland et al., 2019).

Qualitative Feedback Survey

Following each experiment (speech, water), participants answered the brief question, “In this section of the experiment, would you say that you typically perceived: a) a rhythm, b) a melody, c) both rhythm and melody, or d) none of the above?” Responses were coded on a binary scale with the variables “Rhythm” and “Melody.” For example, if a participant responded “both” in a given condition, they were assigned a “1” for Rhythm and a “1” for Melody. Conversely, if a participant responded “neither” they were assigned a “0” for both variables.

Music Experience Survey

Last, all participants completed an online music experience survey with select questions from the Munich Music Questionnaire (Brockmeier et al., 2000) and the Ollen Musical Sophistication Index (Ollen, 2006). The survey consisted of three sections: 1) general questions related to music listening and importance of music in daily life, 2) experience with music with a cochlear implant (CI group only), and 3) musical training. For analyses, we used a binary scale (−1 = non-musician, 1 = musician) which was based on the yes/no survey question: “Do you currently consider yourself a musician?” All NH controls (n = 30) and all but one CI participant (n = 29) completed the survey. Questions are in supplemental Table S1.

Procedure and Test Environment

This study was conducted in accordance with the guidelines and regulations of Vanderbilt University’s IRB. All participants provided written informed consent and were compensated for 1–1.5 h of their time. Stimuli were presented in stereo from two Yamaha Model HS8 powered loudspeakers, positioned at approximately 45° and 315° azimuth from participants who were seated in a sound-attenuated booth throughout testing.

Analysis

First, we analyzed ratings using linear mixed effects models with the fitlme function in MATLAB 2018b. Model comparisons were made using likelihood ratio tests and AIC/BIC comparisons (the compare function in MATLAB). Categorical variables were effect coded as follows:

GroupNH controls: −1CI users: 1

Conditionwater: −1speech: 1

Timepre-repetition: −1post-repetition: 1

We investigated effects from the main model using post-hoc tests, correcting for multiple tests using a Bonferroni correction for significance, and assessing for normality of variables using qqplot and the Anderson-Darling Test (adtest function in MATLAB). In cases where assumptions of normality were not met, we conducted non-parametric post-hoc tests (Wilcoxon signed rank or Mann–Whitney U-test); otherwise parametric t-tests were employed.

For both groups, we then fit separate linear mixed effects models with frequency discrimination threshold and musical experience as potential predictors of musicality ratings (i.e., post-repetition – pre-repetition = ΔRating). For the CI group, we explored whether the following variables were associated with ΔRating: duration of hearing loss, presence/absence of acoustic amplification during testing – as was the case for the 6 CI recipients with acoustic hearing preservation in the implanted ear(s) – and daily hearing condition (i.e., whether individuals had acoustic hearing available during daily life).

Lastly, we assessed qualitative feedback ratings using Fisher’s exact tests to assess group differences in the proportion of responses in each category (rhythm, melody).

Results

Replication of Rowland et al., 2019

To establish the validity of the current experiment, we compared the results from our NH controls (mean age = 50.8 years) to the younger participants in (Rowland et al., 2019) (mean age = 19–20 years old). In (Rowland et al., 2019), mean musicality ratings (i.e., ΔRating) were 0.26 in the speech condition and 0.47 in the water condition compared to 0.25 and 0.36, respectively, in the present study. Independent samples t-tests confirmed that, despite a 30+ year age difference, these two cohorts were not significantly different from one another (speech: t(58) = 0.14, p = 0.89; water: t(58) = 1.71, p = 0.092), suggesting that age does not affect the magnitude of the sound-to-music effect (Figure 3).

Figure 3. Comparison of results from (Rowland et al., 2019) to the older cohort of normal hearing participants from this study. We replicated results for both speech and water conditions with no significant differences between the two cohorts. Left figure panel reproduced with permission from (Rowland et al., 2019).

The Sound-To-Music Effect in Cochlear Implant Users

For speech stimuli, the mean musicality ratings (i.e., ΔRating) were 0.18 (SE = 0.025) for CI users and 0.25 (SE = 0.043) for NH controls in the present study (as noted above). For water stimuli, mean musicality ratings were 0.19 (SE = 0.032) for CI users and 0.36 (SE = 0.048) for NH controls (Table 2; Figure 4).

Figure 4. Comparison of pre-repetition and post-repetition ratings for CI users (red) and NH controls (purple) in the speech condition (left) and water condition (right). Perceptual ratings ranged from 0 (exactly like speech/not at all like music) to 1 (exactly like singing/exactly like music). Both groups experienced the sound-to-music effect (i.e., a positive slope). Error bars are standard error of the mean.

Table 2. Descriptive statistics for the repetition experiments. Mean (standard error) and 95% confidence intervals.

Group Condition – Time	M (SE)	95% CI
CI users
Speech – Pre	0.086 (0.017)	[0.051, 0.12]
Speech – Post	0.26 (0.026)	[0.21, 0.31]
Speech - ΔRating	0.18 (0.025)	[0.12, 0.23]
Water – Pre	0.18 (0.032)	[0.11, 0.24]
Water – Post	0.37 (0.040)	[0.28, 0.45]
Water - ΔRating	0.19 (0.032)	[0.12, 0.26]
NH controls
Speech – Pre	0.069 (0.018)	[0.031, 0.11]
Speech – Post	0.32 (0.046)	[0.22, 0.41]
Speech - ΔRating	0.25 (0.043)	[0.16, 0.33]
Water – Pre	0.15 (0.037)	[0.071, 0.22]
Water – Post	0.50 (0.056)	[0.39, 0.62]
Water - ΔRating	0.36 (0.048)	[0.26, 0.46]

The full main model (fit by maximum likelihood) included perceptual ratings (averaged over items) as the dependent variable, fixed effects for group (CI users, NH controls), condition (speech, water) and time (pre-repetition, post-repetition) as well as their interactions, and random intercepts for participant and participant varying by group:

model1 <- fitlme(data, Rating ~ 1+ Group*Condition*Time + (1|participant: Group) + (1|participant))

The main effect of repetition (i.e., Time) was significant ($\mathit{\beta }$ = 0.12, p < 0.001) and there was no main effect of group ($\mathit{\beta }$ = −0.019, p = 0.32). Post-hoc tests comparing pre-and post-repetition ratings, separately for each group and condition, confirmed that in all cases, post-repetition ratings were higher, suggesting that CI users and NH controls are able to experience the effect (CI users’ speech: Z = 4.39, p<0.001; NH controls speech: Z = 4.34, p < 0.001; CI users’ water: Z = 4.31, p < 0.001; NH controls water: Z = 4.68, p < 0.001; tests applied a Bonferroni correction for multiple tests). Ratings for both groups were greater for water compared to speech (main effect condition, $\mathit{\beta }$ = −0.057, p < 0.001). There was a significant group × time interaction, and post-hoc tests showed this was due to higher post-repetition ratings (across speech and water condition together) in the NH control group (t(117) = −2.12, p = 0.036). No other interactions were significant. See Table 3 for full results of Model1.

Table 3. Fixed effects results for main sound-to-music effect model (Model1).

	β estimate	SE	df	t	p	95% CI
Intercept	0.24	0.019	230	12.85	<0.001	[0.20, 0.28]
Group	−0.019	0.019	230	−1.00	0.32	[−0.055, 0.018]
Condition	−0.057	0.0099	230	−5.77	<0.001	[−0.076, −0.038]
Time	0.12	0.0099	230	12.29	<0.001	[0.10, 0.14]
Group * Condition	0.0090	0.0099	230	0.91	0.36	[−0.010, 0.029]
Group * Time	−0.030	0.0099	230	−3.02	0.0028	[−0.049, −0.010]
Condition * Time	−0.016	0.0099	230	−1.59	0.11	[−0.035, 0.0038]
Group * Condition * Time	0.012	0.0099	230	1.20	0.23	[−0.0076, 0.031]

Notes. Beta estimates are unstandardized.

Model1 provided a better fit to the data than a) a model without interactions amongst fixed effects (p = 0.0093), and b) a null model without any fixed effects, but with the same random effects structure as Model1 (p < 0.001). See Table 4 for model comparison statistics. We also ran a version of Model1 with covariates for age and gender, which were not significant predictors of Rating. Last, we ran a version of Model1 excluding the two CI users with neurological confounds, and found results were nearly identical to the full Model1. Therefore, all subsequent models were run with the full cohort of CI users (n = 30).

Table 4. Model fit statistics for the full Model1 and reduced versions of the model.

	AIC	BIC	Log likelihood
Full model Rating ~1+ Group*Condition*Time + (1|participant: Group) + (1|participant)	−122.53	−84.34	72.27
Reduced model Rating ~1+ Group+Condition+Time + (1|participant: Group) + (1|participant)	−117.09	−92.79	65.55
Null model 1+ (1|participant: Group) + (1|participant)	−1.68	12.21	4.84

As a complementary analysis to further understand musicality ratings between groups, we ran a model with the magnitude of the repetition effect, ΔRating, as the dependent variable. This differed from Model1 in that Model1 used absolute ratings as the dependent variable, whereas this analysis used difference values as the dependent variable. In both cases, ratings were averaged across all items in a given condition. In this analysis of ΔRating, NH controls had higher slopes than CI users (main effect of group, $\mathit{\beta }$ = −0.060, p = 0.0082). Again, we found a main effect of condition ($\mathit{\beta }$ = −0.031, p = 0.035) and no significant group × condition interaction ($\mathit{\beta }$ = 0.024, p = 0.10). Overall, the results from this model parallel those of Model1. See Table S2 in the Supplement for full model results.

To test whether there was a disproportionate subset of “non-perceiving” participants between the two groups, we analyzed individual data (Figure 5). Non-perceivers were participants with a mean ΔRating of 0.05 or less on the 0–1 continuous rating scale. In the speech condition, five CI users and nine NH controls met this criterion, as did eight CI users and six NH controls in the water condition. The number of non-perceivers was not significantly different between groups in either the speech condition (χ² = 1.49, p = 0.22) or water condition (χ² = 0.47, p = 0.49), suggesting a similar proportion of individuals who did and did not perceive the illusion.

Figure 5. Individual participant data for the repetition experiments (speech left, water right). Each participant’s post and pre-repetition ratings (averaged over all trials) are plotted in gray, with the group means overlayed in color (CI users in red, NH controls in purple). There is significant individual variability in the magnitude of the effect.

Frequency Discrimination

After winsorizing, frequency discrimination threshold summary statistics for the two groups were as follows: CI users: mean = 25.62 Hz (0.98 semitones), SE=5.03; NH controls: mean = 5.68 Hz (0.22 semitones), SE=2.11 (Figure 6). Not surprisingly, a Welch’s two-sample t-test indicated that CI users have significantly poorer frequency discrimination (i.e., higher thresholds) than NH controls (t(37.5)=3.66, p<0.001, CI: 8.90–30.98). Descriptive statistics for UWCAMP were as follows: 262 Hz condition (mean ± SD = 49.33 ± 37.64 Hz; mean = 2.99 semitones); 330 Hz condition (mean ± SD = 57.55 ± 74.53 Hz; mean = 2.78 semitones); 391 Hz condition (mean ± SD = 79.86 ± 94.06 Hz; mean = 3.22 semitones). These thresholds are largely consistent with previous UWCAMP data (Drennan et al., 2015; Kang et al., 2009).

Figure 6. Comparison of frequency discrimination thresholds, after winsorizing, for CI users (n = 29) and NH controls (n = 29). Each bar represents one participant’s threshold for distinguishing pitches above/below a 440 Hz reference tone (black line). CI users (red) had significantly higher thresholds than NH controls (purple; p<0.001).

We then fit separate models for the CI (model2) and NH (model3) groups, with frequency discrimination threshold as a fixed effect in the model and a random intercept for participant. The dependent variable in these models was ΔRating.

model2 <- fitlme(CI_data, ΔRating ~1+ Condition*Freq_Discrim + (1|participant))

model3 <- fitlme(NH_data, ΔRating ~1+ Condition*Freq_Discrim + (1|participant))

In the CI group there was neither an effect of frequency discrimination threshold ($\mathit{\beta }$ = −0.0014, p = 0.11) nor condition ($\mathit{\beta }$ = −0.011, p = 0.65) on ΔRating. In the NH control group, there was no effect of frequency discrimination on ΔRating, but there was a significant main effect of condition ($\mathit{\beta }$ = −0.067, p = 0.014) indicating slopes were higher for water compared to speech, consistent with our previous models. There were no significant interactions for either group (model results in Table S3). These results indicate that frequency resolution did not affect individuals’ perception of the sound-to-music effect.

Relationship Between CI users’ Clinical Factors and the Sound-To-Music Effect

Next, we explored whether clinical variables within the CI group were associated with the magnitude of the sound-to-music effect (i.e., ΔRating). This model was based on the twenty-six individuals who had all clinical data available.

model4 <- fitlme(CI_data, ΔRating ~1+ Condition + Duration Hearing Loss + Hearing Condition + Condition Tested + (1|participant))

None of these clinical factors – duration of hearing loss, presence/absence of acoustic hearing during experimental testing, presence/absence of acoustic hearing in daily life – were significantly associated with ΔRating, and condition (i.e., speech, water) was also not significantly associated with ΔRating. This suggests that clinical variability in acoustic hearing and duration of hearing loss did not impact musicality ratings (model results in Table S4).

Musical Experience (ME)

Seven participants in the CI group and eight participants in the NH group reported having musical experience (ME). ME was entered as a fixed effect in order to understand how musical experience influenced perceptual musicality ratings:

model5 <- fitlme(data, ΔRating ~1 + Group*Condition + ME + (1|participant:Group) + (1|participant))

There was not a main effect of musical experience ($\mathit{\beta }$ = 0.0066, p = 0.80) indicating that musicians and non-musicians alike perceived the sound-to-music effect. The main effect of condition was not significant ($\mathit{\beta }$ = −0.027, p = 0.062). Additionally, there was a main effect of group ($\mathit{\beta }$ = −0.060, p = 0.0089), indicating NH controls had higher slopes than CI users, and a marginally significant group × condition interaction ($\mathit{\beta }$ = 0.028, p = 0.051). For full model results, see Table S5. In addition to our binary scale, we created a cumulative score which included six questions from the musical experience survey, each normalized to a 0–1 scale. Results were almost identical when we used this more nuanced measure of musical experience. Overall, these results suggest that musicianship did not influence musicality ratings.

Qualitative Feedback Responses

In the speech condition, there were no group differences for the number of rhythm and melody qualitative ratings (OR = 0.72, p = 0.50). Similarly, the water condition had no group differences for the number of rhythm and melody responses (OR = 0.96, p = 1.00). These results suggest that participants in both groups were qualitatively experiencing similar percepts (Figure 7), which is consistent with their quantitative ratings (Figure 4). Because there were seemingly more “rhythm” responses for both groups in the water condition compared to speech, we did an exploratory Z-test for proportions which indicated that participants did perceive rhythmic cues to be more apparent than melody cues in the water condition (z=-6.12, p < 0.001). As a complementary approach, we then compared the proportion of total rhythm responses to a proportion of 0.5 (i.e., an assumed equal number of rhythm and melody responses) and again found rhythm cues to be more apparent (z = 4.33, p < 0.01). The number of rhythm and melody responses did not differ from one another in the speech condition (z = −1.58, p = 0.11) and did not differ when comparing to an assumed equal number of responses between the two (z = 1.12, p = 0.26).

Figure 7. Qualitative feedback ratings for speech condition (left) and water condition (right) for both CI users and NH controls. Following each experiment, participants answered the question, “would you say that you typically perceived: a) a rhythm, b) a melody, c) both rhythm and melody, or d) none of the above?” results indicate similar responses in both CI users and NH controls.

Experience with Music in CI Users

Because music satisfaction is typically decreased in CI users, we wanted to characterize this tendency in our cohort. When surveyed about music listening satisfaction, CI users reported a range of ratings from 1 to 10 on a scale of “very unsatisfied” to “very satisfied.” The median response was 4 (SD=2.65). After separating participants by whether they did (n = 17) or did not (n = 11) have residual acoustic hearing, we found that the median satisfaction ratings were 4 and 3, respectively (Figure 8).

Figure 8. Histograms of responses to questions in the music experience survey, provided by CI users. Participant numbers are included on each figure panel. Vertical dotted lines represent the median and both questions were on a 10-point likert scale. panel A. “how satisfied are you with how music sounds through your cochlear implant?” panel B. “Currently how often do you listen to music?” (red) and “before hearing loss, how often did you listen to music?” (purple).

Ratings for current time spent listening to music ranged from 1 (“never) to 10 (“often) and the median response was 7 (SD=2.85). Participants reported more time spent listening to music before their hearing loss began, with a median response of 9 (SD = 2.70) on the same scale (Figure 8). Overall, CI users reported more time spent listening to music prior to their hearing loss and low satisfaction with music after CI activation.

Discussion

Given the difficulty that CI users face with music perception, we questioned whether they could perceive musicality through repetition in the speech-to-song illusion, thereby circumventing device limitations for encoding spectral elements like pitch. In line with our hypothesis, we found that CI users experienced the musical transformation for both speech and non-speech (i.e., dripping water) stimuli, but at a lower magnitude compared to NH controls. We also found that frequency discrimination, musical training, and clinical variables within the CI group were not associated with musicality ratings. CIs provide a strong clinical model for disentangling the effects of spectral and temporal information in an acoustic signal, and our results suggest that intact rhythm perception is sufficient for experiencing this auditory illusion.

In our prior work, a deconstructed speech signal with either rhythmic or spectral content alone could elicit the sound-to-music effect, suggesting that repetition-induced musicality may be a broader principle of the auditory system (Rowland et al., 2019). Others have suggested that a possible mechanism for this illusion could be disinhibition of neural circuitry underlying pitch/melody perception (Deutsch et al., 2011). Based on the present findings, pitch information does seem to contribute to the magnitude of the sound-to-music effect, as the addition of spectral cues (i.e., available to NH controls but not to CI users) led to an increased magnitude, or slope, of musicality ratings. However, since CI users also experienced the sound-to-music effect, temporal/rhythmic cues do seem sufficient to perceive the effect. The stimulus manipulations in (Rowland et al., 2019) (i.e., percussive cross-stick samples at speech event onsets) highlight the importance of rhythmic cues in the sound-to-music effect – results that we replicate here in a clinical population. Our findings suggest that the auditory system may be relying more on temporal envelope cues rather than fine structure for the perception of this illusion.

For both groups, musicality ratings were higher for water than for speech, which parallels the condition effect sizes also reported in (Rowland et al., 2019). We replicated prior results (Rowland et al., 2019) in our older, NH control group which suggests, along with other findings (Mullin et al., 2021), that age does not affect the magnitude of the sound-to-music effect. Furthermore, our replication highlights that musicality ratings are highly stable across participants for a given stimulus (Tierney et al., 2018a).

In the qualitative post-repetition survey, CI users and NH controls reported experiencing similar percepts (Figure 7) which aligned well with their quantitative ratings. Both groups reported perceiving rhythm and melody in the speech condition, while in the water condition, rhythmic cues were significantly more apparent. Environmental sounds can often be identified with temporal/rhythmic cues, conveyed by the sound envelope, even in the absence of detailed spectral information (Gygi et al., 2004). The sound envelope of speech contains relatively slow amplitude fluctuations over time while the sound envelope of environmental sounds is typically much faster. As a result, these two categories can be successfully distinguished by this characteristic alone (Reed & Delhorne, 2005), despite the varied sounds (e.g., water dripping, a dog barking, ambulance siren, etc.) and associated acoustics that constitute “environmental sounds.” CI users have been shown to have relatively good environmental sound perception for sounds with distinct temporal envelopes (Inverso & Limb, 2010; Reed & Delhorne, 2005; Shafiro et al., 2022). Our qualitative findings provide evidence that rhythmic cues are both important and apparent to listeners perceiving the sound-to-music effect.

We also found that about one-quarter of the participants in both groups were “non-perceivers,” indicating notable individual differences in the sound-to-music effect. A few papers have reported on individual differences for this effect (Mullin et al., 2021; Tierney et al., 2021), though relatively more have used linear mixed effects models as we have done to account for individual participant behavior (Groenveld et al., 2020; Jaisin et al., 2016; Margulis & Simchy-Gross, 2016; Simchy-Gross & Margulis, 2018; Tierney et al., 2018a). While the speech-to-song illusion is indeed a robust phenomenon reported in the literature, there are substantial individual differences in who is susceptible to experiencing this musical transformation.

Next, we examined whether other factors, including frequency discrimination, clinical characteristics, and musical experience would influence musicality ratings. We hypothesized that CI users with better frequency discrimination (i.e., lower thresholds) would have greater musicality ratings. However, frequency discrimination was not associated with musicality ratings in either group. This was surprising for the CI group in particular given the considerable variability in frequency discrimination thresholds (Figure 6); however, other work in normal hearing participants also did not find a relationship between musicality ratings and frequency discrimination (Tierney et al., 2021).

Within the CI group, none of the clinical variables we measured was significantly associated with musicality ratings. These variables included duration of significant hearing loss, presence/absence of acoustic hearing during experimental testing, and presence/absence of acoustic hearing in daily life (i.e., residual hearing). This was at first surprising given other work showing that duration of deafness and acoustic hearing are associated with clinical outcome measures for speech recognition (Blamey et al., 2012; Holden et al., 2013; Lazard et al., 2012; Roditi et al., 2009). But, more recent work suggests that duration of deafness may not play a large role in functional outcomes when accounting for other factors such as daily CI processor use (DeFreese et al., 2023). The associations between musicality ratings in our experiment and both duration of hearing loss and residual hearing were, however, trending in the anticipated directions (though not statistically significant, see Table S4). It is possible that our sample size (n = 30) was too small to detect such relationships.

As hypothesized, musical experience was not associated with musicality ratings. This aligns with prior work showing that musical training does not correlate with the magnitude of the sound-to-music effect (Tierney et al., 2021) and that musicians and non-musicians alike perceive the effect (Mullin et al., 2021; Vanden Bosch Der Nederlanden et al., 2015). Recent work does show, however, that beat and tone perception skills (i.e., musical aptitude), rather than formal musical training, is associated with greater magnitude of the sound-to-music effect (Tierney et al., 2021) and enhanced neural coding for speech (Mankel & Bidelman, 2018). Conversely, one study showed that musical sophistication (as measured by the continuous and multidimensional Gold-MSI scale (Müllensiefen et al., 2014)) was associated with reduced speech-to-song musicality ratings (Groenveld et al., 2020). Future work could assess whether musical aptitude and musical sophistication, rather than formal musicianship, are associated with the sound-to-music effect in CI users.

Last, we assessed music satisfaction and experience in CI users and found low satisfaction with music listening post implantation and less time spent listening to music following hearing loss (Figure 8). This is consistent with widely reported outcomes of reduced satisfaction/appreciation of music and less time spent listening to music after CI activation (Drennan et al., 2015; Fuller et al., 2022; Gfeller et al., 2000; Leal et al., 2003; Riley et al., 2018). Thus, the cohort of CI users in this study appear representative of the broader CI population in regard to low music satisfaction.

One methodological limitation of our work is that participants always completed the water condition first, followed by the speech condition. Testing order bias could be contributing to our finding that, overall, water ratings were higher than speech ratings. However, we do not necessarily think this is the case because in (Rowland et al., 2019) water ratings were also greater than speech ratings, and that study was a between-subjects, rather than a within-subjects, design. Also, prior work suggests that “it is hard to ‘unhear’ the illusion” (Groenveld et al., 2020) such that when participants heard a more “musical” condition first, they tended to give higher musicality ratings for subsequent stimuli. If this were the case in our data, we would have observed higher musicality ratings for speech (the second condition participants completed) compared to water (the first, and indeed more musical condition participants completed). Instead, we did not find that musical expectations generalized between stimulus types. Nonetheless, it would be important for future studies employing within-subjects designs in larger participant samples to counterbalance conditions.

A second limitation of this work relates to the on-screen scale where participants made perceptual ratings. These scales, with the endpoints “exactly like speech/exactly like singing” and “not at all like music/exactly like music” (Figure 2) could suggest a change in perception over a trial that might influence participants to respond in a positive way. While this is a possibility, we think this was unlikely in the majority of cases as a) there were several “non-perceiving” participants who did not move their mouse over the course of the experiment and b) the initial mouse position defaulted to the middle of the scale, precluding any biases about participant starting points/changes in perception over time.

Future Directions

A crucial next step for this work is to better understand the neural correlates and neurophysiological mechanisms underlying the sound-to-music effect. To our knowledge, there are only two studies that have explored the sound-to-music effect with neuroimaging (i.e., fMRI) in normal hearing individuals (Hymers et al., 2015; Tierney et al., 2013). These studies found that speech perceived as song, compared to normal speech, activated bilateral regions in the superior temporal and middle temporal gyri along with some frontal and parietal regions – findings that align with brain regions involved in song perception more generally (Schön et al., 2010; Whitehead & Armony, 2018). The sound-to-music effect provides a unique opportunity to further understand the relative overlap (Peretz et al., 2015) and dissociation (Chen et al., 2023; Norman-Haignere et al., 2015) of music elements (rhythm, melody) and speech/language processing in the brain. Neuroimaging studies could also help determine what factors may be at play for perceivers and non-perceivers of the sound-to-music effect.

Future work could use time-resolved methods such as magnetoencephalography (MEG) or electroencephalography (EEG) to understand the neural time-course corresponding to the change in perception from speech/water to song/music. When acoustic features are carefully matched, cortical tracking of song is more precise than cortical tracking of speech in difficult listening conditions, but is similar in easy listening conditions (Vanden Bosch der Nederlanden et al., 2020). Other work has found that cortical tracking for music and speech rhythm is modulated by musical training (Harding et al., 2019). During the transformation from speech to song, in which no acoustic changes occur, it would be interesting to investigate the degree of neural phase-locking over the course of repetitions and understand how phase-locking patterns map on to behavioral perceptions of speech and song. In CI users, neuroimaging techniques such as EEG/MEG and fMRI are more limited due to the electrical and ferromagnetic components of the implant. Techniques such as functional near infrared spectroscopy (fNIRS) could allow for a more nuanced understanding of the cortical regions that may come online during stimulus repetition in the sound-to-music effect in CI users.

In addition to future directions for neuroimaging, our work could inform new clinical directions for making music accessible to CI users. Composers could modify music to sound more pleasant for CI users; already, some work has designed deep neural networks that, in real time, remix pop songs and other music to sound more enjoyable (Gajęcki & Nogueira, 2018; Tahmasebi et al., 2020). Here, we show that CI users perceive more musicality after multiple, unalerted repetitions of a single stimulus, suggesting that music producers and composers could better incorporate this common tool into music tailored specially for this population. These newer approaches in music composition can be pursued in parallel with continued improvements to cochlear implant technology. Such work will hopefully spark collaborations between musicians, composers, audiologists, scientists, and patient stakeholders, bringing together interdisciplinary perspectives to advance quality of life and music listening for CI users.

Conclusion

Cochlear implant users experience challenges with music perception due to both physiological limitations (e.g., channel interaction) as well as engineering limitations (e.g., envelope-based rather than fine structure coding). As a result, many CI recipients report low satisfaction with music listening (Riley et al., 2018). In this study, we asked whether repetition of short, looped spoken phrases and environmental sound clips could be a useful tool for eliciting music perception in this population. We found that CI users did perceive the musical transformation for both speech and water droplets. While CI technology continues to improve features for music perception, repetition incorporated into music specially composed for CI users may provide a promising way for them to experience music.

Author Contributions

AVK, IMB, JR, and RHG designed the study. AVK, IMB, AJD acquired data. AVK, IMB, and AJD conducted data analysis. ALH provided input on statistical analyses and data interpretation. AVK and IMB drafted the manuscript and RHG, JR, MTW, and RLG provided guidance and feedback during manuscript preparation. All authors approved the manuscript for publication.

Acknowledgments

We thank Michael Burchesky, AuD and Rayah Kirby for assistance with data collection, Dr. Chris Stecker for creating the frequency discrimination task, and Dr. Dan Gustavson for data analysis suggestions. We also thank all of our participants. The color palette used in the figures is from nationalparkcolors by Katie Jolly: https://github.com/katiejolly/nationalparkcolors.

Disclosure statement

Author RHG was a consultant for Skylark Bio as well as a member of the Audiology Advisory Board for Advanced Bionics and Cochlear Americas at the time of publication. No competing interests are declared for any other authors.

Data availability statement

Stimulus files and deidentified participant data are available at the following Open Science Framework repository: https://osf.io/dujt3/.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/25742442.2024.2313430

Additional information

Funding

This work was supported by a National Science Foundation Graduate Research Fellowship Program award and NIDCD F31DC020112 to AVK, NIDCD F31DC015956 to IMB, VICTR Award VR52433 to AD, NIDCD R01 DC016977 to RLG, and NIDCD R01 DC009404 to RHG. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funders.