Using haptic stimulation to enhance auditory perception in hearing-impaired listeners

ABSTRACT

Introduction

Hearing-assistive devices, such as hearing aids and cochlear implants, transform the lives of hearing-impaired people. However, users often struggle to locate and segregate sounds. This leads to impaired threat detection and an inability to understand speech in noisy environments. Recent evidence suggests that segregation and localization can be improved by providing missing sound-information through haptic stimulation.

Areas covered

This article reviews the evidence that haptic stimulation can effectively provide sound information. It then discusses the research and development required for this approach to be implemented in a clinically viable device. This includes discussion of what sound information should be provided and how that information can be extracted and delivered.

Expert opinion

Although this research area has only recently emerged, it builds on a significant body of work showing that sound information can be effectively transferred through haptic stimulation. Current evidence suggests that haptic stimulation is highly effective at providing missing sound-information to cochlear implant users. However, a great deal of work remains to implement this approach in an effective wearable device. If successful, such a device could offer an inexpensive, noninvasive means of improving educational, work, and social experiences for hearing-impaired individuals, including those without access to hearing-assistive devices.

KEYWORDS:

• Cross-modal
• electro-haptic stimulation
• haptic sound-localization
• multisensory
• neuroprosthetic
• somatosensory
• tactile
• vibrotactile
• hearing aid
• cochlear implant

1. Introduction

Over the past half a century, dramatic advances in hearing-assistive device technology have enabled it to transform the lives of people with hearing impairment. One prominent example is the cochlear implant (CI), which enables severely-to-profoundly hearing-impaired individuals to perceive sounds through electrical stimulation of the auditory nerve. The CI stands as one of modern medicine’s greatest achievements, allowing users to follow a conversation in a quiet environment with a similar accuracy to normal-hearing listeners [e.g. 1]. However, significant limitations to CIs remain, with users often having considerable difficulties locating and segregating sounds [2,3]. Similar issues are experienced by hearing aid users, though to a lesser extent [3,4]. These limitations lead to impaired threat detection and an inability to understand speech in noisy environments, such as schools, restaurants, and busy workplaces.

Recently, a new approach has been proposed that uses ‘electro-haptic stimulation’¹ [5], whereby sound information that is poorly transferred by the electrical CI signal is provided through haptic stimulation. Exciting new evidence indicates that electro-haptic stimulation (EHS) can substantially improve speech-in-noise performance [5–8] and sound localization [9,10], as well as increasing sensitivity to more basic sound properties, such as pitch [11]. If effective, this approach could be delivered noninvasively and inexpensively in a compact wrist-worn device.

In addition to improving performance for hearing-assistive device users, haptic devices might be used to aid those currently unable to access hearing-assistive technology. It is estimated that around 99% of potential CI candidates worldwide cannot access a CI [12]. Furthermore, for those in low-to-middle income countries that can access a CI, surgical complication rates are around double that in high-income countries [13–16]. However, the main prohibitive factor is cost. In India, for example, the personal average annual income is less than 2,000 USD, whereas the cost of a CI (not including hospital fees) is between 12,000 and 25,000 USD [17]. The consequences of this untreated hearing loss are substantial. Young children with unmanaged hearing loss typically have large deficits in language and cognitive development and low educational attainment [18–22]. Children that have a disabling hearing loss in low-to-middle income countries are also unlikely to complete primary education [23]; strikingly, in India, less than a third of hearing-impaired children are enrolled in school at any level and less than 2% receive higher secondary education or above [17]. Hearing-impaired adults in low-to-middle income countries have a much lower employment rate, and those that are employed tend to be in lower-grade occupations [20,24]. This means that, while hearing loss is often a result of poverty, it is also often a cause of poverty [15,20,25]. The development of low-cost haptic devices to aid those with hearing impairments who are unable to access hearing-assistive devices could therefore have a substantial positive impact on quality of life, particularly in low-to-middle income countries.

This review is divided into two parts. The first examines existing work on the use of haptic stimulation to aid hearing-impaired listeners, whilst the second discusses how the promising work already undertaken could be translated into a clinically viable haptic device. It considers what sound information is most beneficial to hearing-impaired listeners, how to provide that information, and what the necessary requirements are for a successful haptic device.

2. Background

2.1. Enhancement of speech-in-noise performance

Work using haptic stimulation to aid those with hearing impairment dates back to at least the 1920s, when a desktop haptic device that stimulated the fingers was trialed to support deaf children in the classroom [26–29]. For deaf individuals who were simultaneously lip reading, this device was reported to increase the number of words recognized by around 20% when there was no background noise. Later, beginning in the late 1960s, researchers in the visual sciences used a similar approach for blind individuals, delivering visual information through haptic stimulation on the fingers or back [30–33]. Using this approach, participants were able to perceive depth and perspective, judge the speed and direction of a rolling ball, recognize faces and common objects, and complete complex inspection-assembly tasks. Interestingly, after training, participants reported that they experienced these ‘images’ as being externalized in front of them, rather than being located at the haptic stimulation site. These influential studies from both the auditory and visual sciences helped trigger an expansion of research into the use of haptic stimulation to aid deaf individuals, which peaked in the 1980s and 1990s [34,35]. The ‘tactile aids’ that were developed showed significant promise. One study, for example, showed that after extensive training with the Queen’s Tactile Vocoder device, it was possible to learn a vocabulary of 250 words [36,37]. However, in parallel to the development of tactile aids, CI technology underwent a revolution [1]. By the mid-1990s, outcomes for CI users had substantially outstripped those achieved by tactile aids [1,34]; by the early 2000s, the success of the CI had caused the development and use of tactile aids to all but cease.

EHS uses haptic stimulation to augment CI listening, rather than as an alternative to the CI. To assess the potential for a new generation of haptic devices to aid hearing-impaired individuals, it is important to consider the limitations that prevented the widespread use of tactile aids in the 1980s and 1990s. One limitation was that these devices were not compact or discreet, and required a large battery pack that frequently needed to be recharged. For example, the body-worn processor unit alone for the Siemens Minifonator measured 84 × 82 x 30 mm and for the Tactaid II measured 93 × 57 x 23 mm [38]. Another issue was that the electronics, microphones, and haptic stimulators were all connected by wires. This made many tactile aids difficult to self-fit, uncomfortable to wear, and raised safety concerns (for example, that wires might get caught on objects such as cups and saucepans). A further important limitation was the impossibility of performing advanced signal-processing. Many of these limitations are now considerably reduced due to the substantial developments in motor, battery, microprocessor, and wireless-communication technology. The time therefore seems right for a new generation of compact, discreet haptic devices to support the hearing impaired.

While tactile aids of the 1980s and 1990s were ineffective in noisy environments, two recent studies have shown that haptic stimulation can be used to improve speech-in-noise performance in CI users [7] and normal-hearing listeners [39]. However, there are two significant limitations to these studies. Firstly, haptic stimulation was delivered to the fingertip, which would disrupt many everyday activities if deployed in a clinical device; secondly, the haptic signal was extracted from the clean speech signal (without background noise), which is not available in the real world. If the clean speech signal was available, then this signal would simply be presented to the listener through the hearing aid or CI.

More recently, it has been shown that haptic signals extracted from speech in noise and delivered to the wrists can improve speech-in-noise performance for CI users [5,6,8]. These studies showed benefits across participants who used a range of CI devices (from MED-EL, Advanced Bionics, and Cochlear Ltd). In one study, where speech and noise were both presented from directly in front of the listener, CI users recognized 8% more words in noise with EHS, with some participants recognizing over 20% more words [5]. Another study explored whether EHS improved speech recognition when speech and noise were spatially separated. This study focused on the 95% of CI users that are only implanted in one ear [40]. The speech was presented from directly in front of the listener and the noise was presented either to the implanted side or to the non-implanted side. For both noise positions, CI users’ speech-reception thresholds in noise were improved by around 3 dB when EHS was provided [8]. In these studies of EHS enhancement, the signal processing was computationally lightweight so that it could be applied in real-time on a compact device. This demonstration of an effective and clinically viable approach marks an exciting advance in the translation of EHS from a research finding to an effective clinical tool.

The speech-in-noise performance benefit measured for EHS with spatially separated sounds is comparable to the improvement observed when patients use two implants rather one [see 8 for discussion,41,42]. However, implantation of a second device is expensive, risks loss of residual hearing and vestibular dysfunction, and limits access to future technologies and therapies. A noninvasive, inexpensive haptic device may therefore be an attractive alternative to a second implant.

For the many that have not received a second implant, another approach used to improve speech-in-noise performance is the mounting of an additional microphone behind the non-implanted ear. The audio from this microphone is transmitted to the implant so that the signals from the implanted and non-implanted sides can be combined. This contralateral routing of signal (CROS) approach aims to reduce the negative effects of the acoustic head-shadow when a sound of interest is on the non-implanted side. One study established whether CROS microphones benefit speech-in-noise performance when speech is presented in front of the listener and noise is presented either to the side with the implant or to the opposite side [43]. Unexpectedly, no benefit of the CROS microphone was found when the noise was on the implanted side and the CROS microphone was found to impair performance when the noise was on the non-implanted side. Another study found that CROS microphones did not affect speech-in-noise performance when the speech and noise were both in front of the listener, and reduced performance when the speech was in front and the noise was on the implanted or non-implanted side [44]. EHS, on the other hand, has been shown to produce clear benefits in each of these three speech and noise configurations. Other studies have found considerable benefits of CROS microphones under different conditions, such as when speech is located on the non-implanted side and noise comes from loudspeakers all around the listener [45,46]. To date, no studies have assessed EHS benefits under comparable conditions.

2.2. Enhancement of sound localization

In addition to studies showing that tactile aids can be used to provide speech information, a small number of studies showed that haptic stimulation on the fingertips could be used to locate sounds [47–51]. However, despite this early promise, haptic sound-localization remains little studied. Building from this work, it was recently shown that EHS can be used to dramatically improve sound localization in CI users [9]. In this study, the haptic signal was derived from the audio received by behind-the-ear hearing-assistive devices and delivered to each wrist. This allowed participants to access intensity differences between the ears [52], which are key cues for sound localization. In CI users who were implanted in one ear (unilateral CI users), even without training, EHS was found to reduce RMS error in sound localization from 47° to 29°, making their performance similar to CI users implanted in both ears [bilateral CI users; 3,53]. After a small amount of training with EHS (lasting around 15 minutes), performance improved substantially, becoming similar to that of bilateral hearing-aid users [3,54]. Another recent study, which used a similar approach but with a more sophisticated signal-processing strategy, found still greater haptic sound-localization accuracy [10]. Researchers have explored whether CROS microphones improve sound localization for unilateral CI users, but found no clear benefit [55].

The same EHS approach used for haptic sound-localisation has been shown to enhance speech-in-noise performance for spatially separated sounds [8]. The signal-processing approach is also similar to EHS approaches that have been shown to enhance speech-in-noise performance for co-located sounds [5,6]. Future work should aim to unify these promising signal-processing strategies.

2.3. Enhancement of music perception

CI users frequently suffer from an inability to appreciate and enjoy music [56]. This is primarily due to the implant’s inability to provide frequency information, which conveys critical melody, harmony, and tonality information, and is important for sound segregation [56–58]. Some studies have shown evidence that melody recognition can be improved using haptic stimulation at either the fingertip [59] or wrist [60]. Another study showed that a haptic device on the forearm could be used to substantially improve discrimination of changes in fundamental frequency (an acoustic correlate of pitch). Participants were able to discriminate fundamental-frequency shifts of just 1% [11], which is less than the smallest pitch change found in most western melodies and substantially better than typical CI users [61,62]. This performance was maintained even in the presence of high levels of inharmonic background noise (with signal-to-noise ratios as low as −7.5 dB). However, an important challenge for this and other approaches will be to extract sound information for a single harmonic sound against a background of other harmonic sounds, such as in a polyphonic musical piece. This may be aided by the recent emergence of object-based audio encoding for music, film, and gaming, which gives access to individual sounds within a musical piece or auditory scene [e.g. 63,64].

For the promising findings discussed to be successfully translated into a clinically viable haptic device, there are several important questions that must be addressed: (1) how will the audio signal be acquired; (2) how will this audio signal be processed and converted to haptic stimulation; (3) how and where will haptic stimulation be delivered; and (4) what are the key specifications for a successful haptic device? These questions will be considered in the following section.

3. Priorities for haptic provision and device design

3.1. Audio signal acquisition

The first challenge for a haptic device will be how to capture the audio that is transformed to haptic stimulation. In one proposed approach, the audio signal is streamed from behind-the-ear CIs or hearing aids that are either already worn by the user or are fitted in addition to an existing device [5,6,8,9]. One advantage of this approach is that technology already deployed in hearing-assistive devices, such as beamforming [2,65], can be exploited. In beamforming, the difference in the arrival time at multiple microphones mounted within a single device is used to steer the maximum sensitivity toward the sound source of interest (typically in front of the listener) and reduce sensitivity to sources from other locations (typically to the back and sides). This approach has been shown to substantially improve speech-in-noise performance [66]. Another highly effective approach used with hearing-assistive devices is remote microphones, which are placed close to the sound source of interest [66]. Remote microphones, such as the Roger Pen or Oticon ConnectClip, use Bluetooth or radio to stream audio directly to the hearing-assistive device. A haptic device that streamed audio from a hearing-assistive device could benefit from this existing technology.

Streaming audio from hearing-assistive devices has further advantages. Firstly, the haptic device could benefit from some of the signal processing already performed by the hearing-assistive device (such as pre-emphasis and microphone frequency-response correction filtering). Secondly, if audio is streamed from hearing-assistive devices behind each ear, haptic devices will have access to spatial-hearing cues (such as intensity differences between the ears), which has been exploited in previous work to improve sound localization [9,10,52]. Finally, streaming audio from the same source as the hearing-assistive device will maximize the correlation between audio and haptic signals, which is critical for effective multisensory integration [67–71].

For audio streaming from a hearing-assistive device to be viable for real-world use, low-power wireless streaming technology is required. One new technology, which is available in many of the latest hearing-assistive devices, is Bluetooth Low Power (LE). Bluetooth LE has greatly reduced power consumption compared to classic Bluetooth, allows higher-quality audio streaming, and supports multiple simultaneous data streams. An alternative to Bluetooth LE, which is used by Advanced Bionics and Phonak for streaming between hearing-assistive devices, is low-frequency radio. Low-frequency radio allows extremely low-latency data transfer but has high power consumption. Further work is required to establish the most effective technology for streaming between hearing-assistive and haptic devices.

An alternative to streaming audio from behind-the-ear devices is to mount microphones either on the haptic device or on another part of the body. A microphone mounted on a wrist- or hand-worn device might allow the user to direct the microphone toward a talker or other sound source of interest. However, arm movements, such as when walking or gesticulating, may lead to unwanted distortion of the audio signal. A newly released wrist-worn haptic device, the ‘Buzz’ (Neosensory, San Francisco, USA), has microphones mounted on top of the device. In informal real-world trials by the author and colleagues, this device was found to be frequently triggered by clothing moving against the device and to be highly susceptible to wind noise. It was also found to be excessively triggered by impulsive sounds, particularly when the hands manipulated objects during activities such as typing or cooking. These issues would be reduced or avoided by streaming audio from behind-the-ear devices, which use advanced techniques to suppress wind noise and impulsive sounds [72]. A combination of microphones mounted on the device or body and on hearing-assistive devices might also be considered, particularly as having access to audio from microphones at multiple sites might aid noise reduction [e.g. 73,74].

3.2. Signal processing

Once the audio has been received by the haptic device, the next consideration is how it should be processed. The first possible approach is not to process it at all, and to rely on the skin to extract the most important sound features [27,29,39,75]. One major limitation of this approach is that the skin is insensitive to vibration at frequencies higher than around 500 Hz [76], where a large amount of speech energy resides [77]. To overcome this issue, one tactile aid transposed sound at higher frequencies down to lower frequencies [78]. Nonetheless, using this approach, important stimulus features are likely to be masked or to be impossible for the tactile system to extract [79,80].

Another approach is to extract key sound features from the audio signal and map them to the haptic signal. It is likely to be important to provide sound features that give frequency information, such as the fundamental frequency of the sound of interest. Hearing impairment almost always leads to a reduced ability to discriminate sounds at different frequencies [81]. For CI users, frequency discrimination is typically particularly poor [82]. This can impair talker age, sex, and accent identification [83,84] as well as perception of speech prosody, which allows listeners to distinguish emotions (e.g. anger from sadness), intention (e.g. sarcastic from sincere), statements from questions, and nouns from verbs (e.g. ‘object’ from ‘object’) [85–88]. Frequency information is also critical to separating sounds that occur at the same time [58,89] and to music perception [56]. One priority for haptic devices should therefore be provision of frequency information.

Another important feature is how sound changes in amplitude over time (the amplitude envelope). Hearing impairment almost always leads to a reduction in the dynamic range available to the listener (the difference between detection threshold and uncomfortably intense stimulation). The dynamic-range available to hearing-impaired listeners is typically around half that of normal-hearing listeners [90]. The dynamic-range available for electrical stimulation in CI users, however, is around just an eighth of that for normal-hearing listeners [91–93]. Ability to discriminate sounds at different intensities is also typically severely impaired in CI users [94]. Encouragingly, the dynamic range for vibro-tactile stimulation is around four times the dynamic range available through electrical stimulation with a CI [52]. The tactile system also has excellent intensity discrimination, which is comparable to that of the healthy auditory system [52,95–99] and is highly sensitive to amplitude envelope modulations at the frequencies that are most important for speech recognition [100,101]. A second priority for a haptic device should therefore be provision of amplitude envelope information.

In line with these priorities, many tactile aids that aimed to enhance lip-reading in deaf individuals extracted frequency or amplitude information [36,102]. Previous studies have compared speech recognition when providing the fundamental frequency or amplitude envelope of the speech through audio, either in isolation [103] or in addition to CI-simulated audio [104,105]. Similar benefit to speech reception thresholds was found for each feature. However, the fundamental frequency provided more information about vowel duration and stress, whereas the amplitude envelope provided more information about consonant place, manner, and voicing [103]. This is consistent with the finding that, while each feature provides similar overall benefit to speech-in-noise performance, the provision of both features together provides most benefit [104].

Like tactile aids, recent studies showing benefit of EHS to speech-in-noise performance in CI users have also extracted frequency or amplitude information. Huang, Sheffield [7] showed benefit to speech-in-noise performance in CI users by presenting the fundamental frequency through haptic stimulation. Changes in fundamental frequency were delivered through changes in the frequency of haptic stimulation on the fingertips. Besides the issues already discussed, regarding the stimulation site and the deriving of the haptic signal from clean speech, delivering information through changes in haptic stimulation frequency is likely to lead to information being lost due to the skin’s poor frequency resolution [79]. One way that some devices have overcome this issue is by mapping frequency to location on the skin. A recent study used the newly developed mosaicOne_B device, which has an array of haptic stimulators on the forearm and uses a novel approach for mapping fundamental frequency to stimulation location [11]. This device was shown to be highly effective at delivering fundamental-frequency information, and was robust to background noise. Future work should evaluate whether the mosaicOne_B can be used to enhance speech-in-noise performance.

Other researchers that have shown EHS can enhance speech-in-noise performance for CI users have primarily focused on providing speech amplitude-envelope information [5,6,8]. In these studies, information was also provided about the relative sound energy across either four [5,8] or seven [6] frequency bands, which were selected to contain substantial speech energy. Frequency and amplitude information were delivered through changes in the haptic stimulation intensity of tones focused within the frequency range where sensitivity is high. The frequency separation between these tones meant that they were expected to be individually discriminable. However, as argued above, it may have been possible to transfer more frequency information through a spatial, rather than frequency, mapping [11]. The three studies that have shown improved speech-in-noise performance by providing amplitude envelope information through haptic stimulation have derived their haptic signal from speech in noise, rather than from clean speech [5,6,8]. To do this, two of these studies [5,6] used a simple noise-reduction approach that relied on the speech signal being more intense than the background noise. This is adequate for enhancing speech-in-noise performance for CI users, who typically struggle even when speech is substantially louder than the background noise [5,8]. However, it may not be suitable for hearing aid users, who are typically able to follow speech in situations where the noise is louder than the speech [4]. Future work should assess the effectiveness of more sophisticated methods for extracting signals in noise to widen the applicability of this approach [106,107].

Another important feature is sound location. In normal-hearing listeners, the origin of a sound is determined primarily by assessing differences in the intensity and arrival time between the ears. As previously discussed, highly accurate sound localization has been shown using haptic stimulation derived from audio received by behind-the-ear devices [9,10]. In this work, sounds were located using intensity differences across the wrists [52], which matched the sound intensity differences across the ears. The differences in arrival time between the ears were also provided through haptic stimulation, but these differences were much smaller than can likely be discriminated by the tactile system [108,109]. Future work could explore methods for enhancing spatial-hearing cues to further improve haptic sound-localization [110–112]. One approach that might be explored is to remap time difference cues to intensity differences so that they can be effectively extracted by the tactile system.

Any haptic signal-processing that is deployed must be computationally lightweight. This is to avoid incurring a delay in the arrival of the haptic signal that could disrupt binding of auditory, visual (e.g. lip reading), and haptic information. It will also be important for allowing the signal-processing unit to be compact and power efficient. There is encouraging evidence that a processing delay of tens of milliseconds may be acceptable, although there is insufficient evidence currently to establish this with confidence. One line of evidence comes from research studying the influence of haptic stimulation (air puffs) on the perception of aspirated and unaspirated syllables [113]. In this work, it was found that the influence of haptic stimulation was not significantly reduced when haptic stimulation was delayed by up to 100 ms. Other work has shown evidence of ‘temporal recalibration’, where consistent delays of several tens of milliseconds between correlated sensory inputs are rapidly corrected for in the brain so that perceptual synchrony is retained [114–117]. If haptic stimulation can be delayed from the audio and visual signal by tens of milliseconds, then this could allow for sophisticated signal-processing strategies to be implemented in haptic devices.

Another technology that might maximize the effectiveness of signal-processing regimes is low-latency data streaming between haptic devices. This could be achieved using radio or Bluetooth LE technology, which is discussed in the Audio signal acquisition section. One way in which streaming between devices may be important is for linking signal-processing that adjusts the signal intensity or delay, such as compressors, to avoid distortion of spatial hearing cues [118].

3.3. Signal delivery

3.3.1. Stimulation method

Once the signal has been processed, the next consideration is how it should be delivered. Haptic stimulation has traditionally been delivered either through electro-tactile stimulation, whereby a current is passed through the skin, or vibro-tactile stimulation, whereby the skin is mechanically indented. The usable frequency and amplitude ranges for electro-tactile stimulation are substantially smaller than for vibro-tactile stimulation [119–122]. Furthermore, because electro-tactile stimulation depends on the electrical resistance of the skin, it is strongly affected by its moisture content and by small changes in the stimulation location [119,120,123]. Because of the limited frequency and amplitude range for electro-tactile stimulation, sound information has typically been delivered using arrays of electrical stimulators, with sound features mapped to changes in stimulation location and pulse rate [124,125]. Besides these limitations, there are also safety concerns with electrical stimulation that do not apply to vibro-tactile stimulation. Firstly, because the fingers have a lower electrical resistance than most other body parts, devices designed for other body parts must ensure that the electrical contacts cannot be touched by the user’s finger [126]. Secondly, if mounted on the chest, electro-tactile devices may not be suitable for those with pacemakers. One advantage of using electrical stimulation is that it may require less power, and therefore allow a longer battery life for the device [126]. However, given the limitations and additional safety considerations, vibro-tactile stimulation appears to be a more suitable stimulation method.

Recent developments in haptic motor and driver technology have made it possible for precisely controlled vibro-tactile stimulation to be delivered in compact devices at a low cost. Because of their higher power-efficiency, linear resonant actuators (which generate vibration through a voice coil moving a mass) may be preferred to eccentric rotating mass motors (which generate vibration through rotation of an unbalanced load). Piezoelectric motors also have high power-efficiency but are often expensive. The response latency and precision of waveform tracking for linear resonant actuators and eccentric rotating mass motors can be improved using overdrive and active-breaking techniques. Overdrive involves temporarily driving the motor above its rated voltage to reduce the time it takes to rise to its target intensity. Active breaking involves applying a reverse voltage to reduce the time the motor takes to fall to its target intensity. Application of these techniques using the latest haptic-driver technology may be important for achieving sufficiently precise speech amplitude-envelope tracking.

In addition to providing vibro-tactile stimulation, the Tactile and Squeeze Bracelet Interface (Tasbi), which was recently developed by Facebook Reality Labs, modulates the amount of pressure applied [127]. This prototype device, which was developed to enhance interactions in virtual environments, has a tensioning mechanism that adjusts the amount of ‘squeeze’ as well as six linear resonant actuators spaced around the wrist. One way in which squeeze intensity could be used in a haptic device for the hearing impaired is to provide information about absolute sound intensity. This would allow vibro-tactile stimulation to be focused on providing detailed information about more subtle local amplitude changes. Squeeze feedback could also be effective for supporting music, film, and video games as it has been argued that it elicits emotional responses and is less attention demanding than vibro-tactile stimulation [127–129].

3.3.2. Stimulation site

After establishing the most appropriate stimulation method, the stimulation site must then be considered. A suitable site will be sufficiently sensitive to allow sound information to be effectively transferred, whilst allowing easy device self-fitting, high comfort, and minimal disruption to common activities. Some recent studies have provided haptic stimulation to the fingertip [6,7,130], because it is highly sensitive and contains a high density of tactile receptors [131]. However, the fingertip does not seem an optimal site for real-world use as it is frequently involved in everyday tasks. An alternative site, also used in recent studies, is the wrist [5,8–10]. Although the wrist has higher vibro-tactile detection thresholds than the fingertip [132] and a lower density of tactile receptors [131], there is evidence that intensity discrimination is enhanced at the wrist compared to the fingertip, and that frequency discrimination and temporal-gap detection are similar [132]. Moreover, the wrist would seem a practical site for a real-world application. Wrist-worn devices are familiar, esthetically unobtrusive, do not impede everyday tasks, and are easy to self-fit.

Figure 1 shows the mosaicOne_C, a wrist-worn haptic device for augmenting CI listening that is currently under development. Building on the approach used in the mosaicOne_B device [11], which is worn on the forearm, fundamental frequency can be mapped to stimulation location around the wrist using four vibro-tactile motors. The perception of haptic stimulation can be created at a continuum of positions around the wrist by panning between the motors, which maximizes the resolution of the device. The Buzz, another wrist-worn haptic device for enhancing auditory perception, also has multiple motors arranged around the wrist. The precise signal-processing strategy used to convert audio to haptic stimulation is not in the public domain, but the Buzz does not map the fundamental frequency of a sound to a position on the wrist. Other multi-motor prototype wrist-worn devices have been developed for other applications, such as enhancing virtual and augmented reality (e.g. the Tasbi, discussed above), delivering more detailed notifications and alerts [133], or improving color discrimination in color-blind people [134].

Figure 1. Image of the mosaicOne_C wrist-worn haptic device currently under development as part of the Electro-Haptics Research Project (www.electrohaptics.co.uk) at the University of Southampton (UK). Four haptic motors are housed around a rubber wrist-strap. Image reproduced with permission of Samuel Perry and Mark Fletcher

One potential limitation of providing haptic stimulation at the wrist or finger is the frequent movements and changes in relative position during many activities. This could distort sound information, particularly if transmitted through differences in stimulation across the hands or wrists. This idea is supported by work showing that crossing the arms impairs temporal-order judgments for haptic stimulation across the hands [135,136], although it is not clear whether this can be overcome with training. Other evidence suggests that changes in relative arm position do not impair the perception of intensity difference cues, which are used for haptic sound-localization [9,10]. For example, one study found that haptic intensity perception on one hand was modulated by haptic stimulation on the other hand, but that this modulation did not depend on the relative positions of the hands [137]. However, further work is required to properly assess the impact of body motion on the transfer of sound features through haptic stimulation.

Given the possibility that changes in the relative position of haptic devices might impair information transfer, sites whose relative positions are more fixed should be considered. Previously, tactile aids have been developed that provide stimulation on the sternum [138], abdomen [139], or back [140]. Wilska [141] compared the sensitivity of different sites. He found the sensitivity of the sternum to be quite similar to the wrist, the abdomen to have much lower sensitivity, and some areas of the back to be less sensitive than the wrist or sternum but substantially more sensitive than the abdomen. Other potential sites for haptic stimulation might be the biceps or feet. Like the back, these sites are less sensitive than the wrist or sternum but are more sensitive than the abdomen. While many of these candidate sites benefit from allowing devices to be discreet, some may raise difficulties for self-fitting or lead to uncomfortable feelings of restrictedness that were reported by some users of body-worn tactile aids.

For devices that map changes in stimulus features to changes in location of stimulation, it is also important to consider the spatial acuity of the tactile system at different sites. The ability to discriminate two spatially separate stimuli varies substantially across different parts of the body. For example, spatial acuity is high at the fingertip, is reduced on the forearm, and is reduced further still on the shoulders [142]. It should be noted however, that there is more space available for across-site stimulation on the forearm and shoulder than on the fingertip. As well as careful selection of stimulation site, devices using spatial mapping of stimulus features should consider the decline in spatial acuity with age [143], ensuring that motors are sufficiently spaced to retain performance in older populations.

3.4. Device specifications

Several additional specifications must be met if a haptic device is to be clinically successful. One important issue is power management. Hearing-assistive devices target a minimum battery life of 14 hours, so that a typical user (who sleeps for 8 hours each day) needs only to charge their device overnight. However, modern devices using lithium-ion batteries often last several days on a single charge. With careful power management and use of low-power motor (e.g. linear resonant actuators) and wireless (e.g. Bluetooth LE) technology, as well as computationally lightweight signal processing, a haptic device that meets the required battery-life is readily achievable. The Buzz wrist-worn haptic device, for example, can be continuously used for more than 24 hours with a single charge.

Other important considerations for haptic-device design are esthetic attractiveness, compactness, discreetness, and comfort. It will be important for any haptic device to be lightweight and have a small footprint, although the precise acceptable form-factor will no doubt be influenced by the amount of benefit the device gives. A compact and lightweight device can readily be produced using recently developed low-cost, compact motor and haptic-driver technology in combination with the battery, wireless, and signal-processing technology already implemented in hearing-assistive devices. A common complaint about tactile aids was that they highlighted that the user had a hearing impairment. This could be an issue for devices fitted at sites where they are likely to be visible, such as the wrist. However, given the current prevalence of smartwatches, a wrist-worn device with a sufficiently modern design (like that shown in Figure 1) may be acceptable.

Another important feature of any device will be ease of use for the patient and clinician. This will include already mentioned considerations, such as ease of self-fitting, but may also mean the inclusion of adjustable device settings through easy to use and understand buttons on the device or a linked smartphone app. It is also possible that device tuning, based on the user’s vibro-tactile detection and discomfort thresholds, will be required to maximize comfort and the dynamic range available to the device. To facilitate uptake, tuning routines for clinicians or users must be fast and intuitive. It is also possible that the optimal haptic signal-processing strategy will depend on the user’s hearing-assistive device type and programming. In this case, firmware updates that adjust the haptic signal-processing strategy could be sent from the hearing-assistive device when a new haptic device is paired with it. This would require either close collaboration between hearing-assistive and haptic device manufacturers, or for hearing-assistive device manufacturers to develop their own haptic devices. However, it is important to note that across a number of studies that have shown clear benefits of EHS for a range of CI devices, there was no individual tuning of haptic stimulation [5,7–9]. Furthermore, despite substantial variation in vibro-tactile detection thresholds, no correlation between the size of the benefit of EHS and detection threshold has been found [5,6,8–10]. It is therefore possible that effective haptic devices might be developed that required little or no individual tuning.

Finally, additional features might be added to haptic devices to assist in daily life. For example, the device might connect to a range of smart devices within the Internet of Things to improve awareness and safety. These might include doorbells, telephones, baby monitors, ovens, and wake-up, intruder, fire, or carbon monoxide alarms. The effectiveness of some of these additional features will partially depend on the haptic device having a long battery-life or allowing easy switching of battery units.

4. Conclusion

Exciting new evidence has recently emerged showing that providing missing sound-information through haptic stimulation could be highly effective in augmenting hearing-assistive devices. This approach could also be used to aid the many millions of hearing-impaired people worldwide who cannot access hearing-assistive technology. So far, the approach has shown particular promise for CI users, for whom impressive improvements to speech-in-noise performance and spatial hearing have been demonstrated. These laboratory findings must now be reproduced in the real-world with a device that is appropriate for clinical use. The technology required to develop such a device is already available. However, a large amount of work remains to establish the best way to effectively acquire and process the audio signal, the optimal device configuration, and the most suitable stimulation site. Furthermore, an effective device will likely require the combining of cutting-edge motor, battery, microprocessor, and wireless communication technology. If this can be achieved, then such a device could provide a noninvasive, low-cost means of substantially improving outcomes for hearing-impaired listeners.

5. Expert opinion

It is predicted that the number of people with a disabling hearing loss will nearly double in the next 30 years [20]. There is therefore a rapidly growing population that could potentially benefit from the use of haptic stimulation to provide auditory information. It seems likely that haptics can provide most benefit to those with severe-to-profound hearing impairments, who either have CIs or would be CI candidates. For those fortunate enough to have access to CIs, an effective haptic device could significantly increase spatial awareness and the ability to hear in noisy environments. It could also offer an inexpensive means to acquire the benefits of a second CI without the need for an expensive second surgery. This could substantially reduce costs for individuals and healthcare services. However, many people across the world cannot access facilities for implanting a CI or providing a hearing aid, with cost being a major prohibitive factor. In India, for example, the cost of getting a CI is several times the personal average income [17], making them unaffordable for the majority of candidates. For these people, an effective haptic device might offer an affordable means of recovering critical access to the auditory world. This could allow children and adults far greater access to education, work, and leisure and thereby substantially their improve quality of life.

Currently, the main barrier to uptake of this approach is the absence of an effective, clinically approved haptic device. If an effective device was available that was inexpensive, comfortable, discreet, easy for the user to self-fit, and easy for the clinician to tune to the individual, then it is difficult to see significant barriers to uptake. Substantial work remains, however, to establish the optimal signal-processing strategy and device configuration to maximize benefit for both hearing-assistive device users and those who cannot access hearing-assistive technology. There are also significant challenges ahead in designing and manufacturing a suitable haptic device, carrying out carefully controlled large-scale real-world trials, and obtaining clinical approval. All these challenges, however, can be met.

Within the next five years, a significant expansion in the number of researchers working in this area is anticipated. As the field grows, the range of outcome measures used to assess the benefits of haptic stimulation to hearing is also expected to increase. For example, it will likely soon be understood whether haptic stimulation can be used to reduce listening effort and improve access to speech prosody. Advanced neuroimaging methods, such as near-infrared spectroscopy and electroencephalography, will also likely be deployed so that the underlying mechanisms behind haptic enhancement of hearing can be understood. The biggest development, however, is expected to be the production of an effective device and the translation from laboratory testing to real-world trials. To develop such a device will require the bringing together of several cutting-edge technologies. This technology will likely include 3D-printing, compact power-cells, low-latency data streaming, microprocessors, haptic drivers, and micro-motors. It will be critical for clinicians, engineers, researchers, and industry to work closely together. By doing this, it seems likely that, within the next five years, we will see a clinically approved haptic device to enhance auditory perception in hearing-impaired listeners.

Article highlights

• Recent studies have shown compelling evidence that haptic stimulation can be used to enhance spatial hearing and speech-in-noise performance for cochlear implant users. Haptic stimulation might also have utility for hearing aid users, particularly for improving spatial hearing, as well as for those without access to hearing-assistive devices.

• Laboratory studies are required to establish the limits of this approach, such as how much delay there can be between the audio and haptic signal before the benefits of haptic stimulation decrease. These studies will be critical for informing haptic device design.

• Significant questions remain regarding how best to acquire the audio signal that is converted to haptic stimulation, how best to process and deliver the haptic signal, and the precise specification for a successful device.

• The technology required to develop a device that meets the anticipated requirements already exists.

• Experiments have so far been confined to the laboratory and field trials are required to fully establish the efficacy of the approach.

Declaration of interest

M Fletcher is employed by the University of Southampton but his salary is funded by the William Demant Foundation. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Acknowledgments

My sincerest thanks to Carl Verschuur for his continued support, to Beeps Perry for the many insightful discussions, to Andriano Brookmani for help with the manuscript text, and to Alex and Helen Fletcher for their patience and support during the writing of this manuscript.

Additional information

Funding

This paper was funded by William Demant Foundation, Kongebakken 92765, Smørum.

Notes

1. Other researchers have used the term ‘electro-tactile stimulation’. The term ‘electro-haptic stimulation’ is preferred as electro-tactile stimulation is commonly used to refer to electrical stimulation of the skin, rather than to using tactile stimulation to augment CI listening.