Preliminary assessment of the feasibility of using AB words to assess candidacy in adults

Abstract

Background: Adult cochlear implant (CI) candidacy is assessed in part by the use of speech perception measures. In the United Kingdom the current cut-off point to fall within the CI candidacy range is a score of less than 50% on the BKB sentences presented in quiet (presented at 70 dBSPL).

Goal: The specific goal of this article was to review the benefit of adding the AB word test to the assessment test battery for candidacy.

Results: The AB word test scores showed good sensitivity and specificity when calculated based on both word and phoneme scores. The word score equivalent for 50% correct on the BKB sentences was 18.5% and it was 34.5% when the phoneme score was calculated; these scores are in line with those used in centres in Wales (15% AB word score).

Conclusion: The goal of the British Cochlear Implant Group (BCIG) service evaluation was to determine if the pre-implant assessment measures are appropriate and set at the correct level for determining candidacy, the future analyses will determine whether the speech perception cut-off point for candidacy should be adjusted and whether other more challenging measures should be used in the candidacy evaluation.

keywords:

• Cochlear implant candidacy
• AB words
• BKB sentences

Introduction

Current practice in the United Kingdom (UK) for assessing candidacy for cochlear implants (CIs) was determined based on research conducted over 10 years ago (UK study group, 2004a). The project involved the collaboration of 13 UK CI centres working with academics from the Medical Research Council Institute of Hearing Research to determine the appropriate limits for candidacy for CIs in adults. All study centres used the same testing apparatus, and the same standard measures for assessing CI candidacy. This included the Bamford, Kowal, and Bench (BKB) sentences presented in quiet and without any visual information from the talker's face or lips.

The vocabulary used in the development of the BKB sentences was taken from the utterances of eight to 15-year-old partially hearing children when describing everyday scenarios. The sentences could therefore be used with the majority of implant users pre- and post-implant. The scoring of BKBs was based on the percentage of keywords correct; the recommended cut-off criterion for falling within the CI candidacy range was set at <50% correct. This criterion was adopted by the National Institute for Health and Care Excellence (NICE) in 2009 (NICE, 2009) and is in place today.

The use of monosyllabic word tests or speech in noise assessments to assess outcomes post-implantation was uncommon in the UK in 2004, because the post-operative performance of implantees listening to sentences in quiet was rarely, if ever, constrained by ceiling effects. With improvements in patient outcomes over the years, the assessment of adult patients has become more sophisticated and sensitive to improvements in post-operative performance levels. These changes in outcome over the years have been attributed to changes in candidacy with individuals with more residual hearing are being implanted, improved surgical practice, better device fitting, more sophisticated rehabilitation approaches and device development (Blamey et al., 2013).

In the UK, CI teams struggle with the current candidacy criteria because they do not permit all individuals whom they currently consider as likely candidates to be implanted. In particular, two issues arise from the use of BKB sentences alone for assessing candidacy. The first issue is that they use highly predictable materials because the sentences contain contextual information. Some candidates may only have minimal access to speech cues but because they have refined cognitive skills, they are able to work out the content of the sentence in spite of the severe loss of auditory information. While this skill is unlikely to provide sufficient benefit for speech perception in everyday complex listening situations, it may result in too high a test score for them to be eligible for implantation.

The second issue is that BKB sentence scores do not reflect the hearing abilities of some patients at the lower end of the performance scale. Individuals who do not speak English as their primary language may not have the same abilities as a native English speaker to fill in gaps in predictable sentences because their linguistic knowledge is poorer. The same issue arises in pre-lingually deaf adults who may not have a high levels of language sufficient to be able to get a representative score on the BKB sentences. Craddock et al. (2016) suggested that the CUNY audio-visual sentences might be more appropriate for this group.

Doran and Jenkinson (2016) reported on the additional criteria used by the Welsh CI teams. Individuals who have borderline BKB scores (<60%) and had low-frequency residual hearing are evaluated with a second speech perception measure, the Arthur Boothroyd (AB) monosyllabic word test. Doran and Jenkinson (2016) based their cut-off levels on the extremely conservative level of the 10th percentile of post-operative scores for both BKB sentences (50%) and AB words (15%) and used these as their criteria, this approach followed a procedure recommended by Dowell et al. (2004). The advantages of the AB words are that they are potentially a better measure of speech audibility because context cues are not available and are therefore not subject to the predictability effects associated with the BKB sentences. However, just like the BKB sentences, the confidence intervals for AB words scores are also extremely wide and dependent upon the number of items used (Thornton and Raffin, 1978) and setting an absolute single value for candidacy may not be appropriate. For example, a patient whose ‘true’ performance level is at 50% on the BKB sentences may actually achieve scores between approximately 40–60%. In Wales, the additional rule is applied for individuals who score greater than 50% on the BKB sentences; if their AB word score is less than 15% then they are considered to be potential candidates for CI, to be reviewed by the multi-disciplinary CI team who make candidacy recommendations based on a range of carefully conducted assessments.

Currently, the recommendation for scoring the AB words is to use the ‘word’ score; i.e. the proportion of whole words reported correctly. However, a score based on the proportion of phonemes reported correctly (‘phoneme’ score) may give a finer resolution of performance ability. This approach to scoring may also be more appropriate for those patients with English as a second language or pre-lingually deafened patients in whom the capacity to report whole words may not relate well to audibility.

It is known that the there is a strong linear relationship between BKB sentence and the AB word scores (Vickers et al., 2009). The use of both assessments to verify candidacy has great potential for improving the quality and reliability of the candidacy assessments. An additional assessment provides greater clinical confidence in the eligibility of an individual because of this ‘second look’ at performance. Exploratory analyses were therefore conducted to assess whether the 15% AB word score described by Doran and Jenkinson (2016) determines eligibility in a similar way to the 50% BKB sentences score.

Ten UK CI centres have been engaged in collecting clinical data to determine if the BKB sentence test is still the most appropriate pre-operative measure for determining CI candidacy for adults in the UK. The ultimate goal of the data collection is to determine if the actuarial model for candidacy for CIs in adults in the UK (UK study group 2004b) should be updated. However, the specific goal of this phase of the project was purely an exploratory analysis to evaluate the potential usefulness of the AB words.

Method

Study participant inclusion criteria

There were 134 data sets collated by December 1st 2015. Ninety seven of those had scores for both BKB sentences and AB words in quiet; 37 were missing data on one or both measures. This represented all data sets entered and verified at the point of analysis. Participants had all been seen by a participating adult implant programme and were able to undergo speech perception testing. Data was collected at pre-operative assessment and post-operatively at 3 and 9–12 months, however only pre-operative results are analysed here. Each centre registered the data collection with their individual Hospital Research and Development boards as part of a multi-site service evaluation for the British Cochlear Implant Group.

Speech perception test delivery

The BKB sentences and AB words were delivered in sound-treated rooms at the individual hospitals and presented at 70 dBSPL. The BKB sentences were scored by the number of key words correct and the AB word test was scored by both phoneme and word. For the AB words three lists of 10 words were used so 30 words were presented in total.

Demographic information

Data were also collected on home language, aetiology, duration of profound deafness and age at implantation.

Analysis

The scores on these measures were evaluated in two ways. First, the scores for all participants were reviewed to determine if their AB words or BKB sentences fell within the candidacy range proposed by Doran and Jenkinson (2016); i.e. <50% on BKB sentences and/or <15% on AB words. It should be noted that Doran and Jenkinson (2016) actually excluded individuals from their analysis based on specific demographic requirements (e.g. pre-lingual deafness or long duration of deafness) but for this analysis the goal was to determine how well the criteria worked with the entire group of implant candidates, so individuals were not excluded based on any demographic factors. If there were any anomalies in the speech perception scores the demographic factors were explored to determine if the findings could be explained and thus highlighting areas to be included in the final actuarial model.

Secondly, comparisons of the AB word and BKB sentence scores were made using Receiver operating characteristic (ROC) curve analysis. Patients were first classified as either eligible or ineligible based on their BKB sentences score, where eligible patients were those with a score less than 50% correct, in line with NICE guidance. An ROC curve was then fitted to this binary variable and their corresponding scores on the AB words test. Two analyses were performed: one using the word score and another using the phoneme score. These ROC curves were fit and compared using the pROC package for the R statistical programming environment (Robin et al., 2011). The ‘optimal’ word and phoneme cut-off scores for the AB word test were determined as those scores that simultaneously maximized the sensitivity and specificity of the test. In this context, sensitivity refers to the test's ability to identify eligible patients and specificity to its ability to identify ineligible patients.

Results

The mean age at implantation was 57 years (median 61 years; range 16–88 years) and the mean duration of deafness was 22.5 years (median 15 years; range 0–80 years). Eight individuals had English as an additional language (EAL) and five people used British Sign Language as their primary language. Table 1 shows the number of individuals who would pass or fail the BKB sentences (pass cut-off level was ≥50%) or the AB words (pass cut-off level was >15%). If the AB words were used as the only candidacy measure fewer people would fall within the candidacy range, i.e. 78 versus 85 people with the BKB sentence criteria. If both assessments were used together an additional one person would be considered for implantation on top of the current BKB cut-off at 50%.

Table 1 Number of individuals who would or would not fall within the implant candidacy range based on BKB and AB word cut-offs

Table 2 provides the demographic details of the individuals who had different pass/fail criteria based on the two measures.

Table 2 Demographic details for nine individuals who had different pass/fail criteria for BKBs and AB words

The single individual whose scores would fall within candidacy with the use of AB words is shown on the first row of Table 2. This individual was pre-lingually deafened and had a conductive component to the hearing loss. They were able to detect phonemes but could not identify the words. This individual did remarkably well with the BKB sentences possibly because of the vocabulary level of the materials being appropriate for children. The eight other individuals were those who passed the AB words and failed the BKB sentences, they either did not have English as their primary language or were older than the mean age of the group. Both aspects might make the audibility of phonemes and words easier than understanding the BKB sentences, where gap filling might be necessary to correctly perceive the sentences. Within the dataset forty individuals were over the UK retirement age (65-years-old).

It should also be noted that for some tasks the scores were borderline (between 50 and 60% on BKB sentences (six people) and between 15 and 33% on AB words (11 people)).

Fig. 1 plots the ROC curves for the word and phoneme scores derived from the AB words test. It would be straightforward to choose a very high AB word or phoneme score as a cut-off that would successfully identify all patients with BKB scores <50%; i.e. eligible patients. Similarly, it would be easy to choose a very low word/phoneme score as a cut-off that would successfully identify all patients with BKB scores ≥50%; i.e. ineligible patients. ROC curves are used to identify the cut-off score that tries to do both of these tasks simultaneously; i.e. the score that would classify patients as eligible or ineligible as similarly as possible to the NICE criterion level of <50% on the BKB sentence test. The closer the curves are to the top-left corner of the Fig. 1, the more similar they are to the BKB 50% cut-off at determining candidacy.

Figure 1 Receiver operating characteristic (ROC) curves for the word (solid curve) and phoneme (dashed curve) scores from the AB word test. The symbols mark the score that maximizes the sensitivity and specificity of each scoring method; i.e. the cut-off score that identifies eligible and non-eligible candidates as close as possible to the BKB sentence test using the 50% cut-off criterion

Both word and phoneme scores showed high sensitivity (accuracy with identifying eligible patients) and specificity (accuracy with identifying ineligible patients). The optimal cut-off word score was found to be 18.5% (sensitivity 90.9%, specificity 96.3%) and the optimal phoneme score was 34.5% (sensitivity 100%, specificity 93.0%). A paired comparison of the two curves using DeLong's test (Delong et al., 1988) revealed no significant difference (Z = −1.02, P = 0.31), suggesting that both word and phoneme scores assessed candidacy in a similar way.

Discussion

This exploratory analysis was conducted to determine if the data suggest that the addition of the AB words to the candidacy test battery or even the replacement of BKB sentences with AB words would be appropriate. The candidacy criteria recommended by Doran and Jenkinson (2016) were used to determine how the numbers of candidates would change if the AB word criteria were added to the BKB criteria or if the AB words were used alone. The analysis showed that their suggested cut-off level for AB words was conservative and that fewer people would fall within the candidacy range if only the AB words (<15%) were used. This might suggest that the relationship between the two measures was not appropriately set. Further exploratory analyses based on ROC curve analysis produced an AB word cut-off (18.5%) which was marginally higher but very similar to the 15% suggested. This shows that the BKB and AB tests produce quite consistent data when assessing candidacy across different CI centres and patient catchment areas.

At the current 50% BKB cut-off, both word and phoneme scores from the AB word test seem to be highly sensitive and specific. This suggests that they may be suitable replacements/or additions to the test battery for those individuals where there might be concerns over the validity of the BKB sentence test; e.g. EAL, pre-lingually deafened, first-language BSL user.

Current results suggest that it may be productive to use ROC analysis to compare BKB and AB word tests. The future analysis of the BCIG service evaluation data will replicate the current approach with a larger sample of data. Both pre- and post-implantation data can be used to ensure that the entire performance range is covered. A larger sample would also allow the analysis to be conducted on only the subset of post-lingually deafened patients with English as a first-language to eliminate any concerns about the validity of BKB scores. The cut-off produced by that data could then be applied to the sub-population of patients where BKB scores may not be valid to see whether AB word scores produce a similar proportion of eligible candidates.

This analysis has demonstrated that the AB words may have potential as a measure in the candidacy test battery. There is no evidence at this stage to suggest that it should replace the BKB sentences, but the exploration did identify cases (EAL, older adults, BSL users and individuals with high levels of cognitive processing) where it could be beneficial as an additional tool for assessment. The absolute values that are appropriate for candidacy and the actuarial equation will re-evaluated in the future analysis.

Disclaimer statements

Contributors None.

Funding None.

Conflicts of interest None.

Ethics approval Service evaluation approval at individual hospitals.