Reliability of data on percent consonants correct and its associated quality indicator in the Swedish cleft lip and palate registry

Abstract

Background

Data in national health care quality registries must be valid and reliable in order to enable open comparisons of results.

Aim

To assess the reliability of data on percent consonants correct (PCC) and its associated quality indicator ≥86% correct consonants in the Swedish quality registry for patients born with cleft lip and palate (CLP) registry.

Methods

Six independent speech-language pathologists re-assessed the audio recordings of 96 five-year-olds with PCC data in the CLP registry. Target consonants of a single-word picture-naming test were phonetically transcribed, and PCC was calculated. The reliability of PCC data was assessed with the intraclass correlation coefficient (ICC). The reliability of the quality indicator ≥86% correct consonants was assessed with point-by-point percentage agreement and Cohen’s kappa.

Results

Intra- and inter-judge agreement for PCC was excellent with ICCs above 0.9, and so was the agreement of data from the CLP registry and the six judges’ re-assessments. The percentage agreement between all judges and the CLP registry for the quality indicator ≥86% correct consonants was poor (67%). However, in 88% of the cases, results from four judges and the CLP registry agreed, corresponding to good agreement. The mean of all kappa values for six judges and the CLP registry corresponded to good agreement (0.72).

Conclusions

The results indicate the PCC data in the CLP registry and the quality indicator ≥86% correct consonants to be reliable. When differences in outcome between treatment centres are detected, the raw data collected should always be re-examined before drawing definitive conclusions.

Keywords:

• Percent consonants correct
• cleft lip and palate
• reliability
• quality registry

Introduction

The objective of Swedish health care quality registries is to enable comparison and open reporting of results between different counties and hospitals, and to promote quality control, research, and improvement of treatment [1]. To fulfil this purpose, the registered data must be valid and reliable. In this study, the reliability of data on percent consonants correct (PCC) in the Swedish quality registry for patients born with cleft lip and palate (CLP) registry and its associated quality indicator, ≥86% correct consonants, was assessed.

About one out of 500 Swedish children is born with CLP [2]. Depending on the extent of the cleft, CLP can affect several structures and functions to varying degrees. If the cleft involves the palate, palatal surgery is needed to create conditions for typical speech development. We still do not know the best timing, technique, or staging for surgery [3–5], and different procedures for palatal repair are used globally [6]. In Sweden, primary palatal surgery is performed in one stage between 9 and 15 months of age, or two stages, with soft palate closure at 6 months of age and hard palate closure at about 2 years of age [7]. Even though the cleft in the palate is closed at an early age, speech deviances may arise, such as oral and non-oral misarticulations, and also speech symptoms directly related to insufficient velopharyngeal function, such as hypernasality, audible nasal air leakage, and weak articulation [8]. Phonological processes are also more frequent in children with cleft palate than without at 5 years of age [9].

Since 2016, the CLP registry annually reports treatment results from all six Swedish CLP centres [7]. Data on babbling and early speech are recorded at 18 months of age, and speech outcome is recorded at follow-up at 5, 10, 16, and 19 years of age. The speech variables in the registry at age 5 are PCC, percent non-oral errors, and perceived velopharyngeal competence (VPC) [7]. In addition, results from the Intelligibility in Context Scale [10], a caregiver-reported validated measure of functional intelligibility, are registered. The working group within the framework of the International Consortium of Health Outcome Measures has recommended PCC, VPC, and the Intelligibility in Context Scale for cleft palate speech evaluation [11]. For a description of all the registered variables in the CLP registry, see Klintö et al. [7].

To assess the severity of speech disorders, PCC was originally developed by Shriberg and Kwiatkowski [12] as a measure of the proportion of correctly produced consonants in transcriptions of conversational speech. In PCC, both phonetic and phonological errors are scored as incorrect. Over the years, researchers have used PCC and modifications of PCC in several studies when evaluating cleft palate speech; see, for example, [5,13–15]. The PCC outcome in the CLP registry is based on target consonants in a single-word picture-naming test included in the Swedish test for assessment of nasality and articulation – SVANTE [8]. The design of SVANTE follows principles of cross-linguistic speech assessment [16]. As in GOS.SP.ASS/CAPS-A used for English-speaking patients with cleft palate [17,18], speech patterns are categorised according to the place of articulation, and if they are active or passive [19,20]. Only active speech characteristics, i.e. articulatory and phonological errors, are scored as incorrect in the PCC measure used in the CLP registry. Thus, passive cleft speech characteristics are not included in the assessment of PCC [8].

Agreement within and between assessors is crucial for the reliability and usefulness of results in research on cleft palate speech [21,22]. Usually, agreement is higher in PCC than point-by-point transcriber agreement since when calculating PCC, correct or incorrect are the only options, whereas, in transcription, there are multiple options [23]. In a previous study, inter-judge agreement for PCC, based on 30 target consonants in the SVANTE minimum standard set for cross-linguistic comparison [8], was considered reliable [24]. The PCC-based binary quality indicator, ≥86% correct consonants, was also assessed. The cut-off of 86% was chosen since it corresponds to −2 standard deviations (SD) from the norm data of five-year-olds without CLP [8]. However, in the previous study, this quality indicator was not considered fully reliable [24]. For children born in 2014 and onwards, it was decided to base the PCC outcome in the CLP registry on 59 target consonants in the SVANTE single-word picture-naming test, instead of 30, as an attempt to increase the reliability of the quality indicator. Since 2019, the quality indicators Absence of non-oral speech errors and Competent or marginally incompetent VPC [24] are published online [25]. If the quality indicator ≥86% correct consonants also proves to be reliable, this quality indicator may also be published online.

Objective

The main objective was to investigate the reliability of data on PCC and its associated quality indicator in the Swedish CLP registry. Research questions were:

1. Is data on PCC in the CLP registry and the quality indicator ≥86% correct consonants reliable when based on 59 target consonants?

2. Is there any difference in agreement of PCC between and within judges when the calculation is based on 30 or 59 target consonants?

Methods

Participants

All children born in 2014 with a cleft palate with or without cleft lip, with audio recordings and registered PCC data in the Swedish CLP registry at age 5 years participated. This resulted in 96 children.

Ethical approval

The Ethics Review Authority in Sweden approved the study (reference no. 2020-00227). The consultation group for quality registries, care databases, and preparation in the Skåne Region approved data access (reference no. 123-20).

Registry data

Data were retrieved from the CLP registry via the Record Centre South, Lund, Sweden.

Setting and recording equipment

CLP team speech-language pathologists (SLPs) audio recorded the speech in connection with the routine follow-up at 5 years ± 6 months, at one of six regional CLP centres, in a university hospital setting. Speech recordings were performed in a quiet room using an audio recorder (Zoom H4n, Hauppauge, NY; TASCAM HD-P2, Montebello, CA) or a PC with Soundswell software (Saven Hitech, Stockholm, Sweden), in combination with a condenser microphone (Røde NT4, Sydney, Australia; Sony ECM-MS957, Tokyo, Japan; Pearl CC3, Åstorp, Sweden; Sennheiser MKE 2 P-C, Hannover, Germany).

Speech material

SVANTE [8] was used for elicitation of speech. SVANTE consists of a single-word picture-naming test, a sentence repetition task, and a thematic picture to elicit conversational speech. In this study, only data based on the single-word picture-naming test were investigated. The 59 first words were composed of one or two syllables, and no nasal consonants. Most words contained only one high-pressure consonant (i.e. stop or the fricatives /s/ and /f/), which was the target consonant, placed in a linguistically stressed position. In order to obtain a sufficient number of words, a few words with two pressure consonants were also included. All target consonants were realised three times in word-initial position, and most of them also twice in word-medial and word-final positions respectively in different words to detect possible variations in production in the child [8].

Phonetic transcription and calculation of PCC

The same SLP who met the child and audio recorded the speech also performed a perceptual assessment from the audio recording and recorded the speech results in the CLP registry. The SLPs transcribed the target consonants of the 59 first words in the SVANTE single-word naming test [8], with semi-narrow transcription according to the International Phonetic Alphabet [26]. If the phonetic symbol differed from the target phonetic symbol, the production was scored as incorrect. Distorted s-sounds were also scored as errors. Audible nasal air leakage and reduced pressure on consonants were not scored as errors. From the transcriptions of the target consonants, the SLPs calculated PCC by dividing the number of correct target consonants with the total number of elicited target consonants [8].

Re-assessments of recordings

Audio recordings of 96 children who had come for their standard assessments at 5 years of age were used. Personal information and other speech material than the SVANTE single-word picture-naming test were deleted from the recordings. Twenty-nine randomly chosen recordings (30%) were duplicated for investigation of intra-judge agreement.

Six independent SLPs specialised in cleft palate speech, one from each participating CLP centre, assessed the audio-recordings using headphones (AKG K182, AKG K271 MkII, Creative Aurvana Live, Denon AH-D1001, Sennheiser HD 280 pro, Yamaha HPH-MT7).

They transcribed the first 59 target consonants of the SVANTE single-word picture-naming test [8] according to the International Phonetic Alphabet [26]; an additional five words with nasal consonants were also transcribed [8]. From the transcriptions, they calculated PCC based on the 59 first target consonants as described above. They also calculated PCC based on the SVANTE minimum standard word set, containing 26 target consonants with high intraoral pressure, two target consonants with low intraoral pressure, and two nasal target consonants [8]. The same scoring guidelines were used for calculation of PCC based on both 59 and 30 target consonants.

Statistical analysis

Absolute agreement between the data in the CLP registry and re-assessments for PCC and intra- and inter-judge agreement for PCC was calculated using the single-measures intraclass correlation coefficient (ICC) with a two-way mixed-effects model. The agreement of the binary quality indicator ≥86% correct consonants was assessed by point-by-point percentage agreement and Cohen’s kappa. In order to assess the agreement of all SLPs and registry data at the same time, the mean of all kappa values from pair-wise comparisons between judges and between individual judges and registry data was calculated [27]. The results were interpreted according to Cicchetti [28] (Table 1).

Table 1. Interpretation of levels of kappa (k) statistics, intraclass coefficients (ICCs), and percentage agreement according to Cicchetti [28].

Levels of k or ICC	Levels of observed agreement (%)	Levels of clinical or practical significance
<.40	<70	Poor
.40–.59	70–79	Fair
.60–.74	80–89	Good
.75–1.00	90–100	Excellent

Results

Reliability of re-assessments

PCC

Intra-judge agreement for PCC based on 59 target consonants for 29 randomly selected children was excellent (Table 2). ICC values varied between 0.97 and 0.99, with 95% confidence intervals (CIs) between 0.93 and 0.99. The corresponding ICC values for PCC based on 30 target consonants were excellent, but somewhat lower, and varied between 0.94 and 0.97 with CIs between 0.86 and 0.99 (Table 2).

Table 2. Intrajudge agreement for the dichotomised quality indicator ≥86% correct consonants as calculated with Cohen’s kappa and percentage agreement, and percent consonants correct (PCC) based on 59 (PCC 59) and 30 (PCC 30) target consonants respectively as calculated with intraclass correlation coefficient (ICC).

	Quality indicator			PCC 59		PCC 30
	%	k	95% CI	ICC	95% CI	ICC	95% CI
SLP1–SLP1	97	.93	.79–1	.98	.97–.99	.97	.93–.99
SLP2–SLP2	86	.67	.38–.97	.99	.97–.99	.97	.93–.98
SLP3–SLP3	90	.79	.57–1	.98	.95–.99	.94	.86–.97
SLP4–SLP4	97	.93	.78–1	.97	.94–.99	.96	.92–.98
SLP5–SLP5	93	.86	.68–1	.97	.93–.99	.95	.9–.98
SLP6–SLP6	93	.86	.67–1	.98	.96–.99	.96	.92–.98

CI: confidence interval; SLP: speech-language pathologist.

Inter-judge agreement for PCC based on 59 target consonants was excellent (ICC 0.92; CI 0.87‒0.95), and so was agreement for PCC based on 30 target consonants, although somewhat lower (ICC 0.9; CI 0.84‒0.93). Inter-judge agreement on the number of reported elicited target consonants was excellent when PCC was based on 59 target consonants (ICC 0.97, CI 0.95‒0.98) and 30 target consonants (ICC 0.94, CI 0.92‒0.96).

The quality indicator ≥86% correct consonants

Intra-judge agreement calculated with percentage agreement and Cohen’s kappa among judges for the quality indicator ≥86% correct consonants, based on 59 target consonants, was excellent in five cases and good in one case (Table 2). Percentage agreement between judges was excellent in five cases, good in seven cases, and fair in three cases (Table 3). Inter-judge agreement calculated with Cohen’s kappa was excellent in six cases, good in six cases, and fair in three cases (Table 3).

Table 3. Agreement for the quality indicator ≥86% correct consonants, when pair-wise comparisons of reassessments by speech-language pathologists (SLP1, SLP2, SLP3, SLP4, SLP5, and SLP6) and data from the Swedish Quality Registry for cleft lip and palate registry (QR) were performed, as calculated with percentage agreement and Cohen’s kappa.

	%	k	95% CI
SLP1–SLP2	78	.58	.42–.74
SLP1–SLP3	95	.89	.8–.98
SLP1–SLP4	94	.86	.76–.97
SLP1–SLP5	93	.85	.74–.96
SLP1–SLP6	83	.67	.53–.82
SLP2–SLP3	83	.67	.53–.82
SLP2–SLP4	78	.58	.42–.74
SLP2–SLP5	81	.63	.48–.79
SLP2–SLP6	82	.64	.49–.8
SLP3–SLP4	89	.76	.62–.89
SLP3–SLP5	92	.83	.72–.94
SLP3–SLP6	84	.69	.55–.83
SLP4–SLP5	93	.85	.74–.96
SLP4–SLP6	79	.59	.43–.75
SLP5–SLP6	87	.73	.6–.87
QR-SLP1	89	.77	.64–.9
QR-SLP2	79	.59	.43–.75
QR-SLP3	90	.79	.66–.91
QR-SLP4	87	.72	.58–.86
QR-SLP5	90	.79	.66–.91
QR-SLP6	80	.61	.45–.76
Mean	–	.72	.58–.85

CI: confidence interval.

Comparison of registry data and data from re-assessments

PCC

Agreement of the data from the CLP registry for PCC based on 59 target consonants and the six SLPs was excellent, with an ICC value of 0.9 (CI 0.87‒0.94). Furthermore, agreement between the CLP registry data and the six SLPs for the number of reported elicited target consonants for PCC based on 59 target consonants was excellent (ICC 0.83, CI 0.79‒0.87).

The quality indicator ≥86% correct consonants

Percentage agreement between individual SLPs and the CLP registry for the quality indicator ≥86% correct consonants was excellent in two cases, good in three cases, and fair in one case (Table 3). When calculated with Cohen’s kappa, agreement between the CLP registry and the individual SLPs was excellent in three cases, good in two cases, and fair in one case (Table 3).

When percentage agreement was calculated for the quality indicator ≥86% correct consonants, including all SLPs and the CLP registry at the same time, agreement was poor (67%). In 79% of the cases, five SLPs agreed with the CLP registry, corresponding to fair agreement, and in 88% of the cases, four SLPs and the CLP registry agreed, corresponding to good agreement. The mean of all kappa values was 0.72, which corresponds to good agreement.

Discussion

Reliability of PCC

In the present study, intra- and inter-judge agreement for PCC based on 59 consonants was excellent. Thus, the PCC variable used in the CLP registry can be considered a reliable overall measure of articulatory and phonological accuracy when evaluating cleft palate speech. Intra- and inter-judge agreement for PCC based on 30 consonants was also excellent, although somewhat lower. One explanation for the better agreement when 59 target consonants were used may be that the effect of chance agreement decreases, i.e. the results on agreement become more stable if the calculation is based on more data points.

Reliability of the quality indicator ≥86% correct consonants

When the results from all the judges and the CLP registry were compared at the same time, agreement was poor (67%) for the quality indicator ≥86% correct consonants. However, when the number of judges was decreased to five and four the percentage agreement between judges and the registry increased to 79% and 88%, respectively. Thus, the number of judges may affect the degree of agreement. A disadvantage with percentage agreement is that it does not take chance agreement into account [27]. For assessment of inter-judge agreement, including several judges, the mean of the kappa values from all pair-wise comparisons may be calculated [27]. In the present study, this resulted in good agreement, with the lower limit of the 95% CI just below the interval of good agreement and the upper limit within the interval of excellent agreement.

The impact of internal standards of judges

Although the reliability of data on PCC and its associated quality indicator was considered reliable, the judges’ individual internal standards when scoring PCC may affect the results to a certain degree. Thus, in the scientific evaluation of cleft palate speech, the use of several independent trained listeners is recommended; see, for example, Sell [22], and not only one listener as in the CLP registry.

It cannot be ruled out that the internal standards of what to score as correct or incorrect in PCC may vary for a judge over time. Furthermore, new SLPs in the CLP teams need training before registering data in a CLP registry. We have therefore developed national guidelines to maintain a high degree of agreement of the speech assessments on which CLP registry data are based. For SLPs inexperienced in rating cleft palate speech, it is mandatory to complete introductory training in cleft palate speech analysis under the supervision of an experienced colleague. All SLPs who collect and analyse data for the CLP registry should be given the opportunity by the employer to participate in annual national meetings for CLP SLPs, where listener calibration among SLPs is carried out. Furthermore, at each regional CLP centre, the SLPs should have the opportunity to perform listener calibration on a regular basis.

The fact that different judges may have different internal standards when performing perceptual speech evaluation, underlines the importance of re-analysing the raw data if differences between CLP centres are detected based on speech data from a national health care quality registry. This should be done before drawing definitive conclusions about treatment results and whether there is a need to change treatment protocols.

Other considerations when using PCC for speech evaluation

Sell and Sweeney [30] highlighted the need to report agreement of the number of elicited target consonants, since it may vary between judges and affect the results. In the present study, inter-judge agreement on the number of reported elicited target consonants was excellent when PCC was based both on 59 target consonants and on 30 target consonants.

The PCC measure used in the CLP registry does not adjust for age-appropriate speech deviances. In Swedish-speaking children, simplifications of /s/ are age-appropriate at 5 years of age [8]. /s/ constitutes about 10% of the 59 target consonants in SVANTE, and disagreement among Swedish-speaking SLPs in the transcription of /s/ has previously been observed [24]. As speech deviances decrease with age [8,29], agreement among SLPs performing perceptual assessment of cleft palate speech is presumed to increase with the age of the patient. Thus, at the next follow-up included in the CLP registry, at 10 years of age, an even higher inter-judge agreement for PCC and the quality indicator may be expected.

The risk of chance agreement is high when calculating PCC, and this limitation has been highlighted by other researchers [23,30]. One also needs to be aware of PCC being a rough measure of speech compared to measures used to classify different types of speech errors into different categories. Different speech errors can have different degrees of severity, and this is not reflected in the PCC measure [30]. Despite these limitations, PCC was included in the CLP registry as an overall measure of articulatory and phonological accuracy which is robust and with as high reliability as possible. When performing open comparisons of clinical data among health care units, the reliability of the variables used has high priority [31]. In the CLP registry, we have refrained from using variables that provide more detailed information, as it may then be more difficult to achieve high inter-judge agreement for speech data.

Advantages of using registry data in health care research

Results from health care registries are not derived from the stringent methodology used in well-designed/high-quality prospective scientific studies [22]. Despite this, there are advantages of using high-quality registry data in health care research. This is especially evident in research areas where patient groups are small and heterogeneous, such as the CLP population, where data must be collected for a long period of time to obtain a larger set of children. When collecting data over a long period of time, it is difficult to keep other factors constant, such as surgical methods and methods for data collection, and this may affect outcomes [29]. Thus, multicentre studies are warranted to recruit participants for study within a reasonable time-period. A well-designed national health care registry, such as the CLP registry, with a high coverage and reporting degree [7], can make a valuable contribution to knowledge about a specific patient group since it allows for research on larger groups of patients. Results from CLP registry studies may indicate speech outcomes in different subgroups of children with cleft palate, and if differences are discovered, one can proceed with in-depth research studies in the field.

The CRANE database in the United Kingdom, which started in 2000, has published several health care process-related studies, for example, on hospital care [32] and grommet surgery in children with orofacial clefts [33]. Recently, treatment results from the CRANE database of maxillary growth and speech in 5-year-olds with unilateral CLP were published [34]. Unfortunately, the speech data in the CRANE database and the CLP registry are not comparable, although the speech assessment procedures are based on the same principles for speech analysis [19,20]. The Norwegian CLP registry that started in 2011 uses the same variables and quality indicators for speech evaluation as the Swedish CLP registry [35]. Evaluation of the reliability of speech data in the Norwegian CLP registry is ongoing [35], and in the future, it may be possible to compare data between the Swedish and Norwegian CLP registries.

Conclusions

The results indicate PCC data in the Swedish CLP registry and the quality indicator ≥86% correct consonants to be reliable. Nevertheless, different judges may have different internal standards when performing perceptual speech evaluation, such as phonetic transcription and PCC scoring. Therefore, it is important to re-analyse raw data if differences in results are detected between CLP centres, before drawing definitive conclusions about treatment results.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Kristina Klintö

Kristina Klintö is a SLP, Associate Professor, and project manager for the Swedish CLP registry. She is responsible for speech variables, and collection and analysis of speech registry data from Skåne University Hospital. She performed perceptual assessment and was responsible for statistical calculations and results. She also wrote the first draft of the manuscript.

Emilie Hagberg

Emilie Hagberg is a SLP, and PhD student. She is responsible for speech variables, and collection and analysis of speech registry data from Karolinska University Hospital. She performed perceptual assessment and contributed to the writing of the manuscript.

Christina Havstam

Christina Havstam is a SLP, and Associate Professor. She is responsible for speech variables, and collection and analysis of speech registry data from Sahlgrenska University Hospital. She contributed to the writing of the manuscript.

Cecilia Nelli

Cecilia Nelli is a SLP and Master of Science. She is responsible for speech variables, and collection and analysis of speech registry data from Linköping University Hospital. She performed perceptual assessment and contributed to the writing of the manuscript.

Åsa Okhiria

Åsa Okhiria is a SLP, and PhD student. She is responsible for speech variables, and collection and analysis of speech registry data from Uppsala University Hospital. She performed perceptual assessment and contributed to the writing of the manuscript.

Karin Brunnegård

Karin Brunnegård is a SLP, and PhD. She is responsible for speech variables, and collection and analysis of speech registry data from University Hospital of Umeå. She performed perceptual assessment and contributed to the writing of the manuscript.