Speech and communication classification of children with cerebral palsy: Novice rater agreement and clinical utility

Abstract

Purpose

To examine novice inter-rater agreement and clinical utility perspectives for speech and communication classification of children with cerebral palsy (CP).

Method

Twenty-one clinicians (speech-language pathologists [SLPs] n = 11; physiotherapists [PTs] n = 5; occupational therapists [OTs] n = 5) novice to the Viking Speech Scale (VSS), Functional Communication Classification System (FCCS), and Communication Function Classification System (CFCS) rated eight unfamiliar children with CP (8–16 years) following classification orientation. Inter-rater agreement was examined between (a) novices, (b) novice SLPs vs. PTs and OTs, and (c) novice vs. expert (kappa statistics). Utility perceptions were scored regarding classification terminology, ease of use, assistive decision-making resources, and construct validity and were analysed using Kruskal–Wallis H-tests.

Result

Rating agreement between novices was substantial (VSS, k = 0.72, 95% CI [0.53–0.92]) to moderate (FCCS, k = 0.44, 95% CI [0.23–0.65]; CFCS, k = 0.45, 95% CI [0.18–0.71]), and almost perfect between novice and expert ratings (VSS, k_w = 0.89, 95% CI [0.86–0.92]; FCCS, k_w = 0.89, 95% CI [0.86–0.92]; CFCS, k_w = 0.86, 95% CI [0.82–0.91]). Statistically significant differences, presented highest to lowest, were found for clinical utility: terminology (VSS, FCCS, CFCS; p = 0.02), assistive decision-making resources (FCCS, VSS, CFCS; p = 0.009), and construct validity (FCCS, CFCS, VSS; p < 0.001).

Conclusion

Novice raters achieved substantial agreement for speech classification, supporting utilisation in clinical, research, and CP register activities. Orientation to communication classification constructs, content, and instructions is recommended for novice raters.

Keywords:

• Cerebral palsy
• children
• Functional Communication Classification System (FCCS)
• Communication Function Classification System (CFCS)
• Viking Speech Scale (VSS)

Introduction

Cerebral palsy (CP) is the most common physical disability in childhood (Oskoui, 2016). Interprofessional practice is recommended to support each child’s functional potential (Damiano et al., 2021). Effective team practice requires a shared language to describe and understand a child’s everyday performance. From the late 1990s, functional classification systems were introduced for children and youth with CP to describe levels of performance in gross motor (Palisano et al., 1997, 2008), fine motor (Eliasson et al., 2006), eating and drinking (Sellers et al., 2014), motor speech (Pennington et al., 2013), and communication skills (Barty et al., 2016; Caynes et al., 2019, 2021; Hidecker et al., 2011). Functional classification replaces more general terms previously used to describe CP severity (e.g. mild, moderate, severe), with the aim to provide a consistent and shared language between families, clinicians, and researchers to describe child function, support intervention selection, and guide long-term planning for children with CP (Rosenbaum et al., 2014). In addition, a shared language is vital for monitoring population-based cohorts in surveillance research (Paulson & Vargus-Adams, 2017) and identifying relationships between communication and functional outcomes in other types of research (McCartney, 2019). Functional classification systems typically consist of four to five level descriptions for everyday activity, ranging from least to most restricted function. Communication performance is assessed from direct clinical observation, and/or consultation with a caregiver, and/or case notes (Pennington, 2016).

This paper focuses on two areas of function, motor speech and communication function, and the corresponding classifications. Data from population-based cohorts indicates that over half of school-aged children with CP present with a motor speech impairment (Nordberg et al., 2013; Påhlman et al., 2019), however, prevalence has been found to be much higher when standardised speech and perceptual measures are used (Mei et al., 2014, 2020). Mei et al. (2020) reported that 82% of 4- to 6-year-old children with CP recruited from a population-based CP registry from one Australian state (n = 84) presented with delayed or disordered speech production. Numbers with communication difficulties vary. Parkes et al. (2010) reported that 42% of young children with CP (n = 1357, median age 5.1 years) experienced expressive communication difficulties based on parent interviews, whereas Mei et al. (2016) reported 61% of 5- and 6-year-olds in a state-based cohort (n = 84) presented with a receptive and/or expressive language impairment based on standardised assessment.

The multifaceted nature of communication function has been captured by three classifications for children and youth with CP. The Viking Speech Scale (VSS; Pennington et al., 2013) describes motor speech intelligibility and its use has been supported by research led by the Surveillance of Cerebral Palsy in Europe (SCPE; Virella et al., 2016). The Functional Communication Classification System (FCCS; Barty et al., 2016; Caynes et al., 2019, 2021) and the Communication Function Classification System (CFCS; Hidecker et al., 2011) describe everyday communication performance with familiar and unfamiliar partners using typical communication modes including augmentative and alternative communication (AAC). Together, these systems provide a holistic description by describing different aspects of communication function. The FCCS classifies expressive communication function according to the variety of communication purposes expressed, level of independence, and overall effectiveness in daily interactions across partners and settings. The CFCS classifies the combination of expressive and receptive communication function using a “global judgement” of a child’s ability to alternate sender and receiver roles at a comfortable pace in daily interactions across partners.

Whilst there is some overlap in what the FCCS and CFCS classify (i.e. expressive communication performance), there are differences. Whereas the FCCS focuses on functional communication for daily activity and participation, the CFCS places emphasis on communication function, that is, sender/receiver skills and the pace of communication. Functional communication as classified by the FCCS involves the ability to express a wide range of communication purposes, such as to request, label, describe, comment, question, direct, express humour, and express emotion. Hence there is a benefit in using both tools in practice, particularly given the time efficient way they can be applied and can be classified using the same, single, quality observation sample. Both classifications take ∼5–10 min to apply when the relevant background information is available and/or completed in collaboration with a parent.

The VSS, FCCS, and CFCS describe different aspects of communication performance. Pennington et al. (2020) completed research that confirms the divergent validity of these classifications, which means that each classification contributes different information on a child’s communication performance and profile, which supports the use of all three systems for a holistic description of communication. For this reason, the combined use of these classifications has been an international recommendation (Pennington et al., 2020; Virella et al., 2016) to ensure adequate data is captured regarding communication development and impairment-related data for CP registers and to be used alongside other CP classifications to form part of the common dataset for CP (Pennington et al., 2020).

For optimal clinical practice, it is essential that both experienced and novice raters can use these classification scales to: (a) classify speech and communication performance accurately; (b) identify potential speech and communication limitations and barriers with respect to production (speech and AAC), purposes expressed, pace, partners (familiar and unfamiliar), and places (home, school, and community); and (c) identify if extra supports are required (e.g. AAC, communication partner training). To support translation to practice, it is important that healthcare professionals can apply each classification easily and accurately, especially when they are novice users of speech and communication classification scales. Supplementary Appendix 1 provides a summary of the three classifications including tool purpose, content, administration, and intended population age range for use.

Preliminary evidence of the agreement between speech-language pathologists (SLPs) and other healthcare professionals who were novice in using communication classifications for children with CP was provided by research conducted in the development of each tool. Reliability for the VSS, FCCS, and CFCS were initially examined in English. Inter-rater reliability between SLPs and other healthcare professionals has been reported as moderate to substantial (k = 0.58–0.78) for the VSS for children with CP aged 4 to 13 years (Pennington et al., 2013), almost perfect (k = 0.94) for the FCCS for children with CP aged 4 and 5 years (Barty et al., 2016), and substantial (k = 0.66–0.77) for the CFCS for children with CP aged 2 to 18 years (Hidecker et al., 2011). The SCPE further examined the psychometric properties of the VSS, FCCS, and CFCS using scales translated into Danish, Latvian, Lithuanian, Norwegian, European Portuguese, European Spanish, and Swedish by comparing ratings from SLPs (n = 143) and other healthcare professionals (n = 146) for children with CP aged 4–13 years (Virella et al., 2016). Inter-rater agreement was similar across tools (VSS, k = 0.51; FCCS, k = 0.42; CFCS, k = 0.37; CFCS-SCPE version, k = 0.40; Virella et al., 2016). Although the SCPE study found these tools to be valid and reliable, inter-rater agreement was only fair (CFCS, CFCS-SCPE) to moderate (FCCS, VSS). Thus, there is a need for further research that investigates inter-rater agreement to support accurate speech and communication classification of children with CP, especially for clinicians who are novice raters, i.e. those who are inexperienced with using communication classification systems. Furthermore, research to date has obtained data from SLPs and other healthcare professionals who were observing and rating familiar children, i.e. children from their own caseloads. It is unclear whether the agreement is similar when SLPs and other healthcare professionals observe and rate unfamiliar children. In addition, previous research has not included separate analyses for the agreement between novice raters (agreement) vs. between novice and expert raters (accuracy). Further research is required to address these gaps for all three scales.

The clinical utility of communication classification tools is also critical to consider, for example, factors such as ease and duration of application, rating systems, and ability to interpret descriptions of everyday speech and communication function. In the SCPE study, over half of SLPs and other healthcare professionals considered the three classifications easy to apply, with other healthcare professionals rating the VSS and FCCS as easier to apply than the CFCS (Virella et al., 2016). The VSS and FCCS were considered to provide appropriate descriptions of speech intelligibility and expressive communication, respectively. However, no tool alone was considered to adequately describe the full activity of communication. At that time the SCPE authors recommended the use of the VSS, in conjunction with either the FCCS or CFCS, to classify all aspects of communication (Virella et al., 2016). More recent research has recommended that concurrent use of all three classifications is beneficial to obtain a holistic description of speech and communication function (Pennington et al., 2020).

The aim of this study was to examine the agreement and accuracy of novice raters when classifying unfamiliar children using the VSS, FCCS, and CFCS. Objectives were to determine: (a) agreement between novice ratings across disciplines (SLPs, physiotherapists [PTs], occupational therapists [OTs]); (b) agreement between novice groups, i.e. SLP ratings vs. combined PT and OT ratings, to determine if there were differences across groups with different background knowledge in communication; (c) agreement between novice ratings and the expert rating (accuracy); and (d) perspectives of novice raters regarding classification utility.

Method

Design

In this study, we examined novice rater agreement and accuracy when using the VSS, FCCS, and CFCS to classify unfamiliar children with CP. Ethics approval was gained from the National Health and Medical Research Council (NHMRC) registered Human Research Ethics Committees of The University of Queensland (2020/HE002821) and Cerebral Palsy League (CPL-2020-005).

Participants

Paediatric SLPs, PTs, and OTs were recruited through the distribution of study fliers via professional networks, social media, and email. Health professionals were included if they self-identified as novice users of the VSS, FCCS, and CFCS. They were excluded if they self-reported “extensive experience” with any of the speech and communication classification systems.

The term novice was used in this study to refer to a clinician who reported having nil to limited experience using a communication classification in clinical practice (i.e. not more than one or two attempts at using); that is, a non-expert rater. The expert rater in this study, the chief investigator (CI; first author), had extensive experience with motor, manual, and speech and communication classifications for both clinical and research applications and was undertaking research in communication classification.

Measures

Video segments of eight children aged 8 to 15 years were selected by the chief investigator (KC) from a previous FCCS research database (see Table I for child characteristics). The video segments were ∼5 min in duration and captured a one-to-one interaction between the CI and the child. As the CI was an unfamiliar communication partner meeting the child for the first time, the interaction was facilitated via play-based scenarios using the pragmatics elicitation protocol (PEP; Caynes et al., 2019). The PEP was developed for a previous study to provide opportunities to observe an indicative sample of 13 everyday communication functions (e.g. greeting, requesting, directing, asking for clarification, giving an explanation, etc.; see Supplementary Appendix 2 for the PEP description).

Table I. Characteristics of child video samples, n = 8 (%).

Sex
Male	4 (50)
Female	4 (50)
Viking Speech Scale (VSS) level
I	1 (13)
II	2 (25)
III	2 (25)
IV	3 (38)
Functional Communication Classification System (FCCS) level
I	1 (13)
II	2 (25)
III	2 (25)
IV	2 (25)
V	1 (13)
Communication Function Classification System (CFCS) level
I	1 (13)
II	2 (25)
III	2 (25)
IV	2 (25)
V	1 (13)
Primary expressive mode group (PEMG)^a
I	3 (38)
II	0 (0)
III	2 (25)
IV	2 (25)
V	1 (12)
Gross Motor Function Classification Scale (GMFCS) level
I	1 (13)
II	0 (0)
III	1 (13)
IV	5 (63)
V	1 (13)
Manual Ability Classification System (MACS) level
I	2 (25)
II	0 (0)
III	1 (13)
IV	3 (38)
V	2 (25)
Motor type
Spasticity	7 (75)
Dystonic	1 (13)
Motor distribution
Quadriplegia	5 (63)
Diplegia	3 (38)
Comorbidities^b
Epilepsy	3 (38)
Visual impairment^c	2 (25)
Hearing impairment^c	1 (13)
Intellectual impairment^c	6 (75)

^aPEMG (Caynes et al., 2021). I = Use speech as primary expressive mode; do not use/need AAC. II = Use speech as primary expressive mode; may use AAC to augment. III = Use/need AAC as primary expressive mode; use of/have potential to use speech or sign to augment (>5 spoken words or signs). IV = Use/need AAC as primary expressive mode; no or limited use of speech or signing (≤5 spoken words or signs). V = Do not use speech or AAC as an expressive mode; partners may use AAC to augment communication and interaction.

^bAustralian Cerebral Palsy Register (ACPR) classification: 1 = no impairment; 2 = some impairment (resolved epilepsy, probably borderline or no intellectual impairment and aided sensory function); and 3 = known impairment.

^cIncludes “some” and “known” impairments. AAC supports included symbol communication books, speech generative devices, and key word signs.

The eight videos represented two examples of children classified at FCCS Levels II-IV, and one example of a child classified at FCCS Level I and Level V. The CI completed the VSS, FCCS, and CFCS following the initial interaction with the child. The CI classified the eight children using the CFCS for this study from video review. The CI ratings for all three communication classifications were used as the gold-standard expert ratings. The CI ratings for all three tools were verified by a second expert rater (DB), who classified the eight children from video review. Both DB and KC were SLPs with over 20 years of clinical experience with children with CP (DB = 22 years; KC = 31 years). The child examples represented all VSS and CFCS levels.

Each child was rated by participants using the following three classification tools. Descriptor levels for each classification are listed for comparison in Table II.

Table II. Level descriptors for the Viking Speech Scale (VSS), Functional Communication Classification System (FCCS), and the Communication Function Classification System (CFCS).

Level	Viking Speech Scale (VSS)	Functional Communication Classification System (FCCS)	Communication Function Classification System (CFCS)
I	Speech is not affected by motor disorder.	An effective communicator in most situations.	Effective sender and receiver with unfamiliar and familiar partners.
II	Speech is imprecise but usually understandable to unfamiliar listeners.	An effective communicator in most situations, but does need some help.	Effective but slower paced sender and/or receiver with unfamiliar and familiar partners.
III	Speech is unclear and not usually understandable to unfamiliar listeners out of context.	An effective communicator in some situations. Can communicate a small range of topics/messages to most familiar people.	Effective sender and receiver with familiar partners.
IV	No understandable speech.	Assistance is required in most situations, especially with unfamiliar people and environments. Communicates daily routine needs and wants with familiar people.	Inconsistent effective sender and/or receiver with familiar partners.
V		Communicates unintentionally with others using movement and behaviour.	Seldom effective sender and receiver, even with familiar partners.

Note. Viking Speech Scale (Pennington, 2016); Functional Communication Classification Scale (Barty et al., 2016; Caynes et al., 2019, 2021); Communication Function Classification Scale (Hidecker et al., 2011).

Viking Speech Scale (VSS)

The VSS is a motor speech classification for children with CP aged 4 years or over. The VSS classifies motor speech intelligibility according to an unfamiliar listener across four descriptor levels as outlined in Table II (Pennington et al., 2013).

Functional Communication Classification System (FCCS)

The FCCS is used to classify the everyday communication competence of children with CP from 2 to 18 years across five performance levels as outlined in Table II. One rating is assigned to reflect typical communication performance using all expressive modes including speech, non-verbal communication, and AAC. Distinctions between levels are determined by the variety of communication purposes expressed by the individual, their level of independence, and the degree of assistance they require from a familiar partner for successful interactions (Barty et al., 2016; Caynes et al., 2019, 2021).

Communication Function Classification System (CFCS)

The CFCS is used to classify the everyday communication performance of children from 2 to 18 years across five descriptor levels as outlined in Table II. Distinctions between levels are determined by an individual’s ability to alternate as a sender and receiver using all expressive modes, effectiveness with familiar and unfamiliar communication partners, and the pace of communication (Hidecker et al., 2011).

Procedure

Participants attended a community or university setting for two 1.5-hr sessions, one week apart. During Session 1, the CI collected participant written consent and demographic characteristics including discipline background; years’ experience as a healthcare professional; years’ experience supporting children with CP; current workplace sector (health, education, private practice, other); caseload (predominant age range, percentage of children with CP); awareness of classifications (yes or no); and clinical experience (0 = nil, 1 = limited, 2 = moderate, 3 = extensive) applying motor (Gross Motor Function Classification system [GMFCS]; Manual Abilities Classification System [MACS]), speech (VSS), and communication (FCCS, CFCS) classifications. They then were introduced to the purpose of communication classification for children with CP and the three classification constructs (two slides), before instructions for each classification were given. This introduction was considered important to orient participants to considerations pertaining to classification in general (i.e. it is not an assessment of ability, but a description of what a child does [daily performance] vs. can do [capacity], etc.; classification is not an assessment, it is not based on age-expected performance). Secondly, particpants were provided with instructions for the first communication classification system (randomised to FCCS or CFCS). Thirdly, they were asked to rate four case videos using that system. Fourthly, they were provided with instructions for the second communication classification system and then, finally, asked to rate four further videos with that system. In Session 2, the CI firstly oriented participants to the VSS and secondly reviewed communication classification instructions. Participants were then asked to review and rate the eight videos in counterbalanced order using the VSS plus either the FCCS or CFCS, whichever was not rated for each video in Session 1. After all case videos were rated with all systems, participants were then asked to rate four clinical utility statements for each classification system on a 7-point Likert scale (from 1 = strongly disagree to 7 = strongly agree). The statements were: (a) the terminology used was easy to understand; (b) it was easy to apply this tool to classify children’s communication performance; (c) the additional resource (FCCS Prompt Sheet/VSS Expanded Descriptions/CFCS Flowchart) assisted decision making; and (d) the tool provides an appropriate description of children’s communication performance. One final open-ended question was asked for each tool: “Do you have any suggestions for improving the [tool name]?”

Analysis

Summary statistics are presented as frequencies (percentages). Inter-rater agreement between combined SLP and allied health professional (AHP) novice ratings was examined using percentage concordance and Fleiss kappa statistics. Fleiss kappa statistics enable measurement of agreement between multiple raters (Fleiss, 1971; Fleiss et al., 2003). Novice rater accuracy, defined as the agreement between expert and novice ratings, was assessed using the quadratic weighted kappa statistic. Quadratic weights are mostly used for summarising agreement on an ordinal scale as used in the current study (Warrens, 2012). Results were interpreted according to criteria by Landis and Koch (1977): almost perfect = 1.00–0.81, substantial = 0.80–0.61, moderate = 0.60–0.41, fair = 0.40–0.21, and slight = 0.20–0.00. Categorical ratings for clinical utility were compared between scales using Kruskal-Wallis H-tests. Data analysis was performed using Stata Statistical Software v14 (College Station, TX, USA). Qualitative comments regarding clinical utility were summarised using content analysis (Graneheim & Lundman, 2004). This qualitative analysis involved identifying meaning units within individual responses, which were condensed and grouped into categories. To increase the rigour of data analysis, coding was subsequently examined by the second and last authors (TR, LJ).

Result

Participant characteristics

Participants were 21 healthcare professionals, similarly distributed between SLPs (n = 11) and the combined PT and OT group (n = 10; PT = 5, OT = 5) from a variety of work settings (see Table III for characteristics of SLPs and AHPs). Clinical experience ranged from 6 to 41 years. The AHP group demonstrated greater experience working with children with CP overall (≥11 years CP experience: PTs and OTs = 7/10, SLPs = 4/11; CP caseload: PTs and OTs = 10/10, SLPs = 6/11).

Table III. Characteristics of speech-language pathologists (SLPs) and other allied health professionals (AHPs).

	SLP (n = 11)	AHP (n = 10)		SLP (n = 11)				AHP (n = 10)
a. Workplace			b. Professional experience in years
Private	5	2	0–10	3				2
University	3	4	11–20	2				2
Health	1	3	21–30	4				2
Community	1	1	31–40	2				3
Education	1	0	41+	0				1
c. Cerebral palsy experience in years			d. Cerebral palsy percentage caseload
0–10	7	3	0%	5				0
11–20	1	2	1–10%	5				6
21–30	3	3	11–25%	1				2
31–40	0	2	26–75%	0				1
41+	0	0	76 + %	0				1
e. Classification awareness			f. Experience	N	L	M	E	N	L	M	E
GMFCS	8/11	10/10	GMFCS	6	5				2	3	5
MACS	3/11	10/10	MACS	11					6		4
CFCS	10/11	7/10	CFCS	5	5	1		3	7
FCCS	10/11	7/10	FCCS	5	5	1		3	7
VSS	6/11	3/10	VSS	10	1			7	3

Note. Classification awareness: GMFCS = Gross Motor Function Classification System; MACS = Manual Ability Classification System; CFCS = Communication Function Classification System; FCCS = Functional Communication Classification System; VSS = Viking Speech Scale. Numbers in SLP (n = 11) and AHP (n = 10) groups aware of each classification. Experience with classifications: N = nil; L = limited; M = moderate; E = Extensive.

We found that professionals may have been aware of speech and communication classifications, but did not necessarily know the content and/or differences between classifications. Most professionals reported being aware of the motor (GMFCS: SLPs = 8/11, PTs and OTs combined = 10/10) and communication (FCCS, CFCS: SLPs = 10/11, PTs and OTs combined = 7/10) classification systems. All PTs and OTs, and less than one-third of SLPs, were aware of the Manual Ability Classification System (MACS: SLPs = 3/11, PTs and OTs combined = 10/10), with approximately half of the SLPs and one-third of PTs and OTs aware of speech classifications (VSS: SLPs = 6/11, PTs and OTs = 3/10). All professionals reported nil to limited clinical experience with speech and communication classification, except one SLP who reported moderate experience with communication but not speech classification. The majority of PTs and OTs (8/10) reported moderate to extensive clinical experience with gross motor classification, compared to less than half of SLPs (5/11) who reported limited experience. All SLPs reported nil clinical experience with the MACS, whereas PTs and OTs combined reported either limited (6/10) or extensive experience (4/10).

Novice rating agreement

Inter-rater agreement between combined novice ratings (SLP and AHP, n = 21) was substantial for the VSS (k = 0.72), and moderate for the FCCS (k = 0.44) and CFCS (k = 0.45). Inter-rater agreement within SLP (n = 11) and PT and OT (n = 10) groups was substantial for the VSS (SLPs: k = 0.70; PTs and OTs: k = 0.72), moderate for the FCCS (SLPs: k = 0.43; PTs and OTs: k = 0.49), and fair to moderate for the CFCS (SLPs: k = 0.38; PTs and OTs: k = 0.50; Table IV).

Table IV. Agreement for the (a) VSS (b) FCCS and (c) CFCS between (i) combined novice (SLP and AHP) and expert ratings (Quadratic Kappa 95%CI) and (ii) combined novice (SLP and AHP) ratings (Fleiss Kappa).

(a) VSS
(i)	Combined novice ratings
Expert ratings	I	II	III		IV	Expert totals
I	7	14				21
II		24	18			42
III		2	38		2	42
IV					63	63
Novice totals	7	40	56		65	168
Expert and combined EA 79% (132/168); k = 0.70 [0.61–0.78]; kw = 0.89 [0.86–0.92]
(b) Combined		SLP			AHP
k = 0.72 [0.53–0.92]		k = 0.70 [0.48–0.93]			k = 0.72 [0.52–0.92]
(b) FCCS
(i)	Combined novice ratings
Expert ratings	I	II	III	IV	V	Expert totals
I	21					21
II	13	21	7	1		42
III		5	26	11		42
IV			3	30	9	42
V				9	12	21
Novice totals	34	26	36	51	21	168
Expert and combined EA 65% (110/168); k = 0.56 [0.47–0.65]; kw = 0.89 [0.86–0.92]
(ii) Combined		SLP			AHP
k = 0.44 [0.23–0.65]		k = 0.43 [0.22–0.64]			k = 0.49 [0.25–0.74]
(c) CFCS
(a)	Combined novice ratings
Expert ratings	I	II	III	IV	V	Expert totals
I	21					21
II	5	27	10			42
III		4	23	14	1	42
IV			5	24	13	42
V		1		7	13	21
Novice totals	26	32	38	45	27	168
Expert and combined EA 64% (108/168); k = 0.52 [0.42–0.61]; kw = 0.86 [0.82–0.91]
(ii) Combined		SLP			AHP
k = 0.45 [0.18–0.72]		k = 0.38 [0.12–0.64]			k = 0.50 [0.19–0.80]

Note. SLP = speech-language pathologist; AHP = allied health professional; EA = exact agreement.

Grey boxes indicate exact agreement between expert and novice ratings.

Novice and expert rating agreement

There was almost perfect inter-rater agreement (accuracy) between combined novice ratings (n = 21) and the expert rating for all systems (VSS, k_w = 0.89; FCCS, k_w = 0.89; CFCS, k_w = 0.86). Exact agreement (100%) between novice and the expert ratings was achieved for VSS Level IV (no understandable speech) and FCCS and CFCS Level I (effective communication). Exact agreement percentages were higher for motor speech (VSS 79%) compared to communication levels (FCCS 65%; CFCS 64%; Table IV).

There was an almost perfect inter-rater agreement between the expert and SLP ratings (VSS, k_w = 0.89; FCCS, k_w = 0.87; CFCS, k_w = 0.84) and the expert and AHP ratings (VSS, k_w = 0.89; FCCS, k_w = 0.90; CFCS, k_w = 0.88; Table V).

Table V. Agreement for the (a) VSS, (b) FCCS and (c) CFCS between expert and (i) SLP ratings and (ii) AHP ratings. Classification System (CFCS)

(a) VSS	SLP ratings						AHP ratings
Expert ratings	I	II	III		IV	Expert totals	Expert ratings	I	II	III		IV	Expert totals
I	4	7				11	I	3	7				10
II		12	10			22	II		12	8			20
III		1	20		1	22	III		1	18		1	20
IV					33	33	IV					30	30
SLP totals	4	20	30		34	88	AHP totals	3	20	26		31	80
Expert and SLP EA 78% (69/88); kw = 0.89 [0.84–0.94]							Expert and AHP EA 79% (63/80); kw = 0.89 [0.84–0.94]
(b) FCCS	SLP ratings						AHP ratings
Expert ratings	I	II	III	IV	V	Expert totals	Expert ratings	I	II	III	IV	V	Expert totals
I	11					11	I	10					10
II	10	9	3			22	II	3	12	4	1		20
III		4	13	5		22	III		1	13	6		20
IV			2	15	5	22	IV			1	15	4	20
V				7	4	11	V				2	8	10
SLP totals	21	13	18	27	9	88	AHP totals	13	13	18	24	12	80
Expert and SLP EA 59% (52/88); kw = 0.87 [0.84–0.91]							Expert and AHP EA 72% (58/80); kw = 0.90 [0.85–0.95]
(c) CFCSCommunication Function Classification System	SLP ratings						AHP ratings
Expert ratings	I	II	III	IV	V	Expert totals	Expert ratings	I	II	III	IV	V	Expert totals
I	11					11	I	10					10
II	3	14	5			22	II	2	13	5			20
III		3	10	8	1	22	III		2	9	9		20
IV			3	13	6	22	IV			2	11	7	20
V		1		4	6	11	V				3	7	10
SLP totals	14	18	18	25	13	88	AHP totals	12	15	16	23	14	80
Expert and SLP EA 61% (54/88); kw = 0.84 [0.76–0.92]							Expert and AHP EA 62% (50/80); kw = 0.88 [0.84–0.93]

Note. SLP = speech-language pathologist; AHP = allied health professional; EA = exact agreement.

Grey boxes indicate exact agreement (95% CI).

Participant perspectives regarding clinical utility

Quantitative results for participant ratings for the four clinical utility statements are summarised in Figure 1 and reported below. In addition, a total of 18 suggestions for tool improvement were received, with the greatest number of suggestions relating to the CFCS (n = 12), followed by the FCCS (n = 3) and VSS (n = 3). Meaning units within the 18 suggestions were grouped into four categories: (a) ability/difficulty understanding terminology or concepts (seven comments), (b) challenges using the tool (three comments), (c) amount and format of information in the assistive decision-making resources (six comments), and (d) appropriateness of content to describe communication performance (two comments).

Figure 1. Distribution (%) of participant agreement on a 7-point Likert scale for four utility statements from strongly disagree (1) to strongly agree (7).

Terminology

Regarding ease of understanding classification terminology, FCCS and VSS ratings were scored higher than the CFCS rating (p = 0.02). Seven qualitative comments related to difficulties understanding terminology or concepts. For example, five participants (SLP = 3; OT = 1; PT = 1) expressed difficulty with CFCS terms. Three of these comments (SLP = 1; OT = 1; PT = 1) related to difficulty conceptualising the CFCS terms “sender/receiver”—e.g. “Different terminology perhaps? ‘Sender and receiver roles’ a little difficult to conceptualise. Not really clear what this is” (OT), and two (SLP = 2) with the terms “familiar/unfamiliar”. Two participants (SLP = 1; OT = 1) requested definitions or examples of terminology, including an SLP for the CFCS term “effective” and an OT for the VSS term “context.”

Ease of use

Ease of use was equivalent between classifications (p = 0.11). Three of the comments stemmed from “challenges using the tool,” with one comment received for each of the three tools. One SLP commented it was difficult to provide a rating from only four levels using the VSS and suggested having a 5-point rather than a 4-point scale, one OT reported difficulty with the multiple options for CFCS Level IV, and one SLP found the FCCS Prompt Sheet slightly confusing.

Assistive decision-making resources

Overall, novices rated the FCCS Prompt Sheet as providing the most assistance in decision-making, followed by the VSS Expanded Descriptions, then the CFCS Flowchart (p = 0.01). Six qualitative comments were provided regarding the amount and format of information in the assistive decision-making resources. For example, two participants suggested having additional descriptions and/or probing questions for the CFCS (PT = 1; OT = 1), “I did have some difficulties deciding between levels so maybe a few more descriptor/probing questions through the (CFCS) flowchart would help” (OT), and one participant for the VSS, “a little bit more description of each of the (VSS) levels” (OT). Another participant preferred the FCCS written description compared to the picture/symbol cues used for the CFCS: “written description (FCCS) was more helpful than the picture/symbol cue (CFCS) for me” (SLP).

Two participants (OT = 2) commented on the amount and format of information in the FCCS Prompt Sheet, sharing it was “wordy” and the flow format prompts between levels, “íf no…then, if yes,” were “a bit confusing.”

Appropriateness of content to describe communication performance

Novices rated the FCCS higher than the CFCS followed by the VSS as providing an appropriate description of a child’s communication performance (p < 0.001). Two comments related to the appropriateness of content to describe communication performance.

One participant suggested additional content for the CFCS level descriptors: “pace is not the only significant factor perhaps level of support and context could be considered” (SLP). In contrast, one participant suggested that too many aspects were considered in CFCS level descriptors, i.e. receptive and expressive communication components: “possibly subscales trying to consider too many aspects at once” (SLP).

Discussion

Our data shows that novice raters can reach substantial agreement when rating motor speech performance (VSS) and moderate agreement when rating communication performance (FCCS, CFCS) for unfamiliar children with CP. Novice raters can also achieve almost perfect agreement with expert ratings for accuracy, for both motor speech and communication classifications. These results were achieved when rating short videos of children interacting with an unfamiliar partner following brief orientation to each classification system. These findings indicate that the VSS, FCCS, and CFCS classification systems all demonstrate sufficient agreement and accuracy when applied by novice raters to be utilised in clinical, research, and population register activities.

Inter-rater agreement between novice raters tended to be slightly higher in the current study compared to previous SCPE research (Virella et al., 2016). In this study, agreement was substantial for the VSS, and moderate for the FCCS and CFCS. The SCPE data indicated moderate agreement for the VSS, moderate for the FCCS, and fair for the CFCS (Virella et al., 2016). One potential explanation may be that our study tested systems in their original English version, whereas the SCPE study included seven translated versions of each classification. Second, clinicians in the SCPE study rated children without training, whereas in the current study a training presentation, consistent in content and duration, regarding each tool’s instructions was provided before participants rated child video samples. It seems that the orientation to each tool before administration resulted in an improved agreement in our study compared to Virella et al. (2016), however, this would need to be verified in a larger scale study. Other factors in the SCPE study may have included the larger number (n = 155) and younger age of children (SCPE mean = 6 years; this study mean = 11 years 9 months). It is also unknown if the health professionals in the SCPE and current study differed in the amount and currency of CP experience and if that may have influenced inter-rater agreement.

The differences in agreement between novice ratings vs. between novice and expert ratings reflect the statistical measurements used. A Fleiss kappa was required to examine agreement between the multiple novice raters, whereas quadratic weighted kappa was applied to examine agreement between combined novice and expert ratings. Quadratic weighted kappa incorporates “partial credit” for near agreements, with smaller discrepancies given more weight, e.g. between one level of a classification system (Graham & Jackson, 1993). Although caution interpreting quadratic kappa is recommended (Palisano et al., 2018), it is the most suited analysis for determining the accuracy of ratings, where disagreement by one level is less clinically significant compared to disagreement over several levels.

When comparing between SLPs and the combined PTs and OTs, we noted the PTs and OTs in our study achieved slightly higher agreement than SLPs when comparing ratings with the expert ratings across classifications. This demonstrates the tools are clear to professionals with a range of clinical backgrounds and the orientation to each tool appeared adequate to facilitate accurate ratings. In addition, it is possible that although PTs and OTs had not been trained in speech and communication, their clinical experience with children with CP may have assisted accurate classification of motor speech and communication performance. Over two-thirds of the combined PT and OT group had 11 or more years’ experience supporting children with CP compared to approximately one-third of the SLP group, and all PTs and OTs had children with CP on their caseloads compared to approximately half of the SLP group. In addition, the PTs and OTs in this study were more aware and experienced than the SLPs in the application of classification systems for other areas of function, such as using the GMFCS and MACS.

In our study, we found higher inter-rater agreement for the VSS compared to the FCCS and CFCS, which suggests that novices find it easier to classify simple motor speech compared to more complex communication performance. This finding has been supported by other researchers (Hustad et al., 2016; Virella et al., 2016). In an inter-rater reliability study by Hustad et al. (2016) involving the VSS, CFCS, and Speech Language Profile Groups (SLPG) classifications, the authors suggested that rating one variable (functional speech) using the VSS is easier and thus more reliable than rating the more complex entity of communication with respect to sender and receiver roles.

The highest agreement for motor speech classification in our study was for children with the poorest speech intelligibility, i.e. minimal (Level III, n = 2) or no understandable speech (Level IV, n = 3). In contrast, Hustad et al. (2016) reported the lowest level of inter-rater agreement at VSS Level III. This may reflect differences in the video interaction between studies, i.e. a child and parent dyad in the study by Hustad et al. compared to the child and unfamiliar therapist in the current study. A lower than anticipated agreement between novice and expert ratings for Level I (no speech impairment) in our study may have been influenced by the case example. It is possible that the child’s speech pronunciation in English may have been inadvertently interpreted by some participants as motor speech imprecision (Level II), noting the child was exposed to a language other than English at home. To address this, novice and unfamiliar raters may benefit from an expanded description in the VSS of “imprecise” to emphasise articulation precision and speech clarity, distinct to pronunciation differences across cultures.

In contrast to speech classification, which involves observation of one component, communication classification requires observation of several dynamic components as well as an evaluation of overall effectiveness with familiar and unfamiliar partners. The FCCS requires observation of the variety of communication functions expressed and independence in interactions, whereas the CFCS requires observation of pace and ability to alternate between sender and receiver roles. When rating communication, novice raters showed 100% agreement when rating effective communication (FCCS and CFCS Level I). This indicates competent communication is easy to recognise from a video even when the participant is unfamiliar to the rater. Levels with the lowest agreement differed between classifications and rater groups (FCCS: SLP = Level V, AHP = Level II; CFCS: SLP and AHP = Level III). Typically, inter-rater agreement is stronger at the extreme ends of classification scales (Palisano et al., 2018). In this study, however, over half of SLP raters (64%) rated a stronger communication performance (FCCS Level IV) for the child classified at FCCS Level V, compared to 20% of AHP raters. This may have been due to the supportive interaction style of the SLP communication partner in the video and/or SLPs observing and crediting a greater range of vocal and movement behaviours with communicative intention. In contrast, 40% of AHP raters appeared to have the most difficulty classifying children who were effective but needed some assistance, FCCS Level II. For the CFCS, raters experienced most difficulty assigning CFCS Level III, which involves determining if a child is an “effective sender and receiver with familiar partners.” This may reflect the feedback from some of the health professionals (SLP = 1, OT = 1, PT = 1) that the CFCS terms effective, and sender and receiver required clarification. These terms may not be easily understood in the context of classifying functional communication and require further explanation and/or examples to support accurate classification. Overall, the slightly lower agreement for Levels II–III also highlights the importance of carefully considering a child’s communication capability with familiar and unfamiliar partners and allowing adequate time for them to incorporate speech, gesture, and aided communication for interaction.

The aim of this study was to examine health professionals’ perspectives regarding the clinical utility of each classification, which is a factor that determines clinical uptake. Our results indicated that the VSS and FCCS terminology were easier to understand compared to the CFCS, the FCCS Prompt Sheet provided the most support in assisting classification, and the FCCS was rated as the best tool for an appropriate description of the communication function. Suggestions were made for the VSS and CFCS regarding terminology, and for the CFCS and FCCS regarding decision-making resources. Other authors have highlighted the importance of clear instructions for communication scales. For example, researchers from the Cerebral Palsy Follow-Up Program (CPUP) recommended “more explicit instructions” (Kristoffersson, 2020, p. 937) and “educational interventions for some raters” (Kristoffersson, 2020, p. 937) for the CFCS. This suggestion is important to consider in revisions or updates of any of the scales.

Our data suggests that novice raters can apply the VSS, FCCS, and CFCS accurately. This means they can be used alongside other motor CP classifications to support a holistic understanding of child function, inter-professional practice, collaborative goal setting, and workload prioritisation. It is acknowledged that all classifications may not be required in every research or clinical circumstance. For example, if speech intelligibility was the only component of communication performance that required examination, then the VSS would suffice. However, if an understanding of how speech intelligibility impacts on expressive communication function and the level of support required from a familiar partner, then the VSS and FCCS would be recommended. Alternatively, if the pace of exchange between sender and receiver roles were to be examined, then the CFCS and VSS would be recommended. All classifications are freely available, and both the FCCS and CFCS have been validated for children from 2 to 18 years; however, only the FCCS has examined concurrent validity with pragmatic assessments for both younger (2- and 3-year-old) and older (5–18 years) children with CP. Organisational support through knowledge translation approaches across disciplines is required to integrate speech and communication classification into daily practice (Ketelaar et al., 2008).

This is the first study to our knowledge that has examined novice rater agreement and accuracy when using the VSS, FCCS, and CFCS to rate speech and communication for unfamiliar children with CP. Strengths included the participation of SLPs, PTs, and OTs who were novice in using speech and communication classification in clinical practice and the analysis of novice accuracy compared to expert ratings. While the agreement was high for novice SLP, PT, and OT raters, future research could explore the influence of clinical experience, especially with children with CP, across larger groups of raters. Although the professional group sample is small, this study has identified differences between discipline groups, which would benefit from further examination in a larger-scale study. As these systems are designed to be used reliably by a variety of people, it was important to see if there were significant differences between professional groups with different background knowledge in communication. We found PTs and OTs achieved greater agreement with expert ratings, which demonstrates these tools are easy to use by all professionals. However, it also raised the question of whether the greater experience PTs and OTs had with motor and manual classification in practice may have supported their accuracy of applying communication classification. This requires further examination and has implications for education and translation of classification used in practice.

It is acknowledged that capturing a representative sample of communication performance from a short video segment poses some limitations. However, the three communication classifications have been developed to be used in several ways, including: (a) direct observation of interaction, and/or (b) parent report regarding child communication performance with familiar and unfamiliar partners, and/or (c) from a file audit. In CP surveillance and research, classification is not always made through direct observation, e.g. CP registers most often classify children from a file audit and/or a phone call with a parent by an administrator.

Although participants rated children with high accuracy from a brief video of an interaction with an unfamiliar therapist, future studies could examine agreement after viewing video footage with both familiar and unfamiliar partners and from using other sources of information. A larger number of child videos, as well as longer segments showing interaction across partners and settings, for classification would provide raters with the opportunity to observe a greater diversity of the communication abilities of children with CP, as well as support further examination of agreement at each level. In this study, the number of videos was limited by the time availability of the health professionals.

Examination of the use of case notes for speech and communication classification would be particularly beneficial, as some international CP registers classify solely from information extracted from health records. The development of best-practice guidelines for achieving consensus between raters would also be helpful for practices where classification is achieved through collaboration with parents and other team members (Bartlett et al., 2016). Finally, future research could explore the potential benefits of training with video examples of children of different ages, using a variety of communication modes.

Conclusion

Overall, the VSS, FCCS, and CFCS showed almost perfect agreement between expert and novice ratings when classifying motor speech and communication of unfamiliar children from video samples following orientation to each tool. Agreement between novice raters themselves was moderate to substantial depending on the tool used, and novices provided suggestions to improve each tool. Practice recommendations to improve agreement between novices for communication classification include orientation to classification constructs and terminology, and viewing video examples of children with different ages, communication abilities, and use of AAC. The combined use of the VSS, FCCS, and CFCS is recommended in future clinical practice, research, and CP registers to extend understanding of the communication profile of children with CP and to assist in identifying specific child and communication partner interaction skills that require augmentation and support for each performance level.

Acknowledgements

We would like to thank the SLPs and AHPs who generously contributed their time to participate in this study, and the children and families from CPL—Choice, Passion, Life—who provided consent for their videos to be used for continuing this communication classification research. Thanks and acknowledgement to Debbie Burmester (DB), the Senior SLP from Choice, Passion, Life, for providing expert confirmation of VSS, FCCS, and CFCS ratings for the archetypal case videos.

Disclosure statement

No potential conflict of interest was reported by the author(s).