Measurement accuracy: Enabling human rights for Fijian students with speech difficulties

Abstract

Purpose: The Universal Declaration of Human Rights enshrines the rights to communication and education and measuring access to these rights for children with disabilities is fundamental. The UNICEF/Washington Group Child Functioning Module (CFM) is being promoted to measure progress against the Sustainable Development Goals for children with disabilities. This cross-sectional diagnostic accuracy study in Fiji compares parent and teacher CFM responses to the Intelligibility in Context Scale for 463 primary-aged students with and without speech difficulties.

Method: Receiver operating characteristic curves were constructed to analyse CFM accuracy and determine optimal cut-off points; inter-rater reliability between teachers and parents was calculated.

Result: Parent responses to the CFM speech questions achieved an area under the curve of 0.98, indicating “excellent” diagnostic accuracy. Teachers achieved 0.92 (“very good”). The Youden Index identified the optimal cut-off as “some difficulty”.

Conclusion: The CFM appears effective when used by parents or teachers for distinguishing between children with and without speech difficulties. While identified as the optimal cut-off statistically, the “some difficulty” category identifies too many children without speech difficulties to be simplistically applied to funding eligibility. The CFM should be used as a screening tool, followed by further assessment to confirm eligibility.

Keywords:

• Universal Declaration of Human Rights
• United Nations
• Article 19
• disability
• disaggregation
• Child Functioning Module
• Education Management Information System

Introduction

The rights to communication and education, originally outlined in the Universal Declaration of Human Rights (United Nations, 1948), were reinforced in the Convention on the Rights of Persons with Disabilities (United Nations, 2006). The current global development framework, the Sustainable Development Goals, strongly supports these rights for children with disabilities and encourages all countries to prioritise spending to achieve the goals. The government of Fiji ratified the Convention on the Rights of Persons with Disabilities in March 2017 and is committed to inclusive and quality education. For children with speech difficulties, this includes providing learning support needs such as augmentative and alternative communication and access to communication specialists such as speech-language pathologists. However, providing resources requires means of estimation and accountability, for which accurate numbers are required. Methods to identify children with disabilities in schools in Fiji have been relatively ad hoc and standardised clinical assessments are seldom performed (Sprunt, Hoq, Sharma, & Marella, 2017).

This study was undertaken within an education sector strengthening program funded by the Australian government. An intention of the Fiji government and the program was to establish a means for disability-disaggregation of the government’s Education Management Information System, which collects and integrates information about education activities. The government recognised the need for a valid and reliable disability identification tool feasible for urban, rural and remote schools. The tool selected as a candidate for this purpose was the Child Functioning Module (CFM) as it is relatively short, can be administered without medical expertise, and enables categorisation by severity of functional difficulty.

Child Functioning Module

To improve international comparability of child disability statistics, UNICEF and the Washington Group on Disability Statistics developed the CFM (available from www.washingtongroup-disability.com). It was designed as an interviewer-administered tool for parents/caregivers for incorporating into population censuses and surveys (Crialesi, De Palma, & Battisti, 2016). There is consensus by United Nations agencies and disability peak bodies that the CFM be used to compare children with and without disabilities to disaggregate the Sustainable Development Goal indicators (UNDP et al., 2016).

The CFM was developed with the International Classification of Functioning, Disability and Health for Children and Youth (World Health Organization, 2007) as its conceptual framework and includes 24 questions covering 14 functioning domains, including communication. Items are scored on a Likert scale of severity: no difficulty, some difficulty, a lot of difficulty and cannot do at all, and children are considered at risk of disability if the proxy has responded “a lot of difficulty” or “cannot do at all” to any one item.

The CFM has undergone extensive cognitive testing (Massey, Chepp, Zablotsky, & Creamer, 2014; Mont, Alur, Loeb, & Miller, 2016) and field testing (Crialesi et al., 2016). Detailed domain-specific field tests are unpublished. Visser et al. (2016) assessed diagnostic accuracy of the CFM in a population of 2–4 year olds in rural South Africa, using access to a care dependency grant as the proxy for the reference standard test as it is based on a medical assessment of disability. Sensitivity/specificity of the full CFM was 0.60/0.84. However, transferability of these findings is questionable as the study only had five children with disabilities. In the context of a population survey (Mactaggart, Cappa, Kuper, Loeb, & Polack, 2016; Mactaggart, Kuper, Murthy, Oye, & Polack, 2016), the 2014 draft CFM was tested in Cameroon and India against clinical tests for vision, hearing, musculoskeletal impairment and history of epilepsy. Findings for the full CFM highlighted a large proportion of people with disabilities were missed at the “a lot of difficulty” cut-off level. Authors recommended that people identified having “some difficulty” should receive subsequent clinical assessments to identify a higher proportion of people with disabilities. This study did not test the CFMs ability to identify children with speech difficulties.

Data used in Education Management Information Systems are completed by schools, so it is critical that the CFM performs acceptably when used by teacher respondents; this has not been tested previously. In order to ensure communication and education rights of children with speech difficulties are met, it is critical they are identified in data systems. This study aimed to test the validity of the CFM in identifying children with speech difficulties and reliability between teacher and parent responses.

Study objectives

This paper reports findings from within a broader study that examined the entire Child Functioning Module (Sprunt et al., 2017). Study objectives for this paper were:

1. To determine the diagnostic accuracy and optimal cut-off level of the CFM in identifying primary school children with speech difficulties in Fiji compared to a reference standard, the Intelligibility in Context Scale.

2. To determine the inter-rater reliability of the CFM between teachers and parents in Fiji for identifying primary school children with speech difficulties.

Method

Study design and sampling

Approvals were obtained from the University of Melbourne’s Human Research Ethics Committee and the Fiji Ministry of Education’s ethics committee and all subjects had written consent. This study required responding to two questionnaires (see Procedures); respondents were parents/caregivers and teachers of each child. This paper uses the CFM speech questions plus additional questionnaire items on learning support needs, assistive technology and hearing examination results.

A cross-sectional study was undertaken in Fiji from March–July 2015. Children were recruited from five mainstream schools and 10 special schools in Fiji covering the four administrative divisions. Sample size was estimated based on minimum number required to achieve sensitivity or specificity of 0.85 (Flahault, Cadilhac, & Thomas, 2005). A minimum of 52 children with speech difficulties (cases) and 52 children without speech difficulties (controls) were required. All children (5–15 years) in the special schools and all children in the mainstream schools previously identified by the school to have disabilities, plus selected controls with no disabilities matched by age, sex, ethnicity and location were invited to participate.

Participant demographics

The sample included 463 children with mean ± SD age of 10.2 ± 2.6 years (range: 5–15) in classes 1–8. Of these, 55.3% (n = 256) were male and 63.5% (n = 294) were from i-Taukei (Fijian) ethnic background (Table I). Of the parents/caregivers, more than half were mothers (57.7%, n = 267), 22.5% (n = 104) fathers and 19.9% (n = 92) grandparents, aunties, uncles and guardians. Ninety-nine teachers participated.

Table I. Demographic characteristics of student participants (n = 463).

Participant characteristics	n	%
Sex
Male	256	55.3
Female	207	44.7
Age (years)
5–7 years	94	20.3
8–9 years	102	22.0
10–11 years	99	21.4
12–13 years	108	23.3
14–15 years	60	13.0
Ethnicity
i-Taukei (Fijian)	294	63.5
Indo-Fijian	150	32.4
Other	19	4.1
Type of school
Special	227	49.0
Mainstream primary	236	51.0
Parent/guardian respondent
Mother	267	57.7
Father	104	22.5
Other*	92	19.9
Highest level of education of parent (n= 448)
Primary	106	22.9
Secondary	268	57.9
Higher education	74	16.5
Area of Residence
Urban	106	22.9
Peri-urban	177	38.2
Rural	119	25.7
Remote	61	13.2

*Other: grandparent, aunty, uncle, guardian.

Procedure

Index test: Child Functioning Module

The index test was the Child Functioning Module (CFM, 5–17 years) draft current at February 2015, with permission from UNICEF and the Washington Group. The wording of the two questions on speaking are: (1) “When (child’s name) speaks, does he/she have difficulty being understood by people inside of this household?”, and (2) “When (child’s name) speaks, does he/she have difficulty being understood by people outside of this household?”. The questions aim to distinguish between people who are more familiar and less familiar to the child. For each question, the interviewer states: “Would you say (name) has: no difficulty, some difficulty, a lot of difficulty or cannot do at all?”

The CFM was translated from English to Fijian and Fiji-Hindi by two separate bilingual speakers for each language, back-translated, then pretested according to the guidelines of the Washington Group (UNESCAP, 2010) using cognitive interviews with 10 teachers and 10 parents of children with disability. On the teacher questionnaire, questions were adapted to: (1) “When (child’s name) speaks, does he/she have difficulty being understood by people inside of his/her main classroom?”, and (2) “When (child’s name) speaks, does he/she have difficulty being understood by people outside of his/her main classroom?”, which distinguishes between understanding by people more familiar and less familiar to the child.

Reference Standard Test: Intelligibility in Context Scale

The Intelligibility in Context Scale (ICS) (McLeod, Harrison, & McCormack, 2012), administered to parents, was the reference standard test. This comprises seven questions related to frequency with which different people understand the child, using a Likert scale from “always” to “never”. English, Fijian and Fiji-Hindi versions (available http://www.csu.edu.au/research/multilingual-speech/ics) were used.

The ICS was selected for several reasons: at time of data collection, there were no speech-language pathology services in Fiji and no speech assessment tools developed or validated in Fiji (Hopf, McLeod, & McDonagh, 2017). It can be administered by non-specialists. It can be used irrespective of language or number of languages spoken by the child (McLeod, Crowe, & Shahaeian, 2015; Washington, McDonald, McLeod, Crowe, & Devonish, 2017), which is important in Fiji where many people are multilingual (Hopf, McLeod, & McDonagh, 2018). It assesses intelligibility and comprehensibility, which are comparable constructs to CFM questions on difficulty being understood when speaking. The ICS had already been rigorously translated into Fijian and Fiji-Hindi and has been widely used both with children with speech sound disorders (McLeod et al., 2012; Ng, To, & McLeod, 2014) and with typically developing speech (McLeod et al., 2012; Ng et al., 2014; Phạm, McLeod, & Harrison, 2017). It has good internal consistency, sensitivity, and a significant difference in scores between children with and without speech sound disorders (McLeod et al., 2015). The Hopf, McLeod, and McDonagh (2017) Fijian ICS validation study with 65 typically developing children aged 5–10 showed mean scores of 4.6 for main language and 4.4 for Fiji-English, which were not influenced by age, gender, school year, main language spoken or socioeconomic indicators.

The ICS has some limitations as a reference standard for this diagnostic accuracy study. It is usually used with 4–5 year olds. Correlation coefficients for criterion validity were weak to moderate (Evans, 1996) across a range of ICS studies (McLeod et al., 2015; McLeod et al., 2012; Neumann, Rietz, & Stenneken, 2016; Ng et al., 2014). In the only study reporting ICS sensitivity and specificity using receiver operating characteristic (ROC) curve analysis (Ng et al., 2014), diagnostic accuracy of the ICS was 0.69, which is below the ≥0.70 cut-off for the category labelled “fair” (Hsieh, 2012). That study included 72 Cantonese-speaking pre-school children (39 with and 33 without speech sound disorders) and the reference standard was the Hong Kong Cantonese Articulation Test. Implications of these limitations relate to the importance of diagnostic certainty of the ICS and presented some challenges in using the ICS to distinguish cases from controls.

Determination of cases and controls using the ICS

The ICS must convincingly distinguish cases from controls to enable measurement of sensitivity and specificity of the index test, the CFM. To achieve diagnostic certainty, the meaning of the ICS scores in the sample was interrogated by analysing five variables from the dataset. Four variables were from the parent and teacher questionnaires (learning support needs, assistance required with communication, levels of difficulty being understood and use of relevant assistive devices) and the fifth being presence of hearing impairment identified in audiometric assessment (>40dBA) (Sprunt et al., 2017). If children scored positively on one or more of these variables, they were coded as “potential speech sound disorder”. Of the 263 children coded “potential speech sound disorder”, 97.7% scored positively on more than one variable. The spread of “potential speech sound disorder” across the ICS scores was examined to identify an ICS cut-off score, which distinguished cases from controls.

The ICS cut-off for cases and controls was determined a posteriori and the data and method used to determine the cut-offs are presented together for clarity. Figure 1 series 1 shows the distribution of all student scores on the ICS (n = 463), where 5 indicates the highest and 1 the lowest score of intelligibility. Figure 1 series 2 shows the percentage of the 263 children with “potential speech sound disorder” across ICS scores. To illustrate, of the children with the lowest possible ICS score (1.0), 100% had a “potential speech sound disorder”, whereas of the children with the highest ICS score (5.0), only 2% had a “potential speech sound disorder”.

Figure 1. Series 1 shows the distribution of Intelligibility in Context Scale scores across the whole sample (n = 463). Series 2 shows the percentage of children with potential speech sound disorder (Group 1, Group 2, Group 3), distributed across Intelligibility in Context Scale scores (n = 263).

This distribution identified three distinct groupings: Group 1 (n = 71, 96% with potential speech sound disorder) scored <2.5; Group 2 (n = 134, 42.5% with potential speech sound disorder) scored 2.5 < 4.0; and Group 3 (n = 257, 3.1% with potential speech sound disorder) scored ≥4.0. As seen from the percentages in each group, Group 1 (cases) and Group 3 (controls) provide clear classification, but Group 2 is a mixture of children with and without potential speech sound disorder, who would require further assessment to avoid diagnostic uncertainty. Group 2 was excluded to reduce uncertainty in case/control allocation.

Implementation of index test and reference standard

Data were collected through assessment camps over two to five days at each school. The CFM and ICS were interviewer-administered with a parent/caregiver for each child, using the parents’ preferred language. The parent was requested to consider the ICS in relation to the child’s main language, similar to work by Hopf, McLeod, and McDonagh (2017). Teachers self-administered the CFM during the camp or within a week. Teachers and parents were blinded to each other’s responses.

Analysis

Estimating and comparing measures of diagnostic accuracy

Using IBM SPSS Statistics 24, descriptive statistics were calculated for participant demographics and cross-tabulations used to explore distribution of CFM responses across cases and controls. Spearman’s Rho correlation coefficient was used to analyse the relationship between parent responses on being understood by people inside (more familiar) and outside (less familiar) the household, and similarly between teacher responses on being understood by people inside (more familiar) and outside (less familiar) the main classroom. It was also used to analyse the relationship between parent and teacher responses using a combined variable (recommended by the Washington Group), which took the highest level of difficulty reported for inside or outside. Correlation coefficients were classified as very high (0.90–1.00), high (0.70 < 0.90), moderate (0.50 < 0.70), low (0.30 < 0.50) and negligible (0.00 < 0.30) (Hinkle, Wiersma, & Jurs, 2003).

Sensitivity was calculated as proportion of children identified by the CFM (at different cut-offs) having difficulty being understood, out of the total number of “cases” (children with ICS score <2.5). Specificity was calculated as proportion of children identified by the CFM (at different cut-offs) not having difficulty being understood, out of the total number of “controls” (children with ICS score ≥4.0).

ROC curves were constructed separately for parent and teacher responses to determine the area under the curve, which is an overall figure of diagnostic accuracy combining sensitivity and specificity. Area under the curve interpretations were classified as excellent (0.96–1.0), very good (0.9 < 0.96), good (0.8 < 0.9), fair (0.7 < 0.8), poor (0.6 < 0.7), and useless (0.5 < 0.6) (Hsieh, 2012). ROC curves used dichotomous clinical variables, differentiated as case or control. The Youden Index, based on the optimal balance between sensitivity and specificity on the ROC curve, was calculated to determine cut-off points for each question and respondent type (parent/teacher). Sensitivity, specificity and likelihood ratios were calculated for each respondent type, for each of the two CFM questions (inside/outside) and for the combined variable. Likelihood ratios indicate how many times more (or less) likely a test result of a given level is obtained when the target condition is present than when it is absent. Likelihood ratios are independent of prevalence of the target condition (Spitalnic, 2004) and were therefore used instead of predictive values.

Result

Parent responses were available for 71 cases and 257 controls and teacher responses for 67 cases and 203 controls.

Correlations

There was “very high” correlation between the two questions on the parent version (n = 328; r = 0.91; p < .001) and a “high” correlation between the two questions on the teacher version (n = 270; r = 0.80; p < .001). Using the combined variable (inside and outside), parent responses showed a “high” correlation with teacher responses (n = 269; r = 0.73; p < .001).

Cross tabulation of the CFM results by the ICS results

Table II presents the spread of CFM responses across the ICS scores. Parents reported the vast majority of controls as having “no difficulty” being understood by people inside the household (93.0%) and outside the household (89.1%) and more than 66% of the cases had “a lot of difficulty” (23.9% inside; 31.0% outside) or “cannot do at all” (42.3% inside; 43.7% outside). “Some difficulty” described over 25% of the cases and very few of the controls on the two questions.

Table II. Cross tabulation of Child Functioning Module (CFM) results by Intelligibility in Context Scale (ICS) results for Group 1 (cases, with speech difficulties) and Group 3 (controls, with typical speech).

	Intelligibility in Context Scale score, n(%)
	1.0–2.43 (Group 1)		4.0–5.0 (Group 3)
Child Functioning Module	Inside	Outside	Inside	Outside
Parent (n = 328)*
No difficulty	4 (5.6%)	0 (0.0%)	239 (93.0%)	229 (89.1%)
Some difficulty	20 (28.2%)	18 (25.4%)	17 (6.6%)	26 (10.1%)
A lot of difficulty	17 (23.9%)	22 (31.0%)	1 (0.4%)	2 (0.8%)
Cannot do at all	30 (42.3%)	31 (43.7%)	0 (0.0%)	0 (0.0%)
Total	71 (100.0%)	71 (100.0%)	257 (100.0%)	257 (100.0%)
Teacher (n = 270)^#
No difficulty	17 (25.4%)	4 (6.0%)	164 (80.8%)	159 (78.3%)
Some difficulty	21 (31.3%)	22 (32.8%)	35 (17.2%)	40 (19.7%)
A lot of difficulty	19 (28.4%)	32 (47.8%)	4 (2.0%)	4 (2.0%)
Cannot do at all	10 (14.9%)	9 (13.4%)	0 (0.0%)	0 (0.0%)
Total	67 (100.0%)	67 (100.0%)	203 (100.0%)	203 (100.0%)

*Level of difficulty being understood by people: inside the household/outside the household.

# Level of difficulty being understood by people: inside the main classroom/outside the main classroom.

Teachers reported cases as having lower levels of difficulty being understood by people inside and outside the main classroom than parents inside and outside the household (Table II). This may be due to teachers in special schools in this study having sign language and augmentative and alternative communication skills. On the other hand, teachers ranked controls as having slightly more difficulty being understood than parents. Teachers described nearly a third of cases and close to 20% of controls as having “some difficulty”.

Diagnostic accuracy of the CFM

Table III shows results of the ROC curve analysis. For parent responses, the area under the curve was ≥96% for all variables (inside, outside and combined variable), which means it is an “excellent” test in distinguishing between children, who score <2.5 and those who score ≥4.0 on the ICS, as a proxy for children with and without speech difficulties respectively. For each of these variables, the Youden Index showed “some difficulty” as the optimal cut-off. Higher cut-offs increase specificity but sensitivity is compromised considerably. The cut-offs “a lot of difficulty” and “cannot do at all” showed positive likelihood ratios (+LR) ranging from 64.2 to 116.3 indicating “large and conclusive” (OMERAD, 2008) results, however confidence intervals were very wide. Parent responses for “inside” show less convincing + LR, but still conclusive at 12.8 at the cut-off “some difficulty” with much narrower confidence intervals. The most conclusive –LRs are at the cut-off “some difficulty”.

Table III. Area under the curve, sensitivity, specificity, likelihood ratios and optimal cut-off points on the Child Functioning Module (CFM), parent vs. teacher responses.

Cut-off points	Sensitivity % (95% CI)	Specificity % (95% CI)	+LR (95% CI)	-LR (95% CI)
Parent (n = 328) Cases n = 71 (21.65%) Controls n = 257 (78.35%)
Inside household	AUC 0.96, SE: 0.015^a, CI (0.93, 0.98)^b, p < .0001, Youden Index 0.870*
≥Some difficulty*	0.94 (0.86–0.98)	0.93 (0.89–0.96)	12.81 (8.3–19.8)	0.06 (0.02–0.2)
≥A lot of difficulty	0.66 (0.54–0.77)	0.99 (0.97–1.00)	85.39 (21.3–343.0)	0.34 (0.2–0.5)
≥Cannot do at all	0.42 (0.31–0.55)	1.00 (0.98–1.00)	109.01 (15.1–785.6)	0.58 (0.5–0.7)
Outside household	AUC 0.98, SE: 0.00579 ^a, CI (0.959, 0.992)^b, p < .0001, Youden Index 0.888*
≥Some difficulty*	1.00 (0.95–1.00)	0.89 (0.84–0.92)	8.90 (6.3–12.5)	0.00
≥A lot of difficulty	0.75 (0.63–0.84)	0.99 (0.97–1.00)	64.20 (20.7–199.4)	0.26 (0.2–0.4)
≥Cannot do at all	0.44 (0.32–0.56)	1.00 (0.98–1.00)	112.65 (15.6–811.0)	0.57 (0.5–0.7)
Combined^c	AUC 0.98, SE: 0.00582 ^a, CI (0.958, 0.992)^b, p < .0001, Youden Index 0.880*
≥Some difficulty*	1.00 (0.95–1.00)	0.88 (0.83–0.92)	8.32 (6.0–11.6)	0.00
≥A lot of difficulty	0.76 (0.65–0.85)	0.99 (0.97–1.00)	65.41 (21.1–203.0)	0.24 (0.2–0.4)
≥Cannot do at all	0.45 (0.33–0.57)	1.00 (0.98–1.00)	116.28 (16.2–836.3)	0.55 (0.4–0.7)
Teacher (n = 270) Cases n = 67 (24.81%) Controls n = 203 (75.19%)
Inside classroom	AUC 0.81, SE: 0.0312^a, CI (0.761, 0.857)^b, p < .0001, Youden Index 0.554*
≥Some difficulty*	0.75 (0.64–0.85)	0.81 (0.75–0.86)	3.88 (2.8–5.3)	0.31 (0.2–0.5)
≥A lot of difficulty	0.43 (0.31–0.56)	0.98 (0.95–1.00)	21.97 (8.0–60.2)	0.58 (0.5–0.7)
≥Cannot do at all	0.15 (0.07–0.26)	1.00 (0.98–1.00)	–	0.85 (0.8–0.9)
Outside classroom	AUC 0.92, SE: 0.0192^a, CI (0.881, 0.950)^b, p < .0001, Youden Index 0.724*
≥Some difficulty*	0.94 (0.85–0.98)	0.78 (0.72–0.84)	4.34 (3.3–5.7)	0.08 (0.03–0.2)
≥A lot of difficulty	0.61 (0.49–0.73)	0.98 (0.95–1.00)	31.06 (11.6–83.5)	0.40 (0.3–0.5)
≥Cannot do at all	0.13 (0.06–0.24)	1.00 (0.98–1.00)	–	1.00 (1.0–1.0)
Combined^c	AUC 0.91, SE: 0.0198^a, CI (0.874, 0.944)^b, p < .0001, Youden Index 0.699*
≥Some difficulty*	0.94 (0.85–0.98)	0.76 (0.69–0.82)	3.90 (3.0–5.0)	0.08 (0.03–0.2)
≥A lot of difficulty	0.63 (0.50–0.74)	0.97 (0.94–1.00)	21.21 (9.4–47.7)	0.38 (0.3–0.5)
≥Cannot do at all	0.16 (0.09–0.28)	1.00 (0.98–1.00)	–	0.84 (0.8–0.9)

AUC: area under receiver operating characteristic curve; CI: confidence interval; positive likelihood ratio (+LR = Sn/(1-Sp); negative likelihood ratio (-LR = (1-Sn)/Sp; *Youden Index indicates the optimal cut-off point.

a DeLong, DeLong, and Clarke-Pearson (1988).

b Binomial exact.

c Responses to the questions on difficulty being understood inside or outside the household (parent) or classroom (teacher) are combined, using the highest level of difficulty reported on either question.

For teacher responses, the area under the curve was lower than for parents, with 0.81 for inside, 0.92 for outside and 0.91 for the combined variable, signifying a “very good” test in distinguishing between children who score <2.5 and those who score ≥4.0 on the ICS, as a proxy for children with and without speech difficulties respectively. As with parent responses, the Youden Index showed “some difficulty” as the optimal cut-off for teacher responses. Again, higher cut-offs increase specificity, but sensitivity is appreciably compromised. +LRs only showed a “small increase in the likelihood” (OMERAD, 2008) of SD at the cut-off “some difficulty”, but show “large and conclusive” increases in the likelihood of SD at the cut-off “a lot of difficulty”. -LRs for teacher responses were more conclusive at the level of “some difficulty” but only when “outside the classroom” is utilised.

Figure 2 provides a visual representation of the ROC curve comparison. The combined parent variable performs similarly to the parent outside variable, offering the best diagnostic accuracy. The most accurate teacher question is “outside the main classroom”. The least accurate question is teacher responses to “inside the main classroom”. The sensitivity of the “outside” question is higher than the “inside” question for both parent and teacher responses, which is logical. UNICEF/Washington Group (2015) note that children with speech difficulties may be able to communicate with household members attuned to their gestures or speech idiosyncrasies, but may face more significant difficulties with people less familiar to them. This means a child with moderate speech difficulty may not be identified by the “inside” question but is more likely to be identified by the “outside” question.

Figure 2. Receiver operating characteristic curves comparison of CFM questions on difficulty being understood.

Discussion and conclusion

The CFM was designed to be undertaken with parents/caregivers in censuses and surveys. This study evaluated whether the CFM could be used for a different purpose – by teachers to identify children with speech difficulties, for the purpose of disability disaggregation of Fiji’s Education Management Information System. The study investigated the accuracy of the CFM against a reference standard, the ICS.

Interpreting the diagnostic accuracy results requires consideration of the main study limitation – lack of certainty about whether children with ICS scores 2.5–3.86 should be regarded as a case or a control. This meant excluding 134 children with those scores, leaving 71 cases and 257 controls, resulting in analysis that is undoubtedly based on cases with higher levels of impairment. This increases the risk of “spectrum effect”, a sampling bias that can lead to higher estimates of sensitivity and specificity (Rutjes, Reitsma, Vandenbroucke, Glas, & Bossuyt, 2005). For future use of the ICS as a reference test in limited resource settings, a subsequent speech assessment for children scoring between 2.5–3.86 would enable including all participants thus mitigating “spectrum effect”, whilst avoiding the costs of conducting speech assessments with the entire sample.

Results showed the CFM has acceptable accuracy when used by either parents or teachers. For parent responses areas under the curve ranged from 0.96–0.98, indicating an “excellent test” for either the “inside” or “outside” questions or the combined variable, which takes the highest level of difficulty reported for either of these two questions. Whilst accuracy of teacher responses was lower than parents, the CFM still appears to be a “very good test” (area under the curve =0.92) when undertaken by teachers using the “outside the main classroom” question or the combined variable. The “inside the main classroom” question has a lower accuracy (area under the curve =0.81), which may imply redundancy of this question. Before assuming it is redundant, further work is needed to explore its use in informing learning support provision. The higher agreement amongst parent responses than teachers is consistent with findings in the Harrison, McLeod, McAllister, and McCormack (2017) study comparing parent and teacher responses on the Parent Evaluation of Developmental Status to clinical diagnosis by a speech-language pathologist. Further study is recommended to understand factors influencing the differences between teacher and parent accuracy.

Initial findings from this study suggest the “outside” question may be more sensitive than the “inside” question at all cut-off levels. The implication for national surveys and censuses, for which the CFM is designed, is that it may be reasonable to only include the “outside” question if cost per question is an imperative. Research on larger datasets is recommended to investigate this issue.

The optimal cut-off to identify children with speech difficulties was “some difficulty” for both parent and teacher responses as determined by the Youden Index. Negative likelihood ratios also pointed to “some difficulty” as the optimal cut-off, however, the positive likelihood ratios challenged these findings, indicating a far higher likelihood of children having speech difficulties at the cut-off “a lot of difficulty”. However, as noted by Sonis (1999), likelihood ratios at extreme values can be imprecise due to sparse data resulting in wide confidence intervals, which was a limitation of this study. A larger sample size or further analysis collapsing the categories “a lot of difficulty” and “cannot do at all” may enable narrower confidence intervals and less ambiguousness around the likelihood ratios.

Providing support to teachers in using the CFM and instigating further assessment and referral processes are important features of a disability disaggregation system. These are highlighted by two issues in this study: (1) the large size of the “inconclusive” group (n = 134); and (2) despite the optimal cut-off being statistically identified as “some difficulty”, this masks the reality that 22% of cases and 38% of controls were labelled “some difficulty” by teachers. That is, a policy that simply uses the cut-off “some difficulty” to identify children with speech difficulties is not satisfactory for determining funding allocations. To ensure that the rights and needs of all children with speech difficulties are met, and that resources are spent on children who meet eligibility standards, the MoE must continue to implement its recently developed system (Access to Quality Education Program, 2017) of further assessing relevant children identified through the student learning profile form. This form uses CFM as its basis and incorporates additional information on learning support needs. Research on effectiveness of this system is critical.

This study included participants with overlapping speech difficulties, hearing impairment and difficulties learning, the associations between which have been previously documented (Dittrich & Rona, 2008; McKean et al., 2017; McLeod & McKinnon, 2007). We did not attempt to test the accuracy of the CFM in identifying children with a diagnosable speech disorder, nor children with speech difficulties in the absence of concomitant conditions. Research comparing the CFM to full speech-language assessment results is recommended to establish which types of communication disorders are picked up through the CFM. As the Ministry of Education has instigated a policy with a conservative cut-off “some difficulty”, which prompts further assessment, the detection of broader communication disorders will be enabled. Acknowledging the lack of permanent speech-language pathology services in Fiji (Hopf & McLeod, 2015; Hopf, McLeod, McDonagh, Wang, & Rakanace, 2017), wherever possible children with speech difficulties should be supported by communication specialists.

Declaration of interest

No conflict of interest is reported.

Acknowledgements

Authors thank students, families and staff of the study schools, partner agencies Pacific Disability Forum and Pacific Islands Forum Secretariat, staff from Access to Quality Education Program and especially to Kitione Ravulo.

Additional information

Funding

This work was supported by Department of Foreign Affairs and Trade, Australian Government [grant number 66440].