Inter-rater reliability, intra-rater reliability and internal consistency of the Brisbane Evidence-Based Language Test

Figures & data

Table 1. Stratification levels of participant sample by Brisbane EBLT score.

Characteristic	Strata Level
	Stratum 1	Stratum 2	Stratum 3	Stratum 4^a
Participant performance as % of overall Brisbane EBLT score (n = 100)	0–25%	26–50%	51–75%	76–100%
Raw Brisbane EBLT score range within strata level	<42	48–90	94–135	>138
Strata size as representative of total participant sample (n = 100)	10	20	17	53
Intra-rater study strata size (n = 35)	3	7	6	19
Inter-rater study strata size (n = 15)	1	3	3	8

^aStratum 4 was the largest group within the sample. This group included participants both with and without mild language conditions as determined by the reference-standard language measure in the original diagnostic accuracy study [5].

Table 2. Characteristics of the stroke participant sample.

Characteristic	Inter-rater reliability study (n = 15)	Intra-rater reliability study (n = 35)^a
Age	66.13 (SD 11.52) (range 44–83)	66.90 years (SD 15.74) (range 35–87)
Education (school and tertiary formal education only)	11.86 (SD 2.92) (range 7–17)	11.25 years (SD 3.74) (range 3–18)
Sex	Males 60% (9)	Males 63% (22)
	Females 40% (6)	Females 37% (13)
Language	Monolingual (English) 87%) (13)	Monolingual (English) 77% (27)
	Bi or multilingual 13% (2)	Bi or multilingual 23% (8)
Handedness	Right 80% (12)	Right 83% (29)
	Left 20% (3)	Left 11% (4)
	Ambidextrous 0% (0)	Ambidextrous 6% (2)
Average Brisbane EBLT score (possible range 0–258)	121.9 (SD 52.4) (range 37–197)	128 (SD 55.04) (range 24–202)
Presence of language impairment diagnosis (as per validation reference standard)	Impaired language 93% (14)	Impaired language 77% (27)
	Language intact 6% (1)	Language intact 23% (8)
Stroke type	Infarct 87% (13) 2 thrombolysis Haemorrhagic 13% (2)	Infarct 91% (32) 1 clot retrieval 1 clot retrieval and thrombolysis Haemorrhagic 9% (3)
Target lesion site	Left cerebral hemisphere 80% (12)	Left cerebral hemisphere 86% (30)
	Left subcortical 27% (4)	Left subcortical 31% (11)
	Note: 1 had both cerebral hemisphere and subcortical involvement	Note: 6 had both cerebral hemisphere and subcortical involvement

^aIntra-rater participant sample included the 15 participants from the inter-rater reliability, plus an additional 20 randomized participants.

Table 3. Characteristics of inter-rater and intra-rater clinician raters (n = 15).

Experience level	Average age (μ) years (SD)	Average number of years since graduation (SD) (range)
<5 years (n = 5)^a	31.6 (14)	0.5 (0.5) (range 0–5)
5–10 years (n = 5)	31.8 (4.81)	7.9 (1.75) (range 5.5–10)
>10 years (n = 5)	44 (12.35)	16.75 (3.77) (range 12–35+)

^aTwo inter-rater clinicians aged 23 and 30 with 0 (new graduate) and 2 years’ clinical experience respectively also participated in the intra-rater study.

Table 4. Inter-rater reliability per Brisbane EBLT subtest.

Brisbane EBLT Subtest	No. of output responses	Reliability coefficient	Correlation (Kappa or ICC)	95% Confidence Interval	Mean Inter-Rater Agreement	Complete Agreement
Perceptual subtests 1–6
Copying gestures	3	Kappa^a	0.199	0.028–0.271	92%	80%
Object to object matching	3	Kappa	0.964	0.933–1.000	100%	100%
Demonstrating object use	3	Kappa	0.672	0.587–1.000	93%	86%
Demonstrating gestures from pictures	3	Kappa	0.629	0.541–0.738	89%	73%
Object to picture matching	3	Kappa	0.738	0.015–0.777	97%	86%
Picture to picture matching (semantic links)	6	ICC^b	0.874	0.781–0.946	88%	60%
Perception section total	16	ICC	0.704	0.546–0.859	70%	40%
Auditory comp subtests 7–14
Following commands	9	ICC	0.947	0.903–0.978	70%	26%
Yes / No Questions	13	ICC	0.968	0.940–0.987	86%	66%
Identifying pictures by description	7	ICC	0.931	0.874–0.971	91%	73%
Identifying objects by function	3	Kappa	0.340	0.321–0.340	96%	93%
Odd one out	3	Kappa	0.778	0.640–0.829	85%	46%
Complex questions	7	ICC	0.977	0.957–0.991	85%	60%
Complex questions self-report	2	Kappa	0.971	0.960–1.000	98%	92%
Synonyms	3	Kappa	0.703	0.632–0.771	83%	46%
Auditory comp section total	41	ICC	0.982	0.966–0.993	41%	0%
Verbal expression subtests 15–29
Counting 1 to 10	2	Kappa	0.877	0.699–1.000	94%	86%
Sentence completion	3	Kappa	0.960	0.912–1.000	99%	93%
Personal questions	7	ICC	0.833	0.709–0.933	76%	40%
Repetition	5	ICC	0.829	0.698–0.935	77%	46%
Object naming	3	Kappa	0.847	0.748–0.878	94%	66%
Naming actions (verbs)	3	Kappa	0.878	0.833–0.881	94%	80%
Picture naming	5	ICC	0.962	0.926–0.986	91%	73%
Naming objects from around the room	5	ICC	0.903	0.825–0.961	87%	60%
Naming gestures	3	Kappa	0.833	0.786–0.985	90%	66%
Verbal fluency (both items)	<40	ICC	0.991	0.982–0.996	58%	26%
Picture description	17	ICC	0.963	0.931–0.985	51%	13%
Picture description self-report	2	Kappa	0.958	0.941–1.000	97%	84%
Picture description self-report (new)	2	Kappa	0.026	0.054–0.012	97%	83%
Word definitions	5	ICC	0.913	0.844–0.963	72%	40%
Similarities and differences	3	Kappa	0.585	0.543–0.699	74%	40%
Proverbs	4	ICC	0.770	0.630–0.895	74%	46%
Verbal expression section total	51	ICC	0.994	0.989–0.997	23%	0.06%
Reading subtests 30–40
Object to word matching	3	Kappa	0.810	0.707–0.947	92%	80%
Single word reading	3	Kappa	0.723	0.378–0.924	94%	86%
Word to picture matching	7	ICC	0.917	0.851–0.965	84%	66%
Following written commands	5	ICC	0.961	0.928–0.984	89%	66%
Sums	3	Kappa	0.810	0.780–0.845	91%	73%
Reading sentence aloud	2	Kappa	0.771	0.739–0.788	85%	46%
Medicine label	5	ICC	0.896	0.816–0.956	75%	46%
High level sentence comp.	3	Kappa	0.912	0.892–0.952	94%	73%
Written paragraph comp. self-report	2	Kappa	0.898	0.771–1.000	96%	80%
Written paragraph comp. self-report (new)^c	2	Kappa	N/A	Missing data	94%	66%
Written paragraph comp. total	22	ICC	0.979	0.961–0.991	59%	40%
Written paragraph inference	2	Kappa	0.713	0.438–0.821	88%	66%
Reading section total	24	ICC	0.982	0.967–0.993	0.08%	0%
Writing subtests 41–40
Drawing in mouth	2	Kappa	0.007	−0.032–0.00	98%	86%
Copying	3	Kappa	0.524	0.185–0.620	89%	66%
Writing name	3	Kappa	0.639	0.525–0.850	86%	60%
Writing gender and address	5	ICC	0.932	0.877–0.972	84%	53%
Writing to dictation	7	ICC	0.972	0.949–0.989	83%	46%
Written object naming	2	Kappa	0.891	0.865–0.931	94%	86%
Written gesture naming	2	Kappa	0.890	0.797–0.953	94%	86%
Sentence construction	14	ICC	0.954	0.915–0.981	53%	13%
Sentence construction self-report	2	Kappa	0.951	0.922–1.000	97%	83%
Writing section total	30	ICC	0.978	0.958–0.991	33%	0%

^aKappa adjusts for the level of agreement that can be expected to occur by chance alone. This index however is affected by prevalence totals and has difficulty making distinctions between participants of a population in which those distinctions are rare [35]. Kappa values therefore can be misleadingly low if a large majority of ratings are at the highest or lowest level [17,35]. In these circumstances, the mean percentage agreement and percentage of complete agreement represent more accurate indications of level of reliability [17].

^bSubtests with high numbers of possible response categories result in reduced probability of raters recording exactly the same score. The mean percentage and complete agreement for the subtests is therefore substantially reduced, despite the majority of ICC estimates falling within the almost perfect range [20]. For these categories, ICC correlations should be interpreted as the true estimate of inter-rater reliability [17].

^cOnly participants who responded “yes, they had difficulty” with the previous question were asked this question (resulting in minimal data).

Table 5. Intra-rater reliability per Brisbane EBLT subtest.

Brisbane EBLT Subtest	Correlation (ICC)	95% Confidence Interval
Perceptual subtests 1–6
Copying gestures	0.920	0.868–0.952
Object to object matching	1.0	N/A
Demonstrating object use	0.973	0.955–0.984
Demonstrating gestures from pictures	0.923	0.873–0.954
Object to picture matching	0.822	0.721–0.892
Picture to picture matching (semantic links)	0.984	0.973–0.990
Perception section total	0.979	0.963–0.987
Auditory comprehension subtests 7–14
Following commands	0.983	0.970–0.990
Yes / No Questions	0.986	0.975–0.992
Identifying pictures by description	0.959	0.931–0.977
Identifying objects by function	0.958	0.929–0.976
Odd one out	0.971	0.951–0.983
Complex questions	0.974	0.955–0.985
Complex questions self-report	0.926	0.552–0.992
Synonyms	0.972	0.953–0.983
Auditory comprehension section total	0.994	0.990–0.996
Verbal expression subtests 15–29
Counting 1 to 10	0.997	0.967–0.999
Sentence completion	1.0	N/A
Personal questions	0.979	0.964–0.988
Repetition	0.989	0.980–0.993
Object naming	0.966	0.943–0.981
Naming actions (verbs)	0.963	0.937–0.979
Picture naming	0.991	0.983–0.994
Naming objects from around the room	0.984	0.972–0.991
Naming gestures	0.956	0.927–0.974
Verbal fluency (both items)	0.934	0.888–0.962
Picture description	0.976	0.959–0.986
Picture description self-report	0.999	0.991–0.999
Picture description self-report (new)	0.927	0.781–0.978
Word definitions	0.959	0.929–0.976
Similarities and differences	0.913	0.856–0.949
Proverbs	0.873	0.794–0.925
Verbal exp. section total	0.983	0.970–0.990
Reading subtests 30–40
Object to word matching	0.983	0.970–0.990
Single word reading	0.982	0.969–0.989
Word to picture matching	0.986	0.977–0.992
Following written commands	0.959	0.929–0.976
Sums	0.941	0.902–0.966
Reading sentence aloud	0.994	0.969–0.999
Medicine label	0.951	0.917–0.972
High level sentence comprehension	0.949	0.914–0.970
Written paragraph comprehension self-report	0.996	0.955–0.999
Written paragraph comprehension  self-report (new)^a	N/A	Missing data.
Written paragraph comprehension total	0.986	0.976–0.992
Written paragraph inference	0.995	0.969–0.999
Reading section total	0.994	0.990–0.997
Writing subtests 41–49
Drawing in mouth	0.994	0.878–0.999
Copying	0.835	0.742–0.899
Writing name	0.891	0.825–0.935
Writing gender and address	0.954	0.923–0.972
Writing to dictation	0.989	0.981–0.993
Written object naming	0.999	0.999–0.999
Written gesture naming	0.974	0.883–0.994
Sentence construction	0.992	0.986–0.995
Sentence construction self-report	0.996	0.996–0.996
Writing section total	0.993	0.987–0.996

^aOnly participants who responded “yes, they had difficulty” with the previous question were asked this question (resulting in minimal data).

Rohde A, Doi S, Worrall L, et al. Development and diagnostic validation of the Brisbane Evidence-Based Language Test. Disabil Rehabil. 2020. Available from: https://mc.manuscriptcentral.com/dandr

 Google Scholar

Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43:543–549.

 PubMed Web of Science ®Google Scholar

Graham M, Milanowski A, Miller J. Measuring and promoting inter-rater agreement of teacher and principal performance ratings. Center for Educator Compensation Reform; 2012. p. 1–33. Available from: files.eric.ed.gov/fulltext/ED532068.pdf

 Google Scholar

Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174.

 PubMed Web of Science ®Google Scholar