Adding a fourth rater to three had little impact in pre-linguistic outcome classification

Figures & data

Table 1. Examples of different approaches to assessment of inter-rater reliability of phonetic transcription of pre-linguistic speech

Number of raters	Reliability observations	Study
Outcome based on transcription from a single primary rater	Second rater in 10–20% of the material	Stoel-Gammon, 1985; Stoel-Gammon, 1988; Vihman & Greenlee, 1987; Robb & Bleil, 1994; Chapman, Hardin-Jones, Halter, 2003; Kern et al., 2009.
Outcome based on two or more primary raters transcribing different parts of the study material. Differences can be related to the primary raters’ different internal standards.	Parts of the material was re-observed by co-rater or by an independent rater	MacNeilage et al., 1997; Gildersleeve-Neumann et al., 2000; McCune & Vihman, 2001; Masataka, 2001
Outcome based on two raters on the entire material	Inter-rater reliability calculated between the raters. Try to reach consensus, if not a third rater act as arbitrator	Hutters et al., 2001; Lohmander et al., 2011; Lieberman & Lohmander, 2014

Table 2. Scenarios with 3 raters

	Scenario	Outcome (taking the majority)
1.	All 3 say present	Present
2.	2 say present and 1 say absent orunsure	Present
3.	1 say present and 2 say absent orunsure	Absent
4.	All 3 say absent	Absent

Table 3. Scenarios with 4 raters

	Scenario	Outcome (taking the majority)
1.	All 4 say present	Present
2.	3 say present and 1 say absent or unsure	Present
3.	2 say present and 2 say absent or unsure	Cannot determine outcome, arbitrator needed
4.	1 say present and 3 say absent or unsure	Absent
5.	All 4 say absent or unsure	Absent

Table 4. Summary statistics of the mean absolute difference in canonical babbling ratio between 3 and 4 raters

Recording	N	Median	Mean	SD	Minimum	Maximum
1	109,620	0.03	0.03	0.02	0	0.11
2	109,620	0.01	0.01	0.01	0	0.07
3	109,620	0.02	0.02	0.01	0	0.06
4	109,620	0.01	0.02	0.02	0	0.13

Figure 1. Distribution of the mean absolute difference in canonical babbling ratio (CBR) between 3 and 4 raters by recording 1 to 4 (top to bottom)

Table 5. Summary statistics of distribution of absolute differences in mean consonant inventory between each combination of 3 raters and on introducing a fourth rater

Recording	N	Median	Mean	SD	Minimum	Maximum	Count of absolute differences ≥1 N (%)
1	109,620	0.17	0.17	0.13	0	0.75	0 (0.00%)
2	109,620	0.25	0.25	0.17	0	1.00	1 (0.001%)
3	109,620	0.17	0.23	0.18	0	0.92	0 (0.00%)
4	109,620	0.25	0.32	0.26	0	1.58	3031 (2.77%)

Figure 2. Distribution of the number of times, when adding a fourth rater to each combination of 3 raters, new sounds were identified for recording 1 to 4 (top to bottom)

Table 6. Summary statistics for the number of times the introduction of a fourth rater (27 possibilities for each combination) identified new sounds

Recording	N	Median	Mean	SD	Minimum	Maximum
1	4060	23	22.86	3.26	15	27
2	4060	15	14.90	4.15	3	27
3	4060	7	9.24	5.50	0	27
4	4060	17	15.27	6.50	0	27

Table 7. Unique syllable types for recording 1 listed across all raters and across language by rater. Syllables identified by at least 10% of the 30 raters are included. Syllables are listed in order of highest to lowest frequency

Recording 1 – infant from Norway
		Language by rater
	Overall n = 30	Brazilian Portuguese n = 5	British English n = 10	Danish n = 4	Norwegian n = 4	Swedish n = 7
Syllable
wa	30 (100%)	5 (100%)	10 (100%)	4 (100%)	4 (100%)	7 (100%)
ma	7 (23%)	3 (60%)	1 (10%)	1 (25%)	1 (25%)	1 (14%)
ja	5 (17%)	2 (40%)	2 (20%)	1 (25%)	0 (0%)	0 (0%)
na	4 (13%)	2 (40%)	0 (0%)	1 (25%)	0 (0%)	1 (14%)

Table 8. Unique syllable types for recording 2 listed across all raters and across language by rater. Syllables identified by at least 10% of the 30 raters are included. Syllables are listen in order of highest to lowest frequency

Recording 2 – infant from UK
		Language by rater
Syllable	Overall n = 30	Brazilian Portuguese n = 5	British English n = 10	Danish n = 4	Norwegian n = 4	Swedish n = 7
ga	29 (97%)	5 (100%)	9 (90%)	4 (100%)	4 (100%)	7 (100%)
ja	26 (87%)	4 (80%)	9 (90%)	4 (100%)	4 (100%)	5 (71%)
da	25 (83%)	4 (80%)	9 (90%)	4 (100%)	3 (75%)	5 (71%)
wa	26 (87%)	5 (100%)	9 (90%)	3 (75%)	3 (75%)	6 (86%)
ka	8 (27%)	1 (20%)	2 (20%)	0 (0%)	2 (50%)	3 (43%)
na	8 (27%)	1 (20%)	1 (10%)	1 (25%)	1 (25%)	4 (57%)
ha	7 (23%)	1 (20%)	4 (40%)	1 (25%)	1 (25%)	0 (0%)
ma	7 (23%)	2 (40%)	2 (20%)	0 (0%)	1 (25%)	2 (29%)
ba	4 (13%)	0 (0%)	2 (20%)	1 (25%)	1 (25%)	0 (0%)

Table 9. Unique syllable types for recording 3 across all raters and across language by rater. Syllables identified by at least 10% of the 30 raters are included. Syllables are listen in order of highest to lowest frequency

Recording 3 – infant from Sweden
		Language by rater
Syllable	Overall n = 30	Brazilian Portuguese n= 5	British English n = 10	Danish n = 4	Norwegian n = 4	Swedish n = 7
ja	30 (100%)	5 (100%)	10 (100%)	4 (100%)	4 (100%)	7 (100%)
ba	28 (93%)	5 (100%)	9 (90%)	4 (100%)	3 (75%)	7 (100%)
ma	27 (90%)	5 (100%)	9 (90%)	3 (75%)	4 (100%)	6 (86%)
wa	26 (87%)	3 (60%)	9 (90%)	4 (100%)	4 (100%)	6 (86%)
na	12 (40%)	2 (40%)	5 (50%)	0 (0%)	2 (50%)	3 (43%)
ha	11 (37%)	0 (0%)	4 (40%)	3 (75%)	2 (50%)	2 (29%)
va	4 (13%)	0 (0%)	3 (30%)	0 (0%)	0 (0%)	1 (14%)
da	3 (10%)	0 (0%)	2 (20%)	0 (0%)	1 (25%)	0 (0%)

Table 10. Unique syllable types for recording 4 listed across all raters and across language by rater. Syllables identified by at least 10% of the 30 raters are included. Syllables are listed in order of most identified to least

Recording 4 – infant from Denmark
		Language by rater
Syllable	Overall n = 30	Brazilian Portuguese n = 5	British English n = 10	Danish n = 4	Norwegian n = 4	Swedish n = 7
ga	18 (60%)	4 (80%)	5 (50%)	2 (50%)	3 (75%)	4 (57%)
ʔa	18 (60%)	2 (40%)	6 (60%)	3 (75%)	4 (100%)	3 (43%)
ha	14 (47%)	1 (20%)	6 (60%)	1 (25%)	2 (50%)	4 (57%)
ma	13 (43%)	4 (80%)	4 (40%)	2 (50%)	0 (0%)	3 (43%)
na	14 (47%)	4 (80%)	3 (30%)	2 (50%)	2 (50%)	3 (43%)
wa	14 (47%)	2 (40%)	5 (50%)	1 (25%)	3 (75%)	3 (43%)
ja	7 (23%)	1 (20%)	4 (40%)	1 (25%)	1 (25%)	0 (0%)
ka	5 (17%)	0 (0%)	1 (10%)	2 (50%)	1 (25%)	1 (14%)
ŋa	6 (20%)	1 (20%)	4 (40%)	0 (0%)	0 (0%)	1 (14%)
da	5 (17%)	2 (40%)	0 (0%)	1 (25%)	1 (25%)	1 (14%)

Table 11. The duplicated result on canonical babbling ratio (CBR), size of syllable inventory (Cons invent) and type of consonants (Cons type) in recording 2

		N	Median	Mean	SD	Min	Max	Count of absolutedifferences ≥1 N(%)
CBR	Recording 2	109,620	0.01	0.01	0.01	0	0.07
	Reassessment	109,620	0.01	0.02	0.01	0	0.06
Cons invent	Recording 2	109,620	0.25	0.25	0.17	0	1.00	1 (0.001)
	Reassessment	109,620	0.17	0.21	0.16	0	0.92	0 (0.00)
Cons type	Recording 2	4060	15	14.90	4.15	3	27
	Reassessment	4060	14	14.35	4.14	2	27

Stoel-Gammon, C. (1985). Phonetic inventories, 15–24 months: A longitudinal study. Journal of Speech, Language, and Hearing Research, 28(4), 505–512. https://doi.org/10.1044/jshr.2804.505

 Web of Science ®Google Scholar

Stoel-Gammon, C. (1988). Prelinguistic vocalizations of hearing-impaired and normally hearing subjects: A comparison of consonantal inventories. Journal of Speech and Hearing Disorders, 53(3), 302–315. https://doi.org/10.1044/jshd.5303.302

 PubMedGoogle Scholar

Vihman, M. M., & Greenlee, M. (1987). Individual differences in phonological development: Ages one and three years. Journal of Speech, Language, and Hearing Research, 30(4), 503–521. https://doi.org/10.1044/jshr.3004.503

 Web of Science ®Google Scholar

Robb, M. P., & Bleil, K. M. (1994). Consonant inventories of young children from 8 to 25 months. Clinical Linguistics & Phonetics, 8(4), 295–320. https://doi.org/10.3109/02699209408985314

 PubMed Web of Science ®Google Scholar

Chapman, K. L., Hardin-Jones, M., & Halter, K. A. (2003). The relationship between early speech and later speech and language performance for infants with cleft lip and palate. Clinical Linguistics & Phonetics, 17(3), 173–197. https://doi.org/10.1080/0269920021000047864

 PubMed Web of Science ®Google Scholar

Kern, S., Davis, B., & Zink, I. (2009). From babbling to first words in four languages: Common trends across languages and individual differences. In F. D’Errico & J.-M. Hombert (Eds..), Becoming Eloquent. Advances in the emergence of language, human cognition, and modern cultures (pp. 205–232). John Benjamins Publishing Company. http://www.rex.kb.dk

 Google Scholar

MacNeilage, P. F., Davis, B. L., & Matyear, C. L. (1997). Babbling and first words: Phonetic similarities and differences. Speech Communication, 22(2–3), 269–277. https://doi.org/10.1016/S0167-6393(97)00022-8

 Web of Science ®Google Scholar

Gildersleeve-Neumann, C. E., Davis, B. L., & MacNeilage, P. F. (2000). Contingencies governing the production of fricatives, affricates, and liquids in babbling. Applied Psycholinguistics, 21(3), 341–363. https://doi.org/10.1017/S0142716400003039

 Web of Science ®Google Scholar

McCune, L., & Vihman, M. M. (2001). Early phonetic and lexical development: A productivity approach. Journal of Speech, Language, and Hearing Research, 44(3), 670–684. https://doi.org/10.1044/1092-4388(2001/054)

 PubMed Web of Science ®Google Scholar

Masataka, N. (2001). Why early linguistic milestones are delayed in infants with Williams syndrome: Late onset of hand banging as a possible rate–limiting constraint on the emergence of canonical babbling. Developmental Science, 4(2), 158–164. https://doi.org/10.1111/1467-7687.00161

 Web of Science ®Google Scholar

Hutters, B., Bau, A., & Brøndsted, K. (2001). A longitudinal group study of speech development in Danish infants born with and without cleft lip and palate. International Journal of Language & Communication Disorders, 36(4), 447–470. https://doi.org/10.1080/13682820110074999

 PubMed Web of Science ®Google Scholar

Lohmander, A., Olsson, M., & Flynn, T. (2011). Early consonant production in Swedish infants with and without unilateral cleft lip and palate and two-stage palatal repair. The Cleft Palate-craniofacial Journal, 48(3), 271–285. https://doi.org/10.1597/09-105

 PubMed Web of Science ®Google Scholar

Lieberman, M., & Lohmander, A. (2014). Observation is a valid way of assessing common variables in typical babbling and identifies infants who need further support. Acta paediatrica, 103(12), 1251–1257. https://doi.org/10.1111/apa.12776

 PubMed Web of Science ®Google Scholar