Adding a fourth rater to three had little impact in pre-linguistic outcome classification

ABSTRACT

The consequence of differing levels of agreement across raters is rarely studied. Subsequently, knowledge is limited on how number of raters affects the outcome. The present study aimed to examine the impact on pre-linguistic outcome classifications of 12-month-old infants when using four raters compared to three. Thirty experienced Speech and Language Therapists (SLTs) from five countries assessed 20 minute video recordings of four 12-month-old infants during a play session with a parent. One recording was assessed twice. A naturalistic listening method in real time was used. This involved: (1) assessing, each syllable as canonical or non-canonical, and (2) following the recording, assessing if the infant was babbling canonically and listing the syllables the infant produced with command. The impact that four raters had on outcome, compared to three, was explored by classifying the outcome based on all possible combinations of three raters and determining the frequency that the outcome assessment changed when a fourth assessor was added. Results revealed that adding a fourth rater had a minimal impact on canonical babbling ratio assessment. Presence/absence of canonical babbling and size of consonant inventory showed a negligible impact on three out of four recordings, whereas the size of syllable inventory and presence/absence of canonical babbling was minimally affected in one recording by adding a fourth rater. In conclusion, adding a forth rater in assessment of pre-linguistic utterances in 12-month-old infants with naturalistic assessment in real time does not affect outcome classifications considerably. Thus, using three raters, as opposed to four, is recommended.

KEYWORDS:

• Inter-rater-reliability
• pre-linguistic syllables
• real-time listening

Acknowledgments

The authors acknowledge the SLTs in our partner centres in Brazil, Denmark, Norway, Sweden and UK for participating in calibration of data recording and assessment. Specifically we would like to thank Prof Kevin Munro for assistance with the manuscript and the SLTs who performed the assessments: Liz Albery, Silvia Helena Alvarez Piazentin-Penna, Malin Appelqvist, Ragnhild Aukner, Pia Bodling, Joan Bogh Nielsen, Melanie Bowden, Karin Brunnegård, Haline Coracine Miguel, Line Dahl Jorgensen, Josefine Enfält, Ana Paula Fukushiro, Cristina Guedes de Azevedo Bento Goncaleves, Jorunn Lemvik, Louise Leturgie, Eva Liljerehn, Natalie Lodge, Siobhan Mcmahon, Kathryn Patrick, Nina-Helen Pedersen, Ginette Phippen, Liisi Raud Westberg, Lucy Rigby, Anne Roberts, Lucy Smith, Helene Soegaard, Maria Sporre, Ann-Sofie Taleman, Jorid Tangstad, Stephanie Van Eeden, Renata Yamashita

Disclosure of interest

The authors report no conflict of interest. This publication was made possible by Grants Number R21DE15128, U01DE018664 and U01DE018837 from the National Institute of Dental and Craniofacial Research (NIDCR). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIDCR.

Additional information

Funding

This work was supported by the National Institute of Dental and Craniofacial Research [R21DE15128, U01DE018664 and U01DE018837].