A Tutorial on Reliability Testing in AAC Language Sample Transcription and Analysis

Abstract

Establishing reliability is an essential step in language sample transcription and analysis. This tutorial provides an illustration of replicable procedures for reliability testing during transcription and analysis of language samples generated by people who use augmentative and alternative communication (AAC) systems. Statistical measures used for testing agreement between raters coding categorical data are summarized. Detailed procedures for reliability testing in AAC language sample transcription and analysis are provided, beginning with the collection of raw language sample data. Procedures include guidelines for (a) establishing inter-judge agreement during the transcription process, and (b) using Cohen's kappa to establish inter-rater reliability during deeper analysis of transcribed utterances. All procedures are demonstrated in a case example using language samples from children who use AAC.

Keywords::

• Reliability
• Language sample
• Utterance
• Augmentative and alternative communication
• Kappa

Notes

Acknowledgements

This study was completed in partial fulfillment of the first author's PhD program in the Department of Communication Science and Disorders at the University of Pittsburgh.

The authors wish to thank Sean, Roger, and their family for providing access to the data analyzed in this study. The authors would also like to thank Mike Dickey, Connie Tompkins, and Helen Stickney for their guidance and feedback, and Zlatana Nenova and Milon Volk for their assistance with this project.

This study was approved by the Institutional Review Board at the University of Pittsburgh, IRB protocol number PRO1001053. Parental consent and child participant assent were obtained.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.

Notes

1. Base SAS is a product of SAS Institute, Inc., Cary, NC.

2. SPSS Statistics is a product of IBM, Amronk, NY.

3. The Unity series of Minspeak Application Programs are registered trademarks of Semantic Compaction Systems, Inc., Pittsburgh, PA.

4. SALT is a product of SALT Software, LLC, Madison, WI.

5. PeRT is a product of the AAC Institute, Pittsburgh, PA.

6. An alternative measure, weighted kappa, could also be used if a variable such as number of morphemes is treated as an ordinal variable with rank-ordered categories. Weighted kappa assigns weights to each type of disagreement to quantify the magnitude of discrepancies.

7. PASS Sample Size Software is a product of NCSS, LLC, Kaysville, UT.

8. Prevalences of categories are reported in the order shown in Table II.