A Tutorial on Reliability Testing in AAC Language Sample Transcription and Analysis

References

• Adamson, L. B., Romski, M., Deffebach, K., & Sevcik, R. A. (1992). Symbol vocabulary and the focus of conversations: Augmenting language development for youth with mental retardation. Journal of Speech and Hearing Research, 35, 1333–1343.
   PubMedGoogle Scholar
• Banajee, M., Dicarlo, C., & Stricklin, S. B. (2003). Core vocabulary determination for toddlers. Augmentative and Alternative Comm unication, 19, 67–73.
   Google Scholar
• Barfay, E., & Donner, A. (2000). The effect of collapsing multinomial data when assessing agreement. International Journal of Epidemiology, 29, 1070–1075.
   PubMed Web of Science ®Google Scholar
• Byrt, T., Bishop, J., & Carlin, J. (1993). Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46, 423–429.
   PubMed Web of Science ®Google Scholar
• Bonvillian, J., Dooley, T., Emmons, H., Jack, A., Kissane, N., & Loncke, F. (2008). The development of a simplified manual sign communication system for special populations. Paper presented at the Clinical AAC Research Conference, Charlottesville, VA.
   Google Scholar
• Brown, R. (1973). A first language: The early stages. Cambridge, MA: Harvard University Press.
   Google Scholar
• Carletta, J. (1996). Assessing agreement on classification tasks: The Kappa statistic. Computational Linguistics, 2, 249–254.
   Google Scholar
• Cohen, J. (1960). A coefficient of agreement for normal scales. Educational and Psychological Measurement, 20, 37–46.
   Web of Science ®Google Scholar
• Cone, J. D. (1977). The relevance of reliability and validity for behavior assessment. Behavior Therapy, 8, 411–426.
   Web of Science ®Google Scholar
• de Vet, H. C. W., Terwee, C. B., Knol, D. L., & Bouter, L. M. (2006). When to use agreement versus reliability measures. Journal of Clinical Epidemiology, 59, 1033–1039.
   PubMed Web of Science ®Google Scholar
• Dunn, G. (1989). Design and analysis of reliability studies: The statistical evaluation of measurement errors. London, England: Edward Arnold.
   Google Scholar
• Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56, 52–64.
   Web of Science ®Google Scholar
• Flack, V. F, Afifi, A. A, Lachenbruch, P. A, & Schouten, H. J. A. (1988). Sample size determinations for the two rater kappa statistic. Psychometrika, 53, 321–325.
   Web of Science ®Google Scholar
• Hanser, G. A., & Erickson, K. A. (2007). Integrated word identification and communication instruction for students with complex communication needs: Preliminary results. Focus on Autism and Other Developmental Disabilities, 22, 268–278.
   Google Scholar
• Hegde, M. N. (2003). Clinical research in communicative disorders: Principles and strategies (3rd ed.). Austin, TX: PRO-ED, Inc.
   Google Scholar
• Heilmann, J., Miller, J. F., Iglesias, A., Fabiano-Smith, L., Nockerts, A., & Andriacchi, K. D. (2008). Narrative transcription accuracy and reliability in two languages. Topics in Language Disorders, 28, 178–188.
   Web of Science ®Google Scholar
• Higginbotham, D. J. (2009). In-person interaction in AAC: New perspectives on utterances, multimodality, timing, and device design. Perspectives on Augmentative and Alternative Communication, 18, 154–106.
   Google Scholar
• Higginbotham, D. J., & Engelke, C. R. (2013). A primer for doing talk-in-interaction research in augmentative and alternative communication. Augmentative and Alternative Communication, 29, 3–19.
   PubMed Web of Science ®Google Scholar
• Higginbotham, D. J, Lesher, G. W., Moulton, B. J., & Rinkus, G. J. (2002). Automated data logging in augmentative communication. In J. Winters, C. Robinson, R. Simpson, & G. Vanderheiden (Eds.), Emerging and accessible telecommunications, information and healthcare technologies – Engineering challenges in enabling universal access (pp. 49–82). Arlington, VA: RESNA Press.
   Google Scholar
• Hill, K. (2001). The development of a model for automated performance measurement and the establishment of performance indices for augmented communicators under two sampling conditions. Dissertation Abstracts International, 60, 2293.
   Google Scholar
• Hill, K. (2004). Augmentative and alternative communication and language: Evidence-based practice and language activity monitoring. Topics in Language Disorders, 24, 18–30.
   Web of Science ®Google Scholar
• Hill, K., Kovacs, T., & Shin, S. (2014). Reliability of brain computer interface language sample transcription procedures. Journal of Rehabilitation Research & Development, 51, 579–590.
   PubMedGoogle Scholar
• Hill, K., & Romich, B. (1999). A proposed standard for AAC and writing system data logging for clinical intervention, outcomes measurement, and research. Paper presented at the RESNA ‘99 Annual Conference, Arlington, VA.
   Google Scholar
• Hill, K., & Romich, B. (2001). A language activity monitor for supporting AAC evidence-based clinical practice. Assistive Technology, 13, 12–22.
   PubMed Web of Science ®Google Scholar
• Hill, K., & Romich, B. (2003). PeRT (Performance Report Tool): A computer program for generating the AAC performance report. Pittsburgh, PA: AAC Institute.
   Google Scholar
• Kottner, J., Audigé, L., Brorson, S., Doner, A., Gajewski, B. J., Hróbjartsson, A., … & Streiner, D. L. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. Journal of Clinical Epidemiology, 64, 96–106.
   PubMed Web of Science ®Google Scholar
• Kovacs, T., & Stickney, H. (2012). Acquisition of subject-auxiliary inversion in a child using AAC: A corpus analysis and clinical implications. Proceedings of the 15th Biennial Conference of the International Society for Augmentative and Alternative Communication. Pittsburgh, PA.
   Google Scholar
• Landis, J. R., & Koch, G. G. (1977). The measure of observer agreement for categorical data. Biometrics, 33, 159–174.
   PubMed Web of Science ®Google Scholar
• Loban, W. (1976). Language development: Kindergarten through grade twelve. Urbana, IL: National Council of Teachers of English.
   Google Scholar
• MacWhinney, B. (2000). The CHILDES project: Tools for analyzing talk (3rd ed.). Mahwah, NJ: Erlbaum.
   Google Scholar
• Marvin, C. A., Beukelman, D. R., & Bilyeu, D. (1994). Vocabulary-use patterns in preschool children: Effects of context and time sam pling. Augmentative and Alternative Communication, 10, 224–236.
   Google Scholar
• Miller, J. F. (1981). Assessing language production in children: Experimental procedures. Needham Heights, MA: Allyn and Bacon.
   Google Scholar
• Miller, J. F. (2009). SALT 2010 English version. A clinical tool for assessing the language of speakers whose primary language is English – user guide. Muscoda, WI: SALT Software, LLC.
   Google Scholar
• Ortloff, M. (2010). Language development in physically and motor impaired children without speech: A case study of a child using a speech-generating device (Master's thesis). University of Cologne, Cologne, Germany. Retrieved from http://www.linguist.org/AAC.html
   Google Scholar
• Portney, L. G., & Watkins, M. P. (2009). Foundations of clinical research: Applications to practice (3rd ed.). Upper Saddle River, NJ: Prentice-Hall.
   Google Scholar
• Posner, K., Sampson, P., Caplan, R., Ward, R., & Cheney, F. (1990). Measuring inter-rater reliability among multiple raters: An example of methods for nominal data. Statistics in Medicine, 9, 1103–1115.
   PubMed Web of Science ®Google Scholar
• Romski, M., Sevcik, R. A, Robinson, B., & Bakeman, R. (1994). Adult-directed communications of youth with mental retardation using the system for augmenting language. Journal of Speech and Hearing Research, 37, 617–628.
   PubMedGoogle Scholar
• Santelmann, L., Berk, S., Austin, J., Somashekar, S., & Lust, B. (2002). Continuity and development in the acquisition of inversion in yes/no questions: Dissociating movement and inflection. Journal of Child Language, 29, 813–842.
   PubMed Web of Science ®Google Scholar
• Sevcik, R. A., Romski, M., Watkins, R. V., & Deffebach, K. P. (1995). Adult partner-augmented communication input to youth with mental retardation using the system for augmenting language (SAL). Journal of Speech and Hearing Research, 38, 902–912.
   PubMedGoogle Scholar
• Shriberg, L., Kwiatkowski, J., & Hoffman, K. (1984). A procedure for phonetic transcription by consensus. Journal of Speech and Hearing Research, 27, 456–465.
   PubMedGoogle Scholar
• Sim, J., & Wright, C. (2005). The kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Physical Therapy, 85, 257–268.
   PubMed Web of Science ®Google Scholar
• Stromswold, K., & Pinker, S. (1986). Questions about WH questions. Paper presented at the 11th Annual Boston University Conference on Language Development, Boston, MA.
   Google Scholar
• Suen, H. K., & Ary, D. (1989). Analyzing quantitative behavioral observation data. New York: Psychology Press.
   Google Scholar
• Weinberg, A. (1990). Markedness vs maturation: The case of subject-auxiliary inversion. Language Acquisition, 1, 165–194.
   Google Scholar
• Wilkinson, K. M., Romski, M., & Sevcik, R. A. (1994). Emergence of visual-graphic symbol combinations by youth with moderate or severe mental retardation. Journal of Speech and Hearing Research, 37, 883–895.
   PubMedGoogle Scholar