Finding the experts in the crowd: Validity and reliability of crowdsourced measures of children’s gradient speech contrasts

References

• Amorosa, H., Benda, U. von, Wagner, E., & Keck, A. (1985). Transcribing phonetic detail in the speech of unintelligible children: A comparison of procedures. British Journal of Disorders of Communication, 20(3), 281–287.
   PubMedGoogle Scholar
• Becker, M., & Levine, J. (2010). Experigen–an online experiment platform. Retrieved from https://github.com/tlozoot/experigen
   Google Scholar
• Boersma, P., & Weenink, D. (2015). Praat: Doing phonetics by computer [computer program]. Version 5.4.20, Retrieved September 26, 2015, from http://www.praat.org/
   Google Scholar
• Boyce, S., & Espy-Wilson, C. Y. (1997). Coarticulatory stability in American English /r/. The Journal of the Acoustical Society of America, 101(6), 3741–3753.
   PubMed Web of Science ®Google Scholar
• Crump, M. J., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS ONE, 8 (3), e57410.
   PubMed Web of Science ®Google Scholar
• Dalston, R. M. (1975). Acoustic characteristics of English /w, r, l/spoken correctly by young children and adults. The Journal of the Acoustical Society of America, 57(2), 462–469.
   PubMed Web of Science ®Google Scholar
• Delattre, P., & Freeman, D. C. (1968). A dialect study of American r’s by X-ray motion picture. Linguistics, 6(44), 29–68.
   Google Scholar
• Edwards, J., Gibbon, F. E., & Fourakis, M. (1997). On discrete changes in the acquisition of the alveolar/velar stop consonant contrast. Language and Speech, 40(2), 203–210.
   Google Scholar
• Gibbon, F. E. (1999). Undifferentiated lingual gestures in children with articulation/phonological disorders. Journal of Speech, Language, and Hearing Research, 42(2), 382–397.
   PubMed Web of Science ®Google Scholar
• Gibson, E., Piantadosi, S., & Fedorenko, K. (2011). Using Mechanical Turk to obtain and analyze English acceptability judgments. Language and Linguistics Compass, 5(8), 509–524.
   Google Scholar
• Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213–224.
   Web of Science ®Google Scholar
• Hagiwara, R. (1995). Acoustic realizations of American /r/as produced by women and men. UCLA Working Papers in Phonetics, 90, 1–187.
   Google Scholar
• Hewlett, N., & Waters, D. (2004). Gradient change in the acquisition of phonology. Clinical Linguistics & Phonetics, 18 (6–8), 523–533.
   PubMed Web of Science ®Google Scholar
• Hitchcock, E. R., & Koenig, L. L. (2013). The effects of data reduction in determining the schedule of voicing acquisition in young children. Journal of Speech, Language, and Hearing Research, 56(2), 441–457.
   PubMed Web of Science ®Google Scholar
• Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14(3), 399–425.
   Web of Science ®Google Scholar
• Ipeirotis, P. G., Provost, F., Sheng, V. S., & Wang, J. (2014). Repeated labeling using multiple noisy labelers. Data Mining and Knowledge Discovery, 28(2), 402–441.
   Web of Science ®Google Scholar
• Julien, H., & Munson, B. (2012). Modifying speech to children based on their perceived phonetic accuracy. Journal of Speech Language and Hearing Research, 55(6), 1836–1849.
   PubMed Web of Science ®Google Scholar
• Klein, H. B., Grigos, M. I., McAllister Byun, T., & Davidson, L. (2012). The relationship between inexperienced listeners’ perceptions and acoustic correlates of children’s /r/productions. Clinical Linguistics & Phonetics, 26(7), 628–645.
   PubMed Web of Science ®Google Scholar
• Lee, S., Potamianos, A., & Narayanan, S. (1999). Acoustics of children’s speech: Developmental changes of temporal and spectral parameters. The Journal of the Acoustical Society of America, 105(3), 1455–1468.
   PubMed Web of Science ®Google Scholar
• Lennes, M. (2003). Collect formant data from files.praat. Retrieved October 8, 2015, http://www.helsinki.fi/lennes/praat-scripts/public/collectformantdatafromfiles.praat
   Google Scholar
• Li, F., Edwards, J., & Beckman, M. E. (2009). Contrast and covert contrast: The phonetic development of voiceless sibilant fricatives in English and Japanese toddlers. Journal of Phonetics, 37(1), 111–124.
   PubMed Web of Science ®Google Scholar
• Macken, M. A., & Barton, D. (1980). The acquisition of the voicing contrast in English: A study of voice onset time in word-initial stop consonants. Journal of Child Language, 7(01), 41–74.
   PubMed Web of Science ®Google Scholar
• Massaro, D. W., & Cohen, M. M. (1983). Categorical or continuous speech perception: A new test. Speech Communication, 2(1), 15–35.
   Web of Science ®Google Scholar
• Maxwell, E. M., & Weismer, G. (1982). The contribution of phonological, acoustic, and perceptual techniques to the characterization of a misarticulating child’s voice contrast for stops. Applied Psycholinguistics, 3(01), 29–43.
   Web of Science ®Google Scholar
• McAllister Byun, T., Halpin, P., & Harel, D. (2015). Crowdsourcing for gradient ratings of child speech: Comparing three methods of response aggregation. In Proceedings of the 18th International Congress of Phonetic Sciences, Glasgow, UK.
   Google Scholar
• McAllister Byun, T., Halpin, P., & Szeredi, D. (2015). Online crowdsourcing for efficient rating of speech: A validation study. Journal of Communication Disorders, 53, 70–83.
   PubMed Web of Science ®Google Scholar
• McAllister Byun, T., & Hitchcock, E. R. (2012). Investigating the use of traditional and spectral biofeedback approaches to intervention for /r/misarticulation. American Journal of Speech-Language Pathology, 21(3), 207–221.
   PubMed Web of Science ®Google Scholar
• McAllister Byun, T., Hitchcock, E. R., & Swartz, M. T. (2014). Retroflex versus bunched in treatment for /r/ misarticulation: Evidence from ultrasound biofeedback intervention. Journal of Speech, Language, and Hearing Research, 57(6), 2116–2130.
   PubMed Web of Science ®Google Scholar
• McGowan, R. S., Nittrouer, S., & Manning, C. J. (2004). Development of [ɹ] in young, midwestern, American children. The Journal of the Acoustical Society of America, 115 (2), 871.
   PubMed Web of Science ®Google Scholar
• Munson, B., & Brinkman, K. N. (2004). The influence of multiple presentations on judgments of children’s phonetic accuracy. American Journal of Speech-Language Pathology, 13(4), 341–354.
   PubMed Web of Science ®Google Scholar
• Munson, B., Johnson, J. M., & Edwards, J. (2012). The role of experience in the perception of phonetic detail in children’s speech: A comparison between speech-language pathologists and clinically untrained listeners. American Journal of Speech-Language Pathology, 21(2), 124–139.
   PubMed Web of Science ®Google Scholar
• Munson, B., Schellinger, S., & Urberg Carlson, K. (2012). Measuring speech-sound learning using visual analog scaling. Perspectives in Language Learning and Education, 19, 19–30.
   Google Scholar
• Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.
   Web of Science ®Google Scholar
• Pisoni, D. B., & Tash, J. (1974). Reaction times to comparisons within and across phonetic categories. Attention, Perception, & Psychophysics, 15(2), 285–290.
   PubMed Web of Science ®Google Scholar
• Pitt, M. A., & Shoaf, L. (2002). Linking verbal transformations to their causes. Journal of Experimental Psychology: Human Perception and Performance, 28(1), 150–162.
   Web of Science ®Google Scholar
• Richtsmeier, P. T. (2010). Child phoneme errors are not substitutions. In Toronto Working Papers in Linguistics, 33 (pp. 343–358). Toronto, Canada: Department of Linguistics, University of Toronto.
   Google Scholar
• Scobbie, J. (1998). Interactions between the acquisition of phonetics and phonology. In M. Gruber (Ed.), Proceedings of the Chicago Linguistic Society 34, part 2: Papers from the panels (pp. 343–358). Chicago, IL: The Chicago Linguistic Society.
   Google Scholar
• Scobbie, J., Gibbon, F. E., Hardcastle, W., & Fletcher, P. (2000). Covert contrast as a stage in the acquisition of phonetics and phonology. In M. Broe & J. Pierrehumbert (Eds.), Papers in Laboratory Phonology V (pp. 194–207). Cambridge, UK: Cambridge University Press.
   Google Scholar
• Shriberg, L. D., Flipsen, P. J., Karlsson, H. B., & McSweeny, J. L. (2001). Acoustic phenotypes for speech-genetics studies: An acoustic marker for residual /r/distortions. Clinical Linguistics & Phonetics, 15(8), 631–650.
   Web of Science ®Google Scholar
• Shriberg, L. D., Paul, R., & Flipsen, P. (2009). Childhood speech sound disorders: From postbehaviorism to the postgenomic era. Speech Sound Disorders in Children, 1–33.
   Google Scholar
• Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86 (2), 420.
   PubMed Web of Science ®Google Scholar
• Sprouse, J. (2011). A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods, 43(1), 155–167.
   PubMed Web of Science ®Google Scholar
• Tyler, A. A., Edwards, M. L., & Saxman, J. H. (1990). Acoustic validation of phonological knowledge and its relationship to treatment. Journal of Speech and Hearing Disorders, 55(2), 251–261.
   PubMedGoogle Scholar
• Tyler, A. A., Figurski, G. R., & Langsdale, T. (1993). Relationships between acoustically determined knowledge of stop place and voicing contrasts and phonological treatment progress. Journal of Speech Language and Hearing Research, 36(4), 746–759.
   Web of Science ®Google Scholar
• Weismer, G. (1984). Acoustic analysis strategies for the refinement of phonological analysis. Phonological Theory and the Misarticulating Child, 30–52.
   Google Scholar
• Werker, J. F., & Logan, J. S. (1985). Cross-language evidence for three factors in speech perception. Perception & Psychophysics, 37(1), 35–44.
   PubMed Web of Science ®Google Scholar
• Young, E., & Gilbert, H. (1988). An analysis of stops produced by normal children and children who exhibit velar fronting. Journal of Phonetics, 16(2), 243–246.
   Web of Science ®Google Scholar