The roles of suprasegmental features in predicting English oral proficiency with an automated system

References

• Anderson-Hsieh, J., Johnson, R., & Koehler, K. (1992). The relationship between native speaker judgments of nonnative pronunciation and deviance in segmentals, prosody, and syllable structure. Language Learning, 42(4), 529–555. doi:10.1111/j.1467-1770.1992.tb01043.x
   Web of Science ®Google Scholar
• Anderson-Hsieh, J., & Venkatagiri, H. (1994). Syllable duration and pausing in the speech of Chinese ESL speakers. TESOL Quarterly, 28(4), 807–812. doi:10.2307/3587566
   Web of Science ®Google Scholar
• Arias, J. P., Yoma, N. B., & Vivanco, H. (2010). Automatic intonation assessment for computer aided language learning. Speech Communication, 52(3), 254–267. doi:10.1016/j.specom.2009.11.001
   Web of Science ®Google Scholar
• Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater® V. 2. The Journal of Technology, Learning and Assessment, 4(3), 3–31.
   Google Scholar
• Bernstein, J., Van Moere, A., & Cheng, J. (2010). Validating automated speaking tests. Language Testing, 27(3), 355–377. doi:10.1177/0265532210364404
   Web of Science ®Google Scholar
• Black, M. P., Bone, D., Skordilis, Z. I., Gupta, R., Xia, W., Papadopoulos, P., … Narayanan, S. S., (2015). Automated evaluation of non-native english pronunciation quality: Combining knowledge-and data-driven features at multiple time scales. Proceedings of Interspeech. Dresden, Germany: 16th Annual Conference of the International Speech Communication Association (pp. 493–497). Retrieved from http://toc.proceedings.com/29080webtoc.pdf
   Google Scholar
• Boersma, P., & Weenink, D. (2014). Praat: Doing phonetics by computer (version 5. 3.83). [Computer program]. Retrieved from https://praat.en.softonic.com/mac
   Google Scholar
• Bradac, J. J., Cargile, A. C., & Hallett, J. S. (2001). Language attitudes: Retrospect, conspect, and prospect. In W. P. Robinson & H. Giles (Eds.), The new handbook of language and social psychology (pp. 137–154). New York, NY: Wiley.
   Google Scholar
• Brazil, D. (1997). The communicative value of intonation in English. Cambridge, England: Cambridge University Press.
   Google Scholar
• Breiman, L. (1996). Bias, variance, and arcing classifiers (Technical Report 460). Berkeley, CA: Department of Statistics University of California.
   Google Scholar
• Černý, V. (1985). Thermodynamical approach to the traveling salesman problem: An efficient simulation algorithm. Journal of Optimization Theory and Applications, 45(1), 41–51. doi:10.1007/BF00940812
   Web of Science ®Google Scholar
• Cheng, J. (2011). Automatic assessment of prosody in high-stakes English tests. Proceedings of Interspeech. 13th Annual Conference of the International Speech Communication Association (pp. 1589–1592), Portland, OR. Retrieved from http://www.isca-speech.org/archive/archive_papers/interspeech_2012/i12_1299.pdf
   Google Scholar
• Cheng, J., D’Antilio, Y. Z., Chen, X., & Bernstein, J. (2014). Automatic Assessment of the Speech of Young English Learners. Proceedings of the 9th Workshop on Innovative Use of NLP for Building Educational Applications (pp. 12–21). Association for Computational Linguistics, Baltimore, MD. Retrieved from http://acl2014.org/acl2014/W14-18/W14-18-2014.pdf#page=28
   Google Scholar
• Cheng, W., Greaves, C., & Warren, M. (2008). A corpus-driven study of discourse intonation. Philadelphia, PA: John Benjamins.
   Google Scholar
• Cutler, A., Dahan, D., & Donselaar, W. (1997). Prosody in the comprehension of spoken language: A literature review. Language and Speech, 40(2), 141–201. doi:10.1177/002383099704000203
   PubMed Web of Science ®Google Scholar
• Derwing, T. M., Munro, M. J., & Wiebe, G. (1998). Evidence in favor of a broad framework for pronunciation instruction. Language Learning, 48(3), 393–410. doi:10.1111/lang.1998.48.issue-3
   Web of Science ®Google Scholar
• Evanini, K., & Wang, X. (2013). Automated speech scoring for non-native middle school students with multiple task types. Proceedings of Interspeech. 14th Annual Conference of the International Speech Communication Association (pp. 2435–2439). Lyon, France. Retrieved from http://evanini.com/papers/evaniniWang2013toefljr.pdf
   Google Scholar
• Field, J. (2005). Intelligibility and the listener: The role of lexical stress. TESOL quarterly, 39(3), 399–423.
   Web of Science ®Google Scholar
• Fulcher, G. (1996). Does thick description lead to smart tests? A data-based approach to rating scale construction. Language Testing, 13(2), 208–238. doi:10.1177/026553229601300205
   Google Scholar
• Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., & Pallett, D. S. (1993). DARPA TIMIT acoustic-phonetic continuous speech corpus CD-ROM. NIST speech disc 1-1.1. NASA STI/Recon Technical Report N, 93, 27403.
   Google Scholar
• Giles, H., & Powesland, P. F. (1975). Speech style and social evaluation. London, England: European Association of Experimental Social Psychology.
   Google Scholar
• Gimson, A. C., & Ramsaran, S. (1970). An introduction to the pronunciation of English. London, England: Edward Arnold.
   Google Scholar
• Ginther, A., Dimova, S., & Yang, R. (2010). Conceptual and empirical relationships between temporal measures of fluency and oral English proficiency with implications for automated scoring. Language Testing, 27(3), 379–399. doi:10.1177/0265532210364407
   Web of Science ®Google Scholar
• Goldman-Eisler, F. (1968). Psycholinguistics: Experiments in spontaneous speech. London, England: Academic Press.
   Google Scholar
• Gumperz, J. J. (1982). Discourse strategies. Cambridge, United Kingdom: Cambridge University Press.
   Google Scholar
• Hahn, L. (2004). Primary stress and intelligibility: Research to motivate the teaching of suprasegmentals. TESOL Quarterly, 38(2), 201–223. doi:10.2307/3588378
   Web of Science ®Google Scholar
• Happel, B. L., & Murre, J. M. (1994). Design and evolution of modular neural network architectures. Neural Networks, 7(6), 985–1004. doi:10.1016/S0893-6080(05)80155-8
   Web of Science ®Google Scholar
• Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning. New York, NY: Springer.
   Google Scholar
• Hewings, M. (1995). Tone choice in the English intonation of nonnative speakers. International Review of Applied Linguistics, 33(3), 251–266.
   Web of Science ®Google Scholar
• Hincks, R. (2005). Measures and perceptions of liveliness in student oral presentation speech: A proposal for an automatic feedback mechanism. System, 33(4), 575–591. doi:10.1016/j.system.2005.04.002
   Google Scholar
• Iwashita, N. (2010). Features of oral proficiency in task performance by EFL and JFL learners. Proceedings of the 2008 Second Language Research Forum (pp. 32–47). Cascadilla Proceedings Project, Somerville, MA. Retrieved from http://www.lingref.com/cpp/slrf/2008/paper2383.pdf
   Google Scholar
• Iwashita, N., Brown, A., McNamara, T., & O’Hagan, S. (2008). Assessed levels of second language speaking proficiency: How distinct? Applied Linguistics, 29(1), 24–49. doi:10.1093/applin/amm017
   Web of Science ®Google Scholar
• Jin, T., & Mak, B. (2012). Distinguishing features in scoring L2 Chinese speaking performance: How do they work? Language Testing, 30(1), 23–47. doi:10.1177/0265532212442637
   Web of Science ®Google Scholar
• Johnson, D. O., & Kang, O. (2015a). Automatic prosodic tone choice classification with Brazil’s intonation model. International Journal of Speech Technology, 19(1), 95–109. doi:10.1007/s10772-015-9327-z
   Web of Science ®Google Scholar
• Johnson, D. O., & Kang, O. (2015b). Automatic prominent syllable detection with machine learning classifiers. International Journal of Speech Technology, 18(4), 583–592. doi:10.1007/s10772-015-9299-z
   Web of Science ®Google Scholar
• Johnson, D. O., & Kang, O. (2016). Automatic detection of Brazil’s prosodic tone unit. Proceedings of Speech Prosody (pp.287–291), ISCA, Boston, MA, USA. Retrieved from https://drive.google.com/file/d/0B5tX0NnTCf-IV0wwbUw3a2FGWVE/view
   Google Scholar
• Johnson, D. O., & Kang, O. (2017). Comparison of algorithms to divide noisy phone sequences into syllables for automatic unconstrained English speaking proficiency scoring. The Artificial Intelligence Review, 49, 1–24. doi:10.1007/s10462-017-9594-y
   Google Scholar
• Johnson, D. O., Kang, O., & Ghanem, R. (2016). Improved automatic English proficiency rating of unconstrained speech with multiple corpora. International Journal of Speech Technology, 19(4), 755–768.
   Google Scholar
• Kahn, D. (1976). Syllable-based generalizations in English phonology (Vol. 156). Bloomington, IN: Indiana University Linguistics Club.
   Google Scholar
• Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Westport, CT: American Council of Education and Praeger Series on Higher Education.
   Google Scholar
• Kang, O. (2010). Relative salience of suprasegmental features on judgments of L2 comprehensibility and accentedness. System, 38(2), 301–315. doi:10.1016/j.system.2010.01.005
   Web of Science ®Google Scholar
• Kang, O. (2012). Relative impact of pronunciation features on ratings of non-native speakers’ oral proficiency. In J. Levis & K. LeVelle (Eds.), Proceedings of the 4th Pronunciation in Second Language Learning and Teaching Conference (pp. 10–15). Ames, IA: Iowa State University. Retrieved from http://jlevis.public.iastate.edu/pslltconference/4th%20Proceedings/Kang%20PSLLT%202012.pdf
   Google Scholar
• Kang, O., & Johnson, D. O. (2015). Comparison of inter-rater reliability of human and computer prosodic annotation using Brazil’s prosody model. English Linguistics Research, 4(4), 58–68. doi:10.5430/elr.v4n4p58
   Google Scholar
• Kang, O., & Pickering, L. (2013). Using acoustic and temporal analysis for assessing speaking. In A. Kunnan (Ed.), Companion to Language Assessment (pp. 1047–1062). London, England: Wiley-Blackwell.
   Google Scholar
• Kang, O., Rubin, D., & Pickering, L. (2010). Suprasegmental measures of accentedness and judgments of language learner proficiency in oral English. The Modern Language Journal, 94(4), 554–566. doi:10.1111/modl.2010.94.issue-4
   Web of Science ®Google Scholar
• Kang, O., & Wang, L. (2014). Impact of different task types on candidates’ speaking performances. Research Notes, 57, 40–49.
   Google Scholar
• Kirkpatrick, S., & Vecchi, M. P. (1983). Optimization by simulated annealing. Science, 220(4598), 671–680. doi:10.1126/science.220.4598.671
   PubMed Web of Science ®Google Scholar
• Knowles, G. (2016). Patterns of spoken English: An introduction to English phonetics. New York, NY: Routledge.
   Google Scholar
• Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. The International Joint Conference on Artificial Intelligence, 14(2), 1137–1145.
   Google Scholar
• Kormos, J., & Denes, M. (2004). Exploring measures and perceptions of fluency in the speech of second language learners. System, 32(2), 145–164. doi:10.1016/j.system.2004.01.001
   Google Scholar
• Leacock, C. (2004). Scoring free-responses automatically: A case study of a large-scale assessment. Examens. 1, 3. Retrieved from https://www.ets.org/Media/Research/pdf/erater_examens_leacock.pdf
   Google Scholar
• Leacock, C., & Chodorow, M. (2003). C-rater: Automated scoring of short-answer questions. Computers and the Humanities, 37(4), 389–405. doi:10.1023/A:1025779619903
   Google Scholar
• Loukina, A., Zechner, K., Chen, L., & Heilman, M. (2015). Feature selection for automated speech scoring. Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications (pp. 12–19). Association for Computational Linguistics, Denver, CO. Retrieved from http://www.aclweb.org/anthology/W15-0602
   Google Scholar
• Maier, A., Hönig, F., Zeisser, V., Batliner, A., Körner, E., Yamanaka, N., … Nöth, E. (2009). A language-independent feature set for the automatic evaluation of prosody. Proceedings of Interspeech (pp. 616–619). International Speech Communication Association, Brighton, UK. Retrieved from http://toc.proceedings.com/06816webtoc.pdf
   Google Scholar
• MathWorks, Inc. (2013). MATLAB release 2013a. [Computer program]. Retrieved from https://www.mathworks.com/products/new_products/r2013a-transition.html
   Google Scholar
• Nakajima, S., & Allen, J. (1993). A study on prosody and discourse structure in cooperative dialogues. Phonetica, 50(3), 197–210. doi:10.1159/000261940
   Web of Science ®Google Scholar
• Nakatsuhara, F. (2006). The impact of proficiency-level on conversational styles in paired speaking tests. Research Notes, 25, 15–20.
   Google Scholar
• OEPT. (2009). Retrieved from http://www.purdue.edu/oepp/oept/index.html
   Google Scholar
• Ostendorf, M., Price, P. J., & Shattuck-Hufnagel, S. (1995). The Boston University radio news corpus. Linguistic Data Consortium, 1–19.
   Google Scholar
• Pearson. (2010). The Official Guide to Pearson Test of English Academic. New York, NY: Author.
   Google Scholar
• Pearson Education, Inc. (2015). Versant english test. Retrieved from https://www.versanttest.com/products/english.jsp
   Google Scholar
• Pickering, L. (2001). The role of tone choice in improving ITA communication in the classroom. TESOL Quarterly, 35(2), 233–255. doi:10.2307/3587647
   Web of Science ®Google Scholar
• Pickering, L. (2004). The structure and function of intonational paragraphs in native and nonnative speaker instructional discourse. English for Specific Purposes, 23(1), 19–43. doi:10.1016/S0889-4906(03)00020-6
   Web of Science ®Google Scholar
• Pirt, G. (1990). Discourse intonation problems for nonnative speakers. In M. Hewings (Ed.), Papers in discourse intonation (pp. 145–156). Birmingham, AL: English Language Research.
   Google Scholar
• Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., Hannemann, M., Motlicek, P., Qian, Y., Schwarz, P. & Silovsky, J. (2011). The Kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding (No. EPFL-CONF-192584). IEEE Signal Processing Society.
   Google Scholar
• Powers, D. M. (2011). Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation. Journal of Machine Learning Technologies, 2(1), 37–63.
   Google Scholar
• Préfontaine, Y., Kormos, J., & Johnson, D. E. (2015). How do utterance measures predict raters’ perceptions of fluency in French as a second language? Language Testing, 33(1), 53–73. doi:10.1177/0265532215579530
   Web of Science ®Google Scholar
• Purpura, J. E. (2004). Assessing grammar. Cambridge, England: Cambridge University Press.
   Google Scholar
• Purpura, J. E., Brown, J. D., & Schoonen, R. (2015). Improving the validity of quantitative measures in applied linguistics. Language Learning, 65(1), 37–75. doi:10.1111/lang.12112
   Google Scholar
• Quinlan, J. R. (1999). Simplifying decision trees. International Journal of Human-Computer Studies, 51(2), 497–510. doi:10.1006/ijhc.1987.0321
   Web of Science ®Google Scholar
• Riggenbach, H. (1991). Towards an understanding of fluency: A microanalysis of nonnative speaker conversation. Discourse Processes, 14(4), 423–441. doi:10.1080/01638539109544795
   Web of Science ®Google Scholar
• Rounds, P. (1987). Characterizing successful classroom discourse for NNS teaching assistant training. TESOL Quarterly, 21(4), 643–672. doi:10.2307/3586987
   Web of Science ®Google Scholar
• Rudner, L. M., Garcia, V., & Welch, C. (2006). An evaluation of IntelliMetric™ essay scoring system. The Journal of Technology, Learning and Assessment, 4(4), 3–21.
   Google Scholar
• Sun, Y. (2011). The influence of the social interactional context on test performance: A sociocultural view. The Canadian Journal of Applied Linguistics, 14(1), 194–221.
   Google Scholar
• Truckenbrodt, H. (2002). Upstep and embedded register levels. Phonology, 19(1), 77–120. doi:10.1017/S095267570200427X
   Google Scholar
• Tusing, K. J., & Dillard, J. P. (2000). The sounds of dominance: Vocal precursors of dominance during interpersonal influence. Human Communication Research, 26(1), 148–171.
   Web of Science ®Google Scholar
• Wagner, M., & Watson, D. G. (2010). Experimental and theoretical advances in prosody: A review. Language and Cognitive Processes, 25(7), 905–945. doi:10.1080/01690961003589492
   PubMedGoogle Scholar
• Wennerstrom, A. (1994). Intonational meaning in English discourse: A study of nonnative speakers. Applied Linguistics, 15(4), 399–421. doi:10.1093/applin/15.4.399
   Web of Science ®Google Scholar
• Wennerstrom, A. (2001). The music of everyday speech. New York, NY: Oxford University Press.
   Google Scholar
• Wichmann, A. (2000). Intonation in text and discourse. London, England: Longman.
   Google Scholar
• Xi, X., Higgins, D., Zechner, K., & Williamson, D. M. (2008). Automated scoring of spontaneous speech using SpeechRaterSM v1. 0. ETS Research Report Series, 2008(2), i–102. doi:10.1002/j.2333-8504.2008.tb02148.x
   Google Scholar
• Zechner, K., Higgins, D., Xi, X., & Williamson, D. M. (2009). Automatic scoring of non-native spontaneous speech in tests of spoken English. Speech Communication, 51(10), 883–895. doi:10.1016/j.specom.2009.04.009
   Web of Science ®Google Scholar
• Zechner, K., Xi, X., & Chen, L. (2011). Evaluating prosodic features for automated scoring of non-native read speech. IEEE Workshop on Automatic Speech Recognition & Understanding, 461–466. 10.1109/ASRU.2011.6163975
   Google Scholar