Does accuracy matter? Methodological considerations when using automated speech-to-text for social science research

ABSTRACT

The analysis of spoken language has been integral to a breadth of research in social science and beyond. However, for analyses to occur with efficiency, language must be in the form of computer-readable text. Historically, the speech-to-text process has occurred manually using human transcriptionists. Automated speech recognition (ASR) is advertised as an efficient and inexpensive alternative, but research shows this method of speech-to-text is prone to error. This paper investigates the viability of using error prone ASR transcriptions as part of the methodological process of language analysis. Results show that at the individual feature level, analysis of ASR transcriptions differ dramatically from human transcriptions. However, when the same features are used for classification, a common machine learning task, performance results between ASR and human transcriptions are similar. We present these findings and conclude with a discussion on the methodological considerations for researchers who opt to use automated speech recognition for social science research.

KEYWORDS:

• Linguistic analysis
• text-to-speech
• LIWC
• machine learning
• automated speech recognition

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Steven J. Pentland

Steven J. Pentland is an assistant professor of Information Technology Management at Boise State University. His research leverages information systems technology to evaluate interpersonal communication. Much of his work uses automated verbal and nonverbal feature extraction technology to predict constructs such as deception, source credibility, hireability, and intelligence. Pentland’s research has appeared in outlets such as the Journal of Management Information Systems, IEEE Transactions on Affective Computing, and Frontiers in Psychology.

Christie M. Fuller

Christie M. Fuller is an associate professor of Information Technology at Boise State University. Her research focuses on text and data analytics, deception detection, virtual teams, and research methods, specifically common method variance. Her research has been published in the Information Systems Journal, Decision Support Systems, IEEE Transactions on Professional Communication, Organizational Research Methods and other international journals. Fuller’s research in deception, analytics and research methods also appears in numerous conference proceedings and book chapters. Her research has been sponsored by the Defense Academy for Credibility Assessment, the Air Force Office of Scientific Research, and the Air Force Research Laboratory.

Lee A. Spitzley

Lee A. Spitzley is an assistant professor of Information Security and Digital Forensics at the University at Albany School of Business. He received his Ph.D. in Management Information Systems from the University of Arizona. His research interests include understanding how language and nonverbal behavior reflects underlying psychological states and traits, linking language measures to psychological constructs, and using these methods to solve real-world problems in the areas of financial fraud, job interviews, and group interactions.

Douglas P. Twitchell

Douglas P. Twitchell PhD is an associate professor of Information Technology Management in the College of Business and Economics at Boise State University. Dr. Twitchell has published articles on information security, deception detection, and classification techniques in journals including the Journal of Management Information Systems, Information Systems Journal, and IEEE Intelligent Systems. He enjoys teaching information security, programming, cloud computing, and other information systems topics.