The impact of chatbot conversational skill on engagement and perceived humanness

ABSTRACT

Conversational agents (CAs)—frequently operationalized as chatbots—are computer systems that leverage natural language processing to engage in conversations with human users. CAs are often operationalized as chatbots which are used for many applications including technical support, customer service, and digital personal assistants. Despite their widespread use, little research to date has investigated how improving the conversational skill of a CA impacts user perceptions of the agent. To elucidate this relationship, this research uses Social Presence Theory to describe how conversational skill influences perceived social presence and ultimately anthropomorphism of a chatbot. We conducted a series of studies in which 450 participants interacted with CAs exhibiting varying levels of conversational skill. We show that people perceive a more skilled CA to be more socially present and anthropomorphic than a less skilled CA. This research advances the knowledge of computer-human interface in information systems, as CA research to date has largely focused on the technical challenges rather than the behavioral questions of how users interact with CAs.

KEYWORDS:

• Conversational agents
• system humanness
• anthropomorphism
• user engagement
• social presence
• chatbots

Notes

1. During the design of the pilot and study 1, we did not feel it was possible for responses to be tailored and not exhibit variability. We would like to thank the reviewer who challenged us on this point, leading to this fourth condition explored in Study 2.

Additional information

Funding

This research was funded by a grant from the Nebraska Research Initiative.

Notes on contributors

Ryan M. Schuetzler

Ryan M. Schuetzler ([email protected]; corresponding author) is an Assistant Professor of Information Systems at Brigham Young University. He received his Ph.D. from University of Arizona. His research focuses on conversational agents, collaboration, and how users behave when interacting with artificial intelligence. Dr. Schuetzler’s research has been published in Journal of Management Information Systems, Decision Support Systems, The Accounting Review, Computers in Human Behavior, and numerous other journals and conferences. He serves as president of the Midwest USA chapter of the Association for Information Systems.

G. Mark Grimes

G. Mark Grimes ([email protected]) is Assistant Professor of Decision and Information Sciences in the Bauer College of Business at the University of Houston. He received his Ph.D. in Management Information Systems from the University of Arizona. His research focuses on conversational agents, information systems security, and analysis of human-computer interaction behaviors to detect changes in emotional and cognitive states. Dr. Grimes’s research has been published in Journal of the Association for Information Systems, Decision Support Systems, Computers in Human Behavior, and other journals and conference proceedings, and has been presented to various industry and government organizations.

Justin Scott Giboney

Justin Scott Giboney ([email protected]) is an Assistant Professor of Information Technology & Cybersecurity at Brigham Young University. He received his Ph.D. in Management Information Systems from the University of Arizona. His research focuses on behavioral information security, deception detection, and knowledge-based systems. Dr. Giboney has been an investigator on eight NSF-funded grants on information security, deception and forensics-related technologies and has published 37 papers related to information security, deception detection, and decision support in such journals as MIS Quarterly, Computers & Security, Decision Support Systems, and Computers in Human Behavior, and others.