Introducing CASUX: A Standardized Scale for Measuring the User Experience of Artificial Intelligence Based Conversational Agents

Abstract

Conversational agents are growing in popularity, and as such they must provide a good user experience and meet the needs of the users. Yet, how to measure the user experience in the conversational AI scenario remains an open and urgent question to be solved, which may hinder further empirical studies on human-agent interactions. In fact, there have been very few studies that explore how users perceive interacting with these conversational agents, which is important to ensure their sustainability. Accordingly, in this work, we used a subjective technique by following a measurement approach to develop a standardized measurement instrument/scale called the Conversational Agent Scale for User Experience. In terms of the methodology, we used a mixed-method approach involving an iterative process spanning across six different user studies (three qualitative and three quantitative) at different points of time for the purpose of dimension identification, item generation and subsequent item refinements. As a part of the qualitative studies we conducted a Systematic Literature Review, semi-structured interviews with users of conversational agents, and expert interviews. For the quantitative studies a lab-based experiment was performed for the pre-test, followed by two online surveys as a part of pilot testing and scale development. The qualitative studies initially identified a total of 13 distinct user experience dimensions and 418 measurement items. Finally, the Exploratory Factor Analysis as a part of the main survey resulted in 9 user experience dimensions (practicality perception, proficiency, humanness, sentiment, robustness, etiquette & mannerism, personality, anthropomorphism, and ease of use) and 34 measurement items. This was supplemented by conducting another Confirmatory Factor Analysis for establishing the reliability and validity of the proposed scale, together with checking the model-fit indices. All the 9 dimensions had sufficient validity, and a reasonable level of statistical reliability. Some of the user experience dimensions like humanness, personality, anthropomorphism, and etiquette & mannerism show the uniqueness of the conversational AI scenario from the traditional usage factors used commonly while evaluating graphical user interface-based systems. This work fills the gap of a lack of research on how to classify and measure the conversational experience of users and provides a reference for practitioners and designers in developing these agents and continuously improving the usage experience.

Keywords:

• Artificial intelligence
• conversational agents
• factor analysis
• scale development
• user experience

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was partially supported by the Petchra Pra Jom Klao Doctoral Scholarship Research Funding under King Mongkut’s University of Technology Thonburi (KMUTT).

Notes on contributors

Lawal Ibrahim Dutsinma Faruk

Lawal Ibrahim Dutsinma Faruk, holds a B.Sc. from the University of Portsmouth (United Kingdom) and a master’s from Mae Fah Luang University (Thailand). He is currently pursuing Ph.D. at King Mongkut’s University of Technology Thonburi (Thailand), and his research focuses on enhancing user experience with AI-based voice agents.

Debajyoti Pal

Debajyoti Pal is currently working as a lecturer in the School of IT at King Mongkut’s University of Technology Thonburi. His research interests include digital humanism, social implications of emerging technologies, technology adoption, and information systems. He serves as an Associate Editor of several prominent journals from Elsevier and Springer.

Suree Funilkul

Suree Funilkul holds a B.Sc. from Mahidol University, and M.Sc. and Ph.D. from King Mongkut’s University of Technology Thonburi. Currently, she is an Associate Professor with the same university and serves as the Assistant Dean for Quality Assurance. Her research focus is on E-Democracy, and E-Government.

Thinagaran Perumal

Thinagaran Perumal is the recipient of 2014 Early Career Award from IEEE Consumer Electronics Society for his pioneering contribution in consumer electronics. He completed his PhD at Universiti Putra Malaysia. He is currently an Associate Professor attached with the same university. His research interests are towards smart homes and IoT.

Pornchai Mongkolnam

Pornchai Mongkolnam earned his Ph.D. in computer science from Arizona State University, specializing in software engineering, artificial neural networks, and AI. He now serves as an Associate Professor at King Mongkut’s University of Technology Thonburi. His recent works include stress and mood recognition systems and practical applications.