Investigating the Influence of Different Factors on the UX Evaluation of a Mobile Application

Abstract

User eXperience (UX) evaluations play an essential role in the software development process. As the results from such evaluations can drive future releases, it is necessary to identify which factors can substantially change users’ judgments about their experience to have more precise results and understand UX better. This article investigates how interaction sequencing, previous experience, and the number of problems could affect overall satisfaction and the two main UX dimensions: pragmatic and hedonic. We employed three different evaluation methods to evaluate a chatbot-based mobile shopping application. The results revealed that participants with previous experience with similar apps tended to give lower ratings. We also found that as inspectors identify more problems, they tend to rate the pragmatic dimension lower. Finally, we did not identify a significant influence of interaction sequencing on UX evaluation. We discuss the reasons for these results, the implications for practitioners and researchers, and research opportunities.

Acknowledgments

We especially thank professors Edleno Silva de Moura and Altigran Soares da Silva and their students for all the support to carry out this research. We also thank all the subjects who participated in the study.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 https://www.nytimes.com/2021/03/17/technology/learning-apps-students.html.

Additional information

Funding

This research was supported by the Brazilian funding agencies FAPEAM through process number 062.00150/2020, and FAPESP through grant #2020/05191-2, the Coordination for the Improvement of Higher Education Personnel-Brazil (CAPES) financial code 001, and CNPq processes 204081/2018-1/PDE, 314174/2020-6, 308513/2020-7. This research was also funded by Samsung Electronics of Amazonia Ltd., according to Article 48 of Decree n° 6.008/2006 (SUFRAMA), under the terms of Federal Law n° 8.387/1991, through agreement n° 003/2019 (PROPPGI), signed with ICOMP/UFAM.

Notes on contributors

Walter T. Nakamura

Walter T. Nakamura holds a PhD in Informatics from the Federal University of Amazonas (UFAM) and is an adjunct professor at the Federal University of Technology-Paraná (UTFPR). His research interests include the intersection between Software Engineering, Human-Computer Interaction, and Machine Learning fields to develop quality products.

Leonardo C. Marques

Leonardo C. Marques is a PhD student in Computer Science from the Federal University of Amazonas (UFAM). He is currently researching on UX of the crowd, in addition to investigating how to evaluate the UX when the app is a part of the interaction, but it is not the main interaction itself.

David Redmiles

David Redmiles is a Professor in the Department of Informatics at the University of California, Irvine (UCI) in the Donald Bren School of Information and Computer Sciences (ICS). His current research focuses on distributed and collaborative software engineering, especially the aspects of awareness and trust among collaborators.

Elaine H. T. de Oliveira

Elaine H. T. de Oliveira is an associate professor at the Federal University of Amazonas (UFAM) and holds a PhD in Informatics in Education from the Federal University of Rio Grande do Sul (UFRGS). Research interests: Informatics in Education, Artificial Intelligence applied to Education, Cognitive Science, and Adaptive Systems.

Tayana Conte

Tayana Conte is an associate professor at the Federal University of Amazonas (UFAM), Institute of Computing (IComp). Her research interests include the intersection between Software Engineering and Human-Computer Interaction, Software Quality, Human-Centered Computing, and Empirical Software Engineering.