Assessing User Satisfaction in the Era of User Experience: Comparison of the SUS, UMUX, and UMUX-LITE as a Function of Product Experience

Abstract

Nowadays, practitioners extensively apply quick and reliable scales of user satisfaction as part of their user experience analyses to obtain well-founded measures of user satisfaction within time and budget constraints. However, in the human–computer interaction literature the relationship between the outcomes of standardized satisfaction scales and the amount of product usage has been only marginally explored. The few studies that have investigated this relationship have typically shown that users who have interacted more with a product have higher satisfaction. The purpose of this article was to systematically analyze the variation in outcomes of three standardized user satisfaction scales (SUS, UMUX, UMUX-LITE) when completed by users who had spent different amounts of time with a website. In two studies, the amount of interaction was manipulated to assess its effect on user satisfaction. Measurements of the three scales were strongly correlated and their outcomes were significantly affected by the amount of interaction time. Notably, the SUS acted as a unidimensional scale when administered to people who had less product experience but was bidimensional when administered to users with more experience. Previous findings of similar magnitudes for the SUS and UMUX-LITE (after adjustment) were replicated but did not show the previously reported similarities of magnitude for the SUS and the UMUX. Results strongly encourage further research to analyze the relationships of the three scales with levels of product exposure. Recommendations for practitioners and researchers in the use of the questionnaires are also provided.

ACKNOWLEDGMENTS

We thank Dr. James R. Lewis, senior human factors engineer at IBM Software Group and guest editor of this special issue, for his generous feedback during the preparation of this paper.

Additional information

Notes on contributors

Simone Borsci

Simone Borsci is a Research Fellow in Human Factors at Imperial College of London NHIR-Diagnostic Evidence Cooperative group. He has over 10 years of experience as psychologist and HCI expert in both industry and academia. He has worked as the UX lead of the Italian Government’s working group on usability, and as a researcher at University of Perugia, Brunel University, and Nottingham University.

Stefano Federici

Stefano Federici is currently Associate Professor of General Psychology at the University of Perugia. He is the coordinator of a research team of CognitiveLab at University of Perugia (www.cognitivelab.it). His research is focused on assistive technology assessment processes, disability, and cognitive and human interaction factors.

Silvia Bacci

Michela Gnaldi is currently an Assistant Professor of Social Statistics at the Department of Political Sciences of the University of Perugia. Her main research interest concerns measurement in education. On this topic, she participated in several research projects of national interest in Italy and in the UK, where she has been working as a statistician and researcher at the National Foundation for Educational Research.

Michela Gnaldi

Silvia Bacci has been an Assistant Professor of Statistics at the University of Perugia. Her research interests concern latent variable models, with a special focus on models for categorical and longitudinal/multilevel data, latent class models, and item response theory models. Now she participates in a FIRB project funded by the Italian government.

Francesco Bartolucci

Francesco Bartolucci is Full Professor of Statistics at the Department of Economics of University of Perugia. He is the Principal Investigator of the research project “Mixture and latent variable models for causal inference and analysis of socio-economic data” (FIRB 2012 - “Futuro in ricerca” – Italian Government).