Psychometric Evaluation of the EMO and the SUS in the Context of a Large-Sample Unmoderated Usability Study

Abstract

This article describes the psychometric properties of the Emotional Metric Outcomes (EMO) questionnaire and the System Usability Scale (SUS) using data collected as part of a large-sample unmoderated usability study (n = 471). The EMO is a concise multifactor standardized questionnaire that provides an assessment of transaction-driven personal and relationship emotional outcomes, both positive and negative. The SUS is a well-known standardized usability questionnaire designed to assess perceived usability. In previous research, psychometric evaluation using data from a series of online surveys showed that the EMO and its component scales had high reliability and concurrent validity with loyalty and overall experience metrics but did not find the expected four-factor structure. Previous structural analyses of the SUS have had mixed results. Analysis of the EMO data from the usability study revealed the expected four-factor structure. The factor structure of the SUS appeared to be driven by item tone. The estimated reliability of the SUS (.90) was consistent with previous estimates. The EMO and its subscales were also quite reliable, with the estimates of reliability for the various EMO scales ranging from .86 to .96. Regression analysis using SUS, EMO, and Effort as predictors revealed different key drivers for the outcome metrics of Satisfaction and Likelihood-to-Recommend. The key recommendations are to include the EMO as part of the battery of poststudy standardized questionnaires, along with the SUS (or similar questionnaire), but to be cautious in reporting SUS subscales such as Usable and Learnable.

Additional information

Notes on contributors

James R. Lewis

James R. Lewis is a senior human factors engineer (at IBM since 1981), currently focusing on the design/evaluation of speech applications. He has published influential papers in the areas of usability testing and measurement. His books include Practical Speech User Interface Design and (with Jeff Sauro) Quantifying the User Experience.

Joshua Brown

Joshua Brown is a senior user experience researcher (at State Farm since 2013), currently focusing on home and business insurance, as well as agency production systems. He has provided user experience research and consulting to various projects previously and currently in development at State Farm.

Daniel K. Mayes

Daniel K. Mayes is a manager in the research department (at State Farm since 2005), currently focusing on customer segmentation and its application to business initiatives. He has published groundbreaking papers in the areas of cognitive psychology and emotional research. His previous experience also includes applying novel right brain constructs to other business-to-customer applications such as video game design/evaluation.