Reliability of Perceived Usability Assessment via Crowdsourcing Platform: Retrospective Analysis and Novel Feedback Quality Inspection Method

ABSTRACT

This paper discusses the differences in perceived usability evaluations between crowdsourcing platform and laboratory testing. A retrospective evaluation measurement method was adopted. A SUS quality inspection method with two new sets of inspection items was established to more precisely observe certain low-quality feedback. We also tested a combination of the traditional check method (Items 2 and 3) and all inspection item pairs to observe the cleaning effect. The results of this study suggests that (i) crowdsourcing platform scores (using SUS) are higher than laboratory test scores, though their quality may not be as good, while the difference in CGS was only one grade. (ii) Using one inspection item pair is an acceptable cleaning method, but it may lead to over-cleaning. (iii) The internal relevance of the inspection item pairs is very high. The SUS quality inspection method proposed in this paper can improve the accuracy of perceived usability evaluations. (iv) The completion-time-based cleaning method may not require setting a maximum time in retrospective usability testing.

Additional information

Notes on contributors

Wang Yuhui

Wang Yuhui is a researcher of the Industrial Design Department, Huazhong University of Science and Technology. His research interests HCI design, usability assessment and product design.

Lei Tian

Lei Tian Lei is an associate professor at the School of Mechanical Science and Engineering, Huazhong University of Science and Technology, where he is Secretary. His research covers information visualization in medicine and engineering, usability, and mobile HCI.

Liu Xinxiong

Liu Xinxiong is a Professor of Industrial Design Department, Huazhong University of Science and Technology. His research interests usability and product design.