Challenges to Assessing Usability in the Wild: A Case Study

Abstract

This article describes one part of a human factors study conducted over 3 months in a petro-chemical manufacturing plant in Australia. The project had two purposes, namely, to identify issues to be included in a training course for plant operators and to identify low-level usability-related software issues that might be rectifiable prior to system implementation. After interviewing 28 operators and eight managers, the operators were observed on the job while interacting with the old system. Finally, the 3-part usability assessment comprising 2 expert inspections and a user-based quasi-walkthrough was conducted. As the study took place shortly before a new, off-the-shelf automated manufacturing system was implemented, it was not possible to test an interactive version, relying instead exclusively on static screens. This made it impossible to provide user performance data, which could have helped to convince management of the seriousness of certain problems. One of these proved so severe that an engineer had to be present 24/7 in the control room for 6 months following system cutover because the operators were unable to achieve the required product quality. Based on the data, suggestions are made for expanding the usability construct to include assessment of perceived technology usefulness and to refine the concept of attitude in mandatory settings.

ACKNOWLEDGEMENTS

The helpful comments by two anonymous reviewers of an earlier draft of this article are gratefully acknowledged. I also thank Jim Lewis for his patience in nurturing this along.

Additional information

Notes on contributors

Gitte Lindgaard

Gitte Lindgaard is currently Professor of Strategic Design at Swinburne University of Technology and a Distinguished Research Professor at Carleton University. She has published over 240 peer-reviewed articles and has served on several international HCI journal editorial boards since 1988. Her current research interests include human decision making, human performance, and the link between aesthetics, UX, Usability.