Comparative usability evaluation (CUE-4)

Abstract

This paper reports on the approach and main results of CUE-4, the fourth in a series of Comparative Usability Evaluation studies. A total of 17 experienced professional teams independently evaluated the usability of the website for the Hotel Pennsylvania in New York. Nine teams used usability testing while eight teams used expert reviews. The CUE-4 results document a wide difference in resources applied and issues reported. The teams reported 340 different usability issues. Only nine of these issues were reported by more than half of the teams, while 205 issues (60%) were uniquely reported, that is, no two teams reported the same issue. A total of 61 of the 205 uniquely reported issues were classified as serious or critical problems. The study also shows that there was no practical difference between the results obtained from usability testing and expert reviews for the issues identified. It was not possible to prove the existence of either missed problems or false alarms in expert reviews. The paper further discusses quality measures for usability evaluation productivity.

Keywords:

• Usability testing
• Expert review
• Evaluation
• Usability evaluation methods
• User interface inspection
• Heuristic inspection

Acknowledgements

Thanks to Chauncey Wilson, Robin Jeffries, Hannu Koskela, S⊘ren Lauesen, Ron Perkins, Chris Rourke, Josh Seiden, John Karat and the anonymous reviewers for thoughtful comments on early versions of this paper.