![Sample our Engineering & Technology journals, sign in here to start your access, latest two full volumes FREE to you for 14 days]({"type":"image" , "src" : "/sda/5933/TOC-Subject-Sample-2021-Engineering-Tech.jpg"})
ABSTRACT
The evaluator effect names the observation that usability evaluators in similar conditions identify substantially different sets of usability problems. Yet little is known about the factors involved in the evaluator effect. We present a study of 50 novice evaluators' usability tests and subsequent comparisons, in teams and individually, of the resulting usability problems. The same problems were analyzed independently by 10 human–computer interaction experts. The study shows an agreement between evaluators of about 40%, indicating a substantial evaluator effect. Team matching of problems following the individual matching appears to improve the agreement, and evaluators express greater satisfaction with the teams' matchings. The matchings of individuals, teams, and independent experts show evaluator effects of similar sizes; yet individuals, teams, and independent experts fundamentally disagree about which problems are similar. Previous claims in the literature about the evaluator effect are challenged by the large variability in the matching of usability problems; we identify matching as a key determinant of the evaluator effect. An alternative view of usability problems and evaluator agreement is proposed in which matching is seen as an activity that helps to make sense of usability problems and where the existence of a correct matching is not assumed.
Notes
Acknowledgments. We are grateful for comments on a draft of this article by Morten Hertzum, Niels Jacobsen, Effie Law, Peter Naur, and Mikael Skov. We also wish to thank the evaluators in the experiment and the persons who assisted with the independent-expert matching: Hasse Clausen, Torkil Clemmensen, Morten Hertzum, Rune Høegh, Janne Juul Jensen, Jesper Kjeldskov, Marta Lárusdóttir, Mie Nørgaard, Mikael Skov, Jan Stage, Georg Strøm, and Tobias Uldall-Espersen.
Support. This work was supported by grant #2106-04-0022 from the Danish Research Council.
Authors' Present Addresses. Kasper Hornbæk, Department of Computer Science, University of Copenhagen, Njalsgade 128, DK-2300 Copenhagen, Denmark. E-mail: [email protected]. Erik Frøkjær, Department of Computer Science, University of Copenhagen, Njalsgade 128, DK-2300 Copenhagen, Denmark. E-mail: [email protected].
HCI Editorial Record. First manuscript received January 12, 2006. Revisions received November 14, 2006 and May 15, 2007. Accepted by Judith Olson. Final manuscript received August 13, 2007.— Editor