Abstract
The usability of IT security management (ITSM) tools is hard to evaluate by regular methods, making heuristic evaluation attractive. In this article, we explore how domain specific heuristics are created by examining prior research in the area of heuristic and guideline creation. We then describe our approach of creating usability heuristics for ITSM tools, which is based on guidelines for ITSM tools that are interpreted and abstracted with activity theory. With a between-subjects study, we compared the employment of the ITSM and Nielsen's heuristics for evaluation of a commercial identity management system. Participants who used the ITSM set found more problems categorized as severe than those who used Nielsen's. We analyzed several aspects of our heuristics including the performance of individual participants using the heuristic, the performance of individual heuristics, the similarity of our heuristics to Nielsen's, and the participants' opinion about the use of heuristics for evaluation of IT security tools. We then discuss the implications of our results on the use of ITSM and Nielsen's heuristics for usability evaluation of ITSM tools.
NOTES
Notes
1 See CitationCarroll and Rosson (1992) for details of the claims analysis method.
2 A theoretical interpretation or explanation of a delimited problem in a particular area (CitationCharmaz, 2006).
3 A theoretical rendering of a generic issue or process that cuts across several substantive areas of study (CitationCharmaz, 2006).
4 HCI-Sec is a mailing list for those who do research on usability of security technologies.
5 There was one outlier with 8 years of professional computer security experience in the ITSM condition. Removing the outlier changes the average years of professional computer security experience to 0.46, and variance to 0.44.
6 To allow comparison, and because the mentioned experiments employed more evaluators, we assumed that the total number of problems in each experiment was equal to the problems found by aggregate size of 14.
Background. This article is based on the doctoral research of the first author.
Acknowledgments. We thank study participants for their time, and members of the Laboratory for Education and Research in Secure Systems Engineering who provided valuable feedback on the earlier drafts of this article. Cormac Herley provided feedback in May 2010 on the design of the project. We thank Robert Biddle for his insightful feedback on several occasions throughout the project. Valuable comments from the reviewers helped us to improve the article significantly.
Funding. This research has been partially supported by CA Technologies and by the NSERC Internetworked Systems Security Network (ISSNet).
HCI Editorial Record. First manuscript received December 16, 2011. Revisions received February 4, 2013, and June 5, 2013. Accepted by Clayton Lewis. Final manuscript received June 18, 2013. — Editor
Additional information
Notes on contributors
Pooya Jaferian
Pooya Jaferian ([email protected], ece.ubc.ca/~pooya) is a human–computer interaction researcher with an interest in usable privacy and security; he is a Ph.D. Candidate in the Department of Electrical and Computer Engineering, University of British Columbia.
Kirstie Hawkey
Kirstie Hawkey ([email protected], web.cs.dal.ca/~hawkey) is a human–computer interaction researcher with an interest in usable privacy and security; she is an Assistant Professor in the Faculty of Computer Science, Dalhousie University.
Andreas Sotirakopoulos
Andreas Sotirakopoulos ([email protected]) is a computer engineer with an interest in usable privacy and security; he is a systems administrator in VoiceWeb S.A.
Maria Velez-Rojas
Maria Velez-Rojas ([email protected]) is a human factors researcher with an interest in design of usable and efficient visualization techniques of complex IT environments; she is a Senior Research Engineer with CA Labs, San Jose.
Konstantin Beznosov
Konstantin Beznosov ([email protected], konstantin.beznosov.net) is a computer security researcher with an interest in usable privacy and security; he is an Associate Professor at the Department of Electrical and Computer Engineering, University of British Columbia.