Heuristics for Evaluating IT Security Management Tools

Abstract

The usability of IT security management (ITSM) tools is hard to evaluate by regular methods, making heuristic evaluation attractive. In this article, we explore how domain specific heuristics are created by examining prior research in the area of heuristic and guideline creation. We then describe our approach of creating usability heuristics for ITSM tools, which is based on guidelines for ITSM tools that are interpreted and abstracted with activity theory. With a between-subjects study, we compared the employment of the ITSM and Nielsen's heuristics for evaluation of a commercial identity management system. Participants who used the ITSM set found more problems categorized as severe than those who used Nielsen's. We analyzed several aspects of our heuristics including the performance of individual participants using the heuristic, the performance of individual heuristics, the similarity of our heuristics to Nielsen's, and the participants' opinion about the use of heuristics for evaluation of IT security tools. We then discuss the implications of our results on the use of ITSM and Nielsen's heuristics for usability evaluation of ITSM tools.

NOTES

Notes

¹ See Carroll and Rosson (1992) for details of the claims analysis method.

² A theoretical interpretation or explanation of a delimited problem in a particular area (Charmaz, 2006).

³ A theoretical rendering of a generic issue or process that cuts across several substantive areas of study (Charmaz, 2006).

⁴ HCI-Sec is a mailing list for those who do research on usability of security technologies.

⁵ There was one outlier with 8 years of professional computer security experience in the ITSM condition. Removing the outlier changes the average years of professional computer security experience to 0.46, and variance to 0.44.

⁶ To allow comparison, and because the mentioned experiments employed more evaluators, we assumed that the total number of problems in each experiment was equal to the problems found by aggregate size of 14.

Background. This article is based on the doctoral research of the first author.

Acknowledgments. We thank study participants for their time, and members of the Laboratory for Education and Research in Secure Systems Engineering who provided valuable feedback on the earlier drafts of this article. Cormac Herley provided feedback in May 2010 on the design of the project. We thank Robert Biddle for his insightful feedback on several occasions throughout the project. Valuable comments from the reviewers helped us to improve the article significantly.

Funding. This research has been partially supported by CA Technologies and by the NSERC Internetworked Systems Security Network (ISSNet).

HCI Editorial Record. First manuscript received December 16, 2011. Revisions received February 4, 2013, and June 5, 2013. Accepted by Clayton Lewis. Final manuscript received June 18, 2013. — Editor

Additional information

Notes on contributors

Pooya Jaferian

Pooya Jaferian ([email protected], ece.ubc.ca/~pooya) is a human–computer interaction researcher with an interest in usable privacy and security; he is a Ph.D. Candidate in the Department of Electrical and Computer Engineering, University of British Columbia.

Kirstie Hawkey

Kirstie Hawkey ([email protected], web.cs.dal.ca/~hawkey) is a human–computer interaction researcher with an interest in usable privacy and security; she is an Assistant Professor in the Faculty of Computer Science, Dalhousie University.

Andreas Sotirakopoulos

Andreas Sotirakopoulos ([email protected]) is a computer engineer with an interest in usable privacy and security; he is a systems administrator in VoiceWeb S.A.

Maria Velez-Rojas

Maria Velez-Rojas ([email protected]) is a human factors researcher with an interest in design of usable and efficient visualization techniques of complex IT environments; she is a Senior Research Engineer with CA Labs, San Jose.

Konstantin Beznosov

Konstantin Beznosov ([email protected], konstantin.beznosov.net) is a computer security researcher with an interest in usable privacy and security; he is an Associate Professor at the Department of Electrical and Computer Engineering, University of British Columbia.