Usability Problem Reports for Comparative Studies: Consistency and Inspectability

ABSTRACT

This study explores issues of consistency and inspectability in usability test data analysis processes and reports. Problem reports resulting from usability tests performed by three professional usability labs in three different countries are compared. Each of the labs conducted a usability test on the same product, applying an agreed test protocol that was collaboratively developed by the labs. Each lab first analyzed their own findings as they always do in their regular professional practice. A few weeks later, they again analyzed their findings, but then everyone applied the same method (SlimDEVAN: a simplified version of DEVAN, a method developed for facilitating comparison of findings from usability tests in an academic setting). It was found that levels of agreement between labs did not improve when they all used SlimDEVAN, suggesting that there was inherent subjectivity in their analyses. It was found that consistency of single analyst teams varied considerably and that a method like SlimDEVAN can help in making the analysis process and findings more inspectable. Inspectability is helpful in comparative studies based on identified usability problems because it allows for tracing back findings to original observations as well as for laying bare the subjective parts of the data analysis.

Notes

¹Although log files with SlimDEVAN codes were made for all 8 participants, Team B accidentally reported problems for only 6 participants.

²From now on the person who makes the comparisons between the team reports is referred to as the “comparer” (for reasons of brevity)

Acknowledgments . Many thanks to the anonymous reviewers and to review editor Clayton Lewis for their well-considered and helpful comments and suggestions on an early draft of this article. We also thank the test participants for their efforts in the usability tests; Deana McDonagh, Pelin Gültekin, and Zeynep Karapars for their contribution to the user tests; Nigel Bevan for his contributions in the early phases of the collaborative project; and Jans Aasman for comments on an early draft of this article.

Support . The research reported in this article was funded by the European Union's “Bridges of Knowledge Programme.”

Authors' Present Addresses. Arnold P.O.S. Vermeeren. TU Delft, Faculty of Industrial Design Engineering, Landbergstraat 15, NL 2628 CE Delft, The Netherlands. E-mail: [email protected]. Jelle Attema, EPN | Platform voor de informatiesamenleving, Post box 16168, NL 2500 BD Den Haag, The Netherlands. E-mail: [email protected]. Evren Akar, Department of Industrial Design, Middle East Technical University, Inonu Bulvari, 06531, Ankara, Turkey. E-mail: [email protected]. Huib de Ridder, TU Delft, Faculty of Industrial Design Engineering, Landbergstraat 15, NL 2628 CE Delft, The Netherlands. E-mail: [email protected]. Andrea J. van Doorn. TU Delft, Faculty of Industrial Design Engineering, Landbergstraat 15, NL 2628 CE Delft, The Netherlands. E-mail: [email protected]. Çiğdem Erbuğ Faculty of Architecture, Department of Industrial Design, METU-BILTIR/UTEST Product Usability Unit, Middle East Technical University, 06531 Ankara Turkey. E-mail: [email protected]. Ali E. Berkman, ODTÜ Mimarlýk Fakültesi, Endüstri Ürünleri Tasarýmý Böl. No. 21, 06531 Çankaya - Ankara Turkey. E-mail: [email protected]. Martin C. Maguire, ESRI (Ergonomics and Safety Research Institute), Loughborough University, Ashby Road, Loughborough, LE11 3TU United Kingdom. E-mail: [email protected].

HCI Editorial Record . First manuscript received June 17, 2005. Revision received October 31, 2007. Accepted by Clayton Lewis. Final manuscript received March 3, 2008. —Editor