Argumentation Models for Usability Problem Analysis in Individual and Collaborative Settings

ABSTRACT

Consolidating usability problems (UPs) from problem lists from several users can be a cognitively demanding task for evaluators. It has been suggested that collaboration between evaluators can help this process. In an attempt to learn how evaluators make decisions in this process, the authors studied what justification evaluators give for extracting UPs and their consolidation when working both individually and collaboratively. An experiment with eight novice usability evaluators was carried out where they extracted UPs and consolidated them individually and then collaboratively. The data were analyzed by using conventional content analysis and by creating argumentation models according to the Toulmin model. The results showed that during UP, extraction novice usability evaluators could put forward warrants leading to clear claims when probed but seldom added qualifiers or rebuttals. Novice usability evaluators could identify predefined criteria for a UP when probed and this could be acknowledged as a backing to warrants. In the individual settings, novice evaluators had difficulty in presenting claims and warrants for their decisions on consolidation. Although further study is needed, the results of the study indicated that collaborating pairs had a tendency to argue slightly better than individuals. Through the experiment novice evaluators’ reasoning patterns during problem extraction and consolidation as well as during their assessment of severity and confidence could be identified.

Additional information

Notes on contributors

Ebba Thora Hvannberg

Ebba Thora Hvannberg is a professor of computer science, School of Engineering and Natural Sciences, University of Iceland. Her research areas include human–computer interaction and software engineering. The main focus has been on methods for usability evaluation in various areas such as e-learning management, air-traffic control, and crisis management.

Effie L.-C. Law

Effie L.-C. Law is a full professor in human–computer interaction, University of Leicester, UK. Her main research interest is usability and user experience methodologies, which are applicable to various domains, including technology-enhanced learning, health care, and cultural heritage. Her current focus is on automatic emotion analysis with multisensory data and on the relation between emerging technology and well-being.

Gyda Halldorsdottir

Gyda Halldorsdottir is a freelance working project manager, University of Iceland and University Hospital of Iceland. Her research areas include HCI and health informatics. The main focus has been on usability evaluation, air traffic control, crisis management, and data management.