Analyses of user rationality and system learnability: performing task variants in user tests

Abstract

No systematic empirical study on investigating the effects of performing task variants on user cognitive strategy and behaviour in usability tests and on learnability of the system being tested has been documented in the literature. The current use-inspired basic research work aims to identify the underlying cognitive mechanisms and the practical implications of this specific endeavour. The focus of our work was to assess user rationality and system learnability. The software application tested was a multilingual learning resource repository. Eleven German and eleven Slovenian participants were involved in two user tests (UTs). Usability problems (UPs) identified in two quasi-isomorphic tasks were categorized with respect to a scheme of associated skills. Actions of the two tasks of each of the 22 users were segmented and coded according to a scheme of cognitive activities. Results showed that generally the users adopted different strategies for working out the given task and its variant, and that the system could be proved learnable. User Rational Action Model and implications for future research on user tests are inferred.

Keywords:

• Rationality
• Learnability
• Usability evaluation
• Task variant
• Situated cognition
• Mental models

Acknowledgements

We are very grateful to the insightful comments given by the three anonymous reviewers on the draft of this paper. Thanks should also go to the test participants who voluntarily took part in the usability evaluation tests. We would like to express our gratitude to the two projects: COST Action 294 (http://www.cost294.org) and PROLEARN (http://www.prolearn-project.org), which enabled the collaboration between the three authors.

Notes

¹Here we differentiate two related but distinct concepts to avoid any confusion: (a) a system's learnability is defined as the ease with which one learns to operate a given system effectively. Normally it is measured in terms of the time or effort taken to get accustomed to the system and its operation and how easy it is to remember operational details; (b) a system's suitability for learning is defined as the extent to which the system can enable a user to learn specific concepts. This quality is primarily determined by the instructional design underlying the system. In this paper, we address only (a) but not (b).

²Transfer, being an age-old theoretical and practical problem, has fallen in and out of the central focus in the history of psychological research. In the mid-1990s heated debates on transfer were instigated by the Situated Cognition movement (Gruber et al. 1999). However, detailed descriptions are beyond the scope of this paper.

³It is a well-recognized fact that thinking aloud is not a perfect method in the usability research, especially when time measurement is involved (van den Haak et al. 2003), because concurrent verbalization and task performance are two processes that can interfere with each other. An elaborated description on this topic, however, will prolong the length of the paper. A caveat is made here that time-on-task (TOT), which equals TCT minus the verbalization time, may be more accurate than TCT. Due to the relatively small amount of verbalization in our cases, we believe that the difference between TOT and TCT will be insignificant.

⁴The TOTE (Test-Operate-Test-Exit) model (Miller et al. 1960) is fundamental to cognitive psychology and the information processing framework. Hence, there is no doubt that our User Rational Model, just like the contemporary work on user modeling in HCI, is related to TOTE. However, we move beyond TOTE and other similar work (e.g. Blandford et al.2001) to address user action by incorporating the concept of situated cognition, given the understanding that human behaviour is not entirely plan-based (Suchman 1987). Further, we embrace the hybridized model that amalgamates the cognitivist-rationalist and situated-constructivist paradigms (section 1), and the User Rational Action Model can well summarize our understanding in this regard.