Identifying Usability Issues for Personalization During Formative Evaluations: A Comparison of Three Methods

Abstract

A personalized system is one that generates unique output for each individual. As a result, personalization has transformed the interaction between the user and the system, and specific new usability issues have arisen. Methods used for evaluating personalized systems should be able to reveal the issues and problems specifically associated with personalization. Therefore this study evaluated three of the most common test methods used to detect usability problems in a personalized search engine. This was done by comparing the comments generated from thinking-aloud, questionnaires, and interviews. Questionnaires and interviews appear to be more useful for assessing specific usability issues for personalization, whereas thinking-aloud generates more comments on the usefulness of the system in the intended context of use and identifies the most critical and serious problems. Interviews, on the other hand, appear to yield a disproportionate number of positive comments. During the formative evaluation of a personalized system it is best to use a combination of thinking-aloud and questionnaires. This article concludes with a summary of implications for practitioners.

Acknowledgments

We thank our colleagues from the Institute for Information Processing and Microprocessor Technology at the Johannes Kepler University in Linz, Austria, for making the Prospector system available for this study, as well as for their assistance and useful suggestions during the evaluation. We are also grateful to the external usability experts who helped in constructing interreliability scores.