Ingredients and Meals Rather Than Recipes: A Proposal for Research That Does Not Treat Usability Evaluation Methods as Indivisible Wholes

Abstract

To better support usability practice, most usability research focuses on evaluation methods. New ideas in usability research are mostly proposed as new evaluation methods. Many publications describe experiments that compare methods. Comparisons may indicate that some methods have important deficiencies, and thus often advise usability practitioners to prefer a specific method in a particular situation. An expectation persists in human–computer interaction (HCI) that results about evaluation methods should be the standard “unit of contribution” rather than favoring larger units (e.g., usability work as a whole) or smaller ones (e.g., the impact of specific aspects of a method). This article argues that these foci on comparisons and method innovations ignore the reality that usability evaluation methods are loose incomplete collections of resources, which successful practitioners configure, adapt, and complement to match specific project circumstances. Through a review of existing research on methods and resources, resources associated with specific evaluation methods, and ones that can complement existing methods, or be used separately, are identified. Next, a generic classification scheme for evaluation resources is developed, and the scheme is extended with project specific resources that impact the effective use of methods. With these reviews and analyses in place, implications for research, teaching, and practice are derived. Throughout, the article draws on culinary analogies. A recipe is nothing without its ingredients, and just as the quality of what is cooked reflects the quality of its ingredients, so too does the quality of usability work reflect the quality of resources as configured and combined. A method, like a recipe, is at best a guide to action for those adopting approaches to usability that are new to them. As with culinary dishes, HCI needs to focus more on what gets cooked, and how it gets cooked, and not just on how recipes suggest that it could be cooked.

Acknowledgments

We thank members of MAUSE Working Group 2 for their contributions to the development of these analyses. In particular, Dominic Furness and Tobias Uldall-Espersen made contributions to parts of the WG2 report (Cockton & Woolrych, 2009) from which some presented material is derived. An archive of the MAUSE project, with all deliverables, can be found at http://www.cost294.org. The follow on COST project, TwinTide (http://twintide.org/), is focused on domain specific resources and requirements that influence the transferability of methods between different domains. Some of the presented material has benefitted from discussions with TwinTide members.