Conducting In Situ Evaluations for and With Ubiquitous Computing Technologies

Abstract

To evaluate ubiquitous computing technologies, which may be embedded in the environment, embedded in objects, worn, or carried by the user throughout everyday life, it is essential to use methods that accommodate the often unpredictable, real-world environments in which the technologies are used. This article discusses how we have adapted and applied traditional methods from psychology and human-computer interaction, such as Wizard of Oz and Experience Sampling, to be more amenable to the in situ evaluations of ubiquitous computing applications, particularly in the early stages of design. The way that ubiquitous computing technologies can facilitate the in situ collection of self-report data is also discussed. Although the focus is on ubiquitous computing applications and tools for their assessment, it is believed that the in situ evaluation tools that are proposed will be generally useful for field trials of other technology, applications, or formative studies that are concerned with collecting data in situ.

Notes

¹Other researchers have also used WOz-style evaluations in situ. Examples are gathering requirements for organization-wide groupware systems (White & Lutters, 2003), simulating location information (Dearman, Hawkey, & Inkpen, 2005), and capturing images with a mobile field guide (Davies, Cheverst, Dix, & Hesse, 2005).

²The My Experience (Me) Toolkit and iESP are available from http://seattleweb.intel-research.net/projects/esm/