A Computational Model of “Active Vision” for Visual Search in Human–Computer Interaction

Abstract

Human visual search plays an important role in many human–computer interaction (HCI) tasks. Better models of visual search are needed not just to predict overall performance outcomes, such as whether people will be able to find the information needed to complete an HCI task, but to understand the many human processes that interact in visual search, which will in turn inform the detailed design of better user interfaces. This article describes a detailed instantiation, in the form of a computational cognitive model, of a comprehensive theory of human visual processing known as “active vision” (Findlay & Gilchrist, 2003). The computational model is built using the Executive Process-Interactive Control cognitive architecture. Eye-tracking data from three experiments inform the development and validation of the model. The modeling asks—and at least partially answers—the four questions of active vision: (a) What can be perceived in a fixation? (b) When do the eyes move? (c) Where do the eyes move? (d) What information is integrated between eye movements? Answers include: (a) Items nearer the point of gaze are more likely to be perceived, and the visual features of objects are sometimes misidentified. (b) The eyes move after the fixated visual stimulus has been processed (i.e., has entered working memory). (c) The eyes tend to go to nearby objects. (d) Only the coarse spatial information of what has been fixated is likely maintained between fixations. The model developed to answer these questions has both scientific and practical value in that the model gives HCI researchers and practitioners a better understanding of how people visually interact with computers, and provides a theoretical foundation for predictive analysis tools that can predict aspects of that interaction.

Notes

Background. This article is based on the doctoral thesis of the first author.

Acknowledgments. Thanks to Beth Halverson, Richard Young, Dario Salvucci, and an anonymous reviewer for their feedback on this article.

Support. This work was supported by Office of Naval Research grants N00014-02-10440 and N00014-06-10054, and by National Science Foundation grant IIS-0308244; all were awarded to the University of Oregon with Anthony J. Hornof as the principal investigator. The opinions expressed here do not necessarily reflect the views of the funding agencies.

HCI Editorial Record. First manuscript received April 3, 2009. Revision received August 2, 2010. Final received September 29, 2010. Accepted by Richard Young.