Automated Protocol Analysis

Abstract

Over the past 8 years, The TextLab Research Group within the Department of Computer Science at the University of North Carolina has developed a collection of tools and techniques for recording users' interactions with graphics-based direct manipulation computer systems in machine-readable form and for automatically analyzing and displaying those data. This article describes these tools, discusses their methodological context, and considers their implications for software design and studies of human-computer interaction. Tools discussed include the following: tracking users' behaviors and producing a machine-recorded protocol at the level of users' actions, replaying users' sessions from the protocol data, modeling users' strategies using formal cognitive grammars, analyzing user sessions by parsing them with the grammars, and displaying results in visual form- both static and animated- to facilitate interpretation and understanding by researchers. These tools are placed in a methodological context by reviewing issues associated with concurrent think-aloud, keystroke, X-Windows, and video protocols; other support systems for working with these forms of protocol data are also reviewed. The discussion concludes with our reflections on the methodology and its application to computer systems and research objectives different from our own.