Eliciting knowledge for software development

Abstract

A current bottleneck in the automation of cognitive tasks, such as software development, is the lack of available, standardized, reliable and valid methods for extracting knowledge from experts. This paper discusses the development of Computer Aided Protocol (CAP) to automatically collect the general and specific cognitive task components of subjects performing a programming task. The effectiveness of CAP is evaluated in a statistically balanced experimental design (n = 30) by comparing it to traditional protocol analysis and a control group. Results indicate that while neither treatment significantly altered the solution process, CAP was able to collect the lower level commands while protocol analysis collected only 56% of these lower level commands. However, protocol analysis was able to obtain significantly more high level goals than CAP. This work suggests that the integration of both protocol and CAP for knowledge extraction would provide more effective information for the development of expert systems than is feasible with either system alone.