Work-Domain Experts as Evaluators: Usability Inspection of Domain-Specific Work-Support Systems

Abstract

Can work-domain experts generate high-impact evaluation results when used as evaluators in usability inspections of domain-specific work-support systems? This study investigated this question empirically. Three applications were evaluated with group-based expert walkthroughs. The walkthroughs were conducted under 2 conditions—with either work-domain experts or usability experts as evaluators. The condition with usability experts served as the background on which to evaluate the performance and impact of the condition with work-domain experts. The work-domain experts identified a smaller number of items (user problem and/or design suggestion) than the usability experts. However, the items identified by the work-domain experts were on average classified as more severe, and the developers (groups or organizations carrying out subsequent development) gave higher priority to items identified by work-domain experts. As a consequence of the higher severity classification and priority given to the work-domain experts' items, it was concluded that work-domain experts may indeed produce high-impact evaluation results when used as evaluators in a usability inspection. The conclusion opens up exciting method development possibilities in the area of usability inspection methods. The study's research design also represents a fresh research approach to the evaluation of usability evaluation methods, utilizing the impact of the evaluation results in the subsequent development process as an evaluation criterion.