Evaluating the Downstream Utility of User Tests and Examining the Developer Effect: A Case Study

Abstract

In this article, a case study on evaluating the downstream utility of user tests performed on a digital library with reference to the developer effect is reported. Downstream utility is defined as the effectiveness with which the resolution to a usability problem (UP) is implemented, and developer effect is defined as developers' bias toward fixing UPs with particular characteristics. To measure the effectiveness of the user tests, the actual impacts of fixing or not fixing the UPs identified were analyzed. To address a theoretical void in studying the persuasive power of usability evaluation results, Information Integration Theory was employed. Six research questions that predict the persuasiveness of different qualities of usability problems to induce fixes and the effectiveness of such fixes were investigated. Multiperspective data have been collected from usability specialists, the development team of the digital library, and its old as well as new users. Implications for reporting UPs and future research are inferred.