A Taxonomy of Evaluation Methods for Information Systems Artifacts

Abstract

Artifacts, such as software systems, pervade organizations and society. In the field of information systems (IS) they form the core of research. The evaluation of IS artifacts thus represents a major issue. Although IS research paradigms are increasingly intertwined, building and evaluating artifacts has traditionally been the purview of design science research (DSR). DSR in IS has not reached maturity yet. This is particularly true of artifact evaluation. This paper investigates the “what” and the “how” of IS artifact evaluation: what are the objects and criteria of evaluation, the methods for evaluating the criteria, and the relationships between the “what” and the “how” of evaluation? To answer these questions, we develop a taxonomy of evaluation methods for IS artifacts. With this taxonomy, we analyze IS artifact evaluation practice, as reflected by ten years of DSR publications in the basket of journals of the Association for Information Systems (AIS). This research brings to light important relationships between the dimensions of IS artifact evaluation, and identifies seven typical evaluation patterns: demonstration; simulation- and metric-based benchmarking of artifacts; practice-based evaluation of effectiveness; simulation- and metric-based absolute evaluation of artifacts; practice-based evaluation of usefulness or ease of use; laboratory, student-based evaluation of usefulness; and algorithmic complexity analysis. This study also reveals a focus of artifact evaluation practice on a few criteria. Beyond immediate usefulness, IS researchers are urged to investigate ways of evaluating the long-term organizational impact and the societal impact of artifacts.

KEY WORDS AND PHRASES:

• artifact evaluation
• content analysis
• design evaluation
• design science
• system design
• taxonomy

Notes

1. The term “compositional style”, commonly used in music, was suggested by one reviewer. It has already been applied in IS [35]. In art, composition is “the harmonious arrangement of the parts of a work […] in relation to each other and to the whole” (collinsdictionary.com). Compositional styles in IS artifact evaluation are typical ways of assembling the six dimensions of the taxonomy to form evaluation methods.

Additional information

Notes on contributors

Nicolas Prat

Nicolas Prat (corresponding author; [email protected]) is an associate professor of information systems at ESSEC Business School, France. He holds a Ph.D. in information systems from the University of Paris Dauphine and a Master of Science in management from ESSEC Business School. His research interests include information systems design, business intelligence and analytics, knowledge management, and data provenance. His work has appeared in journals such as Decision Support Systems, Data and Knowledge Engineering, Expert Systems with Applications, and others, and in the proceedings of major academic conferences, including the International Conference on Conceptual Modeling (ER), Hawaii International Conference on Systems Sciences (HICSS), the Workshop on Information Technologies and Systems (WITS), and the European Conference on Information Systems (ECIS).

Isabelle Comyn-Wattiau

Isabelle Comyn-Wattiau is a professor of information and computer systems at the Conservatoire National des Arts et Métiers in Paris (CEDRIC-CNAM Research Center) and at ESSEC Business School. She received an M.S. and a Ph.D. in computer science from the University of Paris 6. Her research interests include information systems design, data warehouse design and redesign, and database integration. She has published more than seventy journal and conference papers on information and database systems. Her research has appeared in journals such as Decision Support Systems, Data and Knowledge Engineering, Expert Systems with Applications, and major academic conferences including the International Conference on Conceptual Modeling (ER) and the European Conference on Information Systems (ECIS).

Jacky Akoka

Jacky Akoka is a professor and holds the Chair in Information Systems at the Conservatoire National des Arts et Métiers in Paris (CEDRIC-CNAM). He received an M.S. in computer science and a doctoral degree in computer science and operations research from the University of Paris 6. He also received a Ph.D. in MIS from MIT Sloan School of Management. He is a professor of information systems at Télécom École de Management (Institut Mines Télécom). He has published more than 120 conference and journal papers on information and decision systems. His research interests include information systems methodologies, decision support systems, and data warehouse design and implementation. His research has appeared in journals such as Decision Support Systems, IEEE Transactions on Computers, Information and Management, Data and Knowledge Engineering, and others, and in the proceedings of major academic conferences including the International Conference on Conceptual Modeling (ER) and the European Conference on Information Systems (ECIS).