Data Cleansing Decisions: Insights from Discrete-Event Simulations of Firm Resources and Data Quality

Abstract

The cost of poor data quality has been measured in the billions of dollars annually. However, deriving coherent data cleansing strategies to improve data quality is challenging because it is often difficult to justify the financial and human capital cost involved in cleaning data. But those who have planned and designed an effective approach to cleaning data report significant benefits. Although extant literature has extensively focused on data quality issues, little attention has been directed toward providing decision-making techniques that help practitioners determine the cost and benefits of adopting data-cleansing approaches. This study advances an approach that illustrates how discrete-event simulation can be used as a decision tool for making data-cleansing decisions, by understanding the interactions among the firms' resources and performance outcomes. To our knowledge, this is one of the first studies to apply discrete-event simulation for evaluating data-cleansing approaches. The article contributes to an understanding of how various organizational resources interact within, and between, two data-cleansing approaches to drive performance outcomes. Simulation approaches such as the one examined here reveal how the complexity of interactions among such factors can produce results that are difficult to anticipate using other approaches.

Keywords:

• data quality
• data cleansing
• discrete-event simulation
• data warehouse
• data integration
• data profiling
• dirty data

Notes

¹The Extract Transform and Load approach, commonly referred to as ETL, is a data integration approach. It focuses on accessing, cleaning, and consolidating enterprise wide data into a single data repository such as a data warehouse.