Workflow management for ETL development

Abstract

The management of data, including collection, control, movement, transformation and storage, has moved from a background consideration to the forefront of maintaining and increasing a business’ competitive edge. Accordingly, the processes, technologies, activities, stakeholders and governance around developing Extract, Transform and Load (ETL) applications have increased in number and complexity. To ensure ETL solutions can be delivered at the speed of business while maintaining quality, availability and stability of the live environment, these constraints must be dealt with as efficiently as possible. Simply removing steps is often not possible due to separation of duty, legal or business/project/information technology (IT) process best-practices requirements. Therefore, taking action to improve and then manage the process touch-points, workflows and inputs/outputs becomes critical to success. This article proposes a Project Lifecycle Integration Tool (PLIT) which centralizes processes and information, manages and automates workflow, and logs information necessary to monitor and improve quality of processes and output in support of an ETL development environment.

Keywords:

• workflow
• workflow management
• ETL
• data warehouse
• project life cycle

Acknowledgments

The authors wish to thank the anonymous referees for their useful comments which have led to this improved version of the article.