Big Data? Statistical Process Control Can Help!

Abstract

“Big data” is a buzzword these days due to an enormous amount of data-rich applications in different industries and research projects. In practice, big data often take the form of data streams in the sense that new batches of data keep being collected over time. One fundamental research problem when analyzing big data in a given application is to monitor the underlying sequential process of the observed data to see whether it is longitudinally stable, or how its distribution changes over time. To monitor a sequential process, one major statistical tool is the statistical process control (SPC) charts, which have been developed and used mainly for monitoring production lines in the manufacturing industries during the past several decades. With many new and versatile SPC methods developed in the recent research, it is our belief that SPC can become a powerful tool for handling many big data applications that are beyond the production line monitoring. In this article, we introduce some recent SPC methods, and discuss their potential to solve some big data problems. Certain challenges in the interface between the current SPC research and some big data applications are also discussed.

Keywords:

• Covariates
• Data-rich applications
• Dynamic processes
• Feature extraction
• Image data
• Spatio-temporal data

Acknowledgments

We thank the editor, the associate editor, and two anonymous referees for many constructive comments and suggestions, which improved the quality of the article greatly.

Additional information

Funding

This research is supported in part by an NSF grant.