Online monitoring of big data streams: A rank-based sampling algorithm by data augmentation

Abstract

In many applications of modern quality control, process monitoring involves a large number of process variables and quality characteristics. Practitioners are desired to attain complete information about the process in order to assure quick detection of shifts that may possibly occur at any variable. However, full information is not always available during online monitoring of big data streams due to limitations of monitoring resources in practice. In this paper, a rank-based monitoring and sampling algorithm based on data augmentation is proposed to quickly detect the mean shifts in a process when only a limited portion of observations are available online. Specifically, at each observation time, the proposed method will automatically augment information for unobservable variables based on the online observations, and then intelligently allocate the monitoring resources to the most suspicious data streams. Comparing to the existing literature, this method is able to accurately infer the status of all variables in a process based on a small number of observable variables and effectively construct a global monitoring statistic with the proposed augmented vector, which leads to a quick detection of the out-of-control status even if limited shifted variables are observed in real time. Simulation studies as well as a real case study on real-time solar flare detection are conducted to demonstrate the efficacy and applicability of the proposed method.

Keywords:

• cumulative sum (CUSUM)
• data augmentation
• partial observations
• rank-based monitoring
• statistical process control (SPC)

Acknowledgments

The authors thank two anonymous reviewers who provided helpful comments on the manuscript that led to significant improvement of the article.

Additional information

Funding

The authors gratefully acknowledge the support provided in part by the National Science Foundation under grant NSF CMMI-1362529, the 3M, and the Air Force Office of Scientific Research under award number FA9550-18-1-0145.

Notes on contributors

Xiaochen Xian

Dr. Xiaochen Xian is an assistant professor in the Department of Industrial and Systems Engineering, University of Florida. Her email address is [email protected].

Chen Zhang

Dr. Chen Zhang is currently an assistant professor in the Department of Industrial Engineering, Tsinghua University. She is a member of ASQ. Her email address is [email protected].

Scott Bonk

Mr. Scott Bonk is a product manager at Belvedere Trading, LLC. His email address is [email protected].

Kaibo Liu

Dr. Kaibo Liu is an associate professor in the Department of Industrial and Systems Engineering, University of Wisconsin–Madison. He is a member of ASQ. His email address is [email protected].