Robust change detection for large-scale data streams

Abstract

Robust change point detection for large-scale data streams has many real-world applications in industrial quality control, signal detection, and biosurveillance. Unfortunately, it is highly nontrivial to develop efficient schemes due to three challenges: (1) the unknown sparse subset of affected data streams, (2) the unexpected outliers, and (3) computational scalability for real-time monitoring and detection. In this article, we develop a family of efficient real-time robust detection schemes for monitoring large-scale independent data streams. For each data stream, we propose to construct a new local robust detection statistic called the $L_{\alpha }$-CUSUM (cumulative sum) statistic that can reduce the effect of outliers by using the Box-Cox transformation of the likelihood function. Then the global scheme will raise an alarm based upon the sum of the shrinkage transformation of these local $L_{\alpha }$-CUSUM statistics to filter out unaffected data streams. In addition, we propose a new concept called false alarm breakdown point to measure the robustness of online monitoring schemes and propose a worst-case detection efficiency score to measure the detection efficiency when the data contain outliers. We then characterize the breakdown point and the efficiency score of our proposed schemes. Asymptotic analysis and numerical simulations are conducted to illustrate the robustness and efficiency of our proposed schemes.

Keywords:

• Breakdown point
• change detection
• large-scale data
• robustness

SUBJECT CLASSIFICATIONS:

• 62L15
• 60G40

FUNDING

R. Zhang's research was supported in part by National Science Foundation (NSF) grant ECCS-2112740. The work of Y. Mei was supported in part by NSF grants 1830344 and 2015405.

DISCLOSURE

The authors have no conflicts of interest to report.