Large-Scale Datastreams Surveillance via Pattern-Oriented-Sampling

Abstract

Monitoring large-scale datastreams with limited resources has become increasingly important for real-time detection of abnormal activities in many applications. Despite the availability of large datasets, the challenges associated with designing an efficient change-detection when clustering or spatial pattern exists are not yet well addressed. In this article, a design-adaptive testing procedure is developed when only a limited number of streaming observations can be accessed at each time. We derive an optimal sampling strategy, the pattern-oriented-sampling, with which the proposed test possesses asymptotically and locally best power under alternatives. Then, a sequential change-detection procedure is proposed by integrating this test with generalized likelihood ratio approach. Benefiting from dynamically estimating the optimal sampling design, the proposed procedure is able to improve the sensitivity in detecting clustered changes compared with existing procedures. Its advantages are demonstrated in numerical simulations and a real data example. Ignoring the neighboring information of spatially structured data will tend to diminish the detection effectiveness of traditional detection procedures. Supplementary materials for this article are available online.

Keywords:

• Design-adaptive test
• Generalized likelihood ratio method
• Kernel smoothing
• Optimal sampling
• Sequential change detection
• Statistical monitoring

Supplementary Materials

The supplementary materials contain some lemmas and additional simulation results.

Acknowledgments

The authors thank the editor, associate editor, and three anonymous referees for their many helpful comments that have resulted in significant improvements in the article. This work was completed when Ren was a postdoctoral researcher at Pennsylvania State University.

Additional information

Funding

Ren and Li’s research were supported by NSF grants DMS 1820702, DMS 1953196 and DMS 2015539. Zou was supported by NNSF of China grants 11931001, 11690015, 11925106, 11771332 and NSF of Tianjin 18JCJQJC46000. Chen was partially supported by Singapore AcRF Tier 1 grant R-266-000-123-114.