Monitoring of high-dimensional and high-frequency data streams: A nonparametric approach

ABSTRACT

High-dimensional and high-frequency data streams (HHDS) have become widely available with the prevalence of data-acquisition technologies. Because HHDS have the characteristics of high dimensionality and temporal correlation, traditional statistical process control (SPC) techniques cannot be used directly to address the problem of monitoring HHDS. Most extant monitoring methods commonly assume that the process observations are independent or can be described by some parametric models and face the challenge posed by high-dimensional data. Little research has been conducted on the robust monitoring of HHDS that can simultaneously accommodate high dimensions and high frequencies. Therefore, in this paper, we propose a novel nonparametric approach for monitoring HHDS, based on data decorrelation, dimension reduction, and SPC. Specifically, process observations are first sequentially decorrelated and standardized using Cholesky decomposition, and then the $\mathit{k}$-nearest neighbors classification algorithm is applied to transform uncorrelated high-dimensional data into one-dimensional data. Finally, the traditional cumulative-sum procedure is used for online monitoring, based on an empirical log-likelihood ratio test. Numerical studies have shown that our monitoring scheme is sufficiently reliable and efficient for the online monitoring of HHDS. An illustrative example about the Dow Jones 30 industrial stock prices is presented to further validate the proposed approach.

KEYWORDS:

• Data decorrelation
• high-dimensional and high-frequency data streams
• $\mathit{k}$-nearest neighbors algorithm
• nonparametric control charts
• statistical process control

Acknowledgements

The authors would like to thank the editors and two anonymous referees for many helpful comments and suggestions, which have greatly improved the quality of the paper.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Natural Science Foundation of China under grant numbers 71902138, 72002066, 71701188, 72231005, 72261147706, and 72032005, and the State Administration of Science, Technology and Industry for National Defense under grant numbers JSZL2021204B001 and JSZL2022204B005.

Notes on contributors

Zhiqiong Wang

Zhiqiong Wang is an associate professor at the School of Management, Tianjin University of Technology, China. He received his Ph.D. degree in management science and engineering from Tianjin University, China, in 2018. His major research interests include quality control and management, statistical process control, and change-point detection.E-mail: [email protected]

Xin Li

Xin Li received her Master’s degree in management science and engineering from Tianjin University of Technology, China, in 2024. Her major research interests are quality management and statistical process control.E-mail: [email protected]

Ying Wang

Ying Wang is an associate professor at the School of Management, Tianjin University of Technology, China. She received her Ph.D. degree in 2008 from Tianjin University in China. Her major research interests include quality management and quality control, service quality management, and education quality management. E-mail: [email protected]

Yanhui Ma

Yanhui Ma is a professor at the School of Management, Tianjin University of Technology, China. He received his Ph.D. degree in 2008 from Tianjin University in China. His major research interests include quality control and management, supply chain quality management, and machine learning.E-mail: [email protected]

Li Xue

Li Xue is a full Professor and Associate Dean of the School of Management Engineering at the Zhengzhou University of Aeronautics, China. Her major research areas include quality control and management, and big data analytics. She has published over 40 papers in these areas, and her research has been well supported by grants from the National Science Foundation of China, Aeronautic Science Foundation of China, and other grant agencies.E-mail: [email protected]