Contextual anomaly detection for high-dimensional data using Dirichlet process variational autoencoder

Abstract

Due to recent advances in sensing technologies, response measurements of various sensors are frequently used for system monitoring purposes. However, response data are often affected by some contextual variables, such as equipment settings and time, resulting in different patterns, even when the system is in the normal state. In this case, anomaly detection methods that do not consider contextual variables may be unable to distinguish between abnormal and normal patterns of the response data affected by the contextual variables. Motivated by this problem, we propose a method for contextual anomaly detection, particularly in the case where the response and contextual variables are both high-dimensional and complex. The proposed method is based on Variational AutoEncoders (VAEs), which are neural-network-based generative models suitable for modeling high-dimensional and complex data. The proposed method combines two VAEs: one for response variables and the other for contextual variables. Specifically, in the latent space of the VAE for contextual variables, we model the latent variables using a Dirichlet process Gaussian mixture model. Consequently, the effects of the contextual variables can be modeled using several clusters, each representing a different contextual environment. The latent contextual variables are then used as additional inputs to the other VAE’s decoder for reconstructing response data from their latent representations. We then detect the anomalies based on the negative reconstruction loss of a new response observation. The effectiveness of the proposed method is demonstrated using several benchmark datasets and a case study based on a global tire company.

KEYWORDS:

• Contextual anomaly detection
• Dirichlet process mixture model
• high-dimensional data
• variational autoencoder

Data availability statement

The datasets in Section 5 are available at http://archive.ics.uci.edu/ml, and the dataset in Section 6 is not publicly available due to confidentiality.

Acknowledgments

The authors would like to thank the referees, the associate editor, and the editor for reviewing this article and providing valuable comments.

Funding

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2018R1C1B6004511, 2020R1A4A10187747).

Additional information

Notes on contributors

Hyojoong Kim

Hyojoong Kim received a BS degree in industrial engineering from Hanyang University in Korea and an M.S. degree in industrial and systems engineering from KAIST. He is currently a PhD candidate in industrial and system engineering at KAIST. His research interests include machine learning and applied statistics.

Heeyoung Kim

Heeyoung Kim received a BS degree in industrial engineering from KAIST, MS degrees in statistics and industrial engineering from the Georgia Institute of Technology and KAIST, and a PhD degree in industrial engineering from the Georgia Institute of Technology. She is an associate professor with the Department of Industrial and Systems Engineering, KAIST. She was a Senior Member of Technical Staff with AT&T Laboratories. Her research interests include applied statistics and machine learning.