Concept Drift Monitoring and Diagnostics of Supervised Learning Models via Score Vectors

Abstract

Supervised learning models are one of the most fundamental classes of models. Viewing supervised learning from a probabilistic perspective, the set of training data to which the model is fitted is usually assumed to follow a stationary distribution. However, this stationarity assumption is often violated in a phenomenon called concept drift, which refers to changes over time in the predictive relationship between covariates X and a response variable Y and can render trained models suboptimal or obsolete. We develop a comprehensive and computationally efficient framework for detecting, monitoring, and diagnosing concept drift. Specifically, we monitor the Fisher score vector, defined as the gradient of the log-likelihood for the fitted model, using a form of multivariate exponentially weighted moving average, which monitors for general changes in the mean of a random vector. In spite of the substantial performance advantages that we demonstrate over popular error-based methods, a score-based approach has not been previously considered for concept drift monitoring. Advantages of the proposed score-based framework include applicability to broad classes of parametric models, more powerful detection of changes as shown in theory and experiments, and inherent diagnostic capabilities for helping to identify the nature of the changes.

KEYWORDS:

• Control chart
• Multivariate EWMA
• Predictive model
• Score function

Supplementary Materials

In the online supplementary materials of this article, we provide a file containing derivations of Fisher Decoupling Equations (10)(10) $$\begin{array}{c}\Delta \mathit{\theta }={\mathbb{I}}^{-1}({\mathit{\theta }}^{(0)})E_{{\mathit{\theta }}^{(1)}}[\mathit{s}({\tilde{\mathit{\theta }}}^{(0)};(\mathit{X},Y))],\hfill \end{array}$$(10) and (11)(11) $$\begin{array}{c}\Delta \tilde{\mathit{\theta }}={\tilde{\mathbb{I}}}^{-1}({\mathit{\theta }}^{(0)})E_{{\mathit{\theta }}^{(1)}}[\mathit{s}({\tilde{\mathit{\theta }}}^{(0)};(\mathit{X},Y))],\hfill \end{array}$$(11) , examples demonstrating effectiveness of our methods, simulation study results, and a discussion on potential extensions of our methods. Additionally, the zip file contains python code and data necessary to reproduce some results presented in this article.

Additional information

Funding

This work was funded in part by the Air Force Office of Scientific Research Grant (#FA9550-18-1-0381), which we gratefully acknowledge. This work was also supported by a startup grant (#DMR200021) from the Extreme Science and Engineering Discovery Environment (XSEDE) (Towns et al. 2014), which is supported by the National Science Foundation (#ACI-1548562). We also acknowledge the many helpful comments from the Editor (Dr. Roshan Joseph) and the anonymous Associate Editor and Referees.