Operational outlier detection

Abstract

In most physical sciences, vast amounts of data are collected which must be edited for erroneous data points (or outliers) before they can be analyzed. The nature of the data is such that standard data-editing procedures are not applicable, for the data typically have a varying mean and covariance structure. Furthermore, data sets must be analyzed in real-time. In this paper we propose a data adaptive approach and present two methods which address these conditions and perform operational data editing. The first method divides the data into small segments to fit low order polynomials and splines. The second procedure is based on a technique which, in the Electrical Engineering literature, is known as Linear Adaptive Prediction (LAP). This method treats each data value as it becomes available to decide whether or not it is an outlier.

Keywords:

• Data segmenting
• low order polynomials
• cubic splines
• data adaptive procedures
• linear adaptive prediction
• Kalman filtering

^*Now with CIRES, University of Colorado/NOAA, Boulder CO 80309

^**The National Center for Atmospheric Research is sponsored by the National Science Foundation

^*Now with CIRES, University of Colorado/NOAA, Boulder CO 80309

^**The National Center for Atmospheric Research is sponsored by the National Science Foundation

Notes

^*Now with CIRES, University of Colorado/NOAA, Boulder CO 80309

^**The National Center for Atmospheric Research is sponsored by the National Science Foundation