Comparing Classical and Resistant Outlier Rules

Abstract

In this article we compare Rosner's (1983) extension of the outlier t test (the generalized ESD procedure) to the boxplot rules, Tukey's (1977) resistant outlier labeling approach. The underlying principles are contrasted, and the behavior of the procedures is compared in a variety of sampling situations. To facilitate power-type comparisons of procedures that are based on different paradigms, the conditional outside rate, a performance criterion for assessing outlier labeling procedures, is introduced. This quantity represents the probability that an observation, y_i , is labeled an outlier, conditional on its distance from the center, ε = y_i ε μ. Some exact expressions, as well as simple approximations, and a Monte Carlo swindle are developed. The conditional outside rate, together with the outside rate per observation and the some-outside rate per sample, are applied to compare performance of the classical and boxplot rules, based on Monte Carlo simulation of samples of sizes n = 10, 30, and 50 from SLASH contaminated Gaussian distributions. The resistant behavior of the boxplot rules is contrasted with the nonresistant outlier t test. The ESD procedures are then compared with the boxplot rules. Suitably chosen versions of these rules are found to perform in a way similar to boxplot rules in non-Gaussian situations and better in the pure Gaussian case. Relative merits of the two approaches from various other perspectives are also reviewed briefly.

Key Words:

• Exploratory data analysis
• Masking
• Outliers