Outlier-free merging of homogeneous groups of pre-classified observations under contamination

ABSTRACT

We study the problem of merging homogeneous groups of pre-classified observations from a robust perspective motivated by the anti-fraud analysis of international trade data. This problem may be seen as a clustering task which exploits preliminary information on the potential clusters, available in the form of group-wise linear regressions. Robustness is then needed because of the sensitivity of likelihood-based regression methods to deviations from the postulated model. Through simulations run under different contamination scenarios, we assess the impact of outliers both on group-wise regression fitting and on the quality of the final clusters. We also compare alternative robust methods that can be adopted to detect the outliers and thus to clean the data. One major conclusion of our study is that the use of robust procedures for preliminary outlier detection is generally recommended, except perhaps when contamination is weak and the identification of cluster labels is more important than the estimation of group-specific population parameters. We also apply the methodology to find homogeneous groups of transactions in one empirical example that illustrates our motivating anti-fraud framework.

KEYWORDS:

• Anti-fraud analysis
• cluster-wise regression
• fair-price estimation
• F-test
• forward search
• international trade
• masking and swamping
• MM-estimation
• outlier detection

2010 MATHEMATICS SUBJECT CLASSIFICATION:

• 62J05
• 62J20
• 62H30
• 62P20
• 62F35

Acknowledgements

The authors are grateful to Domenico Perrotta and Marco Riani both for their specific comments and for broader discussion of the topic addressed in this work. They also thank one referee and Prof. Andrei Volodin for several helpful suggestions on a previous version of this manuscript.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. The complete description for this CN code is: slag, ash and residues (other than from the manufacture of iron or steel), containing mainly copper.

2. The average number of nominated outliers is given by $n(1-{\pi }_0)\cdot \mathrm{size}+n{\pi }_0\cdot \mathrm{power}$.

Additional information

Funding

The work described in this article has been conducted under the support of the Administrative Arrangement SI2.741170 between the European Anti-Fraud Office (OLAF) and the Joint Research Centre of the European Commission (AMT5, Phase I and Phase II projects), in the framework of the Hercule III Programme on the protection of the financial interests of the European Union.