Performance Is Not Enough: The Story Told by a Rashomon Quartet

Abstract

The usual goal of supervised learning is to find the best model, the one that optimizes a particular performance measure. However, what if the explanation provided by this model is completely different from another model and different again from another model despite all having similarly good fit statistics? Is it possible that the equally effective models put the spotlight on different relationships in the data? Inspired by Anscombe’s quartet, this article introduces a Rashomon Quartet, that is a set of four models built on a synthetic dataset which have practically identical predictive performance. However, the visual exploration reveals distinct explanations of the relations in the data. This illustrative example aims to encourage the use of methods for model visualization to compare predictive models beyond their performance.

KEYWORDS:

• Data visualization
• Explanation
• Model visualization
• Statistical learning

Acknowledgments

We appreciate the careful reviews provided by two anonymous reviewers, who discussed in great detail the weaknesses of the presented models and the potential for better ways to tell the story of the Rashomon Quartet. The results and plots in the article were produced using these R packages: DALEX (Biecek 2018), partykit (Hothorn and Zeileis 2015), randomForest (Liaw and Wiener 2002), neuralnet (Fritsch, Guenther, and Wright 2019), ggplot2 (Wickham 2016), GGally (Schloerke et al. 2023).

Disclosure Statement

We have no conflicts of interest to disclose.

Notes

1 Linear models are a very versatile family; here we consider only its simplest (default) subset of models with a linear relationship between x_i and y.

2 For illustrative purposes, we have limited ourselves to trees with a maximum depth of 3.

3 Consistently obtaining results in neural network fits is still hard, so this fit was ensured by controlling the random number seed and parameterization.

Additional information

Funding

The first three authors contributions were financially supported by SONATA BIS grant 2019/34/E/ST6/00052 from the Polish National Science Centre (NCN).