From Black Box to Shining Spotlight: Using Random Forest Prediction Intervals to Illuminate the Impact of Assumptions in Linear Regression

Abstract

We introduce a pair of Shiny web applications that allow users to visualize random forest prediction intervals alongside those produced by linear regression models. The apps are designed to help undergraduate students deepen their understanding of the role that assumptions play in statistical modeling by comparing and contrasting intervals produced by regression models with those produced by more flexible algorithmic techniques. We describe the mechanics of each approach, illustrate the features of the apps, provide examples highlighting the insights students can gain through their use, and discuss our experience implementing them in an undergraduate class. We argue that, contrary to their reputation as a black box, random forests can be used as a spotlight, for educational purposes, illuminating the role of assumptions in regression models and their impact on the shape, width, and coverage rates of prediction intervals.

Keywords:

• Interactive visualization
• Machine learning
• Statistics education

Supplementary Materials

Supplementary materials include 1) additional examples from the simulation app, illustrating different scenarios involving violations of model assumptions, 2) full results from the student response survey, and 3) instructions for the lab activity accompanying the labs.

Acknowledgments

We acknowledge and thank two anonymous reviewers, an associate editor and an editor, whose suggestions helped us strengthen the apps and the article.

Disclosure Statement

The authors report there are no competing interests to declare.