A Primer on Visualizations for Comparing Populations, Including the Issue of Overlapping Confidence Intervals

ABSTRACT

In comparing a collection of K populations, it is common practice to display in one visualization confidence intervals for the corresponding population parameters θ₁, θ₂, …, θ_K. For a pair of confidence intervals that do (or do not) overlap, viewers of the visualization are cognitively compelled to declare that there is not (or there is) a statistically significant difference between the two corresponding population parameters. It is generally well known that the method of examining overlap of pairs of confidence intervals should not be used for formal hypothesis testing. However, use of a single visualization with overlapping and nonoverlapping confidence intervals leads many to draw such conclusions, despite the best efforts of statisticians toward preventing users from reaching such conclusions. In this article, we summarize some alternative visualizations from the literature that can be used to properly test equality between a pair of population parameters. We recommend that these visualizations be used with caution to avoid incorrect statistical inference. The methods presented require only that we have K sample estimates and their associated standard errors. We also assume that the sample estimators are independent, unbiased, and normally distributed.

Keywords:

• Confidence intervals
• Overlap
• Two-sample test
• Visualizations

Acknowledgments

The views expressed are those of the authors and not necessarily those of the U.S. Bureau of the Census. The authors are grateful to their colleagues (Derrick Simmons, Jun Shao, Carolina Franco, Josh Tokle, Pat Hunley, and Sarah Wilson) who read various drafts and to the referees and editors for many helpful suggestions.