Measure of Strength of Evidence for Visually Observed Differences between Subpopulations

Abstract

For measuring the strength of visually observed subpopulation differences, the Population Difference Criterion is proposed to assess the statistical significance of visually observed subpopulation differences. It addresses the following challenges: in high-dimensional contexts, distributional models can be dubious; in high-signal contexts, conventional permutation tests give poor pairwise comparisons. We also make two other contributions: Based on a careful analysis we find that a balanced permutation approach is more powerful in high-signal contexts than conventional permutations. Another contribution is the quantification of uncertainty due to permutation variation via a bootstrap confidence interval. The practical usefulness of these ideas is illustrated in the comparison of subpopulations of modern cancer data.

Keywords:

• Balanced permutations
• Confidence intervals
• Correlation adjustment
• High dimension
• Population criterion difference

Acknowledgments

We thank anonymous reviewers whose constructive comments helped us to greatly improve the presentation of our results.

Disclosure Statement

The authors report there are no competing interests to declare.

Additional information

Funding

Jan Hannig’s research was supported in part by the National Science Foundation under grant no. DMS-1916115, 2113404, and 2210337. J.S. Marron’s research was partially supported by the National Science Foundation grant no. DMS-2113404. Katherine A. Hoadley’s research was partially supported by grant no. U24 CA264021.