Using a baseline with the probability of agreement to compare distribution characteristics

Abstract

Comparing the center or spread of two samples is a common problem with established solutions in the statistics literature. There exist many methods for comparing two samples, including parametric and non-parametric hypothesis tests, equivalence tests, and the probability of agreement. In this paper, we propose a new probability of agreement solution that combines the merits of the existing methods to allow flexible comparisons across a variety of distributional characteristics, the ability for the user to specify and explore what sizes of difference are practically important, with a straightforward decision rule for easy implementation. We focus on applications where interest lies in comparing a new or “comparison” sample with an existing “reference” sample. The new approach introduces a baseline summary that allows one to contextualize an observed difference between two samples by also considering the anticipated difference between two samples taken from the same distribution, accounting for uncertainty due to sample size and the context-specific equivalence threshold. The method is evaluated with a detailed simulation study and then illustrated with an example comparing two samples of powder, where the goal is to evaluate if multiple aspects of the distribution of the particle sizes for a new sample match those of the established reference sample. A Shiny app implementing the methods is provided to facilitate convenient use for new applications.

Keywords:

• Baseline comparison
• CDF comparison
• double-bootstrap
• practical significance
• probability of agreement
• quantile comparison

Additional information

Funding

The work of Lu Lu was supported by Air Force Office of Scientific Research (AFOSR) grant FA2386-18-1-4104.

Notes on contributors

Lu Lu

Lu Lu is an Associate Professor of Statistics in the Department of Mathematics and Statistics at the University of South Florida. She was a postdoctoral research associate in the Statistical Sciences Group at Los Alamos National Laboratory. Her research interests include reliability analysis, design of experiments, response surface methodology, survey sampling, multiple objective/response optimization.

Christine M. Anderson-Cook

Christine M. Anderson-Cook is a Research Scientist in the Statistical Sciences Group at Los Alamos National Laboratory. Her research areas include reliability, design of experiments, multiple criterion optimization, and response surface methodology. She is a Fellow of the American Statistical Association and the American Society for Quality.

Nathaniel T. Stevens

Nathaniel T. Stevens is an Assistant Professor of Statistics in the Department of Statistics and Actuarial Science, and Director of the Business and Industrial Statistics Research Group at the University of Waterloo. His research interests lie at the intersection of data science and industrial statistics; his publications span topics including experimental design and A/B testing, social network modeling and monitoring, survival and reliability analysis, measurement system analysis, and the development of estimation-based alternatives to traditional hypothesis testing.

Luke Hagar

Luke Hagar is a PhD student at the University of Waterloo in the Department of Statistics and Actuarial Science. His current research interests include design of experiments, hypothesis testing, and computational inference.