Measuring the robustness of predictive probability for early stopping in two-group comparisons

Abstract

Physical experiments are often expensive and time-consuming. Test engineers must certify the compatibility of aircraft and their weapon systems before they can be deployed in the field, but the testing required is time consuming, expensive, and resource limited. Adopting Bayesian adaptive designs is a promising way to borrow from the successes seen in the clinical trials domain. The use of predictive probability (PP) to stop testing early and make faster decisions is particularly appealing given the aforementioned constraints. Given the high-consequence nature of the tests performed in the national security space, a strong understanding of new methods is required before being deployed. Although PP has been thoroughly studied for binary data, there is less work with continuous data, where many reliability studies are interested in certifying the specification limits of components. A simulation study evaluating the robustness of this approach indicates early stopping based on PP is reasonably robust to minor assumption violations, especially when only a few interim analyses are conducted. The simulation study also compares PP to conditional power, showing its relative strengths and weaknesses. A post-hoc analysis exploring whether release requirements of a weapon system from an aircraft are within specification with desired reliability resulted in stopping the experiment early and saving 33% of the experimental runs.

Keywords:

• Bayesian adaptive design of experiments
• conditional power predictive probability
• reliability

Acknowledgements

The authors thank J. Gabriel Huerta from Statistical Sciences organization at Sandia National Laboratories for his helpful comments and edits. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government. IR 1717497.

Data availability statement

The data used for the application is proprietary information and cannot be shared. The simulated data is available upon request.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Daniel Ries

Dr. Daniel Ries is Principal Member of the Technical Staff at Sandia National Laboratories in the Statistics and Data Analytics Department. His statistical research areas include developing explainable AI, incorporating uncertainty quantification in machine learning models, Bayesian modeling, and spatio-temporal modeling. He also serves as Adjunct Clinical Assistant Professor at the University of Illinois Urbana-Champaign. Daniel received his PhD in statistics from Iowa State University in 2017.

Victoria R. C. Sieck

Dr. Victoria R. C. Sieck is an Adjunct Professor at the Air Force Institute of Technology (AFIT), and a Director of Operations in the United States Air Force (USAF). Her research interests include design of experiments, and developing innovate Bayesian approaches to DoD testing. As an Operations Research Analyst in the USAF, her experiences in the USAF testing community include being a weapons and tactics analyst and an operational test analyst. Victoria received her PhD in Statistics from the University of New Mexico in 2021.

Philip Jones

Philip Jones works for Sandia National Laboratories Aircraft Compatibility Department since 2015. He specializes in large scale mechanical compatibility testing for system integration and certification. He received a Master of Science in Mechanical Engineering from Virginia Tech in 2015 with a background in mechatronics systems.

Julie Shaffer

Julie Shaffer is a Principal Member of the Technical Staff at Sandia National Laboratories in the Aircraft Compatibility Department. Her work focuses on mechanical interface and environmental compatibility testing with the Department of Energy and Department of Defense systems. Julie received her master's degree in mechanical engineering from the University of Wisconsin-Madison in 2013.