Analysis of an Intervention for Small Unmanned Aerial System (SUAS) Accidents: A Case Study Involving Simpson's Paradox

Abstract

ABSTRACT We investigate a process intervention designed to lower small unmanned aerial system (SUAS) mishap rates. The intervention implemented review boards, required various certifications, and produced apparent decreases in flight failures and flight damage, albeit at increased costs. However, we demonstrate how the results actually represent an undetected example of Simpson's Paradox in which aggregated results yield differing conclusions than disaggregated results. We use the Wilson score method to produce confidence intervals on the difference in mishap rates both before and after the intervention. Using this method, we show that no mishap rate reduction was truly attributable to the process intervention. Our case study reinforces the importance of using deeper statistical knowledge when making policy decisions based in part on the statistical analysis of empirical data.

KEYWORDS:

• accident analysis
• intervention
• mishap prevention
• Simpson's Paradox
• small unmanned aerial system (SUAS)
• unmanned aerial vehicle (UAV)

ACKNOWLEDGMENTS

Grateful acknowledgment is due the Air Force Research Laboratory's Munitions Directorate for providing the data for this analysis. The authors also thank Harris Butler for his technical contributions to this analysis.

ABOUT THE AUTHORS

S. E. Wolf is a Senior Business Intelligence Analyst at Pinnacle Entertainment. A former Captain in the U.S. Air Force, he holds an MS in Operations Research from the Air Force Institute of Technology.

R. R. Hill is a professor of operations research in the Department of Operational Sciences, Air Force Institute of Technology. He holds a Ph.D. in industrial and systems engineering from The Ohio State University. He is an associate editor for Naval Research Logistics, the Journal of Simulation, and Military Operations Research. He runs the Science of Test Research Consortium in addition to running the graduate program, Test and Evaluation Certificate Program.

J. J. Pignatiello is a Professor of Operations Research and the head of the Department of Operational Sciences at the Air Force Institute of Technology. He earned his Ph.D. in Industrial and Systems Engineering from The Ohio State University. His research is primarily focused on the areas of statistical process monitoring, design and analysis of experiments, and reliability engineering. He is a Fellow of both the American Society for Quality and the Institute of Industrial Engineers.