Designed Experiments for the Defense Community

ABSTRACT

The areas of application for design of experiments principles have evolved, mimicking the growth of U.S. industries over the last century, from agriculture to manufacturing to chemical and process industries to the services and government sectors. In addition, statistically based quality programs adopted by businesses morphed from total quality management to Six Sigma and, most recently, statistical engineering (see Hoerl and Snee 2010). The good news about these transformations is that each evolution contains more technical substance, embedding the methodologies as core competencies, and is less of a “program.” Design of experiments is fundamental to statistical engineering and is receiving increased attention within large government agencies such as the National Aeronautics and Space Administration (NASA) and the Department of Defense. Because test policy is intended to shape test programs, numerous test agencies have experimented with policy wording since about 2001. The Director of Operational Test & Evaluation has recently (2010) published guidelines to mold test programs into a sequence of well-designed and statistically defensible experiments. Specifically, the guidelines require, for the first time, that test programs report statistical power as one proof of sound test design. This article presents the underlying tenents of design of experiments, as applied in the Department of Defense, focusing on factorial, fractional factorial, and response surface design and analyses. The concepts of statistical modeling and sequential experimentation are also emphasized. Military applications are presented for testing and evaluation of weapon system acquisition, including force-on-force tactics, weapons employment and maritime search, identification, and intercept.

KEYWORDS:

• factorial design
• optimal design
• power
• response surface methodology
• space filling design
• test and evaluation

Notes

¹In reality, because the 16 trials might take 8-10 days to complete, the design might be further blocked in groups of 4-8. Additionally, it would be a good practice to replicate one or more points to objectively estimate pure error.

²In DOE terminology, this is a Resolution V design. One can estimate main effects clear of all but a single four-way interaction, and each two factor interaction is aliased with a single three-factor interaction. Sparsity of effects has empirically found these higher order interactions to be rare.

^a CI = confidence interval.

Additional information

Notes on contributors

Rachel T. Johnson

Dr. Rachel T. Silvestrini (née Johnson) is an Assistant Professor in the Operations Research Department at the Naval Postgraduate School. She received her B.S. in Industrial Engineering from Northwestern University and her M.S. and Ph.D. from Arizona State University. Her research and teaching interests are in statistics and operations research with focus in design of experiments.

Gregory T. Hutto

Gregory T. Hutto is the Wing Operations Analyst for the Air Force's 46 Test Wing at Eglin AFB. He is a past Director and member of the Military Operations Research Society. Mr. Hutto has more than 21 years experience applying the principles of experimental design to military test and evaluation projects ranging from basic laboratory science efforts to large scale military exercises.

James R. Simpson

Dr. James R. Simpson is Chief Operations Analyst for the Air Force's 53rd Test Management Group at Eglin AFB, FL. He is Adjunct Professor at the University of Florida, served formerly as Associate Professor at Florida State University and Associate Professor at the Air Force Academy. He is Chair of the ASQ Journal Editors' Committee, and serves on the ASQ Publication Management Board. He earned a B.S. in Operations Research from the Air Force Academy, an M.S. in OR from the Air Force Institute of Technology, and a Ph.D. in IE from Arizona State University.

Douglas C. Montgomery

Dr. Douglas C. Montgomery is Regents' Professor of Industrial Engineering and Statistics, ASU Foundation Professor of Engineering, and Co-Director of the Graduate Program in Statistics at Arizona State University. He received a Ph.D. in engineering from Virginia Tech. His professional interests are in statistical methodology for problems in engineering and science. He is a recipient of the Shewhart Medal, the George Box Medal, the Brumbaugh Award, the Lloyd S. Nelson award, the William G. Hunter award, and the Ellis Ott Award. He is one of the current chief editors of Quality & Reliability Engineering International.