Case study on applying sequential analyses in operational testing

References

• Ahner, D. K., and C. R. Parson. 2016. Workshop report: Test and evaluation of autonomous systems. https://www.afit.edu/stat/statcoe_files/TE/%20Auto/%20Syst/%20Workshop/%20DASDDTE/%20Memo.pdf.
   Google Scholar
• Avery, M. R., J. R. Simpson, and H. M. Wojton. 2021. Determining how much testing is enough: An exploration of progress in the department of defense test and evaluation community. The ITEA Journal of Test and Evaluation 42 (1):39–47.
   Google Scholar
• Box, G. E. 1999. Statistics as a catalyst to learning by scientific method part II–a discussion. Journal of Quality Technology 31 (1):16–29. doi:10.1080/00224065.1999.11979890.
   Web of Science ®Google Scholar
• Box, G. E. P., and K. B. Wilson. 1951. On the experimental attainment of optimum conditions. Journal of the Royal Statistical Society: Series B (Methodological) 13 (1):1–38. doi:10.1111/j.2517-6161.1951.tb00067.x.
   Web of Science ®Google Scholar
• Box, G. E., and P. Y. Liu. 1999. Statistics as a catalyst to learning by scientific method part I–an example. Journal of Quality Technology 31 (1):1–15. doi:10.1080/00224065.1999.11979889.
   Web of Science ®Google Scholar
• Chernoff, H. 1959. Sequential design of experiments. The Annals of Mathematical Statistics 30 (3):755–70. doi:10.1214/aoms/1177706205.
   Google Scholar
• Freeman, L. 2020. Test and evaluation for artificial intelligence. Insight 23 (1):27–30. doi:10.1002/inst.12281.
   Google Scholar
• Freeman, L. J., T. Johnson, M. Avery, V. B. Lillard, and J. Clutter. 2018. Testing defense systems. In Analytic methods in systems and software testing eds. R. S. Kenett, F. Ruggeri and F. W. Faltin. Wiley. doi:10.1002/9781119357056.ch18.
   Google Scholar
• Ghosh, B. 2014. Sequential analysis-historical. In Wiley StatsRef: Statistics Reference Online, eds. N. Balakrishnan, T. Colton, B. Everitt, W. Piegorsch, F. Ruggeri and J. L. Teugels. Wiley. doi:10.1002/9781118445112.stat02062.
   Google Scholar
• Govindarajulu, Z. 1975. Sequential statistical procedures. New York, NY: Academic Press.
   Google Scholar
• Hansen, P. R. 2005. A test for superior predictive ability. Journal of Business & Economic Statistics 23 (4):365–80. doi:10.1198/073500105000000063.
   Web of Science ®Google Scholar
• Johnson, N. L. 1961. Sequential analysis: A survey. Journal of the Royal Statistical Society: Series A (General) 124 (3):372–411. doi:10.2307/2343243.
   Google Scholar
• Johnson, T. H., L. Freeman, J. Hester, and J. L. Bell. 2014. A comparison of ballistic resistance testing techniques in the department of defense. IEEE Access 2:1442–55. doi:10.1109/ACCESS.2014.2377633.
   Web of Science ®Google Scholar
• Lai, T. L. 2001. Sequential analysis: Some classical problems and new challenges with rejoinder. Statistica Sinica 11:303–408.
   Web of Science ®Google Scholar
• Lovell, M. C. 1983. Data mining. The Review of Economics and Statistics 65 (1):1–12. doi:10.2307/1924403.
   Web of Science ®Google Scholar
• Medlin, R., J. Dennis, K. Pagán-Rivera, and L. Wilkins. 2021. A review of sequential analysis. Institute for Defense Analyses D-20487. https://idalink.org/D-20487.
   Google Scholar
• MIL-HDBK-781A. 1996. Reliability test methods; plans, and environments for engineering development, qualification, and production. Washington, DC.
   Google Scholar
• Montgomery, D. C. 2020. Design and analysis of experiments. 10th ed. Hoboken, NJ: John Wiley & Sons.
   Google Scholar
• Myers, R. H., D. C. Montgomery, and C. M. Anderson-Cook. 2016. Response surface methodology: Process and product optimization using designed experiments. Hoboken, NJ: John Wiley & Sons.
   Google Scholar
• National Research Council. 1998. Statistics, testing, and defense acquisition: New approaches and methodological improvements. Washington, DC: The National Academies Press.
   Google Scholar
• Porter, D. J., Y. K. Pinelis, C. M. Bieber, H. M. Wojton, M. O. McAnally, and L. J. Freeman. 2019. Operational testing of systems with autonomy. Institute for Defense Analyses. http://www.jstor.org/stable/resrep22754.
   Google Scholar
• Robbins, H. 1952. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society 58 (5):527–35. doi:10.1090/S0002-9904-1952-09620-8.
   Web of Science ®Google Scholar
• Romano, J. R., and M. Wolf. 2005. Stepwise multiple testing as formalized data snooping. Econometrica 73 (4):1237–82. doi:10.1111/j.1468-0262.2005.00615.x.
   Web of Science ®Google Scholar
• Simpson, J. 2018. Testing via sequential experiments best practice and tutorial. https://www.afit.edu/stat/statcoe_files/Testing_via_Sequential_Experiments_Best_Practice.pdf.
   Google Scholar
• Wald, A. 1945. Sequential tests of statistical hypotheses. The Annals of Mathematical Statistics 16 (2):117–86. doi:10.1214/aoms/1177731118.
   Google Scholar
• Wallis, W. A. 1980. The statistical research group, 1942–1945. Journal of the American Statistical Association 75 (370):320–30.
   Web of Science ®Google Scholar
• White, H. 2000. A reality check for data snooping. Econometrica 68 (5):1097–126. doi:10.1111/1468-0262.00152.
   Web of Science ®Google Scholar