Profiles in the Teaching of Experimental Design and Analysis

Figures & data

Table 1 Categorized list of literature on DOE pedagogy.

Category	References
General DOE Pedagogy	Hunter (1977); Easterling (2004); Goos and Leemans (2004)
DOE Projects and Case Studies	Hunter (1977); Kenett and Steinberg (1987); Box (1992); Mackisack (1994); Anderson-Cook (1998); Nolan and Speed (1999); Anderson-Cook and Dorai-Raj (2001); Howley (2003); Binnie (2004); Steiner et al. (2007); Lawson et al. (2011); Dunn (2013); Kuiper (2016); Kuhnt and Coleman (2020, 2021); Pyott (2021); Woodard (2023)
DOE Simulators	Bulmer (2003, 2012); Schrevens et al. (2004); Darius, Portier, and Schrevens (2007); Muske and Myers (2007); Steiner and MacKay (2009); Bulmer and Haladyn (2011); De Ketelaere et al. (2014); Kuiper (2016); Reis and Kenett (2017); Gramacy (2020)
Teaching DOE to non-Statistics Students	Pollock, Ross-Parker, and Mead (1979); Deming and Morgan (1983); Fillebrown (1994); Antony and Capon (1998); Zolman (1999); Lin Ho, Lyn Nge, and Hong Chua (2004); Lye (2005); Hiebert (2007); Hane (2007); Stafford, Goodenough, and Davies (2010)

Table 2 Learning outcome ranking frequencies (n $=$ 45) ${}^{\mathrm{a}}$.

	Importance
	1st	2nd	3rd	4th
Understand basic design and analysis principles	39	3	2	1
Be able to use software to design and analyze experiments	1	29	13	2
Be able to perform experiments by actually collecting data	4	6	17	18
Understand the theoretical underpinnings of DOE	1	7	13	24

^a Note that 45 of the 50 survey participants responded to this question.

Fig. 1 Topics by course. Visualization of which topics covered (dark gray) versus not covered (light gray) in each course. Respondents indicated that courses 1–5 are Boxian, 6–23 are Fisherian, and 24–50 are Boxian/Fisherian. Rows are sorted from most to least prevalent.

Fig. 2 Teaching methods by course. Visualization of which teaching methods are used (dark gray) versus not used (light gray) in each course. Respondent 21’s nonresponse is indicated by white squares. Respondents indicated that courses 1–5 are Boxian, 6–23 are Fisherian, and 24–50 are Boxian/Fisherian. Rows are sorted from most to least prevalent.

Fig. 3 Assessment method by course. Visualization of which assessment methods are used (dark gray) versus not used (light gray) in each course. Respondents indicated that courses 1–5 are Boxian, 6–23 are Fisherian, and 24–50 are Boxian/Fisherian. Rows are sorted from most to least prevalent.

Table 3 Basic comparison of the four courses described by the authors.

Category	Smucker	Stevens	Asscher	Goos
Theme	“Designed so that Statistics Students Will Not be a Part of the Replication Crisis”	“Designed using Online Experimentation as a Source for Examples, Assignment Problems, and Project”	“Designed so that Engineering Students can use DOE in Industry and Research”	“Designed so Students are Equipped to Design Experiments in Both Standard and Unusual Settings”
Institution and Department	Miami University, Department of Statistics	University of Waterloo, Department of Statistics and Actuarial Science	Kinneret College on the Sea of Galilee, Quality and Reliability Engineering; also Technion - Israel Institute of Technology	KU Leuven, Faculty of Bioscience Engineering & Leuven Statistics Research Center
Audience	Advanced statistics undergraduates; first-year statistics master’s students	Advanced undergraduates, masters and PhD; statistics, data/ computer/ actuarial science, engineering	Quality and reliability engineering undergraduates; undergraduate and graduate engineering students	Bioscience engineering undergraduates; M.Sc. students in statistics and data science
Assumed Prerequisites	Calculus, Probability; Linear Regression with matrix algebra	Calculus, Linear Algebra, Statistics and Probability, Linear Regression	Introductory Probability and Statistics, Linear Regression and ANOVA	Linear Regression with matrix algebra
Course frequency	1–2 sections per year	2 sections per year	1 section per year	1 section per year
Typical class size	30	150	20	120
Software	SAS; JMP	R	JMP	JMP
Textbook	Montgomery; Oehlert; Dean, Voss, and Draguljić (all optional)	Montgomery; Wu and Hamada (both optional)	Montgomery (optional)	Goos & Jones (compulsory)

Hunter, W. G. (1977), “Some Ideas about Teaching Design of Experiments, with 25 Examples of Experiments Conducted by Students,” The American Statistician, 31, 12–17. https://www.tandfonline.com/doi/abs/10.1080/00031305.1977.10479185. DOI: 10.1080/00031305.1977.10479185.

 Web of Science ®Google Scholar

Easterling, R. G. (2004), “Teaching Experimental Design,” The American Statistician, 58, 244–252. https://www.tandfonline.com/doi/abs/10.1198/000313004X1477. DOI: 10.1198/000313004X1477.

 Web of Science ®Google Scholar

Goos, P., and Leemans, H. (2004), “Teaching Optimal Design of Experiments Using a Spreadsheet,” Journal of Statistics Education, 12(3). DOI: 10.1080/10691898.2004.11910631.

 Google Scholar

Kenett, R. S., and Steinberg, D. M. (1987), “Some Experiences Teaching Factorial Design in Introductory Statistics Courses,” Journal of Applied Statistics, 14, 219–227. https://www.tandfonline.com/doi/abs/10.1080/02664768700000027. DOI: 10.1080/02664768700000027.

 Google Scholar

Box, G. E. (1992), “Teaching Engineers Experimental Design with a Paper Helicopter,” Quality Engineering, 4, 453–459. https://www.tandfonline.com/doi/abs/10.1080/08982119208918925.

 Google Scholar

Mackisack, M. (1994), “What is the Use of Experiments Conducted by Statistics Students?” Journal of Statistics Education, 2, 1–15. https://www.tandfonline.com/doi/full/10.1080/10691898.1994.11910461.

 Google Scholar

Anderson-Cook, C. M. (1998), “Designing a First Experiment: A Project for Design of Experiment Courses,” The American Statistician, 52, 338–342. https://www.tandfonline.com/doi/abs/10.1080/00031305.1998.10480592. DOI: 10.1080/00031305.1998.10480592.

 Web of Science ®Google Scholar

Nolan, D., and Speed, T. P. (1999), “Teaching Statistics Theory through Applications,” The American Statistician, 53, 370–375. https://www.tandfonline.com/doi/abs/10.1080/00031305.1999.10474492. DOI: 10.1080/00031305.1999.10474492.

 Web of Science ®Google Scholar

Anderson-Cook, C. M., and Dorai-Raj, S. (2001), “An Active Learning in-Class Demonstration of Good Experimental Design,” Journal of Statistics Education, 9. DOI: 10.1080/10691898.2001.11910645.

 Google Scholar

Howley, P. P. (2003), “Teaching How to Calibrate a Process Using Experimental Design and Analysis: The Ballistat,” Journal of Statistics Education, 11. DOI: 10.1080/10691898.2003.11910709.

 Google Scholar

Binnie, N. (2004), “Using EDA, ANOVA and Regression to Optimise Some Microbiology Data,” Journal of Statistics Education, 12, 2. DOI: 10.1080/10691898.2004.11910738.

 Google Scholar

Steiner, S. H., Hamada, M., Giddings White, B. J., Kutsyy, V., Mosesova, S., and Salloum, G. (2007), “A Bubble Mixture Experiment Project for Use in an Advanced Design of Experiments Class,” Journal of Statistics Education, 15. DOI: 10.1080/10691898.2007.11889458.

 Google Scholar

Lawson, J., Aggarwal, P., Leininger, T., and Fairchild, K. (2011), “Characterizing Variability in Smestad and Grätzel’s Nanocrystalline Solar Cells: A Collaborative Learning Experience in Experimental Design,” Journal of Statistics Education, 19, https://www.tandfonline.com/doi/abs/10.1080/10691898.2011.11889598. DOI: 10.1080/10691898.2011.11889598.

 Google Scholar

Dunn, P. K. (2013), “Comparing the Lifetimes of Two Brands of Batteries,” Journal of Statistics Education, 21. https://www.tandfonline.com/doi/abs/10.1080/10691898.2013.11889666. DOI: 10.1080/10691898.2013.11889666.

 Google Scholar

Kuiper, S. (2016), “Stat2Labs Front Page,” Stat2Labs, available at https://stat2labs.sites.grinnell.edu/.

 Google Scholar

Kuhnt, S., and Coleman, S. (2020), “Hands-On Projects for Teaching DoE” [Conference Presentation], ENBIS-20 Online, available for ENBIS Members at https://enbis.org/media-centre/sonja-kuhnt-shirley-coleman-hands-on-projects-for-teaching-doe/.

 Google Scholar

Kuhnt, S., and Coleman, S. (2021), “Hands-On Projects for Teaching DoE” [Conference presentation], ENBIS-21 Online, available for ENBIS Members at https://enbis.org/media-centre/sonja-kuhnt-shirley-coleman-hands-on-projects-for-teaching-doe-2/.

 Google Scholar

Pyott, L. (2021), “Tennis Anyone? Teaching Experimental Design by Designing and Executing a Tennis Ball Experiment,” Journal of Statistics and Data Science Education, 29, 22–26. DOI: 10.1080/10691898.2020.1854638.

 Web of Science ®Google Scholar

Woodard, V. (2023), “Five Hands-on Experiments for a Design of Experiments Course,” Journal of Statistics and Data Science Education. DOI: 10.1080/26939169.2023.2195889.

 Web of Science ®Google Scholar

Bulmer, M. (2003), “Growing Virtual Plants for Teaching and Learning Statistics,” in Proceedings of the Apple University Consortium Academic and Developers Conference, pp. 3-1–3-7. Available at https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=f09b284b8cf5f5e9e73fad981425a9e5c769b77e.

 Google Scholar

Bulmer, M. (2012), “Virtual Worlds for Teaching Statistics,” International Journal of Innovation in Science and Mathematics Education, 11. https://openjournals.library.sydney.edu.au/CAL/article/view/6061.

 Google Scholar

Schrevens, E., Portier, K., Darius, P., Cooman, A., Tenorio, J., and Medina, A. (2004), “Simulation-Based Environments for Practicing Data-Collection Skills in Greenhouse Experimentation,” in International Conference on Sustainable Greenhouse,” Acta Horticulturae, pp. 871–876. https://www.actahort.org/books/691/691_108.htm. DOI: 10.17660/ActaHortic.2005.691.108.

 Google Scholar

Darius, P. L., Portier, K. M., and Schrevens, E. (2007), “Virtual Experiments and Their Use in Teaching Experimental Design,” International Statistical Review, 75, 281–294. https://onlinelibrary.wiley.com/doi/10.1111/j.1751-5823.2007.00028.x. DOI: 10.1111/j.1751-5823.2007.00028.x.

 Web of Science ®Google Scholar

Muske, K. R., and Myers, J. A. (2007), “A Realistic Experimental Design and Statistical Analysis Project,” Chemical Engineering Education, 41, 31–38. https://journals.flvc.org/cee/article/view/122472.

 Google Scholar

Steiner, S. H., and MacKay, R. J. (2009), “Teaching Variation Reduction Using a Virtual Manufacturing Environment,” The American Statistician, 63, 361–365. https://www.tandfonline.com/doi/abs/10.1198/tast.2009.08042. DOI: 10.1198/tast.2009.08042.

 Web of Science ®Google Scholar

Bulmer, M., and Haladyn, J. K. (2011), “Life on an Island: A Simulated Population to Support Student Projects in Statistics,” Technology Innovations in Statistics Education, 5. https://escholarship.org/uc/item/2q0740hv. DOI: 10.5070/T551000187.

 Google Scholar

De Ketelaere, B., Siebertz, K., van Bebber, D., and Rutten, K. (2014), “The Garden Sprinkler: An Interactive Web-Based Application for Teaching Design of Experiments,” ICOTS9 Proceedings, https://iase-web.org/icots/9/proceedings/pdfs/ICOTS9_C266_RUTTEN.pdf.

 Google Scholar

Reis, M., and Kenett, R. S. (2017), “A Structured Overview on the Use of Computational Simulators for Teaching Statistical Methods,” Quality Engineering, 29, 730–744. https://www.tandfonline.com/doi/abs/10.1080/08982112.2016.1272122. DOI: 10.1080/08982112.2016.1272122.

 Web of Science ®Google Scholar

Gramacy, R. B. (2020), “A Shiny Update to an Old Experiment Game,” The American Statistician, 74, 87–92. https://www.tandfonline.com/doi/abs/10.1080/00031305.2018.1505659. DOI: 10.1080/00031305.2018.1505659.

 Web of Science ®Google Scholar

Pollock, K. H., Ross-Parker, H. M., and Mead, R. (1979), “A Sequence of Games Useful in Teaching Experimental Design to Agriculture Students,” The American Statistician, 33, 70–76. https://www.tandfonline.com/doi/abs/10.1080/00031305.1979.10482663. DOI: 10.1080/00031305.1979.10482663.

 Web of Science ®Google Scholar

Deming, S. N., and Morgan, S. L. (1983), “Teaching the Fundamentals of Experimental Design,” Analytica Chimica Acta, 150, 183–198. https://www.sciencedirect.com/science/article/abs/pii/S0003267000854707. DOI: 10.1016/S0003-2670(00)85470-7.

 Web of Science ®Google Scholar

Fillebrown, S. (1994), “Using Projects in an Elementary Statistics Course for Non-Science Majors,” Journal of Statistics Education, 2. DOI: 10.1080/10691898.1994.11910470.

 Google Scholar

Antony, J., and Capon, N. (1998), “Teaching Experimental Design Techniques to Industrial Engineers,” International Journal of Engineering Education, 14, 335–343. https://enbis.org/wp-content/uploads/2018/09/3299_4928249026.pdf.

 Web of Science ®Google Scholar

Zolman, J. F. (1999), “Teaching Experimental Design to Biologists,” The American Journal of Physiology, 277, S111–S118. https://journals.physiology.org/doi/abs/10.1152/advances.1999.277.6.s111. DOI: 10.1152/advances.1999.277.6.S111.

 PubMedGoogle Scholar

Lin Ho, S., Lyn Nge, W., and Hong Chua, K. (2004), “The Catapult Project: An Innovative Approach for Learning Statistical Design of Experiments,” in 2004 IEEE International Engineering Management Conference (IEEE Cat. No. 04CH37574), 3, 1056–1060, https://ieeexplore.ieee.org/document/1408853.

 Google Scholar

Lye, L. M. (2005), “Tools and Toys for Teaching Design of Experiments Methodology,” in 33rd Annual General Conference of the Canadian Society for Civil Engineering.

 Google Scholar

Hiebert, S. M. (2007), “Teaching Simple Experimental Design to Undergraduates: Do Your Students Understand the Basics?” Advances in Physiology Education, 31, 82–92. https://journals.physiology.org/doi/full/10.1152/advan.00033.2006. DOI: 10.1152/advan.00033.2006.

 PubMed Web of Science ®Google Scholar

Hane, E. N. (2007), “Use of an Inquiry-Based Approach to Teaching Experimental Design Concepts in a General Ecology Course,” Teaching Issues and Experiments in Ecology, 5, 1–19. https://www.esa.org/tiee/vol/v5/research/hane/pdf/Hane_2007.pdf.

 Google Scholar

Stafford, R., Goodenough, A. E., and Davies, M. S. (2010), “Assessing the Effectiveness of a Computer Simulation for Teaching Ecological Experimental Design,” Bioscience Education, 15, 1–9. DOI: 10.3108/beej.15.1.

 Google Scholar