A Hierarchical Expected Improvement Method for Bayesian Optimization

References

• Benassi, R., Bect, J., and Vazquez, E. (2011), “Robust Gaussian Process-based Global Optimization Using a Fully Bayesian Expected Improvement Criterion,” in International Conference on Learning and Intelligent Optimization, pp. 176–190, Springer.
   Google Scholar
• Brunner, T. A., Menon, V. C., Wong, C. W., Gluschenkov, O., Belyansky, M. P., Felix, N. M., Ausschnitt, C. P., Vukkadala, P., Veeraraghavan, S., and Sinha, J. K. (2013), “Characterization of Wafer Geometry and Overlay Error on Silicon Wafers with Nonuniform Stress,” Journal of Micro/Nanolithography, MEMS, and MOEMS, 12, 1–13–13. DOI: 10.1117/1.JMM.12.4.043002.
   Web of Science ®Google Scholar
• Bull, A. D. (2011), “Convergence Rates of Efficient Global Optimization Algorithms,” Journal of Machine Learning Research, 12, 2879–2904.
   Web of Science ®Google Scholar
• Carlin, B. P., and Louis, T. A. (2000) Bayes and Empirical Bayes Methods for Data Analysis (Vol. 88), Boca Raton: Chapman & Hall/CRC.
   Google Scholar
• Castillo, I., Schmidt-Hieber, J., and Van der Vaart, A. (2015), “Bayesian Linear Regression with Sparse Priors,” The Annals of Statistics, 43, 1986–2018. DOI: 10.1214/15-AOS1334.
   Web of Science ®Google Scholar
• Chen, J., Mak, S., Joseph, V. R., and Zhang, C. (2020), “Function-on-Function Kriging, with Applications to Three-Dimensional Printing of Aortic Tissues,” Technometrics, 63, 384–395. DOI: 10.1080/00401706.2020.1801255.
   Web of Science ®Google Scholar
• Chen, R.-B., Wang, W., and Wu, C.-F. J. (2017), “Sequential Designs based on Bayesian Uncertainty Quantification in Sparse Representation Surrogate Modeling,” Technometrics, 59, 139–152. DOI: 10.1080/00401706.2016.1172027.
   Web of Science ®Google Scholar
• COMSOL. (2018). COMSOL Multiphysics [textregistered] v. 5.4.
   Google Scholar
• Cressie, N. (1991). Statistics for Spatial Data, New York: Wiley.
   Google Scholar
• Dasgupta, T., Ma, C., Joseph, V. R., Wang, Z., and Wu, C. F. J. (2008), “Statistical Modeling and Analysis for Robust Synthesis of Nanostructures,” Journal of the American Statistical Association, 103, 594–603. DOI: 10.1198/016214507000000905.
   Web of Science ®Google Scholar
• Deville, Y., Ginsbourger, D., and Roustant, O. (2019), “Kergp: Gaussian Process Laboratory,” https://cran.r-project.org/web/packages/kergp.
   Google Scholar
• Doucet, A., Godsill, S. J., and Robert, C. P. (2002), “Marginal Maximum a Posteriori Estimation using Markov Chain Monte Carlo,” Statistics and Computing, 12, 77–84.
   Web of Science ®Google Scholar
• Feliot, P., Bect, J., and Vazquez, E. (2017), “A Bayesian Approach to Constrained Single-and Multi-Objective Optimization,” Journal of Global Optimization, 67, 97–133. DOI: 10.1007/s10898-016-0427-3.
   Web of Science ®Google Scholar
• Frazier, P. I., Powell, W. B., and Dayanik, S. (2008), “A Knowledge-Gradient Policy for Sequential Information Collection,” SIAM Journal on Control and Optimization, 47, 2410–2439. DOI: 10.1137/070693424.
   Web of Science ®Google Scholar
• Gelman, A. (2006), “Prior Distributions for Variance Parameters in Hierarchical Models,” Bayesian Analysis, 1, 515–534. DOI: 10.1214/06-BA117A.
   Web of Science ®Google Scholar
• Handcock, M. S., and Stein, M. L. (1993), “A Bayesian Analysis of Kriging,” Technometrics, 35, 403–410. DOI: 10.1080/00401706.1993.10485354.
   Web of Science ®Google Scholar
• Jin, R., Chang, C.-J., and Shi, J. (2012), “Sequential Measurement Strategy for Wafer Geometric Profile Estimation,” IIE Transactions, 44, 1–12. DOI: 10.1080/0740817X.2011.557030.
   Web of Science ®Google Scholar
• Jones, D. R., Schonlau, M., and Welch, W. J. (1998), “Efficient Global Optimization of Expensive Black-Box Functions,” Journal of Global Optimization, 13, 455–492. DOI: 10.1023/A:1008306431147.
   Web of Science ®Google Scholar
• Joseph, V. R., Gul, E., and Ba, S. (2019), “Designing Computer Experiments with Multiple Types of Factors: The MaxPro Approach,” Journal of Quality Technology, 52, 343–354. DOI: 10.1080/00224065.2019.1611351.
   Web of Science ®Google Scholar
• Kearns, M., and Singh, S. (2002), “Near-Optimal Reinforcement Learning in Polynomial Time,” Machine Learning, 49, 209–232. DOI: 10.1023/A:1017984413808.
   Web of Science ®Google Scholar
• Lekivetz, R., and Jones, B. (2015), “Fast Flexible Space-Filling Designs for Nonrectangular Regions,” Quality and Reliability Engineering International, 31, 829–837. DOI: 10.1002/qre.1640.
   Web of Science ®Google Scholar
• Liu, Q., and Ihler, A. (2013), “Variational Algorithms for Marginal MAP,” Journal of Machine Learning Research, 14, 3165–3200.
   Web of Science ®Google Scholar
• Loeppky, J. L., Sacks, J., and Welch, W. J. (2009), “Choosing the Sample Size of a Computer Experiment: A Practical Guide,” Technometrics, 51, 366–376. DOI: 10.1198/TECH.2009.08040.
   Web of Science ®Google Scholar
• Mak, S., and Joseph, V. R. (2018), “Minimax and Minimax Projection Designs Using Clustering,” Journal of Computational and Graphical Statistics, 27, 166–178. DOI: 10.1080/10618600.2017.1302881.
   Web of Science ®Google Scholar
• Mak, S., Sung, C.-L., Wang, X., Yeh, S.-T., Chang, Y.-H., Joseph, V. R., Yang, V., and Wu, C.-F. J. (2018), “An Efficient Surrogate Model for Emulation and Physics Extraction of Large Eddy Simulations,” Journal of the American Statistical Association, 113, 1443–1456. DOI: 10.1080/01621459.2017.1409123.
   Web of Science ®Google Scholar
• Mak, S., and Wu, C.-F. J. (2019), “Analysis-of-Marginal-Tail-Means (ATM): A Robust Method for Discrete Black-Box Optimization,” Technometrics, 61, 545–559. DOI: 10.1080/00401706.2019.1593246.
   Web of Science ®Google Scholar
• Marmin, S., Chevalier, C., and Ginsbourger, D. (2015), “Differentiating the Multipoint Expected Improvement for Optimal Batch Design,” in International Workshop on Machine Learning, Optimization and Big Data, pp. 37–48, Springer.
   Google Scholar
• Mockus, J., Tiesis, V., and Zilinskas, A. (1978), “The Application of Bayesian Methods for Seeking the Extremum,” Towards Global Optimization, 2, 117–129.
   Google Scholar
• Morris, M. D., and Mitchell, T. J. (1995), “Exploratory Designs for Computational Experiments,” Journal of Statistical Planning and Inference, 43, 381–402. DOI: 10.1016/0378-3758(94)00035-T.
   Web of Science ®Google Scholar
• Niederreiter, H. (1992), Random Number Generation and Quasi-Monte Carlo Methods, Philadelphia, PA: SIAM.
   Google Scholar
• Olea, R. A. (2012). Geostatistics for Engineers and Earth Scientists, New York: Springer.
   Google Scholar
• Qin, C., Klabjan, D., and Russo, D. (2017), “Improving the Expected Improvement Algorithm,” in Advances in Neural Information Processing Systems, pp. 5381–5391.
   Google Scholar
• Santner, T. J., Williams, B. J., Notz, W., and Williams, B. J. (2018), The Design and Analysis of Computer Experiments (2nd ed.) New York: Springer.
   Google Scholar
• Schwarz, G. (1978), “Estimating the Dimension of a Model,” The Annals of Statistics, 6, 461–464. DOI: 10.1214/aos/1176344136.
   Web of Science ®Google Scholar
• Scott, W., Frazier, P., and Powell, W. (2011), “The Correlated Knowledge Gradient for Simulation Optimization of Continuous Parameters Using Gaussian Process Regression,” SIAM Journal on Optimization, 21, 996–1026. DOI: 10.1137/100801275.
   Web of Science ®Google Scholar
• Seshadri, P., Yuchi, S., and Parks, G. T. (2019), “Dimension Reduction via Gaussian Ridge Functions,” SIAM/ASA Journal on Uncertainty Quantification, 7, 1301–1322. DOI: 10.1137/18M1168571.
   Web of Science ®Google Scholar
• Singh, R., Fakhruddin, M., and Poole, K. (2000), “Rapid Photothermal Processing as a Semiconductor Manufacturing Technology for the 21st Century,” Applied Surface Science, 168, 198–203. DOI: 10.1016/S0169-4332(00)00590-0.
   Web of Science ®Google Scholar
• Snoek, J., Larochelle, H., and Adams, R. P. (2012), “Practical Bayesian Optimization of Machine Learning Algorithms,” in Advances in Neural Information Processing Systems, pp. 2951–2959.
   Google Scholar
• Srinivas, N., Krause, A., Kakade, S., and Seeger, M. (2010), “Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design,” in Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, USA, pp. 1015–1022. Omnipress.
   Google Scholar
• Surjanovic, S., and Bingham, D. (2015), “Virtual Library of Simulation Experiments: Test Functions and Datasets,” Retrieved August 16, 2019, from http://www.sfu.ca/∼ssurjano.
   Google Scholar
• Sutton, R. S., and Barto, A. G. (2018), Reinforcement Learning: An Introduction, Cambridge, MA: MIT Press.
   Google Scholar
• Wackernagel, H. (1995), Multivariate Geostatistics: An Introduction with Applications, Berlin: Springer.
   Google Scholar
• Wendland, H. (2004), Scattered Data Approximation. Cambridge Monographs on Applied and Computational Mathematics (Vol. 17), Cambridge: Cambridge University Press.
   Google Scholar
• Wynne, G., Briol, F., and Girolami M. (2020), “Convergence Guarantees for Gaussian Process Means with Misspecified Likelihoods and Smoothness,” arXiv preprint arXiv:2001.10818.
   Google Scholar
• Xiu, D. (2010), Numerical Methods for Stochastic Computations: A Spectral Method Approach, Princeton: Princeton University Press.
   Google Scholar
• Zhang, R., Mak, S., and Dunson, D. (2022), “Gaussian Process Subspace Prediction for Model Reduction,” SIAM Journal on Scientific Computing, 44, A1428–A1449. DOI: 10.1137/21M1432739.
   Web of Science ®Google Scholar
• Zhang, Y., Han, Z.-H., and Zhang, K.-S. (2018), “Variable-Fidelity Expected Improvement Method for Efficient Global Optimization of Expensive Functions. Structural and Multidisciplinary Optimization, 58, 1431–1451. DOI: 10.1007/s00158-018-1971-x.
   Web of Science ®Google Scholar