Active Learning for Deep Gaussian Process Surrogates

References

• Ba, S., and Joseph, V. R. (2012), “Composite Gaussian Process Models for Emulating Expensive Functions,” The Annals of Applied Statistics, 1838–1860. DOI: 10.1214/12-AOAS570.
   Web of Science ®Google Scholar
• Barnett, S. (1979), Matrix Methods for Engineers and Scientists, McGraw-Hill.
   Google Scholar
• Berger, J. O., De Oliveira, V., and Sansó, B. (2001). “Objective Bayesian Analysis of Spatially Correlated Data,” Journal of the American Statistical Association, 96, 1361–1374. DOI: 10.1198/016214501753382282.
   Web of Science ®Google Scholar
• Binois, M., Huang, J., Gramacy, R. B., and Ludkovski, M. (2019), “Replication or Exploration? Sequential Design for Stochastic Simulation Experiments,” Technometrics, 61, 1, 7–23. DOI: 10.1080/00401706.2018.1469433.
   Web of Science ®Google Scholar
• Bornn, L., Shaddick, G., and Zidek, J. V. (2012), “Modeling Nonstationary Processes Through Dimension Expansion,” Journal of the American Statistical Association, 107, 281–289. DOI: 10.1080/01621459.2011.646919.
   Web of Science ®Google Scholar
• Bui, T., Hernández-Lobato, D., Hernandez-Lobato, J., Li, Y., and Turner, R. (2016), “Deep Gaussian Processes for Regression Using Approximate Expectation Propagation,” in International Conference on Machine Learning, 1472–1481. PMLR.
   Google Scholar
• Cohn, D. (1994), “Neural Network Exploration Using Optimal Experiment Design,” in Advances in Neural Information Processing Systems (Vol. 6), 679–686, Morgan-Kaurmann.
   Google Scholar
• Cole, D. A., Christianson, R. B., and Gramacy, R. B. (2021), “Locally Induced Gaussian Processes for Large-Scale Simulation Experiments,” Statistics and Computing, 31, 3, 1–21. DOI: 10.1007/s11222-021-10007-9.
   Web of Science ®Google Scholar
• Cutajar, K., Pullin, M., Damianou, A., Lawrence, N., and González, J. (2019), “Deep Gaussian Processes for Multi-Fidelity Modeling,” arXiv:1903.07320.
   Google Scholar
• Damianou, A., and Lawrence, N. D. (2013), “Deep Gaussian Processes,” in Artificial Intelligence and Statistics, PMLR, pp. 207–215.
   Google Scholar
• Deissenberg, C., Van Der Hoog, S., and Dawid, H. (2009), “EURACE: A Massively Parallel Agent-Based Model of the European Economy,” Applied Mathematics and Computation, 204, 2, 541–552. DOI: 10.1016/j.amc.2008.05.116.
   Web of Science ®Google Scholar
• Dunlop, M. M., Girolami, M. A., Stuart, A. M., and Teckentrup, A. L. (2018), “How Deep Are Deep Gaussian Processes?” Journal of Machine Learning Research, 19, 1–46.
   Web of Science ®Google Scholar
• Dutordoir, V., Knudde, N., van der Herten, J., Couckuyt, I., and Dhaene, T. (2017), “Deep Gaussian Process Metamodeling of Sequentially Sampled Non-Stationary Response Surfaces,” in 2017 Winter Simulation Conference (WSC), IEEE, pp. 1728–1739. DOI: 10.1109/WSC.2017.8247911.
   Google Scholar
• Duvenaud, D., Rippel, O., Adams, R., and Ghahramani, Z. (2014), “Avoiding Pathologies in Very Deep Networks,” in Artificial Intelligence and Statistics, PMLR, pp. 202–210.
   Google Scholar
• Fadikar, A., Higdon, D., Chen, J., Lewis, B., Venkatramanan, S., and Marathe, M. (2018), “Calibrating a Stochastic, Agent-based Model Using Quantile-Based Emulation,” SIAM/ASA Journal on Uncertainty Quantification, 6, 1685–1706. DOI: 10.1137/17M1161233.
   Web of Science ®Google Scholar
• Fei, J., Zhao, J., Sun, S., and Liu, Y. (2018), “Active Learning Methods With Deep Gaussian Processes.” in International Conference on Neural Information Processing, 473–483. Springer, pp. 473–483.
   Google Scholar
• Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. (2013), Bayesian Data Analysis, New York: CRC Press.
   Google Scholar
• Gneiting, T. and Raftery, A. E. (2007). “Strictly Proper Scoring Rules, Prediction, and Estimation,” Journal of the American Statistical Association, 102, 359–378. DOI: 10.1198/016214506000001437.
   Web of Science ®Google Scholar
• Gramacy, R. B. (2007), “tgp: An R Package for Bayesian Nonstationary, Semiparametric Nonlinear Regression and Design by Treed Gaussian Process Models,” Journal of Statistical Software, 19, 1–46. DOI: 10.18637/jss.v019.i09.
   PubMedGoogle Scholar
• Gramacy, R. B. (2016), “laGP: Large-Scale Spatial Modeling via Local Approximate Gaussian Processes in R,” Journal of Statistical Software, 72, 1, 1–46.
   Web of Science ®Google Scholar
• Gramacy, R. B. (2020), Surrogates: Gaussian Process Modeling, Design and Optimization for the Applied Sciences, Boca Raton, FL: Chapman Hall/CRC.
   Google Scholar
• Gramacy, R. B., and Apley, D. W. (2015), “Local Gaussian Process Approximation for Large Computer Experiments,” Journal of Computational and Graphical Statistics, 24, 561–578. DOI: 10.1080/10618600.2014.914442.
   Web of Science ®Google Scholar
• Gramacy, R. B., and Lee, H. K. H. (2008), “Bayesian Treed Gaussian Process Models With an Application to Computer Modeling,” Journal of the American Statistical Association, 103, 1119–1130. DOI: 10.1198/016214508000000689.
   Web of Science ®Google Scholar
• Gramacy, R. B., and Lee, H. K. H. (2009), “Adaptive Design and Analysis of Supercomputer Experiments,” Technometrics, 51, 130–145.
   Web of Science ®Google Scholar
• Havasi, M., Hernández-Lobato, J. M., and Murillo-Fuentes, J. J. (2018), “Inference in Deep Gaussian Processes Using Stochastic Gradient Hamiltonian Monte Carlo,” in Advances in Neural Information Processing Systems, pp. 7506–7516.
   Google Scholar
• Hebbal, A., Brevault, L., Balesdent, M., Talbi, E.-G., and Melab, N. (2021). “Bayesian Optimization Using Deep Gaussian Processes With Applications to Aerospace System Design,” Optimization and Engineering, 22, 321–361. DOI: 10.1007/s11081-020-09517-8.
   Web of Science ®Google Scholar
• Higdon, D., Kennedy, M., Cavendish, J. C., Cafeo, J. A., and Ryne, R. D. (2004), “Combining Field Data and Computer Simulations for Calibration and Prediction,” SIAM Journal on Scientific Computing, 26, 448–466. DOI: 10.1137/S1064827503426693.
   Web of Science ®Google Scholar
• Higdon, D., Swall, J., and Kern, J. (1999), “Non-Stationary Spatial Modeling,” Bayesian Statistics, 6, 761–768.
   Google Scholar
• Higdon, D. M., Lee, H., and Holloman, C. (2003), “Markov chain Monte Carlo-Based Approaches for Inference in Computationally Intensive Inverse Problems” (with discussion), in Bayesian Statistics 7. Proceedings of the Seventh Valencia International Meeting, eds. J. M. Bernardo, M. J. Bayarri, J. O. Berger, A. P. Dawid, D. Heckerman, A. F. M. Smith, and M. West, Oxford: Oxford University Press, pp. 181–197.
   Google Scholar
• Johnson, M. E., Moore, L. M., and Ylvisaker, D. (1990), “Minimax and Maximin Distance Designs,” Journal of Statistical Planning and Inference, 26, 131–148. DOI: 10.1016/0378-3758(90)90122-B.
   Web of Science ®Google Scholar
• Jones, D. R., Schonlau, M., and Welch, W. J. (1998), “Efficient Global Optimization of Expensive Black-Box Functions,” Journal of Global Optimization, 13, 455–492. DOI: 10.1023/A:1008306431147.
   Web of Science ®Google Scholar
• Joseph, V. R., Gu, L., Ba, S., and Myers, W. R. (2019), “Space-Filling Designs for Robustness Experiments,” Technometrics, 61, 24–37. DOI: 10.1080/00401706.2018.1451390.
   Web of Science ®Google Scholar
• Kass, R. E., Carlin, B. P., Gelman, A., and Neal, R. M. (1998). “Markov Chain Monte Carlo in Practice: A Roundtable Discussion,” The American Statistician, 52, 93–100.
   Web of Science ®Google Scholar
• Katzfuss, M. (2013), “Bayesian Nonstationary Spatial Modeling for Very Large Datasets,” Environmetrics, 24, 189–200. DOI: 10.1002/env.2200.
   Web of Science ®Google Scholar
• Kennedy, M. C., and O’Hagan, A. (2001). “Bayesian Calibration of Computer Models,” Journal of the Royal Statistical Society, Series B, 63, 425–464. DOI: 10.1111/1467-9868.00294.
   Web of Science ®Google Scholar
• MacKay, D. J. (1992). “Information-Based Objective Functions for Active Data Selection,” Neural Computation, 4, 590–604. DOI: 10.1162/neco.1992.4.4.590.
   Web of Science ®Google Scholar
• Marmin, S., and Filippone, M. (2018), “Variational Calibration of Computer Models,” arXiv:1810.12177.
   Google Scholar
• Marrel, A., Iooss, B., Laurent, B., and Roustant, O. (2009), “Calculations of Sobol Indices for the Gaussian Process Metamodel,” Reliability Engineering & System Safety, 94, 742–751.
   Web of Science ®Google Scholar
• McKay, M. D., Beckman, R. J., and Conover, W. J. (2000), “A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code,” Technometrics, 42, 55– 61. DOI: 10.1080/00401706.2000.10485979.
   Web of Science ®Google Scholar
• Mehta, P. M., Walker, A., Lawrence, E., Linares, R., Higdon, D., and Koller, J. (2014), “Modeling Satellite Drag Coefficients With Response Surfaces,” Advances in Space Research, 54, 1590–1607. DOI: 10.1016/j.asr.2014.06.033.
   Web of Science ®Google Scholar
• Microsoft and Weston, S. (2020), foreach: Provides Foreach Looping Construct, R package version 1.5.0.
   Google Scholar
• Morris, M. D., and Mitchell, T. J. (1995), “Exploratory Designs for Computational Experiments,” Journal of Statistical Planning and Inference, 43, 381–402. DOI: 10.1016/0378-3758(94)00035-T.
   Web of Science ®Google Scholar
• Murray, I., Adams, R. P., and MacKay, D. J. C. (2010), “Elliptical slice sampling.” In The Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, Vol. 9 of JMLR: W&CP. PMLR, pp. 541–548.
   Google Scholar
• Neal, R. M. (2011). “MCMC Using Hamiltonian Dynamics,” in Handbook of Markov Chain Monte Carlo (Vol. 2), p. 2.
   Google Scholar
• Oakley, J. E., and O’Hagan, A. (2004), “Probabilistic Sensitivity Analysis of Complex Models: A Bayesian Approach,” Journal of the Royal Statistical Society, Series B, 66, 751–769. DOI: 10.1111/j.1467-9868.2004.05304.x.
   Web of Science ®Google Scholar
• Oliver, D. S., Cunha, L. B., and Reynolds, A. C. (1997), “Markov Chain Monte Carlo Methods for Conditioning a Permeability Field to Pressure Data,” Mathematical Geology, 29, 61–91. DOI: 10.1007/BF02769620.
   Google Scholar
• Paciorek, C. J., and Schervish, M. J. (2003). “Nonstationary Covariance Functions for Gaussian Process Regression,” in Proceedings of the 16th International Conference on Neural Information Processing Systems (NIPS’03), Cambridge, MA: MIT Press, pp. 273–280.
   Google Scholar
• Picheny, V., Gramacy, R., Wild, S., and Le Digabel, S. (2016), “Bayesian Optimization Under Mixed Constraints With a Slack-Variable Augmented Lagrangian,” in Advances in Neural Information Processing Systems, pp. 1435–1443.
   Google Scholar
• Radaideh, M. I., and Kozlowski, T. (2020), “Surrogate Modeling of Advanced Computer Simulations Using Deep Gaussian Processes,” Reliability Engineering & System Safety, 195, 106731.
   Web of Science ®Google Scholar
• Rajaram, D., Puranik, T. G., Ashwin Renganathan, S., Sung, W., Fischer, O. P., Mavris, D. N., and Ramamurthy, A. (2021), “Empirical Assessment of Deep Gaussian Process Surrogate Models for Engineering Problems,” Journal of Aircraft, 58, 1, 182–196. DOI: 10.2514/1.C036026.
   Web of Science ®Google Scholar
• Rasmussen, C. E. (2000), “The Infinite Gaussian Mixture Model,” in Advances in Neural Information Processing Systems (vol. 12). MIT Press, pp. 554–560.
   Google Scholar
• Rasmussen, C. E., and Ghahramani, Z. (2002), “Infinite Mixtures of Gaussian Process Experts,” in Advances in Neural Information Processing Systems (vol. 2), Cambridge, MA: MIT, pp. 881–888.
   Google Scholar
• Rasmussen, C. E., and Williams, C. K. I. (2005), Gaussian Processes for Machine Learning, Cambridge, MA: MIT Press.
   Google Scholar
• Salimbeni, H., and Deisenroth, M. (2017), “Doubly Stochastic Variational Inference for Deep Gaussian Processes,” arXiv:1705.08933.
   Google Scholar
• Saltelli, A. (2002). “Making Best Use of Model Evaluations to Compute Sensitivity Indices,” Computer Physics Communications, 145, 280–297. DOI: 10.1016/S0010-4655(02)00280-1.
   Web of Science ®Google Scholar
• Sampson, P. D., and Guttorp, P. (1992), “Nonparametric Estimation of Nonstationary Spatial Covariance Structure,” Journal of the American Statistical Association, 87, 108–119. DOI: 10.1080/01621459.1992.10475181.
   Web of Science ®Google Scholar
• Santner, T., Williams, B., and Notz, W. (2018), The Design and Analysis of Computer Experiments (2nd ed.), New York: Springer–Verlag.
   Google Scholar
• Sauer, A. (2020), deepgp: Sequential Design for Deep Gaussian Processes Using MCMC, R package version 0.1.0.
   Google Scholar
• Schmidt, A. M., and O’Hagan, A. (2003). “Bayesian Inference for Non-Stationary Spatial Covariance Structure Via Spatial Deformations,” Journal of the Royal Statistical Society, Series B, 65, 743–758. DOI: 10.1111/1467-9868.00413.
   Google Scholar
• Sejdinovic, D., Strathmann, H., Garcia, M. L., Andrieu, C., and Gretton, A. (2014). “Kernel Adaptive Metropolis–Hastings,” in International Conference on Machine Learning, pp. 1665–1673. PMLR.
   Google Scholar
• Seo, S., Wallat, M., Graepel, T., and Obermayer, K. (2000), “Gaussian Process Regression: Active Data Selection and Test Point Rejection,” in Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks, vol. 3. IEEE, pp. 241–246.
   Google Scholar
• Shewry, M. C. and Wynn, H. P. (1987), “Maximum Entropy Sampling,” Journal of Applied Statistics, 14, 165–170. DOI: 10.1080/02664768700000020.
   Google Scholar
• Snoek, J., Larochelle, H., and Adams, R. P. (2012), “Practical Bayesian Optimization of Machine Learning Algorithms,” in Advances in Neural Information Processing Systems (vol. 25). Curran Associates, Inc., pp. 2951–2959.
   Google Scholar
• Stein, M. L. (1999), Interpolation of Spatial Data, Springer-Verlag.
   Google Scholar
• Sun, F., Gramacy, R. B., Haaland, B., Lawrence, E., and Walker, A. (2019), “Emulating Satellite Drag From Large Simulation Experiments,” SIAM/ASA Journal on Uncertainty Quantification, 7, 720–759. DOI: 10.1137/18M1170157.
   Web of Science ®Google Scholar
• Wang, Y., Brubaker, M., Chaib-Draa, B., and Urtasun, R. (2016). “Sequential Inference for Deep Gaussian Process,” in Artificial Intelligence and Statistics, PMLR, pp. 694–703.
   Google Scholar
• Yang, J. and Klabjan, D. (2020), “Bayesian Active Learning for Choice Models With Deep Gaussian Processes,” IEEE Transactions on Intelligent Transportation Systems, 22, 1080–1092. DOI: 10.1109/TITS.2019.2962535.
   Web of Science ®Google Scholar
• Zhang, B., Cole, D. A., and Gramacy, R. B. (2021), “Distance-Distributed Design for Gaussian Process Surrogates,” Technometrics, 63, 40–52. DOI: 10.1080/00401706.2019.1677269.
   Web of Science ®Google Scholar