5,426
Views
46
CrossRef citations to date
0
Altmetric
Research Article

State-of-the-art stochastic data assimilation methods for high-dimensional non-Gaussian problems

, , , , , , & show all
Pages 1-43 | Received 05 Jul 2017, Accepted 19 Feb 2018, Published online: 21 Mar 2018

References

  • Ades, M. and van Leeuwen, P. J. 2013. An exploration of the equivalent weights particle filter. Q. J. R. Meteorol. Soc. 139, 820–840.
  • Anderson, J. 2003. A local least squares framework for ensemble filtering. Mon. Wea. Rev. 131, 634–642.
  • Anderson, J. L. 2001. An ensemble adjustment Kalman filter for data assimilation. Mon. Wea. Rev. 129, 2884–2903.
  • Anderson, J. L. 2007. Exploring the need for localization in ensemble data assimilation using a hierarchical ensemble filter. Physica D 230, 99–111.
  • Anderson, J. L. 2009. Spatially and temporally varying adaptive covariance inflation for ensemble filters. Tellus 61A, 72–83.
  • Anderson, J. L. 2010. A non-Gaussian ensemble filter update for data assimilation. Mon. Wea. Rev. 138(11), 4186–4198.
  • Anderson, J. L. 2012. Localization and sampling error correction in ensemble Kalman filter data assimilation. Mon. Wea. Rev. 140, 2359–2371.
  • Anderson, J. L. and Anderson, S. L. 1999. A Monte Carlo implementation of the non-linear filtering problem to produce ensemble assimilations and forecasts. Mon. Wea. Rev. 126, 2741–2758.
  • Bannister, R. N. 2017. A review of operational methods of variational and ensemble-variational data assimilation. Q. J. R. Meteorol. Soc. 143, 607–633.
  • Bengtsson, T., Snyder, C. and Nychka, D. 2003. Toward a nonlinear ensemble filter for high-dimensional systems. J. Geophys. Res. 108, 8775–8785.
  • Bengtsson, T., Bickel, P. and Li, B. 2008. Curse-of-dimensionality revisited: collapse of the particle filter in very large scale systems. IMS Collections: Prob. Stat. Essays Honor David A. Freedman 2, 316–334.
  • Bishop, C. H. and Hodyss, D. 2009. Ensemble covariances adaptively localized with ECO-RAP. Part 2: a strategy for the atmosphere. Tellus 61A, 97–111.
  • Bishop, C. H., Etherton, B. J. and Majumdar, S. J. 2001. Adaptive sampling with the ensemble transform Kalman filter. Part I: theoretical aspects. Mon. Wea. Rev. 129, 420–436.
  • Blockley, E. W., Martin, M. J., McLaren, A. J., Ryan, A. G., Waters, A., and co-authors. 2014. Recent development of the Met Office operational ocean forecasting system: an overview and assessment of the new global FOAM forecasts. Geosci. Mod. Dev. 7, 2613–2638.
  • Bocquet, M. 2011. Ensemble Kalman filtering without the intrinsic need for inflation. Nonl. Proc. Geophy. 18, 735–750.
  • Bocquet, M., Pires, C. A. and Wu, L. 2010. Beyond Gaussian statistical modelling in geophysical data assimilation. Mon. Wea. Rev. 138, 2997–3023.
  • Bolić, M., Djurić, P. M. and Hong, S. 2003. New resampling algorithms for particle filters. Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP ’03). 2003 IEEE International Conference, Vol. 2, IEEE, pp. II--589–592.
  • Brankart, J.-M., Testut, C.-E., Brasseur, P. and Verron, J. 2003. Implementation of a multivariate data assimilation scheme for isopycnic coordinate ocean models: application to a 1993–1996 hindcast of the North Atlantic ocean circulation. J. Geophys. Res. 108(C3), 3074.
  • Browne, P. A. and Wilson, S. 2015. A simple method for integrating a complex model into an ensemble data assimilation system using MPI. Env. Modell. Software 68, 122–128.
  • Brusdal, K., Brankart, J. M., Halberstadt, G., Evensen, G., Brasseur, P., and co-authors. 2003. A demonstration of ensemble based assimilation methods with a layered OGCM from the perspective of operational ocean forecasting systems. J. Mar. Syst. 40–41, 253–289.
  • Buehner, M. and Charron, M. 2007. Spectral and spatial localization of background-error correlations for data assimilation. Q. J. R. Meteorol. Soc. 133, 615–630.
  • Burgers, G., van Leeuwen, P. J. and Evensen, G. 1998. Analysis scheme in the ensemble Kalman filter. Mon. Wea. Rev. 126(6), 1719–1724.
  • Campbell, W. F., Bishop, C. H. and Hodyss, D. 2010. Vertical covariance localization for satellite radiances in ensemble Kalman filters. Mon. Wea. Rev. 138, 282–290.
  • Chorin, A. J. and Tu, X. 2009. Implicit sampling for particle filters. PNAS 106, 17249–17254.
  • Chorin, A. J., Morzfeld, M. and Tu, X. 2010. Interpolation and iteration for nonlinear filters. Commun. Appl. Math. Comput. Sci. 5, 221–240.
  • Chustagulprom, N., Reich, S. and Reinhardt, M. 2016. A hybrid ensemble transform filter for nonlinear and spatially extended dynamical systems. SIAM/ASA J. Uncert. Quant. 4, 552–591.
  • Cohn, S. E., Da Silva, A., Guo, J., Sienkiewicz, M. and Lamich, D. 1998. Assessing the effects of data selection with the DAO physical-space statistical analysis system. Mon. Wea. Rev. 126, 2913–2926.
  • Desroziers, G., Berre, L., Chapnik, B. and Poli, P. 2005. Diagnosis of observation, background and analysis-error statistics in observation space. Q. J. R. Meteorol. Soc. 131, 3385–3396.
  • Doucet, A., Godsill, S. and Andrieu, C. 2000. On sequential Monte Carlo sampling methods for Bayesian filtering. Stat. Comput. 10, 197–208.
  • Doucet, A., de Freitas, N., Gordon, N. 2001. Sequential Monte-Carlo Methods in Practice. Springer-Verlag, New York.
  • Evensen, G. 1993. Open boundary conditions for the extended Kalman filter with a quasi-geostrophic ocean model. J. Geophys. Res. 98(C9), 16529–16546.
  • Evensen, G. 1994. Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte-Carlo methods to forecast error statistics. J. Geophys. Res. 99, 10143–10162.
  • Evensen, G. 2003. The ensemble Kalman filter: theoretical formulation and practical implementation. Ocean Dyn. 53, 343–367.
  • Flowerdew, J. 2015. Towards a theory of optimal localisation. Tellus A 67, 25257.
  • Frei, M. and Künsch, H. R. 2013. Bridging the ensemble Kalman and particle filters. Biometrica 100, 781–800.
  • Gaspari, G. and Cohn, S. 1999. Construction of correlation functions in two and three dimensions. Q. J. R. Meteorol. Soc. 125, 723–757.
  • Golub, G. H. and Van Loan, C. F. 1996. Matrix computations, 3rd ed. The Johns Hopkins University Press, Baltimore and London.
  • Gordon, N. J., Salmond, D. J. and Smith, A. F. M. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. IEEE Proc. 140, 107–113.
  • Greybush, S. J., Kalnay, E., Miyoshi, T., Ide, K. and Hunt, B. R. 2011. Balance and ensemble Kalman filter localization techniques. Mon. Wea. Rev. 139, 511–522.
  • Gropp, W., Lusk, E. and Skjellum, A. 1994. Using MPI: Portable Parallel Programming with the Message-Passing Interface The MIT Press, Cambridge, Massachusetts.
  • Hamill, T. M. 2001. Interpretation of rank histograms for verifying ensemble forecasts. Mon. Wea. Rev. 129, 550–560.
  • Hamill, T. M. 2006. Ensemble-based atmospheric data assimilation. In: Predictability of Weather and Climate (eds. T. Palmer and R. Hagedorn). Cambridge University Press, New York, pp. 124–156. chapter 6.
  • Hamill, T. M., Whitaker, J. S. and Snyder, C. 2001. Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev. 129, 2776–1790.
  • Hoteit, I., Pham, D.-T., Triantafyllou, G. and Korres, G. 2008. A new approximate solution of the optimal nonlinear filter for data assimilation in meteorology and oceanography. Mon. Wea. Rev. 136, 317–334.
  • Houtekamer, P. L. and Mitchell, H. L. 1998. Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev. 126, 796–811.
  • Houtekamer, P. L. and Mitchell, H. L. 2001. A sequential ensemble Kalman filter for atmospheric data assimilation. Mon. Wea. Rev. 129, 123–137.
  • Houtekamer, P. L. and Zhang, F. 2016. Review of the ensemble Kalman filter for atmospheric data assimilation. Mon. Wea. Rev. 144, 4489–4532.
  • Hunt, B. R., Kostelich, E. J. and Szunyogh, I. 2007. Efficient data assimilation for spatiotemporal chaos: a local ensemble transform Kalman filter. Physica D 230, 112–126.
  • Ide, K., Courtier, P., Ghil, M. and Lorenc, A. C. 1997. Unified notation for data assimilation: pperational, l sequential and variational. J. Meteor. Soc. Jpn. 75, 181–189.
  • Janjić, T., Nerger, L., Albertella, A., Schroeter, J. and Skachko, S. 2011. On domain localization in ensemble-based Kalman filter algorithms. Mon. Wea. Rev. 139, 2046–2060.
  • Jazwinski, A. H. 1970. Stochastic Processes and Filtering Theory Academic Press, New York.
  • Kalman, R. E. 1960. A new approach to linear filtering and prediction problems. Trans. AMSE --J. Basic Eng. 82(D), 35–45.
  • Kepert, J. D. 2006. Localisation, balance and choice of analysis variable in an ensemble Kalman filter. Q. J. R. Meteorol. Soc. 135(642), 1157–1176.
  • Kepert, J. D. 2009. Covariance localisation and balance in an ensemble Kalman filter. Q. J. R. Meteorol. Soc. 135, 1157–1176.
  • Keppenne, C. L. and Rienecker, M. M. 2002. Initial testing of a massively parallel ensemble Kalman filter with the Poseidon isopycnal ocean circulation model. Mon. Wea. Rev. 130, 2951–2965.
  • Keppenne, C. L. and Rienecker, M. M. 2003. Assimilation of temperature into an isopycnal ocean general circulation model using a parallel ensemble Kalman filter. J. Mar. Syst. 40–41, 363–380.
  • Kirchgessner, P., Nerger, L. and Bunse-Gerstner, A. 2014. On the choice of an optimal localization radius in ensemble Kalman filter methods. Mon. Wea. Rev. 142, 2165–2175.
  • Law, K. J. H. and Stuart, A. M. 2012. Evaluating data assimilation algorithms. Mon. Wea. Rev. 140, 3757–3782.
  • Lawson, W. G. and Hansen, J. A. 2004. Implications of stochastic and deterministic filters as ensemble-based data assimilation methods in varying regimes of error growth. Mon. Wea. Rev. 132, 1966–1989.
  • Lei, J. and Bickel, P. 2011. A moment matching ensemble filter for nonlinear non-Gaussian data assimilation. Mon. Wea. Rev. 139, 3964–3973.
  • Lei, L. and Anderson, J. 2014. Empirical localization of observations for serial ensemble Kalman filter data assimilation in an atmospheric general circulation model. Mon. Wea. Rev. 142, 1835–1851.
  • Lermusiaux, P. F. J. 2007. Adaptive modelling, adaptive data assimilation and adaptive sampling. Physica D 230, 172–196.
  • Lermusiaux, P. F. J. and Robinson, A. R. 1999. Data assimilation via error subspaces statistical estimation, part I: theory and schemes. Mon. Wea. Rev. 127, 1385–1407.
  • Lermusiaux, P. F. J., Robinson, A. R., Haley, P. J. and Leslie, W. G. 2002. Advanced interdisciplinary data assimilation: filtering and smoothing via error subspace statistical estimation. Proceedings of “The OCEANS 2002 MTS/IEEE conference, Holland”, Vol. 230, pp. 795–802.
  • Livings, D. 2005. Aspects of the ensemble Kalman filter [Master’s thesis] Department of Mathematics, University of Reading, UK.
  • Livings, D., Dance, S. L. and Nichols, N. K. 2008. Unbiased ensemble square root filters. Physica D 237, 1021–1028.
  • Lorenc, A. C. 2003. The potential of the ensemble Kalman filter for NWP - a comparison with 4D-Var. Q. J. R. Meteorol. Soc. 129, 3183–3203.
  • Metref, S., Cosme, E., Snyder, C. and Brasseur, P. 2014a. A non-Gaussian analysis scheme using rank histograms for ensemble data assimilation. Nonlin. Processes Geophys. 21, 869–885.
  • Metref, S., Cosme, E., Snyder, C. and Brasseur, P. 2014b. A non-Gaussian analysis scheme using rank histogram for ensemble data assimilation. Nonlinear Proc. Geophys. 21, 869–885.
  • Minamide, M. and Zhang, F. 2017. Adaptive observation error inflation for assimilating all-sky satellite radiance. Mon. Wea. Rev. 145, 1063–1081.
  • Miyoshi, T. and Yamane, S. 2007. Local ensemble transform Kalman filter with an AGCM at a T159/L48 resolution. Mon. Wea. Rev. 135, 3841–3861.
  • Morzfeld, M., Tu, X., Atkins, E. and Chorin, A. J. 2012. A random map implementation of implicit filters. J. Comput. Phys. 231, 2049–2066.
  • Morzfeld, M., Hodyss, D. and Snyder, C. 2017. What the collapse of the ensemble Kalman filter tells us about particle filters. Tellus A 69, 1–14.
  • Nakano, S., Ueno, G. and Higuchi, T. 2007. Merging particle filter for sequential data assimilation. Nonlinear Process. Geophys. 14, 395–408.
  • Nerger, L. 2004. Parallel Filter Algorithms for Data Assimilation in Oceanography Number 487 in Reports on Polar and Marine Research, Alfred Wegener Institute for Polar and Marine Research, Bremerhaven, Germany, [PhD Thesis]. University of Bremen, Germany.
  • Nerger, L. 2015. On serial observation processing on localized ensemble Kalman filters. Mon. Wea. Rev. 143, 1554–1567.
  • Nerger, L. and Hiller, W. 2013. Software for ensemble-based data assimilation systems - implementation strategies and scalability. Comput. Geosci. 55, 110–118.
  • Nerger, L., Hiller, W. and Schröter, J. 2005a. PDAF -- the parallel data assimilation framework: experiences with Kalman filtering. In: Use of High Performance Computing in Meteorology: Proceedings of the Eleventh ECMWF Workshop on the Use of High Performance Computing in Meteorology, Reading, UK, 25--29 October 2004 (eds. W. Zwieflhofer and G. Mozdzynski). World Scientific, Singapore, pp. 63–83.
  • Nerger, L., Hiller, W. and Schröter, J. 2005b. A comparison of error subspace Kalman filters. Tellus 57A, 715–735.
  • Nerger, L., Danilov, S., Hiller, W. and Schröter, J. 2006. Using sea level data to constrain a finite-element primitive-equation ocean model with a local SEIK filter. Ocean Dyn. 56, 634–649.
  • Nerger, L., Janjić, T., Schroeter, J. and Hiller, W. 2012a. A unification of ensemble square root filters. Mon. Wea. Rev. 140, 2335–2345.
  • Nerger, L., Janjić, T., Schröter, J. and Hiller, W. 2012b. A regulated localization scheme for ensemble-based Kalman filters. Q. J. Roy. Meteor. Soc. 138, 802–812.
  • Ott, E., Hunt, B. R., Szunyogh, I., Zimin, A. V., Kostelich, E. J., and co-authors. 2004. A local ensemble Kalman filter for atmospheric data assimilation. Tellus 56A, 415–428.
  • Patil, D. J., Hunt, B. R., Kalnay, E., Yorke, J. A. and Ott, E. 2001. Local low dimensionality of atmospheric dynamics. Phys. Rev. Lett. 86(26), 5878–5881.
  • Penny, S. and Miyoshi, T. 2016. A local particle filter for high-dimensional geophysical systems. Nonlinear Processes Geophys. 23, 391–405.
  • Perianez, A., Reich, H. and Potthast, R. 2014. Optimal localization for ensemble Kalman filter systems. J. Meteorol. Soc. Jpn. 92, 585–597.
  • Petrie, R. E. and Dance, S. L. 2010. Ensemble-based data assimilation and the localisation problem. Weather 65(3), 65–69.
  • Pham, D. T. 2001. Stochastic methods for sequential data assimilation in strongly nonlinear systems. Mon. Wea. Rev. 129, 1194–1207.
  • Pham, D. T., Verron, J. and Gourdeau, L. 1998a. Singular evolutive Kalman filters for data assimilation in oceanography. C. R. Acad. Sci. Ser. II 326(4), 255–260.
  • Pham, D. T., Verron, J. and Roubaud, M. C. 1998b. A singular evolutive extended Kalman filter for data assimilation in oceanography. J. Mar. Syst. 16, 323–340.
  • Poterjoy, J. 2016a. A localized particle filter for high-dimensional nonlinear systems. Mon. Wea. Rev. 144, 59–76.
  • Poterjoy, J. and Anderson, J. L. 2016b. Efficient sssimilation of simulated observations in a high-dimensional geophysical system using a localized particle filter. Mon. Wea. Rev. 144, 2007–2020.
  • Reich, S. 2013. A nonparametric ensemble transform method for Bayesian inference. SIAM Journal on Scientific Computing 4(35), 2013–2024.
  • Sakov, P. and Oke, P. R. 2008. Implications of the form of the ensemble transformation in the ensemble square root filters. Mon. Wea. Rev. 136, 1042–1053.
  • Sakov, P., Counillon, F., Bertino, L., Lisaeter, K. A., Oke, P. R. and co-authors. 2012. TOPAZ4: an ocean-sea ice data assimilation system for the North Atlantic and Arctic. Ocean Sci. 8, 633–656.
  • Silverman, B. W. 1986. Density estimation for statistics and data analysis Chapman and Hall, New Work.
  • Snyder, C., Bengtsson, T., Bickel, P. and Anderson, J. 2008. Obstacles to high-dimensional particle filtering. Mon. Wea. Rev. 136, 4629–4640.
  • Snyder, C., Bengtsson, T. and Morzfeld, M. 2015. Performance bounds for particle filters using the optimal proposal. Mon. Wea. Rev. 143(11), 4750–4761.
  • Stordal, A. S., Karlsen, H. A., Nævdal, G., Skaug, H. J. and Valles, B. 2011. Bridging the ensemble Kalman filter and particle filters: the adaptive Gaussian mixture filter. Comput. Geosci. 15, 293–305.
  • Testut, C.-E., Brasseur, P., Brankart, J.-M. and Verron, J. 2003. Assimilation of sea-surface temperature and altimetric observationsduring 1992–1993 inot an eddy permitting primitive equation model of the North Atlantic ocean. J. Mar. Sys. 40–41, 291–316.
  • Tippett, M. K., Anderson, J. L., Bishop, C. H., Hamill, T. M. and Whitaker, J. S. 2003. Ensemble square root filters. Mon. Wea. Rev. 131, 1485–1490.
  • Tödter, J. and Ahrens, B. 2015. A Second-Order Exact Ensemble Square Root Filter for Nonlinear Data Assimilation. Mon. Wea. Rev. 143(4), 1347–1367.
  • Tödter, J., Kirchgessner, P., Nerger, L. and Ahrens, B. 2016. Assessment of a nonlinear ensemble transform filter for high-dimensional data assimilation. Mon. Wea. Rev. 144, 409–427.
  • Tong, X. T., Majda, A. J. and Kelly, D. 2016. Nonlinear stability and ergodicity of ensemble based Kalman filters. Nonlinearity 29, 657–691.
  • van Leeuwen, P. J. 2003a. A variance-minimizing filter for nonlinear dynamics. Mon. Wea. Rev. 131, 2071–2084.
  • van Leeuwen, P. J. 2003b. Nonlinear ensemble data assimilation for the ocean. Recent Developments in data assimilation for atmosphere and ocean, ECMWF Seminar 8--12 September 2003, Reading, United Kingdom, pp. 265–286.
  • van Leeuwen, P. J. 2009. Particle filtering in geophysical systems. Mon. Wea. Rev. 137, 4089–4114.
  • van Leeuwen, P. J. 2010. Nonlinear data assimilation in Geosciences: an extremely efficient particle filter. Q. J. R. Meteorol. Soc. 136, 1991–1999.
  • van Leeuwen, P. J. 2011. Efficient nonlinear data-assimilation in geophysical fluid dynamics. Comput. Fluids 46, 52–58.
  • van Leeuwen, P. J., Cheng, Y. and Reich, S. 2015. Nonlinear data assimilation. Frontiers in Applied Dynamical Systems: Reviews and Tutorials 2 Springer.
  • van Leeuwen, P. J. and Evensen, G. 1996. Data assimilation and inverse methods in terms of a probabilistic formulation. Mon. Wea. Rev. 124, 2898–2913.
  • Verlaan, M. and Heemink, A. W. 2001. Nonlinearity in data assimilation applications: a practical method for analysis. Mon. Wea. Rev. 129(6), 1578–1589.
  • Vossepoel, F. C. and Van Leeuwen, P. J. 2006. Parameter estimation using a particle method: inferring mixing coefficients from sea-level observations. Mon. Wea. Rev. 135, 1006–1020.
  • Wang, Y., Jung, Y., Supine, T. A. and Xue, M. 2013. A hybrid MPI-OpenMP parallel algorithm and performance analysis for an ensemble square root filter designed for multiscale observations. J. Atm. and Oce. Tech. 30, 1382–1397.
  • Whitaker, J. S. and Hamill, T. M. 2002. Ensemble data assimilation without perturbed observations. Mon. Wea. Rev. 130, 1913–1924.
  • Wikle, C. K. and Berliner, L. M. 2006. A Bayesian tutorial for data assimilation. Physica D, 230(1):1–16.
  • Xiong, X., Navon, I. M. and Uzunoglu, B. 2006. A note on the particle filter with posterior Gaussian resampling. Tellus A 58(4), 456–460.
  • Zhen, Y. and Zhang, F. 2014. A probabilistic approach to adaptive covariance localization for serial ensemble square root filters. Mon. Wea. Rev. 142, 4499–4518.
  • Zhu, M., van Leeuwen, P. J. and Amezcua, J. 2016. Implicit equal-weights particle filter. Q. J. R. Meteorol. Soc. page personal communication.
  • Zupanski, M. 2005. Maximum likelihood ensemble filter: Theoretical aspects. Mon. Wea. Rev. 133, 1710–1726.
  • Zupanski, M., Michael, N. I. and Zupanski, D. 2008. The maximum likelihood ensemble filter as a non-differentiable minimization algorithm. Q. J. R. Meteorol. Soc. 134, 1039–1050.