Improving the condition number of estimated covariance matrices

References

• Axelsson, O. 1996. Iterative Solution Methods. Cambridge: Cambridge University Press.
   Google Scholar
• Bannister, R. N. 2017. A review of operational methods of variational and ensemble variational data assimilation. Q. J. R. Meteorol. Soc. 143, 607–633. doi:10.1002/qj.2982
   Google Scholar
• Bickel, P. J. and Levina, E. 2008. Covariance regularization by thresholding. Ann. Stat. 36, 2577–2604. doi:10.1214/08-AOS600
   Google Scholar
• Bormann, N., Bonavita, M., Dragani, R., Eresmaa, R., Matricardi, M. and co-authors. 2015. Enhancing the Impact of IASI Observations through an Updated Observation Error Covariance Matrix. ECMWF Technical Memoranda. ECMWF, Reading.
   Google Scholar
• Bormann, N., Bonavita, M., Dragani, R., Eresmaa, R., Matricardi, M. and co-authors. 2016. Enhancing the impact of IASI observations through an updated observation error covariance matrix. Q. J. R. Meteorol. Soc. 142, 1767–1780. doi:10.1002/qj.2774
   Google Scholar
• Budd, C. J., Freitag, M. A. and Nichols, N. K. 2011. Regularization techniques for ill-posed inverse problems in data assimilation. Comput. Fluids 46, 168–173. doi:10.1016/j.compfluid.2010.10.002
   Google Scholar
• Campbell, W. F., Satterfield, E. A., Ruston, B. and Baker, N. L. 2017. Accounting for correlated observation error in a dual-formulation 4D variational data assimilation system. Mon. Wea. Rev. 145, 1019–1032. doi:10.1175/MWR-D-16-0240.1
   Google Scholar
• Cheney, W. 2005. Numerical Mathematics and Computing. 5th ed., International ed. Brooks/Cole Cengage Learning, Pacific Grove, CA.
   Google Scholar
• Cordoba, M., Dance, S. L., Kelly, G. A., Nichols, N. K. and Waller, J. A. 2016. Diagnosing atmospheric motion vector observation errors for an operational high resolution data assimilation system. Q. J. R. Meteorol. Soc. 143, 333–341.
   Google Scholar
• Daley, R. 1991. Atmospheric Data Analysis. Cambridge University Press, New York.
   Google Scholar
• Desroziers, G., Berre, L., Chapnik, B. and Poli, P. 2005. Diagnosis of observation, background and analysis-error statistics in observation space. Q. J. R. Meteorol. Soc. 131, 3385–3396. doi:10.1256/qj.05.108
   Google Scholar
• Fan, K. 1959. Convex sets and their applications. Lecture notes. Applied Mathematics Division, Argonne National Laboratory, Lemont, IL.
   Google Scholar
• Fowler, A. M., Dance, S. L. and Waller, J. A. 2018. On the interaction of observation and prior error correlations in data assimilation. Q. J. R. Meteorol. Soc. 144, 48–62. doi:10.1002/qj.3183
   Google Scholar
• Ghil, M. 1989. Meteorological data assimilation for oceanographers. Part I: description and theoretical framework. Dyn. Atmos. Oceans. 13, 171–218. doi:10.1016/0377-0265(89)90040-7
   Google Scholar
• Ghil, M. and Malanotte-Rizzoli, P. 1991. Data assimilation in meteorology and oceanography. Adv. Geophys. 33, 141–266. doi:10.1016/S0065-2687(08)60442-2
   Google Scholar
• Gill, P. E., Murray, W. and Wright, M. H. 1986. Practical Optimization. Academic Press, Amsterdam/London.
   Google Scholar
• Golub, G. H. and Van Loan, C. F. 1996. Matrix Computations. 3rd ed. The John Hopkins University Press, Baltimore, MD.
   Google Scholar
• Gray, R. M. 2006. Toeplitz and circulant matrices: a review. Found. Trends Commun. Inform. Theory 2, 155–239.
   Google Scholar
• Guillet, O., Weaver, A. T., Vasseur, X., Michel, Y., Gratton, S. and co-authors. 2019. Modelling spatially correlated observation errors in variational data assimilation using a diffusion operator on an unstructured mesh. Q. J. R. Meteorol. Soc. 145, 1947–1967. doi:10.1002/qj.3537
   Google Scholar
• Haben, S. A. 2011. Conditioning and preconditioning of the minimisation problem in variational data assimilation. PhD thesis, Department of Mathematics and Statistics, University of Reading, Reading.
   Google Scholar
• Haben, S. A., Lawless, A. S. and Nichols, N. K. 2011a. Conditioning and preconditioning of the variational data assimilation problem. Comput. Fluids 46, 252–256. doi:10.1016/j.compfluid.2010.11.025
   Google Scholar
• Haben, S. A., Lawless, A. S. and Nichols, N. K. 2011b. Conditioning of incremental variational data assimilation, with application to the Met Office system. Tellus A 64, 782–792.
   Google Scholar
• Hamill, T. M., Whitaker, J. S. and Snyder, C. 2001. Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev. 129, 2776–2790. doi:10.1175/1520-0493(2001)129<2776:DDFOBE>2.0.CO;2
   Google Scholar
• Hansen, P. C. 1998. Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion. Society for Industrial and Applied Mathematics, Philadelphia, PA.
   Google Scholar
• Healy, S. B. and White, A. A. 2005. Use of discrete Fourier transforms in the 1D-Var retrieval problem. Q. J. R. Meteorol. Soc. 131, 63–72. doi:10.1256/qj.03.193
   Google Scholar
• Heilliette, S. and Garand, L. 2015. Impact of accounting for inter-channel error covariances at the Canadian meteorological centre. In: Proceedings of the 2015 EUMETSAT Meteorological Satellite Conference, Toulouse, France, p. 8. Online at: https://www.eumetsat.int/website/home/News/ConferencesandEvents/PreviousEvents/DAT_2305526.html
   Google Scholar
• Higham, N. J. 2002. Computing the nearest correlation matrix – a problem from finance. IMA J. Numer. Anal. 22, 329–343. doi:10.1093/imanum/22.3.329
   Google Scholar
• Higham, N. J., Strabić, N. and Šego, V. 2016. Restoring definiteness via shrinking, with an application to correlation matrices with a fixed block. SIAM Rev. 58, 245–263. doi:10.1137/140996112
   Google Scholar
• Hoerl, A. E. and Kennard, R. W. 1970. Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12, 55–67. doi:10.1080/00401706.1970.10488634
   Google Scholar
• Horn, R. A. and Johnson, C. R. 1991. Topics in Matrix Analysis. Cambridge University Press, Cambridge.
   Google Scholar
• Janjić, T., Bormann, N., Bocquet, M., Carton, J. A., Cohn, S. E. and co-authors. 2018. On the representation error in data assimilation. Q. J. R. Meteorol. Soc. 144, 1257–1278. doi:10.1002/qj.3130
   Google Scholar
• Johnson, C., Hoskins, B. J. and Nichols, N. K. 2005. A singular vector perspective of 4D-Var: filtering and interpolation. Q. J. R. Meteorol. Soc. 131, 1–19. doi:10.1256/qj.03.231
   Google Scholar
• Ledoit, O. and Wolf, M. 2004. A well-conditioned estimator for large-dimensional covariance matrix. J. Multivar. Anal. 88, 365–411. doi:10.1016/S0047-259X(03)00096-4
   Google Scholar
• Liu, Z.-Q. and Rabier, F. 2003. The potential of high-density observations for numerical weather prediction: a study with simulated observations. Q. J. R. Meteorol. Soc. 129, 3013–3035. doi:10.1256/qj.02.170
   Google Scholar
• McNally, A. P., Watts, P. D., Smith, J. A., Engelen, R., Kelly, G. A. and co-authors. 2006. The assimilation of AIRS radiance data at ECMWF. Q. J. R. Meteorol. Soc. 132, 935–957. doi:10.1256/qj.04.171
   Google Scholar
• Ménétrier, B., Montmerle, T., Michel, Y. and Berre, L. 2015. Linear filtering of sample covariances for ensemble-based data assimilation. Part I: optimality criteria and application to variance filtering and covariance localization. Mon. Wea. Rev. 143, 1622–1643. doi:10.1175/MWR-D-14-00157.1
   Google Scholar
• Michel, Y. 2018. Revisiting Fisher’s approach to the handling of horizontal spatial correlations of the observation errors in a variational framework. Q. J. R. Meteorol. Soc. 144, 2011–2025. doi:10.1002/qj.3249
   Google Scholar
• Migliorini, S. 2012. On the equivalence between radiance and retrieval assimilation. Mon. Wea. Rev. 140, 258–265. doi:10.1175/MWR-D-10-05047.1
   Google Scholar
• Moodey, A. J. F., Lawless, A. S., Potthast, R. W. E. and van Leeuwen, P. J. 2013. Nonlinear error dynamics for cycled data assimilation methods. Inverse Probl. 29, 025002. doi:10.1088/0266-5611/29/2/025002
   Google Scholar
• Nakamura, G. and Potthast, R. 2015. Inverse Modeling. Bristol: IOP Publishing, pp. 2053–2563.
   Google Scholar
• Pourahmadi, M. 2013. High-Dimensional Covariance Estimation. John Wiley & Sons, Hoboken, NJ.
   Google Scholar
• Prates, C., Migliorini, S., Stewart, L. and Eyre, J. 2016. Assimilation of transformed retrievals obtained from clear-sky IASI measurements. Q. J. R. Meteorol. Soc. 142, 1697–1712. doi:10.1002/qj.2764
   Google Scholar
• Qi, H. and Sun, D. 2010. Correlation stress testing for value-at-risk: an unconstrained convex optimization approach. Comput. Optim. Appl. 45, 427–462. doi:10.1007/s10589-008-9231-4
   Google Scholar
• Schiff, S. J. 2011. Neural Control Engineering: The Emerging Intersection between Control Theory and Neuroscience. Computational Neuroscience. MIT Press, Cambridge, MA.
   Google Scholar
• Simonin, D., Waller, J. A., Ballard, S. P., Dance, S. L. and Nichols, N. K. 2019. A pragmatic strategy for implementing spatially correlated observation errors in an operational system: demonstration for Doppler radar winds. Q. J. R. Meteorol. Soc. 145, 2772–2790. doi:10.1002/qj.3592
   Google Scholar
• Skøien, J. O. and Blöschl, G. 2006. Sampling scale effects in random fields and implications for environmental monitoring. Environ. Monit. Assess. 114, 521–552. doi:10.1007/s10661-006-4939-z
   Google Scholar
• Smith, P. J., Lawless, A. S. and Nichols, N. K. 2018. Treating sample covariances for use in strongly coupled atmosphere-ocean data assimilation. Geophys. Res. Lett. 45, 445–454. doi:10.1002/2017GL075534
   Google Scholar
• Stewart, L. M. 2010. Correlated observation errors in data assimilation. PhD thesis, University of Reading, Reading.
   Google Scholar
• Stewart, L. M., Cameron, J., Dance, S. L., English, S., Eyre, J. and Nichols, N. K. 2009. Observation Error Correlations in IASI Radiance Data. Mathematics Report Series, Report number 1/2009, University of Reading, Reading.
   Google Scholar
• Stewart, L. M., Dance, S. L. and Nichols, N. K. 2008. Correlated observation errors in data assimilation. Int. J. Numer. Meth. Fluids 56, 1521–1527. doi:10.1002/fld.1636
   Google Scholar
• Stewart, L. M., Dance, S. L. and Nichols, N. K. 2013. Data assimilation with correlated observation errors: experiments with a 1-D shallow water model. Tellus A 65, 19546 (14 pp.), doi:10.3402/tellusa.v65i0.19546
   Google Scholar
• Stewart, L. M., Dance, S. L., Nichols, N. K., Eyre, J. R. and Cameron, J. 2014. Estimating interchannel observation-error correlations of IASI radiance data in the Met Office system. Q. J. R. Meteorol. Soc. 140, 1236–1244. doi:10.1002/qj.2211
   Google Scholar
• Tabeart, J. M., Dance, S. L., Haben, S. A., Lawless, A. S., Nichols, N. K. and Waller, J. A. 2018. The conditioning of least squares problems in variational data assimilation. Numer. Linear Algebra Appl. 25, e2165. doi:10.1002/nla.2165
   Google Scholar
• Tabeart, J. M., Dance, S. L., Lawless, A. S., Migliorini, S., Nichols, N. K., and co-authors.2019. The impact of reconditioning of the correlated observation error covariance matrix on the Met Office system. Online at: http://arxiv.org/abs/1908.04071. Submitted.
   Google Scholar
• Tanaka, M. and Nakata, K. 2014. Positive definite matrix approximation with condition number constraint. Optim. Lett. 8, 939–947. doi:10.1007/s11590-013-0632-7
   Google Scholar
• Thiebaux, H. J. 1976. Anisotropic correlation functions for objective analysis. Mon. Wea. Rev. 104, 994–1002. doi:10.1175/1520-0493(1976)104<0994:ACFFOA>2.0.CO;2
   Google Scholar
• Trefethen, L. N. and Bau, D. 1997. Numerical Linear Algebra. Society for Industrial and Applied Mathematics, Philadelphia, PA.
   Google Scholar
• Waller, J. A., Ballard, S. P., Dance, S. L., Kelly, G., Nichols, N. K. and Simonin, D. 2016. Diagnosing horizontal and inter-channel observation error correlations for SEVIRI observations using observation-minus-background and observation-minus-analysis statistics. Remote Sens. 8, 581. doi:10.3390/rs8070581
   Google Scholar
• Waller, J. A., Dance, S. L. and Nichols, N. K. 2016. Theoretical insight into diagnosing observation error correlations using observation-minus-background and observation-minus-analysis statistics. Q. J. R. Meteorol. Soc. 142, 418–431. doi:10.1002/qj.2661
   Google Scholar
• Waller, J. A., Simonin, D., Dance, S. L., Nichols, N. K. and Ballard, S. P. 2016. Diagnosing observation error correlations for Doppler radar radial winds in the Met Office UKV model using observation-minus-background and observation-minus-analysis statistics. Mon. Wea. Rev. 144, 3533–3551. doi:10.1175/MWR-D-15-0340.1
   Google Scholar
• Weston, P. 2011. Progress towards the Implementation of Correlated Observation Errors in 4D-Var. Forecasting Research Technical Report 560, Met Office, Exeter, UK.
   Google Scholar
• Weston, P. P., Bell, W. and Eyre, J. R. 2014. Accounting for correlated error in the assimilation of high-resolution sounder data. Q. J. R. Meteorol. Soc. 140, 2420–2429. doi:10.1002/qj.2306
   Google Scholar
• Yaglom, A. M. 1987. Correlation Theory of Stationary and Related Random Functions, Volume I: Basic Results. Springer-Verlag, New York.
   Google Scholar