Recommendations on statistics and benchmarks to assess photochemical model performance

References

• Baker, K., and P. Scheff. 2007. Photochemical model performance for PM2.5 sulfate, nitrate, ammonium, and precursor species SO2, HNO3, and NH3 at background monitor locations in the central and eastern United States. Atmos. Environ. 41:6185–95. doi:10.1016/j.atmosenv.2007.04.006
   Web of Science ®Google Scholar
• Bessagnet, B., A. Hodzic, R. Vautard, M. Beekmann, S. Cheinet, C. Honoré, C. Liousse, and L. Rouil. 2004. Aerosol modeling with CHIMERE: Preliminary evaluation at the continental scale. Atmos. Environ. 38:2803–17. doi:10.1016/j.atmosenv.2004.02.034
   Web of Science ®Google Scholar
• Boylan, J.W., and A.G. Russell. 2006. PM and light extinction model performance metrics, goals, and criteria for three-dimensional air quality models. Atmos. Environ. 40:4946–59. doi:10.1016/j.atmosenv.2005.09.087
   Web of Science ®Google Scholar
• Brandt, J., J.H. Christensen, L.M. Frohn, C. Geels, K.M. Hansen, G.B. Hedegaard, M. Hvidberg, and C.A. Skjøth. 2007. THOR—An operational and integrated model system for air pollution forecasting and management from regional to local scale. Proceedings of the 2nd ACCENT Symposium, Urbino (Italy), July 23–27, 2007.
   Google Scholar
• Byun, D.W., S.T. Kim, and S.B. Kim. 2007. Evaluation of air quality models for the simulation of a high ozone episode in the Houston metropolitan area. Atmos. Environ. 41:837–53. doi:10.1016/j.atmosenv.2006.08.038
   Web of Science ®Google Scholar
• Dennis, R., T. Fox, M. Fuentes, A. Gilliland, S. Hanna, C. Hogrefe, J. Irwin, S.T. Rao, R, Scheffe, K. Schere, D.A. Steyn, and A. Venkatram. 2010. A framework for evaluating regional-scale numerical photochemical modeling systems. J. Environ. Fluid Mech. 10:471–89. doi: 10.1007/s10652-009-9163-2.
   PubMed Web of Science ®Google Scholar
• Downey, N., C. Emery, J. Jung, T. Sakulyanontvittaya, L. Hebert, D. Blewitt, and G.Yarwood. 2015. Emission reductions and urban ozone responses under more stringent US standards. Atmos. Environ. 101:209–16. doi: 10.1016/j.atmosenv.2014.11.018
   Web of Science ®Google Scholar
• Dunker, A.M. 1981. Efficient calculation of sensitivity coefficients for complex atmospheric models. Atmos. Environ. (Part A) 15:1155–61. doi:10.1016/0004-6981(81)90305-X
   Web of Science ®Google Scholar
• Dunker, A.M., G. Yarwood, J.P. Ortmann, and G.M. Wilson. 2002. Comparison of source apportionment and source sensitivity of ozone in a three-dimensional air quality model. Environ. Sci. Technol. 36:2953–64. doi:10.1021/es011418f
   PubMed Web of Science ®Google Scholar
• Eder, B., D. Kang, R. Mathur, J. Pleim, S. Yu, T. Otte, and G. Pouliot. 2009. A performance evaluation of the National Air Quality Forecast Capability for the summer of 2007. Atmos. Environ. 43:2312–20. doi:10.1016/j.atmosenv.2009.01.033
   Web of Science ®Google Scholar
• Emery, C., E. Tai, and G. Yarwood. 2001. Enhanced Meteorological Modeling and Performance Evaluation for Two Texas Ozone Episodes. Report to the Texas Natural Resources Conservation Commission, prepared by ENVIRON International Corp, Novato, CA. http://www.tceq.state.tx.us/assets/public/implementation/air/am/contracts/reports/mm/EnhancedMetModelingAndPerformanceEvaluation.pdf.
   Google Scholar
• Emery, C., J. Jung, N. Downey, J. Johnson, M. Jimenez, G. Yarwood, and R. Morris. 2012. Regional and global modeling estimates of policy relevant background ozone over the United States. Atmos. Environ. 47:206–217. doi:10.1016/j.atmosenv.2011.11.012
   Web of Science ®Google Scholar
• Foley, K.M., S.J. Roselle, K.W. Appel, P.V. Bhave, J.E. Pleim, T.L. Otte, R. Mathur, G. Sarwar, J.O. Young, R.C. Gilliam, et al. 2010. Incremental testing of the Community Multiscale Air Quality (CMAQ) modeling system version 4.7. Geoscientific Model Dev. 3:205–26. doi:10.5194/gmd-3-205-2010
   Web of Science ®Google Scholar
• Foley, K.M., C. Hogrefe, G. Pouliot, N. Possiel, S.J. Roselle, H. Simon, and B. Timin. 2015. Dynamic evaluation of CMAQ part I: Separating the effects of changing emissions and changing meteorology on ozone levels between 2002 and 2005 in the eastern US. Atmos. Environ. 103:247–55. doi:10.1016/j.atmosenv.2014.12.038
   Web of Science ®Google Scholar
• Gaydos, T.M., R. Pinder, B. Koo, K.M. Fahey, G. Yarwood, and S.N. Pandis. 2007. Development and application of a three-dimensional aerosol chemical transport model, PMCAMx. Atmos. Environ. 41:2594–611. doi:10.1016/j.atmosenv.2006.11.034
   Web of Science ®Google Scholar
• Grell, G.A., S. Emeis, W.R. Stockwell, T. Schoenemeyer, R. Forkel, J. Michalakes, R. Knoche, and W. Seidl. 2000. Application of a multiscale, coupled MM5/chemistry model to the complex terrain of the VOTALP valley campaign. Atmos. Environ. 34:1435–53. doi:10.1016/S1352-2310(99)00402-1
   Web of Science ®Google Scholar
• Grell, G.A., S.E. Peckham, R. Schmitz, S.A. McKeen, G. Frost, W.C. Skamarock, and B. Eder. 2005. Fully coupled “online” chemistry within the WRF model. Atmos. Environ. 39:6957–75. doi:10.1016/j.atmosenv.2005.04.027
   Web of Science ®Google Scholar
• Hakami, A., M.T. Odman, and A.G. Russell. 2003. High-order, direct sensitivity analysis of multidimensional air quality models. Environ. Sci. Technol. 37:2442–52. doi:10.1021/es020677h
   PubMed Web of Science ®Google Scholar
• Harvard University. 2015. GEOS-Chem Online User’s Guide. Atmospheric Chemistry Modeling Group, School of Engineering and Applied Sciences, Harvard University, Cambridge, MA (http://acmg.seas.harvard.edu/geos/doc/man/index.html).
   Google Scholar
• Hogrefe, C., W. Hao, K. Civerolo, J.Y. Ku, G. Sistla, R.S. Gaza, L. Sedefian, K. Schere, A. Gilliland, A., and R. Mathur. 2007. Daily simulation of ozone and fine particulates over New York State: Findings and challenges. J. Appl. Meteorol. Climatol. 46:961–79. doi:10.1175/JAM2520.1
   Web of Science ®Google Scholar
• Hogrefe, C., K.L. Civerolo, W. Hao, J.Y. Ku, E.E. Zalewsky, and G. Sistla. 2008. Rethinking the assessment of photochemical modeling systems in air quality planning applications. J. Air Waste Manage. Assoc. 58:1086–99. doi:10.3155/1047-3289.58.8.1086
   Web of Science ®Google Scholar
• Hogrefe, C., W. Hao, E.E. Zalewsky, J.Y. Ku, B. Lynn, C. Rosenzweig, M.G. Schultz, S. Rast, M.J. Newchurch, L. Wang, P.L. Kinney, and G. Sistla. 2011. An analysis of long-term regional-scale ozone simulations over the northeastern United States: Variability and trends. Atmos. Chem. Phys. 11:567–82. doi:10.5194/acp-11-567-2011
   Web of Science ®Google Scholar
• Hou, X., M.J. Strickland, and K.J. Liao. 2015. Contributions of regional air pollutant emissions to ozone and fine particulate matter-related mortalities in eastern U.S. urban areas. Environ. Res. 137:475–84. doi:10.1016/j.envres.2014.10.038
   PubMed Web of Science ®Google Scholar
• Im, U., R. Bianconi, E. Solazzo, I. Kioutsioukis, A. Badia, A. Balzarini, R. Baro, R. Bellasio, D. Brunner, C. Chemel, G. Curci, et al. 2015. Evaluation of operational on-line-coupled regional air quality models over Europe and North America in the context of AQMEII Phase 2. Part I: Ozone. Atmos. Environ. 115:404–20. doi:10.1016/j.atmosenv.2014.08.072
   Web of Science ®Google Scholar
• Jin, L., N.J. Brown, R.A. Harley, J.W. Bao, S.A. Michelson, and J.M. Wilczak. 2010. Seasonal versus episodic performance evaluation for an Eulerian photochemical air quality model. J. Geophys. Res. Atmos. 115:D09302. doi:10.1029/2009JD012680
   Google Scholar
• Kang, D., R. Mathur, and S.T. Rao. 2010. Real-time bias-adjusted O3 and PM2.5 air quality index forecasts and their performance evaluations over the continental United States. Atmos. Environ. 44:2203–12. doi:10.1016/j.atmosenv.2010.03.017
   Web of Science ®Google Scholar
• Karamchandani, P., K. Vijayaraghavan, S.Y. Chen, C. Seigneur, and E.S. Edgerton, 2006. Plume-in-grid modeling for particulate matter. Atmos. Environ. 40:7280–97. doi:10.1016/j.atmosenv.2006.06.033
   Web of Science ®Google Scholar
• Kim, Y., J.S. Fu, and T.L Miller. 2010. Improving ozone modeling in complex terrain at a fine grid resolution—Part II: influence of schemes in MM5 on daily maximum 8-h ozone concentrations and RRFs (Relative Reduction Factors) for SIPs in the non-attainment areas. Atmos. Environ. 44:2116–24. doi:10.1016/j.atmosenv.2010.02.038
   Web of Science ®Google Scholar
• Koo, B., N. Kumar, E. Knipping, U. Nopmongcol, T. Sakulyanontvittaya, M.T. Odman, A.G. Russell, and G. Yarwood. 2015. Chemical transport model consistency in simulating regulatory outcomes and the relationship to model performance. Atmos. Environ. 116:159–71. doi:10.1016/j.atmosenv.2015.06.036
   Web of Science ®Google Scholar
• Lee, D., D.W. Byun, H. Kim, F. Ngan, S.Kim, C. Lee, and C. Cho. 2011. Improved CMAQ predictions of particulate matter utilizing the satellite-derived aerosol optical depth. Atmos. Environ. 45:3730–41. doi:10.1016/j.atmosenv.2011.04.018
   Web of Science ®Google Scholar
• Liao, K.-J., X. Hou, and D. R. Baker. 2014. Impacts of interstate transport of pollutants on high ozone events over the Mid-Atlantic United States. Atmos. Environ. 84:100–12. doi:10.1016/j.atmosenv.2013.10.062
   Web of Science ®Google Scholar
• Liu, X.-H., Y. Zhang, K.M. Olsen, W.-X, Wang, B.A. Do, and G.M. Bridgers. 2010. Responses of future air quality to emission controls over North Carolina, Part I: Model evaluation for current-year simulations. Atmos. Environ. 44:2443–56. doi:10.1016/j.atmosenv.2010.04.002
   Web of Science ®Google Scholar
• Makar, P.A., W. Gong, C. Mooney, J. Zhang, D. Davignon, M. Samaali, M.D. Moran, H. He, D.W. Tarasick, D. Sills, and J. Chen. 2010. Dynamic adjustment of climatological ozone boundary conditions for air-quality forecasts. Atmos. Chem. Phys. 10:8997–9015. doi:10.5194/acp-10-8997-2010
   Web of Science ®Google Scholar
• Makar, P.A., W. Gong, C. Hogrefe, Y. Zhang, G. Curci, R. Zabkar, J. Milbrandt, U. Im, S. Galmarini, A. Balzarini, et al. 2015. Feedbacks between air pollution and weather, Part 1: Effects on chemistry. Atmos. Environ. 115:499–526. doi:10.1016/j.atmosenv.2014.10.021
   Web of Science ®Google Scholar
• Marmur, A., W. Liu, Y. Wang, A.G. Russell, and E.S. Edgerton. 2009. Evaluation of model simulated atmospheric constituents with observations in the factor projected space: CMAQ simulations of SEARCH measurements. Atmos. Environ. 43:1839–49. doi:10.1016/j.atmosenv.2008.12.027
   Web of Science ®Google Scholar
• Menut, L., R. Vautard, M. Beekmann, and C. Honore. 2000. Sensitivity of photochemical pollution using the adjoint of a simplified chemistry-transport model. J. Geophys. Res. Atmos. 105:15379–402. doi:10.1029/1999JD900953
   Web of Science ®Google Scholar
• National Research Council. 2007. Models in Environmental Regulatory Decision Making. Washington, DC: National Research Council of the National Academies. doi:10.17226/11972.
   Google Scholar
• Odman, T., and Z. Adelman. 2014. Emissions and Air Quality Modeling for SEMAP. Final Report to Southeastern States Air Resource Managers, Inc., December 31, 2014, Georgia Tech, http://semap.ce.gatech.edu/sites/default/files/files/SEMAP-Revised-Final-Report_Final.pdf.
   Google Scholar
• Pun, B.K., C. Seigneur, K. Vijayaraghavan, S.Y. Wu, S.Y. Chen, E.M. Knipping, and N. Kumar. 2006. Modeling regional haze in the BRAVO study using CMAQMADRID: 1. Model evaluation. J. Geophys. Res. Atmos. 111:D06302. doi:10.1029/2004JD005608
   Web of Science ®Google Scholar
• Queen, A., and Y. Zhang. 2008. Examining the sensitivity of MM5-CMAQ predictions to explicit microphysics schemes and horizontal grid resolutions, Part II—PM concentrations and wet deposition predictions. Atmos. Environ. 42:3856–68. doi:10.1016/j.atmosenv.2007.12.066
   Web of Science ®Google Scholar
• Ramboll Environ. 2016. User’s Guide: Comprehensive Air quality Model with extensions, Version 6.30. Ramboll Environ, Novato, CA (www.camx.com).
   Google Scholar
• Rodriguez, M.A., M.G. Barna, K.A. Gebhart, J.L. Hand, Z.E. Adelman, B.A. Schichtel, J.L. Collett, Jr., and W.C. Malm. 2011. Modeling the fate of atmospheric reduced nitrogen during the Rocky Mountain Atmospheric Nitrogen and Sulfur Study (RoMANS): Performance evaluation and diagnosis using integrated processes rate analysis. Atmos. Environ. 45:223–34. doi:10.1016/j.atmosenv.2010.09.011
   Web of Science ®Google Scholar
• Sakulyanontvittaya, T., A. Guenther, D. Helmig, J. Milford, and C. Wiedinmyer. 2008. Secondary organic aerosol from sesquiterpene and monoterpene emissions in the United States. Environ. Sci. Technol. 42:8784–90. doi:10.1021/es800817r
   PubMed Web of Science ®Google Scholar
• Simon, H., K.R. Baker, and S., Phillips. 2012. Compilation and interpretation of photochemical model performance statistics published between 2006 and 2012. Atmos. Environ. 61:124–39. doi:10.1016/j.atmosenv.2012.07.012
   Google Scholar
• Smyth, S.C., W.M. Jiang, D.Z. Yin, H. Roth, and T. Giroux. 2006. Evaluation of CMAQ O3 and PM2.5 performance using Pacific 2001 measurement data. Atmos. Environ. 40:2735–49. doi:10.1016/j.atmosenv.2005.10.068
   Web of Science ®Google Scholar
• Smyth, S.C., W. Jiang, H. Roth, M.D. Moran, P.A. Makar, F. Yang, V.S. Bouchet, and H. Landry. 2009. A comparative performance evaluation of the AURAMS and CMAQ air-quality modelling systems. Atmos. Environ. 43:1059–70. doi:10.1016/j.atmosenv.2008.11.027
   Web of Science ®Google Scholar
• Solazzo, E., R. Bianconi, G. Pirovano, V. Matthias, R. Vautard, K.W. Appel, B. Bessagnet, J. Brandt, J.H. Christensen, C. Chemel, et al. 2012. Operational model evaluation for particulate matter in Europe and North America in the context of AQMEII. Atmos. Environ. 53:75–92. doi:10.1016/j.atmosenv.2012.02.045
   Web of Science ®Google Scholar
• Solazzo, E. and S. Galmarini. 2016. Error apportionment for atmospheric chemistry-transport models – A new approach to model evaluation. Atmos. Chem. Phys. 16:6263–6283 doi:10.5194/acp-16-6263-2016.
   Web of Science ®Google Scholar
• Spak, S.N., and T. Holloway. 2009. Seasonality of speciated aerosol transport over the Great Lakes region. J. Geophys. Res. Atmos. 114:D08302. doi:10.1029/2008JD010598
   Google Scholar
• Stroud, C.A., P.A. Makar, M.D. Moran, W. Gong, S. Gong, J. Zhang, K. Hayden, C. Mihele, J.R. Brook, J.P.D. Abbatt, and J.G. Slowik. 2011. Impact of model grid spacing on regional- and urban- scale air quality predictions of organic aerosol. Atmos. Chem. Phys. 11:3107–18. doi:10.5194/acp-11-3107-2011
   Web of Science ®Google Scholar
• Tang, W., D.S. Cohan, G.A. Morris, D.W. Byun, and W.T. Luke. 2011. Influence of vertical mixing uncertainties on ozone simulation in CMAQ. Atmos. Environ. 45:2898–909. doi:10.1016/j.atmosenv.2011.01.057
   Web of Science ®Google Scholar
• Taylor, K.E. 2001. Summarizing multiple aspects of model performance in a single diagram. J. Geophys. Res. 106:7183–92. doi:10.1029/2000JD900719
   Web of Science ®Google Scholar
• Tesche, T.W. 1988. Accuracy of ozone air-quality models. J. Environ. Eng. ASCE 114:739–52. doi:10.1061/(ASCE)0733-9372(1988)114:4(739)
   Web of Science ®Google Scholar
• Thunis, P., E. Georgieva, and S. Galmarini. 2011. A procedure for air quality models benchmarking. Prepared by the Joint Research Centre, Ispra, Italy ( 16 February 2011). http://fairmode.jrc.ec.europa.eu/document/fairmode/WG1/WG2_SG4_benchmarking_V2.pdf
   Google Scholar
• U.S. EPA. 1990. User’s Guide for the Urban Airshed Model, Volume I: User’s Manual for UAM (CB-IV). U.S. Environmental Protection Agency, Research Triangle Park, NC (EPA-450/4-90-007A).
   Google Scholar
• U.S. EPA. 1991. Guideline for Regulatory Applications of the Urban Airshed Model. U.S. Environmental Protection Agency, Research Triangle Park, NC (EPA-450/4-91-013).
   Google Scholar
• U.S. EPA. 2007. Guidance on the Use of Models and Other Analyses for Demonstrating Attainment of Air Quality Goals for Ozone, PM_2.5, and Regional Haze. U.S. Environmental Protection Agency, Research Triangle Park, NC (EPA-454/B-07-002).
   Google Scholar
• U.S. EPA. 2011a. Policy Assessment for the Review of the Secondary National Ambient Air Quality Standards for Oxides of Nitrogen and Oxides of Sulfur. U.S. Environmental Protection Agency, Research Triangle Park, NC ( EPA-452/R-11-005a, February).
   Google Scholar
• U.S. EPA. 2011b. The Benefits and Costs of the Clean Air Act from 1990 to 2020. U.S. Environmental Protection Agency, Washington, DC (March).
   Google Scholar
• U.S. EPA. 2011c. Policy Assessment for the Review of the Particulate Matter National Ambient Air Quality Standards. U.S. Environmental Protection Agency, Research Triangle Park, NC ( EPA 452/R-11-003, April).
   Google Scholar
• U.S. EPA. 2011d. Air Quality Modeling, Final Rule Technical Support Document. US Environmental Protection Agency, Research Triangle Park, NC (June).
   Google Scholar
• U.S. EPA. 2014. Draft Modeling Guidance for Demonstrating Attainment of Air Quality Goals for Ozone, PM_2.5, and Regional Haze. U.S. Environmental Protection Agency, Research Triangle Park, NC ( December).
   Google Scholar
• Willmott, C.J. 1982. Some comments on the evaluation of model performance. Bull. Am. Meteorol. Soc. 63:1309–13. doi:10.1175/1520-0477(1982)063<1309:SCOTEO>2.0.CO;2
   Web of Science ®Google Scholar
• Wong, D.C., J. Pleim, R. Mathur, F. Binkowski, T. Otte, R. Gilliam, G. Pouliot, A. Xiu, J.O. Young, and D. Kang. 2012. WRF-CMAQ two-way coupled system with aerosol feedback: software development and preliminary results. Geoscientific Model Dev. 5:299–312. doi:10.5194/gmd-5-299-2012.
   Web of Science ®Google Scholar
• Wu, S.-Y., S. Krishnan, and Y. Zhang, V. Aneja. 2008. Modeling atmospheric transport and fate of ammonia in North Carolina—Part I: Evaluation of meteorological and chemical predictions. Atmos. Environ. 42:3419–36. doi:10.1016/j.atmosenv.2007.04.031
   Web of Science ®Google Scholar
• Yu, S., R. Mathur, D. Kang, K. Schere, B. Eder, and J. Pleim. 2006. Performance and diagnostic evaluation of ozone predictions by the Eta-community multiscale air quality forecast system during the 2002 New England Air Quality Study. J. Air Waste Manage. Assoc. 56:1459–71. doi:10.1080/10473289.2006.10464554
   Web of Science ®Google Scholar
• Yu, S., R. Mathur, K. Schere, D. Kang, J. Pleim, and T.L. Otte. 2007. A detailed evaluation of the Eta-CMAQ forecast model performance for O3, its related precursors, and meteorological parameters during the 2004 ICARTT study. J. Geophys. Res. Atmos. 112:D12s14. doi:10.1029/2006JD007715
   Google Scholar
• Yu, S., R. Mathur, J. Pleim, D. Wong, R. Gilliam, K. Alapaty, C. Zhao, and X. Liu. 2014. Aerosol indirect effect on the grid-scale clouds in the two-way coupled WRF–CMAQ: Model description, development, evaluation and regional analysis. Atmos. Chem. Phys. 14:11247–85. doi:10.5194/acp-14-11247-2014. doi:10.5194/acp-14-11247-2014
   Web of Science ®Google Scholar
• Zhang, L.M., M.D. Moran, P.A. Makar, J.R. Brook, and S.L. Gong. 2002. Modelling gaseous dry deposition in AURAMS: A unified regional air-quality modelling system. Atmos. Environ. 36:537–60. doi:10.1016/S1352-2310(01)00447-2
   Web of Science ®Google Scholar
• Zhang, Y., P. Liu, A. Queen, C. Misenis, B. Pun, C. Seigneur, and S.-Y. Wu. 2006. A comprehensive performance evaluation of MM5-CMAQ for the Summer 1999 Southern Oxidants Study episode—Part II: Gas and aerosol predictions. Atmos. Environ. 40:4839–55. doi:10.1016/j.atmosenv.2005.12.048
   Web of Science ®Google Scholar
• Zhang, Y., J.-P. Huang, D.K. Henze, and J.H. Seinfeld. 2007. Role of isoprene in secondary organic aerosol formation on a regional scale. J. Geophys. Res. Atmos. 112:D20207. doi:10.1029/2007JD008675
   Google Scholar
• Zhang, Y., Y. Pan, K. Wang, J.D. Fast, and G.A. Grell. 2010. WRF/Chem-MADRID: Incorporation of an aerosol module into WRF/Chem and its initial application to the TexAQS2000 episode. J. Geophys. Res. Atmos. 115:D18202. doi:10.1029/2009JD013443
   Google Scholar