On the Comparative Assessment of the Performance of Air Quality Models

Abstract

The reliability and accuracy of various atmospheric dispersion models currently being used by regulatory agencies have not been well documented due to a lack of sufficient field data to verify these models, and a lack of well-defined performance measures for evaluation purposes. In this paper, two performance measure approaches—paired and unpaired comparisons—are discussed. These form the basis for assessing model behavior both from a scientific and a regulatory point of view. The tracer gas data base developed in the General Motors sulfate experiment is used to demonstrate the utility of many of these model evaluation approaches.

It is demonstrated that extreme value theory is a powerful technique to assess qualitatively and quantitatively the performance of a model in the upper tail of the concentration cumulative frequency distribution. Preliminary results indicate that any screening of data in model comparisons and evaluations could adversely affect the ability of a model to simulate the processes that produce extreme concentrations. It is suggested that both paired and unpaired comparison techniques be used in order to provide a critical evaluation of the performance of air quality models.