Evaluating the effectiveness of air quality regulations: A review of accountability studies and frameworks

ABSTRACT

Assessments of past environmental policies—termed accountability studies—contribute important information to the decision-making process used to review the efficacy of past policies, and subsequently aid in the development of effective new policies. These studies have used a variety of methods that have achieved varying levels of success at linking improvements in air quality and/or health to regulations. The Health Effects Institute defines the air pollution accountability framework as a chain of events that includes the regulation of interest, air quality, exposure/dose, and health outcomes, and suggests that accountability research should address impacts for each of these linkages. Early accountability studies investigated short-term, local regulatory actions (for example, coal use banned city-wide on a specific date or traffic pattern changes made for Olympic Games). Recent studies assessed regulations implemented over longer time and larger spatial scales. Studies on broader scales require accountability research methods that account for effects of confounding factors that increase over time and space. Improved estimates of appropriate baseline levels (sometimes termed “counterfactual”—the expected state in a scenario without an intervention) that account for confounders and uncertainties at each link in the accountability chain will help estimate causality with greater certainty. In the direct accountability framework, researchers link outcomes with regulations using statistical methods that bypass the link-by-link approach of classical accountability. Direct accountability results and methods complement the classical approach. New studies should take advantage of advanced planning for accountability studies, new data sources (such as satellite measurements), and new statistical methods. Evaluation of new methods and data sources is necessary to improve investigations of long-term regulations, and associated uncertainty should be accounted for at each link to provide a confidence estimate of air quality regulation effectiveness. The final step in any accountability is the comparison of results with the proposed benefits of an air quality policy.

Implications: The field of air pollution accountability continues to grow in importance to a number of stakeholders. Two frameworks, the classical accountability chain and direct accountability, have been used to estimate impacts of regulatory actions, and both require careful attention to confounders and uncertainties. Researchers should continue to develop and evaluate both methods as they investigate current and future air pollution regulations.

Introduction

Air pollution accountability is a growing field that seeks to assess the effectiveness of regulatory actions, with governments and stakeholders that must control their emissions taking particular interest in the outcomes of such studies (Health Effects Institute, 2003). Early work in the field focused on air pollution and health impacts of local regulatory or nonregulatory actions (such as a major local air pollution source shutting down, e.g., due to economic factors). In the last decade, the methods have expanded to include long-term assessments of multiple control programs implemented over large spatial domains.

The accountability chain

The later decades of the 20th century saw major reductions in ambient air pollution concentrations in developed nations as governments enacted regulatory actions to curb emissions and subsequently protect human and environmental health (Hubbell et al., 2009; National Research Council, 2004). These three elements—regulations, emissions, and health—represent the three of the five links in the accountability chain (air quality and exposure are the others; Figure 1, left side), which tracks the effects of an air quality regulation from promulgation to final impacts on public health (Health Effects Institute, 2003). The goal of accountability research is to assess environmental policies by attributing causality (or lack thereof) of changes to links in the accountability chain to a regulatory action, and a successful accountability study should address each link in the chain between the regulatory action and the end point of interest (Health Effects Institute, 2003). At the conclusion of the analysis, study results should be compared with estimates of the proposed benefits of the regulations to air policy makers in decisions regarding future regulations. The majority of studies completed to present define either air pollution concentrations or health outcomes as the end point of interest, whereas a few also assess economic impacts. Numerous studies have been published in this domain since the Health Effects Institute’s (HEI) 2003 report (e.g., Dockery et al., 2013; Kelly et al., 2011; Morgenstern et al., 2012; Peel et al., 2010; Peters et al., 2009; Wong et al., 2012).

Figure 1. Classic accountability chain (left) and direct accountability framework (right). Confounding factors (red portion of the arrows) increasingly obscure the observable signals attributable to regulations (blue portion) between the links in the accountability chain. Further down the chain, the observed response is impacted only partially by regulations. Direct accountability studies seek to directly link regulatory actions to changes in air quality and/or health outcomes. Classic and direct accountability studies strive to take into account as many confounders as practical, although they use different methods to do so. At multiple points in both frameworks, the opportunity exists to compare with each regulation’s proposed benefits and update the approach in new regulatory actions.

Defining causal relationships between any one of the links presents a difficult challenge, and assessing each link in the chain is even more arduous (Bell et al., 2011). Often, studies link changes in the outcome of interest to regulatory actions by making assumptions about the intermediate links in the accountability chain. Most studies identify a regulatory action, then use statistical or deterministic modeling of multiple other links in the chain to associate emissions with air quality and/or air quality with one or more health outcomes (e.g., Clancy et al., 2002; Hou et al., 2010; Kelly et al., 2011; Morgenstern et al., 2012; U.S. Environmental Protection Agency [EPA], 2011, 1999a; Zigler et al., 2012).

The Health Effects Institute (2003) noted that a successful accountability study should address effects of regulatory actions on outcomes at each link in the accountability chain. Confounders at each link cloud the signal, and these factors compound through the length of the chain, increasingly obscuring the signal from the regulation at each link (Figure 1). Further, confounding variables and changes due to a regulatory action may not be wholly independent. For example, electric utilities use a complex decision-making process when determining whether to retrofit or shutter old plants and/or build new ones. Inputs to these decisions include regulations, projected population changes and fuel costs, and geographic constraints, and can include stakeholder review and public oversight (National Research Council, 2004). This multitude of variables that affects these decisions makes it impossible to attribute some changes in emissions wholly to specific regulations.

Given the issues involved in approaching accountability link by link, Zigler and Dominici (2014) suggest that direct accountability (discussed in more detail below) as an attractive, supplementary framework. In short, direct accountability uses statistical methods to determine whether an intervention caused in the outcome of interest (typically changes in air quality or health outcomes), and is less focused on potential impacts of regulations on each link in the accountability chain. Both the classic and direct frameworks are subject to confounding factors (Figure 1). The classic accountability approach takes a more mechanistic approach by establishing relationships links in the chain—between, for example, emissions–air quality relationships and dose-response curves—and applying these relationships to hypothetical counterfactual scenarios. Direct accountability methods approach the problem from a potential-outcomes perspective. These studies define an experiment with an “intervention condition” and a “control condition” such that, assuming a population were randomly assigned between these conditions, the difference in the outcome of interest (usually either air quality or health outcomes) is interpreted as the causal effect of the intervention.

Regulations

The first step in the accountability chain is the adoption of a policy and the ensuing regulations or related interventions. The regulation may or may not call for specific changes in emissions and is, itself, subject to varying levels of success; for example, the degree to which the regulation leads to actions that could result in emission changes. For example, an area being found to be in nonattainment of the National Ambient Air Quality Standards in the United States does not lead directly to a set of specific emission controls—each state that contains a nonattainment area submits a State Implementation Plan (SIP) that details the actions it will take to reduce pollution levels (National Research Council, 2004; Zigler et al., 2016). These actions could include a regulation that specifies legal emission amount and/or controls on a facility. One example of this type of policy in the state of Georgia, the Multipollutant Control Rule, lists specific control technologies for each unit of electric generating units in the Atlanta area, which was in nonattainment for both ozone (O₃) and particulate matter with a diameter less than or equal to 2.5 µm (PM_2.5) when the rule was promulgated in 2008 (Georgia Environmental Protection Division, 2013). Regulations may also lead to or be associated with unintended consequences, such as increased vehicle traffic carrying visitors to the Shanghai World Expo that partially counteracted the effects of temporary restrictions on industrial activity implemented to improve air quality during this event (Hao et al., 2011) and China’s policy of providing coal for heating homes and offices north of the Huai River (Chen et al., 2013).

Emissions

Emissions of air pollutants are affected by multiple factors at a range of time scales from decadal to less than daily, including climate, population growth/decline, changes in the economy (including economic effects of air quality emissions, costs to operate in different areas, and fuel prices), regulatory actions, industry compliance rates, meteorology, weekly demand, and work travel patterns. Because of these factors and the general difficulty and expense of measuring emissions, typically only large sources, such as power plant stacks, are directly measured. Emissions from other sources (e.g., automobiles) are estimated as the sum of multiple smaller or individual sources. It is not yet practical to measure emissions from all of the cars all of the time, so the total fleet emissions are modeled, which introduces uncertainties (EPA, 2011). Uncertainties in data used by those models include emission factors and activities, and studies have identified major biases in emission estimates. For example, in the 1980s, the Southern California Air Quality Study (SCAQS) tunnel studies identified that volatile organic compound (VOC) and carbon monoxide (CO) emissions were much higher than estimated (Lawson, 1990; Pierson et al., 1990), and that a small number of vehicles emitted a high percentage of total mobile emissions (these vehicles were termed “super-emitters”) (Lawson et al., 1990). Current ambient ground- and satellite-based observations studies have found a disagreement of up to a factor of 2 in emission modeling (Anderson et al., 2014; Goldberg et al., 2016; Souri et al., 2016).

The lack of direct measurements makes it challenging to assess how emissions are changing, much less how they are responding to specific regulations. Further, in developed and developing countries alike, emission sources are regulated under a number of statutes implemented at various times and applied over various spatial scales (Health Effects Institute, 2003). Such complexities are likely to increase as interest grows in a multimedia approach to environmental issues (Klausbruckner et al., 2016; Murray, 2013).

Even for sources whose emissions are well characterized, it can be difficult to assess the impacts of regulations. For example, if a regulation would lead to costly controls or plant retrofits, a utility may decide to shutter a plant before they otherwise would “move” emissions, such as by increasing activities in one location and decreasing activities in another, or, in order to avoid New Source Review requirements, keep older plants online (National Research Council, 2004). Other external factors can play a role, such as economic activity that drives reduction in demand during recessions (Chay and Greenstone, 2003) or changing fuel costs, which may or may not be directly linked to regulatory actions (National Research Council, 2004). Quantifying the actual impacts becomes more uncertain as the length of time assessed grows.

Studies have employed various techniques for addressing each of these challenges in order to attribute changes in emissions to specific interventions. For example, estimates of how mobile emissions have changed in response to regulations come from studies on a limited number of vehicles, changes in emission factors and ambient observations (e.g., in tunnels), and emission models (McDonald et al., 2013; National Research Council, 2004). In studies on interventions that occurred over short time periods, for instance, long-term trends in emissions are typically ignored, and baseline emissions are taken as the period before (or after, in the case of interventions that expire) the intervention (Friedman et al., 2001; Hou et al., 2010; Peel et al., 2010). Studies on long-term interventions typically either assess changing emissions over time (e.g., Butler et al., 2011; Gégo et al., 2007; Godowitch et al., 2010) or model an estimate of counterfactual—i.e., what would have happened had there not been an intervention—by assuming a constant emission rate from a baseline year throughout the study period (e.g., Daskalakis et al., 2016; Gégo et al., 2008; Morgenstern et al., 2012).

Air quality

The impacts of regulations and emissions on air quality have been well studied using both empirical (Blanchard et al., 2012; Butler et al., 2011; Harrington et al., 2012; Pierce et al., 2010; Zigler et al., 2012) and nonempirical (i.e., deterministic; e.g., Daskalakis et al., 2016; Simon et al., 2015; Wang et al., 2014) air quality modeling approaches, each of which has strengths and limitations. Empirical air quality models directly use air quality observations that are presumed to have relatively small uncertainties for air quality near the monitor compared with, for example, deterministic models (Friberg et al., 2016). Statistical techniques provide both emission changes—air quality associations and uncertainty estimates. However, statistical linkages to emission changes or the underlying interventions are subject to confounding factors and require a number of modeling assumptions. If not all factors are correctly accounted for and assumptions justified in the approach, the associations will be flawed (Chang et al., 2014a; Harrington et al., 2012; Zigler et al., 2012). Deterministic air quality models include state-of-the-science characterizations of atmospheric processes, but are subject to uncertainties in model inputs and parameterizations, numerical errors, and are often more time- and computationally intensive than empirical approaches (Napelenok et al., 2011; Simon et al., 2012; Wyat Appel et al., 2008).

Two empirically based approaches commonly used for evaluating the impacts of regulations on air quality are before/after or more polluted/less polluted location comparisons and time-series analyses (including source apportionment). Before/after comparisons highlight the change over time, but are particularly susceptible to confounding, as long-term changes in multiple factors (e.g., population, emissions, or climate) can significantly impact results.

Time-series analysis involves relating changes in meteorology and/or emissions to ambient measurements. Methods that link only meteorological variability (not emissions) to ambient concentrations are focused on detrending air pollution in order to assess long-term trends without added interference of meteorology (Camalier et al., 2007; Cox and Chu, 1993; Flaum et al., 1996; Gégo et al., 2007; Henneman et al., 2015; Kuebler et al., 2001; Milanchus et al., 1998; Rao et al., 1995). Other methods seek to directly relate changes in measured concentrations with regulatory actions (Zigler et al., 2012), emission changes (Butler et al., 2011; Harrington et al., 2012), or fuel consumption (Russell et al., 2016). Such methods can be used with counterfactual emission estimates to model potential outcomes that assume no regulations. Source apportionment (the attribution of observed concentrations to specific sources) remains an active area of research (Blanchard et al., 2012; Hopke, 2016; Hu et al., 2014; Yuan et al., 2013). Empirical accountability studies typically define a baseline period of air quality during the time periods before and after an intervention (e.g., Clancy et al., 2002; Friedman et al., 2001; Peters et al., 1996), or, alternatively, in the location of the intervention versus a control region (e.g., Lin et al., 2012; Peel et al., 2010; Wong et al., 1998), and quantify resulting changes. Zigler and Dominici (2014) point out potential issues with the classical approach, including the fact that it relies on extrapolating associative relationships and counterfactual scenarios that cannot be evaluated using measurements.

Deterministic air quality models directly estimate the impacts of emission changes on changes in air quality. Such models make it possible to distinguish the effects of multiple regulations happening simultaneously through the estimation of emission-concentration sensitivities (Napelenok et al., 2011; Simon et al., 2013). These models have been used to estimate counterfactual concentrations (Daskalakis et al., 2016; Gégo et al., 2008; Godowitch et al., 2010; Wang et al., 2014) and are primary tools used to assess proposed regulations (e.g., EPA, 2005a, 2005b, 1997a) and demonstrate future attainment with air quality standards as part of SIPs (National Research Council, 2004).

Exposure/dose

Challenges that arise in estimating exposure (and the subsequent health effects) to air pollution in accountability research are similar to those in other air pollution epidemiological research (Cox and Popken, 2015). Historically, scientists estimate exposure using air quality at a central monitor (or a group of monitors) to approximate exposure for all cases in the sample population, although recent studies have used results from combined multiple models, satellite data, and land use regressions (Chang et al., 2014b; de Hoogh et al., 2014; Lee et al., 2015; Lim et al., 2013; Pachon et al., 2012; Wang et al., 2015). To date, not all of these methods have been tailored to accountability research questions. For daily statistical time-series models, researchers often include multiple lag terms or averaging periods to assess various relationships between health outcomes and measured concentrations. A few studies use population-weighted concentration (Hou et al., 2010) and personal exposure monitoring (Wu et al., 2010). Some studies opt to not assess exposure and instead apply statistical techniques that test for a change in health outcomes rates over the predefined time period of the intervention (Greenstone, 2004; Pegues et al., 2012; Zigler et al., 2016). Specific issues in relating air quality changes to exposures when conducting accountability studies include diagnosing both model uncertainty and parameter uncertainty in model selection (Morgenstern et al., 2012), missing data (Bell et al., 2011; Van Erp et al., 2011), and estimating baseline exposure for the no-control case that accounts for appropriate confounders (Health Effects Institute, 2003).

Health effects

Increased air pollution has been linked to negative health outcomes in numerous studies (Dockery et al., 1993; Garcia et al., 2011; Laden et al., 2006; Lim et al., 2013; S. Lin et al., 2013). Outcomes of greatest interest include morbidity and mortality associated with respiratory symptoms and cardiovascular disease (CVD) (Bell et al., 2011). Acute asthma events, heart rate variability (Wu et al., 2010), birth weight (Rich et al., 2015), and bronchial hyperreactivity (Wong et al., 1998) have been used as well. Most studies relate time periods of regulatory actions or changes in emissions to the health outcome of interest while controlling for environmental or population-related confounders, including meteorology, time trends, location, and age. Researchers then apply these models to a baseline pollution level (either from before/after the intervention of interest or a counterfactual) to estimate the relative risk of the response of interest to the regulatory action. Some extend this to estimate the number of outcomes avoided due the regulation being adopted.

Zigler and Dominici (2014) argued that many studies (particularly those that compare periods before and after a regulatory action) do not account for all of the factors that may impact a changing health response and fail to address the uncertainties in the results relating to such an assumptions. Cox (2013) discussed appropriate statistical methods for determining causality independent of a researcher’s personal biases, such as Granger causality, conditional independence, and counterfactual causality tests.

Scope of this review

This review assimilates work to date in the emerging field of accountability research. The studies are limited to those that assess air quality regulations in terms of at least one of the links in the accountability chain, and the focus is on studies that assess past regulatory action—as opposed to work that projects the effects of future regulations—or related events, such as a recession. A previous review (Bell et al., 2011) discussed publications focused specifically on the health end point. The current review extends their analysis by summarizing work that assesses other steps in the accountability chain, adding studies that have been published since 2011, and discussing future directions and challenges of accountability research.

In the next section, “Accountability Study Approaches,” we present popular approaches taken by previous studies to investigate regulatory impacts on links in the accountability chain. Studies are grouped based on their end point of interest; for example, some studies are interested only in the regulatory impact on air quality, whereas others are interested in the impact on health effects. The following section, “Accountability Case Studies,” reviews specific regulatory actions and the related works. The focus here is on discussing the ability of research methods to control for confounding factors. The next section reviews previous commentaries on accountability studies. In the final two sections, we highlight challenges and future directions in accountability research, and conclude by summarizing the work and commenting on which aspects of accountability research deserve the most attention moving forward.

Accountability study approaches

Most accountability studies choose as an end point one of the links in the accountability chain: emissions, air quality, or health outcomes, although some address each step (Tables 1 and 2). Studies in the list were chosen based on a desire to include work from multiple authors on several major accountability case studies. The list is extensive, but not exhaustive, and the focus is on studies that assess past regulatory actions aimed at improving air quality.

Table 1. Accountability studies with emission and air quality end points.

Author	Year	Policy/event	Geographic scale	Time scale	Emissions	Air quality
Rao and Zurbenko	1994	All regulations, 1983–1991	Single monitor, New York State	Long		Ambient, meteorological detrending
Kuebler et al.	2001	All regulations, 1985–1998	National: Switzerland	Long	Changes in emissions by source over time	Ambient, meteorological detrending
Greenstone	2004	1970 Clean Air Act Amendments	National: U.S.	Long		Ambient, change in SO2 concentrations in designated attainment vs. nonattainment regions
Camalier et al.	2007	All regulations, 1997–2005	Region: Eastern U.S.	Long		Ambient, meteorological detrending
Gégo et al.	2007	NOx SIP Call	Region: Northeastern U.S.	Long	Changes in point-source emissions over time	Ambient, meteorological detrending (including wind direction analysis)
Gégo et al.	2008	NOx SIP Call	Region: Northeastern U.S.	Long	Changes in point-source emissions over time, estimate of counterfactual	Modeled, comparing actual and counterfactual using single meteorological year
Godowitch et al.	2008	NOx SIP Call	Region: Northeastern U.S.	Long	Changes in point-source emissions over time, estimate of counterfactual	Modeled dynamic analysis, comparing change due to emissions vs. meteorology
Ban-Weiss et al.	2008	Mobile regulations, 1997–2006	Local: tunnel	Long	Measured emission factors, before/after comparison
Atkinson et al.	2009	Congestion charging scheme, 2003	City: London	Short		Ambient, before/after comparison
Goodman et al.	2009	Coal sale bans	National: Ireland	Long		Ambient, before/after comparison
W. Wang et al.	2009	Industrial activity reduced for 2008 Olympics in Beijing	City: Beijing	Short		Ambient, before/after comparison, back-trajectory analysis, correlations with meteorology
M. Wang et al.	2009	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short		Ambient, before/after comparison
Wang and Xie	2009	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short	Modeled, before/after comparison	Modeled, before/after comparison
X. Wang et al.	2009	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short	Measured emission factors, before/after comparison	Ambient, before/after comparison
Y. Wang et al.	2009	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short	Modeled, before/after comparison	Modeled, before/after comparison
Kean et al.	2009	Mobile regulations, 1999–2006	Local: tunnel	Long	Measured emissions factors, before/after comparison
2010–2012
Godowitch et al.	2010	Mobile regulations, 2002–2006	Region: Eastern U.S.	Long	Changes in emissions by source over time	Modeled dynamic analysis, comparing change due to emissions vs. meteorology
Dallman and Harley	2010	Mobile regulations, 1996–2006	National: U.S.	Long	Changes in modeled mobile emissions over time
B. Wang et al.	2010	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short	Measured emission factors, before/after comparison	Ambient, before/after comparison, source apportionment
S. Wang	2010	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short	Modeled, before/after comparison	Ambient and modeled, before/after comparison
Lin et al.	2010	All regulations, 2004–2008	Region: Eastern China	Long	Changes over time estimated by satellite	Ambient, change over time (satellite measurements)
Lamsal et al.	2011	All regulations, 2003–2009	Global	Long	Changes over time estimated by satellite
Colette	2011	All regulations, 1998–2007	Continent: Europe	Long	Changes in emissions by source over time	Modeled dynamic analysis, comparing change due to emissions vs. meteorology
Butler et al.	2011	All regulations, 1997–2008	Region: Eastern U.S.	Long	Changes in point-source emissions over time	Ambient, change over time and sensitivity to emissions
Xing et al.	2011	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short	Counterfactual emissions	Modeled dynamic analysis, comparing change due to emissions vs. meteorology
Kelly et al.	2011	Congestion charging scheme introduced in 2003	City: London	Short	Modeled spatial	Ambient and modeled, before/after and control location comparison, source apportionment
Hao et al.	2011	Shanghai World Expo 2010	City: Shanghai	Short		Satellite-measured, before/after comparison
Harrington et al.	2012	1990 CAAA	Region: Eastern U.S.	Long	Counterfactual	Ambient, modeled counterfactual
McDonald et al.	2012	Mobile regulations, 1990–2010	Multicity: U.S.	Long	Changes over time
Shreifels et al.	2012	SO2 reduction goals in the 10th and 11th Five-Year Plans	National: China	Long	Changes over time
Morgentsern et al.	2012	Title IV Phase 2 of the 1990 CAAA	Region: Eastern U.S.	Long	Change over time and counterfactual	Ambient, trends over time and counterfactual
Pegues et al.	2012	1997 8-hr ozone State Implementation Plans	National: U.S.	Long	Modeled, change over time	Ambient, change over time and location
Liu et al.	2012	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short		Ambient and satellite-measured, statistically modeled changes attributable to meteorology vs. emissions
2013–2016
He et al.	2013	All regulations, 1997–2011	Region: Eastern U.S.	Long	Changes in point-source emissions over time	Ambient, change over time
Yuan et al.	2013	All regulations, 1998–2008	City: Hong Kong	Long	Changes over time	Ambient, change over time, source apportionment, and sensitivity to emissions
McDonald et al.	2013	Mobile regulations, 1990–2010	Multicity: U.S.	Long	Changes over time
Kuwayama et al.	2013	Adoption of clean diesel technology at a major shipping port	Local: port	Short	Measured emission factors, before/after comparison	Ambient, before/after comparison, source apportionment
Huang et al.	2013	Shanghai World Expo 2010	City: Shanghai	Short		Ambient, before/after comparison, back-trajectory analysis
Lin et al	2013	Shanghai World Expo 2010	City: Shanghai	Short		Ambient, before/after comparison, back-trajectory analysis
Lurmann et al.	2014	All regulations, 1994–2011	Region: Southern California	Long	Changes over time	Ambient, change over time
Sickles and Shadwick	2014	All regulations, 1990–2009	Region: Eastern U.S.	Long	Changes over time	Ambient, change over time
Wang et al.	2014	SO2 and NOx control policies, 2006–2015	Region: Eastern China	Long	Counterfactual	Modeled counterfactual
Vijayaraghavan et al.	2014	Mobile regulations, 1995–2010	City: Atlanta, GA	Long	Modeled actual	Summertime ozone (detrended)
Harley	2014	California drayage truck regulation	Local: Port of Oakland	Long	Measured emission factors, change over time
Gan et al.	2014	All regulations, 1995–2010	National: U.S.	Long	Changes in point-source emissions over time	Ambient, multiple ground-based and satellite networks, change over time
Gan et al.	2015	All regulations, 1995–2010	National: U.S.	Long	Changes in point-source emissions over time	Modeled, multiple ground-based and satellite networks, change over time
Simon et al.	2015	All regulations, 1998–2013	National: U.S.	Long	Change over time	Ambient, trends over time
Liu et al.	2015	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short		Ambient, neural network analysis and 1-D box model
Russell et al.	2016	Closure of three power plants	City: Pittsburgh, PA	Short	Fuel use used as a proxy	Ambient, before/after comparison, statistically modeled change due to emissions
van der A. et al.	2016	All regulations, 2005–2015	National: China	Long	Changes over time estimated by satellite
Daskalakis et al.	2016	All regulations, 1980–2010	Global	Long	Counterfactual	Modeled counterfactual
Henneman et al.	in review	All regulations, 1993–2013	City: Atlanta, GA	Long	Counterfactual	Ambient, before/after comparison, source apportionment, and counterfactual

Table 2. Accountability studies with health end points.

Author	Year	Policy/event	Geographic scale	Time scale	Emissions	Air quality	Dose/exposure	Health response
Pope	1989	Brief closing of steel mill, 1987–1988	City: Utah Valley	Short	Discussion of local contribution	Ambient, before/after comparison	Monthly mean and lagged mean	Respiratory illness, changes in health response rates before/after
Peters et al.	1996	1990 reduction of sulfur in fuel oil	City: Hong Kong	Short		Ambient, before/after comparison		Household survey, changes in health response rates before/after
EPA Section 812 report	1997b	1970 Clean Air Act Amendments	National: U.S.	Long	Counterfactual	Ambient, change over time and modeled counterfactual	Varied across pollutants	Multiple outcomes, change from observed to counterfactual
Wong et al.	1998	1990 reduction of sulfur in fuel oil	City: Hong Kong	Short		Ambient, before/after and more/less polluted district comparisons		Bronchial hyperreactivity, changes in health response rates before/after and in more/less polluted districts
Friedman et al.	2001	Transportation limits for 1996 Olympics	City: Atlanta	Short	Before/after comparison	Ambient, before/after comparison	Daily, 2- and 3-day lag city average	Asthma events in children, changes in health response rates before/after
Clancy et al.	2002	1990 coal sale ban	City: Dublin	Short		Ambient, before/after comparison		Mortality, changes in health response rates before/after
Hedley et al.	2002	1990 reduction of sulfur in fuel oil	City: Hong Kong	Short		Ambient, before/after and more/less polluted district comparisons		Mortality, changes in health response rates before/after
Chay and Greenstone	2003	1981–1982 recession	National: U.S.	Short		Ambient, before/after comparison	Annual county average	Infant mortality, changes in health response rates before/after
Chay et al.	2003	1970 Clean Air Act Amendments	National: U.S.	Long		Ambient annual average by county	Annual county average	Adult mortality, changes in health response rates before/after
Laden et al.	2006	Reductions in PM2.5, 1979–1998	Multicity: U.S.	Long		Ambient, trends over time	Four annual averaging methods	Mortality, changes in health response rates with changes in concentration
Lee et al.	2007	Transportation limits for 2002 Asian Games	City: Busan, Korea	Short		Ambient, before/after comparison	Daily city average	Asthma events in children, changes in health response rates before/after
Pope et al.	2009	Reductions in PM2.5, 1970s–2000s	National: U.S.	Long		Ambient, trends over time	Annual mean	Mortality, changes in health response rates with changes in concentration
Peters et al.	2009	Rapid industrialization of a city, 1990–2002	City: Erfuhrt, Germany	Long	Change over time	Ambient, trends over time	Daily, single monitor lagged up to 5 days	Risk of death, change over time
2010–2016
Hou et al.	2010	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short	Before/after comparison	Ambient, before/after comparison	Population-weighted exposure	Mortality and morbidity, changes in health response rates before/after
Li et al.	2010	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short		Ambient, before/after comparison	Daily city average	Asthma events, changes in health response rates before/after
Wu et al.	2010	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short		Personal exposure, before/after comparison	Personal exposure monitoring	Association of heart rate variability with PM2.5 exposure
Peel et al.	2010	Transportation limits for 1996 Olympics	City: Atlanta	Short	Traffic counts	Ambient, change over time and location		Relative risk for respiratory hospital visits compared with baseline period
Zigler et al.	2012	1990 CAAA—NAAQS nonattainment designation	Regional: Western U.S.	Long		Ambient, counterfactual 3-yr average	3-yr average county average	Mortality, change from observed to counterfactual
Rich et al.	2012	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short		Ambient, before/after comparison	Daily, single monitor	Multiple health outcomes, changes in health response rates before/after and in various districts
Wong et al.	2012	1990 reduction of sulfur in fuel oil	City: Hong Kong	Short		Ambient, change over time and location	Daily, 1- and 2-day lag city average	Mortality, changes in health response rates before/after
Lin et al.	2013	NOx Budget Trading Program	Regional: New York State	Long		Ambient, before/after and more/less polluted district comparisons	Kriged 3-day moving average	Multiple health outcomes, changes in health response rates before/after and in various districts
Dockery et al.	2013	Coal sale bans	National: Ireland	Long		Ambient, change over time and location		Mortality, changes in health response rates before/after
Chen et al.	2013	Huai River policy	National: China	Long		Ambient, change over time and location	Annual city average (central monitors)	Change in life expectancy between locations
Rich et al.	2015	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short		Ambient, before/after comparison	Daily, single monitor	Change in birth weight associated with improved air quality during specific months in pregnancy
Su et al.	2015	Reduced industrial activity for 2008 Olympics in Beijing	City: Beijing	Short		Ambient, before/after comparison and 72-hr back-trajectory analysis	Daily, single monitor, 0–4 day lag	Cardiovascular disease mortality, changes in health response rates before/after
Gauderman	2015	All regulations, 1994–2011	Multicity, California	Long		Ambient, change over time and location	All regulations, 2005–2015	Lung development associated with increase concentrations
Berhane	2016	All regulations, 1994–2011	Multicity, California	Long		Ambient, change over time and location	All regulations, 2005–2015	Bronchitic symptoms associated with increase concentrations
Zigler et al.	2016	1990 CAAA—various regulations	National: U.S.	Long		Causally modeled		Multiple health outcomes, change from observed to counterfactual

The end points of interest (i.e., emissions, air quality, or health outcomes) listed in Tables 1 and 2 are associated with the depth of analysis at each link in the accountability chain. Columns in Tables 1 and 2 describe the types of analysis at each of the links in the accountability chain for each study. Although health and air quality are the most common end points assessed (indeed, each of the studies that assessed health as an end point made an attempt to characterize the change in air quality associated with the regulatory action of interest), a few studies focused on the effectiveness of regulatory actions at reducing emissions. Implementation time scales investigated by the studies in this review fall into two categories: those implemented over a relatively short-term time frame (i.e., associated with hosting special events, such as the Olympics) and those implemented gradually over a long time scales (on the order of years) (Tables 1 and 2). The studies focused on varied geographical scales, from single cities to global.

The following subsections describe the approach of studies that define each of the three end points, and precede examples of specific case studies.

Emission end point studies

In the United States, multiple studies have assessed the impacts of new regulations and a changing fleet on mobile emissions and have found significant reductions in emissions of multiple pollutants since the 1990s (Ban-Weiss et al., 2008; Dallmann and Harley, 2010; Harley, 2014; Kean et al., 2009; Kuwayama et al., 2013; McDonald et al., 2013, 2012). The National Emissions Inventory (NEI), assembled by the EPA every 3 yr, is one example of an emission end point study (https://www.epa.gov/air-emissions-inventories/national-emissions-inventory). The NEI seeks to quantify emissions from all sources of criteria air pollutants, criteria pollutant precursors, and hazardous air pollutants. Development of the NEI has employed a variety of methods (primarily bottom-up emission inventories) depending on the source to assimilate emission estimates and depends on data from each of the states. An example of bottom-up emission estimates is large point sources, which are required to measure their stack emissions and report these to the EPA (EPA, 2016). A source of confounding in NEI releases over the years is the EPA’s changing methods for quantifying emissions from varying source types. EPA has updated the models it uses for most major emissions sources over the years it has compiled the NEI (EPA, 2015a).

One type of top-down emission estimate is using satellite measurements of column concentrations of air pollutants. Prior to analysis, the raw data from satellites must be processed by models and algorithms that make numerous assumptions, such as the shape of the pollutant’s vertical profile and the relationship between the actual measurement to the pollutant of interest (Streets et al., 2013). Satellite data have the benefits of being available with relatively small time lag than other emission assimilating techniques and covering a wide spatial scale. Limitations include measurement drift, interference from clouds and surfaces with high albedo, and temporal coverage, which depends on each satellite’s orbit. Streets et al. (2013) review multiple studies that have used satellite measurements to track changes in emissions at various spatial and temporal scales (Lamsal et al., 2011; Lin et al., 2010; van der A et al., 2016). Lamsal et al. (2010) compare satellite inventories with bottom-up inventories from different world regions and finds an agreement of 6.0% globally between the two (where 6.0% represents the difference between satellite and bottom-up inventories divided by the bottom-up inventory).

The majority of emission end point accountability studies compare emissions before/during/after an intervention. Studies that assess regulations implemented over many years (e.g., the 1990 Clean Air Act Amendments—CAAA) must account for long-term trends in addition to those impacted by controls, such as trends in fuel use, miles traveled, economic expansions and recessions, etc. Accountability studies focused on events that occur on short time scales (e.g., policies for the Atlanta and Beijing Olympics, which lasted a few weeks) tend to ignore long-term emission trends (Kuwayama et al., 2013; Li et al., 2010). This may be an appropriate assumption for studies of interventions that last only a few weeks, but studies that use long time periods before and after short-term interventions (such as the Dublin coal ban [Clancy et al., 2002; Goodman et al., 2009] and the Hong Kong sulfur ban) should take long-term trends into account.

Other investigators have used measurements at high temporal resolution to estimate changes in emission factors in real-world driving situations (Ban-Weiss et al., 2008; Kean et al., 2009; Kuwayama et al., 2013). Estimates have been made both for fleet averages (Ban-Weiss et al., 2008; Kean et al., 2009) and specific vehicles matched by registration number (Harley, 2014). Alternatively, researchers have applied source apportionment techniques to ambient measurements to estimate emission changes (Huang et al., 2012; Kuwayama et al., 2013; B. Wang et al., 2010). In China, Schreifels et al. (2012) compared sulfur dioxide (SO₂) emission reductions under different periods of environmental policies (Five-Year Plans) and found a 28% increase in emissions during the 10th Five-Year Plan (2001–2005) and a 14% decrease during the 11th Five-Year Plan (2006–2010). The government’s goals were 10% reductions in both of the periods.

For emission estimates made using source-specific emission factors and activity level, it is straightforward to estimate a counterfactual realization of emissions that assumes no added controls under the regulation of emissions and the same activity level, although uncertainties in emission factors and activity levels carry through the calculation. This approach is taken in a number of studies that focus on the air pollution end point (Daskalakis et al., 2016; Gégo et al., 2008; Morgenstern et al., 2012). This approach may, however, ignore concurrent changes or feedback loops, such as changes in fuel prices (e.g., the low natural gas prices seen in the United States in the past decade) and increased/reduced demand due to population shifts related to air quality policies.

Air pollution end point studies

A second group of studies focus on the impacts of emission regulations on air quality. Researchers place as a high priority the comparison of the magnitudes of effects of changing meteorology versus changing emissions on measured or modeled concentrations, and studies with this end point usually assess long-term interventions. Two main approaches are used to address the linkage between emissions and air quality: empirical approaches, including statistical modeling, often with meteorological detrending (Butler et al., 2011; Camalier et al., 2007; Gégo et al., 2007; Henneman et al., in review, 2015; Kuebler et al., 2001; Liu et al., 2012; Rao and Zurbenko, 1994) and chemical transport–modeled dynamic analysis (Colette et al., 2011; Gégo et al., 2008; Godowitch et al., 2010, 2008; Xing et al., 2011). A third approach tests directly whether designations of nonattainment, which required further actions by local governments, has led to the area achieving its goals of improved air quality (Greenstone, 2004; Pegues et al., 2012; Zigler et al., 2016).

Empirical assessments of how the atmosphere responds to controls typically rely on developing statistical relationships of long-term ambient observations with emissions. These relationships, however, are confounded by meteorological variations, particularly in shorter-term studies. Identifying atmospheric responses over shorter periods—e.g., less than multiple years—can be difficult, and over periods of less than 1 yr, meteorological variations dominate (Camalier et al., 2007; Rao and Zurbenko, 1994). Two events and associated accountability studies (described in more detail below) that note the difficulty in accounting for meteorological variability are the Atlanta Olympics Studies (Friedman et al., 2001; Peel et al., 2010) and the Shanghai World Expo (Y. Lin et al., 2013).

Air quality models provide a second, complementary approach to link emissions to air quality changes. These models directly provide emissions–air quality relationships by describing and following emissions, transport, transformation, and fate in the atmosphere. They are, however, subject to uncertainties in inputs and model parameterizations (Napelenok et al., 2011). In dynamic analysis, researchers model two (or more) time periods and assess the ability of the model to capture observed trends across the time span of interest, with a particular interest in controlling for meteorological variability (Colette et al., 2011; Gégo et al., 2008; Godowitch et al., 2010, 2008; Kang et al., 2013; Napelenok et al., 2011; Pierce et al., 2010; Xing et al., 2011). Some studies then repeat the modeling over the same domain with switched meteorological and emission fields (Foley et al., 2015a, 2015b; Godowitch et al., 2008). The differences in modeled concentrations between model runs with the same meteorological inputs and different emissions represent the change attributable to emissions, and the differences between runs with different meteorology and the same emissions are attributable to meteorological changes.

Other air pollution studies seek to link changes in air quality with changes in emissions using statistical modeling techniques. These range from linear models (Harrington et al., 2012; Henneman et al., in review) to principal components analysis (PCA)-related and other source apportionment techniques (Balachandran et al., 2012; Blanchard et al., 2012; Hopke, 2016). Each source apportionment technique carries with it uncertainties, and there have been recent efforts in this domain to combine results from multiple methods (Balachandran et al., 2012; Hopke, 2016).

In general, studies focused on estimating the change in air pollution due to emission changes exert more effort explicitly accounting for differing meteorological conditions, whereas those focused on health do not when assessing changing air quality, although most health studies do account for meteorology in their health models. By more fully accounting for regional meteorological patterns around the time of the 1996 Olympics in Atlanta, Peel et al. (2010) were unable to associate improved air quality with emission reductions, a link that was found earlier by Friedman et al. (2001). The approach used here (i.e., performing similar analyses on the location of interest and control areas not impacted by controls) has been used in a number of accountability studies (e.g., Chen et al., 2013; Kelly et al., 2011; Peel et al., 2010).

Health end point studies

Health outcome studies attempt to answer the question “what change in outcome response (if any) can be associated with the intervention of interest?” Such studies begin with an assessment of the intervention, which usually involves a comparison of air quality and health outcomes before and after the intervention using descriptive statistics (e.g., Dockery et al., 2013; Peters et al., 1996). Some studies couple this with a discussion of the meteorological conditions before, during, and after the intervention. Most employ a regression analysis to link changes in a measure of public health (e.g., morbidity or mortality) to changes in air quality, either by estimating outcome rates before/after an intervention or using direct associations of the outcome with a measure of air quality. Studies that seek to estimate the direct response of the health outcome to air pollution include a measure of the exposure of air pollution (e.g., annual average, multiple day lag, etc.) (e.g., Laden et al., 2006; Lee et al., 2007; Peters et al., 2009; Pope et al., 2009; Wu et al., 2010). Other studies do not estimate air pollution exposure using, for instance, a regression linking concentration metrics with the health outcome of interest. Instead, these studies—termed direct by Zigler and Dominci (2014)—either use factor variables to address changing conditions before and after the intervention or train different models for different time periods (e.g., Clancy et al., 2002; Peel et al., 2010). Regression analyses include many independent variables to reduce effects from confounding factors (Figure 1) that are chosen based on the data available, researchers’ initial understanding of the problem, and the model selection process, which usually involves removing covariates with high standard errors one by one (Pope et al., 2012). Results are reported in the form of change in health response rates, excess risk from air pollution attributable to the intervention, or health response attributable directly to the intervention.

Many accountability studies that assess health impacts do not fully address the relationships between regulations, emissions, air quality, and dose, a recommendation made by the HEI (Health Effects Institute, 2003). Often (particularly in short-term studies), the baseline scenario is taken as the time before the intervention, an assumption that fails to address changes in confounding variables that may occur simultaneously with the intervention (Zigler and Dominici, 2014). In long-term studies, confounders that impact health and are correlated with changing air pollution concentrations (such as access to health care) make finding statistical associations between life expectancy and air pollution policies difficult (Wong et al., 2012). Further, health studies generally use meteorological metrics to control health response models, but do not assess air pollution changes while controlling for meteorology in locations surrounding the study area. Many studies of short-term interventions do not investigate concurrent changes in emissions from sources not related to the intervention of interest.

Multiple studies in the accountability field have found it difficult to attribute significant improvements in air quality or public health attributable to air quality regulations (Cox and Popken, 2015; Dockery et al., 2013; Kelly et al., 2011; Peel et al., 2010; Wong et al., 2012; Zigler et al., 2016). This difficulty, particularly prevalent in studies that diligently control for multiple confounders across domains (location, time, etc.), highlights the challenge of identifying a signal through either the accountability chain or direct frameworks with enough strength to attribute causality.

Recent research has discussed the issues with confounding in statistical models (e.g., from factors listed in Figure 1) and scientist biases that arise when linking health outcomes with air pollution concentrations and estimating a counterfactual (Pope et al., 2012; Cox and Popken, 2015). In a series of publications, Zigler and colleagues discussed the issues surrounding the ability to attribute causality of improvements in air quality or health outcomes to regulatory actions (Zigler and Dominici, 2016, 2014; Zigler et al., 2016, 2012). Zigler and Dominici (2016) argue that the typical approach, which uses models that associate observed concentrations with health outcomes, is appropriate for the question “What is the relationship between exposure to pollution and health outcomes?” They state that this question provides valuable evidence for predicting the impacts of policies indirectly, but posit a different question, “What is the relationship between a specific regulation and health?” to better serve the goal of accountability research. They suggest that newer methods addressing this question should complement older methods, since both are subject to similar sources of confounding.

Zigler and Dominici (2016) and Cox and Popken (2015) apply more advanced statistical tests and methods that go further to assess the causality (instead of the correlation) of air pollution outcomes. Cox and Popken (2015) suggest using such methods as conditional independence and Granger causality tests, which are statistical methods that better account for alternative explanations for correlations between, for example, air pollution and health. Zigler and Dominici (2016) use two statistical methods—principal stratification and causal mediation analysis—to test for causal linkages between long-term air pollution interventions, air quality, and public health.

Comparison with pre-regulation impact assessments

The EPA publishes detailed analyses of the anticipated benefits of a regulation before it is promulgated in regulatory impact assessments (RIAs) (e.g., EPA, 1997a, 1999b, 2005a, 2005b). For air quality regulations, RIAs estimate effects of the regulation on emissions, air quality, and human/ecosystem health. Further, the EPA estimates an expected cost of the regulation based on the controls or other changes needed to meet emission targets.

The 2003 HEI report recommends using outcomes assessed in RIAs as starting points for accountability assessments in order to evaluate the conclusions of the RIA. Harrington et al. (2012) compared the results of their accountability assessment of the Clean Air Interstate Rule (CAIR) with projected impacts reported in the RIA from the same rule (EPA, 2005b). They note that direct comparison is difficult due to differences in the methods used and context of the two studies. The authors compare both the final results (changes in PM_2.5 concentrations in the eastern United States attributable to the 1990 Clean Air Act Amendments) and the sensitivities of PM_2.5 to utility emissions. They find that their post-intervention estimates of air quality changes generally align. Given the current trend of assessing the accountability of longer-term regulatory actions (many of which were accompanied by an RIA), the comparison of accountability analyses with impact estimates prior to the regulation is an important step to assess and improve the regulatory process.

Accountability case studies

This section reviews policy and intervention scenarios that have attracted interest in accountability research. Many of these have been investigated multiple times, some with updated methods each time and some with updated data (e.g., emissions, air pollution measurements, or health outcome data). The interventions are discussed chronologically.

1986–1987 Utah Valley steel mill closing

The Geneva steel mill, a major source of air pollution in Utah Valley, closed for a period of 1 yr spanning 1986–1987 due to a labor dispute (Pope, 1989). Although the closing was not due to a regulation, the analysis conducted on the resulting health impacts is of interest in accountability studies, as similar methods may be used to assess potential health impacts resulting from regulations leading to emission changes. Pope related ambient measurements of PM with a diameter less than or equal to 10 µm (PM₁₀) and respiratory-related hospital admission using a multiple regression model. They used hospital admissions outside the county as a control, and further controlled for temperature cofounders. The researchers observed nearly doubled PM₁₀ concentrations in the valley when the mill was open compared with when it was closed, and found associations between multiple health outcomes and PM₁₀ levels across the time periods before, during, and after the intervention. In later studies, researchers linked PM₁₀ levels to elementary school absences (Ransom and Pope, 1992), daily mortality (Pope et al., 1992), respiratory hospital admissions (Pope, 1991), and preterm births (Parker et al., 2008).

A goal of Pope (1989) study was to determine whether negative health outcomes could be associated with ambient air pollution concentrations over a period of such a large change. The author discussed other sources of emissions in the valley (noting that they were small in comparison with the mill), compared air quality before, during, and after the intervention, used different estimates of dose in the health outcome regression models, and controlled for various confounders. In total, results from 18 models with various combinations of health outcomes and confounders were reported.

The results of this effort laid groundwork for future accountability studies of short-term interventions. Pope (1989) showed that the association of PM₁₀ with various health outcomes was robust to a number of different models across a period of changing emissions, and used similar regressions with health outcomes at nearby hospitals to control for coincidental decreases in influenza when the plant was closed. Later work focused on elementary school absences controlled more thoroughly for location differences by performing the analyses on schools in both more and less polluted districts (Ransom and Pope, 1992). Pope (1989) noted that PM₁₀ may serve as a surrogate measure for other pollutants, but did not find evidence to suggest that PM₁₀ was serving as a surrogate for ozone.

1990 Dublin coal ban

In response to poor air quality in Dublin attributed to the use of coal for heating during the 1980s, the government banned coal sales in the city beginning on 1 September 1990. The subsequent improvement in air quality provided an opportunity to assess the impacts of the ban on public health outcomes. Clancy et al. (2002) compared seasonal black smoke (BS), sulfur dioxide (SO₂), temperature, relative humidity, and nontrauma death rates for 6 yr before the intervention and 6 yr after. Death rates were estimated and adjusted in order to account for other changes happening in Dublin parallel with the ban, including meteorology, changing age distribution, and respiratory disease epidemics (influenza). Other changes that could not be captured were accounted for by adjusting death rates by age-standardized death rates for the rest of Ireland, excluding Dublin. The authors found that BS concentrations decreased 70% after the ban and SO₂ levels were reduced 34%. The ban coincided with a reduction in respiratory and cardiovascular-related death rates similar in magnitude to those found by Pope (1989). They concluded that, even given decreasing death rates throughout Ireland, the coal ban led to substantial improvements in air quality and subsequent reductions in negative health outcomes.

Dockery and colleagues revisited the Dublin coal ban in later studies, along with similar bans in other cities in Ireland that occurred in 1995, 1998, and 2000 (Dockery et al., 2013; Goodman et al., 2009). In their reassessment, which included a detailed sensitivity analysis and a more comprehensive approach for correcting for long-term background trends unassociated with each ban (i.e., by including control locations presumably unaffected by the bans) and other secular effects, they found only a decrease in respiratory mortality rates in Dublin after the 1990 ban, and no significant reductions at other locations after subsequent bans. The interrupted time-series study design used here, in which the health model included a binary variable that indicated controlled versus not controlled periods, is a direct accountability technique linking regulations to health without quantifying the impact of regulations on air quality (the direct accountability framework is discussed in more detail below). The lack of association in the follow-up analysis was linked to better accounting for background health trends in other parts of the country unaffected by the ban (Dockery et al., 2013).

One drawback of the Dockery et al. (2013) study was the lack of air quality measurement data in the control cities (i.e., those that did not implement a coal ban). The authors could draw conclusions on changes in air quality in districts that had the ban using descriptive statistics before and after the intervention, but could not draw conclusions on how changes in these areas were confounded by similar changes seen throughout the country. Taken together, the two studies show how confounders—concurrent changes in societal factors in this case—can obscure results obtained in accountability research. In the later study, more data on both spatial and temporal scales increased the number of viable analyses available to researchers and increased the ability to draw sound conclusions regarding the effectiveness of control programs.

1990 Hong Kong sulfur ban

In 1990, Hong Kong began requiring fuel oil (used for power generation and transportation) sold in the city to be 0.5% by weight sulfur or less. Multiple studies have assessed the impacts of the reduction on air pollution and public health coinciding with the ban (Hedley et al., 2002; Liu et al., 2015; Peters et al., 1996; W. Wang et al., 2009; Wong et al., 2012, 1998). Peters et al. (1996) compared different areas throughout Hong Kong and found that after the ban, more polluted areas experienced large reductions in SO₂ and sulfate concentrations, and subsequent associations with reduced respiratory symptoms in children were found. Peters et al. used multivariate logistic regression to estimate risk of respiratory problems related to a number of factors, including residence district, school, factors representing pre- and post-intervention, and multiple confounding variables. In this study and those that followed, researchers used results from surveys of children in primary school (Wong et al., 2012, 1998).

Hedley et al. (2002) assessed the change in mortality rates associated with the ban using ambient concentrations and death records from all causes. The researchers used a Poisson regression model that accounted for seasonal fluctuations, temperature, and relative humidity changes to estimate the relative change in average annual numbers of deaths in the 5-yr periods before and after the intervention. They found a reduction in seasonal deaths in the first 12 months after the intervention and a return to the expected death rate from the third year to the fifth. The authors concluded that the coal ban did have an effect on air quality and public health outcomes the first year, and the subsequent return to the cyclical pre-intervention death rate was attributed to a delayed death phenomenon—i.e., later deaths were a result of deaths that did not occur the previous year because of air quality improvements. They discussed how secular changes in death rates and other factors may have had an effect on death rates, but did not adjust the results using a control population.

In the most recent follow-up, Wong et al. (2012) used long-term—5 yr on either side of the ban—records of air pollution and mortality to investigate the effect of the intervention on life span. The study met difficulties, including the inability to separate health effects of different pollutants (nickel, vanadium, and SO₂), and large uncertainties caused by high spatial and temporal variability through the study period. In their review of the project, the HEI concluded that the inability to control for potential confounders that correlated with changes in air quality means that there is little hope for estimating changes in life expectancy attributable to changing air pollution—an effect typically estimated using cohort studies—using daily time-series model (Wong et al., 2012; HEI Commentary, 2012).

The varied results in the Hong Kong studies show the difficulty in estimating impacts of regulatory actions, even when large amounts of data are available. An important aspect of each of these studies was the comparison between more and less polluted districts, which showed that more polluted districts experienced larger improvements than previously less polluted districts. These areas, however, are not perfect controls, since they have many differences other than their pollution levels that could impact health outcomes (such as access to health care) (Wong et al., 2012; HEI Commentary, 2012).

1996 Atlanta Olympics

The 1996 Summer Olympics were held in Atlanta, Georgia, during which a local road traffic congestion management strategy was implemented for 17 days that included banning private automobile traffic in the downtown area, new options for public transportation, and alternative work hours for businesses. Friedman and colleagues (2001) compared air quality (multiple pollutants), childhood asthma and nonasthma acute events, meteorology, and vehicular traffic for the period during the Olympics to the 4-week periods prior and post. They found significant decreases in traffic counts along with improvements in both air quality and public health over the period of the Games, but did note that the cause of the change in air quality was likely linked to both decreased emissions and favorable meteorological conditions. They concluded that, even with the influence of ideal meteorological conditions, the policies related to the Games had an impact on air quality and public health in Atlanta.

Peel et al. (2010) revisited the same intervention using an expanded data set of emergency department visits for 1995 through 2004, air quality and meteorological data from multiple monitoring sites in Atlanta and the surrounding counties, and traffic counts at 18 sites within the five counties of interest. Similar to Friedman et al. (2001), Peel et al. (2010) reported reduced concentrations of ozone and other air pollutants in Atlanta during the Games, but further concluded that, in light of lower ozone levels across the southeastern United States and meteorological conditions unfavorable to ozone production across the region, improved air quality during the Games was due to atmospheric conditions and not a change in emissions. They did not find significant reductions in respiratory or cardiovascular emergency department visits in adults or children during the Games after adjusting for seasonal trends and other confounders, but the results were limited by the short time period of interest and low numbers of daily emergency department visits.

Peel et al.’s (2010) study was limited by the intervention’s short duration, low numbers of emergency department visits, lack of control areas for the health analysis, and a difficulty in isolating the impact of the control from typical temporal patterns in health effects (Peel et al., 2010; HEI Commentary, 2010). The Atlanta Olympics studies provide examples of accountability studies that investigate an intervention from the perspectives of both the classic accountability paradigm and using the direct approach (i.e., using statistical methods that link the control period directly to the health outcome of interest, instead of linking health outcomes to ambient pollution levels).

2008 Beijing Olympics

As a condition for hosting the 2008 Olympics, the Chinese government agreed to reduce emissions in order to improve the air quality for both the Games and the ensuing Paralympic Games (Rich et al., 2012). This chain of events led to a number of studies assessing the effectiveness of the policies at reducing air pollution in the city and any associated health benefits (Hou et al., 2010; Li et al., 2010; Liu et al., 2015, 2012, Rich et al., 2015, 2012; Su et al., 2015; B. Wang et al., 2010; M. Wang et al., 2009; S. Wang et al., 2010; Wang and Xie, 2009; W. Wang et al., 2009; X. Wang et al., 2009; Y. Wang et al., 2009; Wu et al., 2010; Xing et al., 2011).

An advantage to the planned emission reductions in Beijing was that researchers had the opportunity to plan air quality observational studies, recruit test subjects beforehand, and test health outcomes during the intervention period—a major shift compared with the historical approach to accountability research of natural experiments designed after an intervention. Researchers planned ambient measurement campaigns (B. Wang et al., 2010; M. Wang et al., 2009; X. Wang et al., 2009) and used measurements of personal exposure (Wu et al., 2010) and health markers (Rich et al., 2012) to assess impacts of emission reduction. Using data on birth weights in the Beijing area, Rich et al. (2015) found that babies in their eighth month of gestation during the Olympics were statistically significantly larger at birth than babies born at the same term the year before and after. Su et al. (2015) related cardiovascular death counts to ambient pollution measurements while accounting for meteorology and air mass origin and found a positive association. Hou et al. (2010) extended the analysis to an economic impact estimated based on value of statistical life and average outpatient costs in Beijing. All of the studies listed above found positive associations between reduced health outcomes and the time period of emission reductions.

Xing et al. (2011) used a chemical transport model to estimate gas and particulate matter sensitivities to emissions and meteorology, and concluded that emission controls implemented during the period reduced the sensitivity of ozone chemistry to VOC emissions in Beijing. They noted the limitations of assuming a linear system in their modeling, but asserted that the results show how reduced emission controls have important effects on atmospheric conditions even in time periods of meteorology favorable to reduced pollution.

Although most studies concluded that emission controls led to improved air quality during the Olympics, Wang et al. (2009) used 4-day air plume back trajectories regressed against PM concentrations to conclude that 40% of the variability in ambient measurements was attributable to meteorology (compared with 16% attributable to emissions). The authors observed statistically significant changes in PM concentrations during the Games compared with before and the Olympic Games, but did not find statistically significant differences between source control and non–source control periods (controls were imposed a few weeks before the Games began). Wang et al. (2009) determined that wind direction from the cleaner territory to the northeast of the city during the Games created favorable conditions that reduced PM concentrations below what they would have been with just the controls.

The Beijing Olympics studies show the expanded types of studies available to scientists who can plan in advance to implement accountability studies. These include special ambient air pollution, personal exposure, and health marker measurement campaigns. Future accountability studies should take advantage of planned rules and plan ahead in a similar fashion, but should extend the planning to measurements of confounding factors (e.g., smoking habits, accessibility of care, etc.).

London congestion charging scheme

On 17 February, 2003, London introduced a congestion charging scheme (CCS) in the center of the city, along with other regional traffic changes in an attempt to reduce traffic within the city during work hours on weekdays (Atkinson et al., 2009; Kelly et al., 2011). This intervention is an example of a policy not specifically designed to improve air quality, but one that may have potential benefits. Atkinson et al. (2009) used ambient concentrations measured at more than 40 monitoring cites in the greater London area to assess impacts of the CCS on air quality in the city. Using comparisons with geometric mean background concentrations from monitors outside the city center, they were able to show evidence for decreased nitrogen oxides (NO_x), O₃, PM₁₀, and CO within the CCS zone; however, given the concurrent traffic changes, the authors were not able to fully attribute causality to the CCS. Later, the same group applied more detailed statistical analyses, including use of estimates of emission changes attributable to the CCS, and again found small improvements in air quality within the CCS zone (Kelly et al., 2011). Further modeling using estimated emissions provided evidence that the changes were attributable to regional air quality regulations, such as the requirement for buses to install particulate filters.

Akin to the CCS, 12 countries across Europe have implemented low emission zones (LEZs), which restrict vehicles based on the emission standards they were designed to meet. Studies to date that investigate the effectiveness of LEZ policies, reviewed by Holman et al. (2015), tend to be similar in approach to the work investigating the London CCS and have produced similarly mixed results.

These studies show the difficulty of quantifying the impacts of a single local regulatory action in the context of broader regional changes, variable meteorology, congruent regulations, and unanticipated institutional or behavioral changes that may be associated with a regulation. The results highlight the need for detailed estimates of uncertainties in measurements and modeling for each link in the accountability chain—the small improvements in air quality attributed to the CCS policy must be viewed with caution due to limited knowledge of the associated uncertainty (Kelly et al., 2011; HEI Commentary, 2011).

2010 World Expo in Shanghai

In 2010, Shanghai hosted the 6-month-long (May–October) World Expo and implemented a number of emission control measures to improve air quality (Hao et al., 2011; Huang et al., 2013, 2012; Y. Lin et al., 2013). The government imposed short-term limits on energy production, industries such as coking and cement-making, transportation, construction, and agricultural burning in the region.

Hao et al. (2011) compared satellite measurements of nitrogen dioxide (NO₂), aerosol optical thickness (AOT), and carbon monoxide (CO) between the same period over the three previous years and 6 months after the Expo. They found reductions in NO₂, AOT, and CO of 8%, 14%, and 12% compared with before the Expo, and subsequent increases in NO₂ and AOT of 20% and 23% after the Expo. Huang et al. (2013) used highly time-resolved measurements of PM constituents to assess ambient pollution concentration trends during the Expo. They used back-trajectory analysis to identify important sources and regions (similar to the Beijing studies, they determined the Central Plains to be a major source). Assessment of the constituent makeup led to the conclusion that controls on stationary sources led to the reduction in sulfate aerosols over the period of the Expo, but found significant increases in nitrate aerosol, which the authors attributed to increased traffic from visitors to the region during the Expo. This study discusses meteorological conditions, but does not adjust for meteorological fluctuations in their analysis.

Y. Lin et al. (2013) used measurements from 53 measurement sites in the Yangtze River Delta region that includes Shanghai and found that air quality was worse in May, June, and October, and that it was improved in the intermediary months. The researchers concluded that the cleaner months aligned with wind patterns from the sea, which led to cleaner air near the coast and deteriorated air quality inland. They attributed lower SO₂ and PM₁₀ concentrations during the Expo to limits imposed on power plants, industry, and construction, and higher NO_x, CO, and O₃ to increased vehicular traffic associated with visitors to the region for the Expo.

The study period for the Shanghai World Expo had the benefit of being longer (6 months) than other short-term intervention studies (e.g., the Atlanta or Beijing Olympics). However, Lin et al.’s investigation into wind patterns led to the conclusion that meteorological variability and regional transport played an important role in the changes in air quality during this period. All three studies discussed here noted the role of an unanticipated outcome of an intervention—i.e., increased traffic due to visitors to the region. Their use of satellite and monitoring data, statistical descriptions, and back-trajectory analysis provide an example of a multiplatform approach to accountability research, which should be employed in further studies as available and applicable.

Port of Oakland diesel trucks retrofit

Shipping ports are a major source of PM air pollution in coastal areas (Harley, 2014; Kuwayama et al., 2013). The State of California instituted controls on multiple sources associated with activities at ports, one of which was the Comprehensive Truck Management Plan, which began in October 2009. The program required that trucks meet emission standards (requiring the use of a diesel particulate filter, or DPF) and register with the port database by April 2010. Kuwayama et al. (2013) applied positive matrix factorization (PMF), a source apportionment technique, to PM measurements near the port for weekdays between 8 March 2010 and 28 March 2010. They used the entire time period to train the PMF model, but only used a subset of days selected for wind speed and direction coming from the port in the analysis. The authors found that contributions of port truck traffic to PM components decreased by amounts ranging from 66% to 87%.

Harley (2014) measured concentrations at high time resolution at a major access road to the same port in 2011 and 2013. Emission factors of passing trucks were estimated from the measurement data and related to the passing vehicle using the registration database, which included information on model year and DPF retrofit status. The authors reported 99% coverage of DPF technologies, and reductions in fleet-averaged emission factors of 76 ± 22% and 53 ± 8% of black carbon and NO_x, respectively, over the period 2009–2013, which align with prior estimates by Kuwayama et al. (2013).

These studies show how before and after measurements can be made directly of emissions related to interventions, again exemplifying a benefit of planning an accountability study before a regulation goes in effect.

China Huai River policy

Chen et al. (2013) performed a study that addresses a markedly different type of regulation, namely, one that had the effect of deteriorating air quality. Under the Huai River policy, the Chinese government provided coal for heating to residents north of the Huai River, where air quality is especially poor. Chen et al. (2013) used two models: first, a conventional model with mortality rates in 90 cities across China as the response and total suspended particles (TSP) and other city-specific observable characteristics that may impact health as covariates, and, second, a two-stage least squares system of equation that leverages a regression discontinuity (RD) approach to test a step function change of air quality and TSP effects on health across the Huai River. The conventional approach yielded the result that a 100 µg m⁻³ increase in TSP was associated with a 0.52- or 0.54-yr decrease in life expectancy for models without and with climate and demographic controls, respectively. The RD approach tests for discontinuity at the Huai River line; for example, the test indicates that the changes in SO₂, NO_x, and expected life expectancy across the line are not statistically significant, but that the changes in TSP and actual life expectancy (5.04 yr, 95% confidence interval [CI]: −8.81, −1.27 yr) across the line are statistically significant. These results suggest a decreased life expectancy of 3.0 (95% CI: 0.4, 5.6) yr for a 100 µg m⁻³ increase in TSP.

This study was able to leverage a policy that affected two populations (i.e., north and south of the Huai River) simultaneously, and the authors designed their study to directly compare the affected population (to the north) with the control (to the south). The authors note that multiple confounding factors are not captured by the model, for instance, if the policy caused altered behavior changes (such as spending more time indoors) or different spending habits (such as on medical care or tobacco) because of the policy’s potential impact on disposal income. Chen et al. (2013) found that the results were robust to numerous sensitivity analyses to test for differences in populations north and south of the river line.

The Chen et al. (2013) study is well thought out and addresses one of the major issues with a number of previous accountability studies, namely, the lack of a suitable control population. Although the RD method design is not applicable to all accountability studies (neighboring control populations are a large part of the study’s success), it offers a number of benefits and should continue to be applied and developed in the accountability field.

Regulations under the 1990 United States Clean Air Act Amendments

Actions taken under the 1990 CAAA presented researchers with an opportunity to extend accountability analysis from the short-term actions described above to actions taken incrementally over many years. Further, nationwide ambient air quality networks made it possible to assess impacts over broad spatial scales. Between 1997 and 2011, estimated nationwide (United States) NO_x and CO emissions dropped by about 50% and 60%, respectively (He et al., 2013), and 95th percentile summertime ozone decreased over the period 1998–2013 by 1–2 ppb yr⁻¹ (Simon et al., 2015).

Given the long time period and economic cost of the intervention, there have been many studies that focus on the emission and air quality end points (Camalier et al., 2007; Gégo et al., 2008, 2007, Godowitch et al., 2010, 2008; Harrington et al., 2012; He et al., 2013; Lurmann et al., 2014; Sickles and Shadwick, 2014; EPA, 2011). The lengthy data record of both air quality measurements and emission estimates allows these researchers to use meteorological detrending and other statistical methods to assess the relative impacts of meteorology and emission changes. Results agree across studies that the regulations have reduced air pollution concentrations.

The EPA undertook a detailed assessment of benefits and costs of the Clean Air Act as part of the requirements under Section 812 of the 1990 CAAA (EPA, 2011). They used modeled actual and estimated counterfactual emissions to drive a chemical transport model for two scenarios: “with-CAAA” and “without-CAAA,” and estimated changes in concentrations, mortality, environmental impacts, costs, and benefits between the two. Estimated direct costs, most of which stemmed from control costs, totaled $53 billion (in 2006 dollars) in 2010. The research used results of cohort air pollution studies and expert elicitations to decide on appropriate concentration-response functions for ozone and PM, and used population-weighted concentrations to estimate dose. These values were used with concentration-response functions and estimates of willingness to pay to estimate cases of “excess premature mortality” per year and the sum of “dollars per mortality avoided” (similar to “value of a statistical life”—VSL). Estimated monetized benefits totaled $1.3 trillion (in 2006 dollars) in 2010. A detailed sensitivity uncertainty analysis to model inputs yielded variability in the results, but it was small compared with the difference between the estimates of total benefits and costs.

Whereas most studies assess trends in emissions and air quality, a body of research uses a potential-outcomes approach that estimates outcomes at links in the accountability chain for developing counterfactual emission scenarios (Daskalakis et al., 2016; Harrington et al., 2012; Henneman et al., in review; EPA, 2011; Wang et al., 2014; Zigler et al., 2012). Morgenstern et al. (2012), for example, developed a statistical model that linked monthly PM_2.5 concentrations with power plant SO₂ emissions and then used the model to estimate counterfactual concentrations. Henneman et al. (in review) extends this analysis to daily PM_2.5 and ozone. Daily results are important in this context for future use in acute health analyses.

Results for health studies are more varied (Cox and Popken, 2015; S. Lin et al., 2013; Pope et al., 2012; EPA, 1997b; Zigler et al., 2012). These studies have noted the difficulty in assessing long-term implementations of regulations due to the inability of statistical models to capture other changing factors that occur, such as shifting baseline health of a population or availability of care, statistical manipulation, or overinterpretation of subgroup analysis (i.e., making definitive conclusions based on models that group health observational data sets by gender, age, geographic areas, etc. [Pope et al., 2012]). They employed alternative methods such as Granger causality tests (Cox and Popken, 2015) and principal stratification (Zigler et al., 2012).

Zigler et al. (2012) sought to attribute causality of improved public health to nonattainment designation for PM₁₀ of counties in the western United States under the 1990 CAAA. They used a statistical method called principal stratification to distinguish causal effects of a regulation on health that are associated with causal effects of the regulation on air quality versus those that are not. Their analysis covered 7 million Medicare enrollees in the western United States and finds that the effect of the nonattainment designation programs was 1.76 fewer deaths per 1000 Medicare beneficiaries.

Zigler et al. (2016) extended their previous efforts by assessing the designation of National Ambient Air Quality Standards (NAAQS) nonattainment on air pollution in various locations using both principal stratification and causal mediation, a similar method that separates effects of an intervention into the “natural direct” and “natural indirect” effects. The authors employed a principal stratification method that matches areas based on their propensity scores—a statistical score that combines multiple confounders into one number—and uses locations not designated as nonattainment areas for the control. Their analysis shows that there is some interference between study and control locations. A limitation of this method is that, in order to compare multiple locations at once, propensity scores are developed across many varying conditions. First, for locations that are in nonattainment that have no matching areas that are in attainment (based on propensity scores), these locations cannot be used. Second, assigning a single score to a region based on a range of factors may dull the impact of one important factor. One important factor may influence air pollution or health outcomes more than many similar attributes. A second limitation is the assumption that there is no unmeasured confounding, meaning that all factors that went into the estimation of propensity scores represent all factors that could confound comparisons.

The California Children Studies investigated impacts of long-term changes in air quality on children health. Gauderman et al. (2015) and Berhane et al. (2016) used surveys from three cohorts of children aged 5–18 between 1993 and 2012 in eight communities in California to assess health impacts of air quality changes associated with the 1990 CAAA. Gauderman et al. (2015) used a linear spline model to estimate impacts of changing air quality on measures of lung development in children aged 11–15. The models adjusted for multiple confounding variables and found statistically significant improvements in lung function attributable to decreased NO₂, PM_2.5, and PM₁₀ concentrations, but the authors note high correlation between these three pollutants. Berhane et al. (2016) used a multilevel logistic model to estimate effects of multiple annual air pollution concentrations. Their model included many covariates to account for multiple potential confounders, including random effects, secondhand smoke, season or month of data collection, and age. From their results, they conclude that decreases in ambient pollution concentrations are associated with decreased bronchitic symptoms in children that are statistically significant. They concede that their results cannot be interpreted as attributing causality, but that they support benefits of air pollution reductions.

Two recent studies address accountability of the CAA on aerosol loading over the United States and relate its potential to impact climate (Gan et al., 2015, 2014). Gan et al. (2014) assessed multiple surface and satellite measurement data sets to assess changes in short-wave radiation and noted a trend of increasing cloud brightness. These studies are notable both for their nationwide coverage and long time period of coverage. Although their analysis is primarily concerned with trends, the authors include detailed discussion of physical and chemical reasons for the observations.

The United States experienced widespread reductions in emissions and subsequent improvements in air quality between when the Amendments became law and the present. Collectively, the long-term Clean Air Act Amendment studies represent an advance over the classical accountability studies that investigated short-term interventions. Results of studies focused on air quality find it straightforward to conclude that the legislation and regulations thereunder led to improvements. Studies that assessed health impacts found mixed results; some found large improvements, whereas others found small or no reductions in health effects. This collection of studies presents an argument for further development of statistical methods that are appropriate for assessing changes in health effects while accounting for the multitude of confounders that change over long time periods on wide time scales.

Previous commentaries on accountability studies

Key points from recent reviews of accountability analyses are discussed below, and a commentary on the direction of the field is provided in the following section.

Health Effects Institute Communications 11, 14, and 15

HEI Communication 11 is intended to promote research aimed at refining methods for performing accountability analyses of current policies (Health Effects Institute, 2003). HEI makes the case for thorough assessment of previous policies, namely, to inform future policy decision-making, and lists a number of opportunities for accountability studies. The work steps through the accountability chain and dissects confounders that influence the chain between the links. The authors list potential issues that affect all accountability studies and provide guidance on opportunities for future work. They present broad recommendations in three areas: (1) development and testing of new study designs, (2) identifying targets for accountability research, and (3) development of systems to track health impacts of regulations. Specific challenges include defining an appropriate baseline of pollutant levels, identifying the impact of mixtures of pollutants versus single pollutants, and isolating the causal pathway between regulations and health outcomes independent of unanticipated changes in personal activities, behaviors, and other confounders. They caution that stakeholders often hold high expectations for accountability research, when the realities of the research may restrict the possibility of providing definitive evidence for efficacy of controls. Since the 2003 report was written, the HEI has funded a number of accountability studies (e.g., Dockery et al., 2013; Kelly et al., 2011; Peel et al., 2010; Peters et al., 2009; Wong et al., 2012; Zigler et al., 2016).

More recently, HEI Communications 14 and 15 review the accountability research carried out to date and make recommendations for future strategies, methods, and opportunities for future accountability research (Health Effects Institute, 2010; Van Erp and Cohen, 2009). Both Communications 14 and 15 note specific challenges in accountability research and make specific recommendations regarding study planning and design, with particular emphasis on designing studies around interventions with high-quality air pollution monitoring data (e.g., highly time and spatially resolved) and large enough reductions in air quality to discern statistically significant effects on health. They recommend assessing as many links in the accountability chain to account for confounding, using proactive (i.e., before the intervention) study design, and continuing developing methods that support causal links between interventions and health outcomes.

Review of accountability studies assessing health end points, 2011

Bell et al. (2011) reviewed accountability studies that assessed health end points. They note that the majority of the accountability studies published up to that point focused on short-term interventions at a local scale, and that extending the analysis to longer time scales and larger distance scales requires the development of more advanced statistical techniques. Increasing numbers of studies have improved their methodologies to take these larger scales into account, such as those discussed above under the Clean Air Act throughout the United States. The authors identify several challenges for studies of this scale (both time and distance). Those discussed in depth include the choice of baseline pollution and health levels, transboundary pollutant transport from outside of the study area, and difficulties in exposure-response studies, such as air pollution modeling and extrapolation between populations and geographic areas.

Bell et al. (2011) note that it can be particularly difficult to determine baseline air pollution levels for secondary pollutants or ones that undergo nonlinear transformations in the atmosphere. They list a number of modeling approaches (e.g., source-receptor matrices and chemical transport models) that are used to estimate air pollution–emissions relationships, but caution that each of these modeling approaches carries with it limitations. The authors recommend further development and evaluation of these models and their extension to exposure estimates. The ability of a model to estimate absolute pollutant concentrations may not be as necessary as the model’s ability to capture changes.

Exposure studies are subject to the same assumptions in accountability research as in more traditional epidemiological research. Examples of such assumptions include the extrapolation of results across pollution levels (e.g., relating to the potential existence of a threshold pollutant level below which no adverse health effects are observed), populations, or time spans. Effects of multipollutant mixtures are difficult to assess because of limited scientific knowledge of interactions, correlation between pollutants, and chemical makeup of pollutants that are characterized together (e.g., PM_2.5 is composed of multiple chemical elements that differ between regions). They note the difficulty in assessing each and every health (and environmental) outcome of interest due to both data limitations and available statistical methods.

They conclude that, given the challenges inherent in accountability analysis, the most robust studies should include detailed assessments of uncertainties and present results from multiple models in order to provide cogent evidence that the policy did/did not impact links in the accountability chain.

American Journal of Epidemiology point-counterpoint, 2014

In 2014, the American Journal of Epidemiology (AJE) published a pair of commentaries in a point-counterpoint exchange that addressed the benefits and limitations of potential-outcomes and prospective research designs (Hubbell and Greenbaum, 2014; Zigler and Dominici, 2014). Zigler and Dominici (ZD) argue that alternative-outcome research adds substance to the current body of air pollution policy research. They note that the majority of the research framed as accountability studies explore indirect relationships between emissions and health outcomes. These studies compare the time during/after the regulatory action (baseline) of interest with a relatable time before or after the action (Figure 1, right side). ZD argue that this ignores potential changes of the exposure–response relationship that may occur during the implementation of the regulatory action (as discussed in this paper). Questions of direct accountability seek to attribute changes in health response to well-defined regulatory actions while controlling for confounders by including comparable populations that are not subjected to the same regulatory action. Instead of quantifying effects of regulatory actions at each link in the chain, confounders are accounted for in statistical models that link the regulation of interest with the end point of interest (usually either air quality or health outcome). They state that direct studies are necessary to advance accountability research and provide important evidence in the present regulatory climate.

In the counterpoint, Hubbell and Greenbaum (2014) concede that the potential-outcomes framework offers an appealing alternative to traditional epidemiological methods that relate air pollution and health. However, they point out that the approach introduces additional factors (in particular data availability and lack of randomness in the assignment of populations to control and study groups) that must be considered in the study design and suggest that potential-outcomes studies would best be served by designing the studies before the intervention of interest in order to best account for confounders. Pope et al. (2012) acknowledge concerns—such as indiscriminant multiple testing of observational data sets, inadequate control of confounding variables, and statistical model manipulation—that relate to the application of observational studies. However, Pope et al. (2012) note that the validity of each study of this type should be judged on its ability to account for confounding.

The direct accountability approach is somewhat at odds with the historical view of the accountability chain, in which each link should be evaluated for its relationship to the link before it. However, the difference in approaches may be subtler; when answering questions of direct accountability, it remains important to account for potential confounders and control for changes over time that are simultaneous with regulatory actions. The confounders may be slightly different; for instance, if a direct accountability study assesses the impact of installing scrubbers on power plants on populations living near power plants, and an important health-related factor—e.g., smoking behavior–—is comparable between the populations, smoking behavior is not a confounder (Zigler and Dominici, 2016).

Challenges and future directions

Two trends are apparent in the accountability assessments above. First, often, one study will assess a regulatory action and determine that the intervention led to a statistically significant change in the response of interest (Clancy et al., 2002; Friedman et al., 2001; Peters et al., 1996; Zigler et al., 2012). Later, using additional data, updated methods, and/or accounting for additional factors, those results are found to be less definitive and potentially invalid (Cox and Popken, 2015; Dockery et al., 2013; Peel et al., 2010; Wong et al., 2012). Second, spatial and temporal scales of accountability studies have expanded, which has increased the number of potential confounders.

Both patterns demonstrate the importance of appropriately accounting for confounding factors when developing models and understanding as much as practicable responses in each step of the accountability chain (Hubbell and Greenbaum, 2014). Examples of important confounders to consider include concurrent regulations, meteorological variability over time/space, interactions between the pollutant targeted by the regulation and other pollutants, and changes in overall health of a population of interest across time and space. In the classical accountability framework, information from these confounders is used to estimate baseline levels at each link in the accountability chain. Typically, researchers use a different type of model for each link, and assessments of both the model outputs (i.e., the counterfactual emissions, air quality, etc.) and the relationships in the model (e.g., the sensitivity of PM_2.5 concentrations to emissions) are necessary components of a full evaluation of baseline levels.

So-called direct accountability studies offer an alternative to complement the classical accountability framework. The direct framework has been used in other fields and is becoming more popular in air pollution accountability. Although it is important that these methods be investigated further, it is important to note that the causal framework used in direct accountability is not applicable to all accountability questions. One example is the situation when suitable control areas or populations do not exist (Zigler et al., 2016; HEI Commentary, 2016). Controlling for confounders is related to selecting an appropriate baseline or control, and statistical methods can be used to address confounding, but neither conventional nor direct methods may not be able to fully account for all confounders (Cox and Popken, 2015; Pope et al., 2012; Zigler and Dominici, 2014).

When feasible, stakeholders should plan approaches to assess a regulation’s efficacy before the regulation is implemented (as in the California Children Studies [Berhane et al., 2016; Gauderman et al., 2015] and the Beijing Olympics, where B. Wang et al. [2010] and X. Wang et al. [2009] began specialized air pollution measurement campaigns and Wu et al. [2010] measured continuous exposure and health markers in taxi drivers during the Olympic period). This will allow for study designs that more fully address confounding (instead of relying on the “natural experiments” that have typically been used up to this point (Cox and Popken, 2015; Hubbell and Greenbaum, 2014). For example, studies can be designed to directly assess each link in the accountability chain (direct emission measurements, expanded air quality observations in areas that are expected to most directly respond to the intervention, as well as control regions, and additional health monitoring).

Any changes attributable to regulations are relative to background pollution levels, which are a function of global emission trends and climate change (Fiore et al., 2015). The interactions between air pollution and climate has been widely studied; however, there exist many uncertainties in the feedback loops (Fiore et al., 2015; Intergovernmental Panel on Climate Change [IPCC], 2013; Tagaris et al., 2007; Weaver et al., 2009). In the United States, the EPA operates the IMPROVE network in remote areas, partially to capture background trends (Sisler and Malm, 2000). These, combined with satellite monitoring and regional and global air quality models, are important tools for characterizing changes in background concentrations.

Uncertainties exist in all observations and models. In accountability research, uncertainties inherent in typical emissions, air quality, exposure, and health outcome data and models increase with each assumption researchers make regarding causality. Uncertainties in epidemiological and other models come in two forms: stochastic, or sampling uncertainty, and uncertainty related to model selection (Health Effects Institute, 2003). Scientists have formal methods for quantifying the former, but the latter is often larger and more difficult to quantify. Overall uncertainty in estimates is an important aspect of interpreting results. Uncertainties can bias results, and differential error can lead to erroneous conclusions when comparing effectiveness across regulations. More work is needed to better quantify uncertainties, and, when possible, correct for errors.

A final aspect of accountability studies is the comparison between the results of the study and the initial proposed benefits of the regulation (Figure 1, dashed lines). This is particularly important, as accountability studies results have the potential to contribute to effective future policies, as, for example, the EPA uses accountability-related methods in its regulatory impact assessments (e.g., EPA, 2015b, 2005b) and its assessment of the 1990 CAAA (EPA, 2011). Few other studies to date, however, have attempted to compare results with the original estimated benefits (alhough there are exceptions, e.g., Harrington et al., 2012; Morgenstern et al., 2012).

Conclusion

As a research field, air pollution accountability has grown in complexity both in terms of the scales of regulations investigated and the analyses performed. The accountability chain serves as one framework for tracking the effects of regulatory actions across multiple outcomes, and future studies should continue to use it as a road map for assessing regulatory actions. A second framework, direct accountability, offers an alternative that should continue to be developed and complement traditional methods. Studies using either approach need to account for relevant confounders when assigning baseline levels and estimating uncertainties with their results. New studies will be enhanced by planning appropriate studies before regulations are introduced in order to enhance data collection of confounding variables as well as outcomes. Further, new tools available to the scientific community, such as satellite measurements and statistical methods that have been applied in other domains offer broad opportunities for the expanded characterization of the efficacy of regulations. These should be evaluated with care in this domain and applied as appropriate. None of the methods reviewed here are applicable to all interventions. The final step in an accountability assessment should be to compare the results with the proposed benefits of the regulation of interest.

Acknowledgment

Laura E. Carpenter helped design Figure 1.

Funding

This material is based upon work supported by Health Effects Institute and the National Science Foundation Graduate Research Fellowship under grant no. DGE-1148903.

Additional information

Funding

This material is based upon work supported by Health Effects Institute and the National Science Foundation Graduate Research Fellowship under grant no. DGE-1148903.

Notes on contributors

Lucas R.F. Henneman

Lucas R.F. Henneman is a Ph.D. candidate in the School of Civil & Environmental Engineering at Georgia Tech. His thesis work is an accountability assessment of the 1990 Clean Air Act Amendments in the Southeastern United States.

Cong Liu

Cong Liu is an associate Professor in Southeast University, China. His main research interests include regional air quality modeling, indoor–outdoor interaction, and gas-particle partitioning.

James A. Mulholland

James A. Mulholland is a Professor in the School of Civil & Environmental Engineering at Georgia Tech. His research is in areas of air quality science and engineering and air pollution health effects.

Armistead G. Russell

Armistead G. Russell is a Professor in the School of Civil & Environmental Engineering at Georgia Tech. His research focuses on atmospheric chemistry and engineering and air pollution health effects.