An evaluation of the robustness of the visual air quality “preference study” method

Abstract

In 2012, the U.S. Environmental Protection Agency (EPA) considered setting a secondary National Ambient Air Quality Standard (NAAQS) for particulate matter (PM) to maintain urban visual air quality (VAQ) above the level that EPA believes results in adverse effects on public welfare. EPA is relying on a type of survey it calls a “VAQ preference study” to determine this level. Individuals are shown photographs of the same vista under a range of different visibility conditions and asked to state whether the VAQ in each photograph is “acceptable” or “unacceptable.” EPA considers the effect on public welfare to be adverse at the VAQ level that at least 50% of respondents deem unacceptable (the “VAQ cutpoint”). Given its central role in setting a NAAQS, the scientific validity of this method is an important question. This study tests the robustness of the VAQ preference study method by replicating the survey instrument from a prior VAQ preference study, and by applying two variants in which the only change was use of a different range of VAQ levels. Tested on split samples, these three variants produced statistically significantly different VAQ cutpoint estimates. In contrast, all three variants produced comparable results for a calibration task at the start of each survey in which respondents were asked to rate the VAQ in each photograph on a scale of 1–7 (without any opinion on the acceptability of each level). The significantly different estimates of VAQ cutpoints across survey variants cannot be attributed to inability on the part of respondents to discern whether they were being shown the entire range of actual visibility conditions. This suggests that VAQ preference surveys do not actually estimate individuals' enduring preferences regarding VAQ, because absolute preferences would not be influenced by the particular levels of VAQ over which their preferences are elicited.

Implications:

In setting its particulate matter secondary National Ambient Air Quality Standard for urban visibility, the U.S. Environmental Protection Agency is relying on a survey method that has people rate a range of urban visibility levels as “acceptable” or “unacceptable.” A test of that survey method using split samples finds its results are not robust to variations in the range of visibility it shows to respondents. This implies that the visibility preference survey method needs more scientific evaluation before it can be assumed to be measuring individual preferences for visibility in a valid manner.

Background

In 2012, the U.S. Environmental Protection Agency (EPA) considered whether to set a secondary National Ambient Air Quality Standard (NAAQS) for particulate matter (PM) that would address impacts to urban visual air quality (VAQ), or “visibility.” “Urban visibility” refers to the aesthetic impacts that people experience in their day-to-day lives in the area where they live and work. A type of survey that is called a “VAQ preference study” has taken a central role in EPA's deliberations and merits scientific scrutiny.

Extensive scientific research has been conducted demonstrating that people can perceive and reliably report differences in visual air quality, and that those reported perceptions can be linked to objective measurements of VAQ such as light extinction (e.g., Malm et al., 1981; Stewart et al., 1983; Middleton et al., 1984). Research has also been conducted demonstrating that contextual features of a vista or landscape can alter peoples' attitudes about its aesthetic quality (e.g., Craik and Zube, 1976; Daniel and Boster, 1976; Malm et al., 1980). However, no research appears to exist that has assessed whether, for a fixed set of contextual features in a given vista, the VAQ preference study method provides a measure of perceived VAQ that adversely affects individuals' sense of well-being that is reliable and consistent with the theoretical concept of individual preferences. VAQ preference studies are unlike studies that attempt to estimate the public's willingness to pay to improve VAQ, which is the standard method by which preferences are measured by environmental economists (Smith et al., 2005). Thus, prior research on reliability of willingness to pay surveys is not relevant to this alternative method. This paper reports on a preliminary investigation of the reliability and validity of results from VAQ preference studies as measures of the theoretical concept of individual preferences.

In the VAQ preference study procedure, individuals are shown photographs of the same vista under a range of different visibility conditions, and asked to state whether the VAQ in each photograph is “acceptable” or “unacceptable.” Using a fixed vista and varying only the light extinction in that photographic image holds constant most of the other contextual features that also are known to affect aesthetic appeal. (The specific contextual elements of the selected view may still affect the sensitivity of individuals' responses to variations in VAQ applied to that one view, but that sensitivity will be constant for a single photographic image.) The most common current use of such surveys is to identify the VAQ level that at least 50% of respondents rate as unacceptable, and to interpret that VAQ level as the point at which the public welfare becomes adversely affected. This estimate made from the survey responses is called the “VAQ cutpoint” in the remainder of this paper.

Prior to the investigation reported in this paper, four VAQ surveys had been performed, all during the period 1991–2002. Each survey was conducted for a different location, using photographs and populations from those locations. The first study was performed with 214 individuals in Denver in 1989 under sponsorship of the Colorado Air Pollution Control Division (Ely et al., 1991). The second study was a 1994 survey of 180 university students in British Columbia (B.C.) funded by the B.C. Ministry of Environment (Pryor, 1996). These first two studies used actual photographs of the same view but taken at different times. Thus, although the vista was fixed, these earlier surveys' photographs had varying lighting and cloud conditions along with the varying light extinction. The third and fourth surveys used photographs that were created using computer-imaging techniques described in Molenar et al. (1994), so that they differed solely in terms of their light extinction. These later two studies were a 2000 pilot study of nine individuals in Washington, DC, funded by the EPA (EPA, 2001), and a 2002 survey of the general population in Phoenix (385 respondents in 27 focus groups) funded by the Arizona Department of Environmental Quality (BBC Research & Consulting, 2003).

Each of these four surveys used a different vista (each being a view of the locale in which the respective sampled population resided). The photographs of the fixed vista that were shown to respondents during each survey reflected a wide range of VAQ levels. In each study, at least some of the VAQ levels were deemed “unacceptable” by almost 100% of the respondents, and at least some of the VAQ levels were deemed “acceptable” by almost 100% of the respondents. Thus, a 50%-acceptable level can be statistically estimated for each study to serve as an estimate of its VAQ cutpoint.

Whether the VAQ preference survey method produces reliable and valid measures of individuals' preferences is an important question that apparently has not been explored before in a controlled manner. Following Craik and Zube (1976), “reliability” refers to whether the method can produce comparable results if applied several times to the same concept. “Validity” refers to whether the method measures that which it purports to measure. Reliability is more straightforward to evaluate than whether a method's measurements relate well to the conceptualization of what it is intended to measure. Craik and Zube (1976) note that the challenges of demonstrating a method's validity “can be overcome by insisting that the measurement system provide a clear and explicit statement of the conceptualization upon which it is based” (p. 35). Thus, this paper's evaluation of the reliability and validity of the VAQ preference survey method starts with a summary of the conceptualization of preferences, as defined in preference theory.

Theoretical and Empirical Considerations

“Preferences” are defined as the ordering of all combinations (or “bundles”) of goods that people could experience or consume such that any combination that ranks higher than another combination generates higher utility or welfare (Hicks, 1956; Samuelson, 1938). The term “goods” here should be viewed as including services, societal amenities, and environmental amenities, such as VAQ. A preference is a ranking of two alternative bundles of goods. In brief, preference theory is a comparative conceptualization.

The validated empirical methods for making inferences about preferences derived from this theoretical conceptualization also rely on comparative information. The basic requirement for measurement of preferences is the existence of a quasi-concave, monotonically increasing utility function, U, such that bundle “x ₁” is preferred to bundle “x ₂” if and only if U(x ₁) ≥ U(x ₂) for all x ₁ and x ₂ (Debreu, 1954). Empirical efforts to measure individuals' preferences are thus conducted by studying the choices that people are willing to make between a specific good of interest and other goods. In the case of traded goods, information about preferences for goods can be inferred from market observations of what people choose to purchase given variations in relative prices for all goods (Samuelson, 1948). The observational data from markets are interpreted in a comparative manner.

Nontraded goods such as VAQ, however, do not generate the market data necessary to infer peoples' preferences for them. Surveys are thus the only option for obtaining information about preferences for such goods. This makes it necessary to consider the best way to ask questions about individuals' preferences; again, a comparative framing would be consistent with the underlying concept. For example, if a survey asks an individual to choose between two bundles of goods, x ₁ and x ₂, and the respondent chooses x ₁, then it can be inferred that she prefers x ₁ over x ₂. By asking a sufficient number of questions involving such comparative choices, it is possible to estimate an individual's preference function for a particular good that has been included in the alternative bundles. However, reliance on surveys creates special challenges that need to be carefully addressed.

Psychologists have long recognized the pitfalls in reliably eliciting both objective and subjective information through surveys. One widely documented problem, often called “framing bias,” occurs when results differ significantly if the survey states or “frames” the question in an alternative manner that should have no bearing on the correct response or the true construct being studied (see, e.g., Tversky and Kahneman, 1981; Shafir et al., 1993; Levin et al., 1998; Schwarz, 1999). An interdisciplinary field of research on survey design has developed and advanced understanding of these concerns over the past two decades. One of the general insights of this literature has been that questions that involve information that is either temporally or cognitively distant for the respondents are the most prone to being biased by the framing of the survey itself (Schwarz, 1999).

Cognitive distance between the concept that is the target of a VAQ preference survey (i.e., personal preferences about VAQ) and the way a VAQ preference survey poses its question is a concern that is readily identified. Given that preference is inherently a comparative concept, it follows that people will have less cognitive difficulty conceptualizing responses to preference-related questions that ask them to make choices between various combinations of that good and others. In other words, because preference is fundamentally a comparative concept, questions about preferences are best posed in comparative terms. In contrast, asking a person to express whether a certain amount of a good is “acceptable” or “unacceptable” to her/him is likely to create greater cognitive difficulty because the form of the question is inconsistent with the preference structure for a normal good. Normal preferences cannot be summarized in such an absolute and noncomparative manner. (Additionally, it should be noted that preference-eliciting questions need to be multidimensional. Comparative questions involving a single good will produce only trivial results, because more of one good will always be preferred to less of that good.)

The dissimilarity between the comparative concept of individual preference and the noncomparative “acceptable/unacceptable” rating task in VAQ preference surveys suggests a potential area for miscommunication in the course of the survey. If some (or all) of the respondents have preferences over VAQ that cannot be expressed in purely dichotomous, noncomparative terms, they will find the question difficult to interpret. Schwarz (1999) explains that when people are asked a question that may initially have “no obvious meaning” to them, they will seek to determine its “pragmatic meaning” (i.e., determine what the questioner is expecting as an answer) by observing other elements of the survey design. Thus, as Schwarz puts it, the question that is actually answered may not be what the questioner intended.

How might the design of a VAQ preference survey affect responses to it? Following are some possibilities, each of which would need to be tested before one can know if they are actually occurring. If respondents have difficulty interpreting the task of rating VAQ in noncomparative terms (i.e., for “acceptability”), they may start to seek cues in survey content to infer pragmatically what kinds of responses are expected. One of the most prominent potential cues available to them will be the range of VAQ shown in the survey. Thus, the range of VAQ presented during the survey might influence respondents' determinations of acceptability, even though in principle, if a person considers a certain level of VAQ to be “acceptable,” that person should deem it “acceptable” no matter what other VAQ levels the survey also has the person rate. (If, alternatively, a subject's responses about acceptable VAQ are influenced by the survey content, then the survey-based findings cannot be generalized to identify a level of visibility that protects the welfare of the public at large.) Further, because it would seem a meaningless exercise for the questioner to be asking this question if all of the VAQ levels were expected to be acceptable (or unacceptable), respondents might reasonably infer that the questioner expects them to determine that at least one VAQ level being shown is “unacceptable.” Also, some respondents may be concerned that if they rate as acceptable every VAQ level shown to them, they would be categorized as having no environmental sensitivity at all; or, that they would be categorized as unable to detect the variations in VAQ conditions in the photographs. Such respondents could alleviate their concerns by adopting a response strategy of identifying as “unacceptable” at least the worst VAQ level in the set that the survey presents to them.

Tests for framing bias can indicate whether respondents are drawing more heavily on cues from the survey design to formulate their responses than on internal preferences that they hold independently of the survey. Testing for existence of framing bias has become common over the past two decades but appears not to have been applied to any of the past VAQ preference studies, which were themselves conducted 10–20 years ago. The need for such testing is apparent, however, after taking note of the inconsistency between preferences as a comparative construct, and the way that VAQ preference studies frame the question as a purely dichotomous, noncomparative judgment. Not only is reliability a concern, but so too is construct validity. It is therefore reasonable to test the method for framing bias now, and that was the purpose of this investigation.

Hypothesis

Given the conceptual concerns raised above, tests were conducted for the possibility that results from VAQ preference surveys based on the protocol described above are subject to framing bias. Evidence of framing bias would exist if the estimate of the VAQ cutpoint can be altered by changing the VAQ range over which respondents are asked to identify “acceptable” and “unacceptable” conditions. Thus, the following no-bias hypothesis is posed for testing:

If H0 is rejected (H0 = 0), this implies the VAQ preference study method is conducted under a setting of bias. H0 is tested by performing controlled variations of one of the survey instruments, in which only the range of VAQ shown to the participants is changed. Rejection of H0 will occur if the estimated VAQ cutpoint is found to be malleable in a predictable and systematic manner when the range of VAQ shown is varied in a systematic way. For example, this would occur if a split sample is subjected to two variants of the survey in which the range of VAQ is broader in one than in the other, and VAQ levels that are deemed “acceptable” under the broader range variant are deemed “unacceptable” under the narrower range variant. This should not occur if individuals' preferences are stable and the survey instrument is successfully eliciting information about those preferences.

Methods

VAQ is usually described with one of three mathematically related metrics. These metrics are visual range, expressed in kilometers or miles; light extinction, expressed in inverse megameters (Mm⁻¹); and the deciview (dv), which expresses VAQ in units that are linear with respect to human ability to perceive the degree of visibility reduction when vista elements are present at every distance between the observer and the visual range. Each metric is just a mathematical transform of the others. Following EPA's lead in analyzing results of prior VAQ preference studies, this paper uses primarily the deciview metric to characterize the VAQ levels in the photographs. However, the statistical conclusions presented here would be the same regardless of which metric is used to characterize those photographs. A lower dv indicates higher VAQ, which is synonymous with “greater visual range,” “less light extinction,” and “better visibility.” An increment of 1 dv implies approximately a 10% change in light extinction.

To investigate the hypothesis stated above, EPA's 2000 Washington, DC, pilot survey (EPA, 2001) was used as the basis for three variants. The first variant (“Variant 1”) was a replica of the original EPA survey using the same set of rating and acceptability questions, and using the same photograph of a Washington, DC, view under the same set of VAQ levels. Figure 1 shows the photograph under the 15.6 dv VAQ level. The photographs used in this study were created using version 2.9.0 of Air Resource Specialists's “WinHaze” software (available at http://vista.cira.colostate.edu/improve/tools/win_haze.htm), whereas the slides used in the original study were prepared by Air Resource Specialists using the same computer-imaging techniques on which WinHaze is based (Molenar et al., 1994).

Figure 1. Washington, DC, vista used in survey, shown at 15.6 dv.

The second survey variant (“Variant 2”) reduced the upper end of the VAQ range shown to respondents, whereas the third variant (“Variant 3”) increased it. All variants maintained the same VAQ lower bound, which was a very clear condition of 8.8 dv (or 163 km). By fixing Variant 1 to mimic the original 2000 survey, one can assess whether preference survey results are similar when applied to different samples of people and at different points in time. By shifting the VAQ range systematically around that of Variant 1, Variants 2 and 3 allow one to test the bias hypothesis, H0.

The survey questionnaire (see Appendix) followed the flow and wording of the original script, and had subjects record their responses in copies of the same booklet used by EPA. Subjects were first shown the photograph with a VAQ of 15.6 dv (Figure 1) and asked a few basic questions about their habits and visibility beliefs. Next, they were shown the full set of photographs one by one, and asked to score the VAQ in each photograph on a Likert-type scale (1 = “very poor” to 7 = “very good”). They could not go back and view any of the slides again or reevaluate their prior responses. After the first rating task had been completed for each photograph, the facilitator then said that they would be shown the same set of slides once more, with the instruction that they would now rate each slide either as “acceptable” or “unacceptable,” “according to whether the visibility is acceptable or unacceptable to you.” Subjects were not allowed to turn back to their earlier ratings during this second iteration through the set of photographs. The original study's VAQ preference survey portion ended there. In the current investigation, subjects also were asked a few follow-up questions that were not asked in the original survey about what they were thinking during the process. Follow-up questions were open-ended; those responses were given orally, and were recorded by the facilitator. No one taking the survey ever saw any photographs other than those shown in the variant to which he/she was assigned. (The protocol of preceding the acceptability question with the 1–7 rating task was used in all four of the existing VAQ preference studies.)

Operationally, the only differences between the original and this survey are the following: (i) the photographs were shown with 35 mm projected slides in the original, whereas this study projected the photographs from .jpg files on a laptop onto a screen in a windowless room; and (ii) the interviews were conducted in a group in the original, whereas this study conducted them individually. It is unclear how questions that might have been asked by respondents during the original survey were addressed. However, although those subjects' ratings were privately recorded in their own response booklets, the group setting means all original nine respondents would have heard any questions from the other participants, and the facilitator's replies. Because the present survey was conducted in one-on-one interviews, none of the other respondents could have received cues from questions asked by others. The facilitator in the present survey answered clarifying questions when asked, but did not directly encourage such questions and followed the script otherwise.

In the original survey and in its replica, Variant 1, the two rating exercises showed 20 different dv levels in 25 separate photographs, with 5 of the photographs being duplicates of 5 others. The VAQ levels ranged from 8.8 dv (163 km visual range) to 38.3 dv (8.5 km visual range), and were presented in a shuffled order (unsorted with respect to VAQ) maintained for every participant. The solid line in Figure 2 shows the dv of each of the 25 photographs in the order that they were shown. Variant 1 used the same order as the original.

Figure 2. Order of VAQ levels in photographs shown in each variant of the survey.

The second variant (Variant 2) showed only those photographs in Variant 1 that had VAQ better than the VAQ cutpoint estimated in the original survey: only VAQ levels better (lower) than 27.1 dv were shown, including the same duplicates that were in this lower dv range. As a result, the number of photographs shown was reduced to 12. They were shown in the same dv order as in Variant 1, by simply removing those above 27.1 dv from the set of photographs shown to respondents taking Variant 1, and retaining the original order of those 12 photographs that remained. The order of the twelve photographs in Variant 2 is shown as the dashed line in Figure 2. One can see that the effect of not showing the photographs above 27.1 dv makes the pattern of “ups and downs” seen by Variant 2 respondents quite different from the pattern seen by those taking Variant 1. For example, the worst VAQ in Variant 1 (38.3 dv) was the first slide shown. In contrast, the worst VAQ in Variant 2 (now 27.1 dv) was the ninth slide shown. Nevertheless, the ordering in Variant 2 was identical to that in Variant 1, but for the omitted VAQ levels.

In the third variant, Variant 3, two photographs with worse (but not unrealistic) VAQ levels than in the original survey were added (42 and 45 dv), while maintaining the same lower bound of 8.8 dv. VAQ conditions of 42 and 45 dv can occur on nonrainy but high-relative-humidity days, even under attainment of the current PM_2.5 (PM with an aerodynamic diameter ≤2.5 μm) NAAQS. In the interests of survey length, three of the VAQ levels from within the original range were dropped (24.5, 15.6, and 11.1 dv), as were the duplicates (which had produced consistent responses in all the prior surveys). To incorporate these two new photographs, the full set of slides to be used in Variant 3 was reshuffled. Any manner of incorporating these new slides, while also removing others, would have forced an effective re-ordering of the slides, and reshuffling seemed the most neutral option for incorporating them. The order used for Variant 3 is shown as the dotted line in Figure 2.

The three variants of the present survey were conducted during February and March 2009. A sample of 64 individuals was drawn from the pool of employees at a business and economics consulting firm and randomly assigned to one of the three variants of the survey. The facilitator approached people individually in their office or its common areas, and asked if they would be willing to take a brief survey. No incentives for participation were provided and those who took the survey did so entirely voluntarily, without knowledge of the purpose of the survey. None had any prior involvement in regional haze or visibility work, including the facilitator. The choice of individuals to approach was not formally randomized, but it was not targeted or selective of any specific types of staff. The resulting sample included research and nonresearch staff, who worked in a variety of practices areas and administrative functions of the company, with seniority ranging from entry-level to senior management. Obviously, all were employed in white collar jobs. Notably, the sample included residents of both Washington, DC (n = 44), and Houston, TX (n = 20). This was done to explore the effect of location of residence on response. Regardless of city of residence, however, all were surveyed using the same photograph of Washington, DC. Details of the sampled population assigned to each variant are available in Smith and Howell (2009).

Variants 1 and 2 were conducted first on two subsamples composed of Houston residents (n = 10 for each variant) and Washington, DC, residents (n = 16 for each variant). Variant 3 was performed subsequently, and exclusively with Washington, DC, residents (n = 12).

Results

As discussed above, Variant 1 was designed to follow the original survey instrument in all its critical attributes. Figure 3 compares the percentage VAQ “unacceptable” results from Variant 1 with those from the original study (the original results are from Exhibit 11 of EPA, 2001, with scores averaged across both ratings in the case of the five duplicate slides). Based on these raw data alone, the VAQ cutpoint appears to fall somewhere between the 27.1-dv and 29.2-dv photographs in both the original and replication surveys. Based on a logit analysis of the raw data that EPA performed, EPA concludes the original study and Variant 1 from this investigation have equal VAQ cutpoints. EPA's t test for their equality produced a P value of 0.15 (EPA, 2010, pp. 2–26). (The individual response data from this investigation had been provided to EPA.) These results represent a remarkable degree of replication, given the differences in the samples and the relatively small sample size of the original pilot study (n = 9). In fact, both studies produced the same pronounced and anomalous dip in the unacceptability rating of the 28.2-dv photograph. The original researchers suggested that this might be a result of slide ordering. The slide with VAQ of 28.2 dv was the second slide shown in the slide order, and directly after a first slide that had the highest dv rating of 38.3 dv. Their hypothesis is that it “looked” relatively much better to survey respondents given they were asked to rate the worst VAQ in the entire questionnaire just prior (EPA, 2001, p. 21). If slides had been rerandomized for each respondent, such ordering effects in the aggregate ratings might have been mitigated.

Figure 3. Comparison of original and Variant 1 results for percent “unacceptable.”

In this investigation, a logit analysis of individual-level responses was applied to fit a smooth S-shaped curve to the raw individual-level response data, followed by estimation of the VAQ cutpoint and its confidence interval using the Krinsky-Robb procedure. These analyses were performed using Stata MP 12.1 for Windows (64-bit) and the wtpcikr module, with 100,000 replications (Stata Corp., College Station, TX). As reported in Table 1 (row 1), the logit/Krinsky-Robb analysis estimated the VAQ cutpoint in Variant 1 to be 29.3 dv, with a 95% confidence interval (CI₉₅) of 28.6–30.0 dv.

Table 1. 95% confidence intervals for each variant's VAQ cutpoint estimated with Krinsky-Robb analysis

City	Lower Bound	Mean	Upper Bound
Variant 1	28.6	29.3	30.0
Variant 2	20.6	21.3	22.0
Variant 3	31.4	32.3	33.2

Variant 2 showed only VAQ levels deemed acceptable by at least 50% of the Variant 1 respondents—the photographs with VAQ less than or equal to 27.1 dv. Therefore, absent any systematic differences of the populations assigned to each variant, under a setting of no bias (H0 = 1), no more than 50% of Variant 2 respondents should assign unacceptability to any of the photographs included in Variant 2. This did not occur at all. Variant 2 responses produced a striking leftward shift in the unacceptability curve that can be seen clearly in the raw response averages graphed in Figure 4.

Figure 4. Raw response percentages by photograph for each of the three variants.

In Variant 2, the five highest-dv conditions shown (i.e., 21.0–27.1 dv) were deemed “unacceptable” by more than 50% of the sample, even though they were rated “unacceptable” by as few as 0% of the respondents in Variant 1. The portion of the respondents under Variant 2 that deemed these five photographs “unacceptable” was between 50 and 70 percentage points higher than under Variant 1, for each one of the photographs. Based on logit/Krinsky-Robb analysis, the estimated VAQ cutpoint fell from 29.3 dv (CI₉₅: 28.6–30.0 dv) in Variant 1 to 21.3 dv (CI₉₅: 20.6–22.0 dv) in Variant 2 (Table 1, row 2). This cutpoint shift is a change in visibility that is highly perceptible to any person with reasonably normal vision: stated as visual range, it means a cutpoint of 45 km in Variant 1 versus only 20 km in Variant 2. These VAQ cutpoints are statistically significantly different from each other. A difference of means t test rejects the hypothesis that they are equal with over 99.9% confidence (t = 15.8). This is consistent with evidence of framing bias (H0 = 0).

Variant 3 was designed to test whether the converse effect (increase of the VAQ cutpoint by widening the range of VAQ shown to respondents) existed. It did. In contrast to Variant 2, which narrowed the range and shifted the VAQ cutpoint downwards, Variant 3 widened the range and shifted the VAQ cutpoint upwards (see dotted line in Figure 4). VAQ levels deemed unacceptable by about 60% of the respondents under Variant 1 (i.e., the 29.2-dv and 30.1-dv photographs) were deemed unacceptable by only about 10% of Variant 3 respondents. Based on the logit/Krinsky-Robb analysis, the VAQ cutpoint rose from 29.3 dv (CI₉₅: 28.6–30.0 dv) in Variant 1 to 32.3 dv (CI₉₅: 31.4–33.2 dv) in Variant 3 (Table 1, row 3). This 3-dv increase in estimated cutpoint is a 25% visual range decrease (from 20 to 15 km) that is highly perceptible as well as statistically significant. A difference of means t test rejects the hypothesis that the Variant 3 cutpoint is equal to the Variant 1 cutpoint with over 99.9% confidence (t = 5.2). This also is consistent with evidence of framing bias (H0 = 0).

Another way of testing whether the results are significantly different from each other is to use the logit estimation procedure to fit smooth S-shaped curves for all three variants simultaneously, while using dummy variables to allow the procedure to fit differently shaped S-curves for each variant, if differences in shape are indicated by the raw data. That is, an S-shaped curve is defined by eq 1.

(1)

The S-shaped curve of eq 1 can be statistically fit to the raw response data after specifying an equation to define its parameter, Y. Any of a number of equations for Y (“models”) could be used, with the best choice to be determined by the statistical fit of each model. Equation 2 shows one obvious option employing the dummy variable concept:

(2)

where dv is the deciview level of each photograph for which a rating (1 = acceptable, 0 = unacceptable) was obtained, “var2” is a dummy variable for responses obtained in Variant 2 and “var3” is a dummy variable for responses obtained in Variant 3, and ϵ is an error term. If Variant 2 and Variant 3 produced responses significantly different from Variant 1, then the estimates of the parameters for the dummy variables, α₂ and α₃, would be significantly different from zero. Two alternative models that were also estimated are in eqs 3 and 4, where different Greek letters are used only to help clarify the specific model that applies to each set of parameter estimates.

(3)

(4)

Tables 2, 3, and 4 provide the parameters estimated for these three alternative logit/dummy models of eqs 2, 3, and 4, respectively. These show that the dummy variables for Variants 2 and 3 are highly statistically significant (P < 0.001), whether the dummy is applied to an interaction term or to the constant term of Y. When combined in a single model, eq 4, the fit is only slightly improved, suggesting that either eq 2 or eq 3 is sufficient; in all cases, the S-curves for Variants 2 and 3 are found to be significantly different from the S-curve for Variant 1. The specific choice of logit/dummy model does not affect the estimate of the VAQ cutpoint, however; all three sets of VAQ cutpoints calculated using the parameters estimated in this logit/dummy analysis are provided in Table 5. They also match the estimates from the logit/Krinsky-Robb analysis (Table 1).

Table 2. Results of logit analysis with dummy variables ( eq 2)

Parameter	Variable	Coefficient	Standard Error	Z-statistic	P >│Z│
α0	Constant	11.720	0.798	14.70	0.000
α1	dv	−0.399	0.026	−15.12	0.000
α2	Var2	−3.200	0.304	−10.54	0.000
α3	Var3	1.165	0.225	5.17	0.000
N observations	1190
Pseudo R²	0.4839

Table 3. Results of logit analysis with dummy variables ( eq 3)

Parameter	Variable	Coefficient	Standard Error	Z-statistic	P >│Z│
β0	Constant	10.94	0.718	15.25	0.000
β1	dv	−0.372	0.024	−15.44	0.000
β2	Var2 × dv	−0.141	0.013	−10.68	0.000
β3	Var3 × dv	0.033	0.007	4.93	0.000
N observations	1190
Pseudo R²	0.4860

Table 4. Results of logit analysis with dummy variables ( eq 4)

Parameter	Variable	Coefficient	Standard Error	Z-statistic	P >│Z│
γ0	Constant	10.02	0.871	11.50	0.000
γ1	dv	−0.342	0.029	−11.72	0.000
γ2	Var2	0.799	1.560	0.51	0.608
γ3	Var3	4.953	2.678	1.85	0.064
γ4	Var2 × dv	−0.166	0.067	−2.48	0.013
γ5	Var3 × dv	−0.122	0.084	−1.45	0.147
N observations	1190
Pseudo R²	0.4895

Table 5. Comparison of mean cutpoints from each alternative estimation method

Variant	Equation 2	Equation 3	Equation 4
1	29.37	29.38	29.30
2	21.36	21.31	21.30
3	32.30	32.24	32.26

Figure 5 plots all three of the S-shaped curves using each of the three logit/dummy models' respective estimated parameters. The differences in the curves estimated for each of the three variants, respectively, are apparent in Figure 5, but the relevant information indicating whether these differences are statistically significant are the Z-statistics for the dummy variable parameter estimates and the goodness of fit (R ²) of each alternative model. As noted above, the dummy variables' Z-statistics indicate significance. Additionally, the R ² values of the three alternative models that include dummy variables are all between 48% and 49%, whereas the R ² for the same model without any dummy variables was only 35%. This means that the choice among the three alternative logit/dummy models makes little difference, but all three represent statistically significant improvements over the assumption that responses to all three variants are equivalent.

Figure 5. Three alternative S-curves estimated for each of the three survey variants.

To sum up, H0 is rejected with high statistical significance. Statistical tests using both a Krinsky-Robb procedure and dummy variables strongly reject the hypotheses that (i) the VAQ cutpoint for Variant 2 equals that of Variant 1, (ii) that the VAQ cutpoint for Variant 3 equals that of Variant 1, and (iii) that the VAQ cutpoint of Variant 3 equals that of Variant 2, with over 99.9% confidence (P < 0.001) in all cases. t tests do not, however, reject the hypothesis that results of Variant 1 are equal to those of the original survey design that it replicated (P = 0.15). Thus, this investigation was able to closely replicate the results of the original study when the framing of the VAQ levels was comparable, despite fairly substantial differences in the populations sampled. This fact reinforces the interpretation of the statistically significant differences across variant designs as evidence rejecting H0 (i.e., H0 = 0). This investigation thus provides substantial evidence in support of the view that the responses to this survey method are subject to attribute framing bias.

The results above are based on pooled responses of all individuals assigned to each survey variant. Demographic explanations for the pattern of results (such as gender, job category, or geography) were explored and none detected. The strong shift in the VAQ cutpoint between Variants 1 and 2 was apparent for all demographic subgroups in the sample, and also between Variants 1 and 3. More details documenting the similarity of response patterns across demographic categories are in Smith and Howell (2009).

The current study was not designed to address reliability of results across surveys using different vistas, but it can address whether the framing bias found for a fixed vista is modified by whether that single vista is near to or far from where the respondent lives. This was done by the inclusion of split samples of people living in Washington, DC, and Houston for Variants 1 and 2. Those results are presented by region in Figure 6. Qualitatively similar response malleability is observable in the raw results for the two different geographical populations. This suggests that survey design changes, not regional heterogeneity of attitudes about VAQ, are responsible for the evidence of framing bias found in this study.

Figure 6. Breakdown of responses for Variants 1 and 2, by residence of respondent.

The primary way regulators have used VAQ preference studies has been to identify a cutpoint where 50% of the population finds the VAQ unacceptable. However, all four of the existing VAQ preference surveys, and this one, first asked respondents to rate the VAQ on a scale of 1 (“very poor” VAQ) to 7 (“very good” VAQ). In the preceding four studies, results from this task were used only to demonstrate that the respondents could properly discern the relative VAQ differences among the photographs, and to explore consistencies across subgroups. In this investigation, results from this task can also be used to assess whether respondents could relate the VAQ levels shown to them in the survey to the range of levels that they experience in daily life. Average Likert scores for each dv level from Variants 1 to 3 are graphed in Figure 7. These results support the view that respondents in this survey also could discern VAQ variations in the photographs they were shown. More importantly, the ratings outcomes shown in Figure 7 also suggest that subjects were aware when the range of VAQ that they were seeing did or did not span the full range of what they see in daily life. For example, the minimum possible average score of 1 was only approached in Variant 3, for the two highest-dv slides in Variant 3. Similarly, when Variant 2 showed VAQ only up to 27.1 dv, the minimum average score was 3 (for the highest dv shown). This indicates that respondents were aware that visibility conditions in daily life do get worse than the worst they were being shown in Variant 2. Even Variant 1 respondents appeared to be aware that the real-world visibility does get worse than the worst VAQ shown to them in that survey (i.e., 38.3 dv), because the lowest average score assigned in Variant 1 was about 2.5, not 1.

Figure 7. Average ratings of VAQ for each photograph, by variant.

There were four introductory questions in the original script, and also a number of follow-up questions that were not asked in the original study. Responses to the introductory questions are presented, geographically disaggregated, in Tables 6 and 7. (Question B was poorly designed and misunderstood by respondents; it is excluded here.) Responses to the introductory questions reveal that respondents were familiar with the view shown and most expressed at least occasional awareness of the quality of visibility conditions around them. Almost half of all respondents reported VAQ to be important or extremely important to them.

Table 6. Summary of responses to introductory Questions A and C

Question	Location	Never	Rarely or Infrequently	Occasionally	Frequently	Other*
A. How often do you see a view like this one?	DC	0%	17%	50%	22%	11%
[i.e., Figure 1]	Houston	0%	15%	50%	35%	0%
C. Generally speaking, how often do you	DC	3%	25%	44%	19%	8%
notice the quality of visibility on a given day?	Houston	0%	25%	45%	30%	0%
Notes: *DC respondents who checked the “Other” box for Questions A and C commented that the survey's definition of “frequently” as “once a day” and “occasionally” as “once a week” were too disparate, and wrote on the survey that they see a view like this, and notice the quality of visibility, a few times a week.

Table 7. Summary of responses to introductory Question D

Question	Location	Not Important	Somewhat Important	Important	Extremely Important
D. How important are the visibility	DC	6%	44%	28%	22%
conditions in [DC/Houston] to you?	Houston	5%	50%	30%	15%

Although VAQ preference studies seek to isolate visibility from concerns about health, this concept was very confusing to the subjects. In response to the follow-up question “Was any part of your assessment impacted by thoughts about health?” 42% of all subjects in Variants 1–3 reported that yes, they had thought about health. Such a large fraction is notable because the script explicitly included instructions to focus only on visibility: the following admonition was given prior to rating the slides, “For this discussion, we want you to focus only on visibility: how well you can see. Our goal is to understand your opinions regarding visibility itself. When you give us your opinion, consider only the visibility.”

Subjects were also asked in follow-up, “In deciding whether a slide was acceptable or unacceptable, what were you thinking about, or what were your criteria?” Most individuals answered that they were considering how hazy it was, how much color they could see, or how far they could see. Some subjects related the vista in the slides to their place of residence, suggesting that attempts to form an opinion about visibility “acceptability” are informed by personal experience. A number of subjects made comments that support the conclusion of framing bias. For example, one subject said, “Since I don't have anything to base [the acceptability rating] on, I'll go with the one in the middle.”

Discussion

This study was conducted to explore a hypothesis about the presence of framing bias in the VAQ preference study protocol, motivated by a broader concern about inconsistency between its questions and preference theory. Based on relatively small samples and three survey variants that differ only in the VAQ range shown to respondents, the no-bias hypothesis, H0, is rejected at the P < 0.001 level of significance. The ability to replicate the estimated VAQ cutpoint (and even apparent ordering effects) of the original survey when applying the identical VAQ framing further supports the interpretation of these malleable results as evidence of a systematic response to the survey design changes. Although the sample in the present survey was not fully representative of the U.S. population as a whole, the effect of representativeness of the sample on the hypothesis test is argued to be immaterial, given the strength of the differences in responses to the three variants, combined with replicability of the original study responses in the one variant with identical framing to that of the original.

The detection of framing bias in this investigation certainly raises concerns with reliability of the VAQ preference method, especially given its specific question format, which was followed without modification here. However, these results also give support to the concern about possible lack of construct validity that originally motivated this test for framing bias, i.e., the inconsistencies noted between the survey instrument design and the theoretical concept it is intended to measure. The evidence that framing bias does exist in this method is necessary but not sufficient to demonstrate lack of construct validity. This evidence does, however, imply that the central focus of further research should be to determine whether the framing bias detected here is caused by the way the instrument requires subjects to express their attitudes about VAQ in the dichotomous, noncomparative format of acceptable/unacceptable. One should test whether other methods of posing the VAQ preference questions that are more consistent with the theoretical construct of preferences would be able to reduce or eliminate the observed framing bias. When even small samples generate a strong and statistically significant change in the estimate of VAQ cutpoint, a meaningful and appropriate way of framing questions regarding VAQ preferences requires further scientific investigation.

It is important to note that the problem of response malleability observed in this study probably cannot be resolved simply by ensuring that the full range of real-world VAQ is included in the survey. For this to be the only remedy needed, one needs to show that the reason people in Variants 2 and 3 made different statements about what is “acceptable” was because they could not discern when they have been shown a range that is less than they experience in daily life. The evidence in Figure 7 indicates the opposite: for each variant, respondents spread their 1–7 ratings over a narrower range when shown a narrower range of VAQ. More importantly, such a response to the findings of this investigation would be failing to account for the possibility that the fundamental problem leading to the detected framing bias is the use of the noncomparative acceptable/unacceptable question to elicit information on individual preferences. If the latter case is true, and the framing bias is evidence that respondents are constructing strategies for answering the survey question that have little or no relationship to their true (i.e., comparative) preference structure, then the specific format of the question that VAQ preference surveys have been asking would be the root problem. Then, even a survey showing the most complete range of VAQ levels would lack construct validity, and not be able to produce valid measurements of individual preferences or identify a point of VAQ degradation at which people become “adversely affected” by VAQ.

An interesting question is whether a survey protocol without the task to rate the photographs on a 1–7 scale prior to the acceptability rating task would alter the response malleability detected in this investigation. Such a protocol would certainly complicate a respondent's attempts to identify at least one of the VAQ levels as unacceptable, because respondents would not start that task with foreknowledge of the range of VAQ they would be asked to rate. However, if construct validity is the cause of the current evidence of framing bias (i.e., that respondents answer the acceptability question by reference to survey cues rather than by finding the answer from within their own preference structure), construct validity cannot be created just by changing the cues available in the survey. In that case, even if malleability might be less evident under the alternative protocol, the VAQ preference study method will continue to lack validity until the acceptability question itself is reframed to better match the theoretical construct of preferences. Thus, it is argued that the foremost priority for further research should be to ascertain whether the present results are indicative of a construct flaw in the acceptability question itself.

Results from this study also reinforce concerns identified in the original EPA study that respondents to VAQ preference studies intertwine health risk concerns with aesthetic concerns, even when instructed to consider only the aesthetic concerns. This has long been recognized as one of the difficulties in the interpretation of results from visibility willingness-to-pay studies, and our results suggest that VAQ preference studies are not immune to this methodological problem. It is possible, and might also merit further research, that respondents are more likely to resort to thoughts about health concerns if health effects are even mentioned during the survey. If so, this would be another source of framing bias in which respondents might be responding to cues provided by the survey instrument itself to answer the cognitively difficult question. However, rather than trying to find further evidence of framing bias, a higher priority for additional research should be to determine whether a mismatch between preference theory and the survey's question might be the root cause of the framing bias.

The use of the same slide ordering for every respondent is a poor survey design element independent of other issues that have been raised in this paper. Survey design standards call for each respondent to observe a different, randomized ordering to average out any ordering effects on responses. Slide ordering effects are suspected to exist, as noted in the Results section. However, this study needed to determine whether an exact replica of the original design could reproduce the original survey's results before it could perform the controlled testing with variants. Although it seems unlikely that the framing bias that was detected in this investigation could be primarily the result of slide ordering effects, any further work on survey design should use survey instruments that properly randomize the order of the photographs, so that each individual within a single variant sees the same photographs but in a different order.

As noted in the Background section, four VAQ preference studies existed prior to this investigation. EPA has performed logit-based statistical analyses of the results of those four studies using dummy variables for each city. EPA reports significant heterogeneity in the results across those studies, with t tests strongly rejecting the null hypotheses that any of the four cities' VAQ cutpoints is equal to that of any of the others (EPA, 2010, pp. 2–29). The 95% confidence intervals for those four cutpoints are provided in Table 8 and illustrated in Figure 8. The differences are not only statistically significant, but physically large. For example, the lower bound of each estimated VAQ cutpoint differs from the upper bound for next closest VAQ cutpoint by between 10% (i.e., more than 1 dv) and 50% (i.e., more than 4 dv). Such variation of results from studies in different cities also needs to be explained, but it is a separate phenomenon from the lack of robustness found in this investigation. Those four studies each used different vistas with very different contextual elements. Distance to horizon, presence of objects in the foreground, clouds, etc., have all been identified in prior research as affecting perception of and attitudes towards VAQ. Thus, the specific contextual differences in the vistas used in each city are likely to be a substantial part of the explanation for the observed inter-city heterogeneity (Malm et al., 2011). However, even if further research produces explanations for the inter-city differences, those insights are not likely to address the significant malleability found in this investigation, because it exists within a single city in which all the contextual elements of the selected vista are held constant for a given dv level shown to respondents in each survey variant. Given the controlled method of this investigation using just a single photograph, the lack of robustness it has found is evidence of framing bias, not of impacts of alternative contextual elements in a photograph.

Table 8. 95% confidence intervals for each city's VAQ cutpoint estimate in EPA (2010)

City	Lower Bound	Mean	Upper Bound
Denver	19.7	19.9	20.1
British Columbia	22.2	22.5	22.7
Phoenix	24.0	24.2	24.3
Washington, DC	28.6	29.1	29.7

Figure 8. Graph of 95% confidence intervals for each city's estimated VAQ cutpoint.

Further study might also be focused on identifying the heuristic decision rules that respondents are using in their attempts to answer the question of “what is acceptable?” One important issue to explore in further study is whether respondents are thinking about the acceptability rating task as an answer that is strictly “for oneself” or “for society.” In the latter case, their responses will not be reflective of personal preferences, defined by individual willingness to make tradeoffs between different bundles of goods, amenities, and services. All of these could be components of a scientific investigation of the psychological response to the acceptability/unacceptability question, and of alternative ways of eliciting information on people's preference structure for VAQ. That research has not been done, but the need for it to establish the validity of a VAQ preference survey methodology has been demonstrated by this investigation.

Conclusion

This study examines the robustness and reliability of responses to VAQ preference studies. It finds that the estimates of the VAQ cutpoint change in a statistically significant manner when the range of VAQ presented is changed, whether by narrowing or widening the range shown. Such malleability is evident in different geographical populations and in other demographic disaggregations. This opens to question whether VAQ preference studies are able to elicit a valid measure of individual preferences. As the first study examining the robustness of the VAQ preference methodology, this study should be categorized as preliminary research and the results warrant confirmation with more study. However, the malleability of responses is so pronounced in each of the demographic subgroups in the sample that it is unlikely that this effect will not reappear in a larger and more representative sample. A more important next step will be to conduct tests of the validity of the way questions are framed in VAQ preference studies, consistent with modern survey design practice. Such tests of question design have not been performed in the past, and current study findings indicate they are needed. The questions raised by the nonrobustness of results from current forms of the VAQ preference survey instrument should be resolved before the method can be said to produce valid indicators of public preferences for urban VAQ.

Acknowledgment

The author thanks the Utility Air Regulatory Group for funding to conduct and document the survey, and the American Petroleum Institute for supplemental funding to prepare this paper. The author also thanks Will Gans, David Montgomery, and two anonymous reviewers for helpful comments and input. Any remaining errors are solely the responsibility of the author.

Appendix

Below is the script of the survey used in this study, which replicates the portions of the original Washington, DC, study that comprised the VAQ preference questions.

1.	I first want to find out whether this common view of Washington, DC, is one that you typically see. [Project slide with 15.7 dv visibility.] This is a pretty typical view of Washington, DC, on a day with good weather, and pretty scenic. [Now have them answer Questions A and B in booklet.]
2.	Visibility a. Air pollutants come from a number of sources, including cars and buses, industry, and so forth. b. Air pollution forms smog (also called ground-level ozone) and soot (made up of very small airborne particles). Smog and soot in the air cause visibility impairment, or haze. c. Natural factors such as humidity and dust can also contribute to haze. Visibility is affected by haze. When it is hazy, you can't see as far, the objects in your view are not as clear and crisp, and colors don't show up as well. Haze is caused when sunlight encounters these tiny particles in the air. Some of these tiny particles reduce visibility by absorbing and others reduce visibility by scattering light. d. You may also know that the same air pollutants that form haze and reduce visibility are of concern because they are harmful to people and the environment. e. For this discussion, we want you to focus only on visibility: how well you can see. Our goal is to understand your opinions regarding visibility itself. When you give us your opinion, consider only the visibility. f. Just focus on how things look: i. how far you can see ii. the clarity iii. the crispness iv. the colors [Now have them answer Questions C and D in booklet.]
3.	Rating of Visibility Conditions a. Now, you will now be shown a series of slides with the same view of Washington we've been looking at. These slides illustrate many different levels of visibility. b. Just rate the slides according to the visibility conditions in the slides: The ratings run from very poor to very good. Very poor is all the way to the left, and very good is all the way to the right
4.	Acceptability/Unacceptability a. Now you will be shown the same set of slides that you just rated. Again each image will illustrate the effect of a different level of visibility. This time, rate the slides according to whether the visibility is acceptable or unacceptable to you.
5.	In deciding whether a slide was acceptable or unacceptable, what were you thinking about, or what were your criteria?