Promising Performance: The Overestimation and Underestimation of Performance Targets

ABSTRACT

This study examines goal setting in the federally funded, state-implemented Weatherization Assistance Program (WAP). Specifically, using federally determined criteria that states are supposed to use to set expected performance targets, I model the predicted performance targets and compare them against the observed performance targets. The purpose is to examine the determinants of performance target overestimation and underestimation in the WAP. The findings suggest that, where credible commitment to achieving performance targets is important (i.e., when grantees want to show top performance to attract grant money and federal partners), grantee governments are more likely to underestimate goals. However, where the aspirational nature of performance targets is more important than the credible commitment of achieving them (i.e., in an election year or partisan congruence with federal government), grantee governments are more likely to overestimate goals.

KEYWORDS:

• Performance measurement
• Gaming
• Energy policy
• Federalism
• Contracting

Notes

^1. INCLUDE A.

^2. The dependent variable in this literature has focused on funding rates and funding choices (for exceptions see citations omitted).

^3. Service areas are aggregated at the county level. For example, if a local government is providing WAP services, they will be responsible for providing it to the entire county. If their service area is multiple counties, they will then provide WAP services for the entire geographic area of that county.

^4. The $5 billion was given to a partnership between Department of Energy (DOE) and the Department of Housing and Urban Development (HUD) but the program continued to be operated through DOE. Of state agencies through which WAP dollars are funneled, 20% of them of social-services oriented, 66% are economic development oriented and 14% are energy-oriented.

^5. States were subject to considerable oversight and guidance from DOE in the performance target-setting process (DOE, 2009). Administrative records show instances where proposed grantee goals were rejected and funds were withheld until the grantee resubmitted the application for additional funds.

^6. In interviews conducted for this research, one DOE staff person explains the metrics as an “attempt to compare apples to apples. When states use wildly different criteria for their units planned, we cannot compare.”

^7. To estimate these predicted performance targets based on DOE expectations, the actual performance targets were regressed against the four aforementioned factors (ARRA allocation, population eligible, and housing density in the service area and the number of service providers in the state). Since multiple WAP service areas are nested in a single state, a hierarchical linear model with random intercepts was employed to estimate the predicted performance targets.

^8. Unsurprisingly, the results of this model had a high level of predictive power. The within-state r-squared was estimated to be 0.894 while the between-state r-squared was estimated to be 0.731; the overall r-squared was 0.862. Only one variable in the estimated model was statistically significant: ARRA allocation was positive and statistically significant but had an extremely small effect. These results suggest that there are important variables not included in the model. Roughly 14 percent of the variation in the performance targets is left unexplained by the model.

^9. May, et al. (2008) discuss this in terms of the organization of attention.

^10. The American Council for an Energy-Efficient Economy ranks Virginia and Missouri 35^th and 44^th (out of the 50 states), respectively on its 2014 energy efficiency scorecard.

^11. Among the Retrofit Ramp-Up recipients were the Michigan, Missouri, and Maryland.

^12. Since states developed the performance targets for the entire ARRA implementation period in 2009, only 2009 data is used here.

^13. This capacity measure is a variation of that used by Carley et al. (2015), who use the money score developed by Pew’s Government Performance Project (GPP) to measure financial management capacity. The overall GPP score is used to operationalize capacity because target setting is expected to be affected by multiple types of state capacity.

^14. Considerable experimentation was undertaken to determine whether to include any random slopes in the model. Using the variables of interest, none of the random effects for slopes were statistically significant. Thus, a random intercepts model was employed.