From the ‘econometrics of capital punishment’ to the ‘capital punishment’ of econometrics: on the use and abuse of sensitivity analysis

Abstract

The academic debate over the deterrent effect of capital punishment has intensified again with a major policy outcome at stake. About two dozen empirical studies have recently emerged that explore the issue. Donohue and Wolfers (2005) claim to have examined the recent studies and shown the evidence not to be robust to specification changes. We argue that the narrow scope of their study does not warrant this claim. Moreover, focusing on our two studies that they have examined, we show the deterrence findings to be robust, while their work has serious flaws and their reporting appears to be selective. The selectivity is biased towards showing ‘no deterrence’.

Acknowledgements

We are thankful to Joanna Shepherd for substantial assistance with this article. Comments by Steven Durlauf, Hugo Mialon, Dale Cloninger and John Donohue are gratefully acknowledged.

Notes

¹ The Solicitor General of the United States, for example, introduced Ehrlich's findings to the Supreme Court in support of capital punishment (Fowler versus North Carolina, 428 US 904, 1976).

² For example, Yunker (1976), Cloninger (1977), Ehrlich and Gibbons (1977), Layson (1985) and Chressanthis (1989) report further deterrence evidence; Bowers and Pierce (1975), Passell and Taylor (1977), Hoenack and Weiler (1980), Leamer (1983), McManus (1985), Cover and Thistle (1988), McAleer and Veall (1989) and Grogger (1990) find no deterrence, and Black and Orsagh (1978) find mixed results.

³ The studies that show deterrence include Brumm and Cloninger (1996), Ehrlich and Liu (1999), Lott and Landes (2000), Cloninger and Marchesini (2001), Yunker (2002), Dezhbakhsh et al. (2003), Mocan and Gittings (2003), Shepherd (2004), Liu (2004), Zimmerman (2004, 2006a), Cloninger and Marchesini (2005), Dezhbakhsh and Shepherd (2006) and Ekelund et al. (2006). The no deterrence studies include Bailey (1998), Albert (1999), Sorenson et al. (1999), Stolzenberg and D’Alessio (2004), Berk (2005), Fagan et al. (2006) and Kovandzic et al. (2009). Results reported by Katz et al. (2003) are mixed, though tending towards nondeterrence. We do not include Fagan (2006) in the above count as it is a compendium of verbal criticisms without any empirical analysis to back the author's assertions. In a recent study that focuses on findings by Dezhbakhsh et al. (2003) and Donohue and Wolfers (2005), Cohen-Cole et al. (2007) attribute the disparate findings in this literature to model uncertainty.

⁴ See Donohue and Wolfers (2005, pp. 794 and 844).

⁵ These two studies are Dezhbakhsh et al. (2003) and Dezhbakhsh and Shepherd (2006). We also need to note that our purpose here is not to offer a point-by-point response to Donohue and Wolfers’ critique of our studies. Our aim is rather to analyse their methods and conclusions.

⁶ Zimmerman (2006b) documents several misrepresentations and errors in Donohue and Wolfers’ study. Cloninger and Marchesini (2007) state that Donohue and Wolfers’ measurement-related criticism of their work is baseless. Mocan and Gittings (2006) also raise doubts about Donohue and Wolfers’ claims.

⁷ This list is not intended to be exhaustive and is drafted drawing on the literature on sensitivity analysis and the authors’ insights.

⁸ For example, in their reply to Donohue and Wolfers’ (2005) criticism of their work, Mocan and Gittings (2006) expose errors that Donohue and Wolfers have made in estimation of risk measure. They further show that appropriate extension of the specification in their original article (Mocan and Gittings, 2003) does not alter their deterrence finding. Zimmerman (2006b) also documents several misrepresentations and errors in Donohue and Wolfers’ study. We must also note that our purpose here is not to offer a point-by-point response to Donohue and Wolfers’ claims about our findings.

⁹ Donohue and Wolfers examine a prepublication version of this study which is available at http://law.bepress.com/cgi/viewcontent.cgi?article=1017&context=alea. While there are some differences between this version and the published version, the main findings are quite similar.

¹⁰ See Donohue and Wolfers (2005, p. 815).

¹¹ See Cloninger and Marchesini (2007) for a discussion of this point.

¹² See Donohue and Wolfers (2005, pp. 796–797).

¹³ See, e.g. Cook (1977), Cook and Weisberg (1982) and Welsch (1982).

¹⁴ See, e.g. Wooldridge (2006, Section 9.4) for the perils of nonrandom data omission.

¹⁵ Dummy-trap refers to a situation where a linear combination of two or more regressors are equal to a column of ones and, thus, perfectly multi co-linear with the regression intercept, making estimation impossible.

¹⁶ See, e.g. Wooldridge (2002, Chap. 1).

¹⁷ This error is due to ignoring specific coding of the STATA command (STATA User's Guide, Release 8, p. 273).

¹⁸ Donohue and Wolfers’ assertion that we used Ordinary Least Squares (OLS) method is incorrect. We used weighted least squares with robust SE and cluster correction. In fact, when attempting to replicate our work, they themselves use weighted least square which is Generalized Least Squares (GLS) and not OLS (Donohue and Wolfers (2005), Table 2 and top of p. 805).

¹⁹ See Donohue and Wolfers (2005, p. 834).

²⁰ This assumption is fairly intuitive. As an analogy, one can point to the diverse effect that 1 min of television advertising has in markets with different population densities, and, therefore, different viewer densities. Advertisers pay more for dense markets because of the larger impact.

²¹ See Donohue and Wolfers (2005, p. 796).

²² See Dezhbakhsh and Shepherd (2006, Table 8).

²³ For these parameter tests, the null is the absence of deterrence and the alternative is the presence of deterrence. Given our interest, the one-sided deterrence alternative is appropriate.

²⁴ Their estimate for moratorium variable is −0.47, which is close to our estimate of 0.4 for the moratorium variable. The difference in sign is trivial and due to their designation of 1 and 0 instead of 0 and 1 for the binary variable.

²⁵ Zimmerman (2006b) has criticized Donohue and Wolfers for mischaracterizing his results and making errors in their analysis.

²⁶ The influence is exerted through changing the makeup of the court system by appointing new judges or prosecutors that are ‘tough on crime’. This affects the justice system and its desire to convict and execute criminals.

²⁷ They admit this point in footnote 84. Also, in our description of the partisan influence (PI), we should have stated ‘six PI variables’ instead of ‘PI’ to make the point clear. Nevertheless, as Donohue and Wolfers acknowledge in p. 823, they had our computer programs which show that we use six PI variables.

²⁸ This is because an incorrect restriction forces all coefficient estimates to have the same value. But the corresponding coefficients are not equal, so imposing equality causes estimation bias. Note that while imposing a wrong restriction causes estimation bias, failure to impose a correct restriction reduces the accuracy of estimation (efficiency loss) (see, e.g. Greene, 2003, Chap. 8).

²⁹ It is also not surprising that they find a strong deterrent effect in such cases, because by construction they create biased estimates that suggest executions take place in nondeath penalty states.

³⁰ For example, the difference-in-difference means analysis reported in their Table 1 panels B, C and D, listed under ‘Our Innovation’, contains an error. What they want to do in panel D is to perform a matched (or paired) comparison, whereby the crime change for each state is subtracted from the crime change for its matched pairs, and then a one-sample location (mean) test is performed on these differences. What they do instead is a two-sample comparison, whereby the overall average obtained for the treatment group is subtracted from the overall average obtained for the control group (averages reported in panel C is subtracted from the corresponding ones in panel B). Calculation of the SE and the t-tests for the two approaches are different; one uses a one-sample t-test and the other, the two-sample t-test, and the results can be different as well (see, e.g. Sheskin, 2004). Correcting this gross error may not alter their finding, but such errors diminish considerably the readers’ trust in a statistical study. Donohue and Wolfers also offer no description of the matching process for states and no list of the matched states.

³¹ See Donohue and Wolfers (2005, Table 7, panels D and E).

³² See Donohue and Wolfers (2005, top of p. 827).

³³ The first pick has a probability of 4/50 and the second pick a probability of 3/49 and the probability of the intersection is 0.0048.

³⁴ In its general form, Hausman specification test involves a statistical comparison of two estimators of the same set of parameter(s). Under the null hypothesis of no misspecification, these two sets of estimates are statistically equal. In the present context, the null hypothesis is that the instruments are valid.

³⁵ The subscripts i and t are dropped and the linear equations are written in the general form for expositional ease.

³⁶ We thank John Donohue for suggesting that we emphasize this data distinction.

³⁷ The first stage involves estimating the reduced forms of Equations 1–4 and the second stage involves reestimating these equations while replacing the endogenous variables on the right hand side with their predicted values from the first stage. Also, appropriate correction is made to the SE to account for the fact that the residuals in the 2SLS estimation is the difference between the left hand side variable and the linear combination of the right hand side variables and not their predicted values, as obtained when least squares is applied in stage 2 of estimation (see, e.g. Davidson and MacKinnon, 1993, Section 7.5).

³⁸ They obtain alternative estimates by (1) excluding PE, JE and PA, or (2) excluding the six PI variables from the set of instruments.

³⁹ It is not clear whether the estimates in their Table 9, which also correspond to what they call instruments, are obtained by including the demographic/economic and trend variables in their estimation. If not, then these results are also invalid.

⁴⁰ See Hausman (1978) or many standard econometric textbooks (e.g. Davidson and MacKinnon, 1993, Chap. 11; Johnston and DiNardo, 1997, Chap. 8; Greene, 2003, Chap. 5).

⁴¹ Even if their comparison estimates were correctly obtained, visual comparison would not be appropriate for a Hausman test, because the variance of the difference between alternative estimates is not the difference in the corresponding variances, so reporting two estimates and the corresponding SE is not very telling, no matter how different they are.

⁴² See Donohue and Wolfers (2005, p. 844).

⁴³ See Section I of this article for a list of these 21 studies. Donohue and Wolfers have been aware of all but perhaps one (Fagan et al., 2006) of these studies which have all been cited in Dezhbakhsh and Shepherd (2006) or Shepherd's congressional testimony that Donohue and Wolfers cite in footnote 11 of their 2006 study. The four studies they examine include Dezhbakhsh et al. (2003), Katz et al. (2003) and Mocan and Gittings (2003), Dezhbakhsh and Shepherd (2006). They also briefly touch on three other studies by Cloninger and Marchesini (2001, 2005) and Zimmerman (2004), without a detailed sensitivity analysis.

⁴⁴ Six of the nine tables reporting their sensitivity checks are devoted to two studies, Dezhbakhsh and Shepherd (2006) and Dezhbakhsh et al. (2003).

⁴⁵ They only sent us their article when it was about to go to print. Specifically, Donohue and Wolfers e-mailed us their article on Friday night, 2 December 2005, asking for input by Monday, 5 December 2005 – the date Stanford Law Review had given them to make further revisions, as they stated in their e-mail. Despite various prior e-mails where they sought our help with data and computer codes, they never communicated to us the exact purpose of their study or their findings until that Friday evening. Moreover, the article was already in the Stanford Law Review format, indicating that they had prepared it much earlier. We note that we sent a preliminary version of this article to Donohue and Wolfers. Donohue provided brief comments that we addressed but Wolfers did not respond in writing, but indicated orally that he did not find our analysis convincing.

⁴⁶ We find it quite amusing that Wolfers, in an interview with Tanner (2007), refers to deterrence findings as ‘flimsy [results] that appeared in second-tier journals’. The studies he refers to have gone through rigorous peer review process, often blind review and appeared in economics peer reviewed journals, while his finding with Donohue, that he uses to call the deterrence studies flimsy, has never been peer reviewed and appears only in a student edited/refereed journal. Moreover, their article appeared online (at SSRN) only after publication and was not to our knowledge presented until after publication.

⁴⁷ We had posted our article online and presented them at several universities and professional meetings, including one session at the American Economics Association Meetings at which Donohue presented the Levitt–Donohue article on abortion.

⁴⁸ Lacetera and Zirulia's (2009) theoretical analysis suggests that peer review is a critical deterrent to scientific misconduct, perhaps even more important than editorial oversight. (We thank Hugo Mialon for bringing this research to our attention.)

⁴⁹ See Donohue and Wolfers (2005, p. 845).