Evaluating Contradictory Experimental and Nonexperimental Estimates of Neighborhood Effects on Economic Outcomes for Adults

Abstract

Although nonexperimental studies find robust neighborhood effects on adults, such findings have been challenged by results from the Moving to Opportunity (MTO) residential mobility experiment. Using a within-study comparison design, this article compares experimental and nonexperimental estimates from MTO and a parallel analysis of the Panel Study of Income Dynamics (PSID). Striking similarities were found between nonexperimental estimates based on MTO and PSID. No clear evidence was found that different estimates are related to duration of adult exposure to disadvantaged neighborhoods, nonlinear effects of neighborhood conditions, magnitude of the change in neighborhood context, frequency of moves, treatment effect heterogeneity, or measurement, although the uncertainty bands around our estimates were sometimes large. Another possibility is that MTO-induced moves might have been unusually disruptive, but results are inconsistent for that hypothesis. Taken together, the findings suggest that selection bias might account for evidence of neighborhood effects on adult economic outcomes in nonexperimental studies.

Keywords:

• Neighborhood effects
• employment
• within-study comparison
• selection bias
• Moving to Opportunity

Disclosure Statement

No potential conflict of interest was reported by the authors.

Supplementary Material

Supplemental data for this article can be accessed here.

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Notes

1. This literature is vast, but important reviews or original contributions have been made in numerous disciplines, such as sociology (e.g., Sampson, 2012; Sharkey & Faber, 2014), psychology (Leventhal & Brooks-Gunn, 2000), economics (Chetty & Hendren, 2018a, 2018b), public health (Kawachi & Berkman, 2003) and housing (Galster, 2017), as well as key reports by the National Academy of Sciences, such as Lynn and McGeary (1990) and Shonkoff and Phillips (2000).

2. For example, see Delgadillo, Coster, and Erickson (2006); Diez-Roux (2001); Ellaway, Anderson, and Macintyre (1997); Ellen and Turner (2003); Goldsmith, Holzer, and Manderscheid (1998); Holzer (1991); Kawachi and Berkman (2003); Leventhal and Brooks-Gunn (2000); O’Regan and Quigley. (1996); Ross (2000); Ross, Reynolds, and Geis (2000); Ross and Mirowsky (2001); Sampson, Morenoff, and Gannon-Rowley (2002); Silver, Mulvey, and Swanson (2002); Vermilyea and Wilcox (2002); Waitzman and Smith (1998); and Yen and Kaplan (1999a, 1999b).

3. Given our focus on the urban poor in the United States, we do not review literature on neighborhood effects in other countries, nor do we cover literature on the effects of ethnic enclaves or neighborhood social networks on the labor market outcomes of recent immigrants. Galster and Hedman (2013) and Galster and Sharkey (2017) include recent reviews of these issues.

4. That is, allowing for separate intercept terms for each individual or family in the study sample to control for time-invariant confounding factors.

5. In our study we use the geocoded locations of MTO addresses as constructed by HUD, which have the advantage of being consistently available for all of the residential addresses we have available for MTO households, but have the disadvantage of perhaps including some additional measurement error relative to other sources of geocoded addresses. This would work in the direction of our understating MTO’s effects on the neighborhood environments of participating households.

6. However, Pinto (2019) and Aliprantis and Richter (2020) seem to find beneficial MTO effects on adult labor market outcomes for the subset of the sample that experienced particularly large neighborhood changes.

7. Previous work on MTO (Kling et al., 2007) did try to compare the effects of a discrete 1/0 residential-move variable with a measure of actual neighborhood poverty rates. The results suggested that neighborhood poverty had much larger effects, although the statistical power of this analysis was somewhat limited.

8. Of course, there is always the chance that in any given sample, random assignment fails to achieve a balance between the treatment and control groups in the distributions of all baseline variables that affect the outcomes of interest. However, previous MTO research has shown that at least among those baseline variables that are captured in the available data, there does seem to be a balance (see, e.g., Orr et al., 2003; Sanbonmatsu et al., 2011).

9. Within-study comparisons began with LaLonde (1986) and Fraker and Maynard (1987) in the context of job training, and have now been extended to a wide range of other applications such as welfare (Michalopoulos et al., 2004), early childhood interventions (Dong & Lipsey, 2018), education (Agodini & Dynarski, 2004; Angrist, Autor, Hudson, & Pallais, 2015; Bifulco, 2012; Fortson, Gleason, Kopa, & Verbitsky-Savitz, 2015; Hallberg, Wong, & Cook, 2016; Jacob, Somers, Zhu, & Bloom, 2016; Wilde & Hollister, 2007), immigration (McKenzie, Stillman, & Gibson, 2010), health (Schneeweiss, Maclure, Carleton, Glynn, & Avorn, 2004), and voting and political behavior (Arceneaux, Gerber, & Green, 2010). See Wong, Steiner, and Anglin (2018) for a recent overview of within-study comparisons, and see Burdick-Will, Jens Ludwig, Raudenbush, Sampson, and Sharkey (2011) for a within-study comparison study that addresses child outcomes using MTO participants in Chicago and Project on Human Development in Chicago Neighborhoods subjects.

10. Unlike with MTO, where HUD set eligibility criteria for the program that included the requirement that heads of household had children, the PSID was intended to be a nationally representative sample, so it includes people without children. (We can see this in the descriptive statistics for the unweighted PSID sample, where the mean number of children is less than 1). The trimming of the PSID sample to make it look more like MTO, which reduces the sample size we use from PSID from 4,299 to 850, reduces the share of adults without children from 59% (unweighted) to 2% (trimmed).

11. The section on Data and Methods includes additional information about weighting, but we note here that our nonexperimental analyses of the PSID and MTO samples are generally unweighted, except for our models that use propensity score weights to produce more comparable groups or treatment dosages, and some of our PSID sensitivity analyses that apply attrition weights. The experimental analysis of the MTO sample applies the MTO probability weights to account for changes in random assignment ratios across randomization cohorts, survey sample selection, and two-phase interviewing.

12. The time period used depends on the year that the adult completed her long-term survey interview. For adults interviewed in 2008 (fielding began in June), we use data for even years in the 1998–2008 range (excluding 1998 for the small number of adults in the Los Angeles site who were randomized in 1998). For adults interviewed in 2009 (and the first few months of 2010, when fielding ended in April), we use data for odd years in the 1999–2009 range.

13. This measure includes tract percentage black, welfare receipt, unemployment rate, female-headed households, and share of residents under 18, in addition to tract poverty rate.

14. The lower reliability for MTO is due to standardizing based on the PSID. Were the MTO measures standardized on the MTO data, the reliability of the index in the MTO data would be 0.76.

15. These are the MTO compliance rates among female MTO adult respondents in the long-term follow-up survey.

16. The greatest threat to internal validity is potential bias from sample attrition. Extensive effort was made to achieve an effective response rate of over 89% for the long-term MTO follow-up study, which was quite similar for each of the three randomly assigned MTO groups (Sanbonmatsu et al., 2011). In addition, the results do not seem to be very sensitive to survey nonresponse, as judged by trying alternative techniques such as weighting and multiple imputation, and through comparisons to estimates of MTO employment and earnings impacts using administrative unemployment insurance data covering the full MTO sample.

17. These assumptions imply that the experience of housing counseling and search induced by assignment to a treatment group did not affect later outcomes if that household did not make a program move. We believe that this assumption is probably not strictly true, but we believe that effects of housing counseling are likely to be orders of magnitude smaller than the effects of moving. When Equations (1)(1) $D=Z{\pi }_1+X{\beta }_1+{\epsilon }_1$(1) and (2(2) $Y=Z{\pi }_2+X{\beta }_2+{\epsilon }_2$(2) ) are estimated using ordinary least squares, this is numerically identical to a two-stage least squares regression of Y on D, with Z used as an instrumental variable for D.

18. Note that because moving with a voucher in MTO is highly correlated with future exposure to neighborhood poverty once we weight the data to align neighborhood poverty gaps, the MTO experimental TOT estimate is equivalent to an instrumental variable analysis that uses the MTO randomization to instrument for neighborhood poverty.

19. One might worry that we are simply matching the treatment dose on neighborhood differences for one neighborhood attribute at a time, so that if it is actually the combination of neighborhood attributes that a family experiences, we might miss that in our analysis. However, below we also present results that focus on matching the MTO and PSID samples on the dosage of the index of multiple neighborhood disadvantage measures suggested by Sampson et al. (2008).

20. PSID analyses are unweighted, except for the models in which the PSID and MTO data have been propensity-score reweighted. That is, we do not use the PSID sampling weights in our regression analyses.

21. We focus on basic demographics for which we can achieve some balance between the samples.

22. We choose a 25% threshold for distinguishing the lower and higher neighborhood poverty groups because this threshold allows us to better align the MTO and PSID samples through weighting.

23. This is the case even when we narrow the PSID sample to African American and Hispanic women with incomes below 200% of the poverty line.

24. Table 1 also shows that MTO respondents live in different regions than PSID respondents, especially when we limit the PSID sample to those living in high-poverty neighborhoods or to African American or Hispanic women. PSID respondents are more likely to live in the South than MTO respondents are. This is a result of the city-stratified sampling design of MTO and subsampling by poverty status in the PSID. Unfortunately, our weighting procedures are unable to correct for such large regional differences (see Online Appendix Table A4). All models control for region. Because the discrepancies between the PSID and experimental MTO estimates are similar to the discrepancies between the MTO experimental and MTO nonexperimental estimates, we are not concerned that the particularities of MTO study cities are driving the PSID–MTO discrepancies.

25. Table 2 uses neighborhood measures that are duration weighted using all MTO addresses, whereas for analyses comparing MTO and PSID below we limit the MTO data to biannual measures to match PSID.

26. Online Appendix Table A1 shows correlations between the tract poverty rate and the other neighborhood characteristics. Percentage poor is highly correlated with welfare receipt, unemployment, and female-headed households, is moderately correlated with percentage youth, and exhibits a low correlation with percentage black in the MTO sample. This explains how MTO reductions in neighborhood poverty are accompanied by reductions in welfare receipt, unemployment, and female-headed households but by smaller reductions in percentage black.

27. Parallel analyses of two other adult outcomes, mental and physical health, are provided in Online Appendix B. These are outcomes for which the existing research finds similar results in experimental and observational studies.

28. Tables 2 and 3 (row A1) show different mean poverty rate TOTs because Table 2 uses the duration-weighted MTO measure, making use of the full MTO address data over the entire follow-up period. Table 3’s measure of neighborhood poverty exposure, in contrast, is constructed so that the MTO sample is parallel to the PSID, so it uses only biannual addresses during the present study’s follow-up time.

29. In row A1 of Table 3, the low-poverty group mean is the experimental complier mean, the poverty rate differential is the TOT for the effect on the mean neighborhood poverty rate during follow-up, and the high-poverty group mean is the implied experimental control complier mean (the experimental complier mean minus the TOT).

30. Given the size of the standard errors around the two sets of estimates, the difference between the estimated effects in MTO vs. PSID, of about 0.11 standard deviations, is not statistically significant.

31. We made an analogous comparison between experimental and nonexperimental estimates within the MTO study using data from 2002 (4–7 years after random assignment), instead of the data collected from 2008 to 2010 (10–15 years after random assignment) focused on in this article, and found a similar but weaker pattern of experimental and nonexperimental results. The labor market in 2002 exhibited lower unemployment than that in 2008–2010, enhancing the external validity of results from the earlier data. We focus on the later data because they have equivalent internal validity and were collected more than twice as long after random assignment—thus providing more comprehensive information about the effects of living in different neighborhoods over time.

32. Online Appendix Tables A5 and A6 track the means and 25th and 75th percentiles of the tract poverty rate, concentrated disadvantage index, years in low-poverty areas, and number of moves across census tracts, as we limit and weight the samples in Table 3.

33. Our analysis purposely aims to analyze a traditional nonexperimental method in which the comparison is made between people above and below thresholds of the neighborhood poverty rate. By its nature, that method has less variance in the neighborhood poverty rate of the low-poverty group for the nonexperimental method than for the experimental method; the subset of the experimental low-poverty group compliers with neighborhood poverty rate above the threshold has no counterpart in the nonexperimental low-poverty group. The reason this might matter is that the effect of the neighborhood poverty rate might be nonlinear. We address that as one of our candidate hypotheses below, but do not find evidence supporting it. The reason, as shown in Online Appendix Table A6, is that the bulk of the distribution is quite similar: The 25th and 75th percentiles for neighborhood poverty, in panel D, for the MTO TOT are .128 and .250 versus .152 and .219, in panel E, for the MTO nonexperimental comparison (a little narrower at the top and bottom but unlikely to generate important nonlinear effects).

34. This interpretation is also consistent with the comparison of the unweighted and weighted MTO experimental group means in Online Appendix Tables A4 and A5, respectively, which show the weighting changed the baseline means for number of children, welfare receipt, education, income, and especially baseline employment. Online Appendix Table A5 also shows that the weighting has not changed the covariate balance between the experimental and control groups within the MTO experimental estimate sample.

35. Throughout our discussion of the results from the weighting procedure, we have focused on aligning the gaps and levels of the means on the neighborhood poverty rate, the concentrated disadvantage index, the number of years in low-poverty neighborhoods, and the number of residential moves. Online Appendix Table A6 shows the 25th and 75th percentiles of the distributions of these four variables when various weighting procedures are applied. This table shows that whenever we are able to balance the means, we are able to balance those percentiles as well.

36. We also examined the results of specifications including fixed effects for quintiles of the propensity score. In most cases, the results were similar to those reported here. In a few cases, the standard errors were much larger and the estimates became uninformative.

Additional information

Funding

Support for this research was provided by a contract from the U.S. Department of Housing and Urban Development (HUD; C-CHI-00808) and grants from the National Science Foundation (SES-0527615), National Institute of Mental Health (R01-MH077026), National Institute on Aging (P30-AG012810, R01-AG031259, and P01-AG005842-22S1), Smith Richardson Foundation (Grant no. 20161249), University of Chicago’s Center for Health Administration Studies, Russell Sage Foundation, and Robert Wood Johnson Foundation. The collection of the PSID data used in this study was partly supported by the National Institutes of Health (under grant no. R01 HD069609) and the National Science Foundation (under award number 1157698). Outstanding assistance with the data preparation and analysis was provided by Joe Amick, Ryan Gillette, Ray Yun Gou, Ijun Lai, Jordan Marvakov, Nicholas Potter, Nathan Weil, Fanghua Yang, Sabrina Yusuf, and Michael Zabek. The survey data collection effort was led by Nancy Gebler of the University of Michigan’s Survey Research Center, under subcontract to our research team. The views expressed in this work are those of the authors and should not be interpreted as those of the Congressional Budget Office or HUD. A restricted-access version of the MTO data used in this article will be provided to the U.S. Department of Housing and Urban Development (HUD). At the time of writing, information about access to the MTO data could be found here: http://www.nber.org/mtopublic/. Researchers may apply for access to PSID restricted data on census tracts through the University of Michigan (see http://simba.isr.umich.edu/restricted/RestrictedUse.aspx for more information).

Notes on contributors

David J. Harding

David J. Harding is Professor of Sociology at University of California Berkeley.

Lisa Sanbonmatsu

Lisa Sanbonmatsu is Director of Research at the Center for Education Policy Research at Harvard University.

Greg J. Duncan

Greg J. Duncan is Distinguished Professor at the School of Education at UC Irvine.

Lisa A. Gennetian

Lisa A. Gennetian Pritzker Associate Professor of Early Learning Policy Studies in the Sanford School of Public Policy at Duke University.

Lawrence F. Katz

Lawrence F. Katz is Elisabeth Allison Professor of Economics at Harvard University a Research Associate at the National Bureau of Economic Research.

Ronald C. Kessler

Ronald C. Kessler is the McNeil Family Professor of Health Care Policy at Harvard Medical School.

Jeffrey R. Kling

Jeffrey R. Kling is Research Director at the Congressional Budget Office and a faculty research fellow at the National Bureau of Economic Research.

Matthew Sciandra

Matthew Sciandra is a Research Public Health Analyst with RTI International.

Jens Ludwig

Jens Ludwig is the Edwin A. and Betty L. Bergman Distinguished Service Professor at the University of Chicago and Research Associate at the National Bureau of Economic Research.