Causal Quartets: Different Ways to Attain the Same Average Treatment Effect

References

• Angrist, J. D., and Pischke, J. S. (2009), Mostly Harmless Econometrics, Princeton, NJ: Princeton University Press.
   Google Scholar
• Anscombe, F. J. (1973), “Graphs in Statistical Analysis,” American Statistician, 27, 17–21. DOI: 10.2307/2682899.
   Web of Science ®Google Scholar
• Baribault, B., Donkin, C., Little, D. R., Trueblood, J. S., Oravecz, Z., Van Ravenzwaaij, D., and Vandekerckhove, J. (2018), “Metastudies for Robust Tests of Theory,” Proceedings of the National Academy of Sciences, 115, 2607–2612. DOI: 10.1073/pnas.1708285114.
   PubMed Web of Science ®Google Scholar
• Bitler, M. P., Gelbach, J. B., and Hoynes, H. W. (2006), “What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments,” American Economic Review, 96, 988–1012. DOI: 10.1257/aer.96.4.988.
   Web of Science ®Google Scholar
• Bryan, C. J., Tipton, E., and Yeager, D. S. (2021), “Behavioural Science is Unlikely to Change the World Without A Heterogeneity Revolution,” Nature Human Behavior, 5, 980–989. DOI: 10.1038/s41562-021-01143-3.
   PubMed Web of Science ®Google Scholar
• Buhl-Wiggers, J., Kerwin, J., Muñoz, J. S., Smith, J., and Thornton, R. (2023), “Some Children Left Behind: Variation in the Effects of an Educational Intervention,” Journal of Econometrics. DOI: 10.1016/j.jeconom.2021.12.010.
   Google Scholar
• Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., and Munafò, M. R. (2013), “Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience,” Nature Reviews Neuroscience, 14, 365–376. DOI: 10.1038/nrn3475.
   PubMed Web of Science ®Google Scholar
• Chatterjee, S., and Firat, A. (2007), “Generating Data with Identical Statistics but Dissimilar Graphics: A Follow Up to the Anscombe Dataset,” American Statistician, 61, 248–254. DOI: 10.1198/000313007X220057.
   Web of Science ®Google Scholar
• Cronbach, L. J. (1982), Designing Evaluations of Educational and Social Programs, San Francisco: Jossey-Bass.
   Google Scholar
• Gelman, A. (2014), “When There’s a Lot of Variation, It Can Be a Mistake to Make Statements About “Typical Attitudes,” Statistical Modeling, Causal Inference, and Social Science. Available at https://statmodeling.stat.columbia.edu/2014/10/08/var/
   Google Scholar
• ——- (2018), “You Need 16 Times the Sample Size to Estimate an Interaction than to Estimate a Main Effect,” Statistical Modeling, Causal Inference, and Social Science. Available at https://statmodeling.stat.columbia.edu/2018/03/15/need16/
   Google Scholar
• Gelman, A., Hullman, J., and Kennedy, L. (2023), “Thinking about Variation When Hypothesizing a Plausible Average Treatment Effect,” Technical Rreport, Department of Statistics, Columbia University.
   Google Scholar
• Heckman, J., and Smith, J. (1995), “Assessing the Case for Social Experiments,” Journal of Economic Perspectives, 9, 85–110. DOI: 10.1257/jep.9.2.85.
   Web of Science ®Google Scholar
• Heckman, J., Smith, J., and Clements, N. (1997), “Making the Most Out of Programme Evaluations and Social Experiments: Accounting for Heterogeneity in Programme Impacts,” Review of Economic Studies, 64, 487–535. DOI: 10.2307/2971729.
   Web of Science ®Google Scholar
• Heiler, P., and Knaus, M. (2022), “Effect or Treatment Heterogeneity? Policy Evaluation with Aggregated and Disaggregated Treatments,” available at https://arxiv.org/abs/2110.01427
   Google Scholar
• Hill, J. L. (2011), “Bayesian Nonparametric Modeling for Causal Inference,” Journal of Computational and Graphical Statistics, 20, 217–240. DOI: 10.1198/jcgs.2010.08162.
   Web of Science ®Google Scholar
• Hofman, J. M., Goldstein, D. G., and Hullman, J. (2020), “How Visualizing Inferential Uncertainty Can Mislead Readers About Treatment Effects in Scientific Results,” in Proceedings of the 2020 ACM Conference on Human Factors in Computing Systems (CHI ’20), p. 327.
   Google Scholar
• Hullman, J., Kay, M., Kim, Y., and Shrestha, S. (2017), “Imagining Replications: Graphical Prediction & Discrete Visualizations Improve Recall & Estimation of Effect Uncertainty,” IEEE Transactions on Visualization and Computer Graphics, 24, 446–456. DOI: 10.1109/TVCG.2017.2743898.
   PubMed Web of Science ®Google Scholar
• Imbens, G. W., and Angrist, J. (1994), “Identification and Estimation of Local Average Treatment Effects,” Econometrica, 62, 467–475. DOI: 10.2307/2951620.
   Web of Science ®Google Scholar
• Kim, Y., Hofman, J. M., and Goldstein, D. G. (2022), “Putting Scientific Results in Perspective: Improving the Communication of Standardized Effect Sizes,” Proceedings of the 2022 ACM Conference on Human Factors in Computing Systems (CHI ’22), p. 625.
   Google Scholar
• Kim, Y., Reinecke, K., and Hullman, J. (2017), “Explaining the Gap: Visualizing One’s Predictions Improves Recall and Comprehension of Data,” in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 1375–1386.
   Google Scholar
• Matejka, J., and Fitzmaurice, G. (2017), “Same Stats, Different Graphs: Generating Datasets with Varied Appearance and Identical Statistics through Simulated Annealing,” in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), pp. 1290–1294.
   Google Scholar
• Shadish, W. R., Cook, T. D., and Campbell, D. T. (2002), Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Boston: Houghton Mifflin.
   Google Scholar
• Wager, S., and Athey, S. (2018), “Estimation and Inference of Heterogeneous Treatment Effects Using Random Forests,” Journal of the American Statistical Association, 113, 1228–1242. DOI: 10.1080/01621459.2017.1319839.
   Web of Science ®Google Scholar
• Yarkoni, T. (2022), “The Generalizability Crisis,” Behavioral and Brain Sciences, 45, E1. DOI: 10.1017/S0140525X21001758.
   Web of Science ®Google Scholar
• Zidar, O. (2019), “Tax Cuts for Whom? Heterogeneous Effects of Income Tax Changes on Growth and Employment,” Journal of Political Economy, 127, 1437–1472. DOI: 10.1086/701424.
   Web of Science ®Google Scholar