Propensity Score Matching for Education Data: Worked Examples

References

• Aiken, L. R. (1974). Two scales of attitude toward mathematics. Journal for Research in Mathematics Education, 5, 67–71. doi: 10.2307/748616
   Google Scholar
• Aiken, L. S., West, S. G., & Millsap, R. E. (2008). Doctoral training in statistics, measurement, and methodology in psychology: Replication and extension of Aiken, West, Sechrest, and Reno’s (1990) survey of PhD programs in North America. American Psychologist, 63(1), 32–50. doi: 10.1037/0003-066X.63.1.32
   PubMed Web of Science ®Google Scholar
• American College Testing Program (1989). ASSET technical manual. Iowa City, IA: Author.
   Google Scholar
• Antonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. (2010). On making causal claims: A review and recommendations. The Leadership Quarterly, 21(6), 1086–1120. doi: 10.1016/j.leaqua.2010.10.010
   Web of Science ®Google Scholar
• Austin, P. C. (2009). Some methods of propensity-score matching had superior performance to others: Results of an empirical investigation and Monte Carlo simulations. Biometrical Journal, 51(1), 171–184. doi: 10.1002/bimj.200810488
   PubMed Web of Science ®Google Scholar
• Austin, P. C. (2011). Optimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studies. Pharmaceutical Statistics, 10(2), 150–161. doi: 10.1002/pst.433
   PubMed Web of Science ®Google Scholar
• Austin, P. C. (2014). A comparison of 12 algorithms for matching on the propensity score. Statistics in Medicine, 33(6), 1057–1069. doi: 10.1002/sim.6004
   PubMed Web of Science ®Google Scholar
• Austin, P. C., & Mamdani, M. M. (2006). A comparison of propensity score methods: A case-study estimating the effectiveness of post-AMI statin use. Statistics in Medicine, 25(12), 2084–2106.
   PubMed Web of Science ®Google Scholar
• Austin, P. C., Grootendorst, P., &Anderson, G. M. (2007). A comparison of the ability if different propensity score models to balance measured variables between treated and untreated subjects: A Monte Carlo study. Statistics in Medicine, 26, 734–753.
   PubMed Web of Science ®Google Scholar
• Barratt, W. (2006). The Barratt simplified measure of social status (BSMSS) measuring SES. Retrieved from: http://www.marson-and-associates.com/syllabi/151BSMSS.pdf
   Google Scholar
• Beaujean, A. A., Hull, D. M., Sheng, Y., Worrell, F. C., Bolen, J., & Verdisco, A. E. (in press). Psychometric properties of the Shipley Block Design task: A study with Jamaican young adults. The Journal of Psychoeducational Assessment, doi: 10.1177/0734282916643439
   Web of Science ®Google Scholar
• Burnham, K. P., Anderson, D. R., & Huyvaert, K. P. (2011). AIC model selection and multimodel inference in behavioral ecology: Some background, observations, and comparisons. Behavioral Ecology and Sociobiology, 65, 23–35. doi: 10.1007/s00265-010-1029-6
   Web of Science ®Google Scholar
• Cochran, W. G. (1965). The planning of observational studies of human populations. Journal of the Royal Statistical Society, 128, 134–155.
   Google Scholar
• Cochran, W. G. (1968). The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics, 24(2), 295–313.
   PubMed Web of Science ®Google Scholar
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Mahwah, NJ: Erlbaum.
   Google Scholar
• Collis, K., & Romberg, T. (1992). Collis-Romberg mathematical problem solving profiles. Hawthorne Victoria, Australia: Australian Council for Educational Research (ACER).
   Google Scholar
• Cox, B. E., McIntosh, K., Reason, R. D., & Terenzini, P. T. (2014). Working with missing data in higher education research: A primer and real-world example. The Review of Higher Education, 37(3), 377–402.
   Web of Science ®Google Scholar
• D’Agostino, R. B. (1998). Propensity score methods for bias reduction in the comparison of treatment to non-randomized control group. Statistics in Medicine, 17(19), 2265–2281.
   PubMed Web of Science ®Google Scholar
• D’Agostino, R. B. (2004). Propensity score estimation with missing data. In A. Gelman & X. L. Meng (Eds.) Missing data and Bayesian methods in practice: Contributions by Donald Rubin’s statistical family (pp. 163–174). New York, NY: John Wiley & Sons.
   Google Scholar
• D’Agostino, R. B. (2007). Propensity scores in cardiovascular research. Circulation, 115(17), 2340–2343.
   PubMed Web of Science ®Google Scholar
• D'Agostino, R. B., & Rubin, D. B. (2000). Estimating and using propensity scores with partially missing data. Journal of the American Statistical Association, 95(451), 749–759.
   Web of Science ®Google Scholar
• Dehejia, R. H., & Wahba, S. (2002). Propensity score-matching methods for nonexperimental causal studies. The Review of Economics and Statistics, 84(1), 151–161.
   Web of Science ®Google Scholar
• Foster, E. M. (2010). Causal inference and developmental psychology. Developmental Psychology, 46(6), 1454–1480. doi: 10.1037/a0020204
   PubMed Web of Science ®Google Scholar
• Frölich, M. (2004). Finite-sample properties of propensity-score matching and weighting estimators. Review of Economics and Statistics, 86(1), 77–90.
   Web of Science ®Google Scholar
• Gelman, A., Su, Y.-S., Yajima, M., Hill, J., Pittau, M. G., & Kerman, J. (2014). Arm: Data analysis using regression and multilevel/hierarchical models. (Version 1.7-07) [Computer software]. New York, NY. Retrieved from http://cran.r-project.org/web/packages/arm/index.html.
   Google Scholar
• Glynn, R. J., Schneeweiss, S., & Sturmer, T. (2006). Indications for propensity scores and review of their use in pharmacoepidemiology. Basic Clinical Pharmacology Toxicology, 98(3), 253–259.
   PubMed Web of Science ®Google Scholar
• Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., & Gough, H. G. (2006). The international personality item pool and the future of public-domain personality measures. Journal of Research in Personality, 40(1), 84–96. doi: 10.1016/j.jrp.2005.08.007
   Web of Science ®Google Scholar
• Harder, V. S., Stuart, E. A., & Anthony, J. C. (2010). Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychological Methods, 15(3), 234–249. doi: 10.1037/a0019623
   PubMed Web of Science ®Google Scholar
• Henson, R. K., Hull, D. M., & Williams, C. S. (2010). Methodology in our education research culture. Educational Researcher, 39(3), 229–240. doi: 10.3102/0013189x10365102
   Web of Science ®Google Scholar
• Henson, R. K., & Williams, C. S. (2006). April). Doctoral training in research methodology: A national survey of education-related degrees. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
   Google Scholar
• Hinkelmann, K., & Kempthorne, O. (2007). Design and analysis of experiments, volume 1: Introduction to experimental design (2nd ed.). New York, NY: John Wiley & Sons.
   Google Scholar
• Ho, D., Imai, K., King, G., & Stuart, E. A. (2007). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis, 15(03), 199–236. doi: 10.1093/pan/mp1013.
   Google Scholar
• Ho, D., Imai, K., King, G., & Stuart, E. A. (2011). MatchIt: Nonparametric preprocessing for parametric causal inference. Journal of Statistical Software, 42(8), 1–28. Retrieved from http://www.jstatsoft.org/v42/i08/paper
   Web of Science ®Google Scholar
• Holland, J. L. (1996). Exploring careers with a typology: What we have learned and some new directions. American Psychologist, 51(4), 397–406.
   Web of Science ®Google Scholar
• Hong, G. (2012). Marginal mean weighting through stratification: A generalized method for evaluating multivalued and multiple treatments with nonexperimental data. Psychological Methods, 17(1), 44–60. doi: 10.1037/a0024918
   PubMed Web of Science ®Google Scholar
• Huppler Hullsiek, K., & Louis, T. A. (2002). Propensity score modeling strategies for the causal analysis of observational data. Biostatistics, 3(2), 179–193.
   PubMed Web of Science ®Google Scholar
• Imai, K., King, G., & Lau, O. (2010). Zelig: Everyone’s statistical software. Retrieved from http://gking.harvard.edu/zelig.
   Google Scholar
• Imbens, G. W. (2003). Sensitivity to exogeneity assumptions in program evaluation. American Economic Review, 93(2), 126–132.
   Web of Science ®Google Scholar
• Institute of Education Sciences (IES). (2006). WWC study design classification. Retrieved from https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_version1_standards.pdf
   Google Scholar
• Institute of Education Sciences (IES). (2017). What works clearinghouse standards handbook (Version 4.0). Washington, DC: Author.
   Google Scholar
• Joffe, M. M., & Rosenbaum, P. R. (1999). Invited commentary: Propensity scores. American Journal of Epidemiology, 150(4), 327–333.
   PubMed Web of Science ®Google Scholar
• Keller, B., & Tipton, E. (2016). Propensity score analysis in R: A software review. Journal of Educational and Behavioral Statistics, 41(3), 326–348. doi: 10.3102/1076998616631744
   Web of Science ®Google Scholar
• Leech, N. L., & Goodwin, L. D. (2008). Building a methodological foundation: Doctoral level methods courses in colleges of education. Research in the Schools, 15(1), 1–8.
   Google Scholar
• Leite, W. (2017). Practical propensity score methods using R. Los Angeles: Sage Publications Inc.
   Google Scholar
• Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data (2nd ed.). Hoboken, NJ: John Wiley.
   Google Scholar
• Luellen, J. K., Shadish, W. R., & Clark, M. H. (2005). Propensity scores: An introduction and experimental test. Evaluation Review, 29(6), 530–558.
   PubMed Web of Science ®Google Scholar
• Murnane, R. J., & Willett, J. B. (2011). Methods matter: Improving causal inference in educational and social science research. New York: Oxford University Press.
   Google Scholar
• Pearl, J. (2009). Causality: Models, reasoning, and inference (2nd ed.). New York, NY: Cambridge University Press.
   Google Scholar
• Perham, H. J. (2010). Quantitative training of doctoral school psychologists: Statistics, measurement, and methodology curriculum. Unpublished master's thesis, Arizona State University, Tempe, AZ.
   Google Scholar
• Peugh, J. L., & Enders, C. K. (2004). Missing data in educational research: A review of reporting practices and suggestions for improvement. Review of Educational Research, 74, 525–556.
   Web of Science ®Google Scholar
• R Development Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.
   Google Scholar
• Rosenbaum, P. R. (1995). Quantiles in nonrandom samples and observational studies. Journal of the American Statistical Association, 90(432), 1424–1431.
   Web of Science ®Google Scholar
• Rosenbaum, P. R. (2002). Observational studies (2nd ed.). New York: Springer.
   Google Scholar
• Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1),41–55.
   Web of Science ®Google Scholar
• Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79(387), 516–524.
   Web of Science ®Google Scholar
• Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.
   Web of Science ®Google Scholar
• Rubin, D. B. (1986). Comment: Which ifs have causal answers? Journal of American Statistical Association, 81, 961–962.
   Web of Science ®Google Scholar
• Rubin, D. B. (1997). Estimating causal effects from large data sets using propensity scores. Annals of Internal Medicine, 127(8_Part_2), 757–763.
   PubMed Web of Science ®Google Scholar
• Rubin, D. B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services and Outcomes Research Methodology, 2, 169–188.
   Google Scholar
• Schneider, B., Carnoy, M., Kilpatrick, J., Schmidt, W. H., & Shavelson, R. J. (2007). Estimating causal effects: Using experimental and observational designs. Washington, DC: American Educational Research Association.
   Google Scholar
• Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
   Google Scholar
• Shipley, W. C., Gruber, C., Martin, T., & Klein, A. M. (2010). Shipley institute of living scale (2nd ed.). Los Angeles, CA: Western Psychological Services.
   Google Scholar
• Storer, B., & Kim, C. (1990). Exact properties of some exact test statistics for comparing two binomial proportions.. Journal of the American Statistical Association, 55, 146–155.
   Google Scholar
• Stouffer, S. A. (1950). Some observations on study design. American Journal of Sociology, 55(4), 355–361.
   Web of Science ®Google Scholar
• Stuart, E. A. (2007). Estimating causal effects using school-level data sets. Educational Researcher, 36(4), 187–198.
   Google Scholar
• Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science: A Review Journal of the Institute of Mathematical Statistics, 25(1), 1–21.
   PubMed Web of Science ®Google Scholar
• Stuart, E. A., & Rubin, D. B. (2007). Best practices in quasi-experimental designs: Matching methods for causal inference. In J. Osborne (Ed.), Best practices in quantitative social science (pp. 155–176). Thousand Oaks, CA: Sage Publications.
   Google Scholar
• Stürmer, T., Joshi, M., Glynn, R. J., Avorn, J., Rothman, K. J., & Schneeweiss, S. (2006). A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. Journal of Clinical Epidemiology, 59, 437–447.
   PubMed Web of Science ®Google Scholar
• Thoemmes, F. J., & Kim, E. S. (2011). A systematic review of propensity score methods in the social sciences. Multivariate Behavioral Research, 46(1), 90–118. doi: 10.1080/00273171.2011.540475
   PubMed Web of Science ®Google Scholar
• Thoemmes, F. J., & West, S. G. (2011). The use of propensity scores for nonrandomized designs with clustered data. Multivariate Behavioral Research, 46(3), 514–543. doi: 10.1080/00273171.2011.569395
   PubMed Web of Science ®Google Scholar
• U.S. Department of Education. (2016). Non-regulatory guidance: Using evidence to strengthen education investments. Retrieved from https://www2.ed.gov/policy/elsec/leg/essa/guidanceuseseinvestment.pdf
   Google Scholar
• Welch, B. L. (1947). The generalization of "Student's" problem when several different population variances are involved. Biometrika, 34(1-2), 28–35. doi: 10.1093/biomet/34.1-2.28
   PubMed Web of Science ®Google Scholar
• West, S., Biesanz, J. C., & Pitts, S. C. (2000). Causal inference and generalization in field settings: Experimental and quasi-experimental designs. In C. M. Reis, H. T. Judd (Ed.), Handbook of research methods in social and personality psychology (pp. 40–84). Cambridge, UK: Cambridge University Press.
   Google Scholar
• West, S. G., Cham, H., Thoemmes, F., Renneberg, B., Schulze, J., & Weiler, M. (2014). Propensity scores as a basis for equating groups: Basic principles and application in clinical treatment outcome research. Journal of Consulting and Clinical Psychology, 82(5), 906–919. doi: 10.1037/a0036387
   PubMed Web of Science ®Google Scholar
• What Works Clearinghouse (WWC). (n.d.). About us. Retrieved from http://ies.ed.gov/ncee/wwc/aboutus/.
   Google Scholar
• Wilcox, R. R. (2003). Applying contemporary statistical techniques San Diego, CA: Academic Press.
   Google Scholar
• Winship, C., & Morgan, S. L. (2014). Counterfactuals and causal inference: Methods and principles for social research (2nd ed.). New York: Cambridge University.
   Google Scholar