The Implications of Population Changes on Generalization and Study Design

References

• Bhattacharyya, A. (1943). On a measure of divergence between two statistical populations defined by their probability distributions. Bulletin of Calcutta Mathematical Society, 35, 99–109.
   Google Scholar
• Buchanan, A. L., Hudgens, M. G., Cole, S. R., Mollan, K. R., Sax, P. E., Daar, E. S., Adimora, A. A., Eron, J. J., & Mugavero, M. J. (2018). Generalizing evidence from randomized trials using inverse probability of sampling weights. Journal of the Royal Statistical Society: Series A (Statistics in Society), 181(4), 1193–1209. https://doi.org/10.1111/rssa.12357
   PubMed Web of Science ®Google Scholar
• Chan, W. (2017). Partially identified treatment effects for generalizability. Journal of Research on Educational Effectiveness, 10(3), 646–669. https://doi.org/10.1080/19345747.2016.1273412
   Web of Science ®Google Scholar
• Chan, W. (2018). Applications of small area estimation to generalization with subclassification by propensity scores. Journal of Educational and Behavioral Statistics, 43(2), 182–224. https://doi.org/10.3102/1076998617733828
   Web of Science ®Google Scholar
• Chan, W. (2020). The effect of distributional overlap on the precision gain of bounds under subclassification. American Journal of Evaluation. Advance online publication. http://arxiv.org/abs/2101.09596
   Google Scholar
• Cole, S. R., & Stuart, E. A. (2010). Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial. American Journal of Epidemiology, 172(1), 107–115. https://doi.org/10.1093/aje/kwq084
   PubMed Web of Science ®Google Scholar
• Djebbari, H., & Smith, J. (2008). Heterogeneous impacts of PROGRESA. Journal of Econometrics, 145(1–2), 64–80. https://doi.org/10.1016/j.jeconom.2008.05.012
   Web of Science ®Google Scholar
• Kaput, J. J. (1997). Rethinking calculus: Learning and thinking. The American Mathematical Monthly, 104(8), 731–737. https://doi.org/10.1080/00029890.1997.11990710
   Web of Science ®Google Scholar
• Kern, H. L., Stuart, E. A., Hill, J., & Green, D. P. (2016). Assessing methods for generalizing experimental impact estimates to target populations. Journal of Research on Educational Effectiveness, 9(1), 103–127. https://doi.org/10.1080/19345747.2015.1060282
   PubMed Web of Science ®Google Scholar
• Konstantopoulos, S., Miller, S., & van der Ploeg, A. (2013). The impact of Indiana’s system of interim assessments on mathematics and reading achievement. Educational Evaluation and Policy Analysis, 35(4), 481–499. https://doi.org/10.3102/0162373713498930
   Web of Science ®Google Scholar
• O’Muircheartaigh, C., & Hedges, L. V. (2014). Generalizing from unrepresentative experiments: A stratified propensity score approach. Journal of the Royal Statistical Society: Series C (Applied Statistics), 63, 195–210.
   Web of Science ®Google Scholar
• Olsen, R. B., Orr, L. L., Bell, S. H., & Stuart, E. A. (2013). External validity in policy evaluations that choose sites purposively. Journal of Policy Analysis and Management, 32(1), 107–121. https://doi.org/10.1002/pam.21660
   PubMed Web of Science ®Google Scholar
• Roschelle, J., Shechtman, N., Tatar, D., Hegedus, S., Hopkins, B., Empson, S., Knudsen, J., & Gallagher, L. P. (2010). Integration of technology, curriculum, and professional development for advancing middle school mathematics: Three large-scale studies. American Educational Research Journal, 47(4), 833–878. https://doi.org/10.3102/0002831210367426
   Web of Science ®Google Scholar
• Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41
   Web of Science ®Google Scholar
• Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
   Google Scholar
• Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science: A Review Journal of the Institute of Mathematical Statistics, 25(1), 1–21. https://doi.org/10.1214/09-STS313
   PubMed Web of Science ®Google Scholar
• Stuart, E. A., Cole, S. R., Bradshaw, C. P., & Leaf, P. J. (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174(2), 369–386. https://doi.org/10.1111/j.1467-985X.2010.00673.x
   Web of Science ®Google Scholar
• Texas Education Agency. (2007). Standards for the operation of school district disciplinary alternative education programs. http://ritter.tea.state.tx.us/rules/tac/chapter103/ch103cc.html
   Google Scholar
• Texas Education Agency. (2018). Enrollment in Texas public schools, 2017–2018. https://tea.texas.gov/sites/default/files/enroll_2017-18.pdf
   Google Scholar
• Tipton, E. (2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38(3), 239–266. https://doi.org/10.3102/1076998612441947
   Web of Science ®Google Scholar
• Tipton, E. (2014). How generalizable is your experiment? Comparing a sample and population through a generalizability index. Journal of Educational and Behavioral Statistics, 39(6), 478–501. https://doi.org/10.3102/1076998614558486
   Web of Science ®Google Scholar
• Tipton, E., Hallberg, K., Hedges, L. V., & Chan, W. (2017). Implications of small samples for generalization: Adjustments and rules of thumb. Evaluation Review, 41(5), 472–505. https://doi.org/10.1177/0193841X16655665
   PubMed Web of Science ®Google Scholar
• Tipton, E., Hedges, L. V., Vaden-Kiernan, M., Borman, G., Sullivan, K., & Caverly, S. (2014). Sample selection in randomized experiments: A new method using propensity score stratified sampling. Journal of Research on Educational Effectiveness, 7(1), 114–135. https://doi.org/10.1080/19345747.2013.831154
   Web of Science ®Google Scholar