A Comparison of Three Methods for Providing Local Evidence to Inform School and District Budget Decisions

References

• Acosta, J., Chinman, M., Ebener, P., Malone, P. S., Phillips, A., & Wilks, A. (2019). Evaluation of a whole-school change intervention: Findings from a two-year cluster-randomized trial of the restorative practices intervention. Journal of Youth and Adolescence, 48(5), 876–890. https://doi.org/10.1007/s10964-019-01013-2
   PubMed Web of Science ®Google Scholar
• Asen, R., Gurke, D., Conners, P., Solomon, R., & Gumm, E. (2013). Research evidence and school board deliberations: Lessons from three Wisconsin school districts. Educational Policy, 27(1), 33–63. https://doi.org/10.1177/0895904811429291
   Web of Science ®Google Scholar
• Athey, S., & Imbens, G. W. (2017). The state of applied econometrics: Causality and policy evaluation. Journal of Economic Perspectives, 31(2), 3–32. https://doi.org/10.1257/jep.31.2.3
   PubMed Web of Science ®Google Scholar
• Athey, S., & Imbens, G. W. (2018). Design-based analysis in difference-in-differences settings with staggered adoption. National Bureau of Economic Research. No. w24963
   Google Scholar
• Augustine, C. H., Engberg, J., Grimm, G. E., Lee, E., Wang, E. L., Christianson, K., & Joseph, A. A. (2018). Can restorative practices improve school climate and curb suspensions? An evaluation of the impact of restorative practices in a mid-sized urban school district. RAND Corporation. https://www.rand.org/pubs/research_reports/RR2840.html
   Google Scholar
• Baisch, M. J., Lundeen, S. P., & Murphy, M. K. (2011). Evidence‐based research on the value of school nurses in an urban school system. Journal of School Health, 81(2), 74–80. https://doi.org/10.1111/j.1746-1561.2010.00563.x
   PubMed Web of Science ®Google Scholar
• Behrman, J. R., Gallardo-Garcia, J., Parker, S. W., Todd, P. E., & Vélez-Grajales, V. (2012). Are conditional cash transfers effective in urban areas? Evidence from Mexico. Education Economics, 20(3), 233–259. https://doi.org/10.1080/09645292.2012.672792
   PubMedGoogle Scholar
• Best, N. C., Nichols, A. O., Waller, A. E., Zomorodi, M., Pierre‐Louis, B., Oppewal, S., & Travers, D. (2021). Impact of school nurse ratios and health services on selected student health and education outcomes: North Carolina, 2011‐2016. Journal of School Health, 91(6), 473–481. https://doi.org/10.1111/josh.13025
   PubMed Web of Science ®Google Scholar
• Best, N. C., Oppewal, S., & Travers, D. (2018). Exploring school nurse interventions and health and education outcomes: An integrative review. Journal of School Nursing, 34(1), 14–27. https://doi.org/10.1177/1059840517745359
   PubMed Web of Science ®Google Scholar
• Bradshaw, C. P., Koth, C. W., Thornton, L. A., & Leaf, P. J. (2009). Altering school climate through school-wide positive behavioral interventions and supports: Findings from a group-randomized effectiveness trial. Prevention Science, 10(2), 100–115. https://doi.org/10.1007/s11121-008-0114-9
   PubMed Web of Science ®Google Scholar
• Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12(3), 133–148. https://doi.org/10.1177/1098300709334798
   Web of Science ®Google Scholar
• Brahim, N., Hensel, S., & Carpenter, K. (2018). Restorative Practices: 2017-2018 evaluation report. JCPS Department of Accountability, Research, and Systems Improvement.
   Google Scholar
• Caldas, S. J. (1993). Reexamination of input and process factor effects on public school achievement. The Journal of Educational Research, 86(4), 206–214. https://doi.org/10.1080/00220671.1993.9941832
   Web of Science ®Google Scholar
• Chang, Y., Hollands, F. M., Holmes, V. R., Shand, R., Evans, P., Blodgett, R., Wang, Y., & Head, L. (2021, April 9–12). Challenges to finding evidence for ESSA: A case study of southern urban district [Paper presentation]. American Educational Research Association Annual Meeting.
   Google Scholar
• Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review, 104(9), 2593–2632. https://doi.org/10.1257/aer.104.9.2593
   Web of Science ®Google Scholar
• Clay, M. M. (1994). Reading recovery: The wider implications of an educational innovation. Literacy, Teaching and Learning, 1, 121–141. https://www.proquest.com/openview/d1d98b40c1b3892705c69661a5131714/1?pq-origsite=gscholar&cbl=39178
   Google Scholar
• Clay, M. M. (1998). Accommodating diversity in early literacy learning. In D. R. Olson & N. Torrance (Eds.), The handbook of education and human development: New models of learning, teaching and schooling (pp. 197–217). Wiley-Blackwell.
   Google Scholar
• Clay, M. M. (2016). An observation survey of early literacy achievement (3rd ed.). Heinemann.
   Google Scholar
• Cost Analysis Standards Project. (2021). Standards for the economic evaluation of educational and social programs. American Institutes for Research. https://www.air.org/sites/default/files/Standards-for-the-Economic-Evaluation-of-Educational-and-Social-Programs-CASP-May-2021.pdf
   Google Scholar
• D’Agostino, J. V., & Harmey, S. J. (2016). An international meta-analysis of Reading Recovery. Journal of Education for Students Placed at Risk (JESPAR), 21(1), 29–46. https://doi.org/10.1080/10824669.2015.1112746
   Web of Science ®Google Scholar
• Darling‐Hammond, L., Newton, X., & Wei, R. C. (2010). Evaluating teacher education outcomes: A study of the Stanford Teacher Education Programme. Journal of Education for Teaching, 36(4), 369–388. https://doi.org/10.1080/02607476.2010.513844
   Web of Science ®Google Scholar
• Darling-Hammond, S., Fronius, T. A., Sutherland, H., Guckenburg, S., Petrosino, A., & Hurley, N. (2020). Effectiveness of restorative justice in U.S. K-12 schools: A review of quantitative research. Contemporary School Psychology, 24(3), 295–308. https://doi.org/10.1007/s40688-020-00290-0
   Google Scholar
• Darnell, T., Hager, K., & Loprinzi, P. D. (2019). The impact of school nurses in Kentucky public high schools. The Journal of School Nursing, 35(6), 434–441. https://doi.org/10.1177/1059840518785954
   PubMed Web of Science ®Google Scholar
• Every Student Succeeds Act, 20 U.S.C. § 6301. (2015). https://www.congress.gov/114/plaws/publ95/PLAW-114publ95.pdf
   Google Scholar
• Frank, S., & Hovey, D. (2014). Return on investment in education: A “system-strategy” approach. Education Resource Strategies (ERS). https://www.erstrategies.org/library/return_on_investment_in_education
   Google Scholar
• Gage, N. A., Larson, A., Sugai, G., & Chafouleas, S. M. (2016). Student perceptions of school climate as predictors of office discipline referrals. American Educational Research Journal, 53(3), 492–515. https://doi.org/10.3102/0002831216637349
   Web of Science ®Google Scholar
• GFOA. (2017). Academic return on investment: Foundations and smart practices. Government Finance Officers Association. https://gfoaorg.cdn.prismic.io/gfoaorg/142dffae-80a3-4878-8c52-6da0a87d42eb_AROIWhite+PaperFINAL.pdf
   Google Scholar
• Goodenough, A., Henry, A., & Cicchiello-Wright, H. (2018). Reading Recovery 2017-18 summary report. Jefferson County Public Schools.
   Google Scholar
• Gottfried, M. A. (2013). Quantifying the consequences of missing school: Linking school nurses to student absences to standardized achievement. Teachers College Record, 115(6), 1–30. https://doi.org/10.1177/016146811311500605
   Web of Science ®Google Scholar
• Gray, A. M., Sirinides, P. M., Fink, R., Flack, A., DuBois, T., Morrison, K., & Hill, K. (2017). Discipline in context: Suspension, climate, and PBIS in the school district of Philadelphia. CPRE Research Reports.
   Google Scholar
• Gregory, A., Clawson, K., Davis, A., & Gerewitz, J. (2016). The promise of restorative practices to transform teacher-student relationships and achieve equity in school discipline. Journal of Educational and Psychological Consultation, 26(4), 325–353. https://doi.org/10.1080/10474412.2014.929950
   Web of Science ®Google Scholar
• Harris, D. N., & Sass, T. R. (2011). Teacher training, teacher quality and student achievement. Journal of Public Economics, 957(7–8), 798–812. https://doi.org/10.1016/j.jpubeco.2010.11.009
   Google Scholar
• Hedges, L. V. (2018). Challenges in building usable knowledge in education. Journal of Research on Educational Effectiveness, 11(1), 1–21. https://doi.org/10.1080/19345747.2017.1375583
   Web of Science ®Google Scholar
• Heinrich, C. J., & Good, A. (2018). Research-informed practice improvements: Exploring linkages between school district use of research evidence and educational outcomes over time. School Effectiveness and School Improvement, 29(3), 418–445. https://doi.org/10.1080/09243453.2018.1445116
   Web of Science ®Google Scholar
• Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), 172–177. https://doi.org/10.1111/j.1750-8606.2008.00061.x
   Web of Science ®Google Scholar
• Hollands, F., Pan, Y., & Escueta, M. (2019). What is the potential for applying cost-utility analysis to facilitate evidence-based decision-making in schools? Educational Researcher, 48(5), 287–295. https://doi.org/10.3102/0013189X19852101
   Web of Science ®Google Scholar
• Hollands, F. M., Leach, S. M., Shand, R., Head, L., Wang, Y., Dossett, D., Hensel, S., … Hensel, S. (2022). Restorative Practices: Using local evidence on costs and student outcomes to inform school district decisions about behavioral interventions. Journal of School Psychology, 92, 188–208. https://doi.org/10.1016/j.jsp.2022.03.007
   PubMed Web of Science ®Google Scholar
• Hollands, F. M., Pan, Y., Shand, R., Cheng, H., Levin, H. M., Belfield, C. R., Kieffer, M., Bowden, A. B., & Hanisch-Cerda, B. (2013). Improving early literacy: Cost-effectiveness analysis of effective reading programs. Teachers College, Columbia University.
   Google Scholar
• Hollands, F. M., Pratt-Williams, J., & Shand, R. (2021). Cost analysis standards & guidelines 1.1. Cost Analysis in Practice (CAP) Project. https://capproject.org/resources
   Google Scholar
• Honig, M. I., & Coburn, C. (2008). Evidence-based decision making in school district central offices: Toward a policy and research agenda. Educational Policy, 22(4), 578–608. https://doi.org/10.1177/0895904807307067
   Web of Science ®Google Scholar
• Hummel-Rossi, B., & Ashdown, J. (2002). The state of cost-benefit and cost-effectiveness analyses in education. Review of Educational Research, 72(1), 1–30. https://doi.org/10.3102/00346543072001001
   Web of Science ®Google Scholar
• International Data Evaluation Center. (2018). Reading Recovery by the numbers. https://readingrecovery.osu.edu/learn/Reading_Recovery_Facts_Figures.pdf
   Google Scholar
• International Institute for Restorative Practices (IIRP). (2018). Behavior Support Systems Model: Restorative Practices implementation guide. https://www.jefferson.kyschools.us/sites/default/files/JCPS%20Restorative%20Practice%20Implementation%20Guide.pdf
   Google Scholar
• Kane, T. J., McCaffrey, D. F., Miller, T., & Staiger, D. O. (2013). Have we identified effective teachers? Validating measures of effective teaching using random assignment. Measures of Effective Teaching Project. Bill & Melinda Gates Foundation. https://files.eric.ed.gov/fulltext/ED540959.pdf
   Google Scholar
• Koedel, C., Mihaly, K., & Rockoff, J. E. (2015). Value-added modeling: A review. Economics and Education Review, 47, 180–195. https://doi.org/10.1016/j.econedurev.2015.01.006
   Web of Science ®Google Scholar
• Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241–253. https://doi.org/10.3102/0013189X20912798
   Web of Science ®Google Scholar
• Lamdin, D. J. (1996). Evidence of student attendance as an independent variable in education production functions. The Journal of Educational Research, 89(3), 155–162. https://doi.org/10.1080/00220671.1996.9941321
   Web of Science ®Google Scholar
• Leach, S. M., Hollands, F. M., Stone, E., Shand, R., Head, L., Wang, Y., Pan, Y., … Pan, Y. (in press). Costs and effects of school-based licensed practical nurses on elementary student attendance and chronic absenteeism.​ Prevention Science.
   Google Scholar
• Leach, S. M., & Yan, B. (2021a). Academic return-on-investment (AROI) and budget decision-making: A research brief. Louisville, KY: Jefferson County Public Schools. https://www.jefferson.kyschools.us/sites/default/files/IES_AROI_Brief_3_Final_Dec_2021.pdf​
   Google Scholar
• Leach, S. M., & Yan, B. (2021b). ITS 2.0 user’s guide. Getting more bang for your buck: Academic return on investment and the Investment Tracking System. Accountability, research, & systems improvement. ​​​Louisville, KY: Jefferson County Public Schools. https://www.jefferson.kyschools.us/sites/default/files/AROI_Brief_1_ARSI_Final.pdf​​
   Google Scholar
• Levenson, N. (2012). Smarter budgets, smarter schools: How to survive and thrive in tight times. Harvard Education Press.
   Google Scholar
• Levenson, N., Baehr, K., Smith, J. C., & Sullivan, C. (2014). Spending money wisely: Getting the most from school district budgets. District Management Council.
   Google Scholar
• Levin, H. M. (1970). A cost-effectiveness analysis of teacher selection. The Journal of Human Resources, 5(1), 24–33. https://doi.org/10.2307/144622
   Web of Science ®Google Scholar
• Levin, H. M. (1975). Cost-effectiveness analysis in evaluation research. In M. Guttentag & E. L. Struening (Eds.), Handbook of evaluation research (Vol. 2, pp. 89–122). Sage Publications.
   Google Scholar
• Levin, H. M. (1988). Cost-effectiveness and educational policy. Educational Evaluation and Policy Analysis, 10(1), 51–69. https://doi.org/10.3102/01623737010001051
   Google Scholar
• Levin, H. M., & McEwan, P. J. (2001). Cost-effectiveness analysis: Methods and applications (2nd) ed.). Sage Publications.
   Google Scholar
• Levin, H. M., McEwan, P. J., Belfield, C., Bowden, A. B., & Shand, R. (2018). Economic evaluation in education: Cost-effectiveness and benefit-cost analysis. Sage Publications.
   Google Scholar
• Living on Campus. (2011, June). College Planning and Management Magazine. 1105 Media.
   Google Scholar
• London, R. A., Sanchez, M., & Castrechini, S. (2016). The dynamics of chronic absence and student achievement. Education Policy Analysis Archives, 24(112). http://dx.doi.org/10.14507/epaa.24.2741112
   PubMedGoogle Scholar
• Lortie-Forgues, H., Sio, U. N., & Inglis, M. (2021). How should educational effects be communicated to teachers? Educational Researcher, 50(6), 345–354. https://doi.org/10.3102/0013189X20987856
   Web of Science ®Google Scholar
• Madison Metropolitan School District. (2014). 1Reading Recovery evaluation. https://accountability.madison.k12.wi.us/files/accountability/2014-11-1%20-%20Reading%20Recovery%20Evaluation.pdf
   Google Scholar
• Magalnick, H., & Mazyck, D., American Academy of Pediatrics Council on School Health. (2008). Role of the school nurse in providing school health services. Pediatrics, 121(5), 1052–1056. https://doi.org/10.1542/peds.2008-0382
   PubMed Web of Science ®Google Scholar
• Mallett, C. A. (2016). Truancy: It’s not about skipping school. Child and Adolescent Social Work Journal, 33(4), 337–347. https://doi.org/10.1007/s10560-015-0433-1
   Web of Science ®Google Scholar
• May, H., Blakeney, A., Shrestha, P., Mazal, M., & Kennedy, N. (2022). Long-term impacts of Reading Recovery through third and fourth grade: A regression discontinuity study from 2011-12 through 2016-17. [Paper presentation]. American Educational Research Association Annual Meeting, San Diego, CA.
   Google Scholar
• Meyer, R. H. (1997). Value-added indicators of school performance: A primer. Economics of Education Review, 16(3), 283–301. https://doi.org/10.1016/S0272-7757(96)00081-7
   Web of Science ®Google Scholar
• Mihaly, K., McCaffrey, D. F., Lockwood, J. R., & Sass, T. R. (2010). Centering and reference groups for estimates of fixed effects: Modifications to felsdvreg. The Stata Journal, 10(1), 82–103. https://doi.org/10.1177/1536867X1001000109
   Web of Science ®Google Scholar
• Moonie, S., Sterling, D. A., Figgs, L. W., & Castro, M. (2008). The relationship between school absence, academic performance, and asthma status. Journal of School Health, 78(3), 140–148. https://doi.org/10.1111/j.1746-1561.2007.00276.x
   PubMed Web of Science ®Google Scholar
• National Center for Education Statistics. (2020). School nurses in U.S. public schools. Data Point. NCES 2020–086.
   Google Scholar
• National Commission on Excellence in Education. (1983). A nation at risk: The imperative for educational reform. The Elementary School Journal, 84(2), 113–130. https://doi.org/10.1086/461348
   Web of Science ®Google Scholar
• National Research Council. (2010). Getting value out of value-added: Report of a workshop. The National Academies Press.
   Google Scholar
• No Child Left Behind Act of 2001, P.L. 107–110, 20 U.S.C. § 6319. (2002).
   Google Scholar
• Organisation for Economic Co-operation and Development. (2015) . Education policy outlook 2015: Making reforms happen. OECD Publishing.
   Google Scholar
• Penuel, W. R., & Farrell, C. C. (2017). Practice partnerships and ESSA: A learning agenda for the coming decade. In E. Quintero (Ed.), Teaching in context: The social side of education reform (pp. 181–200). Harvard Education Press.
   Google Scholar
• Pimentel, S. D., Page, L. C., Lenard, M., & Keele, L. (2018). Optimal multilevel matching using network flows: An application to a summer reading intervention. The Annals of Applied Statistics, 12(3), 1479–1505. https://doi.org/10.1214/17-AOAS1118
   Web of Science ®Google Scholar
• Rice, J. K. (1997). Cost analysis in education: Paradox and possibility. Educational Evaluation and Policy Analysis, 19(4), 309–317. https://doi.org/10.3102/01623737019004309
   Web of Science ®Google Scholar
• RRCNA. (2018). Standards and guidelines of reading recovery in the United States (8th ed.). Reading Recovery Council of North America.
   Google Scholar
• Schmitt, M. C., Askew, B. J., Fountas, I. C., Lyons, C. A., & Pinnell, G. S. (2005). Changing futures: The influence of Reading Recovery in the United States. Reading Recovery Council of North America.
   Google Scholar
• Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
   Google Scholar
• Shand, R., Leach, S., Hollands, F. M., Chang, F., Pan, Y., Yan, B., Head, L., … Head, L. ​ (2022). Program value-added: A feasible method for providing evidence on the effectiveness of multiple programs implemented simultaneously in schools. American Journal of Evaluation. https://doi.org/10.1177/10982140211071017
   Web of Science ®Google Scholar
• Simon, J. (2011). A cost-effectiveness analysis of early literacy interventions. ( Publication No. 865292034) [Doctoral dissertation, Columbia University]. ProQuest Dissertations and Theses Global.
   Google Scholar
• Sirinides, P., Gray, A., & May, H. (2018). The impacts of reading recovery at scale: Results from the 4-year i3 external evaluation. Educational Evaluation and Policy Analysis, 40(3), 316–335. https://doi.org/10.3102/0162373718764828
   Web of Science ®Google Scholar
• Smith, R., & Knapp, K. (2019). Return on Instructional Investment (ROII) model: A practical guide for school leaders. Academy of Educational Leadership Journal, 23(1), 1–11. https://www.proquest.com/docview/2238974843?pq-origsite=gscholar&fromopenview=true
   Google Scholar
• Smith, J. A., & Todd, P. E. (2005). Does matching overcome LaLonde’s critique of nonexperimental estimators? Journal of Econometrics, 125(1–2), 305–353. https://doi.org/10.1016/j.jeconom.2004.04.011
   Web of Science ®Google Scholar
• Sugai, G., & Simonsen, B. (2012). Positive behavioral interventions and supports: History, defining features, and misconceptions. Center for PBIS & Center for Positive Behavioral Interventions and Supports, University of Connecticut. https://assets-global.website-files.com/5d3725188825e071f1670246/5d82be96e8178d30ae613263_pbis_revisited_june19r_2012.pdf
   Google Scholar
• Telljohann, S. K., Dake, J. A., & Price, J. H. (2004). Effect of full-time versus part-time school nurses on attendance of elementary students with asthma. The Journal of School Nursing, 20(6), 331–334. https://doi.org/10.1177/10598405040200060701
   PubMedGoogle Scholar
• Thorsborne, M., & Blood, P. (2013). Implementing restorative practices in schools: A practical guide to transforming school communities. Jessica Kingsley Publishers.
   Google Scholar
• U.S. Department of Education. (2016). Non-regulatory guidance: Using evidence to strengthen education investments. https://ed.gov/policy/elsec/leg/essa/guidanceuseseinvestment.pdf
   Google Scholar
• Ver Hoef, J. M., & Boveng, P. L. (2007). Quasi‐Poisson vs. negative binomial regression: How should we model overdispersed count data? Ecology, 88(11), 2766–2772. https://doi.org/10.1890/07-0043.1
   PubMed Web of Science ®Google Scholar
• Wang, Y., Hollands, F. M., Shand, R., Pratt-Williams, J., Head, L., Kushner, A., & Chang, Y. (2020). Cost of facilities calculator. [Computer app]. Cost Analysis in Practice (CAP) Project. https://capproject.org/resources
   Google Scholar
• Wang, L. Y., Vernon-Smiley, M., Gapinski, M. A., Desisto, M., Maughan, E., & Sheetz, A. (2014). Cost-benefit study of school nursing services. JAMA Pediatrics, 168(7), 642–648. https://doi.org/10.1001/jamapediatrics.2013.5441
   PubMed Web of Science ®Google Scholar
• Weiss, C. H. (1977). Research for policy’s sake: The enlightenment function of social research. Policy Analysis, 3(4), 531–545. https://www.jstor.org/stable/42783234
   Google Scholar
• What Works Clearinghouse. (2013, July). Beginning reading intervention report: Reading Recovery®. http://whatworks.ed.gov
   Google Scholar
• Winsch, B. (2017). Restorative Practices evaluation plan. Jefferson County Public Schools, KY.
   Google Scholar
• Winsch, B. J. (2016). School nurse program evaluation.Department of Data Management, Planning, and Program Evaluation. Jefferson County Public Schools.
   Google Scholar
• Yan, B. (2017). Lessons learned from the budget requests and approvals: 2017–18. Jefferson County Public Schools.
   Google Scholar
• Yan, B. (2018, January). Improve system deficiencies to make stronger budget decisions. School Business Affairs, 84(1), 14–17.
   Google Scholar
• Yan, B. (2020). Issues around using academic return on investment (A-ROI) for informing and improving decisions. Part I: Validity. Jefferson County Public Schools, KY. https://eric.ed.gov/?id=ED603587
   Google Scholar