Lessons for career guidance from return-on-investment analyses in complex education-related fields

Figures & data

Figure 1. Corpus development method overview.

Table 1. Number of experts approached by field.

Field	Requested to review	Provided input
Secondary school career guidance	11	11*
Career guidance or support for unemployed adults	11	7*
Widening participation (campaign to increase education uptake)	6	4
Behaviour in schools (e.g. bullying, social emotional learning)	6	5

Note: Three people counted once in each field, as they provided input on both forms of guidance.

Table 2. Number of papers by field.

Field	Academic literature	Grey literature *
Secondary school career guidance	1 paper Castleman and Goodman (2018)	4 papers Ford et al. (2012) Engelman et al. (2016) Percy (2020) Frontier Economics (2021)
Career guidance or support for unemployed adults	3 papers Osikominu (2013) Fleissig (2014) Maibom et al. (2017)	9 papers Hasluck et al. (2000) Van Reenen (2001) Oxford Economics (2008) Gauge Ireland (2012) Michaelides et al. (2012) Krug and Stephan (2013) Hollenbeck and Huang (2016) ICF International (2016) Percy & Hughes (2022)
Widening participation	1 paper Ravulo et al. (2019)	3 papers IntoUniversity (2010) The Sutton Trust (2010) Beesley et al. (2015)
Behaviour in schools	10 papers Aos et al. (2004) Foster and Jones (2006) Foster and Jones (2015) Belfield et al. (2015) Klapp et al. (2017) Bowden et al. (2020) Huitsing et al. (2020) Turner et al. (2020) Jadambaa et al. (2021) Legood et al. (2021)	1 paper Swain-Bradway et al. (2017)
Total	15 papers	17 papers

Note: Not academically indexed in the Scopus search.

Table 3. ROI by field.

Field	Number of estimates [number of papers]*	Headline ROI Range as a multiplier **	ROI interquartile range and median
Secondary school career guidance	10 [5]	0.7x to 61.5x	1.2x to 2.5x to 4.1x
Career guidance or support for unemployed adults	16 [12]	0.9x to 109.6x	1.9x to 3.2x to 6.7x
Widening participation	5 [4]	0.6x to 43x	2.4x to 4.2x to 5.7x
Behaviour in schools	14 [11]	−0.5x to 104.9x	1.4x to 4.2x to 9.7x

Notes: * Where a paper contains multiple interventions, the most proximate to school-age career guidance is chosen. Where the paper summary reports a range of estimates with equal prominence for the best fit intervention each estimate is counted separately, for example, an upper – and lower-bound estimate or cross-group variation. Inclusion in the corpus is not an endorsement of the method or ROI estimates. In 13 papers, the ROI estimates are calculated manually for the paper out of inputs contained in the original study, such as where a net benefit presentation is used instead (benefits minus costs). ** For example, 1x would mean the return is one times the investment, i.e. breakeven.

Table 4. A framework for ROI in school-based career guidance.

Topic	Questions for ROI users/evaluators	Questions to guide analysts estimating ROI*
In what context is the ROI undertaken?	Is it evaluating a specific intervention or proposing a general theory? What conflicting interests might be present?	- What schools are involved in the ROI? How have they been selected? Is there anything special about them in terms of their engagement in career guidance or any other features? - What other actors are involved in the intervention (e.g. government agencies, employers, career guidance organisations)? What is their interest in the ROI, e.g. hoped-for outcome?
How is the ROI framed conceptually?	Is it based on an explicit theory of change or on the use of available data? From what perspective are the costs and benefits being calculated? E.g. individual, organisational or governmental. What consideration is given to second order effects?	- How are insights from career theory and research used to inform the theory of change? Can evidence on intermediate ROI outcomes, such as progress in vocational identity or labour market understanding, be incorporated into the ROI calculation chain or used as a sense-check on the output? - Can an accounting perspective diagram be used to explain which costs/benefits theoretically apply, including those that cannot be costed or can only be partially addressed? Likely perspectives are the student participant or the government’s fiscal position, as well as the school or local economy.
How are costs captured?	Do they include programme costs, participant costs and any wider social and opportunity costs?	- How far should the existing costs of the school or of other actors be viewed as a baseline that does not need to be costed? For example, are teacher time, the use of school premises, employer volunteer time and student time viewed as a cost? Where these non-direct costs are not costed they should at least be described and recognised in the ROI.
How are benefits captured?	What benefits are identified? How are these benefits monetised and how is potential overlap addressed? E.g. through tax revenues or more subjective monetisation like willingness to pay. Over what time period are benefits captured? How are third party data points justified?	– Should the ROI attempt to capture and monetise the psycho-social impacts of career guidance (e.g. increased self-efficacy; often easier to capture and motivate theoretically), more easily measured short-term impacts such as attendance, attainment or progression to further learning and education, and/or more easily monetised impacts (e.g. labour market outcomes)? - Where multiple benefits are monetised, how is overlap accounted for? (e.g. increased self-efficacy may contribute to improved levels of qualification which may in turn increase salary, but these benefits cannot simply be added together) - What time period is addressed? Where career guidance improves early career outcomes or shifts a career mindset, there can in principle be multi-decade benefits, given longitudinal studies on the stickiness of wages and pathway-dependency of progression. However, at what point should the original guidance benefit be considered fully depreciated? - Consider using a formal search strategy and selection criteria/sensitivity checks for contested third party estimates that strongly influence results, such as intervention impact, the wage returns to qualification gain or lifetime public service cost of youth NEET outcomes
What counterfactual approach is used?	What techniques are deployed to establish a counter-factual? e.g. comparison to benchmarks vs randomised control trials.	- Where intervention schools are compared with other schools it is important to think through what schools and why they are a meaningful comparison. Where randomised approaches are adopted this raises questions about the appropriate level of randomisation (individual, class or school) and ethical and perceptual issues related to who is getting access to the new provision. - If a social or economy-wide perspective is adopted, what qualitative or quantitative account can you make for displacement? For instance, how likely is it that participants’ outperformance is counterbalanced by worse than counterfactual outcomes for the non-participants for the intervention in question? (e.g. networking or CV polish that leads to gaining a job at someone else’s expense, but with no/little productivity difference or search cost reduction)
How is the final ROI estimate sense-checked?	Is it outside of the typical range? If so, is it justified by a strong methodology? What kinds of robustness checks have been conducted?	- This review suggests that anything above a 6x-7x ROI for school-based career guidance should be viewed as unusual and the reasons for this high result discussed. The same is also true for very low ROIs e.g. below 1x. - For ROI calculations with many inputs, consider presenting the overall estimation range using a Monte Carlo calculation with low/med/high values for each input and a simple distributional assumption (e.g. triangular; uniform). - Can the project brief be constructed to include sense checks of calculation output and key inputs with participants, practitioners, funders and stakeholders?

Note: Further to good ROI practice, for instance, benefit-cost analysis methodologies in Levin et al. (2018).

Castleman, B., & Goodman, J. (2018). Intensive college counseling and the enrollment and persistence of low-income students. Education Finance and Policy, 13(1), 19–41. https://doi.org/10.1162/edfp_a_00204

 Web of Science ®Google Scholar

Ford, R., Frenette, M., Nicholson, C., Kwakye, I., Hui, T., Hutchinson, J., Dobrer, S., Fowler, H., & Hebert, S. (2012). Future to discover: Post-secondary impacts report. Social Research and Demonstration Corporation. http://www.srdc.org/publications/Future-to-Discover-FTD–Post-secondary-Impacts-Report-details.aspx.

 Google Scholar

Engelman, A., Morgan, G., Ruthven, M., & Pugh, E. (2016). 2016 legislative report Colorado school counselor corps grant program. Colorado Department of Education. https://www.cde.state.co.us/postsecondary/2016-school-counselor-corps-grant-program.

 Google Scholar

Percy, C. (2020). Personal guidance in English secondary education: An initial return-on-investment estimate. The Careers & Enterprise Company.

 Google Scholar

Frontier Economics. (2021). The economic value of WorldSkills UK. WorldSkills UK.

 Google Scholar

Osikominu, A. (2013). Quick job entry or long-term human capital development? The dynamic effects of alternative training schemes. Review of Economic Studies, 80(1), 313–342. https://doi.org/10.1093/restud/rds022

 Web of Science ®Google Scholar

Fleissig, A. (2014). Return on investment from training programs and intensive services. Atlantic Economic Journal, 42(1), 39–51. https://doi.org/10.1007/s11293-013-9394-y

 Google Scholar

Maibom, J., Rosholm, M., & Svarer, M. (2017). Experimental evidence on the effects of early meetings and activation. The Scandinavian Journal of Economics, 119(3), 541–570. https://doi.org/10.1111/sjoe.12180

 Web of Science ®Google Scholar

Hasluck, C., McKnight, A., & Elias, P. (2000). Evaluation of the new deal for lone parents: Early lessons from the Phase One Prototype - Cost-benefit and econometric analyses research report No. 110. Department of Social Security.

 Google Scholar

Van Reenen, J. (2001). No more skivvy schemes? Active labour market policies and the British new deal for the young unemployed in context. The Institute for Fiscal Studies.

 Google Scholar

Oxford Economics. (2008). Examining the impact and value of EGSA to the NI economy: Final report.

 Google Scholar

Gauge Ireland. (2012). Social return on investment – Final report (December 2012). e-Merge - An Initiative of the Mount Street Trust – Employment Initiative delivered by Ballymun Job Centre. Ballymun Job Centre.

 Google Scholar

Michaelides, M., Poe-Yamagata, E., Benus, J., & Tirumalasetti, D. (2012). Impact of the reemployment and eligibility assessment (REA) initiative in Nevada. IMPAQ International.

 Google Scholar

Krug, G., & Stephan, G. (2013). Is the contracting-out of intensive placement services more effective than provision by the PES? Evidence from a randomized field experiment. IZA Discussion Papers, 7403. Institute for the Study of Labor (IZA).

 Google Scholar

Hollenbeck, K., & Huang, W. (2016). Net impact and benefit-cost estimates of the workforce development system in Washington State. Upjohn Institute Technical Report, 16-033. W.E. Upjohn Institute for Employment Research. https://doi.org/10.17848/tr16-033

 Google Scholar

ICF International. (2016). Social innovation fund – return on investment study for three national fund for workforce solutions (NFWS) workforce partnership programs in Ohio. ICF.

 Google Scholar

Percy, C., & Hughes, D. (2022). Lifelong guidance and welfare to work in Wales: Linked return on investment methodology. In F. Cedefop (Ed.), Towards European standards for monitoring and evaluation of lifelong guidance systems and services (Vol. I, pp. 109–138). Publications Office of the European Union. Cedefop. http://data.europa.eu/doi/10.2801/422672.

 Google Scholar

Ravulo, J., Said, S., Micsko, J., & Purchase, G. (2019). Utilising the social return on investment (SROI) framework to gauge social value in the fast forward program. Education Sciences, 9(4), 290. https://doi.org/10.3390/educsci9040290

 Web of Science ®Google Scholar

IntoUniversity. (2010). Intouniversity social return on investment.

 Google Scholar

The Sutton Trust. (2010). The mobility manifesto: A report on cost-effective ways to achieve greater social mobility through education.

 Google Scholar

Beesley, A., Gulemetova, M., & DiFuccia, M. (2015). Georgia governor’s office of student achievement innovation fund SROI analysis final report. IMPAQ International.

 Google Scholar

Aos, S., Lieb, R., Mayfield, J., Miller, M. G., & Pennucci, A. (2004). Benefits and costs of prevention and early intervention programs for youth. Washington State Institute for Public Policy. https://www.ojp.gov/ncjrs/virtual-library/abstracts/benefits-and-costs-prevention-and-early-intervention-programs-youth.

 Google Scholar

Foster, E. M., Jones, D. E., & Conduct problems prevention research group (2006). Can a costly intervention be cost-effective? Archives of General Psychiatry, 63(11), 1284–1291. https://doi.org/10.1001/archpsyc.63.11.1284

 PubMed Web of Science ®Google Scholar

Belfield, C., Bowden, A. B., Klapp, A., Levin, H., Shand, R. D., & Zander, S. (2015). The economic value of social and emotional learning. Journal of Benefit-Cost Analysis, 6, 508–544. https://doi.org/10.1017/bca.2015.55

 Web of Science ®Google Scholar

Klapp, A., Belfield, C., Bowden, B., Levin, H., Shand, R. D., & Zander, S. (2017). A benefit-cost analysis of a long-term intervention on social and emotional learning in compulsory school. The International Journal of Emotional Education, 9, 3–19. https://repository.upenn.edu/cbcse/7/

 Google Scholar

Bowden, A. B., Shand, R., Levin, H. M., Muroga, A., & Wang, A. (2020). An economic evaluation of the costs and benefits of providing comprehensive supports to students in elementary school. Prevention Science, 21(8), 1126–1135. https://doi.org/10.1007/s11121-020-01164-w

 PubMed Web of Science ®Google Scholar

Huitsing, G., Barends, S., & Lokkerbol, J. (2020). Cost-benefit analysis of the KiVa anti-bullying program in The Netherlands. International Journal of Bullying Prevention, 2, 215–224. https://doi.org/10.1007/s42380-019-00030-w

 Google Scholar

Turner, A. J., Sutton, M., Harrison, M., Hennessey, A., & Humphrey, N. (2020). Cost-effectiveness of a school-based social and emotional learning intervention: Evidence from a cluster-randomised controlled trial of the promoting alternative thinking strategies curriculum. Applied Health Economics and Health Policy, 18(April 2020), 271–285. https://doi.org/10.1007/s40258-019-00498-z

 PubMed Web of Science ®Google Scholar

Jadambaa, A., Graves, N., Cross, D., Pacella, R., Thomas, H., Scott, J., Cheng, Q., & Brain, D. (2021). Economic evaluation of an intervention designed to reduce bullying in Australian schools. Applied Health Economics and Health Policy, 20, 79–89. https://doi.org/10.1007/s40258-021-00676-y

 PubMed Web of Science ®Google Scholar

Legood, R., Opondo, C., Warren, E., Jamal, F., Bonell, C., Viner, R., & Sadique, Z. (2021). Cost-utility analysis of a complex intervention to reduce school-based bullying and aggression: An analysis of the inclusive RCT. Value in Health, 24(1), 129–135. https://doi.org/10.1016/j.jval.2020.04.1839

 PubMed Web of Science ®Google Scholar

Swain-Bradway, J., Lindstrom Johnson, S., Bradshaw, C., & McIntosh, K. (2017). What are the economic costs of implementing SWPIBS in comparison to the benefits from reducing suspensions? PBIS Technical Assistance Center, University of Oregon. https://www.pbis.org/resource/what-are-the-economic-costs-of-implementing-swpbis-in-comparison-to-the-benefits-from-reducing-suspensions.

 Google Scholar

Levin, H. M., McEwan, P. J., Belfield, C., Bowden, A. B., & Shand, R. (2018). Economic evaluation in education: Cost-effectiveness and benefit-cost analysis (3rd ed.). SAGE publications.

 Google Scholar

Data availability statement

All corpus documents are publicly available at the time of writing. Corpus details are available from the corresponding author upon reasonable request.