The Value of Efficiency Measures: Lessons from Workforce Development Programs

ABSTRACT

In 2007, the Employment and Training Administration was required to implement an outcome-based measure or measures of efficiency for its programs. This paper identifies outcome-based efficiency measures for 11 ETA-administered programs. The research involved interviews with state officials; analysis of efficiency measures used by other federal workforce programs; and analysis of cost and outcome data for the 11 ETA programs to investigate the potential for developing regression models to adjust expected performance for states. A key lesson is that it is critical to anticipate the behavioral changes likely to be induced by the performance policies adopted. The paper recommends that efficiency measures be tied to outcome performance measures. It also recommends use of program expenditures as the measure of program costs. Efficiency measures recommended include cost per entered employment, cost per retained in employment, cost divided by post-program (average) earnings, and cost divided by change in earnings.

KEYWORDS::

• costs
• efficiency
• employment and training
• performance measures
• return on investment

Notes

The paper is based on research conducted for the U.S. Department of Labor’s Employment and Training Administration (ETA) in response to a request from the Office of Management and Budget (OMB) for ETA to develop and implement outcome-based efficiency measures for workforce development programs administered by ETA.

Most of this section is based on Barnow and Smith (2004). For a critical review of the performance management movement, see Radin (2006).

The earnings-gain measure was later replaced by a post-program earnings measure.

A related problem is that performance measures must use short-term post-program earnings to measure performance, but the impact of a program is best measured over a longer period. Barnow and Smith (2004) reviewed the literature on the relationship between short-term earnings impacts and long-term impacts, and they found that most studies find a very weak relationship between the two.

See TEIN 5-93, available at: http://wdr.doleta.gov/directives/corr_doc.cfm?DOCN=310, accessed February 20, 2015.

See Bloom, Michalopoulos, Hill, and Lei (2002), Smith and Todd (2005), and Wilde and Hollister (2007). For a more optimistic appraisal, see Cook, Shadish, and Wong (2008).

See Dickinson et al. (1988).

Executive Office of the President, Office of Management and Budget, Program Assessment Rating Tool Guidance No. 2007-7: Improving the Quality of PART Performance and Efficiency Goals, issued December 12, 2007. Available at http://georgewbush-whitehouse.archives.gov/omb/performance/guidance/part_guid_2007-07.pdf, accessed February 20, 2015. The Program Assessment Rating Tool Guidance No. 2007-7 called for a government-wide initiative “to increase the government and public’s ability to assess and improve a program’s effectiveness and efficiency.”

Executive Office of the President, Office of Management and Budget, Program Assessment Rating Tool Guidance No. 2007-7: Improving the Quality of PART Performance and Efficiency Goals, issued December 12, 2007. Available at http://georgewbush-whitehouse.archives.gov/omb/performance/guidance/part_guid_2007-07.pdf, accessed February 20, 2015.

For example, GAO (1998) makes a distinction between three types of performance measures: “Performance measures may address the type or level of program activities conducted (process), the direct products and services delivered by a program (outputs), and /or the results of those products and services (outcomes).”

For additional background on the Common Measures, see Training and Employment Guidance Letter No. 17-05, “Common Measures Policy for the Employment and Training Administration’s (ETA) Performance Accountability System and Related Performance Issues.” Available at http://wdr.doleta.gov/directives/corr_doc.cfm?DOCN=2195, accessed February 20, 2015.

See U.S. Government Accountability Office (2004b).

The term “allocation” is quite often used interchangeably with “allotment,” but in the context of formula programs, the appropriate term is “allotment.”

See U.S. General Accounting Office, 2004a.

Also according to OFAM, the term “drawdown or payment” is sometimes used to reflect the “transfer of cash to a grantee/contractor based on grantee requests/contractor invoices to reimburse the grantee/contractor for expenditures on a valid grant/contract. Source of quotations in footnote and text is OFAM electronic response to a question.

An alternative to collecting cost data by activity would be to take into account the activity mix of participants (e.g., the percentage of individuals trained) when setting performance standards for efficiency measures.

The Federal Trade Act provides special benefits under the Trade Adjustment Assistance (TAA) program to those who were laid off or had hours reduced because their employer was adversely affected by increased imports from other countries. These benefits include paid training for a new job, financial help in making a job search in other areas, or relocation to an area where jobs are more plentiful. Those who qualify may be entitled to weekly Trade Adjustment Allowances (TRA) after their unemployment compensation is exhausted. Additional background on TAA and TRA is available on the U.S. Department of Labor website at http://workforcesecurity.doleta.gov/unemploy/tra.asp.

In addition to regression analysis, other approaches can be used to compare jurisdictions on efficiency. Geys and Moesen (2009) describe and apply two techniques in addition to regression analysis to assess local efficiency—data envelopment analysis (DEA) and free disposal hull (FDH). However, the same data problems we encountered in conducting regression analysis would also apply to these other approaches.

Additional information

Burt S. Barnow is the Amsterdam Professor of Public Service and Economics at George Washington University, and previously was associate director for research at Johns Hopkins University’s Institute for Policy Studies and director of the Office of Research and Evaluation in the Employment and Training Administration. His current research includes the development and evaluation of pilot programs to test self-sufficiency strategies for welfare recipients for the Department of Health and Human Services, an evaluation of the Reemployment and Eligibility Assessment program for the Department of Labor, and an evaluation of the Homeless Veterans Reintegration Program. In 2013, Dr. Barnow co-authored a book on occupational labor shortages and co-edited a volume assessing the implementation and effectiveness of workforce and unemployment insurance provisions of the American Recovery and Reinvestment Act. He received a B.S. in economics from Massachusetts Institute of Technology and a Ph.D. in economics from the University of Wisconsin at Madison.

John Trutko is president of Capital Research Corporation, Inc. He has led and been part of research teams on over 30 past Department of Labor studies, working on a range of process/implementation, outcome, and impact evaluation studies since the mid-1980s and has directed high-level research studies on evaluation of employment and training initiatives focusing on veterans, separating military personnel, disadvantaged adults, youth, dislocated/trade-affected workers, and unemployment insurance claimants. He has authored or co-authored over 75 reports for a wide variety of federal and state government agencies and foundations, including over 30 reports for the Employment and Training Administration. Mr. Trutko has a master’s degree in economics from the University of Sussex (UK) and a bachelor’s degree in political science from the University of Rochester (NY).