Foundational learning program evaluation and dropouts: are dropouts a heterogeneous group?

ABSTRACT

We analyze the characteristics and outcomes of a Canadian foundational learning program's dropouts and compare them with those of the completers. We find significant heterogeneity within dropouts along two dimensions: when they drop out and why. Individuals whose characteristics have been historically associated with greater labour market barriers, and those with lower employability skills are more likely to complete the program. Individuals who face fewer barriers tend to leave at an early stage, while individuals without a high school degree tend to drop out later. Conditional on education, higher employability-skill participants are more likely to leave and return to school.

KEYWORDS:

• Active labour market policies
• dropouts
• policy evaluation

Acknowledgments

This paper emerged from plans to evaluate the FWSP intervention for which some results were circulated in a prior working paper entitled ‘An analysis of a foundational learning program in BC: the Foundations Workplace Skills Program (FWSP) at Douglas College.’ We gratefully acknowledge financing from Human Resources and Skills Development Canada. We have benefitted from the input of Pierre Brochu, Karen Myers, Craig Riddell, Arthur Sweetman, and Jean-Pierre Voyer. The assistance of Pam Tetarenko of Douglas College, who provided us with the data set and informed us about the program, was instrumental. All errors are our own.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Funding

This paper is part of the Adult Learning and Returns to Training Project sponsored by the Canadian Labour Market and Skills Research Network and The Social Research and Demonstration Corporation.

Notes

1 The literature generally makes a distinction between dropouts and attrition. For example, if a participant does not complete the program, but we still observe her outcome of interest (say, her employment status) after the program's end date, we would label this participant as a ‘dropout.’ If we do not observe the outcome of interest, we would label this participant as an ‘attritor,’ whether or not she completes the program. In this paper, we observe some post-treatment outcomes (e.g. employment status) but not others (e.g. post-treatment test scores) for individuals who did not complete the program. As such, our program ‘non-completers’ will be either ‘attritors’ or ‘dropouts’ depending on the outcome we are considering.

2 This scarcity contrasts with the literature on school dropouts, which is quite large. See, e.g. Agasisti, Bolzoni, and Soncin (2023), Eegdeman et al. (2023), and Hermann and Horn (2023) and references therein.

3 Fitzenberger, Osikominu, and Paul (2010) address the choice of program starting time and potentially positive and negative selection into dropout status. Both entries and exits from a German job training programs are endogenously modelled. Based on the same data set, Biewen et al. (2014) take into account dynamic selection into different programs. Dalla-Zuanna and Liu (2019), which is based on US data, deals with variation in time spent in treatment and the potential selection out of the primary intervention into alternative or additional programs. Mealli, Pudney, and Thomas (1996) involves a long-term job training program in the UK. In their joint estimation of the outcomes of the duration of the training spell and the destination state, they observe early termination, but they do not model it. They determine that the impact of that event on the estimated transition probabilities into employment is negative relative to the event of completion, and they do not examine differences between completers and early terminators.

4 For an informative descriptive summary of Canada's rather complicated array of public training and skills development programs, see Jansen et al. (2019).

5 Reading text, document use, numeracy, writing, thinking skills, oral communication, continuous learning, working with others, and computer use.

6 Examples include SUCCESS (United Chinese Community Enrichment Services Society), ISS (Immigrant Services Society), Options Community Services, and PCRS (Pacific Community Resources Society).

7 Those authors label that phenomenon ‘the service gap’; addressing it is the crux of the FWSP.

8 Interviews with program officials indicated that the only alternative programming that existed at the time was some Basic Literacy programs.

9 The goal is not to develop an entirely new career path.

10 A typical schedule might be 16 hours in group activities, 29 hours at home, and 26 hours in lab activities.

11 While we do not observe the type of education pursued by those who returned to school, we can infer it using their career goal. The vast majority of them listed career goals requiring either community college (e.g. accounting clerks, health care assistants, licensed practical nurses) or professional training schools (e.g. carpenters, class 1 truck drivers, culinary artists).

12 Since linking FSWP participants to their employment records is not feasible, we must rely on the information provided by the participants. When participants dropped out, they either provided that information to program staff directly, or to the case workers who originally referred him/her to the FWSP notified program staff. Unfortunately, no information in the data allows us to identify who provided the information (e.g. the participant or the case worker). This missing information should not affect the analysis of the stage at which participants leave the program, as the program administrators track this information. In particular, if individuals lied about having found a job, it would affect our descriptive statistics. Whether it would affect our regression analysis or not depends on whether the likelihood of falsely reporting finding a job is correlated with our predictors. As long as the likelihood of falsely reporting to have found a job is uncorrelated with our predictors, it should not affect our analysis. Alas, there is no way for us to investigate that. Finally, falsely reporting to have returned career planning/search outside the program or failing to complete a phase are not likely, as most of this information is collected by case workers and program administrators.

13 Card, Kluve, and Weber (2018) find that ALMPs have larger impacts in recessionary periods. We have investigated whether labour market conditions can also affect the probability of dropping out of the FWSP. Since age-specific unemployment rates and the month and year dummy variables are highly correlated, including the former as a control variable makes its parameter estimates imprecise. We therefore exclude the age-specific unemployment rate variable from our regressions.

14 Plotting the unconditional normalized TOWES scores leads to the same conclusions (see Onlne Appendix Figure A.1).

15 While the results presented here are based on a linear probability model, they are very similar (in terms of estimates and statistical significance) to those obtained from a probit or a logistic model (see Online Appendix Tables A.7 and A.8).

16 We use the normalized averaged initial TOWES scores to control for TOWES performance given the high positive correlation between the subject-specific TOWES scores.

17 As mentioned above, phase one consists of the initial assessment, phase two consists of ‘portfolio development’ and employability, and phase three deals with the enhancement of foundational skills applied to the work environment.

18 Career planning/search includes individuals who work actively with a case manager, participating in career planning or in job search.

19 The vast majority of observations in our ‘Incomplete’ category come from individuals who did not complete the phase. There are 8 missing observations, 6 ‘no activity,’ and 14 ‘other’ in phase one. In phase two, we have 7 missing observations, 0 ‘no activity,’ and 3 ‘other.’ Finally, there are 29 missing observations, 0 ‘no activity,’ and 1 ‘other’ in phase three.

20 The hiring rate in Brochu and Green (2013) is computed using two-month mini-panels constructed from Labour Force Surveys. Hence, these are month-to-month transitions.

21 Pairwise Kolmogorov-Smirnov tests reject the assumptions that the distributions are equal at a 1% significance level.

22 For the sake of expositional clarity, we do not present the estimates for ‘Some College’ and ‘Some University’ in Figure 3 as they are imprecise (due to the small number of observations) and affect the scale of the x-axis. They are listed in Online Appendix Table A.2.

23 Online Appendix Table A.4 also shows that, if we exclude the control for TOWES scores, native speakers are more likely to drop out early. Again, this comes from the fact that these individuals have significantly higher TOWES scores.

24 The estimated average marginal effects are also presented in Online Appendix Table A.3. As for the timing analysis, we do not present the estimates for ‘Some College’ and ‘Some University’ in Figure 5, but they can be found in Online Appendix Table A.3.

Additional information

Funding

This paper is part of the Adult Learning and Returns to Training Project sponsored by the Canadian Labour Market and Skills Research Network and The Social Research and Demonstration Corporation.