The effects of booster classes in protracted crisis settings: Evidence from Kenyan refugee camps

ABSTRACT

Students in protracted crisis settings often face a range of challenges which combine to yield low education outcomes. This paper presents the results from a randomised controlled trial of weekend and holiday booster classes for 7th and 8th grade girls in Kakuma refugee camp in Kenya, that aimed to improve girls’ education outcomes and increase transition rates from primary to secondary school. While qualitative results suggested numerous advantages of the booster classes, including more freedom to ask questions, smaller class sizes, and kinder teachers, the program did not yield statistically significant effects on learning outcomes, school attendance or noncognitive skills. Mixed-methods research suggests that the limited impacts may stem from implementation challenges including irregular booster class attendance and a lack of appropriate teaching materials. More broadly, the results show the importance of accounting for implementation challenges in the reporting of impact evaluation results.

KEYWORDS:

• Education
• refugees
• learning outcomes
• Kenya
• randomised controlled trial

Acknowledgements

We recognise the contributions of many organisations, without which it would not have been possible to complete this study. Our thanks go to the United Kingdom’s Department for International Development, the United Nations Children’s Fund, and the United Nations High Commissioner for Refugees for the opportunity to carry out this study, and for their financial and technical support. We also acknowledge the members of the Reference Group for the HEA, whose inputs guided the direction of this study. Our acknowledgements would be incomplete without mentioning our team of very able enumerators in Kenya. We acknowledge the input of the team of community mobilisers from the World University Service of Canada.

Disclosure statement

Timothy Kinoti and Darius Isaboke were employed at World University Services of Canada at the time of the study. They contributed with contextual knowledge and helped interpret results, but researchers from American Institutes for Research were responsible for deciding about final conclusions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank Group, its Board of Executive Directors, or the governments they represent.

Notes

1. This study was conducted with ethical approval from American Institutes for Research’s (AIR’s) Institutional Review Board and the National Commission for Science, Technology, and Innovation in Kenya. AIR is registered with the Office of Human Research Protection as a research institution and conducts research under its own Federalwide Assurance. The EERCK program is funded by the United States Department of State’s Bureau of Population, Refugees, and Migration.

2. A related quasi-experimental study used a regression discontinuity design to determine the impact of the same booster education program (with the exception of some differences in the implementation model) in Dadaab refugee camp. The final report of the Humanitarian Education Accelerator presents the results of this study in Dadaab, which did not show statistically significant effects on learning outcomes (De Hoop et al. 2019). We do not present the results of the quasi-experimental study in this paper because of limited statistical power and because of space constraints. This paper does include qualitative results from the evaluation in Dadaab.

3. This makes the program slightly different from remedial education programs that target the lowest-achieving students. In this case, the program targets students at risk of dropping out of school because of nonachievement-related reasons, also because the remedial education is meant to prepare students to apply for a scholarship that is offered through the WUSC scholarship program.

4. WUSC works with Windle International Kenya (WIK) to implement the program.

5. WUSC included the provision of learning materials and snacks to encourage learners to attend classes regularly.

6. Coaches are expected to visit two to three centres per month. During each visit, coaches are expected to sit in during a booster class while the teacher delivers a scheduled lesson and to use a classroom-observation guide to assess teacher practices and the learning environment. After the lesson is over, the coach is expected to meet with the teacher to provide feedback on areas for improvement.

7. Community mobilisers are recruited from the predominant ethnic group surrounding the booster centre. For example, WUSC recruits Somali, Dinka, and Nuer community mobilisers to build trust with parents in the community.

8. The community mobilisers also: monitor booster class attendance rates and establish parent and teacher taskforces to monitor girls’ school and booster class drop-out rates; promote equal opportunity for both girls and boys; offer guidance and counselling to primary- and secondary-school girls work closely with the implementing organisation and education partners in promoting the Girls’ Education Challenge; and conduct awareness sessions on gender and education in the refugee community.

9. These include individual, relational, communal, and cultural resources available to each individual.

10. In 2017, we implemented a 1-day qualitative training for EERCK community mobilisers who then led the data collection under the supervision of WUSC staff. In 2018 we carried out a 3-day training for research assistants recruited for data collection. All audio recordings were transcribed verbatim and coded for analysis. We also triangulated primary data with relevant program documentation such as teacher training reports and classroom observation reports.

11. FGDs typically included 6–8 participants.

12. We use an intent-to-treat approach and do not include these backup students in our analysis.

13. In Dadaab, there were only 747 girls who met the selection critera of scoring above 170 and being at risk of dropping out of school. An RCT with this sample size would have precluded WUSC from meeting its cohort targets. Because of this small sample size, we instead relied on drawing comparisons between those girls that scored above 170 and those that scored just below in the range of 141 to 169 using a regression discontinuity design. We discuss these results in the full report describing the findings of the Humanitarian Education Accelerator (De Hoop et al. 2019).

14. Table 5 presents results from an exploratory analysis examining heterogeneous effects on girls from food-secure households in Kakuma. Although all point estimates are positive, we do not observe any statistically significant impacts for the sample of food-secure households. While we need to exercise major caution in interpreting this finding, impact estimates for the subsample of food-secure households show that program participants from food-secure households who have attended a minimum of 50 hours of remedial education in the year before the survey score 0.20 standardised mean differences higher on the Kenya Certificate of Primary Education exam than girls from food-secure households in the control group. These results are based on an instrumental variable analysis with a constrained sample of girls in food insecure households in which we use the random assignment as an instrument for the likelihood that girls participate more than 50 hours in remedial education. These results are robust to the use of other thresholds with point estimates increasing for a larger number of hours, but the results only becoming statistically significant for a threshold of 50 hours. Importantly, however, we need to exercise major caution in the interpretation of this finding because the results are not robust to the use of the Uwezo test as an outcome measure and because the results are based on a very small sample size of only 170 girls.

Table 5. WUSC KCPE impacts: food secure households.

Additional information

Funding

The work was supported by the UK Department for International Development [43208424].

Notes on contributors

Andrew Brudevold-Newman

Andrew Brudevold-Newman is an Economist in the Africa Gender Innovation Lab at the World Bank. In this role, he leads large-scale studies on women's economic empowerment and provides support to project teams and policy makers to bolster the uptake of effective policies and interventions. He has a broad research agenda focused on entrepreneurship (including training and credit constraints, and soft-skills development), the transition from school to work, and land rights.

Thomas de Hoop

Thomas de Hoop directs a research and evaluation portfolio with a focus on the impact, scalability, and cost-effectiveness of self-help groups, savings groups, and education innovations in South Asia and sub-Saharan Africa. He is the program area lead for AIR’s international projects that focus on food security, agriculture, and nutrition. He is the team leader for an evidence synthesis on the partnership pillar of the sustainable development goals and was the principal investigator for the Evidence Consortium on Women’s Groups.

Chinmaya Holla

Chinmaya Holla is a researcher, specializing in gender and social norms, women’s economic empowerment, education programming in humanitarian settings, and foundational learning. He supports quantitative research design, data collection and analysis for numerous projects and contributes to impact evaluations, evidence syntheses, and cost analyses.

Darius Isaboke

Darius Isaboke is an education specialist with significant experience designing and implementing education programs with a specific focus on Kenyan refugee camps. He is a Program Management Specialist at World University Service of Canada.

Timothy Kinoti

Timothy Kinoti is a Social and Economic Analyst, with over 10 years of experience working with Development Actors in the Civil Society and Social Impact Investors from NGO and Public sectors. He has experience working with displaced communities on Education, Health, Economic Empowerment and Managing Grants. He has worked as a lead consultant and employee, leading teams that had local and international members in both the public and private sectors, in Kenya and Internationally.

Hannah Ring

Hannah Ring has 16 years of professional experience in international development programming and research, and she specializes in the evaluation of education and social protection programs. She’s led multiple mixed methods evaluations of education interventions in sub-Saharan Africa, including field work in more than 15 African countries. She specializes in qualitative research methods for evaluative purposes, including participatory approaches and qualitative data collection with protected groups such as refugees, displaced persons, and minors.

Victoria Rothbard

Victoria Rothbard is a Senior User Experience Researcher within the technology and digital banking space. She previously worked at the American Institutes for Research, where she spent 5 years evaluating the effectiveness of EdTech programs in resource-constrained settings, with a focus on refugee populations. Ms. Rothbard holds a Master’s degree in International Development Studies from the George Washington University.