Results from a First-Year Evaluation of Academic Impacts of an After-School Program for At-Risk Students

Abstract

This article presents the research findings of an evaluation of the academic impacts of 21st Century Learning Centers (CCLC) in Louisiana. Using quasi-experimental design, the article operationalizes academic achievement as core and subject test performance on nationally standardized pre- and posttests (Iowa Test of Basic Skills; ITBS). Based on previous research and evaluation requirements, the article (a) employs outcomes of interest to policymakers (standardized test scores); (b) uses program attendance as a key independent variable; (c) uses efficient methods to control for extraneous impact on the dependent variable; and (d) focuses the evaluation on a specific group of student—at-risk children in Louisiana. Findings indicate that the 21st CCLC program is having a positive academic impact on participants who attend the program for 30 days or more. Further, impacts are shared across specific grantee programs, specific subjects, and subgroupings of students. Finally, the study finds that intensity of attendance is positively related to academic impact.

Notes

¹The Fall test is administered only to program participants and control students. The Spring test is part of a statewide accountability program and, as such, is taken by all students in Grades 3, 5, 7, and 9. The use of the same academic year Fall-to-Spring impact measures is preferable to Spring-to-Fall academic impact measures (Kane, 2004).

²As measured by the Iowa Test of Basic Skills (ITBS). The ITBS forms A and B measure skills and achievements in reading, language arts, math, social studies, science, and information sources. The ITBS is used by a number of school districts nationwide to measure academic success. The Core score is the average of a student's performance in reading, language, and math.

³The U.S. Department of Education defines a participant as any child who has attended the program for 30 days or more (21st CCLCs)

⁴In addition to the independent variables of interest discussed later in this article, the following control variables were used: student gender (0 = boy, 1 = girl); minority status (0 = White, 1 = other); recipient of free or reduced-priced lunch (0 = no, 1 = yes).

⁵To measure the impact of the regional programs, we constructed a model with dichotomous regional variables and interacted them with the participation variable. The coefficient of the interaction term can be interpreted as the academic impact of participants (vs. nonparticipants) in that particular region.

⁶See the Appendix for a qualitative overview of grantee programs.

*Minority = 703 African American, 6 Asian, 1 American Indian, 1 Hispanic.

** Bienville is captured by the intercept.

*Not included in computation of core test NCE score. Boldface items indicate a level of statistical significance at or below .05 for the coefficient using a two-tailed test.

⁷In the case of Bienville parish, the chances of finding significant effects appear to be moderated by the small number of participants.

*Initial academic achievement groups have been categorized by quartiles.

Boldface items indicate a level of statistical significance at or below .05 for the coefficient using a two-tailed test.

⁸To gauge program effects on students of varied achievement levels, we divided the entire student sample into equal quarters based on pretest scores. Although the grouping is somewhat arbitrary, it mirrors other education effects research (Decker et al., 2004).

⁹We also regressed attendance as a single continuous variable on the same regressors. Consistent with the results discussed here, the partial slope coefficient was positive and strongly significant (b = 0.029, t = 4.04). Results do appear to be linear. Functionally transforming the continuous attendance variable did not produce a better explanation of variance in the dependent variable.

¹⁰The method employed here is similar to the technique employed for checking for linearity (and testing for a directional hypothesis) within a single variable. We divided the attendance variable into four discrete groups (control, 30- to 59-day participants, 60- to 89-day participants, and participants who participated for more than 90 days). Each of these dummy variables was included in the basic regression equation. The control group, in this case, is captured by the intercept.

¹¹For instance, attendance regressed as a continuous variable may produce a positive slope coefficient, but the effect may be disproportionately produced by medium or high attendees.

¹²It is possible, of course, that the effect observed here may be due to other influence, such as increased attendance in school. But as Huang et al. (2000) pointed out, attendance in school may be endogenously related to attendance in the program. Causality is complex in this regard and deserves further investigation. Because previous performance was controlled for, however, one may be relatively confident that the impact scores here are not due to a corresponding achievement effect that might accompany higher attendance.