Accountability of the informal: challenges and new directions

Abstract

This article focuses on the assessment of impact of La Clase Mágica (The Magic Class), or LCM, on child participants at a time of demands for accountability in Californian schools. LCM is a bilingual/bicultural, computer-based, after-school programme serving Mexican American and Native American communities in San Diego County since 1989. This article discusses the challenges and lessons drawn from the evaluation of this programme and suggests new directions in the accountability of informal learning environments. In particular, we address the results of the Preschool Study, aimed at evaluating the preschool component of LCM, which worked in collaboration with a local Head Start programme for the past 16 years. We argue that the evaluation of the effectiveness of after-school programmes and other informal learning environments is complexly interrelated to funding and programmatic conditions. However, despite the difficulties encountered during the assessment procedure, we agree on the benefits of incorporating a mixed-method, cross-institutional framework, which can serve as the basis of future evaluations of after-school programmes in different contexts.

Keywords:

• accountability
• preschool children
• after-school programmes
• assessment

Acknowledgements

We are particularly indebted to Neighborhood House for sharing their resources and expertise in making this endeavour possible and for their partnership in serving the needs of low-income communities. We are also thankful to Migrant Education Region IX for participating in this venture so willingly. This study would not have been possible without the invaluable help of the Impact Team members, whose enthusiasm, creativity and perseverance during the design, implementation and analysis phases were priceless in completing the project in a timely manner. Our special gratitude to Massimiliano Marinucci, who assisted us with the quantitative data analysis and patiently listened to our queries. Many thanks to Eva Palomares for presenting this article for the authors at the American Educational Research Association (AERA) Annual Meeting (2007).

Notes

1. The Elementary and Secondary Education Act, reauthorized as the No Child Left Behind Act of 2002, is the main federal law affecting education from kindergarten through high school. Elementary and Secondary Education Act is built on four principles: accountability for results, more choices for parents, greater local control and flexibility and an emphasis on doing what works based on scientific research. For more information, you can visit the following webpage: http://www.ed.gov/nclb/overview/intro/index.html.

2. Here we refer to after-school programmes in the United States whose objectives are to enhance participants' academic skills and in the long run have a positive effect on their school performance. Unlike other after-school programmes, these programmes are organized around enrichment activities that have a defined research and evaluation plan and are often tied to state or federal funding.

3. On its second cycle of funding, the Distributive Literacy Consortium sponsored by the Andrew Mellon Foundation learned that evaluation of the participants' cognitive development, products and language and culture remained speculative at best, given the programme's voluntary nature and institutional constraints.

4. While the increase in high-stakes testing worldwide has led to a rise in after-school offerings, variations in international conceptions of quality of programming, assessment and goals and objectives of both local and state mandates make it difficult to situate our efforts within a larger spectrum of after-school programming.

5. In 24 October 2009, La Clase Mágica celebrated “20 years of Innovation” that tweaked the theories, research and practice to provide cognitively enhancing after-school programming for children from under-served communities.

6. In 2006, La Clase Mágica located in San Diego County embodied a federation of six community-based projects supported by a non-profit organization called CASA (Center for Academic and Social Advancement) and the University of California, San Diego (UCSD), in collaboration with Palomar Community College.

6. Head Start is a federally funded project aimed at low-income families. Head Start and Early Head Start are comprehensive child development programmes that serve children from birth to age five, pregnant women and their families. They are child-focused programmes and have the overall goal of increasing the school readiness of young children in low-income families. For more information, see: http://www.acf.hhs.gov/programs/ohs/.

7. Information about the DRDP can be found at http://www.cde.ca.gov/sp/se/sr/drassessment.asp. It is revised every year by the California Department of Education. Bilingualism, however, was not a focus of this study. The DRDP for preschool children only included one question about the child's linguistic profile, which referred to the child's home language and had to be filled out as part of the observer information. Other questions included in this section were the following: (1) What language(s) do you speak with this child? (2) If you do not speak the child's home language, did anyone assist you who does speak it?

8. The three-quarter course, Practicum in Child Development with a socio-historical focus, was co-sponsored by the Department of Communication and Human Development Program.

9. Neither of two other preschool programmes included computers as part of the institutional activities, nor had more than two computers available for the children's use.

10. The names of these programmes were changed to protect participants' identities.

11. For more information visit the Migrant Education Region IX web page at: http://www.sdcoe.net/ssp/migranted/.

12. Pooling across two years (2004–2005 and 2005–2006) turned out to be useful, because from a statistical point of view bigger sample sizes reduce the effect of random variation on the results, supporting better our conclusions. Also, there is no particular reason to analyse these subgroups separately, thus from now and in order to avoid any possible confusion we will always refer to the pooled sample.

13. We could have considered more sophisticated ANOVA models but in this simple setting it was sufficient to consider t-tests in order to evaluate the effect of MCM programme on each domain.