Bilingual program effectiveness: an evaluation of meta-analytic methods and findings

ABSTRACT

Research syntheses that evaluate bilingual program effectiveness have grown exponentially since the 1980s. Contradictory to earlier anti-bilingual findings, these research syntheses, including statistical meta-analyses, have converged on findings supporting L1 teaching. This study examines the methodological soundness of eight statistical meta-analyses to provide methodology validation to this convergence conclusion. It examines the extent to which important methodological assumptions and conditions meta-analysis builds on are adequately considered and addressed. This examination lends some methodological support to the certainty of L1 teaching’s positive impact. However, it also reveals some concerns, particularly regarding the source of primary studies that these meta-analyses include, consideration of primary studies’ heterogeneity and meta-analyses’ reporting gaps. Based on these findings, I recommend updating research about bilingual program effectiveness, giving greater attention to heterogeneity and developing reporting standards for future meta-analyses.

KEYWORDS:

• Bilingual program effectiveness
• heterogeneity
• meta-analyses
• reporting standard

Disclosure statement

No potential conflict of interest was reported by the author(s).

Geolocation information

South Africa.

Notes

1 L2 is usually English in countries where English is the dominant or medium of instruction in schools. However, in Europe or other countries, L2 can be the respective national language.

2 Interdependence and transfer hypothesis argued that certain L1 knowledge can be positively transferred during the process of L2 acquisition. Time on task argument posited that time spent learning in L1 is time lost learning L2.

3 For example Proposition 227 in California in 1998 and Proposition 203 in Arizona in 2000.

4 Including both program design and implementation and “between schools within the same project, between classrooms within the same school, and between students within the same classroom” (Piper 1984) often ‘in a state of flux … [often change and evolve as they] adapt to local conditions and improvements in design’ (Lam 1992, 197).

5 For example, Baker, Basaraba, and Polanco (2016) listed over 15 types (825–826); or the reasons for incorporating L1, including additive or subtractive, elective or circumstantial, emergent, proficient, or dynamic. This is often linked to whether the program is aimed for majority students or minority students. With minority students, there also transitional models that aim to replace L1 by L2 when students are sufficiently competent with L2 (usually after a few years) and maintenance models where L1 is expected to remain the instructional language for longer years.

6 Cheung and Slavin (2012) reported two studies: one on language of instruction (257–365); the other on effective reading programs for ELLs (365–389). This review only concerns the 1st study per the focus of this study.

7 According to Cohen (1977), .20 is considered a small effect size; .50 medium and .80 large. This guideline needs to be benchmarked in each discipline. In bilingual programs, .20, .50 and .80 was ‘interpreted to roughly equal to two, five and eight months’ advantage for bilingual education programs’ (McField and McField 2014, 269).

8 Slavin and Cheung (2005) and Cheung and Slavin (2012) included both standardized and self-made but only ‘if experimenter-made measures were used, they were included only if there was evidence that all groups focused equally on the same outcomes” (2012, 356). Types of outcome (standardized and others) was coded in Reljić, Ferring, and Martin (2015) but not discussed. Outcome measures were outlined in individual study descriptions in McField (2002), but not discussed and unclear whether it was coded as study variable. However, from information from study description of the primary studies, it seems that majority used standardized test. None of the meta-analyses differentiated the standardized or self-made measures in their analysis. This aspect was not specifically mentioned in Greene (1997), Francis, Lesaux, and August (2006), Rolstad, Mahoney, and Glass (2005) and Willig (1985).

9 Francis, Lesaux, and August (2006) and Slavin and Cheung (2005) focused on L2 literary only and produced a higher effect size (.62 and .45– .53 respectively.) Slavin and Cheung’s (2005) calculation also show the difference between using random effect model (.54) and fixed effect model (.45).

10 Overall L2 literary effect sizes, except that from Rolstad, Mahoney, and Glass (2005).

11 However, discussion and interpretation of the overall effect size as well as effect size for individual studies were often made on the point estimate only (Table 2), probably due to the cumbersome nature of including intervals. There are primary studies with similar effect size magnitude but the effect size was not significant, e.g. for Kaufman (1968), therefore reporting on point estimate was insufficient.

12 Usage of ** and * indicates .01 and .05 p-value.

13 Two-way bilingual programs: .10 (357), paired model: .30, transitional: –.01. No CI or Z scores were given.

14 Although the authors distinguished five types of bilingual programs: transitional, developmental, English as a second language, English only for ELLs and English only for non-ELLs, effect size was calculated only for transitional bilingual programs: –0.1, SD = .45 and developmental bilingual programs: .18, SD = .86 (589).

Additional information

Notes on contributors

Ke Yu

Prof Ke Yu is an associate professor at the Department of Education Leadership and Management. She worked at the Human Sciences Research Council before joining UJ. Her PhD investigated researcher-researched relationship (University of Pretoria). Her Masters and Bachelor degree were in business management and administration in Norway and China respectively. Self-claimed to be a multipotentialite, she is a critical thinker with extensive interdisciplinary knowledge and a wide range of interests. Her publication spans authoring and co-authoring of over 35 book chapters, journal articles, reports and policy briefs and 10 conference presentations. She publishes widely in inter-disciplinary fields, including meta science, knowledge production, research ethics, learning and communication theories, educational technology (and implication for the society), education policy, cultural studies, bilingual education, higher education studies and China.