A Randomized Experiment of a Cognitive Strategies Approach to Text-Based Analytical Writing for Mainstreamed Latino English Language Learners in Grades 6 to 12

Abstract

This study reports Year 1 findings from a multisite cluster randomized controlled trial of a cognitive strategies approach to teaching text-based analytical writing for mainstreamed Latino English language learners (ELLs) in 9 middle schools and 6 high schools. There were 103 English teachers stratified by school and grade and then randomly assigned to the Pathway Project professional development intervention or control group. The Pathway Project trains teachers to use a pretest on-demand writing assessment to improve text-based analytical writing instruction for mainstreamed Latino ELLs who are able to participate in regular English classes. The intervention draws on well-documented instructional frameworks for teaching mainstreamed ELLs. Such frameworks emphasize the merits of a cognitive strategies approach that supports these learners’ English language development. Pathway teachers participated in 46 hrs of training and learned how to apply cognitive strategies by using an on-demand writing assessment to help students understand, interpret, and write analytical essays about literature. Multilevel models revealed significant effects on an on-demand writing assessment (d = .35) and the California Standards Test in English language arts (d = .07).

Keywords:

• Adolescent literacy
• secondary English language arts instruction
• process writing
• strategy instruction
• teacher professional development
• multi-site cluster randomized controlled trials

ACKNOWLEDGMENTS

We gratefully acknowledge funding from the U.S. Department of Education's Institute of Education Sciences, under grant number R305W06016. The views expressed in this paper reflect the opinions of the authors and not the funding agency or the authors' respective institutions. We are grateful to the students, teachers, principals, and administrators in the Santa Ana Unified School District for supporting the implementation of the study.

Notes

We estimated the minimum detectable effect size (Bloom, 2005), which is the smallest true impact that can be detected with 80% power using a two-tailed test with alpha set at .05. In our multisite cluster randomized field trial in which teachers were placed into school by grade blocks and then randomly assigned to conditions, we used Optimal Design (Raudenbush, Liu, Spybrook, Martinez, & Congdon, 2006) to estimate the minimum detectable effect size based on the following design parameters: the number of schools (K = 15), the anticipated number of teacher clusters per site (J = 8), two different estimates of the intra-class correlation (ρ = .05 and .10), the percentage of the variance in the student posttest scores explained by the pretest covariate (R ² = .50), and the power of the blocking variable (B = .05). We used the district's average class size of 30 students to estimate the number of students per cluster (i.e., classroom). Based on the parameters of our study design, there was sufficient power (80%) to detect a standardized mean difference of .12 on the student outcome measures, which is typical of effect sizes generated by randomized experiments of cognitive strategies instruction in the secondary grades (Slavin et al., 2008).

^a n = 51.

^b n = 52.

The description of the CELDT performance levels are as follows: (a) Beginning = Students performing at this level of English language proficiency may demonstrate little or no receptive or productive English skills. They may be able to respond to some communication tasks. (b) Early Intermediate = Students performing at this level of English language proficiency start to respond with increasing ease to more varied communication tasks. (c) Intermediate = Students performing at this level of English language proficiency begin to tailor the English language skills they have been taught to meet their immediate communication and learning needs. (d) Early Advanced = Students performing at this level of English language proficiency begin to combine the elements of the English language in complex, cognitively demanding situations and are able to use English as a means for learning in other academic areas. (e) Advanced = Students performing at this level of English language proficiency communicate effectively with various audiences on a wide range of familiar and new topics to meet social and academic demands. To attain the English proficiency level of their native English-speaking peers, further linguistic enhancement and refinement are necessary.

We used the following algorithm: ([rater 1 + rater 2]/2) + rater 3.

^a n = 42.

^b n = 44.

In our follow-up analyses, we fit a multilevel model where the ALA posttest score was predicted by the pretest classroom mean, the main effect of treatment, six grade-level dummy variables (Grade 7–12; Grade 6 = omitted grade), and interactions between treatment and each of the six grade-level dummy variables. In this model, none of the Treatment × Grade interactions was statistically significant. Moreover, the variance of the treatment effect across blocks was .28, which was similar to the variance of the treatment effect without the interactions as reported in the bottom panel of Table 5.

Table 5 Hierarchical linear models used to predict student-, classroom-, and block-level achievement outcomes for the final sample