Exploring the Impact of a Community School Reform Initiative on the Literacy Achievement of Middle Level English Language Learners

Abstract

Inequitable learning opportunities are a leading contributor to the persistent literacy achievement gaps evident between historically marginalized students and their more affluent peers. This study investigated the impact of a community school reform effort in an urban middle school. The community school initiative focused on meeting the complex academic and non-academic needs of newcomer (i.e., immigrant and refugee) students and families. The researchers compared the community school students to a propensity score-matched group and found that the community school students significantly outperformed the comparison group in English language arts (ELA) from Grade 6 to 8 and had significantly fewer course failures in Grade 8. Results of this study also indicate the promising contributions community school wrap-around services lend to reducing the literacy learning gap between ELL students and their non-ELL peers in the middle grades.

Keywords:

• community schools
• middle grades
• newcomer populations
• propensity score matching
• literacy

Introduction

Literacy has been called “a cornerstone of our freedom” (Ippolito, Steele, & Samson, 2008, p. 1) because those possessing strong literacy skills are better equipped to form opinions based on research and evidence and to effectively advocate for personal or other meaningful causes. In addition to standing as a foundational building block for societal freedom, enhanced levels of literacy are essential for meeting demands of hiring skilled workers in industrialized nations due to persistent technological advancement and globalization (Berman, Bound, & Machin, 1998). Further, elevated levels of overall population literacy have been linked to a country’s economic development (Coulombe, Tremblay, & Marchand, 2004), and individual earnings and occupational outcomes are positively associated with literacy levels even after controlling for educational attainment (Kerckhoff, Raudenbush, & Glennie, 2001). On a global scale, literacy levels are related to various social and health factors. International studies have found self-confidence and self-esteem to be stronger in individuals with higher levels of literacy (Brigham, 2009; Infante, 2000; McDonald & Scollay, 2009; Metcalf & Meadows, 2009; Prins, 2008), and some health issues directly affiliated with literacy levels are life expectancy (Gilbert, Teravainen, Clark, & Shaw, 2018), infant and child mortality rates (Gakidou, 2013), and basic health knowledge needed for making daily healthy decisions (Rowlands et al., 2015).

Importance of Literacy in the Middle Grades

Because literacy affects all domains of life, much research and a significant body of policy in the United States have been devoted to early childhood reading proficiency to ensure all students read on grade level or receive appropriate interventions by third grade (National Conference of State Legislatures (NCSL), 2019). While focusing efforts on improving early literacy skills is imperative, studying reading and writing growth in the middle grades is also important as this is when research suggests students shift between “learning to read and reading to learn” (Chall, 2000, p. 99). In the middle grades, students begin to interact with complex ideas, assess content, and draw conclusions by engaging with written text (Ippolito et al., 2008). Middle school grades in English language arts (ELA) classes act as a proxy for student reading and writing knowledge. Although middle school students’ grades typically decline over the course of sixth to eighth grade (West & Schwerdt, 2012), research on high school and college readiness has shown that both seventh and eighth grade ELA course grades are significantly predictive of students’ ability to be on-track for high school graduation (Allensworth et al., 2014). Additionally, students who fail their middle grades ELA courses are at greater risk for high school drop-out with less than a 25% chance of being on-track for high school graduation (Allensworth et al., 2014; Balfanz, Herzog, & MacIver, 2007; Baltimore Education Research Consortium (BERC), 2011; Neild & Balfanz, 2006).

Inequities in Middle Grades Literacy Achievement

Multiple studies have revealed the importance of middle grades literacy achievement, as shown through course grades, for future high school success and college readiness. Like this study, they were conducted primarily in diverse urban settings (i.e., Chicago, Baltimore, New York, and Philadelphia). This is largely because students from urban, racial/ethnic minority, low-socioeconomic, and English language learning (ELL) backgrounds are among those considered to be educationally disadvantaged (Kretovics, Farber, & Armaline, 1991), consistently demonstrating significantly lower academic proficiency in literacy skills during the middle grades compared to their more affluent peers (NAEP, 2019).

To be sure, this educational gap in literacy is in part a result of educators and policy-makers in the United States failing to provide historically marginalized students with equitable access to resources necessary to facilitate academic success (Ippolito et al., 2008; Vincent, 2000). One proposed method of improving schooling and leveling the playing field for educationally disadvantaged students is through comprehensive community school reform efforts (Provinzano, Sondergeld, & Knaggs, 2020).

Purpose of the Study

This study investigated the impact of a community school reform initiative on student literacy outcomes. The community school initiative was specifically designed to meet the complex needs of a large urban, racial/ethnically diverse, newcomer middle school population. Findings deepen the understanding of ELA outcomes for middle-grade students at community schools, particularly those of newcomer populations. Furthermore, the study design and results lend additional support to establishing collaborative research partnerships between school systems and teams of educational researchers.

Review of Literature

Literacy Skills Development

Strong language and literacy development are central to an individual’s educational success. Knowledge construction for the domains of literacy begins at birth (Gopnik, 2010), and young children work on acquiring language, reading, and writing skills through social interactions with and observations of humans, books, digital sources, and media (Luo, Tamis-LeMonda, & Mendelsohn, 2019). These informal learning experiences set the stage for more formal literacy education children to encounter when they enter elementary school (Skwarchuk, Sowinski, & Lefevre, 2014), where the path toward academic literacy begins. Academic literacy is “reading proficiency required to construct the meaning of content-area texts and literature encountered in school” (Torgesen et al., 2007, p. 3). Elementary school literacy instruction generally focuses on using simple texts to master five essential reading skills: phonological awareness, phonics, fluency, vocabulary, and comprehension (Read Charlotte, 2018). A main goal is for elementary school students to develop skills necessary to fluently read basic texts and understand meaning. By the time students move into the middle grades, there is a shift in literacy education whereby students must gain more advanced reading practices to sustain levels of proficiency (Snow, Porche, Tabors, & Harris, 2007; Swanson et al., 2016). Middle grade students encounter more complex texts, vocabulary, and content. They are further tasked with critically assessing contexts across disciplines, which makes effective literacy skills critical for learning at this level and beyond (Chall, 2000; Ippolito et al., 2008).

Differential Literacy Skills in the Middle Grades

Research shows that middle grades students are quite heterogeneous in terms of their literacy abilities (Cirino et al., 2013; Firmender, Reis, & Sweeny, 2013; NAEP, 2019). While some students can apply their reading skills across content areas to analyze, synthesize, and evaluate texts, many still struggle with word identification (Cirino et al., 2013), fluency, and/or comprehension (Cirino et al., 2013; Firmender et al., 2013). Students who are not meeting reading proficiency standards at the outset of the middle grades will generally not have the capacity to achieve average performance by the end of eighth grade (Allensworth et al., 2014; Kieffer & Marinell, 2012). Further, failure in an eighth-grade ELA course, or grades below a 2.0 (C average), place students at high risk of not graduating from high school and/or being college ready (Allensworth et al., 2014).

At the adolescent level, varying skills in literacy are often related to demographic variables such as racial/ethnic background, socioeconomic level, and ELL status (Strickland & Alvermann, 2004). Most recent National Assessment of Educational Progress reading (NAEP, 2019) and writing (NAEP, 2011) results show that historically persistent educational gaps still exist between educationally disadvantaged eighth-grade students and their more privileged peers. Overall, only 34% of U.S. students were proficient on the 2019 NAEP reading assessment and 27% on the 2011 NAEP writing assessment. There were significant gaps by race/ethnicity in both reading and writing with White students being at or above proficient (42% in reading, 34% in writing) at significantly greater rates compared to Black (15% in reading, 11% in writing) and Hispanic (22% in reading, 14% in writing) students. Large differences in eighth-grade NAEP reading and writing proficiency were also present by free/reduced lunch (FRL) status with only 20% and 12% of those eligible for FRL earning a score of proficient or higher in reading and writing, respectively, compared to 46% (reading) and 37% (writing) of students who were not FRL eligible. The most substantial eighth-grade NAEP reading and writing proficiency gaps were found between ELL (4% in reading, 1% in writing) and non-ELL students (36% in reading, 28% in writing).

Normative Literacy Practices and Inequality

Large proportions of racial/ethnic minority, low-socioeconomic, and ELL students reside in urban school districts (Perkins & Cooter, 2013; Ruiz-de-Velasco & Fix, 2000; Teale, 2009). Research shows literacy development of students in urban schools is affected by many factors including, but not limited to, poverty levels, student mobility, majority-minority populations, inadequately trained teachers, and ineffective school leadership (Perkins & Cooter, 2013). Alvermann (2002) suggested the “reading, writing and other modes of symbolic communication that are often valued differently by people living in different social and economic structures and holding different political views” (p. 4) can contribute to minoritized students’ lower levels of academic performance. This phenomenon is perpetuated by normative literacy practices.

In plain terms, normative literacy means that “good readers” are defined according to teacher’ behavioral expectations of what is “appropriate” given the curricular structure employed: sitting and looking at the teacher during reading aloud, silence during independent reading, or eyes and pencil on the paper during writing workshop (Souto-Manning, Dernikos, & Yu, 2014). Because normative literacy behaviors are most often those demonstrated by students with higher abilities who are more likely than not from middle-class White households (Cheruvu, Goodwin, & Genishi, 2008; Heath, 1982; Moll et al., 2001), students from diverse backgrounds are often negatively impacted. To ensure classrooms do not take on a normative literacy approach to instruction and become spaces that are not neutral learning environments for all learners (Cahnmann-Taylor & Souto-Manning, 2010; Souto-Manning, 2010; Souto-Manning et al., 2014), urban middle-grade teachers need to implement literacy strategies intended to meet the varying educational needs of diverse students in their classrooms.

Community Schools

Disparities in educational opportunity exacerbate social and economic inequalities for historically marginalized youth. Students of color, and those living in poverty, disproportionately attend segregated and under-resourced schools (Green, 2015), yet societal expectations position schools as a pivotal solution to tackling discriminatory conditions and advancing opportunity for youth (Daniel, Quartz, & Oakes, 2019). In reality, if schools are to be actualized as equal and equitable opportunity spaces, they must be equipped with integrated support to address the holistic needs of students and families. One place where this occurs with regularity is a community school. Described as “an approach to better coordinate schools with local community resources and social service agencies” (J.A. Anderson, Chen, et al., 2019, p. 895), community schools are committed to improving educational outcomes for children by offering essential supports that promote learning and have an explicit focus on the relationship between school and community. Most function at the elementary level, yet there are places (e.g., Oakland and New York) operating community schools at the middle and high school levels.

Research on the overall effectiveness of community schools is emerging (Heers, Van Klaveren, Groot, & Maassen van den Brink, 2016). While there is literature focused on the ways community-based organizations and school districts partner to meet the cultural and linguistic needs of newcomer students (Symons & Ponzio, 2019), little research on full-service community schools designed intentionally with immigrant and refugee students in mind has been conducted. An exception is Oakland International High School, a community school for high school ELL newcomer populations (see Kessler, 2009; Maier & Levin-Guracar, n.d.). However, despite the dearth in research specific to this population, Maier, Daniel, Oakes, and Lam (2017) explained community school research shows students who engage with holistic and integrated student supports achieve higher grades and test scores, in part because of the intentional alignment of school-based services with students’ academic and non-academic needs. Although the evidence base for academic outcomes is limited, “a substantial number of academic studies and program evaluations find small but consistent improvements” (Maier et al., 2017, p. 92) for students in community school settings. More specifically, multiple community school evaluation summaries (Blank, Melaville, & Shah, 2003; Dryfoos, 2000) collectively showed more than 50 community schools out of nearly 70 (74%) demonstrated varying degrees of academic gains in reading.

Context of the Study

Immigrants are considered one of the fastest growing demographic groups in the United States, comprising approximately 26% of the population (Pivovarova & Powers, 2019). Moreover, more than 3 million refugees have resettled in the United States since 1975 (Adel et al., 2019), with the majority resettling in highly populated states like California, Texas, New York, Ohio, and Pennsylvania (Radford & Connor, 2016). Many newcomers are school-aged and, oftentimes, school systems are the first U.S. social institution with which they have sustained engagement (Pivovarova & Powers, 2019). Thus, their experiences while matriculated may either support or hinder their integration (Suárez-Orozco, Suárez-Orozco, & Todorova, 2008).

The need for academic services in school for newcomer students is widely understood and expected, as achievement gaps between ELLs and their non-ELL peers remain large and persistent (August & Hakuta, 1997; Cassady, Smith, & Thomas, 2018; Klinger, Artiles, & Mendez Barletta, 2006). However, successful integration depends on more than academic services; it also depends on how well schools assist students and families in overcoming myriad obstacles in and out of school. As such, educational institutions must have systems in place to support newcomers as they navigate complex academic and non-academic challenges. The School District of Lancaster (SDL) does just this.

SDL is situated in Lancaster City, a mid-size city of approximately 59,000 people in South Central Pennsylvania and the county seat for Lancaster County. Since 2013, Lancaster County resettled 20 times more refugees per capita than the rest of the United States (Gray, 2017), however the region has long been a haven for newcomer populations. Support for refugees dates back 300 years (Hawkes, 2017) and is embedded within the social, cultural, and political fabric of community life. Resultantly, SDL is home to an ethnically diverse and culturally rich student population. More than 11,000 students, who together speak 38 different languages with varying dialects, span 20 schools. Published media reports (Miller, 2017) estimate more than 600 SDL students are refugees, and many reside within the boundaries of Reynolds Middle School.

Serving students in Grades 6–8, Reynolds Middle School had the highest number of refugee students compared to other middle schools in SDL at the time of the study. Affordable housing options in sectors of Lancaster City made it an ideal place for newly arrived populations, and publicly available data (Future Ready PA Index, n.d.) indicated nearly 90% of Reynolds’ students were economically disadvantaged, 19% were ELLs, approximately 21% received special education services, and 6% were homeless at the time of the study. To provide students equitable access to services deemed critical to their educational and life success, Reynolds Middle School partnered with multiple community and faith-based organizations to open a community school with a specific emphasis on meeting the needs of refugee students. In 2015, Reynolds Middle School was renamed and relaunched as the Refugee Center and Community School at Reynolds (RCCSR).

RCCSR served as the hub of activity for community partners, which included community refugee resettlement agencies, who initiated relationships between newly resettled families and the school, Southeast Lancaster Health Center, which staffed the on-site health and wellness clinic, and the Literacy Council of Lancaster-Lebanon, which provided services including but not limited to English language acquisition, high school equivalency exam preparation, and career pathways advising. Following a lead agency model, RCCSR partnered early on with the district’s local Intermediate Unit (IU), IU 13, which served as the coordinating partner responsible for managing and leading the multitude of partnerships and services provided in the school and refugee center. In the state of Pennsylvania, local IUs support school districts and serve as a source of collaboration between schools and community agencies providing direct and indirect classroom support. In their capacity as lead agency for RCCSR, IU 13 employed the community school director (CSD), who oversaw all partnerships between the school and outside agencies, served on the school’s leadership team, and worked intimately with families to ensure obstacles to students’ learning were addressed. IU 13 personnel and programs also contributed, in part, to specialized integration assistance offered through the refugee center, specifically cultural navigation, financial literacy workshops, ESL classes for adults, and citizenship classes for adults.

In tandem, the community school and refugee center operating at RCCSR provided an ideal context for examining the influence of community school programming on middle school students’ literacy outcomes. To some extent, RCCSR embodied the sixteen characteristics described in the landmark position paper This We Believe published by the Association for Middle Level Education (formerly National Middle School Association, 2010).¹ One overarching This We Believe characteristic the school and community had diligently worked toward meeting was Culture and Community. This was achieved by implementing community school programming and refugee services that created an inclusive environment for all to assist students as they navigate social, health, and educational services. Table 1 aligns National Middle School Association (2010) Culture and Community characteristics with RCCSR services.

Table 1 Alignment between National Middle School Association (2010) Culture and Community Characteristics with RCCSR Services

Culture and Community Characteristics	RCCSR Services
School environment: School environment is inviting, safe, inclusive, and supportive of all.	Trauma-informed practices Restorative practices Extended day programming ESL summer programming Welcoming schools program
Adult advocate: Every student’s academic and personal development is guided by an adult advocate.	Onsite cultural navigators Refugee and immigrant education outreach program Onsite caring adult program
Guidance services: Comprehensive guidance and support services meet the needs of young adolescents.	School counseling services School social worker services
Health & wellness: Health and wellness are supported in curricula, school-wide programs, and related policies.	Onsite medical clinic Onsite dental and vision services Onsite mental health counseling Food security program
Family involvement: School actively involves families in the education of their children.	Family refugee integration services Family citizenship preparation classes Adult ESL classes Adult GED classes Monthly Parent and Community Engagement Activities
Community & business: School includes community and business partners.	Onsite community school director Onsite refugee center coordinator Refugee integration services

Purpose and Research Questions

The purpose of this research was to study the impact of a middle grade community school designed to meet the multifaceted needs of urban newcomer students on literacy outcomes across middle grades. Further, an aim of this study was to take a deeper look at community school programming impact on ELL classified students on the same literacy outcomes because of the explicit focus on newcomer needs in the community school. Four research questions guided this study, and all variables are operationally defined in Table 2.

• RQ1. Is there a significant difference in ELA end-of-course grades between students in the community school compared to a matched comparison group from Grade 6–8?

• RQ2. Is there a significant difference in ELA end-of-course grades by the intersection of ELL classification and school from Grade 6–8?

• RQ3. Is there a significant difference in Grade 8 ELA course failures between students in the community school compared to a matched comparison group?

• RQ4. Is there a significant difference in Grade 8 ELA course failures by the intersection of ELL classification and school?

Table 2 Variables and Operational Definitions Aligned with Research Questions

Variable Name	Operational Definition	RQs
ELA end-of-course grades	Dependent variable (continuous) – scale of 0.00 to 4.00 with GPAs representing letter grade criteria of 4.00 (A), 3.66 (A-), 3.33 (B+), 3.00 (B), 2.66 (B-), 2.33 (C+), 2.00 (C), 1.66 (C-), 1.33 (D+), 1.00 (D), 0.66 (D-), 0.33↓ (F)	1 & 2
School	Independent variable (dichotomous) – enrolled at RCCSR or matched comparison school during all three years of middle school	1 & 3
Grade level	Independent variable (categorical) – Grade 6, 7, or 8	1 & 2
ELL x school classification	Independent variable (categorical) – RCCSR non-ELL, RCCSR ELL, matched comparison non-ELL, matched comparison ELL	2 & 4
ELA course failures	Dependent variable (dichotomous) – Grade 8 end-of-course ELA grade of an F or higher than F.	3 & 4

Dependent variable choice for this study was driven by both literature and practical circumstances. While teacher-given grades have been described by some as a more subjective assessment of student learning compared to standardized test outcomes (Malouff, 2008), studies have shown that ELA course grades, in particular eighth-grade course failures, are significantly more predictive of high school graduation compared to standardized test scores (Balfanz et al., 2007; Kurlaender, Reardon, & Jackson, 2008; Neild & Balfanz, 2006). Further, in the state of Pennsylvania where this study took place, examining standardized test scores from year to year was simply impossible as immense structural and content changes in state testing occurred (McCorry, 2015). These changes impacted student testing results negatively as cut score bars were also raised. Thus, state testing results were incomparable over time and not appropriately representative of student ability due to “more demanding standards and very ambitious cut scores” (McCorry, 2015) introduced during the study.

Methods

Research Design

Cochran and Chambers (1965) defined observational studies as those that “elucidate cause-and-effect relationships … [when] it is not feasible to use controlled experimentation” (p. 234) due to the impossibility of randomizing treatment and control group participation. Quasi-experimental studies fall under this classification, and they meet the criteria set forth by the Comprehensive School Reform Quality (CSRQ) Center to be considered scientifically based research (SBR) (Crowley & Hauser, 2007). This study implemented an observational quasi-experimental design with the community school initiative as the treatment, literacy measures as outcomes, and students as experimental units. To bolster inferences made from this study’s results due to the lack of participant random assignment, a comprehensive propensity score matching process was undertaken to generate a demographically and academically baseline matched comparison sample. When rigorously developed propensity score-matched samples are used in observational studies, baseline differences between intervention and comparison groups are often eliminated or reduced and are thereby able to mimic findings from experimentally designed research (Austin, 2011).

Data Collection and Sample

Students (N = 206) who had attended the same middle school for the entire three-year period (Grade 6–8) made up the sample for this study. Half were from RCCSR (CS intervention) and half attended two middle schools across the district (matched comparison group). It was important for students in this study to meet the inclusion criteria of “three-year same middle school attendance” to effectively assess the impact of the middle school reform effort over time. Additionally, because there are multiple community schools in the district, students who had attended an elementary community school and students enrolled in one of the other community school middle schools were excluded from the study. Participant inclusion and exclusion criteria were explicitly used to reduce internal validity concerns related to treatment (i.e., community school) exposure. From multiple Excel files with student-level demographic and end of year course grades provided by SDL’s central data office, the research team found 103 RCCSR students who met the criteria for participation in this study.

Once the RCCSR intervention sample was established, a propensity score-matched comparison group was created using SPSS V25 with the FUZZY extension. Logistic regression on the grouping variable was conducted without replacement to generate a propensity variable for matching comparison students to intervention students. Multiple predictors (i.e., matching variables) were used to develop a model based on comprehensive student-level data for matching including fifth-grade math and reading state proficiency testing levels, gender, race/ethnicity, ELL classification, special education classification, and free/reduced lunch status. Matching variables in this study were explicitly selected as most have been shown to be strongly related to educational achievement (U.S. Department of Education, 2004).

Propensity score matching with near neighbor matching technique (0.20 caliper) was implemented and resulted in a unique 1:1 matched sample of comparison students from across the district for each RCCSR student. Accuracy of the propensity score matching was evaluated using chi-square tests to look for significant differences between groups (CS Intervention vs. Matched Comparison) on each Grade 5 demographic and academic variable used in the model. Significant differences were not found by group on any model variables (p > .05), thus indicating baseline group equivalence. See Table 3 for sample demographics. It is important to note that propensity scores used with small samples can be a limitation and result in sample attrition, meaning not enough cases for matching are available for meaningful analysis (Holmes & Olsen, 2010). While there were only 103 students in the intervention group that matched inclusion criteria, there was a sufficient number of students to select for the matched comparison group (n = 816), which allowed for statistically equivalent 1:1 matched samples.

Table 3 Demographic Data for RCCSR (CS Intervention) and Matched Comparison Group (N = 206)

Demographic Variable Values	RCCSR (n = 103) f (%)	Matched Comparison (n = 103) f (%)
GenderFemaleMale	40 (38.8%)63 (62.5%)	41 (40.5%)62 (60.2%)
Race/EthnicityAsianBlack or African AmericanHispanicMulti-racialWhite	3 (2.9%)25 (24.3%)60 (58.3%)1 (1.0%)14 (13.6%)	5 (6.8%)24 (23.3%)49 (47.6%)3 (2.9%)20 (19.4%)
English Language LearnersYesNo	14 (13.6%)89 (86.4%)	16 (15.5%)87 (84.5%)
Special Education StudentsYesNo	14 (13.6%)89 (86.4%)	15 (14.5%)88 (85.4%)
Free and Reduced LunchYesNo	89 (86.4%)14 (13.6%)	89 (86.4%)14 (13.6%)
Grade 5 Reading ProficiencyProficientNot proficient	38 (36.9%)65 (63.1%)	32 (31.1%)71 (68.9%)
Grade 5 Mathematics ProficiencyProficientNot proficient	59 (57.3%)44 (42.7%)	53 (51.5%)50 (48.5%)

Data Analysis

All research questions were analyzed using SPSS V25. Research questions 1 and 2 were both analyzed with Factorial Repeated Measures ANOVAs. RQ1 used a 2-Between, 3-Within Repeated Measures ANOVA, and RQ2 a 4-Between, 3-Within Repeated Measures ANOVA. Assumptions for Repeated Measures ANOVAs include use of dependent samples, normal distribution of dependent variables, and sphericity. Samples in this study were dependent as outcomes assessed over time came from the same students. Repeated Measures ANOVAs are robust to violations of normality with samples of 200 or more (Tabachnick & Fidell, 2001). With more than 200 students in this study, minor deviations in skewness and/or kurtosis were not problematic. Mauchley’s Test of Sphericity indicated no statistical significance (p > .05) meaning no corrections were necessary. Thus, all statistical assumptions were met for implementing Repeated Measures ANOVAs in this study.

Research questions 3 and 4 both implemented Chi-square tests for analysis: 2 × 2 for RQ3 and 2 × 4 for RQ4. Chi-Square tests have two statistical assumptions: (a) independence of observations and (b) no more than 20% of the expected counts are less than five with all individual expected counts one or greater (Yates, Moore, & McCabe, 1999). All participants were only represented in one cell for each analysis, and all expected counts were acceptable rendering chi-square tests appropriate for this study. If significant differences were found in any analysis, corresponding post-hoc tests were run to identify the specific nature of the differences.

Results

ELA End-of-Course Grades

By school and over time. There was a significant difference in ELA average grades by school across Grade 6–8; F(2) = 68.34, p < .001. The effect size was large (η² = 0.279) with 27.9% of the variance in ELA grade differences due to school attending over time. Table 4 presents descriptive statistics for this analysis. Post-hoc analyses further revealed there was a significant difference in ELA end-of-course grades at each grade level with RCCSR students performing significantly stronger than the matched comparison students at each time point (p < .001).

Table 4 Means and Standard Deviations for ELA End-of-Course Grades by School and Grade Level

School	Grade 6M (SD)	Grade 7M (SD)	Grade 8M (SD)
RCCSRMatched comparison	2.43 (1.14)1.78 (1.08)	2.81 (0.95)1.92 (1.13)	2.50 (1.22)0.90 (0.88)

School by ELL classification over time. A statistically significant difference in ELA average grades was found for school x ELL classification across Grade 6–8; F(3) = 22.86, p < .001. A large effect size (η² = 0.279) was noted with 28.2% of the variance in ELA grade differences due to school x ELL classification. Table 5 provides descriptive statistics. Post-hoc analyses showed similar patterns of significant differences by subgroup at each grade level: (a) RCCSR non-ELL and ELL students performed statistically similar to each other; (b) matched comparison non-ELL and ELL students performed statistically similar to each other; and (c) RCCSR students outperformed matched comparison students regardless of ELL Classification (p < .001).

Table 5 Descriptive Statistics for ELA End-of-Course Grades by School/ELL Classification Over Time

School x ELL Classification	Grade 6M (SD)	Grade 7M (SD)	Grade 8M (SD)
RCCSR non-ELLRCCSR ELLMatched comparison non-ELLMatched comparison ELL	2.46 (1.17)2.25 (0.93)1.77 (1.10)1.89 (0.96)	2.79 (0.96)2.83 (0.85)1.90 (1.13)2.03 (0.91)	2.48 (1.21)2.64 (1.32)0.91 (0.89)0.78 (0.83)

Grade 8 ELA Course Failures

By school and over time. Chi-square test results indicate a significant difference in percentage of students at-risk for high school dropout due to eighth grade ELA failures by school; χ²(2) = 21.08, p < .001. RCCSR had less than 10% of students fall in this at-risk category (9.4%), and the matched comparison group had more than a third of students classified at-risk (37.8%). See Figure 1 for a graphical representation of these findings. The effect size was large (η² = 0.279) with 27.9% of the variance in ELA failures due to school.

Figure 1. Comparison of RCCSR and matched comparison sample high school dropout at-risk rates based on 8^th grade ELA failures. Significantly more students in the matched comparison sample cohort were at-risk compared to the RCCSR sample

School by ELL classification. Chi-Square results showed a statistically significant difference in Grade 8 ELA failures identified by school/ELL classification subgroups; χ²(2) = 21.41, p < .001. A large effect size (η² = 0.287) was found with 28.7% of the difference in Grade 8 ELA failures due to school/ELL classification. Post-hoc analyses further showed that RCCSR non-ELL (9.9%) and ELL (6.7%) students had statistically similar failure outcomes (less than 10% failures) (p > .05); comparison non-ELL (37.0%) and ELL (44.4%) students had a statistically similar proportion of Grade 8 ELA failures (i.e., more than a third) (p > .05); and regardless of school/ELL classification, RCCSR students had significantly fewer Grade 8 ELA failures than the matched comparison students (p < .001). Figure 2 provides a graphical depiction of these findings.

Figure 2. Comparison of high school dropout at-risk rates based on Grade 8 ELA failures by the intersection of school and ELL status. Significantly more students in the matched comparison sample cohort (ELL and non-ELL) were at-risk compared to the RCCSR sample (ELL and non-ELL)

Discussion

Differences in ELA Outcomes by School

Research has shown that it is most common for students’ grades to dramatically decline from Grade 6 to Grade 8 (West & Schwerdt, 2012), particularly in urban settings. Interestingly, in our study, all students showed slight increases in ELA grades from Grade 6 to Grade 7, and students’ ELA grades then dipped in Grade 8. However, matched comparison students’ ELA grades significantly dropped by over the middle school timeframe (approximately averages of C- in Grade 6 to D in Grade 8), aligning with prior literature. Contrastingly, RCCSR students defied this expected overall downward trajectory and instead saw slight increases in their ELA grades with average grades rising from an approximate average C+ in Grade 6 to nearly a B- average in Grade 8. While RCCSR ELA results over time are not aligned with typical urban middle school literature, these findings are consistent with community school research on reading outcomes from various grade levels that have demonstrated improvement in this domain across many community schools (Blank et al., 2003; Dryfoos, 2000). Further, these findings fit within literature that has shown when students’ academic and non-academic needs are met through comprehensive school-based services, higher academic outcomes are found (Maier et al., 2017; Sondergeld, Provinzano, & Johnson, 2020).

In addition to the statistically significant difference in ELA grades by group over time, this study’s results are also important from a practical perspective. More than a third of matched comparison students (37.8%) failed Grade 8 ELA and were thus predicted to have more than a 75% chance of high school drop-out and lacking college readiness due to illiteracy issues (Allensworth et al., 2014). In comparison, nearly all RCCSR students (90.6%) passed their Grade 8 ELA course suggesting they are at least minimally on target to graduate from high school based on literacy skills. Therefore, community school efforts at RCCSR seemed to have propelled ELA outcomes of commonly considered educationally disadvantaged students above what would have been expected for them, as benchmarked by matched peer results in this study and noted in prior literature (NAEP, 2011; NAEP, 2019; Strickland & Alvermann, 2004).

Digging Deeper: School Attending vs. ELL Status

In terms of closing educational gaps in literacy achievement for a newcomer population, findings from this study show promise. Interestingly, gaps in literacy abilities between ELL students and their non-ELL peers, as demonstrated by ELA course grades, were not found at any grade level within either middle school. This is contradictory to what past literature has indicated whereby significant academic deficits, including literacy skills, have been documented for ELL students (August & Hakuta, 1997; Cassady et al., 2018; Klinger et al., 2006; NAEP, 2011; NAEP, 2019). Consequently, the middle school that students attended in this study appeared to have had a greater impact on literacy outcomes over ELL status, as revealed through statistically similar ELA grades over time and Grade 8 pass/fail markers between ELL and non-ELL students in their respective schools. Yet, statistically significant and practically meaningful differences in these outcomes existed between schools regardless of ELL status, resulting in RCCSR ELL students performing significantly stronger than both ELL and non-ELL matched comparison students in middle-level literacy skills.

Participant groups in this study had equally large populations of students from marginalized groups (e.g., urban, free/reduced lunch, racial/ethnic minorities, ELL) that often struggle in their literacy skills (NAEP, 2019; Strickland & Alvermann, 2004) due to lack of essential resources needed to enable academic achievement (Ippolito et al., 2008; Vincent, 2000). Literature suggests that inclusive wrap-around services offered at RCCSR for students and families likely contributed to this difference (Heers et al., 2016). Based on our results and supporting literature, we suspect that the school environment, adult advocates, guidance services, health and wellness, family involvement, and community/business support and programs offered at RCCSR—which were aligned with This We Believe (National Middle School Association, 2010)—provided a welcoming culture and community for all students and families which allowed for similar community school impact on students regardless of demographics and background. Future research should focus on qualitative investigations of why these differences were found and how RCCSR programming facilitates these academic achievements within their school for a more comprehensive understanding of community school impact at RCCSR.

Methodological Considerations and Limitations

Studying school reform impact in urban largely racial/ethnic minority schools is complex as there are many interwoven components concurrently in motion that typically cannot be untangled. Further, traditional school quality measures and reference points are problematic as they simply do not provide a robust and equitable assessment (T. Anderson, Blount, et al., 2019). Common measures of school quality are often not well aligned with school reform initiatives (i.e., community schools) to focus more broadly on holistic urban student needs beyond academics alone. In the absence of more appropriate metrics for evaluating urban school reform efforts, there are methodological design options that can offer more accurate study outcomes and inferences using measures that are available if well-designed and thoughtfully conducted.

Implementation of true experimental designs utilizing random assignment of students to educational treatment and control groups is largely impossible in an educational setting (Creswell, 2008) and could be perceived as unethical in certain situations. Thus, true experiments in education are rarely conducted. Instead, it is more common for quasi-experimental research to be performed using intact groups of students for comparison. Although quasi-experimental studies are considered scientifically based research suitable for studying comprehensive school reform initiatives (Crowley & Hauser, 2007), these designs may take on many forms varying in rigor. To address internal validity concerns (i.e., impact of intervention) associated with quasi-experimental research, it is important to control for known extraneous or confounding variables, which when left unaddressed could impact baseline differences and overall inferences (Austin, 2011; Creswell, 2008; Crowley & Hauser, 2007; Ewert & Sibthorp, 2009).

In this study, precautions to minimize what Ewert and Sibthorp (2009) classified as precursor confounding variables were taken through careful propensity score matching of students based on demographic (i.e., SES, ELL status, special education classification, race/ethnicity, gender) and academic (i.e., pre-intervention state standardized test scores in reading and mathematics) variables known for their strong relationship with educational achievement (U.S. Department of Education, 2004). However, as with any propensity score-matched study, it is important to note that if any critical variable was omitted during the matching procedure, groups could remain unbalanced and results could thus be severely biased (Streiner & Norman, 2012). Based on prior research, we believe our propensity score model to be strong in reducing internal validity threats caused by baseline differences, allowing findings from this study to mirror those from a true experiment (Austin, 2011). One-to-one statistical matching of community school students with comparison students statistically similar at baseline provides greater confidence in the reported impact of RCCSR programming than if a non-statistically matched comparison group or cohort had been used. This is particularly true when group propensity scores span a similar range so that potential intervention participants are not dropped from the study due to an unsuitable match (Streiner & Norman, 2012), as was the case in this research.

Additional confounding variables that may arise during an intervention study have been termed concomitant variables (Ewert & Sibthorp, 2009). Common concomitant variables in educational intervention studies are often related to how students are taught. Curriculum is one educational component that could interfere with outcomes if not held constant or an intended piece of the intervention. In our study, there was district-wide curriculum in place aligned with state standards that all teachers were expected to implement. Thus, ELA curriculum was held constant across intervention and comparison groups. For researchers to obtain data and contextual background appropriate for more rigorous educational intervention research to occur, it is critical to form mutually beneficial partnerships with schools and districts as one entity alone typically cannot conduct this level of design, research, and analysis.

Concluding Thoughts

Literacy skills are central to the development of an equitable society where all members have access to skilled jobs that pay a living wage (Berman et al., 1998; Kerckhoff et al., 2001), develop self-confidence related to communication abilities (Brigham, 2009; Infante, 2000; Prins, 2008), and have basic health needs met (Gakidou, 2013; Gilbert et al., 2018; Rowlands et al., 2015). Nevertheless, traditional public schooling has disproportionately invested in children along racial/ethnic and socioeconomic lines (J.A. Anderson, Chen, et al., 2019), resulting in educational gaps in literacy skills during the middle grades between historically marginalized students (i.e., urban, racial/ethnic minorities, ELL) and their more affluent peers (NAEP, 2011; NAEP, 2019; Strickland & Alvermann, 2004).

With the comprehensive approach to community schooling taken by RCCSR focusing on overall student social, emotional, health/wellness, and academic needs with the support from community members and agencies, literacy outcomes were positively impacted. Further, similar achievements were rendered for even newcomer students marginalized across multiple contexts and during the middle grades—a time when it is customary for student grades to plummet. Although literacy successes were realized at RCCSR through the holistic educational approach intrinsic in community schools, there is still much ground to be made. For educationally disadvantaged students to reach literacy levels necessary to be adequately prepared to tackle post-secondary education, the community school impact must be strong enough to sustain through high school. This, however, is an unknown that needs to be investigated further as community schools and research around them grows to greater maturity.

Notes

¹ 16 Characteristics of Successful Middle Schools: 1) Value Young Adolescents, 2) Active Learning, 3) Challenging Curriculum, 4) Multiple Learning Approaches, 5) Varied Assessments, 6) Shared Vision, 7) Committed Leaders, 8) Courageous & Collaborative Leaders, 9) Professional Development, 10) Organizational Structures, 11) School Environment, 12) Adult Advocate, 13) Guidance Services, 14) Health & Wellness, 15) Family Involvement, 16) Community & Business.