Assessing the Impact of an RPP on a Large Urban School District: The Case of CAFÉCS

ABSTRACT

This study analyzes the impact of the Chicago Alliance for Equity in Computer Science (CAFÉCS) Research Practice Partnership (RPP) on the Chicago Public School (CPS) Office of Computer Science (OCS). Using a qualitative analysis drawing on data from leadership team meetings, published articles and presentations, and evaluation reports from 11 years of the partnership, we utilized a framework developed by the CAFÉCS leadership team to document the impact on district (1) Programs, (2) Research, (3) Organizational Structures, and (4) Policies leading to (5) Equitable Results for students, condensed as PROSPER. In particular, we explore the role of the RPP in supporting a 500% increase in graduating students who completed at least one high school computer science (CS) course between 2016 and 2020 in Chicago Public Schools (CPS). This study adds to the existing literature on assessing the impact of RPPs through the development of the PROSPER Framework, which may be a useful tool to help RPPs examine impacts on partner districts.

Introduction

Research-practice partnerships (RPPs) are a potentially useful strategy to engage a diverse group of practice, community, and research professionals to support educational improvement. In recent years, many RPPs have launched, in part due to funders elevating and supporting this collaborative approach to understanding and addressing complex issues facing school districts today (Arce-Trigatti & Farrell, 2018). Scholarly research has begun to document the benefits of engaging in collaborative research, but the research on RPP outcomes and impacts is still nascent. Regardless, many partnerships must document their progress toward goals and justify their benefits to funders and partners. This documentation is typically in the form of annual reports submitted to funders or in partnership updates shared with school boards or district leaders.

This study examines one long-standing partnership’s approach to documenting its impact in a large urban school district. We describe the collaborative development of a framework to assess how the partnership supported improvements to district (1) Programs, (2) Research, (3) Organizational Structures, and (4) Policies leading to (5) Equitable Results (PROSPER) for students. This framework was used to document partnership impact across these areas, drawing on 11 years of RPP-generated dissemination products, grant proposals, evaluation findings, and information-gathering sessions with the leadership team. Through this analysis we aim to support RPPs in the complicated, but much-needed, process of assessing partnership impact.

The context

In 2016, only 2,300 of 20,348 students (approximately 11% of graduates) graduated from Chicago Public Schools (CPS) with at least one high school computer science (CS) credit, and the demographics of students receiving this credit did not match the demographics of the district. For example, female students made up 51% of the CPS population but comprised only 41% of students enrolled in a CS course (McGee, Dettori, & Rasmussen, 2022). In 2020, just four years later 14,000 of 21,069 students (approximately 66% of graduates) graduated from CPS with at least one high school CS credit (McGee, Dettori, & Rasmussen, 2022). More importantly, the demographics of students taking CS matched that of the district, an incredible testament to the district’s commitment to equity and access (McGee, Dettori, & Rasmussen, 2022). This dramatic 500% increase in participation begs the question, what happened?

This large shift in access to computer science was the result of the enactment of a graduation requirement policy that mandated all students in CPS complete one computer science class (or request a waiver if they were participating in a career program, such as International Baccalaureate [IB] or Career & Technical Education [CTE]), starting with the graduating class of 2020. The successful implementation of this policy required a tremendous amount of support and effort from many across the district and city, including efforts of the Chicago Alliance for Equity in Computer Science (CAFÉCS) RPP.

Background: The Chicago Alliance for Equity in Computer Science (CAFÉCS) RPP

The CAFÉCS RPP has been supporting CS educational improvement in the district since 2009, when it was formed by a group of people with a shared goal to ensure all CPS students have access to compelling and relevant CS experiences. This team included CPS CS teacher Don Yanek, district administrator Brenda Wilkerson, and three university computer scientists, Lucia Dettori (DePaul University), Ron Greenberg (Loyola University Chicago), and Dale Reed (University of Illinois Chicago). Over time, this group grew to include CPS employee Andy Rasmussen, as well as Steven McGee, an educational researcher and president of The Learning Partnership.

CS education was gaining national attention as a critical component of a twenty-first-century education, and the National Science Foundation launched the CS for All RPP program in 2016 with the goal to “provide all U.S. students with the opportunity to participate in computer science (CS) and computational thinking (CT) education in their schools at the pre-K–12 levels” (National Science Foundation, 2020). The Chicago team immediately recognized that the RPP model fit their organic approach initiated many years before.

CAFÉCS received one of the few large grants awarded in the first RPP award cycle, and CAFÉCS formalized as an RPP in 2017. At this time, Erin Henrick, President of Partner to Improve (PTI) and lead author of the RPP effectiveness framework (Henrick et al., 2017), joined the partnership as the external evaluator.

As a formalized RPP, the CAFÉCS’ mission is “to engage in research and development that enables CPS to ensure that all students in Chicago participate in engaging, relevant, and rigorous computing experiences, increase opportunities for all students to pursue computing pathways and prepare all students for the future of work” (CAFÉCS, 2023a). Starting in October 2017, the first identified goal of the RPP was to understand and address variation in the implementation of Exploring Computer Science (ECS), the curriculum and teacher professional development program used in the introductory CS course across CPS. Since 2017, CAFÉCS has been continuously funded to support the goals of the CPS Office of CS (OCS). In total the partnership has secured 12 grants totalling more than $10 million in external funding between 2011 and 2022 (CAFÉCS, 2023b).

The members of the RPP intentionally organize their work to address the needs of the OCS, as set forth by OCS leadership, through collaborative research and inquiry. CAFÉCS engages in a variety of collaborative activities to increase the likelihood of impact within the district. For example, the team meets weekly for 90 min to ensure all partners are aware of what is happening in the partnership. These meetings provide a venue for the team to engage in collective decision-making processes around programmatic support, research, and dissemination efforts. The team also holds quarterly half-day leadership retreats to focus on strategic planning and partnership health. This working structure has proved to be successful, and partners regularly celebrate the longevity and success of the partnership in yearly evaluation-focused interviews.

CAFÉCS utilizes a problem-solving cycle (see Figure 1) to understand and address problems of practice facing the CPS Office of Computer Science (Lewis et al., 2022). This cycle includes five phases: (1) the OCS identifying a problem or issue to address with research; the CAFÉCS team (2) first brainstorming potential hypotheses and then collaboratively filtering the hypotheses to determine the appropriate research approach to use to address the concern (e.g., hypothesis testing, research and development, continuous improvement, or evaluation), (3) engaging in data analysis, and (4) discussing implications of findings from the research with the goal that (5) the research informs OCS strategies and next steps for the CAFÉCS research agenda.

Figure 1. The CAFÉCS Problem-solving Cycle.

The core CAFÉCS team has remained constant amid numerous changes within the district. Since 2017, the OCS has had five different directors, and in 2023, the office moved from the Office of College and Career Success, where it was formed, to the Department of STEM within the Office of Teaching and Learning. This most recent move better reflects the district’s vision for CS as a foundational skill for all students and its support for CS integration into all subjects within CPS, facilitating opportunities for cross-curricular integration with math and science.

Given the longevity of the partnership, the abundance of evidence available through internal documents and published study results, and the leadership team’s desire to contribute to the understanding of RPP impact, the CAFÉCS partnership is an ideal case study to examine RPP impact in an urban school district. This study seeks to answer the question: How did the CAFÉCS RPP impact equitable CS outcomes for CPS students?

Literature review

Education RPPs are an important part of the educational ecosystem that connects research, policy, practice, and community work in the United States. The goal of RPPs’ long-term collaborative approach is to address persistent challenges and systemic inequities in schools and communities by supporting “educational improvement or equitable transformation through engagement with research” (Farrell et al., 2021, p. V). RPPs work to accomplish this goal through building trust and cultivating partnership relationships, conducting research to inform action, supporting practice organizations to achieve their goals, producing knowledge to inform educational improvement efforts more broadly, and building the capacity of team members and organizations to engage in partnership work (Henrick et al., 2017). Assessing the impact of RPPs was identified as a key challenge early on, as RPP leaders indicated that aspects of the local context outside of the partnership’s control often strongly influenced the impact of the partnership progress (e.g., high turnover) (Henrick et al., 2017). However, assessing RPP impact and progress toward partnership outcomes was also identified early on as an essential aspect of the work (Henrick et al., 2017). The majority of RPP research to date generally explores RPP processes and ways of working and does not attempt to study RPP outcomes (Schneider, 2020) or answer the question: “What did the partnerships change, not just in terms of research use or service delivery, but in what matters the most, which is improved outcomes for students?” (Schneider, 2018).

Of the handful of studies that do analyze RPP impacts, one study of an RPP informally known as MIST (Vanderbilt University Peabody College, 2024) is relevant to the present study. The study (Henrick et al., 2018) sought to examine the impact of partnership recommendations and found that, while the districts acted upon 67% of the partnership recommendations, only 17% of those implemented recommendations were found to be “successfully implemented.” This analysis led to a deeper understanding of key factors that influenced successful district impact: the recommendations that were implemented successfully directly aligned with current district priorities, and the district RPP members implementing the recommendations had the expertise and authority to support implementation.

PROSPER framework development

Given the challenges to formally assessing RPP impacts, the CAFÉCS leadership and evaluation team decided to tackle this issue by focusing specifically on understanding how the CAFÉCS partnership impacted the district’s goal of providing all students high-quality CS experiences. We began by conceptualizing a framework for understanding how and in what ways the CAFÉCS RPP intended to impact progress toward this goal in CPS. The team engaged in three co-design meetings to develop and iterate on the framework. These co-design meetings were framed around the question, How did the CAFÉCS RPP impact CS equitable outcomes for students in CPS? The team identified the main areas of impact as they reflected on 11 years of grants, publications, evaluation findings, and experiences. This led to the identification of five areas of RPP district impact to investigate further: (1) Programs, (2) Research, (3) Organizational Structure, and (4) Policy leading to (5) Equitable Results. The resulting PROSPER Framework is visualized in Figure 2. This figure depicts the relationships between the RPP and these five areas. As the model shows, RPPs provide resources, research and expertise to impact district programs, organizational structures, and policies in order to achieve equitable results.While we recognize that these five areas are not an exhaustive list of possible impacts an RPP can have on a school district, they are viewed as key levers in the CAFÉCS context. We see this framework as distinct from the updated Henrick et al. (2023) five dimensions of RPP effectiveness framework in that the Five Dimensions Framework outlines broadly what RPPs are aiming to accomplish, while this framework goes deeper into how the partnership aimed to impact the practice organization in order to help the district achieve its goals (dimension three of the Five Dimensions Framework). In what follows, we describe each component of the framework.

Figure 2 The PROSPER Framework.

Programs

One way RPPs can impact a district’s educational improvement and transformation efforts is through district programs. RPPs have supported schools and districts to identify, develop, deliver, and support the implementation of programs (Farrell et al., 2021). Programs can be interpreted broadly and include structured activities involving district students and staff, such as district- and school-level professional development, coaching, out-of-school time activities, implementation of specific curriculum, or combinations of those elements. For example, recent research by Pizmony-Levy et al. (2022) identified direct changes to district professional development programs as a result of the research conducted by their RPP. The MIST RPP discussed above supported improvements to teacher and school leader professional development programs (Henrick et al., 2018).

Research

A second way RPPs can impact a district’s educational improvement and transformation efforts is through providing districts with research on their specific context. Research in this framework includes systematic inquiry conducted by the partnership for the partnership to be used by the partnership. Research is a primary focus of RPPs, and collaborative research that informs action is a standard metric for assessing the health and progress of RPPs (Henrick et al., 2017). The use of research in RPPs can help inform “how educators integrate research in their decisions” (Penuel et al., 2020), and RPPs can support district capacity to effectively engage with research (Sun et al., 2019). RPP research can impact everything from pedagogical practices to policymaking to funding decisions (Farrell et al., 2021). As such, any attempt to understand the impact of an RPP on a school district must include an analysis of what research was conducted, how it was shared, and what programs, policies, and practices the research sought to influence.

Organizational structures

A third way RPPs can impact a district’s efforts to improve and transform education is through a district’s organizational structures. Organizational structures can be defined as district configurations and systems that facilitate how work flows through the school district to achieve its goals. Organizational structures are often influenced by district budgets, and RPPs can provide the budgetary resources to impact district and school organizational structures (Shand et al., 2024). For example, in the short-term, an RPP can provide external funding to add a district- or school-level position to support a particular initiative (e.g., an instructional coach), and in the longer-term, RPPs can use research to identify changes to organizational structures that make a positive difference and advocate for the sustainability of these changes. One clear example of this is the SUCCESS RPP, which provided funding to support university computer science student mentors for teachers implementing a codesigned computer science curriculum. This additional teacher support would not have been possible without the partnership (CE-STEM, 2020).

Policy

A fourth way an RPP can impact a district is through changes to district and school policy. Policy refers to the rules, regulations, procedures, and practices adopted by school district departments and school district personnel as set forth by school boards and district administrators. When considering how an RPP impacts district or school policy, it is important to consider both how RPPs impact policy development and communication before the policy is enacted, as well how RPPs support the implementation after the policy is enacted (McDonnell & Weatherford, 2016). An RPP may produce research to help inform changes to school board policies around curricula, standards, or teacher professional development; however, existing policies may also limit the ability of an RPP to deliver planned programmatic offerings. Pizmony-Levy et al. (2021) provide an example of investigating how an RPP that directly partnered with policymakers provided RPP-generated research to policymakers, who then used that research to inform changes to compensation policies for school sustainability coordinators in New York City.

Equitable results

Finally, a fifth way to examine the impact of an RPP on a school district is through examining how programs, research, organizational structures, and policies support progress toward equitable results. Equitable Results refer to the outcomes that directly address inequities facing students, including equitable access to educational resources and opportunities, equitable participation in those opportunities, and equitable learning outcomes from those opportunities. The CAPE Framework (Fletcher & Warner, 2021) provides one way to document how RPPs can influence programs, research, organizational structures and policies to increase equity in capacity, access, participation, and experience in educational opportunities. Capacity refers to the availability of resources such as teachers, funding, and policies that enable the implementation of inclusive, high-quality instruction. Access is generally defined according to whether students attend a school that offers the focal educational opportunities. Participation refers to students actually enrolling in the focal educational opportunities. Experience refers to assessing equitable learning outcomes among students participating in focal educational opportunities (Fletcher & Warner, 2021).

Recent research on RPP impacts has advanced our understanding of the role of equity within RPPs. Henrick, McGee, and Penuel (2019) explored how RPPs can support equitable relationships, outcomes, and the design of processes and systems. Farrell et al. (2021) describe how RPP goals address inequities either by conceptualizing equity as part of the process of partnering or part of the outcomes the partnership is aiming to achieve. Farrell et al. (2023) go further to address how RPPs embed equity within their missions through their focus on “1) achievement and standardization, 2) identity, culture, and belonging, and 3) power, justice, and anti-racism” (p. 207). Vetter et al. (2022) found that RPPs provided an avenue for addressing some of the inequities identified by practice partners, including student opportunity gaps, historical inequities faced by marginalized students and families, and inequities in district policies, classroom practices, and curricula. Taken together, the literature indicates that broadly speaking, RPPs are prioritizing and emphasizing equity in ways that need to be highlighted and understood more deeply.

Efforts to document RPP impact on student outcomes are critical to advance our understanding of RPPs’ effects and address gaps in the literature. While there are a growing number of studies that document RPPs contributions toward improving student outcomes (Cannata et al., 2019; Heinrich & Good, 2018; Jesson et al., 2018; Wilcox & Zuckerman, 2019), attributing success in this area to the RPP specifically is challenging given the complex nature of systems change (e.g., van Tulder et al., 2016).

As described above, partnerships’ desired impacts on equitable results vary widely and depend on the goals of the RPP. Many CS education RPPs aim to address a long history of gender and racial inequities in CS education. For example, a higher percentage of males are taking the AP CS exam than females (UC Berkeley School of Information, 2021), and representation of African American and Hispanic students among CS AP exam takers is also below their representation in the general population and especially so for exam passers (Ericson, 2019). Diversity gaps also persist in other high school measures (Johnson, 2020) and continue to be reflected in college degree production (Zweben & Bizot, 2022) and workforce representation (Muro et al., 2018; UC Berkeley School of Information, 2021).

Framework use

The five areas for analyzing and understanding RPP district impact described via the PROSPER Framework above can be a useful lens in both formative and summative examinations of RPP impact. As mentioned earlier, these five areas are not intended to be the only ways to assess RPP impact on a district, but they represent key aspects for examining how an RPP aims to add value to its district partners’ efforts. In terms of this study, we use the PROSPER Framework as the basis for a retrospective analysis to determine how the CAFÉCS RPP impacted the CPS goal for all high school students to succeed in meaningful CS experiences.

Study design

This study adds to the existing literature on the effectiveness and impact of RPPs by analyzing 11 years of research and evaluation findings through the PROSPER Framework to address the question: How did the CAFÉCS RPP impact equitable CS outcomes for CPS students?

After conceptualizing the PROSPER Framework, the first step in this analysis process was to engage in a systematic review of CAFÉCS documents. We conducted an audit of the partnership’s work between 2011 and 2022, resulting in the identification of 10 grants awarded to the CAFÉCS partnership between 2011 and 2022, and 34 dissemination products between 2015 and 2022. Of those 34 dissemination products, 29 were included in the final data set. Five products were removed because they were duplicates (e.g., a paper published in a journal that was previously presented at a conference) or they lacked relevance to understanding RPP district impacts. Additional unpublished documents were reviewed for contextual understanding, including contemporaneous meeting notes, annual reports, and evaluation reports. The full set of grants, publications, and unpublished documents were read and analyzed by four of the study’s authors using deductive coding (Creswell & Creswell, 2017) to document evidence of district impact around each of the five PROSPER Framework components in a spreadsheet. The evaluators met three times with two members of the leadership team to review the codes and come to a consensus. In addition, one evaluator shared the table in a CAFÉCS leadership team meeting to get feedback from the full leadership team. The results from the coding process were synthesized using thematic analysis (Creswell & Guetterman, 2019).

Findings

Broadly speaking, this analysis indicated that the CAFÉCS RPP impacted CPS across each of the five elements of PROSPER:

• 1. Programs: CAFÉCS impacted the selection and implementation of CS programs;

• 2. Research: CAFÉCS conducted research to understand and address problems of practice facing the OCS;

• 3. Organizational Structure: CAFÉCS funding impacted the OCS budget, resulting in changes to organizational structures in the OCS;

• 4. Policy: CAFÉCS impacted the enactment and implementation of district policy; and

• 5. the work of CAFÉCS led to Equitable Results in CS education in CPS by expanding capacity to offer CS equitably across the district, increasing equitable access to CS, increasing equitable participation in CS, and ensuring equitable outcomes related to student experiences in CS.

In what follows, we will describe these findings in more detail. We will begin by describing how CAFÉCS impacted district programs, district organizational structures, and district policies. We then describe how the partnership contributed to equitable results in the district. For each of these components, we describe how the CAFÉCS RPP impacted the district with research. More generally, our analysis indicates that CAFÉCS research studied the following: district program development and program implementation, district policy design and implementation, student assessment development and implementation, student outcomes in CS, CS teacher outcomes, and RPP partnership health.

Programs

When engaging in this analysis, we sought to understand How has the CAFÉCS RPP impacted district programs? We found that the CAFÉCS RPP impacted several district programs over the 11 years of this study. Broadly, CAFÉCS influenced district program identification and selection, program funding, program design, and implementation of programs related to CS curricula and instructional materials and CS professional development and instructional support (e.g., coaching and teacher professional learning communities, PLCs).

The CAFÉCS RPP was instrumental in identifying and selecting the program utilized in the introductory CS course, Exploring Computer Science (ECS) (Dettori et al., 2011). After influencing the adoption of ECS, CAFÉCS supported implementation through the design and facilitation of ECS workshops, summer institutes, and teacher PLCs. CAFÉCS grants funded teacher professional development stipends as well (Dettori et al., 2011; McGee & Rasmussen, 2020). CAFÉCS also supported and studied the implementation of an ECS facilitator development program to train teachers to facilitate ECS professional development (PD) sessions (McGee, Wachen, et al., 2019).

CAFÉCS research extensively studied the implementation of the ECS program, including examining outcomes for students who took the introductory course (McGee, Greenberg, et al., 2019); the impact of ECS on attitudes and interest in pursuing a CS major (McGee et al., 2016); factors that correlated with ECS course failure (McGee, Greenberg, et al., 2018); the impact of the ECS course on student CS enrollment (McGee, McGee-Tekula, et al., 2017); persistence among ECS teachers (Shub & Maaz, 2021); and the impact of the CAFÉCS supported ECS PD program (McGee, Dettori, & Rasmussen, 2022). For example, in the first four years of ECS PD implementation, the CAFÉCS supported ECS PD program led to an increase in the number of schools with a sufficient number of qualified teachers to teach ECS from five to 24 schools.

CAFÉCS developed tools and resources to support the successful implementation of the ECS program, including producing instructional videos of ECS instruction (McGee, Dettori et al., 2017), developing and validating a CS student placement exam (McGee, Dettori et al., 2017; McGee et al., 2021), and codesigning a culturally responsive program, Exploring Connections, to support teachers in helping students to make personal connections to the content in the ECS course (McGee, Dettori, et al., 2020). For example, CAFÉCS research found that student performance on the end-of-course ECS assessment was equivalent by race, ethnicity, and gender, indicating that the curriculum was effective across student demographics (McGee, Dettori, et al., 2018). In addition, CAFÉCS developed and studied a hybrid ECS model as a path for credit recovery (Johnson et al., 2021; Johnson, Chapman, et al., 2022; McGee, Dettori, et al., 2018).

Another way CAFÉCS supported the ECS program was to develop and implement an ECS coaching program. This program was designed to be responsive to teacher needs and included engaging in coaching cycles and teacher PLCs. In addition, CAFÉCS provided support to ECS coaches through a coaching PLC, coaching PD, and coaching stipends for attending coaching PD (Dettori et al., 2015; McGee, Dettori, et al., 2020). CAFÉCS-led research then examined the impact of the CS coaching model (Wachen, McGee et al., ,2021; Wachen et al., 2022).

In addition to focusing on the introductory CS course, CAFÉCS supported other programmatic needs as identified by the district office. For example, during the COVID-19 pandemic, when a need was identified related to teacher well-being, CAFÉCS provided CS teachers the opportunity to participate in a Meditation Program (Breathe for Change) (McGee & Rasmussen, 2020). As more students began taking advanced CS courses, CAFÉCS identified what aspects of the ECS program inspired students to pursue advanced CS coursework (McGee et al., 2015) and designed and implemented a teacher summer academy to develop teachers’ conceptual knowledge of CS (McGee, Dettori, Espiritu, et al., 2022). In addition, CAFÉCS supported out-of-school CS programs (McGee, Dettori, Espiritu, et al., 2022) and a CS+math/science integration program (Dettori et al., 2017).

Organizational structures

When engaging in this analysis, we sought to understand How has the CAFÉCS RPP impacted district organizational structures? As described earlier, we define organizational structures as district structures and systems that facilitate how work flows through the school district to achieve its goals. CAFÉCS impacted district organizational structures in three main ways: (1) indirectly influencing the formation of the OCS; (2) funding district personnel positions; and (3) supporting CPS to be a direct recipient of federal funding.

Forming a district office for CS, the OCS, was a key milestone for establishing the district’s commitment and prioritization of CS in CPS. The OCS was established in 2016, seven years after the CAFÉCS team began supporting the initiative to provide CS opportunities to all students in CPS, and it was born out of a need to oversee the implementation of the graduation requirement, which CAFÉCS directly supported. When the OCS was established, the director of the office was Brenda Wilkerson, an inaugural member of the CAFÉCS leadership team.

Over the years, CAFÉCS has funded full- and part-time positions in the district, including coaching positions, a grant coordinator position, and a curriculum developer position (Dettori et al., 2015, 2017; McGee, Dettori, et al., 2018, 2020). Most significantly, three different CAFÉCS grants supported ECS coaching positions, one of which fully supported a full-time district coaching position for two years. In addition, CAFÉCS provided summer stipends for ECS coaches, which provided the district with the capacity to support more teachers during that time.

Another way CAFÉCS impacted organizational structures was to increase organizational capacity to pursue and receive grant funding. With the guidance of CAFÉCS, CPS became eligible to be a National Science Foundation (NSF) awardee, as opposed to a sub-contractor under a university grant. CPS has received over $2 million in direct funding because of the CAFÉCS RPP.

Policy

When engaging in this analysis, we sought to understand How has the CAFÉCS RPP impacted district policies? The primary policy that CAFÉCS impacted was the computer science graduation requirement policy (Johnson, Wachen, et al., 2022; McGee, Dettori, & Rasmussen, 2022). The CAFÉCS-initiated activities of identifying the ECS curricula and PD program in 2011, supporting the CS4All initiative in Chicago in 2013, and establishing the ECS coaching program in 2015, helped build a solid foundation to support CS opportunities in all schools across the district. In addition, CAFÉCS research provided the CPS School Board with the information they needed to feel confident that a graduation requirement could be implemented successfully (McGee, Dettori, & Rasmussen, 2022).

A significant amount of CAFÉCS research focused on understanding the effects of the inception of the CS graduation requirement policy on student CS access and outcomes (Dettori et al., 2011, 2017; McGee et al., 2016; McGee, Dettori, et al., 2017, 2020; Wachen et al., 2022), and the impact of the graduation requirement policy on CS student access and outcomes (McGee, Dettori, & Rasmussen, 2022). This research will be discussed in more detail in the equitable results section below.

To be expected, not all CAFÉCS efforts to change policy worked out. For example, even though evaluation findings indicated that the hybrid ECS credit recovery program was effective, the lack of centralization of credit recovery processes at the district level hindered sustainability for selecting and training teachers and technology related infrastructure. These policies ultimately thwarted the successful ongoing implementation of the program after the pilot ended (Johnson et al., 2021; Johnson, Chapman, et al., 2022).

Equitable results

Perhaps the most important question this study aimed to answer was How has the CAFÉCS RPP impacted equitable results in the district? Given that the central mission of the RPP is to ensure that all students have opportunities for engaging in computing experiences, achieving equitable results for students is at the center of the CAFÉCS mission. The 2019 formative evaluation indicated that the ways in which CAFÉCS made progress toward this goal were through providing resources, expertise, and research to the OCS. This analysis adds to our understanding of how CAFÉCS worked to impact equitable results, detailing how the addition of resources, expertise, and research led to improvements to district programs, organizational structures, and policies. Our analysis indicates that CAFÉCS supported progress toward equitable results via (1) conducting research to identify and understand inequities within the district and (2) engaging in activities to improve programs, policies, and organizational structures in service of equity goals.

Research on equitable results has been ongoing since the inception of CAFÉCS, given the explicit mission of supporting computer science opportunities for all students. Of recent note, CAFÉCS conducted a study (McGee, Dettori, & Rasmussen, 2022) utilizing the four key components of the CAPE Framework (Fletcher & Warner, 2021) to document how the graduation requirement has impacted capacity, access, participation, and experience in computer science. Findings from this CAFÉCS study using the CAPE Framework indicated that the graduation requirement supported the capacity of the district to teach computer science; after four years of the graduation requirement, the number of schools with a sufficient number of qualified teachers more than tripled to 80 schools. Access to computer science increased from 41 schools offering at least one CS course to 107 schools. Excitingly, this analysis indicated that since the graduation requirement was implemented, the participation of students in a CS course reflected the demographics of the district as a whole. Moreover, student participation in additional CS courses has doubled since the enactment of the CS graduation requirement. Finally, a comparison of course performance prior to the graduation requirement and after the graduate requirement found that student experience was equivalent. Despite the large influx of students taking the class because of the graduation requirement, this study found that the student outcomes were equivalent to the students who took the class as an elective prior to the graduation requirement.

Discussion

With over a decade of experience together and more than $10 million in funding brought in to support the district, it is no surprise that CAFÉCS has impacted the district in numerous ways. While the identification and categorization of different areas of impact was helpful for annual reporting and strategic planning, we believe the PROSPER Framework’s greater value lies in providing a lens for RPPs to examine how these areas of impact can inform, interact with, and guide one another.

As the team sat down to reflect on the process of using the PROSPER Framework to understand the impact of RPPs on the district, one theme emerged related to the interconnection and strong interrelationships between programs, research, organizational structures, policies, and equitable results. Progress along each of these components was interrelated with progress along other components, elevating the idea that RPPs can perhaps be most impactful when taking a systems approach to educational improvement and transformation.

This analysis aimed to document how CAFÉCS impacted activities, efforts, and decisions within the Chicago Public Schools Office of Computer Science. In engaging in this effort, the RPP team found it useful to consider district impact by examining changes in programs, organizational structures, policies, and equitable results for students. What is not fully captured in what is shared above is the role of how research supported these changes.

We found a clear connection between CAFÉCS support for the enactment and implementation of the CS graduation requirement policy, which led to the increased capacity of CPS to offer CS. CAFÉCS research has also shown that increased access to CS has led to equitable student participation in CS, equitable course performance, and a broadening of student participation in CS pathways. In this example, CAFÉCS contributed to the enactment of a policy by engaging in research alongside engaging in activities to initiate and improve programs and organizational structures within CPS to ensure equitable results.

It is critical to acknowledge that the work it took to shift the number of students experiencing CS education in CPS was enormous and was accomplished with the help of numerous CS advocates, both within and outside of the district and CAFÉCS partnership (Johnson, Wachen, et al., 2022). This analysis takes a step forward to document the role that CAFÉCS played in this collaborative process. As RPPs engage in the challenging work of educational transformation, considering how to strategically organize efforts to make the most impact is of critical concern.

Study limitations

One important limitation of this study is that most of the documents reviewed for this analysis were co-authored by members of this analysis team, indicating the potential risk of bias. While this may be viewed as a weakness of the study, we also found it to be a strength because it provided an opportunity for the RPP team to critically examine and synthesize across many different publications and presentations produced by the partnership over time.

Conclusion

CAFÉCS supported the growth of CS in CPS by identifying and supporting the implementation of CS curricular programs, conducting collaborative research to address district needs, securing funding to support changes to district organizational structures, influencing the adoption of the CS graduation policy, all while maintaining a focus on helping the district achieve equitable results for the students. But in reality, the impact of this work is best represented through the voice of a CPS student. “I never would have taken Computer Science if not required,” said CPS sophomore Journi Easter, speaking in August 2022 at a gathering of Chicago educational leaders. “I would definitely major in this,” she went on to say. “I like that since the world is evolved around technology … I can deal with it.” Journi’s story reflects the journey of many of the 14,000 CPS students who graduated with a required CS course credit in 2020 (Easter, 2022).

Unlike some other research efforts, RPPs are intentionally designed to address the needs and goals of the partner practice organizations (Henrick et al., 2023). As such, it is important that RPPs are structuring their work to support the partner practice organization in their improvement efforts. Because of the long-standing relationships (over 13 years now) within CAFÉCS, there is clear evidence to show how the RPP provided research to help support and identify improvement strategies to address the needs of the OCS and subsequently engaged in additional research to inform its ongoing adjustments to the identified strategies. We consider this analysis to be one step toward understanding the complexities of RPP district impact and hope that this framework can be used by other RPPs to consider their impact in other district contexts.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This study was supported by funding from the National Science Foundation (NSF) (Awards #2034145, #2034150, #2031480, #2031482, #1842085, #1738572, #1738776, #1738691, #1738515, #1640215, #0837769, #0837480, #0837495, #0837636, #1542971, #1543217, #1138515, #31138417, #2122908). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of NSF.

Notes on contributors

Erin Henrick

Erin Henrick is a senior lecturer in the Department of Leadership, Policy, and Organization at Peabody College of Vanderbilt University and President of Partner to Improve. She is the co-Director of a Research-Practice Partnership between Metro Nashville Public Schools and Peabody College, as well as a RPP evaluator.

Danny Schmidt

Danny Schmidt is a senior research analyst at Partner to Improve, working primarily as an external evaluator for a diverse group of education RPPs across the country. This role affords him the opportunity to understand the perspectives of both researchers and practitioners in a variety of educational RPPs.

Steven McGee

Steven McGee is president of The Learning Partnership. He is a learning scientist that focuses on how to support learning from an individual cognitive perspective as well as from the perspective of systems and social contexts. He is a founding member and research director of the Chicago Alliance for Equity in Computer Science.

Andrew M. Rasmussen

Andrew M. Rasmussen Andy learned about the importance of computer science education in the context of completing his Ph.D. in theoretical chemistry, and since 2015 has been a part of the CS team at Chicago Public Schools (CPS) and the Chicago Alliance for Equity in Computer Science, supporting efforts connected to research, data, and policy. Since 2023 he has had an additional role supporting data systems for teacher recruitment at CPS, and also consults part-time supporting RPP and evaluation work.

Lucia Dettori

Lucia Dettori is the Interim Dean of Jarvis College of Computing and Digital Media at DePaul University, where she is also a Computer Science faculty. Her research extends from computational fluid dynamic to image processing to computer science education. She is a founding member of the Chicago Alliance for Equity in Computer Science.

Ronald I. Greenberg

Ronald Greenberg is a Professor of Computer Science at Loyola University Chicago, where he serves as the Undergraduate Program Director in the Department of Computer Science. He and his students have been engaged in many efforts to broaden participation in computing over nearly two decades. He has been collaborating with colleagues comprising the Chicago Alliance for Equity in Computer Science since the earliest days of this partnership.

Dale Reed

Dale Reed is Clinical Professor in Computer Science (CS) at the University of Illinois Chicago (UIC). He is Curriculum Lead for the Break Through Tech (BTT) initiative which has increased the percentage of women in CS at UIC from 19 to 26% in the four years 2019–2023. He is a founding member of the Chicago CSTA and is Vice-Chair of the Board for AnitaB.org.

Don Yanek

Don Yanek is a professional lecturer in Computer Science at Depaul University. He is co-chair of the Community Building and Outreach committee for the School Of Computing. Don also taught high school Computer Science and served as an instructional coach for the Chicago Public Schools. He is a member of the leadership team for the Chicago Alliance for Equity in Computer Science (CAFÉCS). In addition, Don is a founding member and former President of the Chicago Chapter of the Computer Science Teachers Association.