Exploring shifts in the characteristics of US government-funded science curriculum materials and their (unintended) consequences

Abstract

Grant-funded curriculum development efforts can substantially impact practice and research in science education. Therefore, understanding the sometimes-unintended consequences of changes in grant priorities is crucial. Using the case of two large funding agencies in the United States, the current portfolio review provides insight into these consequences by examining shifts in the characteristics of K-12 science curriculum materials funded during two time periods with differing funding priorities. Findings revealed a move away from comprehensive curricula, increased reliance on technology-based materials, a growing trend towards open access, but also a decrease in teacher supports. While these shifts may enhance teachers’ flexibility to shape curriculum, they also increase the challenge of ensuring curricular coherence. Recommendations are outlined for policymakers, science education researchers, and curriculum developers.

Keywords:

• Science education
• curriculum materials
• portfolio review
• logistic regression

Introduction

Policy makers around the world have an active and enduring interest in science education (Bybee, 2013; Hazelkorn et al., 2015; National Academies, 2007). In the United States, annual government investments for science, technology, and mathematics education are typically in the range of $2.8–$3.4 billion (Gonzalez & Kuenzi, 2014), with a substantial portion of this spending supporting the development of research-based curriculum materials intended to improve the quality of science teaching and learning (Institute of Education Sciences [IES], 2008; Singer & Tuomi, 1999). These curriculum materials provide targeted and detailed support for the enactment of specific classroom practices, and thus constitute important vehicles for reform (Brown, 2009; Carlson & Anderson, 2002; Remillard, Harris, & Agodini, 2014).

Through their programme solicitations, funding agencies can have a large influence on science curriculum materials research and development (De Lucchi & Malone, 2011; Earle, 2011). Funding priorities – and thus the specific curriculum features emphasised in grant programmes’ solicitations – shift over time due to larger policy reforms (e.g. growing importance of science), trends in science education research (e.g. growing emphasis on Nature of Science as well as Science-Technology-Society connections, changing understanding of the nature of engagement in science practices), and/or new technology developments (e.g. digital tools for data sharing and analysis). Because well-intentioned policies often have unintended consequences (Osborne, 2011), it is important to examine the consequences of changing funding priorities (Fensham, 2009). This becomes of particular importance in the context of recent reforms in science education that call for the development of integrated understanding of science content and of the nature, processes and methods of science (e.g. Australian Curriculum, Assessment & Reporting Authority [ACARA], 2017; National Research Council [NRC], 2012; NGSS Lead States, 2013; United Kingdom Department for Education, 2015). These reforms set an ambitious vision for science learning which is certain to influence both funding priorities and curriculum development efforts internationally in the coming years. Therefore, at this moment it is crucial to learn how past funding priorities have shaped the resulting science curriculum materials, as well as to understand their intended and unintended consequences. The present portfolio review addresses this timely issue around the world by examining the characteristics of K-12 science curriculum materials developed with government funding in the United States, with special attention to changes in these characteristics between two periods of time with differing funding priorities.

While the specific details of this case pertain to the United States, this study remains relevant for the many other countries in which funding for research-based curriculum development represents a vehicle for science curriculum reform. Further, it explores shifts in funding priorities which, as is often the case, stem from policy changes and international advances in the field. Thus, examination of how policy changes (grounded in international trends in science education research) are influencing details of curriculum development both directly and indirectly provides crucial considerations for policymakers around the world working to improve science education. It also provides new phenomena for researchers in curriculum studies.

In the following sections, we first discuss the importance of government funding for curriculum materials research and development. We then describe key curriculum characteristics distilled from the literature and used to guide our analysis of the curriculum materials developed by the funded projects.

Why is it Important to fund the development of curriculum materials?

Curriculum materials can be generally defined as resources designed for use by teachers in the classroom to guide their instruction, including textbooks, supplementary units or modules, and instructional media (Remillard et al., 2014). The role curriculum materials play in the classroom is context-dependent. In some classrooms, schools or countries, teachers tend to see curriculum materials primarily as ‘potential’ (Ben-Peretz, 1990). This is especially the case in settings that place high value on teachers’ own professional judgment. The German tradition of Didaktik exemplifies this, with its commitment to (a) helping learners develop and transform themselves (Bildung) as opposed to helping students acquire the ‘legacy of mankind’; (b) the fundamental distinction between content and meaning; and (c) the necessary autonomy of teaching and learning (Hopmann, 2007). In other settings, materials play a more deterministic role. In some cases, this is due to actual policy, in others it has more to do with teacher views of how to best fulfil their role. The influences of the formal and written curricula shift with pendulum swings in government policy (Nieveen & Kuiper, 2012) as well as with teacher perceptions of their own curricular literacy (McKenney, 2017).

Because these materials influence what teachers and students do on a daily basis in the classroom, policymakers frequently rely on them as a mechanism to facilitate educational reform (Ball & Cohen, 1996; Brown, 2009). In the United States, government agencies have established a number of carefully crafted grant programmes to support the development of innovative curriculum materials that enhance classroom instruction, reflect national standards, and incorporate recent advances in disciplinary content, research on teaching and learning, and instructional technologies (NSF, 2002a; IES, 2014). These government-funded programmes yield direct and indirect benefits beyond what commercial publishers provide because they emphasise innovations and experimental approaches to teaching and learning, and they attempt to validate effects of the materials on teaching and learning through research.

Research-based curriculum materials can offer direct benefits to science education. Many of these materials have been shown to result in substantially improved student outcomes, including learning gains (e.g. Cervetti, Barber, Dorph, Pearson, & Goldschmidt, 2012; Harris et al., 2015; Krajcik, McNeill, & Reiser, 2008), and improved attitudes toward science (e.g. Häussler & Hoffmann, 2002; White & Frederiksen, 1998). Well-crafted materials also support teacher learning with regard to reform intentions, subject matter content, pedagogy, or classroom orchestration (Cervetti, Kulikowich, & Bravo, 2015; Davis, Janssen, & Van Driel, 2016; Schneider, Krajcik, & Blumenfeld, 2005). Even though relatively few grant-funded development projects may endeavour to achieve implementation on a large scale, many aim to yield materials of a high quality that could be considered scale-worthy in a subsequent project, and some of these innovative materials eventually become widely used (Banilower et al., 2013).

Indirectly, science education practice is also benefitted by the theoretical understanding derived from the design and testing of grant-funded materials. Such work provides evidence and empirically-grounded theories upon which models and frameworks for learning, teaching and curriculum materials development can be based. This includes insights into student (mis)conceptions about complex science concepts that are important but not traditionally addressed (e.g. Duncan, Rogat, & Yarden, 2009), and instructional approaches that develop students’ cognitive and metacognitive knowledge and skills (e.g. White & Frederiksen, 1998). The design and testing of curriculum materials also yields understanding of teachers’ orientations, pedagogical content knowledge and instructional strategies, as well as how these may change when engaging with reform-based curriculum materials (Cervetti et al., 2015; Leary et al., 2016; Marco-Bujosa, McNeill, González-Howard, & Loper, 2017). Such work also provides opportunities for learning about the facilitation of curriculum reform in varied settings, by investigating how specific features of materials and related activities influence the uptake, sustained use and spread of new pedagogical ideas (e.g. Clarke & Dede, 2009). Moreover, research on the curriculum development process itself can yield insights into effective approaches that may serve as models for curriculum development efforts by others (e.g. Barber, 2015; Davis et al., 2014; Krajcik et al., 2008). In sum, through the materials themselves (directly) and knowledge distilled from its design and testing (indirectly), grant-funded development efforts can have a large impact on science education practice and research.

Purpose of the study

As described above, grant-funded curriculum development efforts can substantially impact practice and research in science education. The characteristics of these materials may be influenced by shifts in funding priorities in response to larger policy changes and advances in the field. Understanding the sometimes unintended consequences of these shifts in emphasis is therefore of paramount importance to inform future funding policies as well as curriculum research and development initiatives. To offer insights into these consequences, the present portfolio review seeks to (1) identify the characteristics of K-12 science curriculum materials developed with government funding in the United States, and (2) examine major shifts in these characteristics between two periods of time with differing funding priorities.

The two major government agencies supporting the development of curriculum materials for K-12 science education in the United States are the National Science Foundation (NSF) and the Institute of Educational Sciences (IES) at the U.S. Department of Education (Feder, Ferrini-Mundy, & Heller-Zeisler, 2011). Both NSF and IES funding programmes experienced major reorganisations around 2005. These reorganisations were followed by shifts in the priorities of grant programmes’ request for proposals, including an increased emphasis on integrating research alongside development (IES, 2008; NSF, 2006; NSF, 2008a), growing support for the development of instructional resources (e.g. replacement or supplementary units, technology tools) instead of comprehensive curricula (NSF, 2006), and recommendations for making materials publicly available (Ainsworth et al., 2005; Borgman et al., 2008). Because these changes to the grant programmes’ requests for proposals would likely influence the characteristics of the resulting curriculum materials, the present study focused on two specific time periods: 2001–2005 (before programme reorganisations) and 2006–2010 (after programme reorganisations). We note that period of investigation pre-dates the release of the broadly influential Framework for K-12 Science Education (NRC, 2012), and only partially overlap with the Common Core State Standards in Mathematics and English Language Arts in the US (http://www.corestandards.org), which may have also influenced science curriculum design in science given the disproportionate weighting of performance in mathematics and English Language Arts in the accountability systems in place at the time.

Theoretical model

Funding the development of research-based curriculum materials contributes to building a knowledge base that supports curriculum reform. In addition to fundamental understanding about learners, teachers and settings in which reform is enacted, research shows that the characteristics of curriculum materials themselves play an important role in affording and constraining students’ and teachers’ opportunities to learn and teach (Brown, 2009; Cohen & Ball, 1999; Remillard, 2005).

Various frameworks have been developed to examine the content and quality of curriculum materials, including the framework used for the review of instructional materials for middle school science conducted by the National Science Foundation (NSF, 1997), the TIMSS curriculum and textbook analysis framework (Schmidt, McKnight, & Raizen, 2002), and the curriculum analysis procedure developed by Project 2061 (Kesidou & Roseman, 2002). However, these frameworks were mainly designed for evaluation of comprehensive curricula. By contrast, grant-funded science curriculum materials include much more than just comprehensive curricula, often focusing on supplementary or replacement units of short duration (e.g. a six-week curriculum unit), but also including ancillary materials to be inserted into a curriculum (e.g. web-based modelling tools). This diversity of grant-funded materials requires a broader framework.

Figure 1 presents the theoretical model underpinning this study. In it, the dashed box depicts an analytical framework for investigating the characteristics of curriculum materials, which includes four aspects that can influence the uptake, use, and instructional outcomes of these materials: format, scope, supports for student learning and supports for teachers. Some features of the awards given could influence the characteristics shown in the analytical framework. The total amount awarded, the type of organisation leading, and the intended use of the funding (research and/or development) stand to influence the design process and through that, the designed materials. Award features influencing design targets, and subsequently the designed materials, include the grade level, student subgroups and science discipline to be reached. We point out that the theoretical model underpinning this study pertains only to characteristics of curriculum materials that are shaped by grant-funding schemes. It is not meant to offer a comprehensive portrayal of all factors that could influence the characteristics of curriculum materials. Each aspect of the analytical framework is elaborated below.

Figure 1. Theoretical model underpinning this study.

Format

Curriculum materials take diverse formats that influence the ways in which content is presented as well as teachers’ decisions about its uptake and use (Grossman & Thompson, 2008; Remillard, 2012). Here, we distinguish four dimensions of format and explain how they could influence use at scale: delivery format, core learning activity format, technology requirements, and dissemination format. In terms of delivery format, curriculum materials for science education have traditionally adopted the form of textbooks, worksheets and/or notebooks that could be accompanied by a kit providing physical resources for science investigations (Davis et al., 2014). Advances in educational technology generated ample new opportunities to present and engage students with scientific phenomena, impacting the ways in which science is taught and transforming the field of curriculum design (Krajcik & Mun, 2014; Linn, Gerard, Matuk, & McElhaney, 2016). For example, computer simulations provide many opportunities to enhance science teaching and learning by re-creating aspects of the real world that would otherwise be too complex, time-consuming, or dangerous to do in a conventional classroom setting (Smetana & Bell, 2012). They also provide opportunities to make learning of abstract concepts more concrete by visualising scientific phenomena that cannot be readily explored using hands-on activities (McElhaney, Chang, Chiu, & Linn, 2015) and by enabling students to easily modify rules and variables to test hypotheses (Blake & Scanlon, 2007).

Especially for science learning, format of core learning activities bears mention. Research has shown that computer simulations can be equally effective as, and sometimes even more effective than, traditional instructional practices in promoting science content knowledge and developing process skills (Smetana & Bell, 2012; Zhang & Quintana, 2012). However, there is an enduring debate in the literature about whether hands-on experimentation should be completely replaced by technology tools (cf. Klahr, Triona, & Williams, 2007; Rutten, van Joolingen, & van der Veen, 2012). Given these trade-offs, educators will make different decisions about if and how to use curriculum materials that involve different combinations of hands-on, hands-off and/or computer-based learning activities. Here, we use the term hands-on activities to refer to specific instructional strategies where students are actively engaged in manipulating physical objects, such as ramps, test tubes or mechanical devices (Klahr et al., 2007). Hands-off activities, on the other hand, is used here to refer to situations in which students engage with the content through lectures, discussions or paper-based materials, such as a textbook or worksheets. Finally, computer-based activities are those in which students interact with the content through the use of computer software such as simulations or educational games.

The technology requirements of the curriculum materials and the core learning activities supported will, in turn, determine the extent to which access to computers and Internet is required. Such features are also important because they can influence teachers’ choices about when and how to use the materials. Although some schools may be better equipped now, an important distinction influencing educational technology use in schools is if computers are required or if, in addition to this, reliable access to Internet is required.

Finally, the ways in which curriculum materials are disseminated (e.g. commercially, publicly, limited to request) can also have an influence on their uptake and use. While curriculum materials that are publicly available might be easier to spread, commercial materials usually provide access to additional supports (e.g. professional development workshops, implementation support, materials kits) that may facilitate sustainable implementation and adoption.

Scope

A second important characteristic of curriculum materials pertains to scope. Scope is defined as the breadth of content covered by the curriculum materials in a given subject area (Grossman & Thompson, 2008; NRC, 2012). Government-funded curriculum materials vary in scope, ranging from supplementary activities or tools targeted towards specific content that teachers can flexibly integrate into their instruction, to comprehensive curriculum materials covering one or multiple years of instruction (Carlson & Anderson, 2002). Variations in scope can be considered on a continuum, in this study we distinguish four: comprehensive curriculum, curriculum sequence, supplementary curriculum units/modules, or supplementary activities/tools.

The scope of curriculum materials can have an impact on teachers’ and students’ learning as well as on the materials’ use and uptake. On the one hand, curriculum materials that are less comprehensive in scope (e.g. stand-alone supplementary curriculum units or activities targeted towards specific content) have the benefit of originality and flexibility: they provide teachers with the possibility to ‘mix and match’ different curriculum materials as they see fit (NRC, 1999), and thus to better align them with the specific needs of their students and classroom contexts. This greater flexibility can, in turn, facilitate the materials’ implementation and uptake (Fishman & Krajcik, 2003). On the other hand, curriculum materials that are more comprehensive in nature may help students develop increasingly sophisticated ideas by ensuring curriculum coherence – that is, the adequate coordination of content and scientific practices across one or several years of instruction (Fortus & Krajcik, 2012; Roseman, Linn, & Koppal, 2008). Moreover, comprehensive materials may provide explicit pedagogical guidance and supports for teachers, thereby also generating opportunities for teacher learning (Grossman & Thompson, 2008).

Supports for student learning

A third important feature of curriculum materials is the integration of supports for student learning. Supports generally enable learners to accomplish tasks that otherwise might be out of their reach (Reiser, 2004). Some supports are gradually withdrawn over time so as to allow students greater responsibility over their own learning (McNeill, Lizotte, Krajcik, & Marx, 2006). Yet, there are numerous types of cognitive supports that do not fade over time but instead provide continuous support that can be instrumental for successfully completing a task (Pea, 2004). Moreover, supports embedded in the curriculum materials can work collectively with teachers, peers and technology resources to address a specific learning need. Tabak (2004) refers to these as ‘synergistic supports’ given that the productive interactions between them may augment each other to produce a robust form of support.

Here we distinguish two kinds of supports for student learning typically embedded in curriculum materials. Cognitive supports (which may be permanent or fade over time) can assist learners by highlighting key ideas or relationships among concepts (e.g. outlines, concept maps), by providing additional information for completing a complex task (e.g. worked examples, hints), or by giving them opportunities to assess what they know and what they do as they learn (e.g. reflection prompts, automated feedback). Accommodation supports provide specific aids for English language learners (e.g. access to content in both native and second language, audio support), learners with different ability levels (e.g. differentiated tasks), or learners with special needs (e.g. graphic organisers, adjustable text size). Research has shown that accommodation supports (Clark, Touchman, Martinez-Garza, Ramirez-Marin, & Drews, 2012; Knight, Spooner, Browder, Smith, & Wood, 2013) as well as continuous cognitive supports (Lee & Songer, 2004; White & Frederiksen, 2000) and scaffolds (Belland, Walker, Olsen, & Leary, 2015; McNeill et al., 2006) can have positive impacts on students’ cognitive and metacognitive outcomes.

Supports for teachers

A fourth key characteristic of curriculum materials pertains to the integration of supports for teachers. Next to workshops, summer institutes and school-based professional development, the implementation of curriculum materials can also be facilitated by written supports embedded in curriculum materials. Here we distinguish two types of written supports for teachers. Procedural supports (e.g. number of class periods required, list of materials and equipment, safety guidelines) render the materials practical for everyday use (Janssen, Westbroek, Doyle, & Van Driel, 2013). Educative curriculum supports, by contrast, focus specifically on providing opportunities for teacher learning (Davis & Krajcik, 2005; Grossman & Thompson, 2008). These educative supports (1) help teachers anticipate student thinking and misconceptions, (2) support teachers’ learning of the subject matter, (3) help teachers consider ways to relate science concepts and practices across units during the year, (4) make visible the rationale behind particular design decisions, and (5) promote teachers’ capacity to implement and adapt the curriculum materials (Davis & Krajcik, 2005). Various studies suggest that educative curriculum supports can have a positive impact on teachers’ learning (e.g. Beyer & Davis, 2009; Drake, Land, & Tyminski, 2014; Marco-Bujosa et al., 2017) and instructional practices (e.g. Arias, Davis, Marino, Kademian, & Palincsar, 2016; Cervetti et al., 2015), thereby also influencing student learning (Bismack, Arias, Davis, & Palincsar, 2015).

Methodology and methods

Case context: US government funding for science education

Government science education funding is largely concentrated at the NSF and the U.S. Department of Education (Feder et al., 2011). Both agencies have historically played key roles in K-12 science education through, among other mechanisms, the funding of various programmes intended to support the development of high-quality curriculum materials.

The National Science Foundation

NSF is an independent government agency created by the U.S. Congress in 1950. Since its early years, the Foundation has supported the development of curriculum materials for science education (NRC, 2007; NSF, 2014). First initiatives date to 1960, when NSF funded various curriculum development projects aimed at bringing scientists together to improve high school curricula (Earle, 2011).

Around the 1990s, the NSF established the Instructional Materials Development (IMD) programme, which produced a number of broadly used elementary science curricula. As time went on, projects funded by this programme focused on the development of comprehensive curricula and supplementary instructional materials that enhance classroom instruction, reflect national standards, and incorporate recent advances in disciplinary content, research on teaching and learning, and instructional technologies (NSF, 2002a). Although IMD also supported applied research projects intended to increase understanding of how teachers, materials, and assessments facilitate student learning, the emphasis was mainly on curriculum development.

While not the primary focus of the programme, some development of classroom materials was also explicit in the early programme solicitations of the Information Technology Experiences for Students and Teachers (ITEST) programme, which sought to expand opportunities to learn about, experience, and use information technologies within the context of informal and formal science, technology, and engineering education (NSF, 2002b). Interestingly, over the years both IMD and ITEST programme solicitations increased their focus on research as a major component next to development. Since 2003, IMD called out for project proposals to develop evidence of impact of the funded materials (NSF, 2003), and encouraged projects to provide research questions and methodologies as part of their proposal to develop materials (NSF, 2004). Similarly, ITEST evolved into a programme that incorporated a separate research strand intended to contribute to the knowledge base about approaches that are most likely to increase science, technology and engineering capacity of the future workforce (NSF, 2008a).

In 2006, the Directorate for Education and Human Resources – which holds the primary responsibility for NSF’s education mission – experienced a substantial reorganisation. The goal of this reorganisation was to increase coordination and coherence across the various education programmes (NSF, 2006). This resulted both in the creation of the Division for Research on Learning in Formal and Informal Settings and in revisions to the Directorate’s programmes. The latter included the adoption of a cycle of innovation and learning as a conceptual framework guiding the coordination of the new division’s programmes. The cycle, adapted from the RAND mathematics Study Panel (RAND, 2003), consists of five key components: (1) developing and testing of new theories and knowledge about teaching and learning; (2) designing and developing instructional materials, measurement tools and methods; (3) implementing innovations and documenting their impact; (4) evaluating the effectiveness of innovations; and 5) synthesising results as well as identifying new insights and questions to inform further research (NSF, 2008b). The complementary nature of the programmes organised around this cycle supports a varied progression of projects that contribute to both knowledge building and practice improvement.

While all programmes in the Division for Research on Learning in Formal and Informal settings are concerned with the five components of the cycle to different degrees, starting in 2006, two programmes stood out for their focus on the design and development components within formal education settings: Discovery Research K-12 (DRK-12) and Innovative Technology Experiences for Students and Teachers (a revised version of prior ITEST solicitations). The DRK-12 programme was created to enable significant advances in K-12 student and teacher learning of the science and mathematics disciplines through research about, and development and implementation of, innovative resources, models, and technologies for use by students, teachers, and policy makers. Overall, DRK-12 projects are intended to advance our knowledge of effective instruction and curriculum design (NSF, 2008b). The revised ITEST programme, on the other hand, specifically examines issues of science learning and motivation for workforce development. The modifications to this programme illustrate the restructuring efforts to meet the demand for a qualified workforce through the development, implementation and study of strategies that encourage K-12 students’ interest in science and engineering, and through research addressing technological workforce issues (NSF, 2008a).

Altogether, from early programme solicitations to more recent ones, NSF has actively supported the development of high-quality and standards-based science curriculum materials. Although general education research programmes funded by NSF could also have a design component, specific support for the design and development of curriculum materials for K-12 science education has been concentrated on a relatively small number of programmes. The priorities established by these programmes have changed over time, with an increasing emphasis on integrating research alongside development.

U.S. Department of Education

The primary research and evaluation arm of the U.S. Department of Education is the IES. Authorised by the Education Sciences Reform Act of 2002, the mission of IES is to provide rigorous evidence on which to ground education policy and practice (IES, 2007). One of the Institute’s major priorities is to support research that contributes to improved academic achievement for all students, and particularly for those whose education opportunities might be hindered because of their socioeconomic status, race/ethnicity, gender, disability, English proficiency, and/or family circumstances (IES, 2007).

With its emphasis on academic achievement, IES established research programmes focused on core academic outcomes, including the Mathematics and Science Education Research programme (in 2003) and its companion programme under Special Education (in 2005). These programmes have supported research on the exploration, development, measurement, and evaluation of curricula and instructional approaches that are intended to improve mathematics and science proficiency from kindergarten through high school (IES, 2009). While research on science interventions is also supported through some of the Institute’s other research programmes (e.g. Education Technology, Cognition and Student Learning), the creation of the Mathematics and Science programmes illustrate growing emphasis on science and mathematics education.

In 2004, IES established five research goals across its programmes: (1) Exploration, (2) Development and Innovation, (3) Efficacy and Replication, (4) Scale-up Evaluation (now called Effectiveness Evaluation), and (5) Measurement. Similar to the innovation and learning cycle adopted by NSF in 2006, the goal structure adopted by IES was designed to span the range from basic research with practical implications to applied research (IES, 2014). The development of innovative curricula and instructional approaches intended to produce beneficial impacts on student academic outcomes is primarily supported by the Development and Innovation goal. A major objective of projects funded under this research goal is to develop robust educational interventions through an iterative cycle of development, implementation, observation and revision (IES, 2009).

Development and Innovation projects are intended to build on prior theoretical and empirical work to propose a theory of change that specifies the underlying process through which key components of an intervention are expected to lead to improved student outcomes (IES, 2014). Awarded projects are also requested to provide evidence about the promise of the intervention for achieving its intended outcomes and the feasibility of implementing it in authentic educational settings. Evidence resulting from these evaluation studies can later be used in support of a subsequent application for an Efficacy and Replication grant, in which researchers examine if and to what extent the developed interventions ultimately produce beneficial impacts on student outcomes (IES, 2009).

Historically science education has not received as much attention from the Department of Education as other disciplines have, in part because IES has a much broader mandate than just mathematics and science. However, general support for the development of innovative curriculum materials grew over the years, and was consolidated with the establishment of the Development and Innovation research goal. In fact, since the Institute established the goal structure for the education research grants, 48% of all IES education research grants focused on the development of innovative curricula, with approximately half of all grants under the Mathematics and Science programmes being awarded under this goal (IES, 2011).

Approach

To identify the characteristics of government-funded K-12 science curriculum materials and examine their changes between 2001–2005 and 2006–2010, a review of NSF and IES funding portfolios was undertaken. The search was limited to projects awarded between 2001 and 2010 for two reasons: (1) to examine the consequences of shifts in NSF and IES funding priorities taking place within this time window, and (2) to ensure access to project documentation (i.e., for projects awarded before 2001 it could be difficult to obtain information about the materials’ features, and projects awarded after 2010 may have lacked the time needed to disseminate results). Below we describe the five steps in our procedure for identifying and selecting particular awards to be included in our analyses. These steps were part of an inductive-deductive analysis (Fereday & Muir-Cochrane, 2006; Patton, 2002; Thomas, 2006) that was influenced by the original focus on curriculum materials for K-12 science (steps 1 & 2) and the kinds of curriculum materials that were regularly funded (step 3). We adhered to systematic review procedures (Petticrew & Roberts, 2008) for the five steps described below as well as the document analysis phase, described thereafter.

Identification of relevant awards and inclusion criteria (Steps 1 and 2)

The overall identification procedure involved two sequential steps. In step 1, separate broadly inclusive searches for NSF and IES awards were conducted. The search for NSF awards was focused on funding programmes that are known for supporting development work and included: IMD, Interagency Education Research Initiative (IERI), Research on Learning and Education (ROLE), Innovative Technology Experiences for Students and Teachers (ITEST), and Discovery Research K-12 (DRK-12). Projects funded by Small Business Innovation Research Initiatives (SBIR) were also included but only when explicitly linked to one of the previously listed programmes. Awards were sought on the official NSF project databases available at www.research.gov and www.nsf.gov, using the name of the programme as a general keyword. For each programme, the results yielded from both databases were merged and duplicates eliminated. The NSF search resulted in 1,144 awards (see Table 1 for an overview).

Table 1. Inclusion and exclusion criteria.

Inclusion criteria	Exclusion criteria
1. The project takes place within the context of K-12 education and in- school activities	Projects concerned with out-of-school/after school programmes, and/or informal learning
	Projects targeted exclusively at pre-school, undergraduate and/or graduate students
2. The project focuses on science education, including combinations of science with other disciplines such as mathematics, engineering or literacy	Projects focused exclusively on K-12 disciplines other than science (e.g. mathematics, engineering)
	Projects where science is a secondary focus (i.e., focus is on other disciplines but with some connections to science)
3. The project explicitly describes the design of curriculum materials (broadly defined, including curriculum frameworks, learning progressions, materials for classroom use) as an important goal	Projects exclusively concerned with teacher professional development, or where curriculum design was primarily a means for teacher professional development
	Projects exclusively concerned with research (e.g. research synthesis, efficacy studies) or where curriculum design was primarily intended to serve research purposes only and not for actual classroom use (e.g. design as treatment)
	Projects concerned with the design of large scale (national or international) benchmark assessments
	Projects concerned with the design of educational policies, recommendations, etc.

Awards by IES were sought on the official database available at http://ies.ed.gov/. In this case, the search focused on research goals that are likely to be associated with curriculum development (i.e., Development and Innovation, Efficacy and Replication, and Scale-up Evaluation) instead of on specific programmes. The primary research goal in combination with ‘science’ was used as a general search term, which then excluded SBIR awards because they were not associated with research goals in the database. The IES search yielded 157 awards.

In a second step, project abstracts were screened to identify awards concerned with curriculum design for K-12 science education. This step was purposefully broad to understand the range of materials funded under these programmes (e.g. curriculum frameworks, learning progressions, materials for classroom use). An award was regarded as relevant once it targeted K-12 science education and had curriculum design as an important goal (see Table 2). Given that our focus was on formal schooling, projects concerned with out-of-school programmes as well as those targeted exclusively at pre-school or (under)graduate students were not included in the review. Similarly, projects were excluded when curriculum design was itself not an important goal, but rather just a means for teacher professional development or testing of new theories. To establish reliability in screening, a subset of 45 abstracts was screened by two independent researchers and differences in judgments discussed until acceptable levels of reliability were obtained (Kappa = .7). If this award selection process had been limited to the often-partial information provided in the abstract, some relevant awards could have been excluded (i.e., false negatives). To prevent this problem, additional information was sought from project websites when abstract information for one or more inclusion criteria was ambiguous, and the project’s relevance was discussed with the entire research team. This filtering step produced 226 awards concerned with curriculum development in science.

Table 2. Overview of search and selection process by funding organisation.

Step	Resulting Set	National Science Foundation (NSF)	Institute of Education Sciences (IES)	Subsequent NFS & IES awards*	Total
1	Awards between 2001–2010 from programmes that fund development (NSF) or research goals relevant to curriculum development (IES)	1144	157	–	1301
2	Awards concerned with K-12 science curriculum materials design (meeting inclusion criteria)	200	26	–	226
3	Awards specifically concerned with development of curriculum materials for classroom use	143	19	–	162
4	Projects concerned with curriculum materials for classroom use (after merging related awards)	129	15	3	147
5	Random sample of projects concerned with curriculum materials for classroom use	77	6	1	84

^* Projects that received subsequent awards from NSF and IES within our 10-year time window.

Sample selection (Steps 3 to 5)

The sample selection procedure involved three additional steps (see Table 1 for an overview). For each of the 226 awards meeting the inclusion criteria, project abstracts and websites were consulted to gather descriptive information about: (1) the type of materials developed (e.g. curriculum framework, learning progression, curriculum materials for classroom use), (2) the target grade level (e.g. elementary, middle, high school), and (3) the science discipline (e.g. physical sciences, life sciences, earth & space sciences). This information was used, on the one hand, to verify the relevance of the projects included in the review (i.e., exclude false positives) and, on the other hand, to inform sample selection.

Given that 72% of the funded K-12 science curriculum design projects were concerned with curriculum materials for classroom use, we decided to focus on this particular subset (N = 162). Next, different awards linked to the same project were merged (step 4). This procedure further reduced the sample to 147 cases. Finally, to make the process of in-depth analysis of project characteristics manageable and ensure efficient use of principal investigators’ time for member-check interviews (see next section), step 5 involved a random selection of projects based on principal investigators (N = 84). When a principal investigator received multiple awards within our 10-year time window, all the awards were included. This was the case for less than 10% of the selected sample (N = 6), and therefore it is unlikely that this sampling approach had an effect on the observed shifts. In fact, chi-square tests revealed no significant differences (using an alpha level of p < .05) between projects included in the random sample (N = 84) and those excluded (N = 63) with regard to metadata collected on all projects (i.e., award start year period, target grade level, and science discipline). Table 2 summarises the five-step process.

Data collection

To characterise the projects and resulting curriculum materials, publicly available samples of the curriculum materials as well as project documentation (e.g. project proposals, evaluation reports, journal articles) were reviewed. We were able to gather samples of curriculum materials for 83% of the projects (N = 70). For the remaining projects (N = 14), we relied on the descriptions available on academic publications, project websites, and/or on the abstract of the project proposal. Information from all available sources was equally weighted and triangulated to ensure accuracy of the coding (described next). If inconsistencies across sources were found, these were always discussed with the principal investigators in a member-check interview.

Document analysis

An instrument was designed to characterise the materials developed by the selected projects. The instrument development also followed an inductive-deductive analysis approach and was informed by: (1) existing literature about key characteristics of (science) curriculum materials (as described in the analytical framework); and (2) inductive analysis of project documentation for identification of common variations in curriculum materials design, thereby allowing unanticipated trends to emerge. This resulted in a draft coding scheme that was extensively discussed with a group of expert curriculum developers to verify the extent to which the codes were both relevant and broadly representative of key characteristics across diverse types of curriculum materials. Refinements were made based on their input.

The final version of the instrument consisted of project metadata as well as sections related to the four sets of key curriculum materials’ characteristics described in the analytical framework (see Figure 1). Project metadata included: award features (award start and expiration year, total amount awarded), research and/or development focus, leading organisation type (led by a development/outreach organisation alone vs. led by or in collaboration with a university), target grade level(s), target student subgroups, and science discipline(s). The characterisation of the curriculum materials’ format included delivery format, dissemination format, core learning activities, and technology requirements (see Table 3). Scope examined the breadth of content covered by the curriculum materials (see Table 4). Supports for student learning focused on cognitive supports and accommodation supports embedded in the student materials (see Table 5). Finally, supports for teaching examined procedural and educative supports available in the teacher materials, as well as continued access to professional development activities after the end of the award (see Table 6), which may be critical for sustaining, deepening, and broadening use of the curriculum materials.

Table 3. Coding scheme for format of curriculum materials.

Code	Description	Examples
Delivery format^a (no; yes)
(Text)book	Printed book designed to support the study of a specific subject	• Textbook • Story book(s)
Other paper-based materials	Paper-based materials designed to support student learning of a particular topic/subject	• Worksheets • Handouts • Investigation notebooks
Digital resources	Educational resources available on the World Wide Web	• Websites • Web-based learning environments • Online databases
Simulation software	A programme that allows the user to observe an operation through simulation without actually performing that operation	• Simulations • Animations • Modelling tools
Intelligent tutors	Computer system that aims to provide immediate and customised instruction or feedback to learners, usually without intervention from a human teacher	• Intelligent virtual assistants that engage students in meaning-making exchanges in which students interactively devise explanations and make predictions.
Educational games	Games explicitly designed with educational purposes	• Adventure games • Virtual reality games
Core learning activities (required; optional)
Hands-off learning activities	Students engage with the content through lectures, discussions and/or paper-based materials such as a textbook or worksheets	• Students read four science books, including Mystery Forces, which challenges them to figure out which forces are at work in a variety of scenarios.
Hands-on learning activities	Students are actively engaged in manipulating physical objects, such as ramps, test-tubes or mechanical devices	• Using metric rulers, magnifying glasses, microscopes, and the identification guides, students investigate the invertebrates they collected in their schoolyard.
Computer-based learning activities	Students engage with the content through the use of educational software such as simulations or educational games	• Students learn Darwin’s model of natural selection using computer-based models depicting interacting organisms and their environments.
Technology requirements (not always required; always required)
Access to computers	Extent to which access to computers is required to implement the curriculum materials	• Simulations • Modelling tools
Access to Internet	Extent to which access to internet is required to implement the curriculum materials	• Online educational game • Online data bases
Dissemination format (no; yes)
Commercial access	Curriculum materials distributed by a commercial publisher after end of award	• Materials available for purchase from [publisher]
Public access	Curriculum materials are available for free download/use after end of award	• Materials available for free download from [university project website] • Materials available for free use from [non-profit research and development organisation website]
Access limited to request	Curriculum materials are not commercially or publicly available after end of award, but can be accessible upon request.	• Principal investigator states that access is provided upon e-mail request.

^a Curriculum materials could be delivered in multiple formats. When this was the case, the main delivery formats were coded as present.

Table 4. Coding scheme for scope.

Code	Description	Examples
Comprehensive curriculum (no; yes)	Series of curriculum units that cover the whole range of content/key understandings for a specific grade level(s) and science discipline(s)	• Three-year curriculum that introduces students to all of the core concepts in inquiry, physical sciences, life sciences, and earth-space sciences found in the national standards for grades 9–12
Curriculum sequences (no; yes)	Short series of curriculum units designed with the intention to be used together and in a particular sequence, since each new unit builds on the understandings developed in the previous one	• A sequence of four consecutive curriculum units across grade six and grade seven aimed to improve students’ cumulative and integrated understanding of energy concepts
Stand-alone supplementary/replacement units (no; yes)	Lessons or instructional activities structured around a specific topic or common theme. Even though multiple units may be developed, these do not need to follow a particular sequence or build on each other	• A 6-week curriculum unit focused on key ideas in chemistry and their application to living systems, in order to prepare middle school students for high school biology
Stand-alone supplementary/replacement activity/tool (no; yes)	Ancillary resources and/or strategies teachers can use in combination with other curriculum materials to facilitate student learning	• Lesson starters • Laboratory activities • Simulations

Table 5. Coding scheme for supports for student learning.

Code	Description	Examples
Cognitive supports (no; yes)	Supports intended to assist learners by highlighting key ideas or relationships among concepts, providing additional information for completing a task, and/or giving opportunities to assess what they know.	• Concept maps/outlines • Worked examples • Hints • Reflection prompts • Automated feedback
Accommodation supports (no; yes)	Specific supports for English language learners, learners with different ability levels and/or learners with special needs	• Translations to other languages • Text-to-speech • Graphic organisers • Adjustable text size • Differentiated tasks for students with different ability levels

Table 6. Coding scheme for supports for teaching.

Code	Description	Examples
Procedural supports (no; yes)
Organisational information	Information about time and resources required to effectively implement the curriculum materials	• Number of class periods required • List of materials and equipment • Safety guidelines • Advanced preparation
Instructional strategies	Specification of techniques or tips teachers can use to facilitate student learning	• Pedagogical suggestions to productively support student learning • Tips for leading whole-class discussions • Recommendations for how to integrate the activities/tools in the curriculum
Access to variations created by others	Examples of how other teachers have adapted the curriculum materials to their specific classroom situations (beyond exchanges during PD workshops) and/or access to a platform where teachers can share their adapted lesson/activity plans	• Lesson plans, worksheets and/or activities created by teachers available for download from project website • Platform where teachers can share their modified lesson plans, worksheets and/or experiences with others
Educative Supports (no; yes)
Unit goals	Brief statements of what science concepts or practices students will be able to learn, which help teachers understand the purposes of the units and concrete instantiations of those concepts and practices	• Overview of learning goals for each curricular unit, lesson or activity
Alignment to standards	Explicit references to the standards addressed in the curriculum materials, which help teachers see concrete instantiations of relatively abstract standards statements	• List of National and/or State content standards addressed in the curriculum materials
Information about students’ ideas	Information that could help teachers to anticipate student thinking and misconceptions	• Overview of pre-requisite knowledge (i.e., what students should already know and/or be able to do) • Overview of common misconceptions (i.e., misunderstandings and/or or difficulties students have shown to have with a particular content/activity in the materials)
Background information	Information that could assist teachers’ learning of the subject matter	• Description of key science concepts addressed in the curriculum materials • Links to websites and/or resources with additional information about the key science ideas addressed in the curriculum materials
Continued access to professional development activities (no; yes)
Access to professional development after end of award	Professional development activities remain available once direct project support ends	• Workshops organised by the publisher • Webinars • Online courses

The review of sample curriculum materials and project documentation was conducted by the first author, and resulted in individual project profiles that were then verified with the principal investigators in a (telephone) interview as a way to (1) verify the validity of the coding scheme (i.e., extent to which the codes adequately represented the project at hand), (2) check the reliability of the coding (i.e., extent to which the coding was accurate), and (3) collect information for missing fields. Member-check interviews were completed for 83% of the projects in the selected sample; a few principal investigators had retired or did not return e-mails. During the interviews, principal investigators were asked to verify if the characterisation of their project for each key feature was accurate and complete. To facilitate recall, principal investigators were provided with short descriptions of their project(s) a few days prior to the interview. These descriptions included references and/or links to the sources used to code each key feature in our instrument. During the interviews, which lasted on average 60 min, the coding of each feature was discussed until principal investigators were satisfied with its characterisation. Errors in the coding were rare, and conversations focused mostly on adding information for missing fields. When principal investigators were unable to recall certain features of the curriculum materials or we had no access to documentation describing them, these were coded as missing values.

Statistical analysis

Descriptive statistics were used to characterise projects with regard to the total amount awarded, grant focus, target grade level, science discipline(s), and scope of the curriculum materials developed. The overall time window was then divided in two time periods (i.e., 2001–2005 and 2006–2010) to examine major shifts in the characteristics of the curriculum materials. To determine the probability that the observed changes in curriculum characteristics might have happened by mere chance or not, Pearson’s chi-square analyses were conducted. When one or more expected frequencies were less than 5, the more conservative Fisher’s exact test was used to determine significance (Agresti, 2007). To ascertain the magnitude of the effect, Cramer’s V was computed. Cramer’s V ranges from 0 (no relationship) to 1 (strong relationship), with values above .3 generally considered as medium effect and above .5 as strong effect (Cohen, 1988).

To further examine possible interactions between the observed shifts in curriculum characteristics, follow-up logistic regression analyses were conducted. Given the large space of possible interactions, our analyses focused on the most plausible connections for each of the observed shifts (see results for details), while leaving out the less plausible ones in order to prevent by-chance relationships (due to conducting large numbers of tests) or very poor statistical power (due to having to correct for many parallel tests).

Results

Characteristics of US government-funded K-12 science curriculum materials

We begin with a characterisation of funding investments and topics to generally describe the context of investigation and enable others to make international comparisons. Overall, the selected projects from IES and NSF represent a cumulative investment of $156.7 million, with each single project being granted on average $1.86 million (ranging between $.3 M and $8.9 M). Thirteen of these projects (15%) obtained multiple awards within our 10-year time window. The focus on research and development activities varied across programmes, although IES and most NSF programmes funded projects that involved both research and development.

Curriculum materials exclusively targeting high school (40%) or middle school (26%) science constituted over half of the cases, whereas only 12% of the projects targeted elementary school, perhaps mirroring the relatively low implementation rate of science in many elementary schools. However, 22% of the projects developed curriculum materials for multiple grade bands, several of them specifically aimed at facilitating the transition from upper elementary to middle school or from middle to high school science. Special attention to students from populations typically underrepresented in science was explicit in almost a third of the projects (29%), which developed curricular supports for minority groups (12%), language learners (11%) and/or students with special needs (6%). This is possibly a reflection of policy efforts to reduce achievement gaps in science education.

More than half of the projects developed curriculum materials for just one of the four main science disciplines, and not in equal proportions: 22% for life sciences, 16% for physical sciences, 11% for earth and space sciences, and 8% for environmental sciences. The remaining projects developed either separate curriculum units for more than one science discipline (e.g. one unit for earth science and one for life science), or curriculum materials that explicitly aimed at integrating two or more science disciplines (e.g. a unit about biochemistry that integrates life and physical sciences content). Although all projects had a primary focus on science, 14% developed curriculum materials aimed at integrating science with literacy (6%), engineering (6%), or math (2%).

The curriculum materials also varied in scope, ranging from a comprehensive curriculum (18%) or curriculum sequences (8%) consisting of a series of articulated curriculum units, to supplementary curriculum units (52%) or activities/tools (21%) that could be used in combination with other curriculum materials. In addition, 8% of the projects also developed an accompanying instructional model or framework, 3% also developed learning progressions, and 3% developed formative assessments to be used in conjunction with the curriculum materials.

Major shifts in the funding portfolio

In this section, we describe major shifts in the characteristics of curriculum materials between 2001–2005 and 2006–2010. Table 7 contains descriptive and inferential statistics from these analyses to rule out by-chance variation over time, and Figure 2 provides an overview of all the statistically significant shifts. This is followed by an examination of possible interactions between the observed shifts.

Table 7. Proportion of projects per time period, χ², p-values, and Cramer’s V for award start period effects on each variable.

	Descriptive statistics		Inferential statistics^b (DF = 1)
Variables grouped by dimension	2001–2005 (%)	2006–2010 (%)	N	χ^2	p	V
Format
Delivery format^a			80
(Text)Book	42	5		16.30	.000	.45
Worksheets or handouts	22	36		1.89	.170	.15
Digital resources (e.g. websites, virtual learning environments)	22	21		.04	.848	.02
Animation, simulation or modelling software	33	36		.08	.777	.03
Intelligent tutors	3	5		–	1.000	.05
Educational games	8	20		2.28	.131	.17
Core learning activities			79
Hands-off learning activities required	71	41		7.32	.007	.30
Computer-based learning activities required	60	84		5.80	.016	.27
Hands-on activities learning activities required	69	48		3.45	.063	.21
Technology requirements			79
Access to computers required	43	68		5.10	.024	.25
Access to Internet required	20	59		12.25	.000	.39
Dissemination format			77
Commercial access after end of award	52	9		17.11	.000	.47
Public access after end of award	33	70		10.48	.001	.37
Access limited to request after end of award	15	20		.36	.550	.07
Scope			84
Comprehensive curriculum	32	7		8.91	.003	.33
Curriculum sequence	5	11		–	.449	.10
Supplementary curriculum units/modules	45	59		1.63	.202	.14
Supplementary activities/tools	18	24		.37	.541	.07
Supports for student learning
Cognitive supports	88	70	75	3.29	.070	.21
Accommodation supports	47	48	72	.01	.936	.01
Supports for teaching
Procedural supports
Organisational information (e.g. time, materials required)	94	83	71	–	.282	.16
Instructional strategies	88	83	73	–	.746	.06
Variations created by others	9	38	74	7.85	.005	.33
Educative supports
Alignment to standards	84	73	73	1.32	.251	.13
Goals	97	88	74	–	.226	.16
Information about students’ ideas (e.g. common misconceptions)	70	39	69	6.76	.009	.31
Background information (e.g. additional information about the content covered)	88	65	72	4.80	.028	.26
Continued access to professional development after end of award	74	36	59	8.84	.003	.39
Research and development focus			84
Research & development as primary activity	47	83		11.63	.001	.37
Led by development/outreach organisation alone	45	22		5.05	.025	.24

^a Curriculum materials could be delivered in multiple formats. When this was the case, the main delivery formats were coded as present..

^b When χ² is not reported (–), p-values are calculated with Fisher’s exact test.

Figure 2. Major shifts in key characteristics of grant-funded science curriculum materials.

Format

Chi-square tests revealed a shift away from books as the preferred delivery format. Forty-two percent of the projects awarded between 2001 and 2005 designed books as opposed to only 5% in the 2006–2010 period. As might be expected, the shift away from books was observed to go hand-in-hand with an increase in the overall number of materials that required computer-based learning activities (from 60% to 84%). However, it is less clear whether and to what extent the award start period was also associated with the presence of hands-on learning opportunities. Even though in absolute terms there was a drop from 69% (2001–2005) to 48% (2006–2010) in materials that required hands-on learning, this difference failed to reach statistical significance (p = .063).

The surge of computer-based learning activities in the later period also resulted in a significant increase in the number of materials that require continuous access to computers and Internet. Sixty-eight percent of projects awarded between 2006 and 2010 designed curriculum materials that always required computer access (versus 43% for 2001–2005), and 59% designed materials that always required Internet (as opposed to only 20% for 2001–2005). This reveals an important shift towards curriculum materials that rely heavily on a robust technology infrastructure in schools, which is still not broadly available in K-12 classrooms.

Our findings also revealed important shifts in a number of features related to the dissemination format of the curriculum materials. Specifically, a relatively strong association was found between the award start period and curriculum materials commercially (p < .001; V = .47) and publicly (p = .001; V = .37) available after the end of the award. Fifty-two percent of the projects awarded between 2001 and 2005 developed curriculum materials that became commercially available (as opposed to only 9% in 2006–2010). Conversely, 70% of the projects awarded between 2006 and 2010 provided public access to the curriculum materials (as opposed to 33% in 2001–2005), revealing a clear trend towards open access in later years.

Scope

Our findings also revealed a significant shift in the scope of the curriculum materials developed by the funded projects. More specifically, the proportion of projects concerned with the design of comprehensive curriculum materials decreased from 32% in the 2001–2005 period to 7% in the 2006–2010 period.

Supports for student learning

The presence of cognitive supports for student learning decreased slightly over the years (from 88% in 2001–2005 to 70% in 2006–2010), but this difference failed to reach statistical significance (p = .070). No significant associations were found between the presence of accommodation supports in learner materials (e.g. differentiated tasks or multiple languages) and the award start period, suggesting that the proportion of projects including such supports remained relatively constant over the years.

Supports for teachers

While no major shifts were observed in the presence of supports for student learning, there were significant changes in the types of teacher supports embedded in the curriculum materials, as well as in the continued availability of professional development activities after the end of award.

Chi-square tests revealed a significant decrease (70% in early years vs. 39% in later years) in materials including explicit information about students’ ideas (e.g. common misconceptions, pre-requisite knowledge) which could help teachers to anticipate what learners may think or do in response to the instructional activities. There was also a statistically significant decrease (88% to 65%) in how often teacher materials included background information that could support teachers’ learning of the subject matter. Yet, the proportion of curriculum materials that provided teachers with access to examples of how others had adapted or modified the instructional activities and materials increased. More recently awarded projects were more likely to provide teachers with such examples (38% for 2006–2010 as opposed to only 9% for 2001–2005), possibly reflecting the increasing ease of distributing such information using the Internet. In addition to these important shifts in the types of teacher supports embedded in the curriculum materials, chi-square tests revealed a significant decrease (from 74 to 36%) in the availability of professional development activities (e.g. workshops, summer institutes, online courses) after the end of the award.

Research and/or development focus

Finally, chi-square tests show a significant association between the award start period and the projects’ research and development focus. Eighty-three percent of the projects awarded between 2006 and 2010 were concerned with research and development rather than development only (as opposed to 47% for 2001–2005), corresponding to the increased emphasis on integrated research and development in the requests for proposals. Possibly as a result of this shift, our findings also revealed a drop in the overall number of projects led by curriculum development or outreach organisations alone (from 45% to 22%), and a consequent increase in projects involving universities.

Interactions between shifts in curriculum characteristics

The shifts described above may have been caused by changes in explicit grant requirements, changes in technical capabilities of the field, or changes in areas of interest for researchers and developers. Moreover, one shift may have caused another shift as a secondary consequence. To examine these interactions, we conducted four follow-up logistic regression analyses (see Table 8).

Table 8. Exp b (i.e., odds ratio) from logistic regression analyses of award start period along with other key characteristics: prediction of presence of student information or conceptual background in the teachers’ materials, and public availability or access to professional development (PD) after the award is over.

Included predictors	Information about students’ ideas in teacher materials	Background information in teacher materials	Publicly available after end of award	Access to PD after end of award
Award start period	.41	.34^t	4.36**	.27*
Comprehensive Curriculum	13.19*	3.52
Organisation type			.67
Commercial access				4.01^t

Note:

t=<.10

* =<.05

** =<.01.

First, we examined whether the decrease in the two forms of teacher supports embedded in the curriculum materials was related to the drop in comprehensive materials, which are more likely to provide detailed guidance for teachers (Grossman & Thompson, 2008). For the presence of information about students’ ideas in teacher materials, only the association with comprehensive curricula made a statistically significant contribution to the prediction (p = .018): comprehensive curriculum materials were 13 times more likely to include information about students’ ideas in teacher materials, and the effect of the award start period was no longer significant. The logistic regression analysis for the presence of background information in teacher materials was inconclusive, although it appears that the shift with time is more likely than an association with comprehensive curricula. That is, the drop in information about students’ ideas is likely to be a consequence of the shift away from comprehensive curriculum materials but the drop in background information was an independent shift.

We also examined if the overall decrease in access to professional development activities (e.g. workshops, online courses) after the end of the award might have been influenced by the drop in commercial materials, whose sales income could have provided a source of funding for the later professional development efforts, as has been the case with some mathematics curricula (e.g. Lappan & Phillips, 2009). However, commercial access did not explain the shift in availability of professional development activities: projects awarded in the 2001–2005 period were still more likely to provide access to professional development activities after the end of award even when commercial access (marginally significant; p = .068) was included in the regression.

Finally, a follow-up logistic regression analysis was conducted to ascertain whether the shift to public curriculum materials was associated with the type of organisation(s) leading the project, since some organisations (e.g. support and outreach centres) may have better infrastructure and less barriers for dissemination activities than others (e.g. university research teams; Jacobson, Butterill, & Goering, 2004). However, the model reveals that only the award start period made a statistically significant contribution to the prediction (p = .004).

Discussion

Around the world, large investments are made to support the development of research-based curriculum materials with the goal of improving science education, either as directly supporting a mandatory national curriculum or by providing resources for teachers curating their own curriculum. The current portfolio review sought to identify the characteristics of K-12 science curriculum materials developed with government funding in the United States, and examine shifts in these characteristics between two time periods (2001–2005 and 2006–2010) with differing funding priorities. Below, we outline major shifts found in the funding portfolios, examine them in light of existing international science education trends, and discuss their intended and unintended consequences for science teaching and learning.

Shift away from comprehensive curriculum

A major shift in the characteristics of government-funded curriculum materials concerns their scope, revealing a significant drop in comprehensive curriculum materials in the 2006–2010 period. This shift may have been a consequence of the increasing costs of producing such materials and the stronghold that publishing companies have on the market, but also of changes in emphases across funding programmes. For example, the NSF-funded IMD programme explicitly supported the development of comprehensive curricula for several school years as well as single modules for one grade level (NSF, 2002a). In contrast, the subsequent DRK-12 programme supported the development and study of resources and tools for use with K-12 students (e.g. replacement units for specific school contents, virtual tools to increase students’ engagement in science; NSF, 2006), with no explicit references to comprehensive curricula. Greater support for comprehensive curriculum materials in earlier years is also reflected on the special solicitation Middle Grades Science Instructional Materials Initiative from the IMD programme in the early 2000s. This two-phase initiative was specifically intended to support the development of a new generation of comprehensive science instructional materials for use in middle grades and the transition to high school (NSF, 2000).

The shift away from US funding of comprehensive curriculum is paralleled by the broader international trend towards using online resource exchange websites which predominantly involve smaller units and activities. This shift is worrisome in light of a current trend in curriculum and teaching research, which is that the importance of curricular coherence is becoming increasingly recognised. Specifically, this shift can have important consequences for the ways curriculum materials are used and for their impact on students’ and teachers’ learning. While supplementary units and activities provide more flexibility that may facilitate uptake, a collection of loose materials that are not part of a comprehensive curriculum may lack coherence. Therefore, a possible consequence of the drop in comprehensive curriculum is that teachers and schools are charged with the responsibility of bringing coherence to a curriculum constructed from separate parts. For most teachers, attaining curricular coherence throughout a unit or school year is challenging enough. Few schools currently have the capacity or organisational routines to tackle coherence across grades or subject areas. And yet, failing to attain coherence could have important consequences for student learning, since this has been identified as critical to learning for understanding (Schmidt, Wang, & McKnight, 2005). The challenge of attaining curriculum coherence becomes even greater with the implementation of recent science education reforms, which require the integration and coordination of science and engineering practices, cross cutting concepts, and disciplinary core ideas across multiyear sequences (cf. NRC, 2012; NGSS Lead States, 2013).

The risk to curricular coherence posed by the overall drop in comprehensive curriculum is further exacerbated by the fact that it was found to be associated with a decrease in curriculum materials that included teacher supports for anticipating student thinking and/or common misconceptions, again mirroring a similar gap in materials shared online in resource exchanges. This causes concern since such supports can assist teachers in bringing coherence to a curriculum made up of separate parts by providing information about relevant pre-requisite knowledge and skills, as well as concrete suggestions for anticipating student thinking and dealing with difficulties students may have with the content. While this shift may have been a result of ceasing to prioritise comprehensive materials development, it does not seem likely that it was fully intended, given that it co-occurred with US states beginning to require teachers to use government-approved textbooks.

Decreasing teacher supports

While the presence of supports for student learning remained relatively constant across the two studied time periods, our findings reveal a significant decrease in the types of teacher supports embedded in the curriculum materials. More specifically, there was an overall drop in the proportion of materials that included information that could help teachers to (1) anticipate student thinking and/or common misconceptions, and (2) become familiar with content addressed in the curriculum materials that may lie outside their expertise. This shift could be attributed, in part, to changes in emphases within request for proposals. For example, IMD programme solicitations explicitly asked proposals to ‘describe the products to be produced (e.g. print, CDROM, web-based) that will support teachers and administrators in effectively implementing the materials’ (NSF, 2002a, p. 9–10). Teacher materials were thus seen as an integral part of instructional materials for students. In DRK-12 programme solicitations a clear distinction was made between resources and tools for use with K-12 students on the one hand, and resources and tools for use with teachers on the other hand (NSF, 2006). By presenting these as different strands, the request for proposals might encourage their separation. But even if separation were intended, it hardly seems likely that decreasing supports for teachers would have been an intended consequence.

Another possible explanation for the decrease in teacher supports is the above-mentioned drop in comprehensive curriculum materials. Our results revealed that comprehensive curriculum materials were more likely to integrate information about students’ ideas in teacher materials. Such educative supports are usually directed towards helping teachers anticipate student thinking and common misconceptions across a larger learning progression. Therefore, these types of supports may be more challenging to integrate into materials that are designed for open and flexible uses.

It seems prudent to consider this shift in light of that fact that it squarely contradicts the current trend in international science education research, which recognises the importance of providing teacher supports (Davis, Palincsar, Smith, Arias, & Kademian, 2017). Studies have shown that educative supports can help teachers to better understand and enact reform-oriented science instruction (Arias et al., 2016; Cervetti et al., 2015; Marco-Bujosa et al., 2017), thereby indirectly influencing student learning (Bismack et al., 2015; Pareja Roblin, Schunn & McKenney, in press). As the new vision for science learning set by recent reform documents is realised in K-12 schools, new curriculum materials will need to provide teachers with the necessary supports and guidance to facilitate their students’ engagement with science content and scientific practices (e.g. ACARA, 2017; NRC, 2012; United Kingdom Department for Education, 2015). A failure to provide these supports can have an important negative impact on both teaching and learning. In the sample studied, a decrease in educative teacher supports occurred in concert with an overall decrease in availability of teacher professional development activities after the end of the award, meaning that teacher support vanished unless it was explicitly built into the materials.

Growing reliance on technology-based curriculum materials

There were also major shifts in the format of the materials as well as in the type of core learning activities. Specifically, our findings revealed a significant shift from books to materials that require computer-based learning activities. This is not surprising given the increasing availability of technology in schools (Banilower et al., 2013), the growing international research on the use of technology in science education (e.g. Chang, Quintana, & Krajcik, 2014; Cheng, Chen, Chu, & Chen, 2015), and the increased support for the development of technology-based materials across the various funding programmes worldwide (Ainsworth et al., 2005; Borgman et al., 2008).

There is no doubt that technology offers ample opportunities to enhance science teaching and learning, and has contributed to transforming the field of curriculum design (Krajcik & Mun, 2014; Linn et al., 2016). However, a key consideration for researchers, curriculum developers and policymakers worldwide is opportunity-cost. Even though technology-based materials can facilitate student conceptual understanding (De Jong, Linn, & Zacharia, 2013), the increased emphasis on technology-based materials may take place at the cost of opportunities for hands-on experimentation, which has been shown to influence student attitudes toward science (Ornstein, 2006). On the other hand, research has shown that scaffolded software tools may improve understanding of the inquiry process (Zhang & Quintana, 2012). Further research should examine how technology-based materials could be combined with hands-on activities to more effectively support various student outcomes.

A reliance on technology-based materials also presents several challenges for effective implementation and uptake of innovative curriculum materials. While the international trend toward facilitating  access to computers and Internet in schools has increased over the last years (Banilower et al., 2013), technology integration still remains a challenge for many schools. In addition to ensuring access, Zhao, Pugh, Sheldon, and Byers (2002) argue that for technology integration to be successful, access needs to be functional (e.g. easy access to hardware, availability of up-to-date software) and must be accompanied by a strong human infrastructure that provides the necessary professional development and technical support. Lack of a suitable technology infrastructure and support structures, in combination with decreased availability of professional development after the end of the award, may result in discontinued use of the curriculum materials after funding ends. Further, the diversity of resources and support available in schools also raises the concern that a heavy reliance on technology for innovative educational resources might widen rather than narrow the digital divide (Hohlfeld, Ritzhaupt, Barron, & Kemker, 2008; Zhang, 2014). If this should prove true, it would almost certainly constitute an unintended consequence.

Shift from commercial to public access

Our analyses also indicated a significant shift from commercial to public curriculum materials, revealing a clear trend towards open access in the 2006–2010 period, mirroring broader trends towards open educational resources (Atkins, Brown, & Hammond, 2007). This shift may have been a natural consequence of the overall trend to increase technology-based materials and the growth of the open access movement. These movements may have also caused a shift in emphasis on requests for proposals concerning the dissemination format of government-funded curriculum materials. While IMD programme solicitations explicitly supported the commercial dissemination of curriculum materials (NSF, 2002a), later policy documents strongly recommend making materials freely available on the web with permission for unrestricted use and recombination (Borgman et al., 2008).

While most schools still rely on published curriculum materials, a recent national study of science and mathematics classrooms in the United States reported a relevant trend. Namely, they found that teachers increasingly report incorporating activities from other sources to supplement their instruction (Banilower et al., 2013). Open access to research-based supplementary curriculum materials can potentially facilitate their spread worldwide, as well as the possibilities to modify, adapt, and extend them to improve their usefulness in the classroom (Borgman et al., 2008; Conole, 2013). However, as access to open resources grows in the US and internationally, finding them and evaluating their educational value becomes increasingly difficult (Cafolla, 2006). The way in which these resources are shared, selected and integrated into classroom instruction can have a large influence on their impact on teaching and learning (Abramovich, Schunn, & Correnti, 2013).

Open access may also present challenges for implementation and uptake, and these are not likely to have been intended consequences. For example, computers servers are not free, and software needs to be updated as operating systems or web-browsers change. Moreover, the curriculum materials themselves may need improvements and updates over time as a result of advances in disciplinary content and new technological developments. Unlike commercial materials that benefit from an income stream that can be used to update and adapt the materials, open educational resources often lack a systematic and reliable infrastructure to ensure sustainability and continuous improvement.

Increased support for research alongside development

Finally, our results revealed a significant increase in projects with a research and development focus, likely as a direct result of the greater emphasis on research-based design that was explicit in US grant requests for proposals between 2006 and 2010 (Earle, 2011; NSF, 2008a). For example, over the years ITEST evolved into a programme that incorporated a separate research strand intended to contribute to the knowledge base about approaches that are most likely to increase capacity of the future workforce (NSF, 2008a). This greater emphasis on research is the result of broader efforts at NSF to identify the factors that lead to successful outcomes, and better understand what it takes to scale-up curriculum interventions (Earle, 2011).

Together with the growing international interest in use-inspired research in general (Stokes, 1997) and design-based research in particular (Anderson & Shattuck, 2012), the trend to focus on research could explain the establishment of new partnerships between researchers and curriculum development or outreach organisations. By bringing together diverse – but complementary – perspectives and sources of expertise, such partnerships can contribute to the quality and innovativeness of the materials (Dede, 2005). It seems quite likely that this may have been an intended consequence.

Yet, research suggests that partnerships that form and end primarily in concert with short-term funding grants tend to lack the necessary time to develop shared goals and consolidate productive collaboration structures (Penuel, Fishman, & Cheng, 2011). Though not as explicitly present in requests for proposals during the time frame studied, this understanding is beginning to be reflected in recent calls. For example, NSF’s DRK-12 recent Implementation and Improvement Studies description specifically requires ‘deep engagement of researchers and practitioners during the collaborative research on problems of practice that are co-defined and of value to researchers and education agencies’ (NSF, 2015, p. 6). Similarly, in 2013 IES launched the Researcher-Practitioner Partnerships in Education Research Programme specifically intended to support partnerships between research institutions and state or local education agencies.

Another challenge derived from integrating research and development is the effective coordination of these activities (Penuel, Fishman, Cheng, & Sabelli, 2011). The scope of project funding and the short project timelines often makes it challenging to balance the need to attend to the practical demands of the design work and the need to preserve the rigour of research (Dede, 2005; Penuel et al., 2011). To better understand the breadth of consequences, and to unpack different consequences of changing who does the work (from design organisations to research organisations) versus what kind of work they do (from design only to research and design), additional investigation is required.

Limitations

A limitation of the current study is a common concern for review studies, and pertains to the scope of the portfolio review. Since our goal was to study key characteristics of designed materials, we focused only on projects awarded before 2010, and therefore cannot report on more recent trends. However, there have not been new funding calls for comprehensive curricula. At the same time, the focus on research together with development, and on openly-shared, online materials has continued. Further, given the slow rate at which research and curriculum materials are disseminated, the observed shifts are likely still reflected in the materials that are currently in the process of moving to scale and in research of current K-12 science curriculum materials. Beyond the issue of temporal scope, given our focus on government funding within specific US agencies and funding programmes, we may have missed other shifts that occurred in materials developed during this time with the smaller amounts of support provided by private foundations or other government agencies. Other countries may have shown different trends in funding for curriculum material development, but it is likely that the kinds of materials being created outside of the US have also been affected by the same kinds of international trends in education research and practice (e.g. growth in resource sharing via the Internet and growth in computer availability in schools). Another related limitation is that we did not have access to samples of curriculum materials for a small proportion of projects in our sample (17%). As a result, these projects had missing information for several of the key curriculum features we investigated.

Finally, a potential limitation of our study pertains to the analytic framework we used to characterise the curriculum materials. Although the framework we used was informed by existing research on curriculum characteristics that may influence uptake, use, and instructional outcomes, it does not capture all the features that might have changed, nor did it capture potential changes in the quality (rather than presence) of the features that were studied. Further, more research is needed to understand which curriculum features might be most important for facilitating deep changes in teaching and learning.

Concluding remarks and implications

The current study identified important shifts in the characteristics of US-government funded K-12 science curriculum materials between two periods of time that differ in their funding priorities. Such shifts are important because they are likely to have complex impacts on learning and teaching. For example, whereas a move away from comprehensive curricula may result in increased flexibility for implementation and uptake, it also places greater responsibility on teachers to ensure curriculum coherence and improved student outcomes – especially when combined with the significant decrease in teacher supports. Moreover, the heavy reliance on technology and the growing trend towards open access may provide new learning opportunities and facilitate spread, while also bringing challenges for (sustainable) implementation.

Several implications can be derived from this study for science education researchers, curriculum developers, and policymakers in the United States and internationally. First, the science education research community should investigate how the shifts described here may impact teaching and learning, with particular attention to the concern of curricular coherence. These studies would in turn provide evidence that could guide policymakers’ decisions and curriculum developers’ work. Second, this study suggests that curriculum developers aspiring to create materials with significant impacts on student outcomes and teaching practices should consider how to design materials that facilitate curricular coherence in general, and successful teacher curation in particular. Finally, the findings of this study show a general trend in grant-funded resources that put more responsibility for shaping curriculum in the hands of teachers and schools. If this is viewed by policymakers to be a desired outcome, then new research funding as well as new development funding opportunities should be provided to better understand and facilitate such school-based curriculum curation.

Disclosure statement

The authors declare that they have no conflict of interest.

Funding

The research reported here was supported by the National Science Foundation, through [Grants #1251562 1252416 and 1252373] to the University of Pittsburgh, TERC and the Lawrence Hall of Science at the University of California-Berkeley.

Notes on contributors

Natalie Pareja Roblin is an educational researcher and teacher educator. Her research focuses on curriculum design, technology integration and teacher professional development, and on how these may be supported by research-practice partnerships.

Christian Schunn is a co-director of the Institute for Learning, senior scientist at the Learning Research and Development Center and a professor of Psychology, Learning Sciences and Policy, and Intelligent Systems at the University of Pittsburgh and Chief Learning Scientist for Peerceptiv. He directs research and design projects in writing, science, mathematics, technology and engineering education.

Debra Bernstein is a senior researcher at TERC. Her research focuses on the creation of STEM learning environments in formal and informal settings.

Susan McKenney is a professor of teacher professionalization, school development and educational technology at the University of Twente in the Netherlands. Her research focuses on these three themes, especially in relation to curriculum design.

Acknowledgements

The research reported here was supported by the National Science Foundation, through Grants #1251562 1252416 and 1252373 to the University of Pittsburgh, TERC and the Lawrence Hall of Science at the University of California-Berkeley. The opinions expressed are those of the authors and do not represent views of the National Science Foundation.