Exploring shifts in the characteristics of US government-funded science curriculum materials and their (unintended) consequences

Figures & data

Figure 1. Theoretical model underpinning this study.

Table 1. Inclusion and exclusion criteria.

Inclusion criteria	Exclusion criteria
1. The project takes place within the context of K-12 education and in- school activities	Projects concerned with out-of-school/after school programmes, and/or informal learning
	Projects targeted exclusively at pre-school, undergraduate and/or graduate students
2. The project focuses on science education, including combinations of science with other disciplines such as mathematics, engineering or literacy	Projects focused exclusively on K-12 disciplines other than science (e.g. mathematics, engineering)
	Projects where science is a secondary focus (i.e., focus is on other disciplines but with some connections to science)
3. The project explicitly describes the design of curriculum materials (broadly defined, including curriculum frameworks, learning progressions, materials for classroom use) as an important goal	Projects exclusively concerned with teacher professional development, or where curriculum design was primarily a means for teacher professional development
	Projects exclusively concerned with research (e.g. research synthesis, efficacy studies) or where curriculum design was primarily intended to serve research purposes only and not for actual classroom use (e.g. design as treatment)
	Projects concerned with the design of large scale (national or international) benchmark assessments
	Projects concerned with the design of educational policies, recommendations, etc.

Table 2. Overview of search and selection process by funding organisation.

Step	Resulting Set	National Science Foundation (NSF)	Institute of Education Sciences (IES)	Subsequent NFS & IES awards*	Total
1	Awards between 2001–2010 from programmes that fund development (NSF) or research goals relevant to curriculum development (IES)	1144	157	–	1301
2	Awards concerned with K-12 science curriculum materials design (meeting inclusion criteria)	200	26	–	226
3	Awards specifically concerned with development of curriculum materials for classroom use	143	19	–	162
4	Projects concerned with curriculum materials for classroom use (after merging related awards)	129	15	3	147
5	Random sample of projects concerned with curriculum materials for classroom use	77	6	1	84

^* Projects that received subsequent awards from NSF and IES within our 10-year time window.

Table 3. Coding scheme for format of curriculum materials.

Code	Description	Examples
Delivery format^a (no; yes)
(Text)book	Printed book designed to support the study of a specific subject	• Textbook • Story book(s)
Other paper-based materials	Paper-based materials designed to support student learning of a particular topic/subject	• Worksheets • Handouts • Investigation notebooks
Digital resources	Educational resources available on the World Wide Web	• Websites • Web-based learning environments • Online databases
Simulation software	A programme that allows the user to observe an operation through simulation without actually performing that operation	• Simulations • Animations • Modelling tools
Intelligent tutors	Computer system that aims to provide immediate and customised instruction or feedback to learners, usually without intervention from a human teacher	• Intelligent virtual assistants that engage students in meaning-making exchanges in which students interactively devise explanations and make predictions.
Educational games	Games explicitly designed with educational purposes	• Adventure games • Virtual reality games
Core learning activities (required; optional)
Hands-off learning activities	Students engage with the content through lectures, discussions and/or paper-based materials such as a textbook or worksheets	• Students read four science books, including Mystery Forces, which challenges them to figure out which forces are at work in a variety of scenarios.
Hands-on learning activities	Students are actively engaged in manipulating physical objects, such as ramps, test-tubes or mechanical devices	• Using metric rulers, magnifying glasses, microscopes, and the identification guides, students investigate the invertebrates they collected in their schoolyard.
Computer-based learning activities	Students engage with the content through the use of educational software such as simulations or educational games	• Students learn Darwin’s model of natural selection using computer-based models depicting interacting organisms and their environments.
Technology requirements (not always required; always required)
Access to computers	Extent to which access to computers is required to implement the curriculum materials	• Simulations • Modelling tools
Access to Internet	Extent to which access to internet is required to implement the curriculum materials	• Online educational game • Online data bases
Dissemination format (no; yes)
Commercial access	Curriculum materials distributed by a commercial publisher after end of award	• Materials available for purchase from [publisher]
Public access	Curriculum materials are available for free download/use after end of award	• Materials available for free download from [university project website] • Materials available for free use from [non-profit research and development organisation website]
Access limited to request	Curriculum materials are not commercially or publicly available after end of award, but can be accessible upon request.	• Principal investigator states that access is provided upon e-mail request.

^a Curriculum materials could be delivered in multiple formats. When this was the case, the main delivery formats were coded as present.

Table 4. Coding scheme for scope.

Code	Description	Examples
Comprehensive curriculum (no; yes)	Series of curriculum units that cover the whole range of content/key understandings for a specific grade level(s) and science discipline(s)	• Three-year curriculum that introduces students to all of the core concepts in inquiry, physical sciences, life sciences, and earth-space sciences found in the national standards for grades 9–12
Curriculum sequences (no; yes)	Short series of curriculum units designed with the intention to be used together and in a particular sequence, since each new unit builds on the understandings developed in the previous one	• A sequence of four consecutive curriculum units across grade six and grade seven aimed to improve students’ cumulative and integrated understanding of energy concepts
Stand-alone supplementary/replacement units (no; yes)	Lessons or instructional activities structured around a specific topic or common theme. Even though multiple units may be developed, these do not need to follow a particular sequence or build on each other	• A 6-week curriculum unit focused on key ideas in chemistry and their application to living systems, in order to prepare middle school students for high school biology
Stand-alone supplementary/replacement activity/tool (no; yes)	Ancillary resources and/or strategies teachers can use in combination with other curriculum materials to facilitate student learning	• Lesson starters • Laboratory activities • Simulations

Table 5. Coding scheme for supports for student learning.

Code	Description	Examples
Cognitive supports (no; yes)	Supports intended to assist learners by highlighting key ideas or relationships among concepts, providing additional information for completing a task, and/or giving opportunities to assess what they know.	• Concept maps/outlines • Worked examples • Hints • Reflection prompts • Automated feedback
Accommodation supports (no; yes)	Specific supports for English language learners, learners with different ability levels and/or learners with special needs	• Translations to other languages • Text-to-speech • Graphic organisers • Adjustable text size • Differentiated tasks for students with different ability levels

Table 6. Coding scheme for supports for teaching.

Code	Description	Examples
Procedural supports (no; yes)
Organisational information	Information about time and resources required to effectively implement the curriculum materials	• Number of class periods required • List of materials and equipment • Safety guidelines • Advanced preparation
Instructional strategies	Specification of techniques or tips teachers can use to facilitate student learning	• Pedagogical suggestions to productively support student learning • Tips for leading whole-class discussions • Recommendations for how to integrate the activities/tools in the curriculum
Access to variations created by others	Examples of how other teachers have adapted the curriculum materials to their specific classroom situations (beyond exchanges during PD workshops) and/or access to a platform where teachers can share their adapted lesson/activity plans	• Lesson plans, worksheets and/or activities created by teachers available for download from project website • Platform where teachers can share their modified lesson plans, worksheets and/or experiences with others
Educative Supports (no; yes)
Unit goals	Brief statements of what science concepts or practices students will be able to learn, which help teachers understand the purposes of the units and concrete instantiations of those concepts and practices	• Overview of learning goals for each curricular unit, lesson or activity
Alignment to standards	Explicit references to the standards addressed in the curriculum materials, which help teachers see concrete instantiations of relatively abstract standards statements	• List of National and/or State content standards addressed in the curriculum materials
Information about students’ ideas	Information that could help teachers to anticipate student thinking and misconceptions	• Overview of pre-requisite knowledge (i.e., what students should already know and/or be able to do) • Overview of common misconceptions (i.e., misunderstandings and/or or difficulties students have shown to have with a particular content/activity in the materials)
Background information	Information that could assist teachers’ learning of the subject matter	• Description of key science concepts addressed in the curriculum materials • Links to websites and/or resources with additional information about the key science ideas addressed in the curriculum materials
Continued access to professional development activities (no; yes)
Access to professional development after end of award	Professional development activities remain available once direct project support ends	• Workshops organised by the publisher • Webinars • Online courses

Table 7. Proportion of projects per time period, χ², p-values, and Cramer’s V for award start period effects on each variable.

	Descriptive statistics		Inferential statistics^b (DF = 1)
Variables grouped by dimension	2001–2005 (%)	2006–2010 (%)	N	χ^2	p	V
Format
Delivery format^a			80
(Text)Book	42	5		16.30	.000	.45
Worksheets or handouts	22	36		1.89	.170	.15
Digital resources (e.g. websites, virtual learning environments)	22	21		.04	.848	.02
Animation, simulation or modelling software	33	36		.08	.777	.03
Intelligent tutors	3	5		–	1.000	.05
Educational games	8	20		2.28	.131	.17
Core learning activities			79
Hands-off learning activities required	71	41		7.32	.007	.30
Computer-based learning activities required	60	84		5.80	.016	.27
Hands-on activities learning activities required	69	48		3.45	.063	.21
Technology requirements			79
Access to computers required	43	68		5.10	.024	.25
Access to Internet required	20	59		12.25	.000	.39
Dissemination format			77
Commercial access after end of award	52	9		17.11	.000	.47
Public access after end of award	33	70		10.48	.001	.37
Access limited to request after end of award	15	20		.36	.550	.07
Scope			84
Comprehensive curriculum	32	7		8.91	.003	.33
Curriculum sequence	5	11		–	.449	.10
Supplementary curriculum units/modules	45	59		1.63	.202	.14
Supplementary activities/tools	18	24		.37	.541	.07
Supports for student learning
Cognitive supports	88	70	75	3.29	.070	.21
Accommodation supports	47	48	72	.01	.936	.01
Supports for teaching
Procedural supports
Organisational information (e.g. time, materials required)	94	83	71	–	.282	.16
Instructional strategies	88	83	73	–	.746	.06
Variations created by others	9	38	74	7.85	.005	.33
Educative supports
Alignment to standards	84	73	73	1.32	.251	.13
Goals	97	88	74	–	.226	.16
Information about students’ ideas (e.g. common misconceptions)	70	39	69	6.76	.009	.31
Background information (e.g. additional information about the content covered)	88	65	72	4.80	.028	.26
Continued access to professional development after end of award	74	36	59	8.84	.003	.39
Research and development focus			84
Research & development as primary activity	47	83		11.63	.001	.37
Led by development/outreach organisation alone	45	22		5.05	.025	.24

^a Curriculum materials could be delivered in multiple formats. When this was the case, the main delivery formats were coded as present..

^b When χ² is not reported (–), p-values are calculated with Fisher’s exact test.

Figure 2. Major shifts in key characteristics of grant-funded science curriculum materials.

Table 8. Exp b (i.e., odds ratio) from logistic regression analyses of award start period along with other key characteristics: prediction of presence of student information or conceptual background in the teachers’ materials, and public availability or access to professional development (PD) after the award is over.

Included predictors	Information about students’ ideas in teacher materials	Background information in teacher materials	Publicly available after end of award	Access to PD after end of award
Award start period	.41	.34^t	4.36**	.27*
Comprehensive Curriculum	13.19*	3.52
Organisation type			.67
Commercial access				4.01^t

Note:

t=<.10

* =<.05

** =<.01.