Environmental education and K-12 student outcomes: A review and analysis of research

ABSTRACT

Many practitioners and researchers describe academic and environmental benefits of environmental education for kindergarten through twelfth grade (K-12) students. To consider the empirical underpinnings of those program descriptions, we systematically analyzed the peer-reviewed literature (1994–2013), focusing on outcomes of environmental education programs with K-12 students. In the resulting sample of 119 articles, we identified 121 unique outcomes, finding that most articles reported positive findings from the programs under study. Reflections stemming from the review highlight the versatility of environmental education, while also suggesting opportunities for bolder and more diversified approaches in research design and thinking.

KEYWORDS:

• environmental education outcomes
• K-12 students
• systematic review

Introduction

In the past 25 years, research in environmental education (EE) has grown in output, scope, and types of design, as well as methods and approaches (Ardoin, Clark, & Kelsey, 2013; Rickinson, 2001; R. B. Stevenson, Wals, Dillon, & Brody, 2013). With this growth, researchers and practitioners find it increasingly challenging to remain aware of current findings and implications for on-the-ground activities. As a result, practices may not reflect recent findings and, at the same time, research may become distanced from practice, hampering innovation in both directions.

Systematic reviews represent one strategy for addressing this proliferation in research (Rickinson & Reid, 2016). The utility of systematic reviews extends beyond the empirical and positivistic realms as they can help map a field while identifying strengths, weaknesses, and gaps. Individual reviews can synthesize and analyze a field's work; complementary reviews can leverage the expertise of researchers involved in the review process (Rickinson & Reid, 2016). As Hart (2003, p. 247) suggests, because authors do not—and indeed cannot—remain invisible in the literature-review process, each review contributes to the research-and-practice dialogue.

Thanks to EE's interdisciplinarity, numerous research areas are ripe for examination in this way (R. B. Stevenson, Wals, et al., 2013). Rickinson (2001), for example, conducted an extensive review of learners and learning in EE. Whereas Stern, Powell, and Hill (2014) investigated programmatic best practices, Williams and Dixon's (2013) systematic review focused on academic outcomes within the context of school gardens.

In working with a range of EE stakeholders, including researchers, practitioners, and funders, we uncovered a need for a more robust understanding of what outcomes are addressed in various sectors of the field. We noted an interest in better understanding how, and under what conditions, EE with school-aged youth relates to diverse outcomes, including non-environmentally related outcomes (e.g., academic achievement, youth development, and social and emotional learning). Ladwig (2010) noted that EE programs measuring environmentally specific outcomes—such as environmental knowledge, attitudes, skills, and behavior—often extend to include both personal and interpersonal outcomes. R. B. Stevenson, Wals, et al. (2013) discuss how EE goes beyond mere understanding and conceptualization to developing learners' agency, including a problem-solving orientation. Through focus groups and surveys with EE participants and practitioners, West (2015) identified a large number and range of outcomes, noting an emphasis on knowledge for both groups, plus social outcomes among participants. Likewise, previous literature has suggested that environmental educators express diverse opinions about both the shorter- and longer-term goals of their work (Ardoin, Biedenweg, & O'Connor, 2015; Hart, 2007; Schusler et al., 2009).

Prior reviews of EE outcomes have primarily focused on the environmental aspects of those efforts; similarly, they have been dominated by broad outcome categories, despite the presence of what Stern, Powell, and Hill (2014, p. 586) call “more nuanced outcomes.” Our review aims to explore those nuances with attention to outcomes other than those considered “traditional” in EE (Stern et al., 2014). Therefore, rather than using a priori coding focused on environmental knowledge, attitudes, behaviors, and skills, we desired to uncover the breadth of EE outcomes under study using an exploratory process similar to Williams and Dixon's (2013) school-gardens review. We aimed to include many studies, similar to the breadth of Rickinson's (2001) sample of 110 publications.

Our review focuses on K-12 students (approximately ages 5 through 18) because they represent a primary audience for EE; as such, EE research and research reviews frequently focus on youth within this band. Similarly, “students,” or those pursuing EE within formal context or structure, are a key audience for many EE stakeholders, such as funders, policymakers, and school administrators. By examining the most recent 20 years of literature related to outcomes of EE programs for K-12 students—and doing so systematically and analytically—we hoped to gain an understanding of existing research within this paradigm and, concurrently, illuminate gaps. This study describes our review process, presents an interpretation of our findings, and compares our findings with those from similar reviews.

We focus on empirical studies that measure outcomes, and we define our coding categories iteratively to reflect what other researchers may indicate as primary outcomes. By using this honed approach, we are not suggesting that an outcome orientation is, or should be, the driving force of EE research. We recognize, rather, that this is one of numerous possible framings and we chose this orientation as a lens that may help disentangle some of the ways in which researchers have examined EE programs.

The following research questions guided the analysis and coding of the articles that resulted from our search:

•	For K-12 audiences, primarily in formal settings, what EE outcomes are researchers seeking to measure?
•	What implications about past, present, and future EE research might a review of EE outcomes research suggest?

Guiding principles of the review

Our work benefited from and was influenced by the methods, findings, and philosophical discussions of previous EE research reviews (e.g., Hart & Nolan, 1999; Rickinson, 2001; Stern et al., 2014; Williams & Dixon, 2013). We also learned from critiques of research reviews in education, more generally, and EE, in particular (e.g., Cooper, 2010; Gough, 2007; Gough, Oliver, & Thomas, 2012; Hallinger, 2013; Marcinkowski, 2003; Rickinson & Reid, 2016). Given others' criticisms (e.g., Cooper, 2010; Gough, 2007) that, historically, many research reviews have not met methodological standards, we familiarized ourselves with discussions of the nature and quality of research reviews. We pursued techniques of systematic reviews that use “systematic and explicit, accountable methods” (Gough et al., 2012, p. 2) and are recognized for transparency, reproducibility, and relevance (EPPI Centre, 2009).

We took a qualitative coding approach, synthesizing studies conducted within different paradigms and using diverse methods. Adapting Marcinkowski's (2003) purposes for a research review, we intended to produce a review that would (a) identify quality EE research that empirically measured outcomes; (b) describe and characterize the EE programs implemented, audiences investigated, research designs and methods used, and findings reported; and (c) summarize and synthesize the resulting data related to trends and gaps.

We endeavor to present our methods clearly and completely. As appropriate, we provide support for our decisions, describing methodological and/or theoretical rationales. We developed those rationales through discussions with research team members and outside colleagues.

Methods

We began by considering previous EE research reviews (e.g., Hart & Nolan, 1999; Rickinson, 2001; Stern et al., 2014; Williams & Dixon, 2013) to identify structural strategies and methodological issues to address. Combined with the nature and scope of our topic, as well as target audiences, those considerations informed our review steps. (See Figure 1 and following.)

Figure 1. Elements of the review process.

Literature search

We performed systematic pilot searches using varying terms, engines, and parameters to identify research focused on EE-related student outcomes with K-12 audiences (e.g., terms at this stage included: environmental education plus learning, student outcomes, outcomes). We examined the resulting output to determine the extent to which we had surfaced relevant articles; those results informed our search process. We selected the EBSCOhost search engine because of its function as a meta-engine that incorporates results from a variety of databases. We used the following databases: ERIC, Education Full Text, GreenFILE, Environment Index, Academic Search Premier, British Education Index, and Africa-Wide Information.

We followed Stern et al. (2014) and used environmental education as the primary term. This limited our results to articles with environmental education in their title, keywords, abstract, article body, or references, or when in a journal with environmental education in its title.¹ To isolate EE articles addressing K-12 student outcomes, we added student as a search term and excluded articles with college or university. This produced a manageable number of results while maintaining a focus on K-12 student outcomes. Our search included the following restrictions: that the articles were peer-reviewed, published between 1994 and 2013,² and written in English. The final search terms and restrictions produced a sample of 2,034 articles for initial review.³

Identifying articles for inclusion

Our research team collaboratively developed a decision tree with criteria that would allow us to determine whether to include each of the 2,034 articles for further review. (See Figure 2.) Each team member read abstracts from assigned subsets of the articles, applying those criteria.

Figure 2. Decision tree to determine inclusion/exclusion.

Some of the most commonly excluded article types were those that lacked an empirical research component; focused on college-aged students, pre-kindergarten students, or educators; or lacked implementation of an EE program or intervention (i.e., were primarily conceptual in nature). When we applied those criteria to the abstracts, we excluded approximately 80% (n = 1,608) of the initial search results.

We more thoroughly reviewed the remaining 20% (n = 426) of articles; a different team member read each of these articles, applying the same exclusion criteria. This second-level review resulted in 142 articles that we coded to a number of categories. (See Table 1.) This process involved a higher level of scrutiny and deeper reading of the articles, leading to exclusion of an additional 23 articles that did not meet our criteria. After these pre-review rounds, we included 119 articles for full coding and analysis.

Table 1. Selected main codes.

Code
Abstract	Examples
Type of program or intervention	Classroom-based
	Extra-curricular
	Home/school
	School-wide
Length of program	One day
	One instructional unit
	School year
Age range of participants	6th grade
	15–17-year-olds
	1st–3rd grades
Geographic distribution	United States
	Australia
	China
Methods	Pre- and post-test
	Interviews
	Delayed follow-up
Research design	Quasi-experimental
	Case study
	Action research
Type of outcome measured	Environmental knowledge
	Action competence
	Cyclic thinking
Key findings	Positive^a
	Null
	Negative

^a As defined by the article authors; that is, we coded the articles as having “positive” findings if the article's authors stated an objective (or objectives) and had findings that they deemed in support of that objective.

Developing coding scheme and process

We developed our coding guide through inductive and deductive means. We began the study with stated objectives to examine outcomes measured; study characteristics, such as geographic location and participants; methods; and findings. Based on those features, we developed broad coding categories. Within the categories, we added codes to describe the programs, research paradigms, methods, and outcomes represented in the articles. (See Table 1.) We coded the data using NVivo 10 for Windows (QSR International), a qualitative data-analysis software package.

To establish inter-coder agreement, three coders initially read and coded the same four articles. The coders discussed quantitative (amount coded) and qualitative (type of information coded) differences in coding and revised accordingly. Articles then were divided between two coders. The research team continuously discussed the categories throughout the coding process; once completed, a single researcher reviewed coding across all articles for consistency.

Results

Describing the sample

Our resulting sample included 119 articles published in 36 journals: 39 (33%) of those articles were published in Journal of Environmental Education; 24 (20%) in Environmental Education Research; 7 (6%) each in International Journal of Science Education and International Research in Geographical and Environmental Education; and 5 (4%) in Applied Environmental Education and Communication. The remaining 31 journals included three or fewer articles each. (See Supplemental Information at https://purl.stanford.edu/xv847bw0331 for full bibliographic information of the reviewed articles.)

We found an increase in the number of peer-reviewed journal articles exploring measureable outcomes of EE with students during the 20-year search parameter period. Nearly half of the articles (52, or 45%) were published between 2009 and 2013 (Figure 3).

Figure 3. Chronological distribution of the 119 articles.

The articles in our dataset reported on studies involving K-12 students from 33 countries. Fifty-six (47%) of the articles focused on students in the United States. Canada and Germany were geographic areas with the next-highest number of articles, although articles reporting on studies from those countries were still notably fewer than those reporting on U.S.-based studies: 8 (7%) and 6 (5%), respectively. (See Supplemental Table A for full list of countries represented in articles.)

Design and methods of reviewed articles

Most studies (78%) employed quasi-experimental design methods, using either pre- and post-measures or multiple groups to explore the influence of EE on specified outcomes. Researchers implementing quasi-experimental studies also used other descriptors for their work that often alluded to the purpose of their research. Those terms included a range of evaluation types, such as design-based and utilization-focused evaluation. The remaining 22% of the articles used a variety of other research designs. (See Table 2.)

Table 2. Research designs used in articles (n = 119).

Research design	Number of articles (% of overall sample)^a
Quasi-experimental	93 (78%)
Case study	8 (7%)
Comparative study	4 (3%)
Action research	3 (3%)
Ethnography	3 (3%)
Phenomenological	2 (2%)
Survey research	2 (2%)
Autoethnography	1 (1%)
Combination of pre-experimental and case study	1 (1%)
Correlational	1 (1%)
Descriptive	1 (1%)

^a Percentages may not sum to 100 due to rounding.

Researchers most frequently collected data by asking students to respond to structured instruments (82% of articles), which included either closed- or open-ended items, or a mix of both. Although many were written instruments, some were administered verbally or electronically. In 29% of the studies, researchers reported conducting interviews, which ranged from highly structured to completely unstructured. As part of observational measures (reported in 19% of the articles), researchers played a range of roles, sometimes acting solely as an observer while other times interacting in more of a participant-observer role. (See Table 3.)

Table 3. Method of data collection (n = 119).

Method of data collection	Examples	Number of articles (% of overall sample)^a
Structured instruments	Knowledge tests, attitudinal scales	97 (82%)
Interviews	Structured interviews, semi-structured interviews, in-depth conversations, informal discussions	35 (29%)
Observation	Participant observation, researcher interaction, video-recording of events	23 (19%)
Review of student work	Reviews of journals, reflective writing, portfolios	15 (13%)
Conceptual activities	Concept maps, word association, drawings, repertory grid	9 (8%)
Group discussions	Focus groups, informal group discussions	9 (8%)
Photovoice	N/A	1 (1%)
Computer logs	N/A	1 (1%)

^a The sum of articles is greater than 119 because a single article often reported on multiple methods of data collection.

Post-program data collection most often occurred immediately after program completion. Of the reviewed articles, 34 (29%) included a delayed post-program measure; 76% of those delayed measures were implemented within one to six months after completion of the EE program. (See Table 4.)

Table 4. Length of follow-up in articles with delayed assessment (n = 34).

Length	Number of articles (% of overall sample)^a
Less than a month	5 (15%)
1 to 6 months	26 (76%)
7 months to 1 year	4 (12%)
2 to 3 years	2 (6%)

^a The sum of articles is greater than 34 because a single article may include follow-up in more than one category of length.

Range of EE programs under study

In terms of target age group, the largest percentage of the articles described programs for middle-school students (ages 11–14, 57%). The next-largest group was elementary (ages 5–11, 47%), with an emphasis on upper-elementary students (ages 9–11). (See Figure 4.) Just over one-third (34%) of the reviewed articles examined EE with high-school students (ages 14–18).

Figure 4. Age range of participants.

Researchers investigated an array of program lengths, types, and settings. Programs ranged from one-time, one-hour experiences to multi-year projects. Other types included daylong field trips, semester-long activities recurring weekly, and weeklong residential programs. The topics covered ranged from defined issues, such as water or recycling, to the environment in general. Despite our emphasis on the K-12 lens, programs occurred both inside and outside school locations, such as aquariums, parks, and museums.

Outcomes under study

We identified 121 unique, empirically measured outcomes within this sample. Using thematic analysis, we organized those outcomes into broader categories, which mirrored outcomes described in guiding documents of the EE field, including the United Nations Educational, Scientific and Cultural Organization's (UNESCO) Tbilisi Declaration (1978) and the North American Association for Environmental Education's (NAAEE) Guidelines for Excellence series (Hollweg et al., 2011; NAAEE, 2010; Simmons, 1995). In addition to traditional EE outcomes of environmental knowledge, attitudes, and behavior, programs also included a focus on non-environmentally related outcomes, such as attitudes toward computers or verbal communication skills. Through this process, we identified the following broad categories: (1) knowledge (including awareness, perceptions, content knowledge, skills knowledge, socio-political knowledge, and issue-specific understandings); (2) dispositions (such as, interest, affect, attitude, and behavioral intentions); (3) competencies (skills, including cognitive and social); (4) behavior (actions); (5) personal characteristics (self-esteem and character development, among others); and (6) multi-domain outcomes (those spanning more than one domain, such as academic achievement, which involves at least knowledge and competencies).

Articles in our sample most frequently addressed changes in knowledge and dispositions: 68% and 61% of the articles, respectively, addressed those areas. Table 5 provides information on outcome domains; the order of the outcomes under each domain reflects the frequency with which articles studied the outcome. We list outcomes by decreasing level of frequency. Within knowledge, for example, articles most frequently examined environmental knowledge, followed by environmental awareness and environmental perceptions. Supplemental Table B lists each study with associated outcomes and author-supplied keywords.

Table 5. Domain of outcome with all examples (n = 119).

Outcome domain	Number of articles measuring specified domain	Percentage of articles measuring specified domain
Knowledge^a	81	68%
Environmental knowledge, Environmental awareness, Environmental perceptions, Ecological knowledge, Perceived knowledge of action, Awareness of global connections, Awareness of social activism, Career knowledge, Chemistry knowledge, Conceptualizations of the environment, Environmental issue awareness, Geography content, Knowledge for action, Knowledge of citizen action skills, Knowledge of environmental actions, Knowledge of environmental issues, Knowledge of environmental action skills, Knowledge of health effects of inhalants, Knowledge of health effects of pollution, Risk knowledge, Science content, Skill knowledge, Understanding of science methods
Dispositions	73	61%
Environmental attitudes, Environmental behavioral intentions, Environmental concern, Locus of control, Connectedness with nature, Environmental sensitivity, Interest in science, Science attitudes, Self-confidence to act, Self-efficacy, Sense of place, Attitudes toward school, Attitudes toward computers, Attitudes toward experiential learning, Attitudes toward inhalants, Attitudes toward the Internet, Attitudes toward traditional teaching, Civic responsibility, Confidence in action skills, Connection to land, Disposition toward critical thinking, Ecological mindset, Empowerment, Environmental identity, Environmental responsibility, Environmental studies self-efficacy, Group locus of control, Interest in chemistry, Interest in environmental issue under study, Interest in fishing, Interest in gardening, Interest in geography, Interest in learning and discovery, Interest in natural world, Intrinsic motivation, Perceptions of control for managing risks to the environment, Personal commitment to aquatic environments, Personal responsibility to manage risks, Preservation values, Priorities for resource management, Reinforcing factors, Relationship to environment, Risk attitudes, Science self-efficacy, Sense of environmental stewardship, Task value, Utilization values, Value of nature
Competencies	31	26%
Decision-making skills, Environmental problem solving, Critical thinking, Perceived skill in use of action skills, Social skills, Fishing skills, Leadership, Problem-solving, Systems thinking, Action planning, Analytical skills, Career planning, Computer skills, Cyclic thinking, Gardening skills, Geography skills, Group work, Higher-order thinking skills, Internet skills, Issue analysis, Issue evaluation, Issue identification, Land subsistence skills, Metacognitive skills, Monitoring environmental problems, Oral communication, Outdoor skills, Participatory capacities, Reading skills, Research skills, Safety skills, Science process skills, Self-directed learning, Technology skills, Writing skills
Behavior	24	20%
Environmental behavior, Water conservation, Battery recycling, Energy savings and carbon emissions reduction, Fishing participation
Multi-domain outcomes	7	6%
Academic achievement, Action competence,^b Environmental stewardship, Mental models
Personal characteristics	3	3%
Self-esteem, Autonomy, Character development, Maturity, Poise

^a The knowledge domain includes cognitive aspects, as well as those that may have an action-orientation, such as knowledge for action and skill knowledge. Based on discussions in the literature of the multi-dimensionality of environmental knowledge (e.g., Frick, Kaiser, & Wilson, 2004; Jensen, 2002), we categorized all objectives related to knowledge—whether cognitive, affective, or skills-related—within the knowledge domain.

^b Although this term implies classification as a “competency,” we categorized action competence as a multi-domain outcome given its definition in the EE literature as a construct involving knowledge, skills, dispositions, and action (Mogensen, 1993, as cited in Fontes, 2004).

In addition to the six domain categories, we parsed outcomes based on area of focus. More broadly, we considered whether the categories explicitly related to the environment. (See Supplemental Table C.)

Program findings

We tallied findings based on domain (Table 6) and area of focus (Table 7). Authors reported positive findings in 94% of articles examining environmental outcomes and 95% of those considering non-environmental outcomes. Authors reported null findings as well: 40% for environmental outcomes and 37% for non-environmental outcomes.

Table 6. Article findings for specified outcome domains.

Outcome domain	Positive (row %)	Null (row %)	Negative (row %)	Number of articles
Knowledge	79 (98%)	20 (25%)	1 (1%)	81^a
Competencies	28 (90%)	8 (25%)	0 (0%)	31^a
Dispositions	64 (86%)	36 (49%)	2 (3%)	73^a
Behavior	20 (83%)	11 (46%)	1 (4%)	24^a
Multi-domain	7 (100%)	1 (20%)	0 (0%)	7^a
Personal	3 (100%)	1 (33%)	0 (0%)	3^a

^a In each row, the sum of articles reporting each level of findings is greater than the total number of articles because researchers often reported more than one type of finding (positive, negative, or null) in each outcome domain.

Table 7. Article findings for environmentally related versus non-environmentally related outcomes.

Area of focus	Positive (row %)	Null (row %)	Negative (row %)	Number of articles
Environmental	102 (94%)	43 (40%)	4 (4%)	108^a
Non-environmental	39 (95%)	15 (37%)	1 (2%)	41^a
Academic	24 (96%)	9 (36%)	1 (4%)	25^a
Civic	12 (92%)	5 (38%)	0 (0%)	13^a
Social	5 (100%)	1 (20%)	0 (0%)	5^a
Personal	3 (75%)	2 (50%)	0 (0%)	4^a
Health	3 (100%)	1 (33%)	0 (0%)	3^a

^a In each row, the sum of articles reporting each level of findings is greater than the total number of articles because researchers often reported more than one type of finding (positive, negative, or null) for each outcome type.

Discussion

Focusing on the content, structure, and outcomes of interest in 119 peer-reviewed articles related to environmental education with K-12 students and published over a 20-year span, our analysis indicates that the field of EE research is expanding across various dimensions, particularly the diversity of outcomes under study. Examining outcomes broadly facilitated consideration of numerous studies, providing a landscape perspective. Had we desired to conduct in-depth analyses on individual outcomes and/or practices, we would have narrowed our focus, as done successfully and helpfully by Leeming, Dwyer, Porter, and Cobern (1993), Stern et al. (2014), and Zelezny (1999), among others.

The number of published articles focused on measurable student outcomes in EE has increased steadily from 1994 to 2013, with 44% of our sampled articles published in the most recent five years of that period. This growth may reflect a wider trend: an increasing emphasis on measurable outcomes in environmental conservation, as well as in U.S. education, more broadly. In the United States, for example, the 2001 No Child Left Behind act (repealed in December 2015) pushed accountability measures, often with quantifiable outcomes, resulting in major influences on both the formal and informal educational sectors. Those influences were also evident—and repercussions continue to be seen—in the funding sector through a growing emphasis on strategic philanthropy, often characterized by measurable outcomes and outcome-based evaluations (e.g., Brest & Harvey, 2008). Alternatively, the growth in EE studies documented in our review may reflect an overall upswing in EE research, writ large (e.g., Marcinkowski et al., 2013; R. B. Stevenson, Brody, Dillon, & Wals, 2013). Regardless of the reason, this expansion of research within EE signals the potential utility of and need for synthetic reviews. As Rickinson and Reid suggested, “Deeper insights and more powerful conclusions may be possible through the pooling and integration of numerous studies, as opposed to the reporting of studies individually” (2016, p. 143).

A focus on EE outcomes

Our systematic search uncovered great variation among the program types highlighted in articles—from hour-long classroom programs, to residential programs, to recurring programs with informal, off-site partners; the diversity of desired outcomes reflects those varied programmatic structures and settings. Documenting certain outcomes, rather than broader categories, highlights the complexity and range of programmatic structures.

Our findings align with prior research (e.g., Ardoin et al., 2015; Ladwig, 2010; Schusler et al., 2009; West, 2015), indicating that EE operates in the educational and environmental realms and beyond. Our reviewed articles suggest a field with currency in a pluralistic, interdisciplinary paradigm, rather than one narrowly focused on a distinct conceptualization of EE (R. B. Stevenson, Wals, et al., 2013). This, perhaps, poses a challenge in characterizing EE, but also indicates a strength: it is a field diverse in form and function (Ardoin, Clark, & Wojcik, 2016).

To manage the large number of outcomes, we encountered and facilitated meaningful discussion and classified outcomes based on type (knowledge, dispositions, competencies, behavior, personal characteristics, and multi-domain), as well as area of focus (environmental, academic, social, civic, health, and personal). When we considered outcomes grouped by type, without regard to area of focus, we found a preponderance of studies with objectives related to knowledge (68%) and dispositions (61%)—perhaps not surprising given our focus on K-12 audiences in formal settings. Within that context, the emphasis is often on knowledge and dispositions, rather than competencies and behaviors, at least in terms of environmentally related concepts and issues. A potential issue related to this structure is a “doom-and-gloom” approach that might overwhelm students without concurrently providing skills and opportunities to undertake meaningful action (Busch, 2016; Chawla & Cushing, 2007).

We might also be seeing the “streetlight effect,” where researchers investigate what is easiest to observe or measure (Freedman, 2010). Researchers may be able to assess more easily knowledge and dispositions using items yielding quantifiable data (although the accuracy of such attitudinal measures may be questionable). By contrast, competencies may require more complicated, nuanced methods. Similarly, measuring changes in behaviors is notoriously difficult given the complexity of environmental behavior (Heimlich & Ardoin, 2008; Hollweg et al., 2011) and challenges related to parsing intentions versus actual behaviors (Fu, Peterson, Kannan, Shavelson, & Kurpius, 2015).

In terms of area of focus for outcomes, not surprisingly, 91% of the articles included environmentally related outcomes. Perhaps of more interest are the 9% of studies that did not measure environmentally related outcomes. A closer review reveals that 10 of those 11 articles spotlighted academic outcomes, including concepts such as systems thinking (Assaraf & Orpaz, 2010), critical thinking (Ernst & Monroe, 2004), decision-making (Gresch, Hasselhorn, Bögeholz, & Helge Gresch, 2013; Utzschneider & Pruneau, 2010), research skills (Cincera & Maskova, 2011), and science-process skills (Haney, Wang, Keil, & Zoffel, 2007). Three studies addressed academic content areas including chemistry (Bartusevica, Cedere, & Andersone, 2004) and mathematics (Haney et al., 2007); Tsevreni (2011) explored civic engagement-related outcomes. In all of those, the authors described the studied outcomes as precursors to environmental action, while also discussing the skills and attitudes as important in non-environmental contexts. As such, although we did not code those 11 studies explicitly as having environmental outcomes, closer inspection suggests an implicit environmental link.

We commonly found overlap among outcomes, although our classification system may not sufficiently reflect those overlaps. Researchers sometimes described measuring intermediary outcomes in place of, or in addition to, measuring an ultimate outcome. They might report, for example, changes in knowledge related to action skills, with the intention of assessing current or future changes in behavior. The question arose, then, as to how to categorize appropriately studies focusing on action-skills knowledge. Do such studies belong in the knowledge domain or, as elements of skills and behaviors, are they more appropriately categorized in the skills or behavioral domain? Although the outcomes may fit in all three, for simplicity's sake and grounded in the notion that knowledge is a multi-dimensional construct (Frick, Kaiser, & Wilson, 2004; Jensen, 2002), we categorized action-skills knowledge, as well as other skills-related knowledge, as dimensions of knowledge.

Our finding that researchers did not measure academic outcomes nearly as frequently as environmentally related outcomes may also reflect our focused coding approach: this emphasis on environmentally related outcomes may initially seem surprising given the K-12 emphasis of the reviewed studies. Rather than academic outcomes not being of importance or interest to EE researchers, we believe that many outcomes classified as environmentally related also may be inherently academic. We classified climate knowledge, for example, as an environmental outcome, but it may also be appropriately considered as an academic outcome in contexts where students learn science-related content through curriculum developed around state standards and addressed in standardized testing.

However, including non-academic outcomes (e.g., environmental, civic engagement, health benefits) as an emphasis of EE programs for the K-12 audiences who are the focus of this review may reflect the push to extend what students learn in school beyond traditional academic outcomes (Ladwig, 2010). The natural overlap between established EE outcomes and academic outcomes (which makes parsing outcomes difficult) may facilitate this trend of redefining what students learn in school.

Although our review primarily intended to illuminate EE-related outcomes of interest to researchers focusing on K-12 students, in the process, we collected data about the findings associated with outcomes. Compiling findings reported in the peer-reviewed literature suggests an overwhelmingly positive influence of EE on K-12 students across a range of outcomes, including those related to the environment and other domains. The lowest-reported success rate within our sample related to outcomes in studies focused on environmental behaviors. Even among those, however, 83% of studies indicated some level of positive findings.

As designed and implemented in our sample, EE appears to be highly successful at meeting knowledge and competency outcomes. Although EE programs are not quite as successful at achieving outcomes involving dispositions and behavior, as stated, findings related to those outcomes were not negative; they were simply more likely to be null. Those null findings might also reflect the actual implementation of EE programs or result from practical difficulties in measuring change in dispositions and behaviors, which are constructs inherently more complex to characterize and measure than knowledge and competencies. Researchers examining how changes in attitudes are measured have reported inconsistent use of instruments, as well as misuse of established scales (Hawcroft & Milfont, 2010; Johnson & Manoli, 2011). A more in-depth analysis of the ways in which the reviewed studies investigated changes in dispositions and behavior may help illuminate measurement effects on reported findings.

Implications for EE research

As suggested by the previous section, reviewing outcomes and findings in this way encourages reflection on the overall state of EE research. Previous authors (e.g., Hart, 2003) have called on EE researchers to critically examine perspectives on and understanding of the philosophical underpinnings of the field and, in the course of that, consider what is meaningful and what counts as evidence (R. B. Stevenson, Brody, et al., 2013). In parallel, as those questions are asked, the repertoire of related research approaches, methods, and tools that align with those outcomes of interest must also be reconsidered, broadened, and diversified (e.g., Carleton-Hug & Hug, 2010; Hart & Nolan, 1999; Stern et al., 2014; Wheeler, Thumlert, Glaser, Schoelhamer, & Bartosh, 2007). Yet, most studies in this review (78%) indicated use of quasi-experimental designs. Perhaps this is not surprising given our initial focus on measurable outcomes: researchers often consider experimental and quasi-experimental designs especially appropriate for evaluating outcomes within that paradigm. A different epistemological and ontological positioning may suggest other approaches, such as ethnography, action research, and narrative; the outcome orientation guiding this review may not highlight studies within those traditions because such research may focus more on process rather than outcome. In terms of research tools, 82% of the studies in our sample collected data using structured instruments, such as questionnaires; again, the preponderance of their use may reflect the outcome orientation.

Continued exploration of our data related to methodology and methods among the reviewed studies suggests some interest in alternative research approaches: of the reviewed studies, 22% employed designs described as case studies, comparative studies, action research, ethnographies, autoethnographies, phenomenological, correlational, and descriptive. Blatt (2013), for example, took an ethnographic approach and used participant observation and cogenerative dialogues to study changes in high school students' environmental identities and behavior as they participated in an environmental science course. Pruneau, Freiman, Barbier, and Langis (2009) pursued an action-research study with Canadian third graders who worked with scientists to address sedimentation in a local river. The researchers used observation, interviews, and questionnaires with drawings, diagrams, and descriptions to assess the intended outcomes of improved problem-solving and representation skills.

The quasi-experimental studies also evidenced varied approaches. Some highlighted researchers' desires to achieve complementary integration of qualitative and quantitative data using drawings, concept maps, word association, repertory grids, and photovoice. Authors described the structured instruments used in 82% of the studies as including open-ended items designed to elicit responses producing qualitative data rather than strictly quantitative data.

We agree with past reviews (e.g., Leeming et al., 1993; Rickinson, 2001; Stern et al., 2014; Wheeler et al., 2007) that have also noted the opportunity for delayed follow-up measures to extend understanding beyond immediate post-program reflections. Only 29% of our reviewed studies included post-programmatic follow-up, with most of those conducted less than six months after the EE experience. Only two studies (Rioux & Pasquier, 2013; Schneller, 2008) included follow-up more than a year after the EE program. One approach to exploring longer-term effects of EE may be the retrospective methods used in Significant Life Experiences studies (e.g., Chawla, 1998; Tanner, 1980; Wells & Lekies, 2006), although researchers (c.f., Chawla, 2001) recognize the challenges of this work, including difficulties of isolating effects of single and/or particular experiences over time.

Our 119-article review sample includes an emphasis (57%) on the middle grades (ages 11 through 14), perhaps suggesting that some may view those as the “golden years” for EE in terms of students' moral development (Kahn & Kellert, 2002; K. T. Stevenson, Peterson, Bondell, Mertig, & Moore, 2013). Leeming et al.'s (1993) EE outcomes review noted that few studies had been conducted and/or reported with young children (ages 8 and under); our findings suggest that this trend has continued. Although theory and research suggest that early childhood is a critical developmental stage for nature contact (c.f., Arnold, Cohen, & Warner, 2009; Cutter-Mackenzie, Edwards, Moore & Boyd, 2014), exploring programmatic outcomes with young students does not appear to be a focal area of EE research. Although some may interpret this as a literature gap, it more likely relates to the process, rather than outcome, orientation of peer-reviewed journal articles discussing early childhood EE.

Future reviews

The many outcomes described in our review speak to the versatility of EE; concurrently, they complicate the task of coalescing the diverse research bases across the field. Few areas of primary focus stand out among EE's outcomes: although we found many studies reporting a range of EE-related positive outcomes, commenting on individual outcomes is more challenging. Selecting precise outcomes of interest—such as critical thinking, self-esteem, or environmental behavior—may help make crisper statements about EE's effects, and facilitate deeper exploration to better assess existing literature.

Another potential future research avenue would be to review literature concerning fields whose outcomes align with, or closely relate to, traditional EE outcomes. Our review identified a handful of studies explicitly designed for dual purposes. In a study of Turkish secondary students learning English, Arikan (2009) shared positive findings from implementing an EE program coupled with a peace-education initiative. Students displayed increased global awareness, demonstrating potential connections between the fields. Stern, Powell, and Ardoin (2011) evaluated a middle-school-focused residential outdoor program emphasizing character development and EE. The authors discuss connections between EE and positive youth development in light of findings related to character development, leadership, and environmental responsibility. Peace education and positive youth development offer two examples, among many, of fields with EE-compatible outcomes.

Limitations and delimitations

Despite our efforts to cast a wide net, we certainly missed relevant research. To be systematic, we only reviewed articles returned in our database search. Although we included seven search engines, even those do not index every journal that publishes EE research. Our search captured only articles that met the criteria as described in the Methods section; although we attempted to be inclusive in our search terms and procedures, the decision tree certainly included subjectivity, such as deciding what counted as “empirically measured” outcomes. In the first vetting round, we only reviewed article abstracts and, in that process, we may have excluded articles meriting further analysis based on the abstract.

Similarly, including only articles published in peer-reviewed journals limited the sample. Publication bias may have influenced an overrepresentation of positive findings, and we necessarily excluded some high-quality work—notably evaluation studies, which may be available only in the grey literature. We retained the peer-reviewed focus, however, with the intention of standardizing the research process and maximizing opportunities for replication.⁴ We felt those benefits outweighed the challenges associated with attempting a grey-literature review—namely, the significant effort required to conduct a robust, systematic grey-literature search (Mahood, Eerd, & Irvin, 2014).

As noted, our sampling process surfaced many more U.S. studies than those from elsewhere. This may be because of the preponderance of U.S. researchers and programs; the U.S. emphasis on “measurable outcomes,” versus a more holistic focus in other countries and regions (P. Hart, personal communication, 2013); our English language-only search criterion; or some combination of factors. This characteristic of our sample prompts further inquiry into mechanisms for international sharing of research.

Due to the range of definitions of EE and the profusion of synonymous terms, we excluded studies that researchers or practitioners may view as EE but that authors did not explicitly label as such in their article. At no time, however, did we exclude studies in which authors labeled programs as EE and used other terms to describe the program. Those other terms included, but were not limited to, education for sustainability, sustainability education, conservation education, outdoor education, garden education, environmental science, and environmental peace education.

Finally, although we sought to include a variety of methods, this particular review included only articles with measured outcomes of EE. We recognize that this created a bias toward certain research designs and data. We, however, conceptualize EE as a holistic endeavor, and we recognize the challenges inherent in evaluating EE based solely on measured outcomes. As Ladwig (2010) notes, measuring student outcomes has not been the main focus of EE research in part because “much of the push in environmental education directly argues against applying technical rationality, input-output models of education” (p. 127). Although we empathize with this view, we also see a time and a place for outcomes research in EE, given the needs of practitioners, administrators, funders, and other EE stakeholders. We believe this review can contribute to a broader conversation and, complementary to more process-oriented research, can lend support to calls for including EE in the formal educational system.

Conclusion

Our systematic review attempts to describe and critically analyze literature related to outcomes of EE in programs designed for K-12 students. The findings suggest that many existing EE programs, which occur across a range of settings and in various configurations, have positive outcomes in terms of environmental knowledge, attitudes, dispositions, and skills for this audience. Moreover, those studies suggest that environmental education can also affect outcomes less directly focused on the environment, such as those related to academic achievement and civic engagement.

By coalescing and analyzing a number of existing empirical studies in this area, we may start to better understand what exists and, by extension, what remains to be explored—or as Reid and Scott describe, the “blind spots, blank spots, and bald spots” (2013, p. 520). The complex nature of environmental issues, coupled with the diversity of EE settings, audiences, and practices, make EE research challenging; the studies in our review sample suggest, however, that many EE researchers have risen to that challenge through pursuing innovative, rigorous research, yet, in the process of conducting our review, we did note some areas warranting further attention, such as a need for enhanced measurement with a focus on behavior and dispositions, as well as an opportunity for longitudinal studies.

Beyond a focus on outcomes, different research approaches—and, indeed, entirely new avenues of research—may more appropriately describe and characterize certain EE experiences. As we cannot solve many, perhaps even most, environmental issues in the short term, EE research perspectives and designs that look beyond immediate outputs and outcomes to consider EE across the lifespan may be more relevant. EE is most often about relationships, processes, and providing opportunities for transformative experiences, rather than a singular focus on a specific outcome; as such, an outcome-oriented research design may distort perceptions of perceived success, while missing the overall richness of experience. We, therefore, support calls for broader perspectives on what is measured and, relatedly, alternative research approaches and methods. As researchers building on and benefiting from the body of systematic reviews conducted by others, we are eager to dialogue with diverse research teams that offer insights into the past and future of EE research. Finally, we acknowledge the continued dedication of EE researchers and practitioners who strive to improve both the environment and the field of education.

Acknowledgments

We are grateful to numerous colleagues who provided feedback on drafts of this paper, especially Bora Simmons and Charlotte Clark. We appreciate the constructive comments from the journal editor and anonymous reviewers; they substantially improved this manuscript. We appreciate Alan Reid's insightful perspective on systematic reviews. We thank Rhoda Wang for research assistance and Wendi Hoover for editorial assistance.

Funding

The Pisces Foundation provided primary support for this study, with additional support from the North American Association for Environmental Education.

Notes

1. By limiting search terms to environmental education, we did not intentionally exclude studies employing related terms such as education for sustainability or conservation education. Although we did not specifically search for those alternates, the final sample includes several studies that describe the educational intervention using those terms, a result of environmental education journals publishing such articles or from those articles including environmental education within the keywords.

2. At the time of the initial search, this represented the most recent 20-year period for which peer-reviewed articles were fully available.

3. We include enough detail so that others may replicate this search, although individual results returned may vary as publishers update databases; moreover, differences in institutional access to databases can affect results.

4. We did not assume that all studies were equal in quality as we acknowledge that the peer-review process is far from perfect. We included criteria to address quality-control issues in our exclusion/inclusion process for each article identified in our search.