Using evaluation data to inform climate education practice

Abstract

A challenge for using evaluation data to improve climate change education in schools is access to an instrument for teachers to measure behavioral changes and their antecedents. We report on teachers’ implementation of a brief survey measuring climate-related beliefs, self-efficacy, intention to act, action, and anxiety before and after their program of climate change education for students (aged 11-14 years, N = 62). Results indicated improvement in self-efficacy but not in students’ intention to act. This prompted teaching staff to generate modifications aimed at fostering greater behavioral change, with staff discussion focused on students’ emotions and revisions characterized by creative, affect-driven approaches. Our paper demonstrates how data from a reproducible evaluation tool can inform teachers’ efforts to improve their practice.

Introduction

There is an established basis for considering that good evaluation data can empower teachers to be professionals through enabling them to reflect on their practice and make improvements. The use of data has been termed ‘one of the most central reform ideas in contemporary school policy and practice’ (Turner & Coburn, 2012, p114), with data-based decision-making becoming increasingly popular as means to improve instructional quality (e.g. Vanlommel et al., 2021). Data arising from sources such as teacher-created tests can be used to adjust, develop, and modify instruction in a range of ways and so improve learning (Grabarek & Kallemeyn, 2020). The idea of using data to improve practice is consistent with action research approaches that combine problem-solving actions with data-driven collaborative analysis to help understand underlying causes (Koshy, 2010). More broadly, it aligns comfortably with notions of the teacher as a reflective practitioner (Schön, 1983) who, often in conversation with other teachers, uses data to analyze the relationship between teaching and learning with the purpose of improving their teaching (Postholm, 2012).

Applying such an approach in climate change (CC) education does, however, face a unique set of challenges. There is a strong argument that classroom data regarding effectiveness in this area should include indicators of behavioral change, and these data are quite different from those traditionally used to inform teachers’ practice. UNESCO’s Action for Climate Empowerment (ACE) guidelines (UNESCO and UNFCCC, 2016) suggests learning objectives for CC education should extend beyond the acquisition of knowledge and understanding to include changes in beliefs and behavior. Indeed, teachers are now expressing a desire to plan and implement action-oriented lessons that promote individual and social behaviors likely to benefit the environment (Dunlop et al., 2022; Howard-Jones et al., 2021) and which encourage feelings of agency that lead to action rather than despair (Ojala, 2023). The research literature supports this stance, reporting CC educational interventions that are successful in generating efficacy and hope, including communicating about CC with friends and family (e.g. Zografakis et al., 2008), collaborative projects in the community (Monroe et al., 2019), and involvement in political action (e.g. Trott, 2019). Consequently, while evaluation of CC educational interventions can benefit from measures of knowledge and understanding, evaluation data relevant to climate action may be considered increasingly relevant. This type of data collection is encountered more frequently in the realm of research than the classroom, involving methods that can be elaborate and time-consuming. To be able to evaluate their classroom approach in these terms, teachers require access to a valid and convenient instrument.

Which indicators might support teachers’ practice in CC education?

If a key objective of CC education is action, indicators of both behavioral change and its antecedents are potentially useful for guiding teachers’ practice. One such antecedent is climate belief about, for example, whether humans can influence climate. Educational knowledge and understanding can influence the climate belief of young people (e.g. Jurek et al., 2022) and, in turn, belief in the effectiveness of climate action can predict secondary students’ willingness to undertake it (Skamp et al., 2013).

Belief, however, is far from sufficient for ensuring action. For example, Skamp et al. (2013) showed that willingness amongst the students in their study lagged behind beliefs. Even when a student believes in the need for action, their willingness to act may depend on their self-efficacy, i.e. their belief they can act effectively. Social cognitive theory suggests self-efficacy operates as a central self-regulatory mechanism of agency (Bandura, 1998) and strongly predicts human behavior (Bandura, 1997). Environmental self-efficacy has been shown to lead to positive behavioral intention and action in areas of recycling (Tabernero & Hernandez, 2011), saving water (Lam, 2006), and greater general pro-environmental behaviors (Abraham et al., 2015; Huang, 2016).

While self-efficacy is a strong predictor of action, the intention of an individual may not follow. Mediating factors include other types of efficacy belief beyond the individual. For example, Choi and Hart (2021) demonstrated that beliefs regarding the effectiveness of collective action influenced an individual’s likelihood of acting to address climate change.

Even when intention has been expressed, climate action may not occur. Studies of academic procrastination suggest contributory factors which may also prove relevant to environmental behavior, including lower self-esteem (Yang et al., 2023), poorer self-regulation (Wolters, 2003), and lower conscientiousness (Wieland et al., 2022).

Finally, given increased attention to the impact of climate change on young people’s mental health (Baker et al., 2021), it would appear prudent for teachers to have access to an indicator of anxiety related to CC when evaluating student outcomes. The fostering of negative emotions can be a temporary and positive outcome of climate change education, since increased concern can help prompt action (Bright & Eames, 2022; Stevenson et al., 2018) and action can help dissipate climate anxiety (Trott, 2022). However, some approaches to climate education might, plausibly, raise concern without empowering students to act, thereby leading to anxiety that is unaddressed. Therefore, monitoring anxiety during a CC program may help safeguard well-being while also providing insight into emotional aspects of climate pedagogy that are particularly challenging for practitioners (Bryan, 2020).

The aim of this study was to investigate the validity and utility of a brief student survey intended for use by teachers to provide insight into their CC education practice. Research instruments have been developed to measure the efficacy of interventions in relation to the above variables but, even when designed for children, these can be lengthy and time consuming for routine classroom use. In consultation with teachers, we developed a survey for secondary school students that measured self-reported behavior and a range of putative antecedents using no more than 2 types of rating scales, fitted on a single side of A4 paper and required no more than 5-10 min to complete. We quantitatively examined the validity of the survey for collecting and monitoring change in the variables amongst a sample of secondary students who pursued a brief program of CC education. Following the program, we qualitatively analyzed discussions involving teaching staff that were informed by survey outcomes to explore the potential of the survey to support data-driven reflective practice in CC education.

Materials and methods

Research design

Quantitative evaluation data were collected by three teachers (including the Head of Department) administering a self-report survey to their students before and after attending a unit of work on CC education within the school’s geography curriculum. The survey included measures of belief, self-efficacy, action, intended action, and anxiety related to climate change.

The program took place in April-May of 2022 in the summer term, with reflection planned for after the summer break. Due to staff turnover, two reflective meetings took place. Since one of the original teachers had left the school, the researcher met with the head of department and the remaining staff member to discuss a summary analysis of the results and their implications for practice. A further meeting then took place between the researcher, head of department and the new member of staff, who was presented with the existing work scheme, the summary analysis of the results and points arising from the previous meeting. These meetings were recorded, transcribed, and subjected to a process of meaning condensation (Malterud, 2012). Specifically, the entire transcription was first read to get a general sense of the whole discussion. Then the text was read again with the specific aim of discriminating ‘meaning units,’ with a focus on how the teaching, impact, and improvement of the unit was being experienced. With meaning units delineated, these were reexamined to identify and synthesize insights more directly, particularly units most revelatory of how the teachers were generating evidence-informed ideas for developing their approach. The quantitative results focused the discussion on how greater intention to act and greater action might be fostered. In other words, the data were used to inspire reflective questioning and hypotheses about instructional practice, which were also informed by pieces of qualitative data including informal, undocumented data arising from teachers’ conversations and observations (Ho, 2022).

The research received ethical approval according to the procedures of the University of Bristol Research Committee (application 009473).

Participants

The participating school was recruited into the study through a call from the UK’s Climate Education Research Network for interested teachers in the South-West of the UK planning to deliver climate change education. The participating cohort of students (N = 108) were 6 classes due to attend the unit as a part of a rolling program that repeated every 3 years, taught by three members of staff. It comprised three classes in Year 9 (aged 13-14 years), one class in Year 8 (aged 12-13 years) and two classes in Year 7 (aged 11-12 years).

Survey instrument

The survey instrument asked for respondents’ agreement to two lists of statements (See Appendix), the first of which employed a 5-point Likert scale. Five belief statements were included that had been validated amongst a sample of teachers in previous work (Howard-Jones et al., 2021) and originally adapted from an NPR/IPSOS poll (NPR/IPSOS, 2019). Three self-efficacy statements were included. Two of these were used by Flora et al. (2014) to assess climate efficacy amongst US high school students. Since one of these statements (and other statements encountered later in the survey) used the phrase ‘carbon footprint,’ an additional statement was included that assessed confidence in explaining this phrase to others. We used the same time references as Flora et al. (2014) to measure self-reported actions undertaken (focusing on what the students did yesterday - to prioritize recent memory) and future intended actions (what the student would be doing in the next three months) but incorporated these in statements suitable for eliciting a level of agreement. We focused on three levels of behavior for actions undertaken and intended actions: individual, with students/friends, and with parents. At the latter two levels, we also chose to differentiate between ‘talking’ and ‘doing’ as undertaken actions, resulting in five action statements overall.

Climate-related anxiety was measured using a form of the Climate Change Worry Scale (Stewart, 2021) that included minor adaptations reflecting the respondents’ age. Specifically, we simplified the statement ‘I feel paralyzed in being able to do anything about it’ to ‘I feel unable to do anything about it.’ We also changed looking for information about CC ‘in the media (e.g. TV, newspapers, internet)’ to ‘in the media (e.g. TV, news, internet),’ given the declining interest in reading newspapers amongst young people (Twenge et al., 2019).

Program of study

The unit began by students building mind maps in pairs to identify their current knowledge about climate change. These maps supported teachers’ explanations of the science of climate change, augmented by videos that also addressed common misconceptions. Students then undertook exercises that included generating their own written explanations of key concepts and arguments about whether all nations shared equal responsibility, supported by world maps showing carbon emissions by nation and per capita. Students were introduced to the broad global effects of CC through an exercise coding impacts in terms of social, economic, and environmental. Students then considered local impacts using an online tool (uk-cri.org) to list changes in climate, transport, agriculture wildfires, and water availability in their county and beyond over the next century, before discussing as a class who would be most affected in the UK and how. Students then calculated their own carbon footprint (using an online tool: footprint.wwf.org.uk) before considering how they might reduce it. Using text-based resources and video, students explored and evaluated how others were addressing climate change, including a range of adaptation, mitigation, and geo-engineering strategies. Finally, the students chose at least one idea that could help their school mitigate CC and wrote a letter to their headteacher promoting its implementation.

Results

Survey data

Survey data from 40 participants were omitted from the analysis due to a lack of either pre-survey or post-survey data related to student absence. The large extent of student absence may be attributed to rising Covid-19 cases during the period of the research. Another three students declined to have their data entered into published research and three students appeared to misunderstand the instructions for completing the survey (e.g. indicating multiple responses for the same question). This resulted in a final sample size of N = 62 for analysis.

Responses to statements within each category were summed to provide a single score for each participant of belief, self-efficacy, intention, action, and anxiety pre- and post-program. Means and standard deviations of these scores are provided in Table 1.

Table 1. Mean scores (with standard deviations in parentheses) recorded before and after the program for climate-related belief, self-efficacy, intention to act, action, and anxiety. Test statistics derived from two-tailed Wilcoxon signed-rank tests.

		Climate-related belief (out of 25)	Self-efficacy (out of 15)	Intention over next 3 Months (out of 15)	Action taken yesterday (out of 25)	Anxiety about climate change (out of 50)
Descriptive Statistics	Mean (SD) before	21.2(2.3)	10.7(2.2)	10.8(2.5)	13.3(4.4)	27.6(8.9)
	Mean (SD) after	21.3(2.3)	11.50(2.2)	10.9(2.2)	14.4(4.2)	26.7(10.4)
	% Change	+0.44	+5.20	+1.27	+4.12	−1.84
Test Statistics	Z	−0.287	−2.977	−0.139	−1.229	−1.613
	p	0.774	0.003*	0.890	0.219	0.107

* = statistically significant (threshold for significance set at 0.01 for multiple tests).

Pre and post components of responses were pooled before McDonald’s Omega was calculated as an indicator of reliability for belief (0.620), self-efficacy (0.770), intention (0.870), action (0.830), and anxiety (0.936). Note that belief falls short of a criterion of 0.70 for acceptable reliability (Nunnally, 1978).

Two-tailed Wilcoxon signed-rank tests (with the threshold for significance set at 0.01 to correct for multiple comparisons) were applied to detect statistical significance between pre and post scores. Only self-efficacy showed a statistically significant change (Z = −2.977, p = 0.003) confirming an overall increase related to experiencing the program of study.

Discussion and reflection on survey outcomes with practitioners

With articles about rising eco-anxiety in the popular press (e.g. Gregory, 2021), staff were relieved to see no increase in mean anxiety levels and that the difference in means ran in the opposite direction. The minimal increase in CC belief mean score was not considered to be of great interest since scores were already high. The increase in self-efficacy was welcomed, but it was noted that the intention to act had changed little. At first sight, this appeared at odds with that fact that the mean value for action was a few percentage points higher than prior to the program (although not reaching statistical significance). However, with no similar increase in intention over the next three months, this was attributed to residual project tasks that students were belatedly completing. Emotional engagement was reflected upon as key in prompting students to act, with discussion of whether the low anxiety levels might also indicate a lack of concern and so help explain the lack of intention to act. In their study of adults who were sent video messages about climate risks and climate actions, Angill-Williams and Davis (2022) also observed increased climate efficacy without increased intention, explaining this in terms of increased efficacy reducing the perceived urgency of action.

In this way, the discussion turned to the issue of why the program did not have a greater impact on action and intention to act, and how it might be modified to do so. It was noted that the program had chiefly delivered knowledge, understanding, and skills, and did not include the type of school and community projects associated in the literature with increased environmental intentions and behavior (Monroe et al., 2019). Neither did it include highly engaging elements that have characterized some other interventions prompting behavioral change, such as simulation games (Dresner, 1990) and ‘edutainment’ (Flora et al., 2014). However, it did include students using a carbon footprint calculator to reflect on their lifestyle and how they might change it, which has featured in a UK intervention during a religious event that increased adults’ desire to take action (Chauhan et al., 2009).

Staff members identified several potential barriers to improving the program, their own solutions to these barriers, and a commitment to implement these. Discussion of staff concerns included reference to government guidance on impartiality and their Prevent Duty¹, uncertainty about what was now considered as a political group and ‘walking into that dodgy territory’ where, accidentally or otherwise, pupils might be prompted to access websites considered by some as extremist (defined by the UK government as ‘vocal or active opposition to fundamental British values’ (HM Government, 2011)).

The discussion about improving the program was informed by the researcher’s reference to some general themes in the CC education research literature: the relationship amongst adolescent students between greater action and increased levels of hope and concern (Stevenson et al., 2018), the opportunities for CC education offered by working across the curriculum (Hawkey et al., 2019), and the motivational value of contexts where students perceive some control over observable outcomes (Gifford, 2011). The researcher also referred to some specific features of CC education implicated in effectiveness, including deliberative discussion, links to personal and local experience, and interaction with scientists and those experiencing CC impacts (Monroe et al., 2019).

Deliberative discussion was considered useful in raising the emotional engagement of students, but with the potential risk of further entrenching perspectives. Staff decided discussion of non-binary questions would reduce divisiveness, e.g. how much meat should be on the school menu rather than whether it should be entirely banned. Staff members also generated two complementary strategies for drawing on students’ personal motivations. First, they would ask students to identify those aspects of CC they found most concerning and to research and present them. This would provide a student-centered and motivating introduction that would also inform later activities, allowing these to be focused on students’ interests. Later, students would develop and present ideas about actions in these areas they were willing to undertake as individuals, leading to a vote that would identify groups of students who might want to pursue these actions collaboratively. Encouraging communication between these groups and the school’s eco-committee would also help translate these ideas into action at a school level. Reflection on research suggesting the positive impact of relating CC to local impacts and personal testimony prompted the idea of incorporating stories from residents on local flood plains who had experienced disastrous flooding in recent years. Interaction with local businesses about how and why they were changing their practices, products, and services for both environmental and economic reasons might also help students reflect on their own role as critical green consumers.

The teachers considered including collaboration with other subject areas, although practical concerns were raised about synchronization/coordination and additional meetings. However, a previous collaboration with the English department had worked well, and research on the value of affective foresight for predicting action (Brosch et al., 2018) prompted the proposal of an imaginative writing exercise on futures impacted by climate change, with students asked to vote for those they considered most likely.

Discussion

Active participation is considered a significant factor in the effectiveness of teacher professional development (Desimone, 2009; Kennedy, 2016), and we found the collection and interpretation of classroom data prompted a productive reflection by teaching staff on how the program might be improved. The approach here was one that anticipated teachers’ decisions arising from deliberate analysis of data but also from intuitive recognition (Vanlommel et al., 2021). For example, an attempt was made to avoid signaling any expectations that decisions required explicit evidence-based justification. Instead, the data were used as a prompt for a discussion that drew in moral arguments, details of teaching contexts, informally collected qualitative evidence, and anecdotes. Importantly, however, the data helped focus the conversation on students’ emotional experience and motivations. This conversation generated additional ideas for identifying motivations through relevant tasks and for allowing these to guide student activities toward more self-determined goals and actions. More generally, the discussion suggested a shift in teachers’ thinking toward the more participatory, interdisciplinary, creative, and affect-driven approaches identified by CC educational researchers as effective in developing attitudes and behavior (Rousell & Cutter-Mackenzie-Knowles, 2020).

Our results suggest the collection and interpretation of quantitative data by teachers for improving CC pedagogy is feasible, but some additional resources may be helpful for teachers attempting to do this independently. The brevity and simplicity of the survey facilitated implementation by teachers before and after the climate change unit, but the reflection that followed involved the researcher analyzing data and drawing attention to the literature. It is difficult to assess how critical this input was, but it has been noted that teachers do not always receive the support they need to use data optimally (Datnow & Hubbard, 2015; Hebbecker et al., 2022). Also, in contrast to the array of CC teaching resources available online, less guidance is available about what constitutes effective CC pedagogy. Provision of accessible resources for understanding factors contributing to the effectiveness of climate change education would, therefore, help teachers interpret evaluation data independently and plan suitable revisions. Although the current study involved a researcher carrying out the statistical analysis, this is not an insurmountable hurdle for teachers to do independently (e.g. Churches & Dommett, 2016) and data collection could be supported by training and/or use of online survey tools (e.g. Qualtrics) and a tailored Google or Excel sheet. Furthermore, the value of statistical testing to practitioners has been challenged (Daniel & De Bruyckere, 2021) and mean values alone may have been equally effective as a prompt for reflection and improvement.

In contrast to previous research measuring climate beliefs of UK teachers (Howard-Jones et al., 2021), Macdonald’s Omega fell short of the acceptability criterion. This suggests the component statements did not sufficiently align to provide a measure of a single unitary concept of climate belief. In other words, children may hold a more divergent and differentiated set of beliefs than adults about climate change, possibly reflecting their level of learning and development. Echoing results of a larger UK study (Etherington, 2022), beliefs were already high amongst the sample at the beginning of the program, so this may not have unduly impacted the quality of reflection arising from the data. It does, however, suggest caution when applying the survey in regions of the world where belief in anthropogenic CC may be lower.

There were no data collected in the current study for knowledge and understanding, which would need to be assessed in a content-specific manner, using the types of methods commonly embedded in school routines and culture. Instead, the study focused on measures of behavioral change and its antecedents, which are more amenable to assessment in a manner that is not content-specific. We recognize options for changing and possibly improving the survey as means to collect such information. For example, collective efficacy can be at least as significant a predictor of pro-environmental behavior as self-efficacy (Chen, 2015). Although belief in collective efficacy might contribute to ratings of the belief statement ‘We could act to slow climate change’ and ‘We could act to lessen the effects of climate change,’ a statement capturing confidence in small groups to bring about more observable and immediate change (e.g. in local policy) might be a useful inclusion. The window of time during which students were asked to retrospectively report their actions might be extended from ‘yesterday.’ This could potentially increase the sensitivity of the survey and capture more infrequent actions, although an important consideration here is that memory can fall off abruptly after about a week (Fisher & Radvansky, 2018). We also chose to largely retain the original wording of statements validated by Flora et al. (2014) but these might be modified further to align statements across variables. For example, items intended to measure behavior might resemble more closely those intended to measure self-efficacy by referring to ‘reducing carbon footprint’ rather than ‘doing something about climate change.’ Similarly, for the sake of validity, we chose to retain all ten statements (with small modifications) of the Climate Change Worry Scale (Stewart, 2021). However, this might be balanced in future by statements measuring hope, which is another emotional component contributing to pro-environmental change of behavior in adolescence (Stevenson et al., 2018). There are also other limitations of our survey in a theoretical sense. The survey allows practitioner discussion to be informed by some antecedents of behavioral change that are less visible than the behavior itself, but nonetheless important. However, the processes by which such change occurs can be complex and involve many other variables. For example, social psychology points to the contribution of social norms and attitudes to behavioral intentions (Ajzen, 1991), with social cognitive models emphasizing the contribution of implicit attitudes and implicit motivation to mediating the relationship between intentions and behavior (More & Phillips, 2022).

Staff turnover prevented the gathering in a single meeting of all staff who had taught the program and would be teaching the revised version the following year. Although losing the experience and reflections of the outgoing staff member was unfortunate, staff retention is currently a common problem in UK schools (Strods et al., 2022). In our study, it drew attention to the potential value of evaluation data for inducting new teaching staff onto a program, helping them to contribute their own evidence-informed suggestions for revision.

Finally, it should be noted that our study was focused on the use of survey data for stimulating and informing teacher reflection. Accordingly, although we sought and found evidence of data-informed reflections and intentions, it is not possible to report on whether or how intentions were enacted. Mindful of the intention-action gap discussed above, future research might valuably address such questions.

Despite these limitations and outstanding questions, we have demonstrated how teachers might conveniently collect and use data when proposing research-informed improvements to CC programs whose aims include climate action.

Disclosure statement

The authors report there are no competing interests to declare.

Notes

1 The UK’s Counter-Terrorism and Security Act 2015 requires specified authorities, including schools, to “have due regard to the need to prevent people from being drawn into terrorism”. This is known as the ‘Prevent duty'.

Appendix.

Survey (with codes in brackets referring to the variable contributed to: Belief (B); self-efficacy (S); intention to act (I); action (A) and anxiety)

Date of Birth: Day___ Month ________ Year ______

Write X here if you do NOT want your responses used by researchers at the University ___

Table

1. How strongly do you agree with these statements?	Strongly Disagree	Disagree	Do not know	Agree	Strongly agree
Climate change is being caused by humans (B)
The world’s climate is changing (B)
We could act to slow climate change (B)
We could act to lessen the effects of climate change (B)
Weather-related incidents are becoming more severe (B)
I am confident I could explain what a carbon footprint is to others (S)
I am confident I could explain global warming to others (S)
I am confident I could help start a project to help others reduce their carbon footprint(S)
In the next 3 months:
I will take action to reduce my own carbon footprint (I)
I will encourage other students/friends to reduce their carbon footprints (I)
I will encourage my parents to reduce their carbon footprints (I)
Yesterday:
I did something about climate change on my own (A)
I talked to other students/friends about climate change (A)
I did something about climate change with other students/friends (A)
I talked to my parents about climate change (A)
I did something about climate change with my parents (A)
2. How often do you do the following? (Anxiety)	Never	Rarely	Sometimes	Often	Almost always
I worry about climate change more than other people
Thoughts about climate change make me worry about what the future may hold (X)
I look for information about climate change in the media (e.g. TV, news, internet)
I worry when I hear about climate change, even if its effects are some time away
I worry that outbreaks of severe weather may be the result of a changing climate
I worry about climate change so much that I feel unable to do anything about it
I worry that I might not be able to cope with climate change
I notice that I have been worrying about climate change
Once I begin to worry about climate change, I find it difficult to stop
I worry about how climate change may affect the people I care about