The impact of development education and education for sustainable development interventions: a synthesis of the research

Abstract

The Sustainable Development Goals set out by the United Nations advocate that all learners will have the knowledge and skills needed to promote sustainable development. Development education, education for sustainable development and global citizenship education are deliberate educational interventions, which all address global justice and sustainability issues. Current research continues to expand our understanding of the field, but there are no recent reviews of the literature addressing the impact of these educational interventions. The objective of this paper is, therefore, to examine the impact of intentional development education, education for sustainable development and global citizenship education. We reviewed 243 abstracts against specific inclusion criteria: 127 from Scopus, 101 from ERIC, and 15 from EBSCO. Of those abstracts, 99 met inclusion criteria and underwent double review, which excluded further studies. Of the final 44 papers included for review, 26 focused on education for sustainable development or environmental educational themes, 12 were global orientated in content, either through development education or global citizenship, and six were intercultural educational interventions. In this paper, we provide an overview of measures of assessment of learning used, review the evidence of the impact on learners, and address some methodological and pedagogical questions arising from the review.

Keywords:

• Research Synthesis
• impact
• development education
• education for sustainable development
• global citizenship education

Introduction

In recent years, the global context of education has brought a new focus to education policy and practice. This ‘global-character’ of contemporary education has become evident in educational policy and discourse, as well as in the practice of teaching development education, and education for sustainable development. The reform processes within education and public spending demand increased transparency regarding accountability, efficiency and measurement. This is reflected in the proliferation of standardised testing programmes such as the OECD’s Programme for International Student Assessment (PISA) and the adoption of international literacy and numeracy testing initiatives which offer comparative scores. The global education reform movement is eager to install a ‘new basis for ethical decision-making … erected by the “incentives” of performance’ (Ball 2003, 218).

Alongside these policy changes, educational interventions for global citizenship take place, thus building skills of analysis and understanding, empathy and efficacy, as well as promoting sustainability and justice. Increased interest in global citizenship and development education has come about as a result of a number of factors, such as, for example, the increasing multicultural nature of societies and the work of international development organisations (Baily, O’Flaherty, and Hogan 2017; O’Flaherty et al. 2017). Greater importance has been placed on highlighting the inequalities that exist in the world and the role we all play in causing or preventing such inequalities (Liddy and Parker Jenkins 2013; McMorrow 2006). The Sustainable Development Goals decided by the United Nations include a goal centred on learners gaining the necessary knowledge and skills to promote sustainable development (UNESCO 2015). Yet, in a policy environment, how is global citizenship and development education work measured and assessed to justify public spending? Can measures of learning and impact adequately account for enhanced levels of civic engagement and social efficacy? How is activism for social and economic change included in performance measures and studies of impact? And do measures of impact gather all learning outcomes? In this paper, we address how these deliberate educational interventions measure and account for their impact, which is demanded in a managerial policy environment.

This paper presents a synthesis of the literature pertaining to the question of the ‘impact’ of deliberate development education ‘interventions’, guided by the following research question: What is the impact of intentional development education interventions? The remainder of the paper is set out as follows; Development Education, Education for Sustainable Development and Global Citizenship Education (hereafter abbreviated to DE/ESD/GCED) are introduced and discussed from thes perspective of policy and practice. Some pertinent information regarding the particular research context, the Republic of Ireland, is also offered. Thereafter, the methodology used to frame the synthesis of the literature is described. Findings are presented in three sections: Forms of Learning Assessment /Assessment of Impact; Education Content; and Intervention Outcomes. Finally, some interpretations of these findings are explored from a local and international perspective. Cognizant of the need for high quality evidence of learning, this research synthesis will provide an up-to-date, comprehensive compilation and review of the research regarding the measurement of output/indicators of learning arising from forms of education which aim to enhance learners’ understanding of the world. It is particularly of relevance in terms of progressing our understanding of DE, ESD, and global citizenship, asking some critical methodological, epistemological and pedagogical questions for policy and practice in these areas.

Development education, education for sustainable development and global citizenship education

More than a century ago, Durkheim (1885, 445) declared that the ‘aim of public education is not ‘a matter of training workers for the factory or accountants for the warehouse but citizens for society’. From a US perspective, Feinberg (2006, xi) draws attention to the ‘shared moral understandings required to sustain and reproduce liberal, pluralist democracies’. Noddings (1997, 27) proposes that a ‘morally defensible aim for education … should be to encourage the growth of competent, caring, loving, and lovable people’ and Cochran-Smith (1999, 116) identifies ‘social responsibility, social change, and social justice’ as key goals of education’.

International policy developments which aim to support these goals of education include the UN Decade of Education for Sustainable Development (UNESCO 2012) and the UN Sustainable Development Goals, which, in goal 4.7 stipulates that by 2030 we must ensure that all learners,

… acquire knowledge and skills needed to promote sustainable development, including among others through education for sustainable development and sustainable lifestyles, human rights, gender equality, promotion of a culture of peace and non-violence, global citizenship, and appreciation of cultural diversity and of culture’s contribution to sustainable development. (UN 2015, 19)

The inclusion of global development topics in education is formally termed development education. It aims to highlight the inequalities and injustices present across our globe, and to advocate action for global social justice. Development education is an educational process that increases awareness and understanding of a rapidly changing, interdependent and unequal world (Irish Aid 2006), while education for sustainable development centres on a new vision of education which empowers learners to assume responsibility for creating and enjoying a sustainable future (UNESCO 2002). These definitions highlight key elements in this work, namely building learners’ knowledge and awareness of global issues; critical thinking and analytical skills; and action for positive social and political change. In recent years there has been a move towards the term global as it seems to be a more relevant and accessible terminology (Bourn 2014a; Bourn 2014b). UNESCO describes global citizenship education in similar terms to development education, reflecting the active role of learners to ‘face and resolve global challenges and ultimately to become proactive contributors to a more just, peaceful, tolerant, inclusive, secure and sustainable world’ (2014, 15).

Whilst each of these educations share some common ground, there are differences in their origins and history, their theoretical basis and pedagogies, and their implementation and adoption into education systems. This paper does not set out to elucidate these differences, neither are the authors trying to minimise them; however, for the purposes of this systematic review, these educations have been brought together to address the question of impact on learning. The commonality, we believe, lies in the inclusion of global themes in content and in teaching approaches. Bourn (2014a, 21, 22) describes pedagogy for global social justice based on four main elements: a sense of global outlook; recognition of power and inequality in the world; a belief in social justice and equity; and a commitment to reflection and dialogue. These educational interventions aim to develop critical awareness of the complexity of global challenges such as poverty, injustice and unsustainability. They engage learners in considering different perspectives, questioning views and biases, and in reflecting on their own roles in perpetuating an unbalanced world. Central to these educations is developing solution-oriented skills such as critical and creative thinking, decision-making and empowerment which are viewed as essential for the sustainable future for the planet (McCloskey 2016).

Yet the learning outcomes from development education cannot be predefined which raises difficulties for assessment of impact. Bourn (2014a) argues that learners engage in debate on development and poverty to deepen their understanding of historical, cultural and social systems in order to address these topics from social justice perspectives. Some commentators critique development education in particular for losing its ‘original radical underpinnings’ (Bryan 2011, 2), and becoming soft rather than critically focused (Andreotti 2006). Others question the potential for learners in a privileged position to develop ‘the knowledge, lived experiences and perspective consciousness’ (Merryfield 2000, 241), while Jefferess is critical of an individual-centred focus to global citizenship, that does not develop any empathy or solidarity with global communities, but ‘reframes humanitarianism and global citizenship education in the terms of the self-help industry’ (2012, 18).

Surely the ultimate impact of these deliberate education interventions would be a just, peaceful and sustainable world, and as this has not been achieved, the question remains as to where do these deliberate education interventions specify their positive impact? Is it through the acquisition of solution-oriented skills and empowerment? Is it through the actions arising from learning in the creation of a just world? Or does it lie in content through measures of awareness and understanding global issues? As development education is conceptualised in terms of the inclusion of global development topics in education, reflective of a particular educative focus, it is important to problematise the notion of ‘impact of these educative experiences’. The use of the term ‘impact’, in a traditional research sense, aligns with ideals of measurement and evidence to support the impact or effect of a particular treatment with a particular group. Within the social sciences, when conducting research to determine positive outcomes as a result of an intervention, randomised experiments are considered best practice. The authors are cognisant of the need for high quality evidence of what works, particularly in light of policy changes towards evidence-based approaches in education. However, due to the distinctive and varying epistemological understandings of DE/ESD/GCED educative interventions, with their focus on process as well as product indicators, for the purposes of this paper, ‘impact’ has been conceptualised in a much broader way: as change in knowledge, skills, attitudes, ethics, actions arising, including both hard and soft measurement outputs, from exams and knowledge tests through to ethical/values measures.

Some relevant Irish context

Similar to other nations, Irish education policy is critiqued as aligned with economic objectives (Hannan 1991) and increasingly influenced by market values and neoliberal thinking (Lingard 2010; Lynch, Grummel, and Devine 2012). The model of accountability, enshrined in the Irish Department of Education and Skills (DES) strategy statements, is viewed as contractual rather than responsive, and predicated on performance rather than process indicators (Gleeson and O’Donnabháin 2009). These strategy statements have been heavily influenced by the neoliberal Lisbon Agenda, which aimed to make the European Union the most successful and competitive economy in the world by 2010, with a focus on human rather than social capital development. In summary, educational discourse in the Republic of Ireland has become increasingly ‘coterminous with the theme of education and the economy’ to the exclusion of civic competence, social and emotional learning and moral development (Corcoran and O’Flaherty 2016, 2017; Leahy, O’Flaherty, and Hearne 2017; O’Flaherty and McGarr 2014; O’Flaherty and Gleeson 2017).

Set against this context, however, there have been a number of educational initiatives in Ireland aimed at incorporating a greater sense of social responsibility and environmental protection in the formal education sector - each with their own history and rationale. Development Education has received prominence due to Irish Aid’s¹ commitment to funding Development Education projects since the 1980s (Irish Irish Aid 2006; Fiedler, Bryan, and Bracken 2011). A review of development education work highlights ‘the integration and acceptance of development education into the mainstream … as a major strength’ in Ireland (Fiedler, Bryan, and Bracken 2011; 49). Development education supports the learner to explore complex, interdependent and inter-related issues such as poverty, inequality, production and consumption, climate change, population growth, migration, homelessness, sustainability, conflict and human rights. (Baily, O’Flaherty, and Hogan 2017; Ubuntu Network 2017). Examples of such curriculum initiatives include, Civic, Social and Political Education (CSPE) introduced as a mandatory examination subject at lower secondary level in Ireland in 1997; Politics and Society launched February 1st 2016 as an optional examination subject in upper secondary level; Intercultural Education Guidelines for Schools (NCCA 2004, 2006). These policy and curriculum shifts have been influenced by factors such as increasing religious and ethnic diversity (O’Flaherty et al. 2017), the work of Irish NGOs, the history of Irish emigration and the multiplicity of cultural ties and political relationships Ireland shares (NCCA 2005). The legacy of educational policy witnessed across the last three decades continues to evolve within the present context, for example, with the launch of the National Strategy on Education for Sustainable Development (DES (Department of Education and Skills) 2011) and more recently the new Development Education Strategy (Irish Aid 2017).

Methods

Given the local and international contexts, a systematic review of the literature was conducted ‘given its strength as a means of establishing a reliable evidence base’ (Davies et al. 2012, 81). The purposes of the qualitative synthesis is to describe the nature of the evidence in the literature, and interpret the possible effect of convergence and divergence among studies. Selecting literature employing systematic procedures using specified criteria reduces the risk of ‘selective’, ‘biased’ or ‘partial’ accounts, accusations, which are frequently levelled at conventional literature reviews (Andrews 2005, p. 404). The current review followed established guidelines (Oxman 1994), employed techniques proposed by Glass, McGaw, and Smith (1981), Lipsey and Wilson (2001). The purpose, therefore, of this qualitative synthesis is to describe the nature of the evidence in the literature. A literature search was performed and no systematic review that summarised and synthesised research on the impact [on learning] of development education interventions was identified.

Emerging research question

The aim of this study was to complete a critical review of the literature pertaining to the question of the ‘impact’ of deliberate development education ‘interventions’. The following research question guided the review of the literature: What is the impact of intentional development education (and ESD/Global Citizenship Education) interventions?

Literature search strategy

The authors conducted a comprehensive initial search to locate all studies that explored the impact of intentional development education (and ESD/Global Citizenship Education) interventions. Electronic searches were made of educational databases (namely ERIC, EBSCO, Scopus – Social Sciences and Humanities only), web-based repositories, and recent tables of contents of key journals. The key search terms used were ‘development education’, ‘education for sustainable development’, ‘global education’, ‘global citizenship’, ‘world studies’. After initial screening (n = 243), two reviewers identified 99 for further scrutiny in full text. These sources were retrieved, read in full and subjected to further screening using inclusion/exclusion criteria, 44 were found to address the research questions. Published papers in academic journals were sought, rather than conference papers, and all studies had to be written in English (but could have taken place in any country). The authors acknowledge the limitations of their language skills, which may have excluded studies from other contexts and may endorse a Western conceptualisation of development, development education and sustainable development.

Selection criteria

The inclusion criteria required that studies focused on assessment of impact of DE/ESD/GCED interventions. In order to be included in this review, studies had to meet the following inclusion criteria: literature published between 2000 and 2014; studies published in refereed journals; focus on students or young people as learners; all disciplines and subject areas were included; studies must present clear measures of impact on learning, rather than being general studies of attitudes; and papers had to give an account of an educational intervention. Studies were excluded if they if they were not published in English; cases studies of change without any form of impact [on learning] assessment; curriculum development initiatives or curriculum audits; policy development or review; descriptive papers including conceptual and theoretical discussions; and attitudinal studies. Technical reports, dissertations, conference proceedings, book chapters or unpublished evaluations, were excluded as the search was limited to academic journal databases. Following application of this inclusion criteria 44 studies qualified for inclusion.

Results

The findings of this study are presented in three sections; Forms of Learning Assessment/Assessment of Impact; Education Content; and Intervention Outcomes. Of the 44 studies included, 26 focused on education for sustainable development or environmental education themes,² 12 were global citizenship or development-orientated in content; and six described intercultural education interventions.

A variety of age and education levels are presented in the final review. Twelve studies reported the results of interventions conducted within higher education, all of which were with undergraduate students across various disciplines including science, health and engineering. A further 11 studies reported findings from interventions completed with students enrolled in pre-service teacher education programmes. Second level students were also represented in the final review, with five studies reporting across middle and high school levels. A further ten studies reported findings pertaining to elementary/primary level respondents whilst Bautista Garcia-Vera (2012) reported findings from both a pre-school (kindergarten) and primary (elementary) setting. Of the remaining studies, one reported results of an intervention undertaken with ‘office staff’ (Rehm 2009) and one was undertaken with volunteers engaged in community work (Ollif 2001) with no age details given in either paper.

Forms of learning assessment /assessment of impact

Many of the papers reviewed utilised formal assessment procedures and data collection tools to measure ‘impact on learning’. The dominant form of data collection employed a pre-post survey (12) or questionnaire (8) which allowed comparison between learners’ knowledge, attitudes and understanding before and after the educational intervention.

Of the 20 papers employing a survey/questionnaire for assessment of/for learning, five employed a secondary form of assessment including interviews with learners, a focus group with learners or analysis of classroom interactions as complimentary data collection (see Table 1 for an overview of studies included). Twelve papers reported analysis of the work of students completed during and after the education intervention. These papers included descriptions of formal assessments of learning (analysis of student exams, homework assignments and classroom talk), student assignments plus learning reflection; drawings and diaries completed by students, student coursework, and student feedback alongside coursework.

Table 1. Research method – assessment of/for learning.

Research approach/design	Data collection tools	Examples
Quasi-experimental approach	Pre-post survey/questionnaire	Olsson, Gericke, and Chang Rundgren (2015), Roesch, Nerb, and Riess (2015), Burmeister and Eilks (2012, 2013), Gresch and Bogeholz (2013), Moore et al. (2012), Moriba et al. (2012), Johnson, Boyer, and Brown (2011), Smeds et al. (2011), Bo-Yuen Ngai and Koehn (2010), McCormack and O’Flaherty (2010), Kennelly, Taylor, and Maxwell (2008), Ioannou et al. (2009)
Quasi-experimental approach – mixed methods	Pre-post survey/questionnaire, interviews with learners; focus group with learners; student diaries; analysis of classroom interactions	Murray, Goodhew, and Murray (2013), Gresch and Bogeholz (2013), Pace (2010), Pipere, Grabovska, and Jonane (2010), Hestness et al. (2011),
Qualitative approach – analysis of student work	Student exams; content of digital interactions; student reflections; learner dairies	Kourti and Androussou (2013), Lencucha (2014);Zeegers and Clark (2014), Seeberg and Minick (2012), Habron, Goralnik, and Thorp (2012)
Qualitative approach – analysis of student teachers’ work	Teacher reflections on the learning activity; Participation/observation; analysis of teacher lesson plans	Koch et al. (2013), Paschall and Wuestenhagen (2012), Winter and Firth (2007), McNaughton (2006, 2010), Ollif (2001)
Qualitative approach	Focus group with students; analysis of classroom talk by students	Niens and Reilly (2012), Bautista Garcia-Vera (2012)

The results presented in Table 1 attest to the wide breadth of assessment modalities used to assess education interventions. It is heartening to note the variety of student-centred approaches employed across intervention designs (student reflections and learner diaries for example).

Six papers either made use of teacher reflections on the learning activity, or used participation observation by the teacher-researcher while one paper used focus groups with students to assess the impact of learning as the sole data collection format. Other assessment modalities included analysis of classroom talk by students in response to the use of audio-visual teaching materials, analysis of teacher lesson plans developed, and student interviews. Three papers assessed the content of online learning programmes. Two papers assessed student concept maps created before and after an education for sustainable development intervention. The inclusion of so many approaches demonstrates the complexity of DE/ESD/GCED knowledge base, and therefore highlights the challenges of measuring interventions aimed at facilitating increased knowledge, understanding, and awareness of action outcomes.

The presence of a large number of quasi-experimental research designs is notable as their increased usage reflects the proliferation of metrics and evaluation of impact of learning (OECD 2013). When designing assessment, educators need to remain cognizant of the complexity of a concept that crosses cognitive and affective domains, therefore assessment modalities need to capture the intended student outcomes across both these domains. If the study is set up as a deliberate education intervention, then quasi-experimental approaches are a suitable method for assessing impact or effect on learners. However other studies make use of traditional assessment approaches, for example, analysis of student exams, which raises the question of whether exams are a suitable form of assessment of DE/ESD/GCED outcomes, and whether cognitive understanding and awareness are central to measurement of outcomes.

Education content

In this section, we present a summary of the educational interventions reviewed. As stated earlier, 26 articles described education for sustainable development or environmental education themes, 12 described global development-orientated content, and six described intercultural education interventions. The following section expounds on each of these categories.

Education for sustainable development

Twenty-six papers addressed education for sustainable development or environmental education themes. Content was delivered using a variety of approaches including blended learning, drama, simulation exercises, decision-making for sustainable development and self-evaluation tools for pre-service teachers (Pace 2010). Themes reflected a broad approach to ESD and environmental education and included genetically modified food (Dovros and Makrakis 2012), energy issues (Pipere, Grabovska, and Jonane 2010; Sakschewski et al. 2014), natural resource management (Koch et al. 2013), and systems thinking (Gresch and Bogeholz 2013). Murray, Douglas-Dunbar, and Murray (2014) researched learner values base with regard to sustainability, while Zeegers and Clark (2014) utilised student evaluations and diaries to assess sustainability awareness.

Development education and global citizenship focus

Twelve papers were categorised as having a global focus through either having development education in the content or global citizenship as the focus of their educational intervention. Of these, 11 were explicitly identified as development education; with three papers sharing a development education and global citizenship content focus. One shared development education with education for sustainable development, and another had an explicit global citizenship focus.

Three papers were categorised as sharing a focus between development education and global citizenship. One paper examined the potential of multimedia learning and simulation exercises to enhance student knowledge of world issues (Ioannou et al. 2009), while another study examined the learning of students in a college of agriculture arising from modules with an international dimension, although no detail of the international content were given (Moriba et al. 2012). Meyer, Sherman, and MaKinster (2006) wrote of a continuing professional development opportunity for teachers through the ‘Japan Bridge’ project, while Johnson, Boyer, and Brown (2011) also used a problem-based online learning tool to examine its potential for global competence.

One paper was categorised as a shared focus between development education with education for sustainable development; Lencucha (2014) studied the impact of a global health module with students through assessment of student work. A further paper stated an explicit global citizenship focus, as it examined the potential for learning through education for global citizenship in a divided society context of Northern Ireland (Niens and Reilly 2012).

Intercultural education oriented

Articles included in the review, which were intercultural education orientated, demonstrated a variety of foci. One paper focused on analysing young people’s intercultural knowledge as enhanced through classroom interventions in Madrid (Bautista Garcia-Vera 2012), while another addressed a US-based study of young people’s knowledge and understanding of indigenous people (Bo-Yuen Ngai and Koehn 2010). Other studies focused on the following areas: developing pre-service teachers’ intercultural awareness and understanding (Kourti and Androussou 2013; Seeberg and Minick 2012); and a discourse analysis of Swedish student teachers web-based forum posts examining their cultural beliefs and values (Myers and Eberfors 2010). One paper was a study of a community of learning in a business environment, which assessed the learning impact of online learning interventions in developing global intercultural understanding (Rehm 2009).

To summarise briefly, the selected papers address a wide variety of content reflective of the diverse and global world in which we live, and the challenges facing our society and planet. Examples include knowledge of sustainability and global development issues, attitudes and awareness, and decision-making for sustainable development and promotion of intercultural knowledge/awareness.

Intervention outcomes

In this section of the findings, we present a description of the intervention types and the particular outcomes attributable to these intervention approaches. Included in the final review were a number of varying intervention approaches and outcomes classified in different ways. Of the 44 studies included in the final synthesis of the research, eight report ‘statistically significant impact’ from pre- to post-assessment following the intervention (see Table 2 for an overview of these studies). Six further studies reported ‘some significant impact’ from pre- to post-assessment following intervention, indicating that statistically significant differences emerged across ‘some’ of the constructs included in assessment (see Table 2). Just four studies reported ‘no significant impact’ from pre- to post-assessment following intervention, indicating that assessment scores did not significantly differ from pre- to post-assessment following intervention (see Table 2).

Table 2. Education intervention outcomes.

Outcome	Intervention approach	Outcome(s)	Examples
	Inclusion of ESD (focus on knowledge, attitudes, behaviours across all three dimensions of sustainability)	Significant differences in students’ values between schools that included ESD and schools that did not include ESD content	Berglund, Gericke, and Chang Rundgren (2014)
	Multimedia-based instructional material	Students in the multimedia group (MG) had marginally larger gains in knowledge and interest than their counterparts in the text group (TG)	Ioannou et al. (2009)
	Global Education Project using a PBL simulation	Significant increases seen from pre- to post survey data in knowledge and skills from this educational programme	Johnson, Boyer, and Brown (2011)
	Global health module	Significant increase in global knowledge; male and female differences emerged; significant inverse relationship between global health knowledge and responsibility	Moore et al. (2012)
Significant impact	Focus on developing international awareness	Developed student awareness of international dimensions of education and globalisation; statistically significant differences emerged in attitudes	Moriba et al. (2012)
	Two types of online discussion forum: one personal Café Talk, one content led module forum	Participants’ attitudes towards group collaboration were generally positive – achieve better results by working collaboratively. E-learning phase was positively evaluated	Rehm (2009)
	Ecology (forest ecosystems) using a PBL simulation – competency orientated lesson	Raised problem-solving abilities – competencies in generating epistemic questions, planning experiments, identifying controls were reported	Roesch, Nerb, and Riess (2015)
	Cross-cultural competence (CCC)	Enhanced affective and cognitive CCC. Recognition of cultural stereotypes	Seeberg and Minick (2012)
Some significant impact	Judged potential solutions to sustainability problems, using a four-point scale of effectiveness across economical/ecological/social dimensions	Situational knowledge – no difference. High for socioeconomic and ecological. Lower for institutional knowledge. Knowledge of scientific concepts increased but small. Gaps in capacity for solving complex resource management problems	Koch et al. (2013)
	Values-based sustainability workshops: intended to help learners clarify their personal values regarding sustainability	Warm relationships, universalism and benevolence recorded increases after sustainability training. No statistically significant differences for power, achievement, excitement, or self-direction	Murray, Douglas-Dunbar, and Murray (2014)
	ESD/EE (Environment Education)	Greater awareness of EE/ESD. Confusion over titles. Lack of political support for EE. Participatory methods required acclimatization. School practice difficult	Pace (2010)
	ESD/EE (Energy Education)	Effectiveness of multi-disciplinary and cross faculty work; need for good training materials; not all innovative methods can be used in class; Latvian teachers not used to participatory methods	Pipere, Grabovska, and Jonane (2010)
	Pedagogical Approaches in Biology	Systems thinking – achievement score and justification score increases. Need for lesson to support stimulation	Riess and Mischo (2010)
	ESD/EE course – equal consideration to the social, economic and environmental aspects	Despite experiencing a pedagogical approach which challenged views by encouraging discussion, debate, and reflection, many of the students still leaned towards an environmentally focused perspective of sustainability	Zeegers and Clark (2014)
No significant impact	Environmental Education Unit	No major impact in willingness to act for the environment (possibly because a high percentage began unit with desire to include environmental issues in their teaching)	Kennelly, Taylor, and Maxwell (2008)
	Japan Bridge project – global education programme	More open minded to Japanese, but not to other cultures. No increase in empathy, non-ethnocentrism or resistance to stereotyping	Meyer, Sherman, and MaKinster (2006)
	Sustainable self-training: personal engagement with sustainability	Limited evidence of change in values from the value surveys; not statistically significant for all including control group. Interviews demonstrated a shift in participants’ values awareness; growing awareness of the significance of their personal values, perspective on their relationship with sustainability, new insights into the meanings and ideas of sustainability	Murray, Goodhew, and Murray (2013)
Positive impact	Development Education	• Positive attitude to integrating DE into teaching • Students report perceived gains in knowledge, are actively involved in the seminars, and show a desire to learn more about global issues	McCormack and O’Flaherty (2010), Pearce (2009), Reich (2012), Lencucha (2014)
	Global Citizenship	• Some theoretical conceptualisations of global citizenship, including an awareness of global issues, understandings of environmental interdependence and global responsibility	Niens and Reilly (2012), Riley (2006)
	Intercultural Education	• Global discussion forums can help students to interpret texts from multiple perspectives, problematize representations and interpretations, and consider how to transform identities and relationships in their intercultural interactions	Bautista Garcia-Vera (2012), Kourti and Androussou (2013), Myers and Eberfors (2010)
	Specific Pedagogical Approach	• Trans-disciplinary role play simulating multilateral negotiations on climate change: high levels of enthusiasm, with several students stating that they changed their own personal behaviours as a result	Paschall and Wuestenhagen (2012), Vanhear and Pace (2008)
		• Vee heuristics and concept mapping: the use of these two tools facilitates the achievement of ESD targets and may, in the long-run, bring about the desired environmental responsible behaviour. This is because these two tools present a process of praxis, and through their use, learners are trained in decision-making, reflective and problem solving skills
	Volunteering	• 40% of participant volunteers began voluntary work after their one-month experience in India. Personal growth was reported, rather than personal change	Ollif (2001)

It is important to foreground reporting of these interventions with some discussion of ‘outcomes’. Where quasi-experimental designs were used, outcomes were determined by pre- and post- tests, and results obtained in an assessment were compared from before and after the experiment or intervention. As with any research experiment of this nature, results of ‘impact’ must be interpreted with some caution. Quasi-experimental approaches are used to determine casual impact of an intervention without random assignment which can lead to some concern regarding internal validity as treatment and control groups may not be comparable at baseline (Cohen, Manion, and Morrison 2011). It may not be possible to credibly demonstrate a causal link between the treatment condition and observed outcomes using a quasi-experimental approach. This is particularly true if there are confounding variables that cannot be measured or controlled. Participant responses to DE/ESD/GCED interventions, therefore, may be plausibly influenced by factors that cannot be easily measured and controlled, for example, the participants’ intrinsic interest in the area.

Positive impact

Of the studies included in the final synthesis of the research, 22 reported a ‘positive impact’ by the deliberate educational intervention. Both Niens and Reilly (2012) and Riley (2006) reported positive impact on learners’ conceptualisations of global citizenship, including an awareness of global issues, understandings of environmental interdependence and global responsibility. Others reported on how learners perceptions have changed, reflective of cultural beliefs and values, and demonstrated greater media awareness (Kourti and Androussou 2013; Bautista Garcia-Vera 2012; Myers and Eberfors 2010).

A number of studies carried out with pre-service teachers demonstrated a positive attitude to integrating DE into their future teaching work (Reich 2012; Lencucha 2014; McCormack and O’Flaherty 2010; Pearce 2009). Other studies showed pre-service teachers interest in teaching on climate change (Paschall and Wuestenhagen 2012; Hestness et al. 2011); higher confidence in teaching global topics, and awareness of resources (Burmeister and Eilks 2012; Kennelly et al. 2012; McCormack and O’Flaherty 2010; Nielsen et al. 2012). Gresch and Bogeholz (2013) reported on increases in sustainability knowledge based on reflections of learners’ decision-making.

One study described ‘problems’ with reporting intervention outcomes (Habron, Goralnik, and Thorp 2012) suggesting difficulties in assessment of competencies and the need for a consistent and valid assessment measure with an agreed rubric, which would ensure stable and rigorous assessment across multiple reviewers.

Table 2 illustrates a general perspective on the main outcomes emerging from the intervention approaches and provides some examples. The intervention outcomes highlight the wide range of factors which can play a role in the delivery of intentional DE/ESD/GCED units of learning.

In reviewing the results presented in Table 2, a number of interesting findings emerge. In terms of the delivery of DE/ESD/GCED content, the results provide support for a number of learning outcomes including: increased awareness of global issues, more developed conceptualisations of global citizenship and increased understanding of environmental interdependence and global responsibility. One must be mindful however, that set against these positive results, the particular frame used to conceptualise these complex concepts is important, otherwise these activities could be seen as an endeavour to simply reproduce ‘northern’ perspectives. This was highlighted in a number of papers with an intercultural education orientation. A similar question must be asked of interventions with a development education focus as to the consistency of approach of conceptualisation of DE and global citizenship.

A number of interventions that report significant or positive impact utilised active learning methodologies including multi-media approaches, problem-based learning (PBL), discussion forums, role-play and concept mapping. This approach is particularly in keeping with the conceptualisation of DE and ESD as utilising active and participatory teaching methods. It is also positive to note that pre-service teacher education studies demonstrated an increased interest and openness to teach these concepts upon graduation.

Those studies that reported ‘no significant impact’ should be interpreted with some caution, as results may have emerged due to a number of compounding factors. For example, base-line figures that may already reflect the hypotheses being tested (Kennelly, Taylor, and Maxwell 2008); impact of specific culture (Meyer, Sherman, and MaKinster 2006) or the metrics used may not capture increases in ‘awareness’ (Murray, Goodhew, and Murray 2013). These results may also suggest a mismatch between competencies or outcomes selected for assessment and those the students acquire. It is important that some consideration be given to how we measure DE/ESD/GCED outcomes. Darling-Hammond (2010) surmised that the use of various different ways to assess ‘effective practice’ was a powerful aggregate for shedding light on performance, thus advocating a variety of assessment modalities.

Discussion

Results from the current review attest to the huge variety of global development content and themes presented which are reflective of the interdependent world in which we live. The diversity of pedagogical approaches employed to deliver content is also reflective of how DE, ESD and GCED are conceptualised in the literature (Andreotti 2006; Hogan and Tormey 2003; Nevin 2008). As only English language studies were selected for inclusion in this review, the authors accept that this limitation may present a mostly northern perspective of these concepts. It is heartening to see a variety of professional disciplines being represented as DE/ESD/GCED is relevant to all. Results also support the selection of appropriate assessment modalities, where mixed methods approaches are advocated – so as to ascertain learning across a variety of domains. Throughout the systematic review process, a number of questions and issues arose for the researchers, pertaining to the assessment and measurement of learning arising from deliberate educational interventions. In this section, we discuss three of these questions of relevance to the DE/ESD/GCED community: the dominant forms of assessment employed (epistemological question); the ways and tools employed for measurement (methodological question); and whether the measurement tools are assessing what is distinctive to DE/ESD/GCED work (pedagogical question).

Epistemological question: are forms of assessment employed relevant and appropriate?

As noted earlier, the dominant form of assessment of impact from the educational intervention utilised quantitative measures, such as a pre/post survey or questionnaire, essentially reflecting a positivist epistemology. Twenty papers employed this quasi-experimental approach. In his review of studies on the impact of environmental education, Rickinson (2001) highlighted that the evidence base for learning is predominately positivist in nature. This finding has been replicated in the present study, as half of the studies reviewed in this paper were framed by a positivist approach. However, a number of studies also used qualitative methods and many studies used multiple methods of data collection to triangulate their findings and to strengthen the rigour of their findings. Thus, it could be argued that some change may be seen in the forms of assessment used. Selection of our research question possibly aligned with more positivist approaches, as we specifically looked for studies of impact and measurement of/for learning. However, some change in types of assessment of/for learning towards the inclusion of qualitative measures is noted.

Additionally, Rickinson (2001) evidenced the use of interpretativist or constructivist epistemological paradigms, as opposed to feminist, poststructural or other epistemologies. In the 44 papers reviewed here, few made a clear explicit statement of the researchers’ epistemological stance, except for one paper framed by a critical pedagogy approach (Myers and Eberfors 2010). DE/ESD/GCED as a deliberate educational intervention can act to challenge existing social, economic and political systems, which perpetuate injustice and inequalities and arguably aligns with a critical pedagogy epistemology (Liddy 2011). For example, a topic such as Fair Trade could be used to redress the imbalances of global trade by guaranteeing a fair and liveable wage for farmers. Yet, the topic can be represented in a manner that does nothing to challenge and change existing economic patterns; rather, Fair Trade could be presented as a necessary and temporary reform. In critical and more politically informed DE/ESD/GCED, there is a necessity to engender critical literacy and address learners’ assumptions about poverty and inequality (Liddy 2014). Otherwise, educators may reproduce the systems and ways of thinking they are trying to question (Andreotti 2006, 49). The lack of explicit statement on epistemological views and knowledge values leads to questions on researcher/educators’ assumptions about the world.

Furthermore, the researcher/educators’ epistemological view of DE/ESD/GCED will also frame how they design assessment and measurement of impact. More explicit statements of researcher/educators’ reflexivity and epistemological views are essential in developing a fuller account of the impact of deliberate DE/ESD/GCED educational interventions. Equally of importance is the researcher/educators’ epistemological views of learning, where we need to question how leaning is conceptualised across these papers – from a cognitive perspective, or reflective of affective and emotional domains? Bourn (2014a) argues for a pedagogy of social justice, where learners explore global issues in their social, cultural and historical context. Rather than prescribing a set interpretation of these contexts, leaners engage and debate in order to develop their own perspective and understanding, thus supporting the selection of outcomes reflective of all learning domains.

Methodological question: are the tools employed for measurement adequate?

This question centres on the tension that exists between the philosophical conceptualisation of DE/ESD/GCED versus the measurement of learning. A performance measurement approach to project management insists on the inclusion and development of indicators of expected change, assessment of baseline, stated targets and validation tools to provide evidence of change. This results-orientated approach emphasises efficiency and accountability in public spending, with clearly defined outputs, and results demonstrating value for money. As budgets shrink, the emphasis is on demonstrating social value and efficiency in spending using objective and quantifiable outputs and results. Beck (2013, 28) argues that often times, policy succumbs to a compulsion ‘to act speedily, in a way that threatens to bypass the rules of democracy.’ Solbrekke and Sugrue (2014, 11) note that this approach is evident in education, where,

the approach nationally and internationally has been to prescribe what the ‘knowledge economy’ requires and to ‘measure’ the ‘success’ or otherwise of schooling [and] higher education, … by externally imposed accountability measures including the necessity for rapid action due to politicians’ need to exercise power of ‘action’.

Performance indicators bring the focus on the product rather than the process of DE/ESD/GCED, a critique that is made of the Irish education system (Gleeson and O’Donnabháin 2009). When it comes to education, both defining outcomes and measuring success are difficult as the process of education is complex and multifaceted (Ball 2008; Liddy 2014). Bourn (2014a) argues strongly that the learning outcomes from development education cannot be predefined; rather, individual learners engage in debates on development and global poverty in order to deepen their understanding of different perspectives and encourage critical reflection. Furthermore, the content of global development is not readily understood, nor can easy solutions be found. Learning about global issues can raise overwhelming and far-reaching concerns, describing a world of ecological risk and threats, without hope or realistic plans for the future. The fundamental questions on our economies, politics and social choices are often left without answers, leaving students feeling overwhelmed, dejected and cynical about their efficacy to make change. Thus, selecting appropriate learning outcomes and indicators for inclusive practices, identity in a global context, or self-confidence in challenging racism or other unjust behaviours is difficult. Additionally, the impact of deliberate DE/ESD/GCED educational interventions may be long-term engagement with issues and questioning assumptions, rather than immediate measurable results. Consequently the development of indicators and outcomes is more complex and relates to the researcher/educators’ definition of development education, as addressed earlier. This product outcome focus misses the distinctiveness of DE/ESD/GCED, where the learning outcomes may be in the form of questioning and activism, rather than immediate or short-term goals. DE/ESD/GCED relies on learner agency and openness to new understandings and reflections on internalised dispositions. The insights gained into participants’ interpretations of global development, inequalities and poverty present a multifaceted picture of our world, which may require time and consideration for impact to be realised. Defining appropriate and adequate measures for impact of DE/ESD/GCED continues to be a challenge.

The authors are also conscious of the ever increasing discourse surrounding the use of Randomized Control Treatments (RCTs). Using RCTs, study participants can either be assigned to the treatment or control group. Whilst the authors acknowledge that this could prove useful in terms of assessment of DE/ESD/GCED interventions as differences observed between groups would be due to chance, rather than to a systematic factor related to treatment (Cohen, Manion, and Morrison 2011). It is important to note that randomization itself does not guarantee that groups will be comparable at baseline, however, any change in characteristics post-intervention is likely attributable to the intervention. In light of the results presented in this review, the authors recommend building a mixed methods research design comprising of RCTs and other interpretivist approaches which lends itself to the existence of multiple realities and experiences that may be viewed differently (Moustakas 1994) and allows for thick descriptions and complex nuanced findings (Dumas and Anderson 2014).

Pedagogical question: are measurement tools assessing what is distinctive to DE/ESD/GCED education?

The ethos of results-orientated approaches reflects a mechanistic education (Liddy 2014) which ultimately is at odds with education tasked with addressing sustainable and environmental challenges (Sterling 2001). The potential for change is limited in a mechanistic education system, as prescriptive forms of education with predetermined learning outcomes are limiting to the process of learning and development of self. As Freire said, ‘Liberating education consists in acts of cognition, not transferrals of information’ (1972, 53). Both Sterling and Freire’s analysis is apt here, as the complexity of education is not easily reductive to numbers and quantifiable indicators.

DE/ESD/GCED stress the importance of active and participatory learning methodologies, yet the majority of the 44 papers included for review reported on work completed in traditional learning environments such as lecture theatres and classrooms. (The exceptions are research based in an NGO setting of an overseas volunteer programme, and an outdoor education setting.) The traditional, formal educational setting has been noted as being problematic for the use of active methodologies (McCormack and O’Flaherty 2010) and the dominance of traditional learning sites is contrary to the inclusion of active and participatory learning, which is central to developing learners’ efficacy in relation to global issues.

One positive note is the use of use of multimedia in both the design of educational interventions, as well as its use as a tool for measurement of learning. Seven papers used online leaning forums; two papers employed online problem-based learning simulations, while another employed a climate change negotiations simulation, supported by a variety of platforms (short story forum, Appropedia, Blackboard). Many of these studies employed analysis of student interactions and conversation threads from these formats. The use of other forms of media was also noted, including audio-visual stories, film and film-making, student drawings and concepts maps. This use of multimedia demonstrates innovation in both pedagogical design and as a research tool, it may also support learner engagement and enhance participation.

It is important to highlight the absence of activism for global change as a learning outcome in studies reviewed. DE, ESD and GCED aim to work towards actions for sustainable and just social change (Irish Aid 2006; UNESCO 2002; UNESCO 2015). Yet, none of the papers cited here examined aspects of action and activism as part of the educational intervention, or as part of the assessment process. This is possibly due to the form of research question asked, yet it is a notable absence. Assessing young people’s knowledge and understanding of global justice may be more straightforward in terms of their cognitive acquisition, but assessing their behaviours and actions for social justice, and their underlying values and attitudes, is far more complex. Action for sustainable and just global change can range from generating greater awareness through letter-writing or social media to personal consumer activism, and from lobbying local politicians to engaging in national policy consultations (Liddy 2013). Yet, none of the 44 papers reviewed here included activism as an output.

Conclusion

Set against a background of austerity measures, the incursion of New Public Management into education policy (Solbrekke and Sugrue 2014) and practice and calls for robust evidence to justify educational change such as the inclusion of global development and sustainable development issues, this paper aimed to review measures of impact of intentional Development Education interventions (including ESD/Global Citizenship Education). Impact was conceptualised as: change in knowledge, skills, attitudes, ethics, and actions arising, including both hard and soft measurement outputs, from exams and knowledge tests through to ethics and values measures and was a necessary requirement for inclusion in this review. Many studies report statistically significant outcomes with others highlighting positive outcomes from their educational interventions.

However, a number of questions have arisen from this review – notably questions of epistemology, methodology and pedagogy. DE/ESD/GCED have multiple learning outcomes based on a complex knowledge base. Epistemological questions became clear from the review process, as many papers reviewed failed to present a clear account of the values and beliefs of the researcher/educator, and displayed a reliance on a positivist epistemology. Methodological questions arose as to whether the assessments modes employed were appropriate for the active and participatory teaching approaches advocated within DE/ESD/GCED, as well as whether they measure the complexity and variability of DE/ESD/GCED outcomes across all learning domains. Finally, pedagogical questions were addressed which noted positive innovations with the use of multimedia, but also the continued dominance of traditional learning sites and the lack of activism for positive social change as a learning outcome.

In conclusion, we note that much has been achieved in addressing, defining and measuring the impact of DE/ESD/GCED. The high number of studies which report both positive and significant learning is welcome. However, designing and employing appropriate and adequate research methods to address the complexity and multiplicity of learning arising from DE/ESD/GCED requires further investigation and innovation. This is especially important as Goal 4.7 of the Sustainable Development Goals is being implemented in the coming years.

Funding

This work was supported by the Ubuntu Network based at the School of Education, University of Limerick.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Joanne O’Flaherty is a lecturer in the School of Education, University of Limerick. She has a primary degree in Physical Education and English. She has worked in a variety of educational settings, including the formal second-level sector and the NGO sector, before joining the University of Limerick faculty. She is responsible for both co-ordinating and disseminating different education modules offered by the School of Education and acts as the Academic Coordinator of the Ubuntu Network. Her research interests include pre-service teacher education, pro-social development, and social justice education, and she has published in these areas.

Mags Liddy completed her PhD thesis at the University of Limerick examining the impact of overseas volunteering as a professional development experience for teachers. She is co-editor of Education that Matters: teachers’ critical pedagogy and development at local and global levels (2013, Peter Lang). She was post-doctoral researcher for the study of characteristic spirit of publicly managed schools in Ireland during 2016.

Notes

1. Irish Aid, a division of the Department of Foreign Affairs and Trade, is the Irish Government’s official aid programme, working on behalf of Irish people to address poverty and hunger in some of the world’s poorest countries (see https://www.dfa.ie/our-role-policies/irish-aid/).

2. It must be noted that our findings are probably skewed by the UN Decade for ESD, which led to the publication of much research work and special journal issues, as well as inspiring new journals such as International Journal of Sustainability in Higher Education (McKeown and Hopkins 2005).