Ocean literacy gamified: A systematic evaluation of the effect of game elements on students’ learning experience

Abstract

Low levels of concern about anthropogenic climate change have been attributed to a range of factors, some of which relate to education. These include people’s lack of understanding and engagement with the multifaceted nature and extent of the problem that it presents to current and future generations. Limited knowledge is also known to be an obstacle to individual behaviour change, with important implications for young people’s perceptions of the urgency to act and awareness of the consequences of their own behaviours. In this study, we explored ways to address low levels of understanding about ocean science dimensions to climate change phenomena, cognisant of a growing awareness that formal education curricula do not adequately engage young people with developing ocean literacy. Participants were a sample of secondary school students (11 to 14 years) in Portugal and the UK. Using a gamified mobile application, it was examined relationships between the use of different game elements such as points, badges and leaderboards, and learning outcomes. Systematic evaluation of each element shows how different game features affected the participants’ learning experience and learning outcomes. Implications for formal and informal marine education, climate education, and how to improve ocean literacy efforts, are also discussed.

Keywords:

• Gamification
• ocean literacy
• educational technologies
• cognitive learning
• motivation

Introduction

There is an increasing emphasis on environmental education. The interest in marine education has roots in the environmental movement of consciousness since the 1960s (Marrero and Mensah 2010; Charlier and Charlier 1971), as an answer to the new conditions of industrialism: “a process which involves, crucially, internalising the notion of limits” (Foster 2001, p. 156). One of the major consequences of this lacking notion of limits affects the ocean. A common denominator of all the issues that have a detrimental effect on the ocean and on climate change is the waste we produce. A “crusade against climate change science stands as a defence of consumption: it stands for our right to consume” (Falasca-Zamponi 2010, p. 3), and waste is a consequence of consumption.

In the last two decades, there has been a growing acknowledgement of the ocean’s vital importance for the well-being of the planet and the current problems threatening its health, such as climate change, evidenced by many news articles and headlines (O’Donnell and Thorne 2020; Boudet et al. 2020; Cowburn, Samoilys, and Obura 2018; McGrath 2018; Trenberth et al. 2018). Despite its intrinsic and extrinsic values, the ocean is now presenting significant signs of change as a consequence of human activities (Winks et al. 2020; Guest, Lotze, and Wallace 2015).

The grassroots for Ocean Literacy started in 2002 (Costa et al. 2020; Santoro et al. 2017). Integrated into the educational practice, it is believed that better public understanding of the ocean plays an important part in resolving complex critical issues like decades of pollution, habitat degradation, overfishing, and now climate change and ocean acidification (Ocean Literacy Network, 2013; Cava et al. 2005). An ocean literate person understands the ocean’s influence on us and our influence on the ocean (Ocean Literacy Network, 2013).

Presently, the protection of the ocean has become one of the most urgent issues in the world environmental agenda. Several studies on marine education and related topics have been addressed by numerous surveys carried out among students (Lin et al. 2020; Tsai 2019; Guest, Lotze, and Wallace 2015; Plankis and Marrero 2012) and teachers (Hartley et al. 2018; Dromgool-Regan, Burke, and McClouglin 2017). The literature has revealed a knowledge gap about ocean science-related topics in several countries (Winks et al. 2020; Ankamah-Yeboah et al. 2020; Fauville et al. 2019; Leitão et al. 2018; Costa and Caldeira 2018; Papathanassiou et al. 2017; Guest, Lotze, and Wallace 2015; Corner, Capstick, and Pidgeon 2014). Yet many people are not aware of the impact of the ocean on their lives (Pantò 2019; Frick, Kaiser, and Wilson 2004) and there is little understanding of marine environmental issues and protection (Fauville et al. 2019). The literature also shows that ocean topics are largely missing from national school curricula in Portugal (Costa et al. 2020; Barracosa et al. 2019) and the UK (Winks et al. 2020).

The Essential Principles and fundamental concepts of Ocean Sciences can be addressed in the different science subjects of the Portuguese and UK curricula education systems (National Curriculum in England: Science Programmes of Study 2015;Ciências Naturais. Documentos curriculares de referência 2018). Although there are overlaps in science and geography about the climate and environmental issues, in the national curricula in both countries, there are no explicit standards focusing on the ocean and related maritime topics at Key Stage 3, 7^th to 9^th grades (aged 11-14 years). The lack of these scientific concepts prevents people from making conscious personal and societal decisions about environmental issues (Goodale 2020; Marrero and Mensah 2010). Furthermore, an ocean literate person not only holds an understanding of the ocean but is also able to make informed and responsible decisions regarding the ocean and its resources. Therefore, engaging learners in experiences related to the ocean could not only improve their literacy but also motivate them to act for the ocean’s well-being United States Commission on Ocean Policy 2004; Daigle 2003).

Education plays a crucial role in improving ocean literacy, ensuring critical thinking, citizenship skills and democratic attitudes and values (Mercer et al. 2017; Cotton and Winter 2010). According to Mercer et al. (2017) to reach effective education in order to influence sustainable behaviour requires the inclusion of more interactive methods, supporting a move towards more practical and learner-centred approaches. Environmental protection and sustainable development lie with a public that is not only informed but also engaged (Daigle 2003) and therefore, more suitable pedagogies based on more interactive approaches are needed (Mercer et al. 2017). Visbeck (2018) stressed that more engagement at secondary school levels to promote ocean literacy was urgently needed. Interactive learning experiences, including multimedia platforms, have been shown to improve active student learning and affect the learning process positively (Cherner, Wu, and Fegely 2019; Haßler, Major, and Hennessy 2016). The motivating and immersive nature of educational games, with actions, challenges, rewards, and the central layer called ‘fun’, have enhanced the use and interest in them (Cheng 2017; Dicheva et al. 2015).

Games have been used to change behaviour and attitudes, improve learning and motivation outcomes in the areas of health, public policy as well as education, training, environmental behaviour and ocean literacy (Veronica and Calvano 2020; Pantò 2019; Kelley et al. 2017; Huynh, Zuo, and Iida 2016; Rebolledo-Mendez et al. 2009; Dieleman and Huisingh 2006). They can promote a better acquisition of knowledge through exploration, action and experimentation (Prensky 2001). This has motivated their application beyond entertainment. One of the widely adopted strategies in this area is gamification. Most scholarly sources agree that gamification is generally defined as including game elements in non-game contexts to improve the user experience and motivation (Dicheva et al. 2015; Werbach and Hunter 2012; Deterding et al. 2011a). The most common strategy used for designing gamified experiences is to use ‘game elements’. In the growing number of educational systems that incorporate game elements to improve the learning experience, points, badges, and leaderboards are some of the most commonly used (Warmelink et al. 2020; Kirillov et al. 2018; Dicheva, Irwin, and Dichev 2018b; Werbach and Hunter 2012; Zichermann and Cunningham 2011). For this reason, a gamification approach was chosen for this study due to its engaging and motivating features, which could enhance the learning experience (Dicheva, Dichev, and Irwin 2018a; Çakıroğlu et al. 2017; Werbach and Hunter 2012), assist the cognitive domain of learning (Kapp 2012) and, consequently, improve the learning outcomes. Through the experience with a gamified application, pupils can learn by doing and by failing, without negative consequences, which is often absent in traditional classroom approaches. When consequences of actions are negative, one learns what not to do and plan alternative strategies and goals (Dieleman and Huisingh 2006). Therefore, students can play, manipulate, experiment and experience what the consequences are or what they might be (Dieleman and Huisingh 2006; Prensky 2001).

Within this context, gamified applications can create new opportunities and challenges inside the education context. Although teachers are core agents of implementing these technologies in classrooms, it is also essential to address factors that could benefit or obstruct their use: school factors (e.g. availability and support of technologies, professional development), student factors (class sizes and student skills) and teacher factors (attitudes, beliefs and teaching experience) (Drossel, Eickelmann, and Gerick 2017). A further requirement is a need for information and communication technologies (ICTs) integration into curriculum providing reasonable goals, content and methods and, at a higher level, the management of the ICT innovation of a school, district, state and national level. The integration of ICT in the classroom is directly related to teaching practices and methods that the teacher wants to apply. Teachers are viewed as both subjects and agents of change and, at this integration level, marine science concepts also face influences in the curricular adoption (Goodale 2020).

Research by Mercer et al. (2017) shows much criticism of the secondary school system for not providing robust education on topics related to climate change and sustainability. The reasons pointed to are that teachers do not feel prepared to teach these topics and “can find the prospect both daunting and challenging due to the subject complexity and perceived controversial nature of the topic as well as its interdisciplinary nature” (Mercer et al. 2017, p. 360). Furthermore, Gotensparre et al. (2017) identified several barriers to ocean literacy when teaching 12-19-year-olds about the ocean in Portugal and the UK. The authors highlight an inability to recognise the importance and value of teaching students about the ocean in the UK. This aggravates other present barriers such as lack of confidence when taking students out of the classroom (regarding health and safety, logistics), competition and lack of clarity/coordination between providers. The Portuguese education system displays barriers of lack of interest from the ministry of education to implement ocean literacy in the national curriculum, lack of a national strategic plan to implement ocean literacy and, lack of obvious connections with the school curricula (Barracosa et al. 2019).

Several resources are available to promote the integration between the education system and the environment. A platform called Digital explorer (Digital Explorer 2018) offers immersive learning experiences related to the ocean to provide the students with the necessary understanding and skillsets to tackle the world’s most pressing issues. People and Planet and Sustainability Exchange (People and Planet 2019; EAUC 2019) are two projects that incorporated Green leaderboards and Green Gown awards to increase participation and interest from educational institutions in sustainability. The Sea Change project aimed to empower ocean literate citizens to take direct and sustainable action towards a healthy ocean, communities and consequently the planet. The project targeted its intervention at three main societal groups: the general public, formal educators, and policymakers. In the UK, the main focus was developing resources for educators and students to introduce ocean literacy topics into curricula (Winks et al. 2020). The ResponSEAble project aimed to raise awareness about the ocean reciprocal influence between humans and the sea. This project developed several communication tools to address the identified ocean literacy gap in the European population. A mobile application was designed and made available online based on the assumption that games can effectively change behaviours (Pantò 2019).

As a growing number of education systems are incorporating game elements to improve the learning experience, this raises the research question: can different game elements impact the learning outcomes and, if so, how? Therefore, this study was conducted to understand if there is any relationship between different game elements, such as points, badges, leaderboards (also known as the PBL triad), and learning outcomes of ocean literacy.

Method

Through a participatory design approach, a gamified Ocean Literacy mobile application was developed combining the findings from previous studies: platforms and skills (Leitão, Turner, and Maguire 2017), learning outcomes (Leitão et al. 2018), learner-centred (Leitão, Maguire, and Turner 2019b), and game effects on different levels of motivation (Leitão, Turner, and Maguire 2019c). The research tool application consisted of a recycling game where the player recycles items falling from the top of the screen into the appropriate recycling bin. On the lower part of the screen, water was shown flowing with healthy fish in it. Failing to throw an item into the correct bin resulted in the water becoming darker, the fish dying, at which point the game ends. To conduct a systematic evaluation, four versions of the application were used to be able to understand the effect of the different game elements on the students’ acquired knowledge about ocean topics. Three versions used of the techniques: points, badges and a leaderboard to give success feedback while the fourth used all three techniques.

The study adopted a mixed methods approach which involved user testing of the app with observation. As well as playing the game, a multiple-choice test was employed for data collection and asking for pupils’ comments concerning their learning experience. The questionnaire was developed based on the topics: eutrophication, sea-level rise and plastic pollution in the ocean (the intended learning outcomes).

The survey was carried out to school children within the age group 11-14 years, corresponding to 7^th to 9^th grades (Portugal) and Key stage three (KS3), i.e. the first three years of secondary school in the UK. The sample consisted of students from a local comprehensive secondary school in the UK (East Midlands) and secondary school students attending classes during a holiday science programme at the Junior University Summer School, Porto University. A total of 69 students participated in the study in Portugal, and 29 students in the UK, randomly selected. The data were collected in July 2019 in both countries.

The sessions were administered in five different classes, each one with a different condition (see Table 1). Normally, the gamified application would be used as a support tool to complement traditional teaching and not be solely used as a learning tool. Therefore, initially, in all the sessions, a class presentation was given on the three ocean literacy topics, with PowerPoint slides as support. This initial 15-minutes presentation, as well as the design of the slideshow, in both countries, were made by experienced science educators. There were five different session conditions: (1) presentation with game (points), (2) presentation with game (badges), (3) presentation with game (leaderboard), (4) presentation with game (all game elements), (5) presentation only. For each of the sessions, pupils were observed by two researchers who recorded behaviour cues and took notes of indicators of engagement, such as comments or positive body language, as well as signs of lack of enjoyment and frustration. Pupils’ behavioural cues may be more reliable than their responses to direct questions about whether they liked something (Hanna, Risden, and Alexander 1997). The questionnaires were anonymous, and permission was obtained from students and parents regarding participation in the study to comply with the research ethics requirements in both countries. In the UK, ethical approval was obtained through the Loughborough University Ethics Committee for research with human participants. In Portugal, ethical approval was obtained from the Ministry of Education with the registration number 0582600003.

Table 1. Session planning.

Session 1 Points	Session 2 Badges	Session 3 Leaderboard	Session 4 PBL	Session 5 Oral E.
Class presentation given to students on topics: • eutrophication • ocean plastic • sea level rise
Students interact with the application with the Points gamified version	Students interact with the application with the Badges gamified version	Students interact with the application with the Leaderboard gamified version	Students interact with the application with all 3 elements gamified version	Students had no interaction with the application
Ocean Literacy questionnaire (6 questions)

As pupils progressed through the game, potentially being motivated by particular game elements, they were shown more information about the ocean, which increased their knowledge. At the end of each session, to evaluate the learning gained during the five sessions, participants answered a series six of multiple-choice questions concerning ocean literacy (Table 2). The questionnaire was developed in cooperation with the Interdisciplinary Centre of Marine and Environmental Research (CIIMAR, University of Porto) (correct answers are shown in boldface):

Table 2. Ocean literacy answers obtained for Points as a game element. Over 80% of correct answers are in bold font, and less than 20% are in red.

	U K = 6 (%)			P T = 14 (%)
Question	Right	Wrong	Don’t know	Right	Wrong	Don’t know
(a) What is eutrophication	50.0	50.0	–	92.8	–	7.10
(b) Nutrient origins	16.6	66.6	16.6	71.4	7.10	21.4
(c) Causes of sea-level rise	100	–	–	100	–	–
(d) Sources of greenhouse gas emissions	50.0	50.0	–	57.1	14.2	28.5
(e) Where the plastic come from	66.6	33.3	–	100	–	–
(f) Percentage fish affected by plastic	83.3	–	16.6	57.1	35.7	7.10
Total	61.0	33.3	5.50	79.7	9.50	10.7

1. What is eutrophication? (An ocean current that runs along the Atlantic Ocean; The process of formation of calcareous structures of shells; An ecological process caused by excess nutrients in a mass of water; A molecular process associated to photosynthesis; Don’t know);

2. The nutrients that originate eutrophication come from… (Industrial waste; Industrial waste and agriculture run-off; Urban and industrial waste; Agriculture run-off, industrial and urban waste; Don’t know);

3. Among the causes that most contribute to the rising sea level are: (River currents caused by heavy rainfall; The movement and rotation of the Earth; Large discharges of water that we make in our homes; Melting of glaciers and of the Antarctic and Greenland ice sheets; Don’t know);

4. One of the major sources of greenhouse gas emissions and consequently the rising sea level is: (Methane emissions generated by the decomposition of paper in landfills; Oxygen emissions produced by plants through photosynthesis; Emissions of carbon dioxide from human breathing; Methane emissions generated by the decomposition of algae in the seas; Don’t know);

5. Where does the plastic that contaminates the ocean come from? (From the waste that we put in the wrong bins and therefore is not recycled; From the garbage that is left on the street and that is carried by wind and rain; From the plastics that, because they are so lightweight, fly from the landfills where they are unduly deposited; All previous options; Don’t know);

6. According to a United Nations report, what is the percentage of fish that is currently contaminated by plastic? (5%, 10%, 20%, 30%, Don’t know)

A model I factorial (orthogonal) analysis of variance (ANOVA) was carried out to investigate differences in the pupils’ learning scores elicited by the various game elements tested, in relation to the country of origin. For this, we calculated the overall score of each student in the learning assessment questionnaire: the number of questions answered correctly divided by the total number of questions in the questionnaire, i.e. six. This score was entered as the dependent variable in the model I factorial ANOVA. The independent variables, i.e. main factors entered in the analysis, were the country (with two categories; UK and PT) and the session (with five categories, i.e. four game element sessions and the classical oral explanation). The variance homogeneity and normality of the dependent variable were checked using the Levene’s test and the Shapiro-Wilk test, respectively. To fulfil the normality assumption, the dependent variable was transformed using the arcsine square root transformation, indicated for proportion variables. The Dunnett test was used in the post-hoc analysis of the test session main factor, to compare the scores in each game element category (Points, Badges, Leaderboard and the PBL triad) against the traditional teaching session, which was taken as the control category. The significance level was set at 0.05. All analyses were carried out with the Statistical Package for the Social Sciences (SPSS v27).

Results

The findings are first presented separately for the different game elements (i.e. Points, Badges, Leaderboard, PBL triad) and the control group, and then analysed globally in a two-way ANOVA design.

Session 1. Game element: Points

Throughout the observation of the tests during all the sessions, it was noticeable that the students did not need help to start interacting with the app and that they quickly learnt how to use it. The multiple-choice test findings (see Table 2) showed that UK pupils had more difficulty with the questions related to the origin of nutrients causing eutrophication, the eutrophication process and the greenhouse gas emissions, with only 16.6% and 50.0% of correct answers, respectively (Table 2). The questions with the highest percentage of correct responses were related to the sea-level rise (100%) and fish contaminated by plastic (>80%). The Portuguese students showed more difficulty with the questions related to greenhouse gas emissions and the percentage of fish contaminated by plastic, with around half of the correct answers. Almost all the students answered correctly the questions related to sea-level rise, the origin of ocean plastic contamination and eutrophication.

In both countries, students reached the screen where the points obtained were the awards displayed, and through this option explored the screens corresponding to the topics of eutrophication, ocean level rise, and plastic ocean (Figure 1). It was observed that students from both countries were curious to read all or most of the information. Some participants made some comments regarding their learning experience: ‘It teaches you how to recycle’; It tells you what goes to the bin’[United Kingdom pupils]; ‘Who doesn’t know can learn about recycling’[Portuguese student]. Pupils showed enjoyment during the experience: ‘Throwing the trash was fun’[PT student] and also regarding the challenge: ‘I liked the different levels and how it got harder’[UK student].

Figure 1. Ocean Literacy application (Leitão, Maguire, and Turner 2019a).

Session 2. Game element: Badges

The findings exposed more difficulty among the UK students in the questions related to greenhouse gas emissions and eutrophication (Table 3). The questions about the sea level rise and the origin of plastic in the ocean had the highest percentages of correct answers. Most of the Portuguese students answered all the questions correctly, with the lowest rate of correct answers over 60%. They had the highest percentage of correct answers about ocean plastic contamination.

Table 3. Ocean literacy answers obtained for Badges as a game element. Legend as in Table 2.

	U K = 6 (%)			P T = 13 (%)
Question	Right	Wrong	Don’t know	Right	Wrong	Don’t know
(a) What is eutrophication	33.3	66.6	–	84.6	7.60	7.60
(b) Nutrient origins	50.0	50.0	–	61.5	30.7	7.60
(c) Causes of sea-level rise	100	–	–	84.6	7.60	7.60
(d) Sources of greenhouse gas emissions	16.6	50.0	33.3	61.5	23.0	15.3
(e) Where the plastic come from	83.3	16.6	–	100	–	–
(f) Percentage fish affected by plastic	83.3	–	16.6	92.3	–	7.60
Total	61.1	30.5	8.3	80.7	11.5	7.60

Every time they reached a different badge, students in both countries read the information about the ocean topics that appeared on the screen ‘congratulations’. In this way, the information came to them not only through the exploration of the different screens where it was made available - the awards screen (Figure 2) but also through this game element. This could be the reason why the learning results were so high in this session. Even when children did not reach the screens with this information, that content popped up to them.

Figure 2. Ocean literacy application screens.

At the end of the application experience, the students stated they understood the educational content well and learned from the interaction with the application: ‘Learned not to leave plastic lying around’; ‘Good and easy to understand’[UK students]; ‘Good way to learn about the topic’[PT student]. Some pupils also made comments about background sounds and the game graphics: ‘Different levels should have different songs’; It should have more levels and more themes’[PT students]. Although one pupil stated ‘Levels are a bit repetitive’[PT student], most of them demonstrated enjoyment and considered they had learnt while interacting with the application. Concluding, the use of badges as the main game element, with the reinforcement of the topic contents, appeared to have a good effect on the learning outcomes.

Session 3. Game element: Leaderboard

When pupils became aware that the leaderboard was the central game element, it was possible to observe some arousal in their behaviour. From all the sessions, this was the one in which the students showed the most excitement. Nevertheless, compared with the other groups, they obtained the worst results regarding the learning outcomes. The UK students revealed more difficulties regarding the questions about the origin of nutrients and ocean plastic, both with less than 20% of correct answers. The highest percentage of correct answers was related to the plastic contamination in fish. Portuguese students had the highest percentage of correct answers to the questions about sea-level rise and plastic contamination in fish. Their lowest percentage of correct answers was 50% (Table 4).

Table 4. Ocean literacy answers for Leaderboard. Legend as in Table 2.

	U K = 6 (%)			P T = 14 (%)
Question	Right	Wrong	Don’t know	Right	Wrong	Don’t know
(a) What is eutrophication	50.0	33.3	16.6	78.5	7.10	14.2
(b) Nutrient origins	16.6	83.3	–	50.0	35.7	14.2
(c) Causes of sea-level rise	50.0	50.0	–	85.7	14.2	–
(d) Sources of greenhouse gas emissions	33.3	50.0	16.6	50.0	42.8	7.10
(e) Where the plastic come from	16.6	83.3	–	78.5	14.2	7.10
(f) Percentage fish affected by plastic	83.3	16.6	–	100	–	–
Total	41.6	52.7	5.50	73.8	19.0	7.10

In this session, students were more animated and interacted well. Both verbal and corporal behaviours were much more active but more focused on the others’ results.

Students did not comment on the learning but the experience. ‘Very, very fun. I would like to be able to go in at any level’; ‘Fun. I would like more levels’[UK students]. ‘Good’ was the most commented among the UK students. The experience among the Portuguese pupils was very similar. ‘What I liked most was when I could hit the right bin’[PT students] was a common statement among Portuguese children. ‘It was addictive’[PT student]. Although most of the pupils reported that the application appeared to be very good in terms of a learning tool, when asked what they liked most about the game, they made no reference to this topic, unlike in the other sessions. In sum, what students enjoyed most was getting a high place on the leaderboard.

Session 4. All three elements: PBL triad

In Session four, students interacted with the full application and demonstrated enjoyment during the experience. Most of the students in both countries explored the ocean topics screens, expressed interest in them and demonstrated to be reading the information related to the ocean topics. The UK students showed lower and highest knowledge levels in the questions related to the origin of nutrients in eutrophication (<20% correct answers) and plastic contamination in fish (>80%) (Table 5). The vast majority of Portuguese students answered the questions correctly.

Table 5. Ocean literacy answers for the PBL triad. Legend as in Table 2.

	U K = 6 (%)			P T = 14 (%)
Question	Right	Wrong	Don’t know	Right	Wrong	Don’t know
(a) What is eutrophication	33.3	66.6	–	100	–	–
(b) Nutrient origins	16.6	83.3	–	64.2	35.7	–
(c) Causes of sea-level rise	50.0	16.6	33.3	100	–	–
(d) Sources of greenhouse gas emissions	50.0	16.6	33.3	64.2	35.7	–
(e) Where the plastic come from	33.3	66.6	–	92.8	7.10	–
(f) Percentage fish affected by plastic	83.3	–	16.6	92.8	7.10	–
Total	44.4	41.6	13.8	85.7	14.2	–

As a learning tool, most students rated it as very good in both countries. They felt they had learnt new recycling information and consequently the effects on the ocean: ‘Very interesting to understand why the environment is being harmed’; ‘Useful recycling information’; ‘A more fun way to learn’[PT students]; ‘Tests your mind power’[UK student]. Additionally, they commented also on the application experience: ‘I liked the sliding action to recycle’; ‘Fun. I wanted to keep going’; ‘Quite good. I liked the shooting items into the bins’[UK students].

In the UK sample, one of the playing platforms sounded relatively higher than the others. Whenever the student with this tablet missed a bin, the consequent negative feedback sound was noticeable to all classmates. This sound provoked a strange reaction in the student, who always responded with discomfort. While negative feedback could enable the user to learn quickly and adjust, in this circumstance, it provoked a loss in the interaction with the system. Although among the UK students, the learning outcomes were not so high as among the Portuguese students, the overall experience showed wide acceptance and enjoyment by children in both countries.

Session 5. Oral explanations: Control group

This fifth session worked as a control group with only the constant condition: presentation on the ocean literacy topics plus slideshow presentation as teaching support (see Table 1). Consequently, students did not interact with the application. In both groups learning take-up was reduced seemingly due to lack of interaction with the application. In the UK, only 20% of the students answered the question about the origin of nutrients correctly, and none answered correctly the questions about the greenhouse gas emissions and ocean plastic (Table 6). Although the total of Portuguese students performed very well in the eutrophication and sea-level rise questions and less well in questions about the nutrients origin, greenhouse gas emissions and ocean plastic. In both countries, engagement decreased as well as students’ participation in the session. It was not possible to observe the enthusiasm of the previous sections conducted with the Ocean Literacy application.

Table 6. Ocean literacy answers given after the oral explanations. Legend as in Table 2.

	U K = 5 (%)			P T = 14 (%)
Question	Right	Wrong	Don’t know	Right	Wrong	Don’t know
(a) What is eutrophication	60.0	40.0	–	100	–	–
(b) Nutrient origins	20.0	40.0	40.0	50.0	42.8	7.10
(c) Causes of sea-level rise	40.0	60.0	–	100	–	–
(d) Sources of greenhouse gas emissions	–	60.0	40.0	57.1	7,10	35.7
(e) Where the plastic come from	–	60.0	40.0	50.0	50.0	–
(f) Percentage fish affected by plastic	40.0	40.0	20.0	71.4	7.10	21.4
Total	26.6	50.0	23.3	71.4	17.8	10.7

Overall results

The statistical analysis of the data by factorial ANOVA, using the pupils’ overall score in the knowledge test as a dependent variable, revealed significant differences between the two countries and among the game elements tested (Table 7). Interestingly, no statistical significance of the interaction term was found, indicating a similar response to the different game elements from pupils of both countries (Table 7). The PT pupils scored significantly higher (∼62%) in the test than those from the UK (p < 0.001, Figure 3), as suggested by the descriptive analysis. Because the PT control group also showed higher classification scores than the UK one, this suggests that Portuguese pupils were generally more knowledgeable about the topics under analysis than their UK counterparts. As to the game elements, the post-hoc analysis by the Dunnett test revealed that Points, Badges and the PBL triad all increased the learning outcome significantly by about 30%, compared to the control group (p < 0.05, Figure 3).

Figure 3. Scores obtained by the 98 pupils in the assessment questionnaire passed after the five testing sessions. Scores are presented according to the results of the model I factorial ANOVA carried out to investigated possible knowledge differences elicited by the game elements, i.e. by significant main factor: country (left) and test session (right). For the test session, the classical teaching session was taken as the control category against which the scores in each game element category (Points, Badges, Leaderboard and the PBL triad) were compared. ** indicates significant differences at p < 0.001; * indicates significant differences at p < 0.05 relative to the control, according to the results of the post-hoc Dunnett test.

Table 7. Results of the model I factorial (orthogonal) ANOVA carried out to investigate differences in the pupils’ learning scores obtained in the knowledge test (dependent variable) elicited by the various game elements tested in relation to the country of origin.

	Sum of Squares	df	Mean Square	F	p
Country	3.12	1	3.12	44.4	<0.001
Game element	0.900	4	0.225	3.28	0.015
Country x Game element	0.291	4	0.073	1.06	0.382
Residuals	6.04	88	0.069

In contrast, no statistically significant differences were found between the leaderboard and the control groups (Figure 3). Altogether, the findings indicate the leaderboard was fun for the pupils but not useful to improve the learning outcomes, compared to the other three elements that were both fun and efficient in improving pupils’ knowledge. The competitive effect as a central role in the application was thus shown to have a negative effect on the learning outcomes.

Discussion

Gamification is commonly described as the use of game elements and game design techniques in non-playing contexts, that is, in situations whose objective goes beyond merely winning the game (Deterding et al. 2011a). This approach has been applied to several fields to improve the user experience, motivation and engagement (Warmelink et al. 2020; Dichev et al. 2015; Werbach and Hunter 2012; Deterding et al. 2011b, 2011a). The most common strategy used for designing gamified experiences is to use game elements. In the growing number of educational systems that are incorporating game elements to improve the learning experience, points, badges, and leaderboards are some of the most commonly used (Warmelink et al. 2020; Kirillov et al. 2018; Dicheva, Irwin, and Dichev 2018b; Werbach and Hunter 2012; Zichermann and Cunningham 2011). The junction of these three elements is known as the PBL triad, and the extensive focus on these elements has given rise to a large part of the criticisms of gamified applications. The perspective of merely adding different game elements to an application without understanding the meaning behind them has raised a lot of criticism. Several authors have stressed that a gamified application is more than just points, badges and leaderboards (Kornevs, Baalsrud Hauge, and Meijer 2019; Liu, Santhanam, and Webster 2017; Chou 2016; Werbach and Hunter 2012). “The least exciting element of any game is the points, badges or leaderboards. People don’t play a game just for points, they play for mastery, to overcome challenges and to socialise with others. The most effective gamification efforts include more than points and badges — they contain elements of a story, challenge and continual feedback as well as a high level of interactivity” (Kapp 2012, p. 46).

The underlying features of the interaction process within the application, such as the sensation of progression or completion, challenge, the aesthetic experience, the sounds and the tangible interactions, can transmit fun emotions. These features called ‘playability’ can enable enjoyable experiences, which are fundamental to the creation of intrinsically motivating learning environments. However, it is crucial to have a balanced association between learning content and enjoyment for a gamified app to be appealing and effective in learning and not purely fun.

Learning to use the application

Students did not need to learn about the gamified application before the sessions were held. They understood the rules of the application and knew how to use it without being instructed. At the first level, brief instructions on the very basic rules of the application were given, and with this knowledge, students started playing and learnt more about it. Therefore, the technology integration level regarding the necessary skills of students and teachers to use the application seems to be present, showing that the results that are aligned with the previous study findings (Leitão, Turner, and Maguire 2017). Moreover, the further requirement of an ICT integration in order to support technologies such as educational tutorials explaining how to use the software could be much more simplified through a gamified approach.

Unlike other software that often needs the user to learn about it before use, in games, a user usually goes directly into the software. When they are confronted with a new item, they need to learn to adapt it with their existing schema by trial and error or mental reflection. Following this, students learnt by making mistakes (e.g. in this game, in bin selection) and by exploring directly during the application experience. This enhancement of the user experience through the constant and instant feedback from user actions to inform him/her about the steps that need to be performed could turn these apps into useful educational tools.

Effect of different game elements

The purpose of the study was to understand if different game elements (PBL triad) could affect the learning outcomes differently. Findings show that the inclusion of the game-like application with different game elements seems to influence the learning outcomes positively. At least, as far as the overall results are concerned, there appears to exist a close correspondence between the different game elements and the learning outcomes.

The learning benefits in the use of points, badges and the PBL triad seems to be consistent among the two countries. It is essential to highlight that badges can be used as different motivational stimuli according to different rewards. Type of rewards can go from extrinsic motivators, such as monetary or status rewards, towards intrinsic motivators. A previous study showed that games have a greater effect on intrinsic motivation than on extrinsic motivation (Leitão, Turner, and Maguire 2019c). However, the prototype version with the leaderboard as the main game element was shown to have a substantial effect on the least autonomous motivation level (external regulation). In the recycling context, this could represent only recycling to satisfy an external demand or to receive some reward (Leitão et al. 2021). Given this, badges were not used as a contingent reward but rather as a tool for raising the awareness of some of the environmental problems we face and how they arise. In summary, at least under these conditions in both countries, the use of points, badges and the PBL triad seemed to improve the learning outcomes.

Implications of the leaderboard

Despite the innate human desire to learn, overcome obstacles, improve ourselves and win, which can be achieved through fun and engaging tasks, the willingness to win in direct competition with participants can undermine learning outcomes. The comparison between different performances made the pupils not concentrate on their own learning development. The concern with staying ahead of colleagues did not allow them time to read the information about the ocean, nor did it allow them to explore the application and all its features. Therefore, this could have influenced the decrease in the learning outcomes results since students did not pay much attention to the information contained in the app and did not read it. The leaderboard as the main element of the game provoked competition among the students, which seemed to be the main reason that led to the decrease in learning outcomes. However, results presented an increase of 5% of correct answers compared with the session with no game elements.

These findings are aligned with the literature, which reported that competition came to the forefront as a negative element (Çakıroğlu et al. 2017; Ibáñez & Delgado-kloos, 2014). This study revealed that the students’ focus was so strong on their competition that it did not allow them to value the experience and context of the game. They were seen to skip or quickly passed the learning text to see their rank on the leaderboard. Thus, when questioned on the educational content, they showed a greater lack of knowledge about the topics and performed worse in answering the questions. Since the leaderboard is one of the most widely used game element among the gamification approaches, it is essential to stress that the use of this game design element in the learning context could provoke a negative effect on the students’ learning process and, thus, a decrease in the attainment of the intended learning outcomes. Nevertheless, Lister (2015) suggests that, due to the immediate feedback provided by these game elements and the possibility to improve their place, the leaderboard can be very motivating for competitive students while less competitive students can demotivate and decrease their performance.

As the game gets used more and more and certain players accumulate many points, it will become harder for new players to get onto the leaderboard. One way to minimise the stress could be to show the player’s name, position and the best score on the board along with the leaders. They can then measure their improvement and see how far they are behind the leaders.

Differences between UK and Portugal results

The PBL triad in the UK shows almost the same effect on the learning outcomes as the leaderboard element. But, on the contrary, this did not happen in the case of the Portuguese students, who obtained the best results. These findings may have been caused by contamination of the leaderboard, since it was also included in this prototype, while among the Portuguese students, it may have gone unnoticed. A possible reason for this effect may be the negative feedback sound of having failed. The sound was too loud on some platforms in the UK, and everyone in the classroom knew when another student failed, creating anxiety in some students. This could have undermined the learning experience and, consequently, the learning outcomes. The UK group was smaller than in Portugal, so this could have created an extra pressure for students; the Portuguese students were a bigger group, and the leaderboard went unnoticed. Another possible reason for this decrease in learning outcomes among the UK students may be that these latter are more vulnerable to this game element than the Portuguese students. Moreover, according to the flow theory (Csikszentmihalyi 2014), anxiety or boredom seem to influence the learning process. More studies are needed, as well as a larger sample, to understand this effect on the results better. However, when looking at the results individually, the PBL triad effect was higher in the UK than in Portugal. The main difference between both countries is not high regarding the effect but regarding the intensity of it. Although the Portuguese students presented higher learning outcomes, it seems like that the effect of the mobile gamified application was stronger among the UK students.

Are the game elements useful in learning?

According to Bogost (2015), even though most of the gamified applications follow patterns of feedback loops and rewards mechanisms to improve the most desired experiences, these systems pay no attention to what makes games powerful and reduce them to meaningless points, levels, and badges. To respond to this point, these game elements were used in a narrative context and also over a game mechanic that a previous study showed to be engaging for the children (Leitão, Maguire, and Turner 2019b). These elements could explain the improvement of the learning outcomes results with the support of the Ocean Literacy application.

Looking at the overall results, the Ocean literacy mobile application integration as a learning resource seemed to improve the acquisition of knowledge, compared with the oral explanations group. The overall results show the average of correct answers was over 70% with the mobile application (all conditions), while only with oral explanation and the slide show sessions, the average was less than 60%. These overall results show that knowledge acquisition through exploration, action and experimentation is more engaging and learning is more effective.

Future research

This research project included pupils 11-14 years old, so some results may not apply to younger children or older teenagers, although the general guidelines on educational games design on fun and playability are likely to be relevant. Research projects which include empirical evaluation in different grades could help to present a broader picture of the use of these technologies in the learning context. Further research is necessary to understand whether gamified applications have different effects on pupils in different grades.

Schools are not the only places where learning occurs, especially in the case of mobile learning applications. Learning takes place in many different contexts, including in terms of flexibility of time and location. Research is needed on how gamified applications can affect learning in contexts involving, for example, after-school programs, homework or outside of educational activities.

Although people’s poor understanding of marine science and ocean issues is identified as an obstacle to personal behaviour changes, behaviour outcomes were not investigated. Future research could be conducted to investigate whether this knowledge and awareness of the consequences of our behaviours can translate into urgency to act and support public policies.

Study limitations

Some limitations of the study should be addressed. The sample was rather small, and there is a risk of participant variables (individual differences between participants) affecting the results between conditions. Additionally, even though two almost equal-sized gender groups participated in the study, the size group in the UK was smaller and could explain some of the differences between the two countries. The study evaluated the effect of different game elements on ocean literacy outcomes. With this sample, it was not possible to find evidence to argue that the use of badges, points or the PBL triad is more effective than any others. Thus, further studies should be carried out with a larger sample to understand better if there are differences in its application or if gamification results from an overall experience with the system.

Due to the restricted availability of the schools to take part in the study, the pupils’ knowledge was not evaluated before they interacted with the research tool. Therefore, it is not possible to identify if the pupils’ ocean knowledge happened during the sessions or not. The differences in students’ prior knowledge may have led to the differences in the answers. To reduce this limitation, a control group session was included, which showed the results without using the game. Additionally, although a random selection of students was made, the selected sample came from a local secondary school in the UK (East Midlands), while in Portugal, the sample was made up of secondary school pupils attending classes at Junior University Summer School, Porto University. The Portuguese sample may have a more substantial science background than the UK students due to their academic aspirations, economic resources, and their closer geographical location to the sea. The metropolitan Porto area has a strong connection to the coastal area, extending through various economic and social activities, which may make students more aware of the ocean and its importance. Given the results described herein and the interest of the app to improve ocean knowledge, additional studies carried out with a larger, more heterogeneous and representative sample, will be helpful to better understand the differences in learning outcomes exhibited by UK and PT pupils.

Lastly, the sound was too loud on some platforms creating anxiety in some students. Therefore, some attention must be paid to the sound volume of the platforms, particularly in small groups, in order to reduce the pressure among the students. The solution in the future would be to place the sound volume under the pupils’ control so they could turn it down or off if they wished.

Conclusion

Considering the introduction of different game elements in the educational context, creating a rich gamified experience is much more than merely adding various game elements to existing products. Moreover, one solution cannot fit all contexts; it is an approach that requires a significant amount of analysis, game design thinking, testing and adjusting. Although digital technologies cannot replace authentic interactions with the ocean, the immersive features of these platforms can have the potential to involve and engage the students with the ocean. Furthermore, they are particularly relevant to address ocean literacy with students living far away from coastal areas, to whom live contact with the sea is seldom possible. Game features such as fun, beautiful and relevant aspects about the ocean, interactions, immediate feedback and the kinds of rewards seem to be the heart of gamified applications.

This study aimed to establish if there is any relationship between the use of different game elements, such as points, badges, leaderboards and the PBL triad, and learning outcomes. Although the samples sizes are not large, the study shows that different game elements can have different effects on the learning outcomes. To answer the research question: can different game elements impact the learning outcomes and the learning experience and, if so, how? Findings revealed that the use of points, badges and the PBL triad can positively affect the learning outcomes and the leaderboards as the main game element seemed to undermine them. Rather than focusing on rewards for motivation students’ learning, it is essential to focus more on how to develop more interesting and engaging learning activities, more autonomy and choice, ensuring that activities are optimally challenging. Stimuli should be sought that favoured collaboration over competition.

The analysis pointed to the same results in both countries regarding the interaction with the game elements, although the Portuguese students exhibited higher learning outcomes, and the effect was stronger on the UK students. This shows that the country did not influence the gamification effect, and the application helped to improve ocean literacy in both countries. Although care is needed when using leaderboards because of the type of competition that these can generate, gamified applications as a learning tool seem to successfully drive engagement, interaction, and the acquisition of knowledge is more effective, at least under these conditions. Many students stated that they felt they had learned a lot about the ocean during the test sessions. However, the Ocean literacy application itself is only part of the learning experience. Involving students and teachers during the class sessions to learn about the ocean environment generally would enhance the quality of the learning.

Acknowledgements

The authors are thankful to the student participants who took part in the study as well as the schools involved. We would also like to thank the Interdisciplinary Centre of Marine and Environmental Research (CIIMAR) for their help.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Funding

This research was supported by the Arts and Humanities Research Council Design Star CDT (AH/L503770/1). The research was also supported by project OceanClass - Do oceano para a sala de aula funded by the Blue Growth Programme of EEA Grants Portugal (EEA.BG.SGS3.022.2019), (2021-2023) and by national funds through FCT (Foundation for the Science and Technology) within the scope of UIDB/04423/2020 and UIDP/04423/2020.