The use of digital games in academic maritime education: a theoretical framework and practical applications

ABSTRACT

In search of a way to bring back the positive aspects of an internship into the curriculum, TU Delft identified serious games as a potential solution. The literature studied showed that games could increase motivation and understanding, leading to improve knowledge retention. This paper has brought these insights together in a framework identifying the benefits that contribute to knowledge retention but also the requirements and risks for the application of serious games to be addressed. These insights were used to explain the success of a longstanding business game course for advanced students first. Next was the development of a virtual reality practice in a first-year course to replace the lost internship. In this case, knowledge retention was improved significantly, albeit only by 5%. Furthermore, in light of the developed framework, the VR simulation is a more balanced approach with fewer risks compared to the more extensive master course.

KEYWORDS:

• Serious games
• gamification
• education
• economics
• technology

1. Introduction

About ten years ago Minors were introduced at the TU Delft, an academic education institute, to ensure that our academic students would receive a broader education and become better prepared for the more dynamic marketplace. A popular term for this is the T-shaped engineer Guest (1991), someone who can look beyond his domain and put his work in a broader perspective. It increases the potential to integrate knowledge and innovate. Important qualities, especially in today’s highly integrated world. However, to maintain the existing depth of knowledge, the Maritime Engineering education cancelled industry practices and reduced the experiments. Despite these best efforts, a loss was observed in the knowledge retention of our students by lecturers in later years. The contribution of practical experience to a broader education, beyond the practical experience, seems to be underestimated. A subject not often discussed in the literature. The available discussions indicate that practical experience may perform a key role in forming a frame of reference (Schunk 2012; Wenger 2011; Zimmerman 1983). This frame of reference is crucial as a background to anchor and integrate the lessons from courses. This background further facilitates knowledge retention by reducing the ‘newness’ of the materials studied by ensuring links to existing similar knowledge are created.

At the higher vocational level or Universities of Applied Science, maritime education consists of both officers’ training and practical ship design education. At this level, practical work is still a key part of education and especially simulators play an important role in the training of officers and port crew. The training from simulators has even resulted in flag states, like the Dutch flag, stating that under the condition of careful preparation and de-brief, simulator training is better than practice. As a result, it has been requested from IMO that one hour in the simulator counts for three hours on board a ship (HETI 2000; 2014). Given the apparent effectiveness of simulators in officers’ education, this idea may also work for the academic level. Could simulations perhaps replace practical experiences for improving knowledge retention, by forming the required frame of reference?

Considering that simulators are a branch of (digital) games, the goal of this paper is to investigate the potential of serious games to support a frame of reference equal to practical work and through that increase knowledge retention. At the moment, academic education in the maritime field (such as naval architecture, maritime economics and marine system integration), rarely makes use of practices or simulators to bring understanding to their students and games could be an efficient measure for this. Therefore, a key question for this paper is, if and how serious games can improve the learning of students.

After a discussion of the methodology in the next section (2), the theoretical background for this study is discussed (Section 3). The resulting framework is first tested on an existing course with a serious game, before being applied to the development of a simulation of practical work (section 4). This is followed by a discussion of the results from the case studies (section 5). While the final part (Section 6) provides a summary of the results, a discussion on contributions to literature and policy and recommendations for further research.

2. Methodology

To investigate the potential of digital games and simulations as a way to increase knowledge retention and as an alternative to practical work, the following methodology has been applied. A broad literature study on education was performed to gain insights into the key roles games can perform in education. The focus was on how these roles can support knowledge retention and insight creation. The literature search first focused on any game-related learning and how such methods impact learning. Papers published between 2010–2020 were reviewed for this purpose. In a second round, with a more profound understanding of different game implementations, the literature linking aspects of games to learning was further elaborated. Making use of references in relevant papers to dig further into individual aspects. This led to an overview of benefits, requirements and risks for the intended goal of knowledge retention. A summary of the key findings is presented in this paper only, as the focus is on the application in the case studies.

In the first case study, these insights from the literature were used to investigate an existing course that has already implemented a game. The focus was on the presence of the identified key elements and linking them to the success of the course. The second case study deals with the issue of the introduction and applies these insights to develop a serious game using these principles. In this case, the before and after situations provide a first insight into the effectiveness of serious games to improve knowledge retention.

3. Theoretical background

In the literature, three concepts related to games are found: Serious games, gamification and game-based learning. Serious games is a name used for any game used to introduce a concept to the players and specifically developed for that purpose. It is often used to change cultures or discuss new perspectives in a safe environment (Fox, Pittaway, and Uzuegbunam 2018; Malone 1980; Prensky 2001; Zhonggen 2019). Game-based learning is a game focused on learning certain materials. The difference is that in a serious game, the game itself is the learning goal, whereas, in game-based learning, the learning goals are supported by creating an unrelated game around it (Hwang and Po-han 2012; Ming-Chaun and Tsai 2013; Plass, Homer, and Kinzer 2015). The third term is gamification, which means the use of game mechanics to enliven the coursework. Usually, this is achieved without actually playing a game. A scoreboard for good behaviour in primary schools is an example, but also the election of an employee of the month, or earning credits and status with activities on your favourite forum. In addition, in this case, there is a clear overlap between this term and the other two (Deterding et al. 2011; Dicheva et al. 2015; Majuri, Koivisto, and Hamari 2018; Nah et al. 2014). An example to illustrate the differences; using a list of equations and asking a student to solve as many as possible in a set time is a form of gamification. However, executed online in a race simulation against other students would be game-based learning and finally being required to do many similar equations, although not explicitly, to take the right decision, could be part of a serious game.

The focus of this paper is on serious games. As Routledge (2016) indicates, games and serious games are used with success in many situations. Such games achieve objectives beyond the initial focus, especially in the form of skills like problem-solving. Despite these advantages, the amount of literature on the effect of serious games in academic engineering education is limited. Often computer games are linked to skills development (e.g. Mayer 2016) but these concern regular games, not serious games. As stated above a serious game has a lesson at the core, while a common game has entertainment at the core of the development. Even when broadening the search beyond serious games, research focuses primarily on motivation according to several overview papers (Caponetto, Earp, and Ott 2014; Deterding et al. 2011; Dicheva et al. 2015; Hallifax et al. 2019; Hořejší et al. 2019; Majuri, Koivisto, and Hamari 2018; Nah et al. 2014; Swacha 2021). These papers discuss the aspects that promote or hinder the motivation of the participants.

On the subject of motivation, almost all aspects of games are found to influence motivation positively. Immersiveness, direct feedback, awards and incremental progress are all examples of key positive enforcements from games. On the other hand, several researchers warn against the use of a leaderboard (Deterding et al. 2011; Toda, Valle, and Isotani 2018). This might result in disengagement for those not ranked at the top of the board. Another important aspect to be aware of is cognitive overload. The fact that students have a limited capacity to absorb new information (Sweller 1988), requires careful consideration when using games in education. Learning the game may take too much effort and result in not learning the intended knowledge. This is a key risk when applying games, simulators or even new software in education, the burden of learning the software is often underestimated by the expert users. When not properly addressed, learning results will be below the expectations or the required effort will far exceed the intended course load. The increase in motivation is welcome and the warnings are important for the applications of all games in education. As stated in the introduction, the focus of this paper is on the potential to increase the understanding and retention of the lessons learnt (Van Roy and Zaman 2018).

Deshpande and Deshpande and Huang (2011) do focus on this in one of the rare overviews of applications found in the literature. Their focus is on simulators, which could be seen as a subset of serious games, aimed at practising certain (complex) skills. Hořejší et al. (2019) discuss the creation of a management and simulator game used in mechanical engineering education at their institute but gives no insight into the impact or success. As mentioned in the introduction also in maritime academies or higher vocational institutes, simulators are used for skills training. Moreover, simulators Charles et al. (2020) and Markopoulos et al. (2019) also discuss the use of VR to train crews in safety and eco-efficiency awareness.

A clear link between games and improved understanding as expected was not identified within the discussed literature. However, considering the broader theories on learning, such a link seems likely. Games and simulators are in essence similar to practices and focus on learning by doing and explorative learning. As recognized by Collins (Collins, Seely Brown, and Holum 1991; Collins and Kapur 2006), we are used to learning skills through apprenticeships. A long time ago, parents and other adults thought you everything in this way. Nowadays you still learn from them, but much of the general skills are formalised in schooling. It is still very effective as it allows the apprentice to look into the head of the master. It not only includes the transfer of information required, but also the sequence required, the reasoning and linking details to their role in the more global system. It is also highly social, which increases motivation. It is however not very economical. In the work of Collins (Collins, Seely Brown, and Holum 1991; Collins and Kapur 2006) these aspects of apprenticeship were studied. A set of 17 principles found in a normal apprenticeship that can be applied to create a cognitive apprenticeship were proposed. A cognitive apprenticeship is a more economic classroom application of the beneficial aspects of an apprenticeship. Many of these principles are not new (Ausubel, Donald Novak, and Hanesian 1968; Black and Wiliam 1998; Dunlosky et al. 2013; Reigeluth and Stein 1983; Rosenshine 2010; Wood, Bruner, and Ross 1976; Craik and Lockhart 1972; Bloom 1984; Ausubel, Donald Novak, and Hanesian 1968; Black and Wiliam 1998; Dunlosky et al. 2013; Reigeluth and Stein 1983; Rosenshine 2010; Wood, Bruner, and Ross 1976; Craik and Lockhart 1972; Bloom 1984), but are linked here in a constructive way. These key ideas will be briefly discussed.

As demonstrated by (Dunlosky et al. 2013), distributed practices and practice testing or self-testing are the best techniques for long-term knowledge retention. Distributed practice is doing something for a short time, with time (e.g. days) in between. Practice-testing is (on multiple occasions) retrieving knowledge from your memory. Both activities are found in industry practice and games, as tasks will be executed multiple times and the results will be assessed by the student and the mentor. This repetitiveness also supports the long-term storage of the knowledge (Craik and Lockhart 1972)

As Reigeluth and Stein (1983) already discovered almost 40 years ago, it is important to relate each of these practices to the global overarching idea and to variate in the application, showing how a technique transfers to other situations, as well as varying in complexity. The relative long timeframe of an internship offers sufficient time for this. When designing it in the form of a game or simulator, special attention needs to be paid to this aspect. Although in essence, studying the whole issue in this way, will increase knowledge retention as a frame of reference is expected to quickly build up on the slightly simplified, but complete issue to be studied (Ausubel, Donald Novak, and Hanesian 1968). This helps to place further elements to be learned in the right context, which in turn facilitates learning.

Another aspect shared by both a practice and a game is that in principle it is a form of self-assessment or formative assessment (Black and Wiliam 1998). As lecturers, we usually only grade the presence, as the students discover the effectiveness of its approach in the execution of the task and are, in most cases, able to assess the level of success achieved themselves.

The final benefit found in practices and apprenticeships is the fact that you are working with small groups and offering tutoring on a one-to-few basis (Bloom 1984; Wood, Bruner, and Ross 1976). This greatly enhances the learning experience. In such a situation the tutor is much better able to switch between instructional methods and offer the right content (knowledge, approach, technique) at the right time. A process that is known as scaffolding (Rosenshine 2010).

Based on this overview of the literature Table 1 was created. The key benefit of serious games for academic education would be higher motivation, especially towards the end of the period (Van Roy and Zaman 2018). Motivation supports learning and lessons are remembered better also due to the experience-based learning and self-assessment options. Finally, as the whole can be studied more easily than in a regular class, a frame of reference is created, which not only supports the current class but also transfers to other subjects as the interaction between them is better understood.

Table 1. Overview of benefits, requirements and risks for successful learning with serious games.

Benefits to learning	Requirements	Risks
Higher motivation	Distributed practising	Cognitive overload
Experience-based learning	Scaffolding	Leaderboard disconnection
Frame of reference	Direct feedback
Self-assessment

To achieve these benefits, the serious game should offer distributed practice, meaning that some form of repetition and variation is required (Dunlosky et al. 2013). Furthermore, the feedback should be direct to support the self-assessment (Black and Wiliam 1998). Mistakes or misconceptions should be clear soon after they are made and not at the end of the course in the exam. Finally, the setting (instructor or in-game) should provide support at the right moment to ensure that maximum learning is achieved (Rosenshine 2010).

Finally, two key risks from gamification were identified as well. A leaderboard might lead to demotivation and disconnection for those not able to achieve a top ranking and might therefore work counterproductive. Furthermore, cognitive overload could become an issue if the game is complex. Learning the game may take too much effort and lessons are not picked up due to that.

Using this framework two implementations of games within the Maritime Engineering curriculum of TU Delft are discussed and evaluated. The use of a serious game to learn about the link between economics and technology towards the end of their studies is a long existing course and is used to validate the framework. The use of VR simulations to learn about shipbuilding at the start of their studies is a new development, created using the insights of this paper and addresses the practice reduction from the introduction.

4. Case studies

For each case study, the situation will be briefly described, followed by the game development (linking to the benefits and requirements) and completed by a short discussion on the impact. The first study is on an already existing situation. As result, the evaluation of the effect cannot be done directly, as the prior situation is not available anymore. Still, it serves as a validation of the framework and provides insight into the useful mechanics of the course. The second situation is the update of an existing course with measurements taken before and after the update. This will provide insight into the potential of games to increase knowledge retention, by comparing the before and after situations.

4.1. Case 1, business simulation in master education

The first case study considers a game that was developed in cooperation between the University of Antwerp (UA) and TU Delft (TUD). The objective of this course is to teach students about the link between economics and technology. The game is provided to both maritime engineering and maritime economics students, as this aspect is relevant for both. In general, students often do not appreciate subjects that are outside of their core curriculum, as they find it difficult to understand the relation (Everingham, Gyuris, and Sexton 2013; Strauss and Grant 2018). This certainly holds for teaching economics to engineering students, based on the author’s experience. The game approach was introduced to increase the motivation for the subject.

In the game, students take on the role of a ship owner and will have to deal with the day-to-day business. Decisions on buying and selling vessels, bidding on contracts, or changing operational areas are all part of the game. The course requires and applies knowledge from earlier years and is provided towards the end of their study. The course that is already present for over 5 years and thus a longer time reference is available. This includes data on performance and evaluations. However, specific data and questionnaires on e.g. motivation are not available and will need to be studied via proxies.

It is a popular course which has led to cooperation with other curricula. As a result, the course is now offered at several maritime education institutions, allowing for an insight into the impact of background on the experience of a game and the results. The course is provided in the following programmes: at the UA it is offered in two master programmes (MLM and C-MAT), while the TUD offers it in one programme (MT-MSc). It is also provided at Erasmus University Rotterdam as part of their Maritime Economics and Logistics master (MEL) and as part of an exchange programme for Maritime Technology master students from the Nanyang Technological University Singapore (NTU). Furthermore, three universities of applied science, Rotterdam Mainport Institute (RMI), Nationale Hogeschool Leeuwarden Stenden (NHL) and the University of Southeast Norway (NSU), also have adopted the course as part of either a minor or their final year programme. At NSU the course was only offered for two years. The fact that the course is provided to students from different backgrounds (technical and economics, higher vocational (applied science) and academic levels), also offers a potential to investigate the cognitive burden as well as loss of motivation.

Before addressing the results, the course is discussed in more detail, to be able to identify the aspects of the framework. The maritime business game was developed as the core of a course to achieve the goal of linking economics and technology. In this course, students take on the role of an aspiring shipowner. As can be seen in Figure 1 the students perform four activities. A trial run (1) to familiarise yourself with the game. This should lower cognitive overload as time is spent in the game before the actual application. The creation of a business plan (2) to study the game world and identify a suitable strategy. The wealth and depth of data are close to what companies have access to and exceed what students normally can access when writing such a plan. Creating a more concrete and realistic experience. Playing the game (3) to test their strategy and writing a year report to reflect on their performance (4). Although most time is spent on the game and the key lessons take place in playing the game, it is a clear form of formative assessment. The direct feedback of the game and the competitive environment and relatively great variation in uncertainty forces students to reflect and discover the economic and technical relations themselves. Finally, the small groups allow the tutor to visit groups frequently and offer advice and structure (Scaffolding).

Figure 1. Set-up of the maritime business game.

In the end, the profit earned only counts for 10%, whereas over 70% of the time is devoted to the game. On the other hand, the two mentioned reports (business plan and year report) form the summative assessment and constitute about 90% of the grade. This is further supported by a clear assurance that failure in the game is not an issue, only not understanding why you failed is.

The assessment of the achievement of the goals (motivation) as well as the mitigation of the risks of disconnection and cognitive overload has been assessed based on the results within the game and the feedback received from the students both in writing and in their grading of the course. Admittedly this is an ex post facto measure and ideally several questionnaires during the course could provide more detailed insight. However, another measure would only relate to one year and using these proxies more years are available for comparison. Furthermore, it is not possible to compare motivation to the situation before this course (it is provided since 2010 in its current form). Unfortunately, as each institute offers different forms of evaluation, the results are indicative but can be compared to other courses at the same institute.

In Table 2 the results from the course evaluations are given. Where possible converted to the grading of 1–10 with 10 being the highest and 1 the lowest. As can be seen over the years the format of evaluation changed within several programmes. Overall the course receives a high appreciation from the students, especially when compared to other courses within the curriculum (e.g. at TU Delft the average rating for obligatory MSc Courses is between 6.5–7.0, at the UA this is around 8.0 for master courses).

Table 2. Summary of student evaluations of the course.

#		2019–2020	2018–2019	2017–2018	2016–2017	Average
1	MBG-TUD	8.4	7.3	7.3	8.1	7.8
2	MBG-UA1	8.8	8.7	9.1		8.9
3	MBG-UA2	9.2	Text feedback, generally more positive than average courses			9.2
4	MBG-MEL	Text feedback, generally more positive than average courses			7.6	7.6
5	MBG-NHL	Not shared in detail, generally positive
6	MBG-RMI	Text feedback, generally positive and in line with other courses in the minor
7	MBG-NSU	Text feedback, very positive and slightly higher than in other exchange courses

All evaluations also offer options for the students to comment on the game, the lecturer, the course structure etc. Based on the evaluation of these comments there is a difference in cognitive load between higher vocational and non-western students versus academic western students. The latter group seldom identifies the complexity of the game as an issue to be solved (<5% of the comments), while within the first group remarks on the complexity and difficulty are much more common (10–30% of the students remark on this. In the view of the author, the large amount of information, in the form of lecture notes and online videos, as well as the large complexity of the game, is less aligned with their regular form of education. To address this, more direct guidance in the videos is implemented, as well as tutorials on the game itself, offering more support to reduce the cognitive load and increase the focus on the content.

Due to the COVID-19 situation the year 2019–2020 was primarily online and this had some impact on the flexibility for offering scaffolding as being in one large room, offers a situation for more frequent interaction. This was observed in the feedback received from the various lectures, showing more frequent measures indicating a cognitive overload. Based on counts of these mentions as fractions of the total counts of feedback submitted, the increase was found to be doubled, from around 10–15% the year before to 25–30% in 2020.

When considering the game results, these can be roughly categorized into three groups, those that excel (achieving a profit), the average group (losses of 1–3 USD mln) and those performing badly (>3 mln USD in loss). Sample sizes vary between 3–20 groups. Based on the experience of the author, the groups that are in the worst performing group often experienced a disconnect.

The classes at UA1 and NHL are on the small side (3–6 groups) and TUD, UA2 and MEL are on the larger side (20–25 groups). The overview in Table 3 presents the division for each year and study. The first percentage is the group with a loss of less than one million, the second is a loss of one to three million and the last percentage is a loss larger than three million. In general, the smaller group of NHL performs very well, but those of C-MAT do not excel. This could be cultural, but also the set-up and time for the course may be of influence here. C-MAT has a very full programme engaging students for 30–35 hours in the week, whereas NHL offers more free space and even tutoring in between the classes.

Table 3. Student performance pre-covid and during covid.

#		2019–2020	2018–2019	2017–2018	2016–2017
1	MBG-TUD	43%, 38%, 9%	50%, 25% 25%	42%, 34%, 23%	50%, 25%, 25%
2	MBG-UA1	17%, 17%, 66%	0%, 50%, 50%	0%, 33%, 67%	0%, 33%, 67%
3	MBG-UA2	36%, 28%, 36%	20%, 20%, 60%	0%, 29%, 71%	0%, 25%, 75%
4	MBG-MEL	12%, 12%, 76%	15%, 70%, 15%	15%, 70% 15%	57%, 7%, 36%
5	MBG-NHL	50%, 25%, 25%	50%, 25%, 25%	N/A
6	MBG-RMI	6%, 29%, 65%	5%, 55%, 40%	0%, 30%, 70%
7	MBG-NSU		11%, 11%, 78%	N/A

In general, the larger courses of TUD, UA2 and MEL perform well, although UA2 had some lower excellence rates in the first two years. At that time, law students were still included in the course and it seems these struggled more with the economic concepts. The larger higher vocational group of RMI seems to struggle more with the game concepts and for USN data of one year was lost, leaving only a single data point, which is close to the regular performance.

As this course is in the last year of most of the studies, knowledge retention cannot be explicitly verified as contact with the students is lost after their graduation. Any voluntary survey would be biased if not almost the entire group fills it in. This was therefore not pursued. Currently, any insight on this is solely based on random encounters with students years after their graduation. In those cases, many still recollect their performance in the game, which is striking, but not a sound scientific base for proof of excellent retention.

4.2. Case 2 practical work in bachelor education

The second case study is related to the use of a combination of practices to mitigate the loss of practical internships in the industry. As indicated in the introduction, over time the bachelor and master programme in Marine Technology has had several revisions. As can be seen in Figure 1 this has led to a decline in practical work over time. In the past, two internships (2 weeks and 10 weeks) were provided. During these internships, students would work in a shipyard as a metal worker (2 weeks 1^st year BSc) and as an engineer (10 weeks 2^nd year MSc). At the moment only 2–3 welding exercises remain. This decline in internships was never desired but rather a necessity as the class size had almost tripled in the same period and as stated also in Figure 2 the amount of steel construction still executed in the Netherlands strongly declined as well. Furthermore, the introduction of Minors led to extra pressure on the content to teach the core content. There were simply not enough possibilities to perform these internships anymore.

Figure 2. An overview of practical work in the marine technology curriculum.

A course in the first year on ship production was chosen for the creation of a virtual alternative. The main goal of this course is to provide insight into ship production activities, as well as introduce the names of parts and elements of the ship. Ideally, a frame of reference is formed by the students, of whom most have not yet seen a shipyard. The goal of this case is to increase knowledge retention and insight of the students through the use of simulators and gamification. Therefore, the set-up presented in Figure 3 was created. It was based on a discussion with students that had taken the internships in the past and designed to cover the lessons learned as they jointly identified them. In this paper, only the end result is presented, but the process consisted of several discussions in different groups to ensure all relevant aspects were addressed. Besides the elements of this overview, there is regular coursework on ship production and construction.

Figure 3. Blended set-up as an alternative for an internship.

Instead of one two-week internship at a shipyard, the students are now presented with a mix of exercises and practices to establish the same learning outcome. The practical work consists of three exercises to learn the limitations of the process, the difficulty of performing it correctly and the time division between pure execution and other aspects to take care of. Alone these practices do offer some reference for later courses but fail to put the work in perspective. To support this both a set of VR exercises have been developed as well as a set of virtual excursions in 360-degree videos. Both form an immersive environment, supporting the perspective and the discovery of the impact of size on all these activities. Where the VR exercises still focus on a set of detailed tasks performed at shipyards, the virtual excursion, offers the global picture and relates all aspects to each other. Finally, the yard simulation would support the understanding of the mutual impact each production phase has on the others. Here the students can manipulate the amount of equipment and personnel and attempt to optimise the total output for their shipyard.

Considering the literature discussed and the description above, it should be clear that any single aspect of the set-up would fail to achieve the intended learning outcome. It is the strength of the combination of exercises that should ensure reaching this goal. The virtual excursions and yard simulation, support building a global representation, whereas the practical work and VR exercises support the more detailed experiences and discovery of issues not visible at the higher level. For these two, the role of the supervisor is crucial, as they need to provide the scaffolding, not only for the exercises themselves but also for linking them to the higher-level image of production and construction. At least all digital aspects contain the crucial gamification elements to support immersion and thus support motivation for learning. Although not mentioned here, the course also contains a yard visit towards the end. The idea here is that with a larger frame of reference from the exercises the students see a lot more during the excursion. Unfortunately, as this development was only implemented in the curriculum of 2020–2021, the excursion was not possible due to COVID-19 restrictions.

As the goal was to create a frame of reference to improve knowledge retention, the key evaluation of this course is comparing the knowledge retained after 1 or 2 years. At the moment only the first assessment in year 2 can be compared. The assessment is a knowledge test in week one of the follow-up course entirely based on the content of the first-year course. In the base case, none of the students had followed the first-year course in the new blended practice set-up. In the second year, about half of the students had followed the new course and the other half had still followed the old course the year before. It should be noted that this is not uncommon within the Netherlands, where the average duration for a bachelor’s diploma is around four years.

As a result, the set of students who followed the new set-up was 33 students, while the set of students that did the previous set-up was 102 students. As can be seen in Table 4, the average of the group has improved and also the minimum and maximum are higher. The improvement of 5% is limited and a larger increase was expected. With (P(T<=t) = 0.041) for a two-sided student T-test the difference is at least significant (at the 5% level).

Table 4. Results of the knowledge retention test in year 2 of the study programme.

	Without VR	With VR
Average	45.5%	50.5%
Max	68.3%	77.5%
Min	9.6%	30%
N	102	33

As indicated this assessment is limited as it only tests one type of knowledge and the sample size is relatively small, with only about half a class in the assessment. Furthermore, the improved frame of reference could contribute to other aspects as well, such as higher final scores, shorter study duration, and fewer resits to name a few. However, the verification of this, requires large data samples, as the changes in tests and variation in students will have an impact as well. Given the recent introduction of this course, such a study is not yet feasible.

5. Discussion

As indicated by Routledge (2016) serious games are powerful tools to increase understanding and insight. However, they also require more complex issues or skills to be useful. As indicated in the literature research, without this, it is not a serious game or simulator, mimicking reality, but a game-based learning tool. This implies it is important to select this approach for suitable subjects, not all courses lend themselves to serious games or simulators. Especially the latter is focused on obtaining a skill or handling perspective and this needs to be present in the learning goals. Furthermore, the applications indicate that serious games in education can be a motivating way for students to engage with the materials. It also matches very well with the blended and flipped classroom approaches and thereby also seems to lend itself to remote and hybrid education. Albeit based on the experience of the MBG during the COVID-19 pandemic, cognitive overload is more likely to occur. In addition, more effort was required from the lecturer in larger courses.

Besides serious games, gamification might offer similar benefits to courses but could be more broadly applicable (Arnab et al. 2015; Caponetto, Earp, and Ott 2014; Deterding et al. 2011; Dicheva et al. 2015; Hallifax et al. 2019; Majuri, Koivisto, and Hamari 2018; Malone 1980; Nah et al. 2014; Ofosu-Ampong 2020; Swacha 2021; Van Roy and Zaman 2018). Especially considering the aspect of motivation over time as shown by Van Roy and Zaman (2018). A similar investigation for the use of games might be interesting, as especially in the MBG the evaluations show that students feel somewhat overwhelmed at the start, perhaps triggering a fight or flight reaction.

Within this paper, the theory of game-based learning was extended with a framework to increase knowledge retention using games or simulations. The framework was verified on an existing course with a game and applied to develop a new simulation for first-year students. The summary of the results is presented in Table 5.

Table 5. Evaluation of theoretical application in practice.

Benefits to learning	Maritime Business Game	VR-Practical
Higher motivation	High appreciation for the course	Not verified
Experience-based learning	Implemented	Implemented
Frame of reference	Assumed, but cannot be verified scientifically.	Supported, but the impact on knowledge retention is limited
Self-assessment	Implemented through the year report	Implemented through short discussions
Requirements
Distributed practising	Implemented with variation, the effect is not studied, due to the absences of follow-up courses	A build-up is implemented, the effect is studied at the start of follow-up courses and showed a slight improvement in the overall course content.
Scaffolding	Provided by the lecturer, not by the game	Provided by the lecturer, not by the game
Direct feedback	Implemented	Implemented
Risks
Cognitive overload	Present in all groups, although more significant in higher vocational and non-western groups	Not observed during the practice and not mentioned in discussions afterwards.
Leader boarddisconnection	There is a loss of motivation for those performing very badly, however, based on interviews, they disconnect already before the game is played.	No direct competition was implemented, as a result, there is no leaderboard and therefore no danger of disconnection.

Based on the experience in the COVID-19 period in the case studies, serious games in a hybrid or online education setting offer effective support for the blended and flipped classroom and should be considered an addition to regular curriculum activities.

From this table, the key lessons are that aiming for the benefits and meeting the requirements are achievable. There is a small increase in knowledge retention, but the assessment of motivation could be further elaborated. An in-class survey might be suitable to gain further insights (Koh et al. 2021; Pallis and Ng 2011), but will need to be developed and validated first. On the subject of risks, cognitive overload is less likely to occur if the applied game is small and simple, but should be monitored and addressed if games become more complex like in case study 1. Disengagement may occur due to this, limiting the insight and causing a drop in motivation.

6. Conclusions

6.1. Summary of findings

This paper identified a framework to explain/identify how serious games can contribute to knowledge retention in higher education. The framework was tested on an existing course that uses a complex game, showing that benefits and requirements identified were present, Risks identified, were recognised and present as well and required active approaches to mitigate.

After this, the framework was used to set up a new course with a serious game in the form of a VR simulation. With pre and post-measurements it was shown that retention was improved by 5%. This difference was significant, but a larger impact was expected. Furthermore, the risks were absent as the serious game was a much smaller aspect of the course.

As a first conclusion, serious games seem to benefit learning in certain applications and smaller games pose fewer risks than larger games. If the increase in retention is also linked to the complexity of the game was not yet investigated.

6.2. Theoretical and policy implications

For theoretical implications, this is one of the few studies that focus on maritime higher design and engineering education instead of its sea-going officers’ training. It is one of the first studies linking knowledge retention to serious games in education in general as well. To do this an initial framework was developed linking beneficial factors for retention to game elements. Secondly, empirical success was demonstrated by the application in a new situation. Offering statistical evidence of increased retention.

Concerning policy implications, the framework developed can be used by other institutes to validate and create serious games in courses. Faculty members will thus be able to select the right mechanisms in their games to increase retention. The actual size of the game, a full course or just an afternoon session, does not matter in that respect.

Secondly, the case studies offer insight into measurements to track impact related to the aspects of the framework. This will allow faculty members and their institutes to track and trace impact, adjusting their work over time, as we are doing at the moment. Adding specific questions related to e.g. cognitive overload in student surveys, could be very relevant, not only for serious games but also for regular projects or practices.

Lastly, especially the requirements and risks identified, demonstrate the importance of the supervisor for a successful application of a serious game. A supervisor will need to be able to provide a mix of group and class tutoring, recognize signs of disengagement and ensure a reconnection to the topic. In addition, due to these limitations, integrative and explorative learning goals are more suitable for gamification. In short, games are not suitable for every situation, the lecturer and content need to align with the requirements.

6.3. Limitations and recommendations

The study also has some limitations. The current results are only based on a limited set of evaluations. Continued monitoring of the results will improve the insights. The development of potential ways to directly monitor motivation, without intrusiveness, would increase the reliability of the current conclusions as well.

Secondly, a further investigation of the differences between the MBG courses and identification of the root causes of these variations would be interesting. Through such studies the suitability of serious games in education could be specified further, extending and improving the current framework.

Finally, this potential link between the complexity and size of the game and knowledge retention could be investigated further. It would be a valuable insight into the costs of game development versus the potential benefits. To investigate if large games are more effective for knowledge retention than a set of small games.

Disclosure statement

No potential conflict of interest was reported by the author.