Mini-games for entrepreneurship in construction: instructional design and effects of the TYCON game

ABSTRACT

Entrepreneurship is crucial for economic growth and employment, but conventional didactical approaches appear ineffective. Effective approaches should include experiential learning from real problems. The serious game under study was developed in the context of entrepreneurship training for construction workers (at European Quality Framework levels 3 to 5). The game offers an immersive scenario with players carrying out authentic tasks in authentic construction business contexts. The article describes important instructional design guidelines applied to assure that tasks and game mechanisms support such more experiential learning. An empirical study (with complete datasets collected from 97 participants across six countries) compared the professional awareness about entrepreneurship before and after gameplay, and analysed computer logging data on game behaviour. We administered a questionnaire after gameplay to inquire about game appreciation. Effective learning from gameplay could indeed be achieved by applying instructional design guidelines. The awareness gain from gameplay was significant (with mean scores on a knowledge test increasing from 45% to 61% correct answers). The games were appreciated (mildly) positive on important game features (flow, authenticity, motivation, usability, learnability and attractiveness). We discuss limitations of the study and recommendations for future research into professional games for awareness raising.

KEYWORDS:

• Serious games
• professional awareness
• experiential learning
• pedagogical scenarios
• authenticity
• construction

1. Introduction

Active learning through meaningful and playful practice has been shown to positively influence professional awareness (e.g. Boersma et al., 2010; Meijers et al., 2013; Sherman et al., 2008). One of the priorities and main VET challenges in the European Union are innovative technology-enhanced approaches to foster key competences in more effective and creative ways (EC, 2020). In this article, we describe the design, development and evaluation of mini-games for entrepreneurship in construction.

There is growing evidence that students experiencing professional challenges during training by playing scenario-based serious games also engage better within future careers. Such “professional games” may provide, when well designed, immersive learning scenarios where students learn to actively acquire and apply their knowledge and skills in authentic contexts. In that way, learning activities become active, relevant and motivating to them, and increase the likelihood that acquired knowledge and skills will actually transfer to real-world situations (Herrington et al., 2003; Hoekstra, 2011). However, many serious games appear hampered by poor instructional design and do not yield many learning effects. So, the main research challenges here are how to effectively design such professional games as more effective, technology-enhanced approaches, and how to obtain evidence that such approaches are indeed effective and appreciated in educational practice.

To address the first challenge, the remainder of this introduction is split into two parts: a short review of professional game-based learning (Section 1.1) and our survey on entrepreneurship skills for construction (Section 1.2). In these sections, we describe the main aspects to take into account when designing professional games of high didactical quality and explain which entrepreneurial competences are involved in the context of this study. The second section then describes the actual design of the mini-games, describing which important instruction design guidelines (from the 4C-ID model) and game mechanisms (using the functionality of the EMERGO platform) have been considered and applied in the game under study. To address the second challenge, the third section describes the setup of the study to evaluate their learning effect and appreciation in practice.

1.1. Game-based learning for professional awareness

Properly designed gaming or playful learning in context is recognized as an activity that can lead to better learning results and motivation when compared to more conventional learning approaches (e.g. Boyle et al., 2016; Garris et al., 2002; Wouters & van Oostendorp, 2013). A recent literature review (Zhonggen, 2019) on serious games over the last decade shows that numbers of empirical studies increased. Enjoyment and motivation were not found to be influencing factors for the learning effect, although this often is thought to be the case. The other way around this effect is reported as resulting from gameplay, next to an efficiency effect (All et al., 2015). Conditionally, researchers report the need for high-quality instructional design methods and evaluation frameworks for the quality assessment of serious games, and some models and frameworks have been suggested (Hainey & Connolly, 2010; Nadolski et al., 2008; Van Merriënboer & Kirschner, 2018).

Adequate relationships between learning attributes and gaming mechanics were found to be critical for the instructional quality of (game) design and desired learning outcomes, and some models for their mapping have been elaborated (e.g. Arnab et al., 2015; Carvalho et al., 2015). Important factors to concern are “whole task” design with subtasks of increasing complexity, decreasing support (scaffolding) and enough variability of practice (Van Merriënboer & Kirschner, 2018); gameplay in an engaging scenario (Nadolski et al., 2008); and authentic problem contexts for gameplay (Herrington et al., 2003). Embedded support, monitoring and assessment (also referred to as internal or stealth assessment) have been found particularly important for skill acquisition (e.g. Caballero Hernández et al., 2017).

Entrepreneurship is expected to respond to the requirements of unexpected events during future working life. The latter requires more and more creative solutions for complex problems and also more young motivated people who are able and willing to become entrepreneurs or possess entrepreneurial skills and know how to use them in different situations (EC, 2014). The need for innovative, TEL-based entrepreneurship education and training (EET) grows and governments support it. The most effective approach to develop entrepreneurship needs to include experiential and task-oriented learning from real business problems and context, e.g. through (virtual) “apprenticeship learning” (Hull University, 2015). This need is fuelled by a general recognition that entrepreneurship is critical for economic growth and employment and plays a vital role in developing more and/or more able entrepreneurs (Kuratko, 2005; Shane & Ventkataraman, 2000).

The specific key target groups in this study are VET apprentices in construction (at EQF levels 3, 4 and 5), their teachers and trainers, and the VET institutions and employers in the construction business. Traditionally, VET training for construction has focused on “hard” skills (like brick-laying and electric wiring skills) and was characterized by a rather hierarchical and “masculine” culture of “just doing without talking that much”. Throughout recent years, also caused by the growing complexity and multi-disciplinarity in the construction business, and other needs revealed by economic and pandemic crises, the importance of “softer” skills and “hands-off” skills (like communication and entrepreneurship) has substantially grown, where conventional “hands-on” training cannot always cater for this change.

1.2. Entrepreneurship competence (for construction)

Entrepreneurship for construction is related but different from leadership and organizational skills, which have been better researched. We carried out desktop research on existing EET offerings at national and local levels in some European countries and administered a survey with stakeholders to design a dedicated competence and assessment framework for entrepreneurial skills in construction. We concluded from our desktop study that there is an overall lack of national strategies and inclusion of entrepreneurship in educational education, although there are many different, isolated and local initiatives. Some initiatives attempted to research EET (EC, 2014; 2016b; Hull University, 2015) and harmonize initiatives for entrepreneurship across domains, but all with a focus on business and administration, like the EntreComp (The Entrepreneurship Competence Framework) project (EU, 2016a). Figure 1 provides the EntreComp “wheel” with entrepreneurship competences. One of the project partners previously led a project called SOCCES (SOCial Competences, Entrepreneurship and Sense of initiative) (e.g. Guilland et al., 2017) that was used as starting point for our survey. SOCCES distinguishes (five) main competences: “creativity and innovation”, “teamwork and collaboration”, “communication”, “critical and analytical thinking” and “positive attitude and initiative”.

Figure 1. The EntreComp “wheel” of entrepreneurship competence.

In early 2019, we carried out the survey with 144 stakeholders from five European countries (Italy, Germany, Spain, UK, Slovenia and Switzerland), of which 37 were teachers, 74 were students and 33 were employers in the construction business. They were asked to express preferences on the provided list of entrepreneurial skills, or add others if needed. The skills that received the highest scores (or were added most often) were placed in their “category” (EntreComp) and competence area (SOCCES). When the highest scores on their defining threads were calculated (in italics), the skills with both highest median scores and highest scores on defining threads were selected (in italics and bold).

Table 1 provides an overview of competences as identified by EntreComp/SOCCES, and the competences finally selected as considered most crucial for construction (and included in our gameplay). Note that we combined “Working with others” and “Mobilizing others/resources” as one competence. When we had selected the most important (five) competences, we determined their learning objectives and performance indicators at relevant EQF levels (for assessment purposes), also based on earlier EntreComp/SOCCES frameworks. For our project and intended game, this entailed intermediate (building independence) EQF levels 3 and 4, and advanced (taking responsibility) EQF level 5. We decided that each competence should be acquired in context (five case leads) at three levels, leading to an initial setup of having 15 mini-games (to be further described in the next section).

Table 1. Selected entrepreneurial competences for gameplay.

Entrepreneurial (sub)competences as identified (by EntreComp and SOCCES projects)
1. Ideas & opportunities	2. Resources		3. Into action
1. Creativity and innovation	2. Teamwork and collaboration	3. Communication	4. Critical and analytical thinking	5. Positive attitude and initiative
Spotting opportunity; Creativity; Valuing ideas; Vision; Ethical and sustainable thinking	Self-awareness and efficacy; Working with others; Financial and economic literacy; Motivation and perseverance	Mobilizing others; Mobilizing resources	Planning and management; Learning through experiences	Taking an initiative; Coping with ambiguity, uncertainty and risks; Working with others
Entrepreneurial competences for construction as identified and selected (by our survey)
Be accountable; Assess consequences and impact of ideas, opportunities and actions; Ethical and sustainable thinking	Inspire, engage and get others on board; Communicate effectively; Motivation and perseverance; Stay focused and don’t give up; Be determined; Mobilizing resources; Believe in yourself and keep developing; Self-awareness and self-efficacy		Learning through experience; Learning by doing; Planning and management; Taking the initiative; Working with others; Taking responsibility; Prioritize, organize and follow up.
4. Ethical and sustainable thinking	5. Motivation and perseverance		Taking the initiative; Learning through experience; Working with and mobilizing others

Table 2 provides an overview of case leads and (overall) learning objectives, based on Table 1.

Table 2. Overview of case leads and their learning objectives.

Competences/case leads	Learning goals	Criteria
Ideas and opportunities
1.5. Ethical and sustainable thinking (Case lead 4: Spain)	Be able to assess the consequences and impact of ideas, opportunities and actions in an ethical and sustainable way	Assess the consequences of ideas that bring value and the effect of entrepreneurial action on the target community, the market, society and the environment; Reflect on how sustainable long-term social, cultural and economic goals are, and the course of action chosen; Act responsibly
Resources
2.2. Motivation and perseverance (Case lead 5: Germany)	Stay focused and don’t give up	Be determined to turn ideas into action and satisfy your need to achieve; Be prepared to be patient and keep trying to achieve your long-term individual or group aims; Be resilient under pressure, adversity, and temporary failure
Into action
3.1. Taking the initiative (Case lead 1: Italy)	Go for it when you see the chance	Initiate processes that create value; Take up challenges; Act and work independently to achieve goals, stick to intentions and carry out planned tasks
3.4. (and 2.3.) Working with and mobilizing others/resources (Case lead 3: UK)	Team up, collaborate and network; Inspire, enthuse and get others on board	Work together and co-operate with others to develop ideas and turn them into action; Network; Solve conflicts and face up to competition positively when necessary; Inspire and enthuse relevant stakeholders; Get the support needed to achieve valuable outcomes; Demonstrate effective communication, persuasion, negotiation and leadership
3.5. Learning through experience (Case lead 2: Slovenia)	Learn by doing	Use any initiative for value creation as a learning opportunity; Learn with others, including peers and mentors; Reflect and learn from both success and failure (your own and other people’s)

The overview of entrepreneurship competences and their case leads formed the basis for further designing our game, which design will be presented in the next section. For this study, we had three research questions. We are interested in the quality of that game design for the effective acquisition and assessment of competences (like entrepreneurship in construction), as well as in the effectiveness and appreciation for this innovative learning approach in practice.

Research questions: The introduction leads us to three research questions for this study:

1. Can we apply important design guidelines (taken from the 4C/ID model for complex learning) in gameplay to achieve more effective learning and to monitor learning progress in entrepreneurship?

2. Does playing the game positively affect professional awareness of entrepreneurship in construction?

3. Is playing the game positively appreciated on important game features (flow, authenticity, motivation, usability and attractivity)?

The next section provides a description of the game in order to answer the first question. The third (Method) section describes how we set up an evaluative study in practice to find answers to the other two questions.

2. TYCON game for entrepreneurship in construction

Crucial for the instructional quality of our TYCON game have been instructional design guidelines taken from the 4C/ID model (for designing complex tasks) and from the EMERGO approach and toolkit (for elaborating and implementing our game designs). A first important research question for us was to see if we could design game mechanisms that assure the key competences can be acquired, monitored and assessed?

The 4C/ID model (Van Merriënboer & Kirschner, 2018) is a highly respected model for instructional design (ID). The model assures the effective acquisition of complex skills in simulated contexts and has been successfully applied to design courses and curricula in various countries and educational sectors. We know of more scarce attempts to also apply the model for game-based learning, like was done for a Mechatronics programme (Van Bussel et al., 2014), and even within EMERGO for a game about research methods and statistics (Van Rosmalen et al., 2014). Without the space to go into detail here, three important guidelines are that “whole tasks” should be grouped in task classes (dealing with the same competence), that consecutive tasks should be of increasing complexity and decreasing support, and that the collection of tasks should have enough “variability of practice”.

We finally implemented 12 mini-games (1.1–5.3), and Table 3 shows how their tasks relate to five case leads or task classes, their levels of complexity (EQF levels 3, 4 and 5) and support levels (from five stars ***** to just one star *). Short descriptions of game mechanics and difficulty causes give an idea of the variety of practice for achieving effective, attractive and playful learning.

Table 3. TYCON game: overview of cases, tasks and game mechanisms.

Case leads	Name mini-game	Game Mechanisms	EQF levels 3–5 (complexity)	Support level	Difficulty causes
1 Italy	1.1 Business opportunities	Simple selection game	3	*****	Easy estimation of business opportunities with a relatively large amount of support from sources
	1.2 Ayusha	Interactive dialogue	4	****	Relatively simple dialogues. The consequences of choices can be foreseen
	1.3 Padua Housing Project	Interactive dialogue + selection game	5	****	Limited support information on which business choices have to be made.
2 Slovenia	2 Smart House	Interactive dialogue + assignments	3	****	Increase in factors that play a role in business choices (when compared to scenario 1)
3 UK	3.1 Project planning	Drag-and-drop game	3	****	Trial and error project planning
	3.2 Team	Ranking game	4	***	Limited support in making choices. Own input becomes more important.
	3.3 Suppliers	Complex calculation	5	*	Complicated calculations on the basis of which a business strategy must be drawn up.
4 Spain	4.1a Target Groups	Selection game (simple)	3	***	Relatively simple statements
	4.1b Impacts	Selection game (medium)	3	***	Average statements
	4.2 The Logic Model	Selection game (complex)	5	**	Complex assignment in which earlier knowledge must be combined in order to estimate impact
5 Germany	5.1 Project Planning	Medium selection game	3	***	Follow up game on 1.1, but with less support and instruction
	5.2 Heritage Authority	Interactive dialogue	4	**	Complex questions
	5.3 Incidents at the worksite	Interactive dialogue with assignments	5	*	Increased amount of complex questions

The EMERGO approach and toolkit (Nadolski et al., 2008) were used to further elaborate the game scripts and to implement and evaluate gameplay. EMERGO (efficient method for experiential education) has been successfully applied for the development of over 70 scenario-based games for professional learning in various domains and educational levels. We followed the three steps of the EMERGO approach in further designing our mini-games on paper before implementing the final scripts in the EMERGO platform.

As overall narrative, we have “junior” (the student as player) start-up in a construction business that will expand across Europe (with junior becoming a “tycoon in construction”). The narrative starts in Italy (start-up doing small renovation project), continues with a more complex smart house project (Slovenia), an even more complex housing project (UK), railway construction impacting many stakeholders (Spain), and finally gets to Germany (a complex cultural heritage project).

Figure 2 contains (six) screengrabs that give an impression of the TYCON gameplay, where Figure 3 contains (another six) screengrabs that give an impression of the TYCON monitoring and assessment mechanisms.

Figure 2. Screengrabs of TYCON gameplay mechanisms.

Figure 3. Screengrabs of TYCON monitoring and assessment mechanisms.

The first screen shows how competences were mapped to five European contexts (or case leads). The second screen shows the gamer’s office (in Italy), while Mr. Buttinski is calling. He is the virtual supervisor (senior in construction) of our gamer (junior in construction) and will provide tasks, guidance and feedback throughout gameplay. The office also contains the tablet on which all resources become available, and work done is saved, as well as dashboard information on progress. Game mechanisms vary, but the third screen shows a dialogue with a game character on the construction site with an urgent problem, where the fourth screen shows the study of potential business plans. Junior receives mails and notifications from other (non-playing) characters to be followed up, like is shown in the fifth and sixth screens. You might notice that junior’s office (in the fifth screen) adapts to the size of the construction business.

The first screen shows feedback during a drag-and-drop task where “junior” has to decide on an adequate structure for talking with a subordinate construction worker who made a mistake. After every mini-game, student’s performance is compared with three non-playing characters representing poor, average and good performance (second screen, fictitious). When the task entails a conversation, feedback on the selected answers might look something like in the third screen. When the task entails planning, feedback might look something like in the fourth screen.

Figure 3 depicts a number of ways the student receives feedback on progress (in individual tasks and towards competences). We developed a scoring mechanism with 100 points for each competence. Although each case lead focuses on one competence (see Table 2), points for each competence can also be earned within other case leads. When the case contains three mini-games (Italy, UK, Germany), the distribution of points over mini-games is 25%−25%−50%; when there are two mini-games (Spain), this is 50-50; and when there is one mini-game (Slovenia), all points are earned at once. After completion of each case, the dashboard shows the amount of competence points earned and their distribution over mini-games (see fifth and sixth screens for the (fictitious) case of less than 20% of 70 points earned for “Taking the initiative” in Italy).

3. Method

Where the game design was quite complex, the setup of our evaluative study was relatively simple. Participants answered awareness questions before and after gameplay to measure learning effects and answered a questionnaire afterwards to determine user satisfaction with important game features. Pre- and post-measures were compared by running paired t-tests, and outcome effects were compared for co-variables like country and role (level) of participants. Satisfaction with game features was described by statistics and correlations, and also compared for different countries and roles (levels) by running ANOVA. We used SPSS version 24 to organize and analyze data. In this section, we now describe participants, procedures and instruments.

3.1. Participants

The participants (n = 97) were recruited by project partners through VET centres for construction: 30,9% were from Italy (on average EQF level 3); 14.4% came from Switzerland; 4.1% from Germany; 5.2% from Spain; and 23.7% from Slovenia (on average EQF level 4). Another 21.6% were construction students from a university in the UK (on average EQF level 5). The majority of participants (84.5%) studied the game in the role of trainee (student); a minority (15.5%) in the role of more experienced peer or trainer (on average level EQF 5). The relatively low numbers (percentages) from Germany and Spain were caused by organizational restrictions (caused by the ongoing pandemic).

3.2. Procedure

Authors provided account information that was linked to various countries and roles (but anonymized), integrated awareness questions (in six languages) within gameplay in the EMERGO environment (in six languages) and linked the LimeSurvey questionnaire (in six languages) from the EMERGO environment. This guaranteed that participants needed to answer questions in order to complete gameplay and assured uniform collection of data. The mini-games available for this study were the prefinal but complete 0.95 version of the TYCON game. Account information was distributed to various VET centres who administered accounts to concrete participants (assuring that personal information remained within centres). Participants could start and finish the game at any moment of convenience within a certain time frame (about two months).

A participant would enter the EMERGO environment with account info, first be asked to provide informed consent, then would first have to answer a (randomized) set of (ten) awareness questions (pre-test), after which they could start gameplay. After having completed 11 out of 12 mini-games, they were asked to answer another (randomized) set of 10 awareness questions (post-test) and finish gameplay. Computer logging data from EMERGO show that participants spent an average of 10 min contesting the awareness questions, and about two and a half hours on gameplay. Upon completion of the last mini-game, they would be directed towards a LimeSurvey questionnaire (containing 80 items). LimeSurvey timestamps estimate this time to complete the questionnaire to have been about 15 minutes on average. Data were extracted from EMERGO and LimeSurvey as Excel files. We normalized all data (all items having the same direction and scale) before comparing and analyzing them.

3.3. Instruments

We used awareness questions, a questionnaire and computer logging to collect data.

Awareness questions: An experienced VET trainer and project member drew up a set of (20) awareness questions about entrepreneurship for construction workers. Every (closed) question has five answer options, of which only one being the right one (r) and four being false (f). The items (and their correct answers) are to be found in Appendix 1. When we tested the internal consistency of these questions, we found an overall (Cronbach’s) α value of   .656, which is not “good” but still “acceptable” (e.g. Pallant, 2001).

Questionnaires: For other outcome measures (appreciation of gameplay features), we used a (validated) questionnaire with 80 items, each using a five-point Likert scale (with values from 1 = completely disagree to 5 = completely agree). We previously used the same questionnaire that had been based on other (validated) scales (Nadolski & Hummel, 2017; Hummel et al., 2020). When we compare the internal consistency measures for this study with those found for that previous study, we observe that Cronbach’s α values remain consistently “good” to “excellent”. We again used 23 items to measure perceived flow in online learning, 19 items to measure perceived authenticity, 13 items to measure motivation, 9 items to measure learnability, 6 items to measure usability and 7 items to measure the attractiveness of playing games. All questionnaire items were (re)calculated in the same direction (from “totally disagree” to “totally agree”), and maximum scores were standardized as 100% (or 100 points). Cronbach’s α values found were “good” to “excellent” for all five-point scales, respectively, α = .940 (previously .920) for flow, α = .974 (previously .937) for authenticity, α = .883 (previously .931) for motivation, α = .768 (previously .762) for learnability, α = .753 (previously .834) for usability and α = .850 (previously .811) for attractiveness. Based on the variance of scores, they all appear to have discriminative power. Average scores could therefore be used for further analyses.

Logging data: A dedicated research data component of the EMERGO authoring environment logged and extracted various gameplay activities as potentially interesting dependent variables. For this study, we have looked into total playtime (computer clock time converted to minutes) and in-game performance score. This performance score is based on monitoring all errors made during gameplay and later transformed into 0–100% scores, with a 100% score indicating that no errors were made by student (most efficient learning) and a 0% score indicating “random behaviour” (or less) by student (least efficient learning). Scores depend on the number of alternatives (nAlt) and the amount of errors (nErr) for each activity. All activities were monitored, including answering MC questions, multi-select tasks, categorizing items (using drag-and-drop tools), selecting text fragments and others.

4. Results

The presentation of our game design (in Section 2) showed that we can positively answer our first research question. By applying important instructional design guidelines for complex learning tasks (taken from the renowned 4C-ID model) and by using important functionalities for game mechanisms (as offered by the dedicated EMERGO platform), it appeared possible to design a high-quality game scenario for our purpose. To have involved entrepreneurial competences clearly established and mapped onto authentic tasks, making it possible to implement mechanisms to monitor and assess learning progress.

We described the setup of our evaluative study (Section 3) that addressed the second and third research questions. The actual learning and appreciation effects of a carefully designed professional game can only be empirically established in educational practice. In this fourth (Results) section, we further focus on our findings on awareness gain as a learning effect from gameplay (Section 4.1 in response to the second research question) and the satisfaction scores on game features (Section 4.2 in response to the third research question).

4.1. Awareness gain

The gain in awareness about entrepreneurship in construction as the effect of gameplay was expressed as deltas (difference) between participants’ average scores on post-test awareness questions (M = 61.03, SD = 25.84) and their average scores on pre-test awareness questions (M = 46.29, SD = 21.47), both percentages of maximum scores possible. This average 1,5 “grade-point” gain found (M = 14.74, SD = 26.34) to be appeared significantly positive when running a paired t-test, with t (96) = 5512 and p < .001.

Upon closer inspection of this learning effect, it was found to be more substantial for trainees (M = 15.12, SD = 27.40) than for more experienced workers or trainers (M = 10.67, SD = 20.62), but this difference for role (level) was not significant. Although participants from all countries positively benefited, average gains were found to be significantly different across countries, with F (5, 91) = 2635, p = .028, η_p² = .012 (small effect size). The average gains per country were around 13% in Switzerland, 12% in Italy, 25% in the UK and 17% in Slovenia. (We note that data for Germany and Spain are unreliable due to low numbers of participants.)

Overall, we see that trainers spent more time on gameplay than trainees and also achieved higher performance scores than trainees, but both differences are not significant. Both time spent on gameplay and performance scores for gameplay are lowest for UK participants when compared to all others (p < .001), so participants in this country appear to have been least motivated to engage in gameplay and/or the study. Paradoxically, their awareness gain nevertheless was highest (almost 25%) of all countries. The latter might be explained by relatively lowest pre-test scores (so relatively much room for improvement) and/or relatively highest education level (EQF 5) when compared to participants from all other countries. When we exclude the (23) UK participants from the dataset, the average time spent on gameplay increases from an average of 135 to 152 min, like we had envisioned it to be upon design.

4.2. Appreciation of gameplay features

Table 4 provides statistics on playtime, performance and appreciation of (six) game features, as were retrieved by calculating average scores on all (six) scales of the post-test questionnaire. Although variance on all measures appears to be very high, general appreciation across all game features is about 65% (“sufficient” till “more than sufficient”). Upon closer inspection, trainers appear to appreciate all game features less than trainees, with differences that only become (or almost) significant for Flow with F (1, 95) = 3469 and p = .066, for Motivation with F (1, 95) = 4335 and p = .040 and for Learnability with F (1, 95) = 3435 and p = .067 (all with small effect sizes).

Table 4. Statistics for main outcome variables (n = 97).

Outcome variable	Min.	Max.	M	δ (SD)
Playtime (minutes)	19.00	839.00	135.54	129.17
Performance	26.00	91.00	63.00	15.83
Flow Authenticity	20.00 20.00	84.40 99.00	60.45 66.93	15.74 18.35
Motivation	23.00	98.40	62.87	16.46
Learnability	20.00	93.40	63.59	12.86
Usability	20.00	96.60	62.61	17.81
Attractiveness	20.00	97.20	66.22	18.85

Again, all appreciation scores from UK participants were significantly lower when compared to other countries. When we exclude the (23) UK participants from the dataset, the overall appreciation of gameplay features increases from 63% (“sufficient”) towards 69% (“more than sufficient”).

Finally, we calculated Pearson’s correlations between all outcome variables (game time, game performance, knowledge gain and the scores on game features). All questionnaire scales are significantly related (with p < .01, two-tailed). Higher performance scores are positively related to more knowledge gain and game appreciation (on all features except for Motivation). However, the amount of time spent does not seem to contribute to any of the outcome variables.

5. Conclusion

We started by arguing why active and experiential learning in authentic contexts is the most effective and motivating way for acquiring professional competences. Serious games hold high potential when their instructional design is of high quality, and we described some approaches and models to safeguard this. We explained the importance of the relation between learning and gaming mechanisms, e.g. to monitor and assess progress on key competences during gameplay.

We described our design of a dedicated game that was developed to foster entrepreneurship competence in the construction business and could positively answer the first research question: Yes, it is possible to implement important instructional design guidelines when developing an entrepreneurship in construction game. We explained how we decided on (five) key competences, and how these were mapped on (five) task classes (Section 1.2). We described how design guidelines were applied within task classes for increasing complexity, decreasing support and enough variability of practice. We presented in-game mechanisms that enabled monitoring and assessing of progress on these key competences (Section 2).

We then described our evaluative study (Section 3), with findings that showed that the game was indeed effective (second question) and appreciated (third question) in practice (Section 4). Our study showed that participants from VET centres for construction indeed gain awareness on entrepreneurship by gameplay, so could also positively answer our second research question. This finding is in line with research, showing that meaningful and playful practice positively influences students’ professional awareness (e.g. Boersma et al., 2010; Meijers et al., 2013; Sherman et al., 2008). We did note high variances in both the time spent on gameplay (with an average of 2.5 h) and in-game performance scores (with an average of 63%), which variables were not found to be related. Differences for roles (levels) and countries were only found to be significantly different for UK participants, who spent the least time and achieved the lowest performance scores.

We could answer our third research question concerning the appreciation (satisfaction) of important game features (flow, authenticity, motivation, learnability, usability and attractiveness) in a moderately positive way. On average, these game features were not considered as “good” but rather as “sufficient” (and some as “more than sufficient”). We again noted a high variance within scores. Trainers appeared more negative overall, with significantly lower scores for perceived flow, motivation and learnability. Again, UK participants scored game features significantly lower.

Although this study has provided us with positive and promising answers to our (three) research questions (and expectations), there remain a number of concerns, interpretations and limitations that should be addressed in future research. First of all, the measures we applied for awareness gain and in-game performance were rather rough-grained. The internal consistency for the set of awareness questions was sufficient, but relatively low. We would need to develop more reliable measures for determining learning effects on knowledge and performance through gameplay (near transfer), but also need objective tasks beyond gameplay that could measure transfer effects, when applying the acquired entrepreneurship competences in other related (immediate transfer) or more different (far transfer) construction business contexts. It is promising to see that participants within the EQF 3 till EQF 5 proficiency levels (including trainers) could achieve significant gains from gameplay. Second, the overall appreciation of game features on average might be more “sufficient”, but is hard to interpret because of the high variance. A closer inspection of other computer-logged data might reveal why performance scores were high, e.g. by looking at which resources were opened for how long, which correct/wrong answers were provided when trying to solve tasks and others. A closer inspection of more qualitative findings obtained from three rounds of piloting (out of scope for this study), reveals that, although a large majority would recommend the TYCON game to peers, some were extremely positive where others were extremely negative. For instance, UK trainers have explained that demotivated students are a general problem, not specifically related to this study. Most positive comments were received for the appealing narrative and cases, and the effective support and monitoring mechanisms. Most negative comments were received about the linearity of the narrative (that should be more flexible) and the cases (which were not considered as realistic). Third, the total number of participants (n = 97) is sufficient but relatively low, and besides is characterized by an unequal distribution across countries (with numbers in two countries too low to be included for between-group comparisons). Restrictions caused by the ongoing COVID-19 pandemic made that some VET centres could not organize their participation (and numbers) as was planned. Although more persons have participated, we finally could only include the data from complete datasets.

We need to follow up on this study with larger numbers of participants and in other contexts, and also by focusing on other effective mechanics for such professional games. Since not all participants equally appreciated the level of linearity of the scenario and amount of control within gameplay, the relation between learner characteristics and learner control seems important. Landers and Reddock (2017) mention the importance of investigating interactions between various types of objective learner control and learner characteristics, or finding the optimal balance between system control and learner control. For instance, applying 4C/ID design principles in our game seems to hold potential for generic use across domains and educational sectors. Such and similar design principles for high-quality awareness-raising games have previously been successfully implemented and evaluated for games promoting awareness about future careers for youth (Hummel et al., 2018), awareness about European identity (Leith et al., 2018) and awareness about the work of psychologists (Hummel et al., 2020). Such games not only aim to increase awareness, but also to make students more interested in future work and more committed to continue their studies. In future studies, we would also like to include more qualitative data from learners in order to be able to explore (the variance in) their perceptions in more depth. Notwithstanding these constraints and still missing data, this study provides other empirical evidence for the enormous potential that professional games hold for more experiential education and further research in this area.

Acknowledgements

This study was made possible thanks to a grant from the European Commission to carry out the “TYCO(O)NSTRUCTOR: Mini-games that Foster Entrepreneurial Competence for the Construction Sector” project (2018-2021). The project proposal and grant number are Erasmus+ 2018-1-NL01-KA202-038926. The project website (with free access to the final mini-games in six languages) can be found at: http://tycon-project.eu.

The project was coordinated and administered by the Open University of the Netherlands who also was responsible for the TYCON game design and development, as well as for the setup of the study reported in this article (the authors). Many actors were involved whom authors would like to thank. We acknowledge Mrs Marlies Timmermans (OUNL) for all guidance on financial and administrative issues. Our (associate) project partners were: Coventry University (CU) – School of Energy, Construction and Environment (UK); West German Chambers of Construction (WHKT) (Germany); Training Centers for Construction (BZB) (Germany); School for Construction Vicenza Andrea Palladio (SCVAP) (Italy), Foundation for Construction Workers (FLC) (Spain); Chambers of Commerce and Industry (of Slovenia) (CCIS); and the University of Applied Sciences and Arts of Southern Switzerland (SUPSI) – Department of Business Economics, Health and Social Care (Switzerland). About 30 project members from eight institutions have directly contributed to the game development by providing the case leads and game content, or by contributing to brainstorm sessions. Without their creative ideas and collaboration, this project would not have been made possible, so thank you all.

An acknowledgement is extended to Mrs Amela Bogdanović (CU) who coordinated the desktop study and survey with stakeholders that led to the competence and assessment framework of the TYCON game (as described in Section 1.2). Another acknowledgement goes to our subcontracted graphical designer Mr Mick Hummel who developed most of the video and artwork for the game. He also arranged all (non-playing) actors and video voice-overs that have contributed to the dynamics of gameplay. Mrs Clara Ines García Ballesteros (FLC) arranged the centralized translation of all video and game dialogue texts by an agency in Madrid. Mr Hub Kurvers (OUNL) set up the Limesurvey questionnaire and link from EMERGO for data collection in this study. Last but not least, we would like to thank about 350 participants (from VET centres and schools for construction) during earlier piloting rounds and during this study, whose comments have helped us improve the quality of gameplay.

The main author has stored the (anonymized) data set (in both Excel and SPSS) associated with this article through DANS in the EASY repository, which dataset is available at: https://doi.org/10.17026/dans-xs7-7tsa. The Ethics committee at Coventry University approved the survey with stakeholders and participation from students under number P80202. At participating VET centers, the responsible officers approved the participation of their trainees in the pilots as part of regular training.

Disclosure statement

No potential conflict of interest was reported by the author(s). There has not arisen any financial interest from the direct application of this research and study.

Additional information

Funding

This work was supported by Erasmus+ [grant number 2018-1-NL01-KA202-038926].

Notes on contributors

Hans G. K. Hummel

Dr Hans G. K. Hummel works for the faculty of Educational Sciences as an associate professor with a research focus on technology-enhanced learning of workplace-based professional skills such as serious games. He is topic leader for “Applied Gaming and Simulation” and an application manager for the EMERGO approach and platform. Dr Aad Slootmaker works for the Expertise Centre for Education as a senior software developer with a focus on dedicated serious games for professional learning. He is the chief architect of the EMERGO authoring and playing environment. Mr Jeroen Storm at the time of the study worked as our colleague at the OUNL and now is an independent graphical designer with a focus on usability studies, serious games and mobile apps for learning.

Appendices

Appendix 1: Entrepreneurial awareness questions

1.	In order to establish, develop and manage a construction company, certain entrepreneurial skills are required. Which of the listed skills does NOT belong to these? □ Use resources responsibly (f) □ Personnel costs as low as possible (r) □ Believe in yourself (f) □ Creativity (f) □ Ethical and sustainable thinking (f)
2.	As an entrepreneur you must above all be active. What aspects are involved? □ Many employees as possible, earn and spend money (f) □ A big house, a fast car and a fashion model (f) □ Only organic food and branded clothing (f) □ Several holiday trips per year (f) □ Only learn what you are particularly interested in (r)
3.	Even as an entrepreneur you have to learn. What does that mean? □ Use only the knowledge from the master school or university (f) □ Mainly drawing knowledge from the Internet (f) □ Reflect on activities and learn from experience (r) □ Secretly spy on other entrepreneurs (f) □ Independent work, making decisions, taking responsibility (r)
4.	Which of the aspects for entrepreneurial success mentioned below is NOT correct? □ Construction (f) □ Opportunity (f) □ Profit maximization (r) □ Planning (f) □ Calculate risk (f)
5.	When developing a PowerPoint presentation to convince potential investors to participate in a new construction project, you have to take into account a few principles. Which of the listed principles is WRONG? □ Rational and concrete examples (f) □ Only 3 key messages (f) □ Good contrast between text and background (f) □ Accompany all slides with music or sounds (r) □ Strong introduction (f)
6.	In order to conduct a convincing sales pitch as an entrepreneur, there are a number of aspects you need to consider. Find out which of the listed aspects does NOT belong to them. □ Enthusiasm and empathy (f) □ Presentation of your service as a customer benefits (f) □ Preparation for queries from the customer (f) □ Persistence to get a definite yes or no (f) □ Not to listen to the customer, prefer to speak for yourself (r)
7.	There are two simple principles to negotiate with the customer. What are they? □ To be quiet and to know your price (r) □ Meet the customer immediately in terms of price and promise higher performance (f) □ To point out to the customer the scarcity of time and to highlight your high costs (f) □ Complaining to customers about the high costs and legal requirements (f) □ Not letting the customer have his or her say and taking him or her by surprise (f)
8.	To enhance the reputation of your company, you can use a simple method. What is it? □ Give the customer a valuable piece of jewellery (f) □ Ask the customer for a recommendation (r) □ Grant the customer at least 20% discount (f) □ Demand the customer to write a comprehensive report about your work in the local newspaper (f) □ Fix your company sign on the customer’s car during the work (f)
9.	To build sustainably, only certain building materials should be used. Which belong to these? □ If possible only solid concrete materials (stability) (f) □ As many plastics as possible (easy to maintain) (f) □ As much steel and glass as possible (modern design) (f) □ If possible only ecological building materials from regional production (r) □ Use as much tropical timber as possible (aid for South America) (f)
10.	For many construction projects, special machinery and vehicles must be used. What properties should they have? □ They should be as forceful as possible (speed) (f) □ They should be as cheap as possible (economy) (f) □ They should be as expensive as possible (values) (f) □ As few employees as possible should be able to operate them (responsibility) (f) □ They should cause as little environmental harm as possible (ecology) (r)
11.	An employee suggests something to improve work productivity. Which entrepreneurial skills are NOT addressed here? □ Entrepreneurial selfishness (r) □ Be curious and open (f) □ Identify, create and seize opportunities (f) □ Be innovative (f) □ Recognize the value of ideas (f)
12.	How should the entrepreneur behave towards a customer where an employee has made a handicraft mistake? □ He should deny everything to the customer (f) □ He should blame the customer himself for the mistake (f) □ He should tell the customer the truth and correct the error (r) □ He should tell the customer not to be so critical (f) □ He should declare to the client that he will fire the employee (f)
13.	To realize larger objects, you have to plan in much more detail than for the construction of individual houses. With which tool can you do this? □ With a special Linux computer (f) □ With a GPS bulldozer (f) □ By using Building Information Modelling (BIM) (f) □ With a Gantt chart (r) □ With a construction site diary (f)
14.	For all construction projects, you also have to deal with financial, tax and legal aspects. Which is NOT one of them? □ Cash flow calculation (f) □ Loan rate calculation (f) □ Penalties for fire safety violations (r) □ Organization of accountancy (f) □ Monitoring building processes (f)
15.	Playing the game, you will learn/you have learned about ethical and sustainable thinking and the so-called The Logic Model. What elements does this model contain? □ Recognize → wait → act → see (f) □ Request → processing → offer → order → implementation (f) □ Sunrise → work → break(s) → work → sunset (f) □ Resources/input → activities → outcome → outputs → impact (r) □ Recognize → learn→ test→ practice (f)
16.	In many construction projects, there are organizational difficulties. Which one is NOT part of it? □ The available processing time is very tight (only a few days) (r) □ The building object is located in a flood area (f) Situation □ The condition of the subsoil is unfavourable for the scaffolding (f) □ There are too many employees in the building office (f) □ The price of diesel is too high (f)
17.	Situation: one of your best construction managers has a private problem. How should you act as a responsible entrepreneur? □ Fire the manager without notice, hire another (f) □ Listen actively, ask discreet questions, offer help, if wanted (r) □ Assemble a team and give advice to the manager (f) □ First listen and explain that it is the manager’s private affair (f) □ Take the manager to a psychologist immediately (f)
18.	Situation: The completion of a construction project is delayed by the COVID-19 pandemic. How should an entrepreneur NOT act in this situation? □ Defining problems, developing and implementing ideas (f) □ Protect your own health and that of your employees and customers (f) □ Guiding and directing protective measures (f) □ Motivating customers to continue working through discounts and employees through bonuses (r) □ Analysis of the situation, make calm decisions, use opportunities (f)
19.	Situation: You receive an order that is actually too big for your company. What can you as an entrepreneur do to fulfil the order? □ Putting together a large team of own, but unqualified employees (f) □ Cooperation with qualified companies (networking) (r) □ Forcing employees to work faster (f) □ Where the customer cannot see it, deliver less quality (f) □ Perform the order only partially and then renegotiate it (f)
20.	Situation: In your company, an African skilled worker is mobbed by the customer. How do you react as an entrepreneur? □ You make it clear to the customer that you accept the diversity of people in your company (r) □ You apologize to the customer for the African (f) □ You ask your employees to frighten the customer (f) □ You accuse the African of provoking the customer and fire him (f) □ You act as if you haven’t heard about it (f)

Appendix 2: Questionnaire game features (scales and items)

Scale “Flow”

1. Playing this game challenged me.

2. Playing this game could provide a good test of my skills.

3. I find that playing this game stretches my capabilities to my limits.

4. I was challenged by this game, but I believed I am able to overcome these challenges.

5. I knew clearly what I wanted to do in this game.

6. I knew what I wanted to achieve in this game.

7. My goals were clearly defined.

8. While playing this game, I had a good idea about how well I was doing.

9. I was aware of how well I was performing in this game.

10. I receive immediate feedback on my actions.

11. My attention was focused entirely on the game that I was playing.

12. When playing this game, I was totally concentrated on what I was doing.

13. When playing this game, I felt in control over what I was doing in the game.

14. I feel comfortable with the controls of this game.

15. I often find myself doing things spontaneously and automatically without having to think.

16. When I play the game, I feel I am in a world created by the game.

17. I kind of forgot about myself when playing this game.

18. I lost the consciousness of my identity and felt like “melted” into the game.

19. When I played this game, I sometimes felt like things were happening in slow motion.

20. When I play this game, I tend to lose track of time.

21. Playing this game is rewarding in itself.

22. I loved the feeling of that performance and want to capture it again.

23. I enjoyed the experience.

Scale “Authenticity”

1. The game provided me with sufficient information about issues in psychology.

2. After playing this game I know better what to think about the work of a psychologist.

3. By playing I gained more insight into the issues involved in the psychological context.

4. The game allowed me to practice my skills with issues in the psychological context.

5. The game made me experience real-life issues and apply knowledge in context.

6. The game makes you study and apply the content in an active way.

7. The game learns you to apply your knowledge in a practical context.

8. The game urges me to reflect and take initiative.

9. The game allowed me to study content from various perspectives and to apply independently.

10. Within this game, I could determine whether I learned enough.

11. The game made me experience in a nice way what the issues in the psychology practice involve.

12. My awareness of the psychology practice was increased in an attractive way by playing the game.

13. This game in fact makes you experience problems with the psychology practice in context.

14. Playing the game made me feel more involved with the professional domain.

15. Playing the game makes you experience content and practice in context.

16. Playing this game taught me that practical problems require an integrative approach.

17. Playing this game taught me that solving practical problems is complex.

18. Playing this game taught me that there are no simple solutions to practical problems.

19. Playing this game taught me that practical problems often are not clearly laid out.

Scale “Motivation”

1. Enjoyed a lot playing this game.

2. The game was fun to do.

3. Playing this game was boring.

4. The game hasn’t got my attention at all.

5. Would like to describe this game as interesting.

6. Put in a lot of effort when playing the game.

7. Was important for me to do well while playing the game.

8. Worked hard when playing the game.

9. Believe this game has been useful for me.

10. Believe that playing this game will be important for my future life.

11. Think playing the game will help me in making future decisions.

12. Think this game is important.

13. Played this game because I didn’t have a choice.

Scale “Learnability”

1. The assignments in the game are complex and challenging enough.

2. It was not made clear enough what was expected of you to do for playing the game.

3. The game content misses relevant content.

4. The game lacks sufficient feedback.

5. The available feedback is very useful.

6. The game should contain more hints and help.

7. The explanations with the assignments were clear enough.

8. The game provides enough structure for efficient learning.

9. The game provides sufficient opportunity to test your knowledge.

10. The game provides sufficient opportunity to test practice.

11. Knew on what I would be assessed while playing the game.

12. Playing, learning and assessment are well integrated into this game.

Scale “Usability”

• 1. The options for operating the interface are well explained.

• 2. Operating this game can sometimes be a frustrating experience.

• 3. Operating this game is easy.

• 4. When operating the game, I had to spend much time correcting.

• 5. Feel I had enough control when playing the game.

• 6. The options for operating the game are according to my needs.

Scale “Attitude”

1. Found the game to have the right balance between learning and playing.

2. Flexibility in a game is important.

3. Games like this make the subject matter more interesting.

4. Games like this make the subject matter more understandable.

5. This game complies with what I feel an ideal game should be like.

6. For doing the assignments the game provided me with sufficient sources of information (documents, video, audio, etc.).

7. Feel playing the game takes too long.