Gamification in medical education: identifying and prioritizing key elements through Delphi method

Figures & data

Figure 1. Study process.

Table 1. The description of the elements which used in the gamification questionnaire.

Elements	Description
Integration with training goal	Each activity designed for training gamification should be closely integrated with the training goals for the purpose of learning, not for fun or playing games.
Game rules	Clear game rules should be set up during the gamification process for the learners to follow.
Rapid feedback	In response to the learner’s performance, set up a rapid feedback mechanism and offer points, praises, chips, or even material rewards. The key is to provide feedback in a timely manner.
Fairness	Fairness is very important in training gamification, when points and rewards are provided by the teacher.
Points or scoring	A scoring mechanism is set up during the training gamification process, giving points to learners depending on their performance. For example, 1000 points are given for answering questions, and 5000 points are rewarded for completing group tasks, etc.
Team work	Teamwork is the basis of training gamification, requiring everyone to work together towards the same goals.
Competition	Design competitive activities as part of training gamification, prompting learners to compete individually or as a team.
Time Pressure	Set up time limits, prompting learners to complete assigned tasks quickly under time pressure.
Increasing difficulty	As part of training gamification, design the learner’s tasks with increasing levels of difficulty.
Freedom to fail	As part of training gamification, create an environment tolerant of mistakes or failure so the learners feel at ease and motivated to learn from their mistakes.
Task with challenging goals	As part of training gamification, design some difficult questions, tasks, or challenging goals that may be answered or completed only after hard work by the learners.
Prize or bonus money	Provide incentives, such as material rewards or cash prizes, to encourage learner participation.
Leaderboard	Display the learner or team performance ranking order to show everyone their own status relative to others in the training class.
Adaptation to difficulty	Make timely adjustments of the game or task difficulty based on each learner’s performance, which may require the use of a software program. Adjust the level of difficulty in any direction, not just from simple to complex.
Experiential activities	As part of training gamification, design various activities suitable for learner participation as well as post-activity evaluation, so learners may learn from the experience. For example, learners may learn the value of teamwork or project management via the cup-stacking activity, or learn to differentiate consultation vs. actual implementation through the activity of walking with a blindfold.
Educational tools or props	As part of training gamification, prepare teaching tools or props to facilitate the implementation of the training course.
Performance status feedback	Provide an index, a number, or some graphics to indicate the learner’s performance status.
Virtual currency or chips	Use coins, pretend-money or chips as scoring tools of training gamification.
Peer rating	Invite learners to evaluate each other as part of the learning process of gamified education.
Storyline	Use a story, background, or dialogues among different players during gamified training to help learners become immersed in the storyline.
Clues	Design a scavenger hunt as part of the gamified training, where the learners need to find a series of clues to discover the instructions for the next set of actions.
Virtual Helper	For training gamification, provide a real or virtual educational helper to answer questions or provide assistance to the learners.
Puzzles	During training gamification, design puzzles or problems for the learners to find the solutions. For example, create a jigsaw puzzle for the learners to solve.
Social network	Establish a channel of communication or social networking mechanism online or offline, so the learners may exchange and connect with each other. For example, set up a physical or virtual social group, or create a group discussion platform, etc.
Progress bar	As part of training gamification, create a status bar display to indicate the learner’s current progress in terms of the percentage of task completion or game progress.
Badges	Reward learners with different badges depending on their performance level, such as badges of gold, silver, bronze, platinum, or diamond. Badges are often given in place of prizes, to indicate hard work or accumulation of achievement in a game in class.
Customization or personalization	Offer a customizable training experience where the learners are able to make their own personalized adjustments and choose their own training content.
Board games	Use board games or commercial video games as part of the gamified training course or experiential activities.
Hierarchy	As part of gamified training, divide learners into different levels corresponding to different evolutions, such that they would obtain different capabilities after advancing to the next level. For example, Pokémon Trainers are divided into 1 ~ 40 levels. After a Trainer accumulates enough Experience Points, they can advance to the next level. Trainers unlock certain Features after they advance to the next level.
Virtual identity	Create a virtual identity or character for each learner in gamified training so they can play different roles, such as a magician, hunter, or animal, etc.
Integration with software apps	Use apps, relevant software, or websites to reach the goals of training gamification.
Virtual treasure	Learners are required to collect virtual treasures in the gamified training course, such as treasures or props in Pokémon.
Virtual reality	Use a 2D or 3D space simulation, virtual reality, or a game world situational simulation as part of the gamified training.
Health score	Assign a virtual health index corresponding to the character played by the learners during gamified training. When the health index reaches zero, it indicates that the character played by the learner needs to perform some task or obtain some material supplements.
Check in	Integrate GPS functionalities with the gamified training course, so the learners may mark their current location or perform a check in.

Table 2. Characteristics of study participants (N = 19).

Code	Gender	Profession	Years of Teaching Experience	Years of Gamification Experience	Fields of Educational Expertise
A	Male	Orthopaedist	10	3	Evidence-Based Medicine
B	Female	Occupational Therapist	7	3	Clinical Practice of Occupational Therapy
C	Male	Emergency Physician	10	3	Medical Care Act
D	Male	Orthopaedist	10	5	Orthopaedics
E	Female	Medical lecturer	21	4	Human Physiology
F	Female	Medical lecturer	12	4	Nutriology
G	Female	Nurse	4	3	Evidence-Based Medicine
H	Female	Case Manager	8	4	Doctor-Patient Communication
I	Male	Family Physician	10	5	Geriatrics
J	Male	Emergency Physician	9	3	Emergency Medicine
K	Male	Pulmonologist	8	3	Thoracic Medicine
L	Male	Pulmonologist	7	4	Interpret of Chest X-ray
M	Female	Dentist	10	3	Dentistry
N	Male	Oral and Maxillofacial Surgeon	18	6	Clinical Practice of Dentistry
O	Female	Nurse	10	4	Home Health Care
P	Male	Physiotherapist	15	3	Clinical Practice of Physical Therapy
Q	Male	Nurse	10	3	Cardiopulmonary Resuscitation
R	Male	Nephrologist	12	5	Clinical Practice of Nephrology
S^†	Male	Emergency Physician	10	7	Emergency Medicine

^†indicated the participant withdrew at the second round period.

Table 3. The results of two round-survey of experts’ opinion.

	Round 1					Round 2
Elements	Mean	S.D.	IQR	≥4 Ratio	Rank	Mean	SD	IQR	≥4 Ratio	Rank
Integration with educational objectives	4.8	0.41	0	100%	1	4.9	0.23	0	100%	1
Game rules	4.8	0.41	0	100%	2	4.9	0.31	0	100%	2
Rapid feedback	4.8	0.41	0	100%	3	4.8	0.42	0	100%	3
Fairness	4.7	0.46	1	100%	5	4.7	0.47	1	100%	4
Points or scoring	4.7	0.44	0.5	100%	4	4.6	0.5	1	100%	5
Teamwork	4.7	0.57	0.5	95%	6	4.6	0.6	1	94%	6
Competition	4.5	0.68	1	89%	7	4.5	0.6	1	94%	7
Time Pressure	4.4	0.67	1	89%	12	4.4	0.68	1	89%	8
Increasing difficulty	4.4	0.59	1	95%	8	4.3	0.67	1	89%	9
Freedom to fail	4.3	0.86	1	84%	13	4.3	0.67	1	89%	9
Task with challenging goals	4.4	0.48	1	100%	11	4.2	0.60	0.75	89%	11
Prize or bonus money	4.1	0.72	1	79%	15	4.0	0.58	0	83%	12
Leaderboard	4.4	0.58	1	95%	11	4.0	0.67	0	78%	13
Adaptation to difficulty	4.1	0.72	1	79%	14	3.9	0.62	0	78%	14
Experiential activities	3.9	1	1.5*	74%	17	3.8	0.53	0.75	72%	15
Educational tools or props	4.4	0.88	1	84%	9	3.8	0.71	1	61%	16
Performance status feedback	4.1	0.91	1.5*	74%	16	3.7	0.56	1	67%	17
Virtual currency or chips	3.7	0.78	1	53%	18	3.4	0.83	1	56%	18
Peer rating	3.7	0.8	1	68%	19	3.4	0.83	1	50%	19
Storyline	3.5	0.75	1	58%	22	3.3	0.82	1	33%	20
Clues	3.6	0.88	1	42%	21	3.2	0.53	0.75	28%	21
Virtual Helper	3.6	0.75	1	53%	20	3.1	0.78	0	22%	22
Puzzles	3.4	0.67	1	47%	24	2.9	0.46	0	6%	23
Social network	3.5	1.19	1	53%	23	2.8	0.69	1	17%	24
Progress bar	3.4	1.04	1	37%	25	2.8	0.76	1	22%	25
Badges	3.3	0.78	1	37%	26	2.7	0.56	1	6%	26
Customization or personalization	3.2	0.89	1	37%	27	2.6	0.60	1	0%	27
Board games	2.8	0.74	1	21%	29	2.6	0.60	1	0%	28
Hierarchy	3.0	0.92	1	26%	28	2.6	0.68	1	0%	29
Virtual identity	2.8	1.06	1.5*	26%	30	2.4	0.76	1	6%	30
Integration with software apps	2.7	0.73	0.5	5%	31	2.4	0.76	1	0%	31
Virtual treasure	2.7	0.86	1	16%	32	2.2	0.69	1	0%	32
Virtual reality	2.5	0.82	1	11%	33	2.2	0.69	1	0%	33
Health score	2.5	0.88	1	5%	34	2.1	0.66	0.75	0%	34
Check in	2.4	0.94	1	11%^‡	35	1.9	0.74	1	0%	35

Total 19 experts participated in the first round, while the 18 experts participated in the second round. At the first round, there 32 items met the criteria of consensus and 3 items did not meet. At the second round, all of the 35 items met the criteria of consensus.

*shown that did not meet the criteria of consensus.

SD: standard deviation; IQR: interquartile range; ≥ 4 Ratio: the numbers of 4 points and above/total numbers x 100%.

Table

Elements	Description	Scoring
Points or scoring	A scoring mechanism is set up during the training gamification process, giving points to learners depending on their performance. For example, 1000 points are given for answering questions, and 5000 points are rewarded for completing group tasks, etc.
Leaderboard	Display the learner or team performance ranking order to show everyone their own status relative to others in the training class.
Badges	Reward learners with different badges depending on their performance level, such as badges of gold, silver, bronze, platinum, or diamond. Badges are often given in place of prizes, to indicate hard work or accumulation of achievement in a game in class.
Task with challenging goals	As part of training gamification, design some difficult questions, tasks, or challenging goals that may be answered or completed only after hard work by the learners.
Hierarchy	As part of gamified training, divide learners into different levels corresponding to different evolutions, such that they would obtain different capabilities after advancing to the next level. For example, Pokémon Trainers are divided into 1~40 levels. After a Trainer accumulates enough Experience Points, they can advance to the next level. Trainers unlock certain Features after they advance to the next level.
Team work	Teamwork is the basis of training gamification, requiring everyone to work together towards the same goals.
Puzzles	During training gamification, design puzzles or problems for the learners to find the solutions. For example, create a jigsaw puzzle for the learners to solve.
Progress bar	As part of training gamification, create a status bar display to indicate the learner’s current progress in terms of the percentage of task completion or game progress.
Social network	Establish a channel of communication or social networking mechanism online or offline, so the learners may exchange and connect with each other. For example, set up a physical or virtual social group, or create a group discussion platform, etc.
Performance status feedback	Provide an index, a number, or some graphics to indicate the learner’s performance status.
Time Pressure	Set up time limits, prompting learners to complete assigned tasks quickly under time pressure.
Storyline	Use a story, background, or dialogues among different players during gamified training to help learners become immersed in the storyline.
Virtual identity	Create a virtual identity or character for each learner in gamified training so they can play different roles, such as a magician, hunter, or animal, etc.
Competition	Design competitive activities as part of training gamification, prompting learners to compete individually or as a team.
Virtual Helper	For training gamification, provide a real or virtual educational helper to answer questions or provide assistance to the learners.
Health score	Assign a virtual health index corresponding to the character played by the learners during gamified training. When the health index reaches zero, it indicates that the character played by the learner needs to perform some task or obtain some material supplements.
Increasing difficulty	As part of training gamification, design the learner’s tasks with increasing levels of difficulty.
Peer rating	Invite learners to evaluate each other as part of the learning process of gamified education.
Virtual currency or chips	Use coins, pretend-money or chips as scoring tools of training gamification.
Virtual reality	Use a 2D or 3D space simulation, virtual reality, or a game world situational simulation as part of the gamified training.
Board games	Use board games or commercial video games as part of the gamified training course or experiential activities.
Customization or personalization	Offer a customizable training experience where the learners are able to make their own personalized adjustments and choose their own training content.
Adaptation to difficulty	Make timely adjustments of the game or task difficulty based on each learner’s performance, which may require the use of a software program. Adjust the level of difficulty in any direction, not just from simple to complex.
Prize or bonus money	Provide incentives, such as material rewards or cash prizes, to encourage learner participation.
Game rules	Clear game rules should be set up during the gamification process for the learners to follow.
Rapid feedback	In response to the learner’s performance, set up a rapid feedback mechanism and offer points, praises, chips, or even material rewards. The key is to provide feedback in a timely manner.
Freedom to fail	As part of training gamification, create an environment tolerant of mistakes or failure so the learners feel at ease and motivated to learn from their mistakes.
Experiential activities	As part of training gamification, design various activities suitable for learner participation as well as post-activity evaluation, so learners may learn from the experience. For example, learners may learn the value of teamwork or project management via the cup-stacking activity, or learn to differentiate consultation vs. actual implementation through the activity of walking with a blindfold.
Integration with training goal	Each activity designed for training gamification should be closely integrated with the training goals for the purpose of learning, not for fun or playing games.
Educational tools or props	As part of training gamification, prepare teaching tools or props to facilitate the implementation of the training course.
Clues	Design a scavenger hunt as part of the gamified training, where the learners need to find a series of clues to discover the instructions for the next set of actions.
Fairness	Fairness is very important in training gamification, when points and rewards are provided by the teacher.
Integration with software apps	Use apps, relevant software, or websites to reach the goals of training gamification.
Virtual treasure	Learners are required to collect virtual treasures in the gamified training course, such as treasures or props in Pokémon.
Check in	Integrate GPS functionalities with the gamified training course, so the learners may mark their current location or perform a check in.

Please add if there are any key gamification elements not included in the list above. Also, are there elements that you think require adjustments or additional descriptions?