Assessment of educational games for health professions: A systematic review of trends and outcomes

Abstract

Background: Traditional lecturing used in teaching has the lowest retention rate; the use of games as part of an instruction method may enhance retention and reinforce learning by creating a dynamic educational environment. This study aims to systematically review the literature on educational games for the health professions to identify trends and investigate assessment tools used to measure its learning outcomes.

Methods: Seven databases were used in the search: ERIC, Education Research Complete, Medline, Medline Complete, Academic Search Complete, The Cochrane Library and PubMed.

Results: The search identified 2865 papers; among them, 1259 were excluded and 22 were evaluated. The selection incorporated five full papers which focused directly on the health professionals. Two studies involved the use of board games and two studies involved card games, crossword puzzles and one study involved a team quiz competition. Overall, studies lacked a strong link between the use of games for both instructional and assessment purposes.

Conclusion: Gaming makes a positive impact on the teaching/learning process. However, existing assessment methodologies have been not fully captured the learning that may occur in these games. Robust research is needed to address the use of games that have been assessed objectively.

Introduction

Educators face various challenges, one of which includes the urge to retain a captivating and enjoyable educational process. Considering the amount of information that health care graduates must learn to function as competent professionals, this challenge remains hard to achieve. Moreover, traditional lectures typically used in teaching continue to be commonly defined as tedious and boring with the lowest retention rate (DeKanter 2004). These issues raise the importance of using an innovative teaching tool that can enhance knowledge retention, stimulate motivation and reinforce learning by creating a dynamic, fun and exciting learning environment through the use of “games” (Royse & Newton 2007).

There are a variety of gaming methods that can be used to educate students. The focus to assess table-top games was to appraise the value of using small group deliberations, multi-player discussions and competitive interactions. Electronic games and simulations were excluded from this review to emphasize the quality of information retained using table-top games in a classroom setting. Electronic games (interactive games provided via digital technology) are another type of games which offer new and engaging teaching method. Their potential has now turned to the training of health professionals and they are already good in assessment. It includes the immersion in the virtual world, which provides safety environment for students. It also allows students to critically analyze the situations, learn from their mistakes and the standardized clinical experience (Bonnel & Smith 2010).

Table

Practice points

Games have positive impact on the teaching/learning process. Games can be used as a context for formative assessment. Assessment tools need to fully capture the learning that may be occurring in the games. Conceptual assessment framework represents six models, which can guide us in the design and assessment of educational games.

The importance of using games as an innovative educational tool has been well addressed; the assessment of its use in enhancing learning is not well-researched. Iseli et al. (2010) argued the effectiveness of games is further complicated by the use of assessments, which might be inappropriate for games that emphasize deeper learning. Yet, as research on the development of advanced statistical and measurement methods are progressing, existing assessment methodologies are not sophisticated enough to fully capture the richness of the learning that may be occurring in these games.

The Conceptual Assessment Framework (CAF) summarizes these assessment aspects by identifying six different types of models, each provides answers to specific questions and assembles a bridge between the assessment argument and the operational aspects of an assessment system as follows (Mislevy et al. 2003): (1) Student Model: “What are we measuring?” defines one or more variables related to the knowledge, skills and abilities to be measured after the use of games; (2) Task Model: “Where do we measure it?” illustrates how to structure the kinds of situations/tasks through games to obtain the needed outcomes; (3) Evidence Model: “How do we measure it?” portrays how observable are the student’s performance in a particular task during gaming. In an operational assessment, evidence rules guide the response scoring process for the learning outcomes; (4) Assembly Model: “How much do we need to measure?” describes how the previous three models must work together to form the psychometric backbone of the assessment; (5) Presentation Model: “How does it look?” explains the mean of which the assessment is delivered; and (6) Delivery Model: “How do all the other five models work together?” incorporates the collection of all the five models and how they will work together. It also describes issues that cut across all of the other models, such as platform, security and timing. These models assemble a bridge between the assessment argument and the operational aspects of an assessment system. This framework provides a structure to build a valid argument on how gaming procedures can lead to potential positive outcomes for teachers, students and future patients.

There are few studies that have directly tested the assessment of using games, with most of the effort focused on its use in teaching. On the other hand, very few papers mentioned using games as formative assessment. Analyzing student errors and providing immediate feedback are hallmarks of formative assessment (Delacruz 2011). Health sciences require an extensive amount of memorization, problem solving and comprehension. There is a solid research consensus that learning outcomes from games are affected by how the instructional methods are integrated in the games and not by the games themselves (Wainess et al. 2011). Therefore, before incorporating educational games into the health science curriculum, or any other setting, the methods and procedures must first be validated.

The growing potential that games have on the learning and assessment of learners raises important questions for the field. This study conducts a systematic review to answer the following research questions:

1. What is the value of educational games as instructional method in the health profession?

2. How can educational games properly be assessed to show that they can produce positive results to better the education of students in the health sciences?

Methods

We define the criteria for inclusion in our review as follows.

Types of participants

Study participants were students from all health professions backgrounds.

Types of interventions

A game is a “generic term that includes board, card and skilled activities (simple popular format) that provide frameworks for inserting content and creating learning activities” (Bradshaw & Lowenstein 2014). The main intervention was an educational table-top game defined as “competitive activity with a prescribed setting constrained by rules and regulations” (Blakely et al. 2009). This type of game was also selected as it requires students to play in teams, foster collaboration and communicate using their problem-solving skills. Simulations and computer games were not included in this review as: (1) they are different and complex forms of games compared to the table-top games, which may introduce additional complexities; (2) its use requires the faculty member to benefit from additional training in order to increase their comfort level with this method; and (3) initial cost.

Outcome measures

Professionals’ knowledge, skills, attitude and satisfaction were used as outcome measures.

Search methods for identification of relevant studies

Databases

Articles were identified by searching seven databases: ERIC, Education Research Complete, Medline, Medline Complete, Academic Search Complete, The Cochrane Library and PubMed. Databases were filtered and limited by English language and peer reviewed journals.

Keywords

The keywords used in the search were “educational gaming”, “educational games”, “teaching tool”, “health professions”, “basic science” and “educational games assessment”.

Dates

Studies published from year 1982 to 2014 were included in the search.

Inclusion/exclusion criteria

Relevant articles were those that (1) focused on the use of specific educational gaming in comparison to the lecture setup, (2) used educational gaming as a teaching tool, (3) applied educational gaming on health professions students, and (4) used outcomes assessment in relation to the designed objectives. The health professions were selected to narrow the scope of the paper so that the findings would be better suited to the field. Irrelevant articles were excluded if the paper (1) used computer based games or simulations, (2) the assessment method was not mentioned, (3) covered subjects that are not related to the health professions, (4) not available in full text but rather abstracts or reviews, and (5) the population composed of teachers only.

Search outcome

The search originally produced a total of 2865 papers. The full study selection is presented in Figure 1. In the end, five articles remained for review of final analysis.

Figure 1. Flow diagram of the study selection process on games used in education of health professions.

Results

For this review, 2865 papers were identified, with 1259 papers excluded during screening and 22 were evaluated. Overall, the selection incorporated five papers in full text (Figure 1), which focused directly on health professionals, mainly on medical and nursing students. This study focuses on identifying themes and trends from these five papers by identifying the types of games used to achieve the learning outcomes and the methods of assessment. All interventions in the selected studies represented a game format. Two studies involved the use of board games and the other three studies involved card games, crossword puzzles and team quiz competitions (Table 1).

Table 1. Summary of the included studies.

Reference	Intervention	Assessment method	Sample size	Topic
Cowen and Tesh (2002)	50 Questions game	Pre-Post test	85 nursing students in two groups(42 in the treatment group and 43 in the comparison group)	Pediatric cardiovascular dysfunction
Ogershok and Cottrell (2004)	Board game	Post-game survey with a five-point Likert scale	37 medical students and 12 residents	Pediatric medicine
Beylefeld and Struwig (2007)	Quiz type board game	Pre-Post surveys	109 third year medical students	Medical microbiology
Graham and Richardson (2008)	Card game	Survey	Not stated	Cultural awareness in nursing
Shah et al. (2010)	Crossword puzzles	Eight item survey instrument	Not stated	Pharmacology and medicinal chemistry of anti-ulcer agents

The first study was by Beylefeld and Struwig (2007), who adopted formal lectures with an informal quiz type board game called, “Med Micro Fun with Facts (MMFWF)”, to teach medical microbiology for third year medical students. The game covered six color-coded categories of the subjects (bacteriology; virology; mycology; parasitology; the laboratory and clinical diagnosis of infectious diseases; and the treatment, management and/or prevention of infectious diseases) along with a seventh category about general medicine. The game consisted of colored squares and triangles to represent the seven content categories; playing tokens (sticks) and reward tokens (doughnuts); and a hundred question and answer cards containing visual material/graphic images on the flip side. There was a mixed approach of assessment including two surveys, a focus group interview, direct observation and the nominal group technique, was applied. Only the surveys and direct observation data were included in the study. The two questionnaire surveys contained validated scales for measuring students' enthusiasm towards medical microbiology and games as a teaching strategy. The first self-administered survey was carried out among students who completed the infectious diseases module in the previous year and had prior exposure to the board game. The second survey was completed after playing the game in three sequential occasions. Respondents indicated that the game produced outcomes that were beyond an increased factual knowledge. Ninety percent of participants indicated that the game was fun and an average of 93% said that the game had made them aware of the factual retention. Twenty-seven percent of respondents indicated that the gaming situation encouraged them to use textbooks, dictionaries and other references as sources of learning. Eighty-three percent of respondents actively participated in small group discussions during the game.

The second study was by Cowen and Tesh (2002), who compared whether gaming combined with lecture was more effective than lecture alone in improving nursing student knowledge about pediatric cardiovascular dysfunction. This was achieved by dividing students into two groups. The first “comparison group” was taught by the traditional lectures and group discussions whereas the second “treatment group” was taught using the same method along with the game in the last 30 min of class. The pediatric cardiac game contained 50 questions to test the students' knowledge in five parts; congenital heart defects, diagnostic tests, congestive heart failure, blood flow and acquired heart diseases. The “winning team” received no money, prizes or grade incentives. The assessment tool was a pre-test one week before the teaching session and a post-test one day after the session. No significant difference was shown in pre-test scores between the two groups in comparison to post-test scores. Comparison group answered 85% of the post-test questions correctly, while the treatment group answered 94% correctly.

The third study was by Graham and Richardson (2008), who tailored a particular game called (BARNGA) for nursing students to facilitate cultural awareness in order to help limit problems stemming from cultural differences. A card game was used while students moved from table (culture) to table (culture) having different rules on each table. To measure the effectiveness of the game, a questionnaire survey was designed. More than half of the survey respondents agreed that the games used enabled them to become a member of another culture. The overall results suggest that the game stimulated strong emotion which raised students' self-awareness and provided new self-insights that would improve future intercultural relationships.

The fourth study was by Ogershok and Cottrell (2004), who used a designed board game for third year medical students and residents to enhance their learning in pediatrics. The players or teams advance through 100 spaces on the board which has four levels of color-coded play: basic, intermediate, advanced and supreme. A total of 400 questions were written from standard pediatric texts commonly used by medical students, residents and faculty. Students' progress in the game by answering questions followed by a post-game survey to evaluate their perceptions on the experience and how much they found it useful and valuable. The survey responses by the students, residents and faculty members suggest that the game was a valuable educational tool and engaged a learning experience. The students universally gave high scores to all 10 survey questions, which reflected that they were motivated to advance their knowledge of pediatrics.

Finally, the fifth study was by Shah et al. (2010), who used a crossword puzzle for three lectures on anti-cancer agents as a part of the gastrointestinal integrated sequence course. Achievement of learning outcomes was assessed through the use of examinations consisting of multiple choice questions. Performance on the examinations related to the pharmacology and medicinal chemistry of anti-ulcer agents represented an average of 82% in 2008 compared to 85% in 2009.

Discussion

Assessment is a national movement that leads the focus of educational institutes' on a path toward quality instead of quantity and on outcomes instead of resources. It is a comparatively simple process that comprises the following steps: setting clear learning objectives, knowing students behaviours that resemble achieving those objectives, identifying an assessment tool for students' achievements and evaluating data and proposing changes (Jasko 1997). These learning objectives can be standardized by using the Conceptual Assessment Framework to appraise the value of gaming in the classroom setting. The framework provides a structure that can be tested to show if games can produce significant positive results to improve the students’ comprehension of the material. This review focused on the assessment of using games as teaching tool in health professions after raising these questions: (1) Is there sufficient evidence to the impact of using games in the teaching/learning process and (2) what are useful methods to evaluate games and its outcomes?

Games are a powerful teaching strategy that develops creative and vigorous learning experiences. It reinforces knowledge, bridges the gap between what is learned; its use engages students, challenges and motivates them to learn dense materials and helps them become more responsible for their own learning (Akl et al. 2013). This requires having the game well-designed and structured clearly with a framework that ensures effective outcomes for both students and teachers (Allery 2004). This in turn leads to other relevant questions: (1) Does the assessment method provide adequate data to analyze the learning outcome? and (2) Does the assessment establish performance standards to help students achieve the learning outcomes?

The search identified five articles in which games were designed specifically for educational purposes. Results of this review showed successful increase in students' knowledge, skills and performance of health professionals by the use of the different games. However, games were neither evaluated through an assessment-oriented framework of reliability nor validity. In addition, the results show that the students are reactive to new learning styles, and are able to increase their knowledge in the subject, but further exploration is needed to see if there are behavioural changes in the learning process that shows definitive results (Kirkpatrick & Kirkpatrick 2013). Reliability testing is required to examine whether the same assessment tool produces consistent results when repeated over time (Graafland et al. 2012). Subsequently, the game must undergo a validation process, preferably not only in the content but rather in one or more of the other sources of validity evidence (response process, internal structure, relationship to other variables and consequences). Moreover, there was no continuous monitoring of progress, and gaining further information to diagnose performance and learning behaviour of students afterwards. Another reason to the difficulty of assessment could be the involvement of several players and the need of deciding a specific outcome to evaluate which may create stress or embarrassment to individuals who may perform poorly (Skiba 2008). From the limited data it can be inferred that educational games can increase the performance of students in the classroom. All five studies showed an increase in student cognition illustrated by higher test scores or encouraged students to use other references as sources of learning. Educational games create a beneficial learning environment by requiring students to take part in question and answer scenarios that reinforce students to retain information. Interactive learning provides immediate feedback to students and teachers to gauge how much information is being understood and where improvements can be made on concepts that were less understood. This is a significant improvement from a traditional lecture setting since it creates more checkpoints to monitor the students’ progress.

There are some limitations to this study. Although this study was able to identify meaningful trends and themes from the five articles reviewed, a wider pool of articles, if identified, may have led to a more robust understanding of the use of educational games in the health professions. The lack of studies also limited our potential to calculate useful effect sizes for comparison. However, these studies also do note that an increased number of papers in educational games is warranted.

There are a number of steps that need to be taken in the assessment of educational games before they can be considered effective. One consideration could be the effectiveness of using games as a learning tool for long-term memory. The studies showed that there was an initial increase in cognitive ability, but there was no follow-up data on how long the information was retained. Additionally, the studies showed improved results in the classroom setting. Further research needs to be done to indicate if the information retained can be later applied in a clinical or practical setting. Therefore, considering these difficulties and constructing valid and reliable assessment tools could make games widely used for the benefit of students and patients. Besides, viewing educational games as formative assessment environments using defined criteria for evaluation strengthens the understanding of feedback in effective game design (Delacruz 2011).

Conclusion

Based on this review, gaming makes a positive impact on the teaching/learning process. However, existing assessment methodologies have been not fully captured the learning that may occur in these games. Robust research is needed to address the use of games that have been assessed objectively. This includes taking into consideration study design, sample size, students and teachers variables, level of knowledge and skills, games description (e.g. objectives, type, rules, material needed to implement it, settings and costs), the evaluation instruments and the statistical methods. In addition, more researches are required on the use of games from the students’ perspectives and linking it with the patients care and outcomes.

Notes on contributors

HIND ABDULMAJED, PhD, is an Assistant Professor, Department of Medical Microbiology and Parasitology at King Abdulaziz University, KSA. She is also doing her master of medical science in medical education at University of Illinois in Chicago in collaboration with King Abdulaziz University in KSA. She designed the conceptual basis for this study, developed the manuscript, and finalized the study.

YOON SOO PARK, PhD, is an Assistant Professor, Department of Medical Education at the University of Illinois at Chicago College of Medicine, Chicago, IL, USA and contributed to conceptualization and revision of the manuscript.

ARA TEKIAN, PhD, MHPE, is an Associate Professor, Department of Medical Education, and Associate Dean for International Affairs at the University of Illinois at Chicago College of Medicine, Chicago, IL, USA and contributed to conceptualization and revision of the manuscript.

Acknowledgements

Sincere gratitude and appreciation are due to Dr Eman Aldigs for her valuable comments, suggestions and editing on preparing the manuscript.

The publication of this supplement has been made possible with the generous financial support of the Dr Hamza Alkholi Chair for Developing Medical Education in KSA.

Declaration of interest: The authors declare that they have no competing interests. The authors alone are responsible for the content and writing of the article.