Assessing Knowledge Retention, With and Without Simulations

ABSTRACT

This study examines the effectiveness of teaching with simulations, compared to active learning without simulations. It utilizes an anonymous extra-credit pop quiz on four topics, each taught with a different method: (1) simulation and in-class debriefing; (2) simulation only; (3) in-class discussions with an accompanying research essay; and (4) in-class discussions only. The study presents a range of assessment techniques used in simulation literature, and suggests the use of the anonymous extra-credit pop quiz as a simple and familiar teaching practice, but an overlooked assessment tool for simulations. The study presents the “Iranian Plane” simulation developed to teach decision making in crisis situations to political science undergraduates. It analyzes empirical evidence on knowledge retention with and without simulations based on students’ performance on the pop quiz 3 months after the simulation. The analysis shows that learning with simulation and debriefing together attains teaching goals set in advance. Simulation with debriefing was the most effective teaching mode for knowledge retention in terms of students’ performance in the quiz, simulation only was almost as successful, but learning without simulation was less efficient.

KEYWORDS:

• Assessment
• crisis
• debriefing
• decision-making
• simulation

Acknowledgments

I would like to thank Mary Jane Parmentier, John Raymond, Elizabeth Mendenhall, Jaime Jackson and three anonymous reviewers for their helpful comments and suggestions. An earlier draft of this study was presented in 2016 at the Annual Meeting of International Studies Association in Atlanta, Georgia.

Notes

Debriefing discussions, especially in online and hybrid settings, can be recorded and used for content analysis on transcripts to increase the validity of the assessment (Leighton and Scholl 2009, e189; Pilkington and Parker-Jones 1996, 4; Ko and Rossen 2017, 78).

For the guidelines and forms for presimulation and postsimulation knowledge quizzes, see Ben-Yehuda, Levin-Banchik, and Naveh 2015, 125–126.

Unless otherwise specified, the study refers to anonymous and nonpunitive pop quizzes.

The semifictional scenario is based partly on an incident that occurred in 1973 and involved Israel and Libya (see http://www.icb.umd.edu/dataviewer/?crisno=251).

See supplemental materials for a more formal presentation of the simulation procedures.

After the simulation ends, I reveal that it might have taken 6 minutes for the civilian airplane to reach Dimona after entering Israeli airspace from its southern border.

The free countdown timer is available at http://www.countdownkings.com/.

On the Libyan plane crisis, see Brecher and Wilkenfeld (1997, 285–86).

I also introduce filters from Allison’s political model by conveying messages to some students, usually the less active ones, saying “remind your teammates that Israel is a democratic state,” or “remind your teammates that parliamentary elections will soon take place.” Then during debriefing we discuss different constraints, including political regime and public opinion, that may shape the decision-making process.

The debriefing takes place during the next class meeting after the Iranian Plane simulation. Ben-Yehuda, Levin-Banchik, and Naveh (2015, 135) suggest that setting the debriefing a few days after the simulation has several advantages, such as allowing time for both students and the educator to rethink the game and prepare for a more qualitative discussion.

This included 10 questions on the decision-making process taught by simulation with debriefing; 10 questions on topics of international crisis taught by simulation only; 10 questions on actors in world politics taught by discussions with a 10-page research essay; and 10 questions on the topic of power in world politics taught by discussions only.

At the time students responded to the pop quiz, they had experienced the “crisis” during the simulation, but had not yet engaged in a debriefing that linked theoretical knowledge on crises with an experience-based one. The debriefing after the Iranian Plane simulation focused on the decision making only.

Additional information

Funding

This work was supported by the Israel Institute Post-Doctoral Research Grant [Grant Number GL#20163].

Notes on contributors

Luba Levin-Banchik

Luba Levin-Banchik (PhD 2016, Bar-Ilan University) is the Israel Institute postdoctoral fellow at the University of California Davis (2016-17) and the University of Toronto (2017-18). Her research interests include enduring rivalries, international crises and terrorism, the Arab-Israel conflict, simulations of international relations, and teaching in higher education. She has published her research in Terrorism and Political Violence, Studies in Conflict and Terrorism, and Media, War & Conflict. She coauthored World Politics Simulations in a Global Information Age, a book on the use of face-to-face and cyber simulations in social science courses, published by the University of Michigan Press in 2015.