Disentangling Perception and Performance: A Natural Experiment on Student Engagement and Learning in Simulations

Abstract

Simulations are increasingly recognized for promoting active learning and student engagement and fostering career skills. Yet whether and how simulations facilitate learning, engagement, or both are still subjects of considerable debate. Many existing studies tend to be observational in design, constrained to a single semester, or reliant on student self-assessment to measure efficacy. We use an experimental approach to demonstrate the effects of classroom simulations on both self-assessment and objective academic performance at a large public university. Over the course of three years, we “treated” some sections of an Introduction to International Relations course (N = 244) with simulations while other “control” sections participated in debates and discussions. We compare self-assessed engagement and motivation from surveys, as well as performance on quizzes and exams in the same course (same instructor, materials, and textbook). Our results demonstrate that students “treated” with the simulations show sustained improvements in academic performance in both iterative (weekly quizzes) and long-term (final exam) contexts, controlling for the mode of delivery. These improvements are also found to be robust for both high and low-performing students in the class. However, we were surprised to find a disconnect between student perceptions of engagement and actual course performance: Self-assessed student engagement and motivation were poor predictors of subsequent academic performance.

Keywords:

• Simulations
• Foreign Policy
• Engagement
• Motivation
• US-China Trade War

Notes

1 IRB approved: February 3, 2022; Study ID: STUDY00148136

2 It should be noted other works contribute to the discussion of how simulation design might be innovated. Notably, Baumann and FitzGibbon (2021) offers the DImE (Design Implementation Evaluation) framework to both implement and evaluate the process. This process is useful in conceptualizing the whole process of simulation implementation and is worth further experimentation.

3 https://modeldiplomacy.cfr.org/preview/101/overview

4 While the United States National Security Council’s membership fluctuates dependent upon the administration, the included potential roles within this simulation include the following: President, Vice President, Secretary of State, Secretary of Defense, Secretary of the Treasury, Chairman of the Joint Chiefs of Staff, Director of National Intelligence, National Security Advisor, Attorney General, White House Chief of Staff, US Permanent Representative to the United Nations, General Advisor to the President, and a “customize-able” category.

5 The PBS documentary is Trump’s Trade War (2019). Sources include news articles, think tank reports, and select speeches by political leaders.

6 Templates of both the USA and China Domestic Negotiation Documents can be found in Appendix: A.4

7 Frieden, Lake, and Schultz (2019) 2019 World Politics: Interests, Interactions, and Institutions introduces students to the basic questions in the field of international relations, focusing on the key questions: (1) who are the key actors in international relations; (2) how do these actors interact, and how are these actions shaped by strategy and preferences; and (3) how do institutions shape interactions? The textbook is further broken into five sections: (1) An Introduction of Concepts; (2) Conflict; (3) International Political Economy; (4) International Law and Norms; and (5) Future Challenges.

8 The course had two GTAs each semester. A total of three GTAs served during the study, one of them taught all three semesters. Lesson plans and grading were standardized across different GTAs during the same semester.

9 In hindsight, we should have also surveyed students in the control group about their engagement with the debates, but we did not collect this data.

10 These questions are outlined in the Appendix.

11 We recognize that this is an imperfect method for identifying gender, much-less gender expression. However, short of re-sampling the same students, which is infeasible, this is our best attempt to capture the measure.

12 Student scoring frequently results in significant clustering and patterns in the residuals, which we found. This results in estimates of the standard errors to be biased downward. To remedy these effects, we ran each of our models with Robust Standard Errors to correct for the biases in the data.

13 “Mean” reports the mean value to survey question on a sliding five-point scale.

14 Students with an average quiz score of 0%–33% are classified as “Low Performing”; 34%–66% as “Mid Performing”; and 67%–100% as “High Performing.”

Additional information

Funding

This work was supported by the University of Kansas Center for Teaching Excellence.

Notes on contributors

Spencer Shanks

Spencer Shanks is a doctoral candidate in the University of Kansas Department of Political Science. Spencer studies leaders in the foreign policy-making process and experimental teaching methods, with a subject matter focus on the U.S.–China relationship.

Jiakun Jack Zhang

Jiakun Jack Zhang, PhD, is an assistant professor of political science at the University of Kansas and the director of the KU Trade War Lab. His research and teaching interests include international political economy, international security, foreign policy analysis, Chinese foreign policy, and U.S.–China relations. His work has been published in the Journal of Contemporary China, Economics and Politics, Business and Politics, and other journals.