American Politics Course Redesigns: The Effect of Propensity Score Matching on Predicting Learning Outcomes

Abstract

Faculty at many colleges and universities are redesigning courses to address differences in student learning preferences. In this study, the researchers surveyed students in two similar American Politics classes. In one, the instructor used a traditional large-class lecture format. In the other, lectures were supplemented with weekly small-group discussions to facilitate more in-depth engagement with course material. We subsequently surveyed students in both classes about their political knowledge gains, perceptions of politics, and views on critical thinking. Due to nonrandom group assignment, we estimate the effects using propensity score matching. Our results indicate that there were different types of students enrolled in each class. Students who experienced the small-group discussions were more likely to score better on critical thinking questions, but they were not as likely to demonstrate more general knowledge about political science in their responses. There are almost no estimation differences based on whether we use propensity score matching or linear regression.

Keywords:

• Propensity score matching
• course redesign
• teaching American Politics

Notes

Notes

1 The measure we used is Cronbach’s alpha, which should be no less than .60 for a reliable scale.

2 Cognate fields included Political Science as well as Communication Studies, Sociology, Criminal Justice, Psychology, and History. A student with a Latin American Studies major with a Political Science minor was coded as “cognate.” A graduate student in the Public Administration program was also coded as “cognate.”

3 Table 1 notes differences in student grades between the treatment and control groups. There is also a theoretical relationship between good grades and increased student knowledge. We did run the model without GPA and there were no significant changes. Therefore, we chose to keep the transcript variables as matching criteria.

4 An evening section of American Politics was also offered, but the evening version traditionally has a much smaller enrollment, so it is not used in the comparison.

5 The “redesigned” section of American Politics was offered 2 days a week at 9:30 a.m. The traditional sections were also offered 2 days a week—one at 9:30 a.m. and one at 12:30 p.m. Since we only offered “redesigned” sections in the morning, we were unable to include as an explanatory variable whether the class was offered in the morning or the evening.

6 The responses from the common bank of 24 multiple choice questions has a Cronbach alpha of .63.

7 The Cronbach’s alpha reliability coefficient is .662.

8 The Cronbach’s alpha reliability coefficient for these statements is .82.

9 Nearest neighbor matching is used to minimize the number of observations excluded due to lack of an exact match.

10 The university does not accept transfer credit wherein a student earns a D or an F. Thus, we may slightly underestimate the number of hours a student has been in school or underestimate the number of previous political science classes the student had taken.

11 While several of the matching variables (e.g., GPA, number of political sciences, number of credit hours) were pretreatment, we included some variables collected in the posttreatment survey (e.g., the degree that they follow local, state, and national news). Obviously, if these criteria are correlated with the outcome variables and influenced by the treatment, this is a limitation of our research design. There is some reason to believe that the variables are not highly correlated with the outcomes. For example, the local and state news indicators are not significant in any of the seven regression models, and following national news is only significant in two of the seven models.

Additional information

Notes on contributors

Martha Kropf

Martha Kropf is a Professor in the Department of Political Science and Public Administration at the University of North Carolina at Charlotte. She studies election administration and voter participation. She is author of Institutions and the Right to Vote in America (2019) and Helping America Vote: The Limits of Election Reform (2012; with David C. Kimball).

Samuel Jacob Grubbs

Samuel Jacob Grubbs is a Lecturer in the Department of Political Science and Public Administration at the University of North Carolina at Charlotte. His research interests include public policy, higher education, and classroom learning.

John Szmer

John Szmer is an Associate Professor in the Department of Political Science and Public Administration at the University of North Carolina at Charlotte. He studies judicial behavior and court administration, with a focus on gender and race. He coauthored The View from the Bench and Chambers (2014; with Jennifer Bowie and Donald Songer).

Beth Elise Whitaker

Beth Elise Whitaker is an Associate Professor in the Department of Political Science and Public Administration at the University of North Carolina at Charlotte. Her research focuses primarily on migration and security issues in Africa.