Challenges and lessons: Design and implementation of a multi-site evaluation of team-based learning

Abstract

The authors of this article describe the design and implementation of a multi-site randomized controlled trial of team-based learning (TBL) application exercises (AE) in introductory economics courses that use TBL. For each of the four study modules that are common across sites, a site is assigned to either the treatment or control version of the module. This design enables the use of a fixed effects model to estimate the effect of treatment on student learning and control for student characteristics. The methodology demonstrates how to extract the benefits of multi-site randomized controlled designs using a minimum of resources. The authors also discuss the challenges and the lessons learned in this study.

Keywords:

• Active learning
• flipped classroom
• randomized controlled trial
• team-based learning

JEL CODE:

• A22
• C90

Notes

1 A copy of the demographic questionnaire is available from the corresponding author upon request.

2 Pacific University IRB approval number: 1421839-1. University of Richmond IRB approval number: URIRB190802.

3 Registry number: AEARCTR-0005225. http://www.socialscienceregistry.org/trials/5225.

4 Actual numbers may vary from expected numbers due to (1) students dropping the course prior to the end of the semester and (2) students declining to release their data to the study. In addition, as discussed below, site #2 withdrew from the study.

5 One site participant received lower than normal student critiques during the study semester, and based on student comments in the critiques, attributed that to the additional assessments. Research on the relationship between the number of assessments in a course and outcomes on student evaluations of teaching does not provide consistent evidence on the relationship between these two variables. See, for example, Shapiro (1990), Centra (2003) and Kramp (2010).

6 https://www.hhs.gov/ohrp/regulations-and-policy/regulations/revised-common-rule-regulatory-text/index.html

7 In the Wozny, Balser, and Ives (2018a, 117) study, short-term assessments were given approximately three lessons after the assessed material was covered in class. Medium-term assessments covered material from the previous eight lessons.

8 A concept test such as the Force Concept Inventory (FCI) (Hestenes, Wells, and Swackhamer 1992) in physics is a scientifically validated assessment that is designed to assess student learning of a given concept (such as Newtonian mechanics). These tests are typically given at the beginning and end of a course as a way to assess student learning of the major concepts in a course (Coletta, Phillips, and Steinert 2007, 235).

Additional information

Funding

This work was supported by the National Science Foundation under IUSE Grant DUE-1712295.