Causal inference in educational effectiveness research: a comparison of three methods to investigate effects of homework on student achievement¹

1. Invited keynote address presented at the second meeting of EARLI SIG 18, Centre for Educational Effectiveness and Evaluation, Leuven, Belgium, August 25–27, 2010. The research reported herein has been financially supported by the Swedish Research Council. This manuscript was accepted under the guest editorship of Jan Van Damme.

Abstract

In educational effectiveness research, it frequently has proven difficult to make credible inferences about cause and effect relations. The article first identifies the main categories of threats to valid causal inference from observational data, and discusses designs and analytic approaches which protect against them. With the use of data from 22 countries which participated both in the Trends in International Mathematics and Science Study (TIMSS) 2003 and TIMSS 2008 with samples of Grade 8 students, 3 different methods are then applied to investigate effects of amount of time spent on homework on mathematics achievement: (a) 2-level regression, which is applied to separate student-level relations from class-level relations; (b) instrumental variables regression, using teacher-reported homework time to instrument student-reported homework time; and (c) a difference-in-differences analysis investigating country-level change between 2003 and 2007. All 3 methods showed that there is a positive effect of homework time on student achievement.

Keywords:

• homework
• mathematics achievement
• TIMSS
• causal inference

Notes

1. Invited keynote address presented at the second meeting of EARLI SIG 18, Centre for Educational Effectiveness and Evaluation, Leuven, Belgium, August 25–27, 2010. The research reported herein has been financially supported by the Swedish Research Council. This manuscript was accepted under the guest editorship of Jan Van Damme.