Causal inference in educational effectiveness research: a comparison of three methods to investigate effects of homework on student achievementFootnote¹

1. Invited keynote address presented at the second meeting of EARLI SIG 18, Centre for Educational Effectiveness and Evaluation, Leuven, Belgium, August 25–27, 2010. The research reported herein has been financially supported by the Swedish Research Council. This manuscript was accepted under the guest editorship of Jan Van Damme.

References

• Angrist , J. D. 2004 . American education research changes tack . Oxford Review of Economic Policy , 20 : 198 – 212 .
   Web of Science ®Google Scholar
• Angrist , J. D. and Pischke , J.-S. 2009 . Mostly harmless econometrics: An empiricist's companion , Princeton, NJ : Princeton University Press .
   Google Scholar
• Baumert , J. , Kunter , M. , Blum , W. , Brunner , M. , Voss , T. , Jordan , A. and Tsai , Y.-M. 2010 . Teachers' mathematical knowledge, cognitive activation in the classroom, and student progress . American Educational Research Journal , 47 : 133 – 180 .
   Web of Science ®Google Scholar
• Blundell , R. and Costa Dias , M. 2009 . Alternative approaches to evaluation in empirical microeconomics . Journal of Human Resources , 44 : 565 – 640 .
   Web of Science ®Google Scholar
• Bollen , K. A. 1989 . Structural equations with latent variables , New York, NY : Wiley .
   Google Scholar
• Cooper , H. 1989 . Homework , White Plains, NY : Longman .
   Google Scholar
• Cooper , H. , Robinson , J. C. and Patall , E. A. 2006 . Does homework improve academic achievement? A synthesis of research, 1987–2003 . Review of Educational Research , 76 : 1 – 62 .
   Web of Science ®Google Scholar
• Creemers , B. P. M. , Kyriakides , L. and Sammons , P. , eds. 2010 . Methodological advances in educational effectiveness research , London, UK : Taylor & Francis . (Eds.). (
   Google Scholar
• De Jong , R. , Westerhof , K. J. and Creemers , B. P. M. 2000 . Homework and student math achievement in junior high schools . Educational Research and Evaluation , 6 : 130 – 157 .
   Google Scholar
• Dettmers , S. , Trautwein , U. and Lüdtke , O. 2009 . The relationship between homework time and achievement is not universal: Evidence from multilevel analyses in 40 countries . School Effectiveness and School Improvement , 20 : 375 – 405 .
   Web of Science ®Google Scholar
• Gustafsson , J.-E. 2007 . “ Understanding causal influences on educational achievement through analysis of differences over time within countries ” . In Lessons learned: What international assessments tell us about math achievement , Edited by: Loveless , T. 37 – 63 . Washington, DC : Brookings Institution Press . (Ed.), (pp
   Google Scholar
• Gustafsson , J.-E. 2010a . Causal inference in international comparative research on student achievement: Methodological challenges and developments , Gothenburg, Sweden. : Invited keynote address presented at the fourth IEA International Research Conference . July
   Google Scholar
• Gustafsson , J.-E. 2010b . “ Longitudinal designs ” . In Methodological advances in educational effectiveness research , Edited by: Creemers , B. P. M. , Kyriakides , L. and Sammons , P. 77 – 101 . London, UK : Taylor & Francis . (Eds.), (pp
   Google Scholar
• Heckman , J. J. 1979 . Sample selection bias as a specification error . Econometrica , 47 : 153 – 161 .
   Web of Science ®Google Scholar
• Heckman , J. J. and Robb , R. 1986 . “ Alternative methods for solving the problem of selection bias in evaluating the impact of treatments on outcomes ” . In Drawing inferences from self-selected samples , Edited by: Wainer , H. 63 – 113 . New York, NY : Springer . (Ed.), (pp
   Google Scholar
• Holland , P. W. 1986 . Statistics and causal inference . Journal of the American Statistical Association , 81 : 945 – 970 .
   Web of Science ®Google Scholar
• Mullis , I. V. S. , Martin , M. O. and Foy , P. 2008 . TIMSS 2007 international mathematics report , Chestnut Hill, MA : TIMSS & PIRLS International Study Center, Boston College .
   Google Scholar
• Mullis , I. V. S. , Martin , M. O. , Gonzalez , E. J. and Chrostowski , S.J. 2004 . TIMSS 2003 international mathematics report , Chestnut Hill, MA : TIMSS & PIRLS International Study Center, Boston College .
   Google Scholar
• Muthén , B. O. 1994 . Multilevel covariance structure analysis . Sociological Methods & Research , 22 : 376 – 398 .
   Web of Science ®Google Scholar
• Muthén , L. and Muthén , B. O. 1998–2008 . Mplus user's guide , Los Angeles, CA : Authors . Version 5
   Google Scholar
• Nordenbo, S. E., Holm, A., Elstad, E., Scheerens, J., Søgaard Larsen, M., Uljens, M., … Hauge, T.E. (2010). Input, process, and learning in primary and lower secondary schools. A systematic review carried out for The Nordic Indicator Workgroup (DNI) (Technical Report Clearinghouse – Research Series 2010 Number 06). Copenhagen, Denmark: Danish Clearinghouse for Educational Research, DPU, Aarhus University. http://www.dpu.dk/fileadmin/www.dpu.dk/omdpu/danskclearinghouseforuddannelsesforskning/udgivelser/SR5-FinalVersion_-_revideret_november_2011.pdf (http://www.dpu.dk/fileadmin/www.dpu.dk/omdpu/danskclearinghouseforuddannelsesforskning/udgivelser/SR5-FinalVersion_-_revideret_november_2011.pdf)
   Google Scholar
• Rubin , D. B. 1990 . Formal mode of statistical inference for causal effects . Journal of Statistical Planning and Inference , 25 : 279 – 292 .
   Web of Science ®Google Scholar
• Rubin , D. B. and Thomas , N. 1996 . Matching using estimated propensity scores: Relating theory to practice . Biometrics , 52 : 249 – 264 .
   PubMed Web of Science ®Google Scholar
• Schlotter, M., Schwerdt, G., & Woessmann, L. (2009). Methods for causal evaluation of education policies and practices: An econometric toolbox (EENEE Analytical Report No. 5 prepared for the European Commission). Retrieved from http://www.eenee.de/portal/page/portal/EENEEContent/_IMPORT_TELECENTRUM/DOCS/EENEE_AR5.pdf (http://www.eenee.de/portal/page/portal/EENEEContent/_IMPORT_TELECENTRUM/DOCS/EENEE_AR5.pdf)
   Google Scholar
• Snijders , T. A. B. and Bosker , R. J. 1999 . Multilevel analysis: An introduction to basic and advanced multilevel modelling , London, UK : SAGE .
   Google Scholar
• Sobel , M. E. 1996 . An introduction to causal inference . Sociological Methods and Research , 24 : 353 – 379 .
   Web of Science ®Google Scholar
• Trautwein , U. 2007 . The homework-achievement relation reconsidered: Differentiating homework time, homework frequency, and homework effort . Learning and Instruction , 17 : 372 – 388 .
   Web of Science ®Google Scholar
• Trautwein , U. and Köller , O. 2003 . The relationship between homework and achievement: Still much of a mystery . Educational Psychology Review , 15 : 115 – 145 .
   Web of Science ®Google Scholar
• Trautwein , U. , Köller , O. , Schmitz , B. and Baumert , J. 2002 . Do homework assignments enhance achievement? A multilevel analysis in 7th grade mathematics . Contemporary Educational Psychology , 27 : 26 – 50 .
   Web of Science ®Google Scholar
• Winship , C. and Morgan , S. L. 1999 . The estimation of causal effects from observational data . Annual Review of Sociology , 25 : 659 – 706 .
   Web of Science ®Google Scholar
• Winship , C. and Morgan , S. L. 2007 . Counterfactuals and causal inference. Methods and principles for social research , Cambridge, UK : Cambridge University Press .
   Google Scholar