Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study

References

• Aleven, V., Roll, I., McLaren, B. M., & Koedinger, K. R. (2016). Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education, 26(1), 205–223. https://doi.org/10.1007/s40593-015-0089-1
   Web of Science ®Google Scholar
• Anderson, J. R. (2013). The architecture of cognition. Psychology Press.
   Google Scholar
• Anderson, J. R., Boyle, C. F., & Reiser, B. J. (1985). Intelligent tutoring systems. Science, 228(4698), 456–462. https://doi.org/10.1126/science.228.4698.456
   PubMed Web of Science ®Google Scholar
• Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4(2), 167–207. https://doi.org/10.1207/s15327809jls0402_2
   Web of Science ®Google Scholar
• Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01
   Web of Science ®Google Scholar
• Beck, J. E., Chang, K.-M., Mostow, J., & Corbett, A. (2008). Does help help? introducing the Bayesian evaluation and assessment methodology. In International conference on intelligent tutoring systems (pp. 383–394). Springer.
   Google Scholar
• Bowers, J., Fredrickson, M., & Hansen, B. (2017). RItools: Randomization inference tools (development version) [R package version 0.2-0]. https://github.com/markmfredrickson/RItools
   Google Scholar
• Caliendo, M., & Kopeinig, S. (2008). Some practical guidance for the implementation of propensity score matching. Journal of Economic Surveys, 22(1), 31–72.
   Web of Science ®Google Scholar
• Cook, T. D., Shadish, W. R., & Wong, V. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27(4), 724–750. https://doi.org/10.1002/pam.20375
   Web of Science ®Google Scholar
• Cook, T. D., Steiner, P. M., & Pohl, S. (2009). How bias reduction is affected by covariate choice, unreliability, and mode of data analysis: Results from two types of within-study comparisons. Multivariate Behavioral Research, 44(6), 828–847. https://doi.org/10.1080/00273170903333673
   PubMed Web of Science ®Google Scholar
• Escueta, M., Quan, V., Nickow, A. J., & Oreopoulos, P. (2017). Education technology: An evidence-based review. Technical report, National Bureau of Economic Research.
   Google Scholar
• Fancsali, S. E., Holstein, K., Sandbothe, M., Ritter, S., McLaren, B. M., & Aleven, V. (2020). Towards practical detection of unproductive struggle. Proceedings of the 21st International Conference on Artificial Intelligence in Education.
   Google Scholar
• Feller, A., Greif, E., Miratrix, L., & Pillai, N. (2016). Principal stratification in the twilight zone: Weakly separated components in finite mixture models. arXiv Preprint arXiv:1602.06595.
   Google Scholar
• Feller, A., Grindal, T., Miratrix, L., & Page, L. C. (2016). Compared to what? Variation in the impacts of early childhood education by alternative care type. The Annals of Applied Statistics, 10(3), 1245–1285.
   Web of Science ®Google Scholar
• Frangakis, C. E., & Rubin, D. B. (2002). Principal stratification in causal inference. Biometrics, 58(1), 21–29. https://doi.org/10.1111/j.0006-341x.2002.00021.x
   PubMed Web of Science ®Google Scholar
• Fred, G. P., & Van Merriënboer, J. J. (1994). Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. Journal of Educational Psychology, 86(1), 122.
   Web of Science ®Google Scholar
• Gabry, J. (2016). bayesplot: Plotting for Bayesian Models [R package version 1.1.0.]. https://CRAN.R-project.org/package=bayesplot
   Google Scholar
• Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2014). Bayesian data analysis (Vol. 2). CRC Press.
   Google Scholar
• Gilbert, P. B., & Hudgens, M. G. (2008). Evaluating candidate principal surrogate endpoints. Biometrics, 64(4), 1146–1154. https://doi.org/10.1111/j.1541-0420.2008.01014.x
   PubMed Web of Science ®Google Scholar
• Goldin, I. M., & Koedinger, K. R. (2012). Vincent Aleven. Learner differences in hint processing. International Educational Data Mining Society.
   Google Scholar
• Hansen, B. B. (2004). Full matching in an observational study of coaching for the sat. Journal of the American Statistical Association, 99(467), 609–618.
   Web of Science ®Google Scholar
• Hansen, B. B. (2011). Propensity score matching to extract latent experiments from nonexperimental data: A case study. In N. Dorans & S. Sinharay (Eds.), Looking back: Proceedings of a conference in honor of Paul W. Holland (Chapter 9, pp. 149–181). Springer.
   Google Scholar
• Hansen, B. B., & Bowers, J. (2008). Covariate balance in simple, stratified and clustered comparative studies. Statistical Science, 23(2), 219–236.
   Web of Science ®Google Scholar
• Hansen, B. B., & Klopfer, S. O. (2006). Optimal full matching and related designs via network flows. Journal of Computational and Graphical Statistics, 15(3), 609–627.
   Web of Science ®Google Scholar
• Hirano, K., & Imbens, G. W. (2004). The propensity score with continuous treatments. Applied Bayesian Modeling and Causal Inference from Incomplete-Data Perspectives, 226164, 73–84.
   Google Scholar
• Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2007). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis, 15(3), 199–236. http://gking.harvard.edu/files/matchp.pdf
   Web of Science ®Google Scholar
• Hong, G. (2015). Causality in a social world: Moderation, mediation and spill-over. John Wiley & Sons.
   Google Scholar
• Hosman, C. A., Hansen, B. B., & Holland, P. W. (2010). The sensitivity of linear regression coefficients’ confidence limits to the omission of a confounder. The Annals of Applied Statistics, 4(2), 849–870.
   Web of Science ®Google Scholar
• Imai, K., Keele, L., Tingley, D., & Yamamoto, T. (2011). Unpacking the black box of causality: Learning about causal mechanisms from experimental and observational studies. American Political Science Review, 105(4), 765–789. https://doi.org/10.1017/S0003055411000414
   Web of Science ®Google Scholar
• Israni, A., Sales, A. C., & Pane, J. F. (2018). Mastery learning in practice: A (mostly) descriptive analysis of log data from the cognitive tutor algebra I effectiveness trial. arXiv:1802.08616.
   Google Scholar
• Jin, H., & Rubin, D. B. (2008). Principal stratification for causal inference with extended partial compliance. Journal of the American Statistical Association, 103(481), 101–111.
   Web of Science ®Google Scholar
• Jo, B. (2008). Causal inference in randomized experiments with mediational processes. Psychological Methods, 13(4), 314–336. https://doi.org/10.1037/a0014207
   PubMed Web of Science ®Google Scholar
• Kalton, G. (1968). Standardization: A technique to control for extraneous variables. Applied Statistics, 17(2), 118–136.
   Google Scholar
• Koedinger, K. R., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review, 19(3), 239–264.
   Web of Science ®Google Scholar
• Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012). The knowledge-learning-instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science, 36(5), 757–798. https://doi.org/10.1111/j.1551-6709.2012.01245.x
   PubMed Web of Science ®Google Scholar
• Leon, A. C., & Hedeker, D. (2005). A mixed-effects quintile-stratified propensity adjustment for effectiveness analyses of ordered categorical doses. Statistics in Medicine, 24(4), 647–658. https://doi.org/10.1002/sim.2042
   PubMed Web of Science ®Google Scholar
• Li, W., & Zhou, X.-H. (2017). Identifiability and estimation of causal mediation effects with missing data. Statistics in Medicine, 36(25), 3948–3965. https://doi.org/10.1002/sim.7413
   PubMed Web of Science ®Google Scholar
• McElreath, R. (2020). Statistical rethinking: A Bayesian course with examples in R and Stan. CRC Press.
   Google Scholar
• McKendree, J. (1990). Effective feedback content for tutoring complex skills. Human-Computer Interaction, 5(4), 381–413.
   Google Scholar
• Mealli, F., & Mattei, A. (2012). A refreshing account of principal stratification. The International Journal of Biostatistics, 8(1), 8.
   Google Scholar
• Mealli, F., Pacini, B., & Stanghellini, E. (2016). Identification of principal causal effects using additional outcomes in concentration graphs. Journal of Educational and Behavioral Statistics, 41(5), 463–480.
   Web of Science ®Google Scholar
• Michael, J. L., & Gutman, R. (2017). Estimation of causal effects with multiple treatments: A review and new ideas. Statistical Science, 32(3), 432–454.
   Web of Science ®Google Scholar
• Miratrix, L., Furey, J., Feller, A., Grindal, T., & Page, L. C. (2018). Bounding, an accessible method for estimating principal causal effects, examined and explained. Journal of Research on Educational Effectiveness, Pages, 11(1), 133–130.
   Web of Science ®Google Scholar
• Neyman, J. (1923). Sur les applications de la théorie des probabilités aux experiences agricoles: Essai des principes. Roczniki Nauk Rolniczych, 10, 1–51.
   Google Scholar
• Nolen, T. L., & Hudgens, M. G. (2011). Randomization-based inference within principal strata. Journal of the American Statistical Association, 106(494), 581–593. https://doi.org/10.1198/jasa.2011.tm10356
   PubMed Web of Science ®Google Scholar
• Page, L. C. (2012). Principal stratification as a framework for investigating mediational processes in experimental settings. Journal of Research on Educational Effectiveness, 5(3), 215–244.
   Web of Science ®Google Scholar
• Pane, J. F., Griffin, B. A., McCaffrey, D. F., & Karam, R. (2014). Effectiveness of cognitive tutor algebra I at scale. Educational Evaluation and Policy Analysis, 36(2), 127–144.
   Web of Science ®Google Scholar
• Pavlik, P. I., Jr, Cen, H., & Koedinger, K. R. (2009). Performance factors analysis – A new alternative to knowledge tracing. Proceedings of the 2009 Conference on Artificial Intelligence in Education (pp. 531–538).
   Google Scholar
• Pustejovsky, J. E., & Tipton, E. (2018). Small-sample methods for cluster-robust variance estimation and hypothesis testing in fixed effects models. Journal of Business & Economic Statistics, 36(4), 672–683.
   Web of Science ®Google Scholar
• Pearl, J. (2011). Principal stratification-a goal or a tool? The International Journal of Biostatistics, 7(1), 20–13. https://doi.org/10.2202/1557-4679.1322
   PubMedGoogle Scholar
• Pustejovsky, J. (2018). clubSandwich: Cluster-robust (sandwich) variance estimators with small-sample corrections [R package version 0.3.2]. https://CRAN.R-project.org/package=clubSandwich.
   Google Scholar
• R Core Team. (2016). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/.
   Google Scholar
• Rasch, G. (1993). Probabilistic models for some intelligence and attainment tests. ERIC.
   Google Scholar
• Rosenbaum, P. R. (2002). Observational studies. Springer.
   Google Scholar
• Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.
   Web of Science ®Google Scholar
• Rubin, D. B. (1978). Bayesian inference for causal effects: The role of randomization. The Annals of Statistics, 6(1), 34–58.
   Web of Science ®Google Scholar
• Rubin, D. B. (2004). Direct and indirect causal effects via potential outcomes. Scandinavian Journal of Statistics, 31(2), 161–170.
   Web of Science ®Google Scholar
• Sales, A., & Pane, J. (2020). The effect of teachers reassigning students to new cognitive tutor sections. Proceedings of the 13th International Conference on Educational Data Mining. International Educational Data Mining Society.
   Google Scholar
• Sales, A. C., Hansen, B. B., & Rowan, B. (2018). Rebar: Reinforcing a matching estimator with predictions from high-dimensional covariates. Journal of Educational and Behavioral Statistics, 43(1), 3–31. https://doi.org/10.3102/1076998617731518
   Web of Science ®Google Scholar
• Sales, A. C., & Pane, J. F. (2019a). The role of mastery learning in an intelligent tutoring system: Principal stratification on a latent variable. The Annals of Applied Statistics, 13(1), 420–443. https://doi.org/10.1214/18-AOAS1196
   Web of Science ®Google Scholar
• Sales, A. C., & Pane, J. F. (2019b). Principal stratification for intelligent tutors: A tutorial. https://github.com/adamSales/tutorial/blob/master/paper.pdf.
   Google Scholar
• Sales, A., Wilks, A., & Pane, J. (2016). Student usage predicts treatment effect heterogeneity in the cognitive tutor algebra I program. Proceedings of the 9th International Conference on Educational Data Mining (pp. 207–214). International Educational Data Mining Society.
   Google Scholar
• Schochet, P. Z. (2015). Statistical theory for the RCT-YES software: Design-based causal inference for RCTs. Mathematica Policy Research.
   Google Scholar
• Schwartz, S. L., Li, F., & Mealli, F. (2011). A Bayesian semiparametric approach to intermediate variables in causal inference. Journal of the American Statistical Association, 106(496), 1331–1344.
   Web of Science ®Google Scholar
• Shadish, W. R., & Steiner, P. M. (2010). A primer on propensity score analysis. Newborn and Infant Nursing Reviews, 10(1), 19–26.
   Google Scholar
• Singh, R., Saleem, M., Pradhan, P., Heffernan, C., Heffernan, N. T., Razzaq, L., Dailey, M. D., O’Connor, C., & Mulcahy, C. (2011). Feedback during web-based homework: the role of hints (pp. 328–336). International Conference on Artificial Intelligence in Education. Springer.
   Google Scholar
• Stekhoven, D. J., & Bühlmann, P. (2012). MissForest-non-parametric missing value imputation for mixed-type data. Bioinformatics, 28(1), 112–118. https://doi.org/10.1093/bioinformatics/btr597
   PubMed Web of Science ®Google Scholar
• Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction, 2(1), 59–89.
   Google Scholar
• VanderWeele, T. J. (2008). Simple relations between principal stratification and direct and indirect effects. Statistics & Probability Letters, 78(17), 2957–2962.
   Web of Science ®Google Scholar
• VanderWeele, T. J. (2011). Principal stratification – Uses and limitations. The International Journal of Biostatistics, 7(1), 1–14.
   Google Scholar
• VanderWeele, T. J. (2012). Comments: Should principal stratification be used to study mediational processes? Journal of Research on Educational Effectiveness, 5(3), 245–249. https://doi.org/10.1080/19345747.2012.688412
   PubMed Web of Science ®Google Scholar
• VanderWeele, T. (2015). Explanation in causal inference: Methods for mediation and interaction. Oxford University Press.
   Google Scholar
• Zeileis, A. (2004). Econometric computing with HC and HAC covariance matrix estimators. Journal of Statistical Software, 11(10), 1–17. http://www.jstatsoft.org/v11/i10/.
   Google Scholar