Multilevel Mixture Modeling with Propensity Score Weights for Quasi-Experimental Evaluation of Virtual Learning Environments

ABSTRACT

With the growing use of virtual learning environments (VLE), innovative methods to evaluate their performance are increasingly needed. A key difficulty in evaluating VLE using system logs is the large heterogeneity of usage patterns. The current study demonstrates an approach to classify complex patterns of student-level and classroom-level usage with latent class analysis, then estimate average treatment effects (ATEs) of membership in student or classroom classes, as well as joint effects. The approach accounts for uncertainty of latent classes with a three-step method, and nonrandom selection into classes using inverse probability weighting. We demonstrate the approach with an analysis of usage of an Algebra VLE and estimate causal effects of latent class membership on a high-stakes Algebra I standardized assessment. Challenges of using system logs for evaluation of VLE are discussed with respect to measurement error, construct validity, latent classes enumeration, and comparison of classes with respect to distal outcomes.

KEYWORDS:

• Causal inference
• program evaluation
• latent class analysis
• multilevel mixture modeling
• virtual learning environments

Notes

¹ Complete code for the analyses shown in this paper, including pre-processing tasks, can be found at https://osf.io/phr4w/.

² The Mplus codes for the multilevel mixture models are available at https://osf.io/phr4w/.

Additional information

Funding

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305C160004 to the University of Florida. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.