Multilevel Mixture Modeling with Propensity Score Weights for Quasi-Experimental Evaluation of Virtual Learning Environments

Abstract

With the growing use of virtual learning environments (VLE), innovative methods to evaluate their performance are increasingly needed. A key difficulty in evaluating VLE using system logs is the large heterogeneity of usage patterns. The current study demonstrates an approach to classify complex patterns of student-level and classroom-level usage with latent class analysis, then estimate average treatment effects (ATEs) of membership in student or classroom classes, as well as joint effects. The approach accounts for uncertainty of latent classes with a three-step method, and nonrandom selection into classes using inverse probability weighting. We demonstrate the approach with an analysis of usage of an Algebra VLE and estimate causal effects of latent class membership on a high-stakes Algebra I standardized assessment. Challenges of using system logs for evaluation of VLE are discussed with respect to measurement error, construct validity, latent classes enumeration, and comparison of classes with respect to distal outcomes.