A Commentary on Construct Validity When Using Operational Virtual Learning Environment Data in Effectiveness Studies

Abstract

Virtual learning environments (VLEs) are increasingly used at-scale in educational contexts to facilitate teaching and promote learning, and the data they produce can be used for educational research purposes. Meanwhile, the U.S. Department of Education's Office of Educational Technology has repeatedly emphasized the importance of using evidence to validate claims from VLE-based educational research. Although VLE data can provide some affordances for conducting educational research, we argue that many challenges can arise with respect to providing evidence for construct validity. The objective of this commentary is to encourage educational researchers using operational, at-scale VLE data to align their data and intended constructs to a theoretical framework of construct validity threats in order to develop a comprehensive set of actionable solutions. We use examples from our research project as a demonstration resource for performing such an alignment.