ABSTRACT
Identifying online users’ contexts can help researchers understand their needs. However, the validity of different methods for identifying the location of online users has been underexplored. This paper proposes using multiple methods and examining their impact on different research questions to determine their validity. It then demonstrates this approach using data from six Massive Open Online Courses (MOOCs) by examining whether different methods produce different results regarding the relationship between SES and participation and engagement in MOOCs. We found that the choice of method impacted the estimated SES of the sample; IP geolocation placed participants in lower SES districts in comparison with their self-reported districts. Using all geolocation methods, we found that our MOOCs’ learners tended to be located in high-SES districts, but the results were inconclusive regarding the relationship between SES and course engagement. Based on this case study, we suggest that using multiple methods produces more robust findings. However, when methods diverge in findings, researchers should consider which method is most suitable for their specific purposes.
Acknowledgement
We are thankful for the support of our research assistants who helped with data preparation including Aria Eppinger, Sydney Dell, Amy Li, and G.R. Marvez.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Additional information
Notes on contributors
Ella Anghel
Ella Anghel is a Ph.D. candidate in the Measurement, Evaluation, Statistics, and Assessment department at Boston College. Her research interests involve validity issues in educational and psychological measurement, particularly in technology-enhanced environments. Her current research focus is on measurement in online educational platforms such as massive open online courses.
Joshua Littenberg-Tobias
Joshua Littenberg-Tobias Ph.D., is the Director of Research and Evaluation at GBH Education where he oversees the research goals and directs educational research activities, and serve as the primary liaison with other research and evaluation organizations. Dr. Littenberg-Tobias has an extensive background in the development and evaluation of K-12 and higher education technology resources. He has also authored numerous peer-reviewed publications and conference presentations on learning science, educational technology, learning analytics, artificial intelligence, and educational equity. Prior to joining GBH, he was a Research Scientist at the MIT Teaching Systems Labs where he oversaw research on large-scale learning in online platforms. Dr. Littenberg-Tobias holds a B.A. from Brown University and a Ph.D. in Educational Research, Measurement, and Evaluation from Boston College.
Justin Reich
Justin Reich is an associate professor of digital media at MIT and the director of the MIT Teaching Systems Lab. He studies learning at scale and teacher learning.