Location in the multiverse of methods: measuring online users’ contexts

ABSTRACT

Identifying online users’ contexts can help researchers understand their needs. However, the validity of different methods for identifying the location of online users has been underexplored. This paper proposes using multiple methods and examining their impact on different research questions to determine their validity. It then demonstrates this approach using data from six Massive Open Online Courses (MOOCs) by examining whether different methods produce different results regarding the relationship between SES and participation and engagement in MOOCs. We found that the choice of method impacted the estimated SES of the sample; IP geolocation placed participants in lower SES districts in comparison with their self-reported districts. Using all geolocation methods, we found that our MOOCs’ learners tended to be located in high-SES districts, but the results were inconclusive regarding the relationship between SES and course engagement. Based on this case study, we suggest that using multiple methods produces more robust findings. However, when methods diverge in findings, researchers should consider which method is most suitable for their specific purposes.

KEYWORDS:

• Geolocation
• Multiverse of methods
• Education
• MOOCs
• SES

Acknowledgement

We are thankful for the support of our research assistants who helped with data preparation including Aria Eppinger, Sydney Dell, Amy Li, and G.R. Marvez.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Ella Anghel

Ella Anghel is a Ph.D. candidate in the Measurement, Evaluation, Statistics, and Assessment department at Boston College. Her research interests involve validity issues in educational and psychological measurement, particularly in technology-enhanced environments. Her current research focus is on measurement in online educational platforms such as massive open online courses.

Joshua Littenberg-Tobias

Joshua Littenberg-Tobias Ph.D., is the Director of Research and Evaluation at GBH Education where he oversees the research goals and directs educational research activities, and serve as the primary liaison with other research and evaluation organizations. Dr. Littenberg-Tobias has an extensive background in the development and evaluation of K-12 and higher education technology resources. He has also authored numerous peer-reviewed publications and conference presentations on learning science, educational technology, learning analytics, artificial intelligence, and educational equity. Prior to joining GBH, he was a Research Scientist at the MIT Teaching Systems Labs where he oversaw research on large-scale learning in online platforms. Dr. Littenberg-Tobias holds a B.A. from Brown University and a Ph.D. in Educational Research, Measurement, and Evaluation from Boston College.

Justin Reich

Justin Reich is an associate professor of digital media at MIT and the director of the MIT Teaching Systems Lab. He studies learning at scale and teacher learning.