Generating actionable predictions regarding MOOC learners’ engagement in peer reviews

References

• Admiraal, W., B. Huisman, and M. Van De Ven. 2014. “Self- and Peer Assessment in Massive Open Online Courses.” International. Journal of Higher Education 3 (3): 119–128.
   Google Scholar
• Alario-Hoyos, C., M. Perez-Sanagustin, C. Delgado-Kloos, H. A. G. Parada, and M. Munoz-Organero. 2014. “Delving Into Participants’ Profiles and use of Social Tools in MOOCs.” IEEE Transactions on Learning Technologies 7 (3). doi:10.1109/TLT.2014.2311807.
   Web of Science ®Google Scholar
• Ashenafi, M. M., M. Ronchetti, and G. Riccardi. 2016. “Predicting Student Progress from Peer-assessment Data.” Proceedings of the 9th international conference on educational data mining, Raleigh, NC, 270–275.
   Google Scholar
• Bekkar, M., H. K. Djemaa, and T. A. Alitouche. 2013. “Evaluation Measures for Models Assessment Over Imbalanced Data Sets.” Journal of Information Engineering Applications 3 (10): 27–39.
   Google Scholar
• Bote-Lorenzo, M. L., and E. Gómez-Sánchez. 2017. “Predicting the Decrease of Engagement Indicators in a MOOC.” Proceedings of the 7th International Conference on Learning Analytics and Knowledge, Vancouver, British Columbia, 143–147.
   Google Scholar
• Bote-Lorenzo, M. L., and E. Gómez-Sánchez. 2018. “An Approach to Build in Situ Models for the Prediction of the Decrease of Academic Engagement Indicators in Massive Open Online Courses.” Journal of Universal Computer Science, 24 (8): 1052–1071.
   Web of Science ®Google Scholar
• Boyer, S., B. U. Gelman, B. Schreck, and K. Veeramachaneni. 2015. “Data Science Foundry for MOOCs.” Proceedings of the IEEE international conference on data science and advanced analytics, Paris, 1–10.
   Google Scholar
• Boyer, S., and K. Veeramachaneni. 2015. “Transfer Learning for Predictive Models in Massive Open Online Courses.” Proceedings of the 17th conference on artificial intelligence in education, Madrid, Spain, 54–63.
   Google Scholar
• Boyer, S., and K. Veeramachaneni. 2016. “Robust Predictive Models on MOOCs: Transferring Knowledge Across Courses.” Proceedings of the 9th international conference on educational data mining, Raleigh, NC, 298–305.
   Google Scholar
• Brinton, C. G., M. Chiang, S. Jain, H. Lam, Z. Liu, and F. M. F. Wong. 2014. “Learning About Social Learning in MOOCs: From Statistical Analysis to Generative Model.” IEEE Transactions on Learning Technologies 7 (4): 346–359.
   Web of Science ®Google Scholar
• Capuano, N., and S. Caballé. 2015. “Towards Adaptive Peer Assessment for MOOCs.” Proceedings of 10th international conference on P2P, parallel, grid, cloud and internet computing, Krakow, 64–69.
   Google Scholar
• Capuano, N., V. Loia, and F. Orciuoli. 2017. “A Fuzzy Group Decision Making Model for Ordinal Peer Assessment.” IEEE Transactions on Learning Technologies 10 (2): 247–259.
   Web of Science ®Google Scholar
• Comer, D. K., C. R. Clark, and D. A. Canelas. 2014. “Writing to Learn and Learning to Write Across the Disciplines: Peer-to-Peer Writing in Introductory-Level MOOCs.” International Review of Research in Open Distance Learning 15 (5): 26–82.
   Google Scholar
• Diez, J., O. Luaces, A. Alonso-Betanzos, A. Troncoso, and A. Bahamonde. 2013. “Peer Assessment in MOOCs Using Preference Learning via Matrix Factorization.” NIPS Work. Data-Driven Educ., no. December, 1–6.
   Google Scholar
• Ding, L., E. Er, and M. Orey. 2018. “An Exploratory Study of Student Engagement in Gamified Online Discussions.” Computers & Education 120: 213–226.
   Web of Science ®Google Scholar
• Er, E., M. L. Bote-Lorenzo, E. Gómez-Sánchez, Y. Dimitriadis, and J. I. Asensio-Pérez. 2017. “Predicting Student Participation in Peer Reviews in MOOCs.” Proceedings of the Second European MOOCs Stakeholder Summit 2017, Madrid.
   Google Scholar
• Er, E., E. Gómez-Sánchez, Y. Dimitriadis, M. L. Bote-Lorenzo, J. I. Asensio-Pérez, and S. Álvarez-Álvarez. 2019. “Aligning Learning Design and Learning Analytics Through Instructor Involvement: A MOOC Case Study.” Interactive Learning Environments 27 (5–6): 685–698.
   Web of Science ®Google Scholar
• Fawcett, T. 2006. “An Introduction to ROC Analysis.” Pattern Recognition Letters 27 (8): 861–874.
   Web of Science ®Google Scholar
• Ferguson, R., and D. Clow. 2015. “Examining Engagement: Analysing Learner Subpopulations in Massive Open Online Courses (MOOCs).” Proceedings of the 5th international conference on learning analytics and knowledge, Poughkeepsie, NY, 51–58.
   Google Scholar
• Formanek, M., M. C. Wenger, S. R. Buxner, C. D. Impey, and T. Sonam. 2017. “Insights About Large-Scale Online Peer Assessment From an Analysis of an Astronomy MOOC.” Computers & Education 113: 243–262.
   Web of Science ®Google Scholar
• Gardner, J., and C. Brooks. 2018. “Student Success Prediction in MOOCs.” User Modeling and User-Adapted Interaction 28 (2): 127–203.
   Web of Science ®Google Scholar
• Gehringer, E. F. 2001. “Electronic Peer Review and Peer Grading in Computer-Science Courses.” ACM SIGCSE Bulletin 33 (1): 139–143.
   Google Scholar
• Hanley, J. A., and B. J. McNeil. 1982. “The Meaning and use of the Area Under a Receiver Operating Characteristic (ROC) Curve.” Radiology 143: 29–36.
   PubMed Web of Science ®Google Scholar
• Hastie, T., R. Tibshirani, and J. Friedman. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. New York City, NY: Springer.
   Google Scholar
• Hattie, J., and H. Timperley. 2007. “The Power of Feedback.” Review of Educational Research 77 (1): 81–112.
   Web of Science ®Google Scholar
• Huisman, B., W. Admiraal, O. Pilli, M. van de Ven, and N. Saab. 2016. “Peer Assessment in MOOCs: The Relationship Between Peer Reviewers’ Ability and Authors’ Essay Performance.” British Journal of Educational Technology 49 (1): 101–110.
   Web of Science ®Google Scholar
• Jiang, S., A. E. Williams, K. Schenke, M. Warschauer, and D. O. Dowd. 2014. “Predicting MOOC Performance with week 1 Behavior.” Proceedings of the 7th international conference on educational data mining, London, 273–275.
   Google Scholar
• Kizilcec, R. F., M. Pérez-Sanagustín, and J. J. Maldonado. 2016. “Self-regulated Learning Strategies Predict Learner Behavior and Goal Attainment in Massive Open Online Courses.” Computers & Education 104: 18–33.
   Web of Science ®Google Scholar
• Kizilcec, R. F., C. Piech, and E. Schneider. 2013. “Deconstructing Disengagement: Analyzing Learner Subpopulations in Massive Open Online Courses.” Proceedings of third international conference on learning analytics and knowledge, Leuven, 170–179.
   Google Scholar
• Kulkarni, C., Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R. Klemmer. 2013. “Peer and Self Assessment in Massive Online Classes.” ACM Transactions on Computer Human Interacttion 20 (6): 1–31.
   Web of Science ®Google Scholar
• Lim, L.-A., Sheridan Gentili, Abelardo Pardo, Vitomir Kovanovic, Alexander Whitelock-Wainwright, Dragan Gaševic, Shane Dawson. Forthcoming. “What Changes, and for Whom? A Study of the Impact of Learning Analytics-Based Process Feedback in a Large Course.” Learning and instruction.
   Web of Science ®Google Scholar
• Lockyer, L., E. Heathcote, and S. Dawson. 2013. “Informing Pedagogical Action: Aligning Learning Analytics with Learning Design.” The American Behavioral Scientist 57 (10): 1439–1459.
   Web of Science ®Google Scholar
• Luaces, O., J. Díez, A. Alonso-Betanzos, A. Troncoso, and A. Bahamonde. 2017. “Content-based Methods in Peer Assessment of Open-Response Questions to Grade Students as Authors and as Graders.” Knowledge-Based System 117: 79–87.
   Web of Science ®Google Scholar
• Luft, C. D. B., J. S. Gomes, D. Priori, and E. Takase. 2013. “Using Online Cognitive Tasks to Predict Mathematics low School Achievement.” Computers & Education 67: 219–228.
   Web of Science ®Google Scholar
• Martinez, M. 2013. “Adapting for a Personalized Learning Experience.” In Reshaping Learning. New Frontiers of Educational Research, edited by R. Huang, Kinshuk, and J. M. Spector, 139–174. Berlin: Springer.
   Google Scholar
• Meek, S., L. Blakemore, and L. Marks. 2016. “Is Peer Review an Appropriate Form of Assessment in a MOOC? Student Participation and Performance in Formative Peer Review.” Assessment and Evaluation in Higher Education 42 (6): 1000 –1013.
   Web of Science ®Google Scholar
• Moreno-Marcos, P. M., C. Alario-Hoyos, P. J. Muñoz-Merino, and C. D. Delgado Kloos. 2018. “Prediction in MOOCs: A Review and Future Research Directions.” IEEE Transactions on Learning Technologies 12 (3): 384–401.
   Web of Science ®Google Scholar
• Moreno-Marcos, P. M., P. J. Muñoz-Merino, C. Alario-Hoyos, I. Estévez-Ayres, and C. D. Kloos. 2018. “Analysing the Predictive Power for Anticipating Assignment Grades in a Massive Open Online Course.” Behaviour & Information Technology 37 (10–11): 1021–1036.
   Web of Science ®Google Scholar
• Mouriño-García, M., R. Pérez-Rodríguez, L. Anido-Rifón, M. J. Fernández-Iglesias, and V. M. Darriba-Bilbao. 2018. “Cross-repository Aggregation of Educational Resources.” Computers & Education 117 (September 2017): 31–49.
   Web of Science ®Google Scholar
• Nicol, D., A. Thomson, and C. Breslin. 2014. “Rethinking Feedback Practices in Higher Education: A Peer Review Perspective.” Assessment and Evaluation in Higher Education 39 (1): 102–122.
   Web of Science ®Google Scholar
• Nosek, B. A., G. Alter, G. C. Banks, D. Borsboom, S. D. Bowman, S. J. Breckler, and S. Buck, et al. 2015. “Promoting an Open Research Culture: Author Guidelines for Journals Could Help to Promote Transparency, Openness, and Reproducibility.” Science 348 (6242): 1422–1425.
   PubMed Web of Science ®Google Scholar
• Ortega-Arranz, A., J. A. Muñoz-Cristobal, A. Martínez-Monés, M. L. Bote-Lorenzo, and J. I. Asensio-Pérez. 2017. “How Gamification is Being Implemented in MOOCs? A Systematic Literature Review.” Proceedings of the 12th European conference on technology enhanced learning, Tallinn, vol. 10474, 441–447.
   Google Scholar
• Pedregosa, F., G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, et al. 2011. “Scikit-learn: Machine Learning in Python.” Journal of Machine Learning Research 12: 2825–2830.
   Web of Science ®Google Scholar
• Phan, T., S. G. Mcneil, and B. R. Robin. 2016. “Students’ Patterns of Engagement and Course Performance in a Massive Open Online Course.” Computers & Education 95: 36–44.
   Web of Science ®Google Scholar
• Piech, C., J. Huang, Z. Chen, C. Do, A. Ng, and D. Koller. 2013. “Tuned Models of Peer Assessment in MOOCs.” Proceedings of 6th international conference on educational data mining, Memphis, Tennessee, 153–160.
   Google Scholar
• Rayyan, S., C. Fredericks, K. F. Colvin, A. Liu, R. Teodorescu, A. Barrantes, A. Pawl, et al. 2016. “A MOOC Based on Blended Pedagogy.” Journal of Computer Assisted Learning 32 (3): 190–201.
   Web of Science ®Google Scholar
• Rice, M. E., and G. T. Harris. 2005. “Comparing Effect Sizes in Follow-up Studies: ROC Area, Cohen’s d, and r.” Law and Human Behavior 29 (5): 615–620.
   PubMed Web of Science ®Google Scholar
• Sanz-Martínez, L., J. A. Muñoz-Cristobal, M. L. Bote-Lorenzo, A. Martínez-Monés, and Y. Dimitriadis. 2017. “Toward Criteria-based Automatic Group Formation in MOOCs.” Proceedings of the 5th European MOOCs stakeholders summit, Madrid.
   Google Scholar
• Shah, N. B., J. K. Bradley, A. Parekh, M. Wainwright, and K. Ramchandran. 2013. “A Case for Ordinal Peer-evaluation in MOOCs.” NIPS Workshop on data driven education, Lake Tahoe, Nevada, 1–8.
   Google Scholar
• Šimundic, A.-M. 2009. “Measures of Diagnostic Accuracy: Basic Definitions.” EJIFCC 19 (4): 203–211.
   PubMedGoogle Scholar
• Suen, H. 2014. “Peer Assessment for Massive Open Online Courses (MOOCs).” International Review of Research in Open and Distributed Learning 15 (3): 312–327. doi:10.19173/irrodl.v15i3.1680.
   Web of Science ®Google Scholar
• Tape, T. G. 2008. “The Area Under an ROC Curve.” Accessed July 22, 2017. http://gim.unmc.edu/dxtests/roc3.htm.
   Google Scholar
• Topping, K. 1998. “Peer Assessment Between Students in Colleges and Universities.” Review of Educational Research 68 (3): 249–276.
   Web of Science ®Google Scholar
• Veeramachaneni, K., U.-M. O’Reilly, and C. Taylor. 2014. “Towards Feature Engineering at Scale for Data From Massive Open Online Courses.” arXiv. https://arxiv.org/abs/1407.5238.
   Google Scholar
• Whitehill, J., K. Mohan, D. Seaton, Y. Rosen, and D. Tingley. 2017a. “Delving Deeper Into MOOC Student Dropout Prediction.” arXiv 21–27. https://arxiv.org/abs/1702.06404.
   Google Scholar
• Whitehill, J., K. Mohan, D. Seaton, Y. Rosen, and D. Tingley. 2017b. “MOOC Dropout Prediction: How to Measure Accuracy?” Proceedings of the 4th ACM conference on Learning@Scale, Cambridge, MA, 161–164.
   Google Scholar
• Xing, W., X. Chen, J. Stein, and M. Marcinkowski. 2016. “Temporal Predication of Dropouts in MOOCs: Reaching the low Hanging Fruit Through Stacking Generalization.” Computers in Human Behavior 58: 119–129.
   Web of Science ®Google Scholar