Peer assessment in MOOCs: Systematic literature review

References

• Anaya, A. R., Luque, M., Letón, E., & Hernández‐del‐Olmo, F. (2019). Automatic assignment of reviewers in an online peer assessment task based on social interactions. Expert Systems, 36(4), Article e12405. https://doi.org/10.1111/exsy.12405
   Google Scholar
• Appiah-Kubi, K., & Rowland, D. (2016). Peer support in MOOCs: The role of social presence.In J.Haywood, V. Aleven, J. Kay, & I. Roll (Eds.), Proceedings of the 3rd ACM Conference on Learning @ Scale (pp. 237–240). Association for Computing Machinery. https://doi.org/10.1145/2876034.2893423
   Google Scholar
• Babik, D., Stevens, S., & Waters, A. E. (2019). Comparison of ranking and rating scales in online peer assessment: Simulation approach. In R. Ferguson, U. Hoppe, & C. Brooks (Eds.), Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 205–209). Association for Computing Machinery. https://doi.org/10.1145/3303772.3303820
   Google Scholar
• Babori, A., Fassi, H. F., & Zaid, A. (2019). Research on MOOCs: Current trends and taking into account of content. In B. Abouelmajd, B. A. Abdelhakim, H. E. Ghazi (Ed.), NISS19: Proceedings of the 2nd International Conference on Networking, Information Systems & Security (pp. 1–9). Association for Computing Machinery. https://doi.org/10.1145/3320326.3320349
   Google Scholar
• Bloom, B. S., & Krathwohl, D. R. (Eds.). (1956). Taxonomy of educational objectives: The classification of educational goals inhandbook Icognitive domain. Longman.
   Google Scholar
• Brown, G. T. L, & Harris, L. R. (Eds.). (2016). Handbook of human and social conditions in assessment. Routledge.
   Google Scholar
• Cambre, J., Klemmer, S., & Kulkarni, C. (2018). Juxtapeer: Comparative peer review yields higher quality feedback and promotes deeper reflection. In R. Mandryk, M. Hancock, M. Perry, & A. Cox (Eds.), Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1–13). Association for Computing Machinery. https://doi.org/10.1145/3173574.3173868
   Google Scholar
• Capuano, N., Loia, V., & Orciuoli, F. (2017). A fuzzy group decision making model for ordinal peer assessment. IEEE Transactions on Learning Technologies, 10(2), 247–259. https://doi.org/10.1109/TLT.2016.2565476
   Web of Science ®Google Scholar
• Cheng, H. F., Yu, B., Park, Y. H., & Zhu, H. (2017). ProjectLens: Supporting project-based collaborative learning on MOOCs. In C. Urrea, J. Reich, & C. Thille (Eds.), Proceedings of the 4th ACM Conference on Learning @ Scale (pp. 253–256). Association for Computing Machinery. https://doi.org/10.1145/3051457.3053998
   Google Scholar
• Cho, K., & MacArthur, C. (2011). Learning by reviewing. Journal of Educational Psychology, 103(1), 73–84. https://doi.org/10.1037/a0021950
   Web of Science ®Google Scholar
• Churches, A. (2008, April 1). Bloom’s taxonomy blooms digitally. Tech & Learning. https://www.techlearning.com/news/bloom39s-taxonomy-blooms-digitally
   Google Scholar
• Conrad, D., & Openo, J. (2018). Assessment strategies for online learning: Engagement and authenticity. Athabasca University Press. https://doi.org/10.15215/aupress/9781771992329.01
   Google Scholar
• Cooper, K., & Khosravi, H. (2018). Graph-based visual topic dependency models. In A. Pardo, K. Bartimote-Aufflick, G. Lynch, S. B. Shum, R. Ferguson, A. Merceron, & X. Ochoa (Eds.), Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 11–15). Association for Computing Machinery. https://doi.org/10.1145/3170358.3170418
   Google Scholar
• Duart J. M., Roig-Vila R., Mengual-Andres S., & Duran M. M. (2017). La calidad pedagógica de los MOOC a partir de la revisión sistemática de las publicaciones JCR y Scopus (2013-2015) [The pedagogical quality of MOOCs based on a systematic review of JCR and Scopus publications (2013-2015)]. Revista Espanola de Pedagogia, 75(1), 29–46. https://doi.org/10.22550/REP75-1-2017-02
   Web of Science ®Google Scholar
• Fidalgo-Blanco, Á., Sein-Echaluce, M. L., & García-Peñalvo, F. J. (2016). From massive access to cooperation: Lessons learned and proven results of a hybrid xMOOC/cMOOC pedagogical approach to MOOCs. International Journal of Educational Technology in Higher Education, 13(1), 1–13. https://doi.org/10.1186/s41239-016-0024-z
   Web of Science ®Google Scholar
• Foley, K., Alturkistani, A., Carter, A., Stenfors, T., Blum, E., Car, J., Majeed, A., Brindley, D., & Meinert, E. (2019). Massive open online courses (MOOC) evaluation methods: Protocol for a systematic review. JMIR Research Protocols, 8(3), Article e12087. https://doi.org/10.2196/12087
   Google Scholar
• Formanek, M., Wenger, M. C., Buxner, S. R., Impey, C. D., & Sonam, T. (2017). Insights about large-scale online peer assessment from an analysis of an astronomy MOOC. Computers & Education, 113, 243–262. https://doi.org/10.1016/j.compedu.2017.05.019
   Web of Science ®Google Scholar
• Gamage, D., Fernando, S., & Perera, I. (2015). Factors leading to an effective MOOC from participiants perspective. In K. P. Hewagamage, N. Ling, & T. K. Shih (Eds.), Proceedings of the 8th International Conference on Ubi-Media Computing, (pp. 230–235). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/UMEDIA.2015.7297460
   Google Scholar
• Gamage, D., Perera, I., & Fernando, S. (2018). Increasing interactivity and collaborativeness in MOOCs using facilitated groups: A pedagogical solution to meet 21st century goals. In C. S. González, M. Castro, & M. L. Nistal (Eds.), Proceedings of the 2018 IEEE Global Engineering Education Conference (pp. 354–367). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/EDUCON.2018.8363324
   Google Scholar
• Gamage, D., Whiting, M. E., Perera, I., & Femando, S. (2019). Improving feedback and discussion in MOOC peer assessment using introduced peers. In S. Nikolic & M. J. W. Lee (Eds.), Proceedings of 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (pp. 357–364). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/TALE.2018.8615307
   Google Scholar
• Gamage, D., Whiting, M. E., Rajapakshe, T., Thilakarathne, H., Perera, I., & Fernando, S. (2017). Improving assessment on MOOCs through peer identification and aligned incentives. In C. Urrea, J. Reich, & C. Thille (Eds.), Proceedings of the 4th ACM Conference on Learning @ Scale (pp. 315–318). Association for Computing Machinery. https://doi.org/10.1145/3051457.3054013
   Google Scholar
• García-Martínez, C., Cerezo, R., Bermúdez, M., & Romero, C. (2019). Improving essay peer grading accuracy in massive open online courses using personalized weights from student’s engagement and performance. Journal of Computer Assisted Learning, 35(1), 110–120. https://doi.org/10.1111/jcal.12316
   Web of Science ®Google Scholar
• Glance, D. G., Forsey, M., & Riley, M. (2013). The pedagogical foundations of massive open online courses. First Monday, 18(5). https://doi.org/10.5210%2Ffm.v18i5.4350
   Google Scholar
• Haddadi, L., Bouarab-Dahmani, F., Guin, N., Berkane, T., & Lazib, S. (2018). Peer assessment and groups formation in massive open online courses. Computer Applications in Engineering Education, 26(5), 1873–1887. https://doi.org/10.1002/cae.22005
   Web of Science ®Google Scholar
• Hicks, C. M., Fraser, C. A., Desai, P., & Klemmer, S. (2015). Do numeric ratings impact peer reviewers? In G. Kiczales, D. M. Russell, & B. Woolf (Eds.), Proceedings of the 2nd ACM Conference on Learning @ Scale (pp. 359–362). Association for Computing Machinery. https://doi.org/10.1145/2724660.2728693
   Google Scholar
• Huisman, B., Admiraal, W., Pilli, O., van de Ven, M., & Saab, N. (2018). Peer assessment in MOOCs: The relationship between peer reviewers’ ability and authors’ essay performance. British Journal of Educational Technology, 49(1), 101–110. https://doi.org/10.1111/bjet.12520
   Web of Science ®Google Scholar
• Joyner, D. (2016). Expert evaluation of 300 projects per day. In J. Haywood, V. Aleven, J. Kay, & I. Roll (Eds.), Proceedings of the 3rd 2016 ACM Conference on Learning @ Scale (pp. 121–124). Association for Computing Machinery. https://doi.org/10.1145/2876034.2893384
   Google Scholar
• Joyner, D., Ashby, W., Irish, L., Lam, Y., Langston, J., Lupiani, I., Lustig, M., Pettoruto, P., Sheahen, D., Smiley, A., Bruckman, A., & Goel, A. (2016). Graders as meta-reviewers: Simultaneously scaling and improving expert evaluation for large online classrooms. In J. Haywood, V. Aleven, J. Kay, & I. Roll (Eds.), Proceedings of the 3rd 2016 ACM Conference on Learning @ Scale (pp. 399–408). Association for Computing Machinery. https://doi.org/10.1145/2876034.2876044
   Google Scholar
• Kitchenham, B. (2004). Procedures for performing systematic reviews (Report No. TR/SE-0401). Keele University & Empirical Software Engineering.
   Google Scholar
• Kolhe, P., Littman, M. L., & Isbell, C. L. (2016). Peer reviewing short answers using comparative judgement. In J. Haywood, V. Aleven, J. Kay, & I. Roll (Eds.), Proceedings of the 3rd 2016 ACM Conference on Learning @ Scale (pp. 241–244). Association for Computing Machinery. https://doi.org/10.1145/2876034.2893424
   Google Scholar
• Kotturi, Y., Kulkarni, C., Bernstein, M. S., & Klemmer, S. (2015). Structure and messaging techniques for online peer learning systems that increase stickiness. In G. Kiczales, D. M. Russell, & B. Woolf (Eds.), Proceedings of the 2nd ACM Conference on Learning @ Scale (pp. 31–38). Association for Computing Machinery. https://doi.org/10.1145/2724660.2724676
   Google Scholar
• Kulkarni, C., Bernstein, M. S., & Klemmer, S. (2015). PeerStudio: Rapid peer feedback emphasizes revision and improves performance. In G. Kiczales, D. M. Russell, & B. Woolf (Eds.), Proceedings of the 2nd ACM Conference on Learning @ Scale (pp. 75–84). Association for Computing Machinery. https://doi.org/10.1145/2724660.2724670
   Google Scholar
• Kulkarni, C., Socher, R., Bernstein, M. S., & Klemmer, S. R. (2014). Scaling short-answer grading by combining peer assessment with algorithmic scoring. In M. Sahami, A. Fox, M. A. Hearst, & M. T. H. Chi (Eds.), Proceedings of the 1st ACM Conference on Learning @ Scale (pp. 99–108). Association for Computing Machinery. https://doi.org/10.1145/2556325.2566238
   Google Scholar
• Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013). Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction, 20(6), Article 33. https://doi.org/10.1145/2505057
   Google Scholar
• Lee, D., Watson, S. L., & Watson, W. R. (2019). Systematic literature review on self-regulated learning in massive open online courses. Australasian Journal of Educational Technology, 35(1), 28–41. https://doi.org/10.14742/ajet.3749
   Web of Science ®Google Scholar
• Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2013). MOOCs: A systematic study of the published literature 2008-2012. The International Review of Research in Open and Distance Learning, 14(3), 202–227. https://doi.org/10.19173/irrodl.v14i3.1455
   Web of Science ®Google Scholar
• Luaces, O., Díez, J., Alonso-Betanzos, A., Troncoso, A., & Bahamonde, A. (2015). A factorization approach to evaluate open-response assignments in MOOCs using preference learning on peer assessments. Knowledge-Based Systems, 85, 322–328. https://doi.org/10.1016/j.knosys.2015.05.019
   Web of Science ®Google Scholar
• Luaces, O., Díez, J., & Bahamonde, A. (2018). A peer assessment method to provide feedback, consistent grading and reduce students’ burden in massive teaching settings. Computers & Education, 126, 283–295. https://doi.org/10.1016/j.compedu.2018.07.016
   Web of Science ®Google Scholar
• Martin, N. I., Kelly, N., & Terry, P. C. (2018). A framework for self-determination in massive open online courses: Design for autonomy, competence, and relatedness. Australasian Journal of Educational Technology, 34(2), 35–55. https://doi.org/10.14742/ajet.3722
   Web of Science ®Google Scholar
• Mendoza, L. B., Ortega, M. P., Hormaza, J. M., & Soto, S. V. (2020). Trends the use of artificial intelligence techniques for peer assessment. In R. Uskenbayeva, Y. Daineko, & S. A. Aljawarneh (Eds.), Proceedings of the ACM 6th International Conference on Engineering & MIS 2020 (pp. 1–7). Association for Computing Machinery. https://doi.org/10.1145/3410352.3410837
   Google Scholar
• Milligan, C., Littlejohn, A., & Margaryan, A. (2013). Patterns of engagement in connectivist MOOCs. Journal of Online Learning and Teaching, 9(2), 149–159. https://jolt.merlot.org/vol9no2/milligan_0613.pdf
   Google Scholar
• Montebello, M., Pinheiro, P., Cope, B., Kalantzis, M., Amina, T., Searsmith, D., & Cao, D. (2018). The impact of the peer review process evolution on learner performance in e-learning environments. Proceedings of the Fifth Annual ACM Conference on Learning at Scale, Article 35, 1–3. https://doi.org/10.1145/3231644.3231693
   Google Scholar
• Papathoma, T., Blake, C., Clow, D., & Scanlon, E. (2015). Investigating learners’ views of assessment types in massive open online courses (MOOCs). In G. Conole, T. Klobučar, C. Rensing, J. Konert, & E. Lavoué (Eds.), Lecture notes in computer science, Vol. 9307. Design for teaching and learning in a networked world (pp. 617–662). Springer. https://doi.org/10.1007/978-3-319-24258-3_72
   Google Scholar
• Petticrew, M., & Roberts, H. (2008). Systematic reviews in the social sciences: A practical guide. John Wiley & Sons. https://doi.org/10.1002/9780470754887
   Google Scholar
• Raman, K., & Joachims, T. (2014). Methods for ordinal peer grading. In S. Macskassy, C. Perlich, J. Leskovec, W. Wang, & R. Ghani (Eds.), Proceedings of the International Conference on Knowledge Discovery and Data Mining (pp.1037–1046). Association for Computing Machinery. https://doi.org/10.1145/2623330.2623654
   Google Scholar
• Raman, K., & Joachims, T. (2015). Bayesian ordinal peer grading. In G. Kiczales, D. M. Russell, & B. Woolf (Eds.), Proceedings of the 2nd ACM Conference on Learning @ Scale (pp. 49–156). Association for Computing Machinery. https://doi.org/10.1145/2724660.2724678
   Google Scholar
• Raman, K., Shivaswamy, P., & Joachims, T. (2012). Online learning to diversify from implicit feedback. In Q. Yang, D. Agarwal, & J. Pei (Eds.), Proceedings of the International Conference on Knowledge Discovery and Data Mining (pp. 705–713). Association for Computing Machinery. https://doi.org/10.1145/2339530.2339642
   Google Scholar
• Reilly, E. D., Stafford, R. E., Williams, K. M., & Corliss, S. B. (2014). Evaluating the validity and applicability of automated essay scoring in two massive open online courses. The International Review of Research in Open and Distance Learning, 15(5), 83–98. https://doi.org/10.19173/irrodl.v15i5.1857
   Google Scholar
• Reimer, Y. J., & Douglas, S. A. (2003). Teaching hci design with the studio approach. International Journal of Phytoremediation, 21(1), 191–205. https://doi.org/10.1076/csed.13.3.191.14945
   Google Scholar
• Richards, D. (2009). Designing project-based courses with a focus on group formation and assessment. ACM Transactions on Computing Education, 9(1), 1–40. https://doi.org/10.1145/1513593.1513595
   Google Scholar
• Sajjadi, M. S. M., Alamgir, M., & Von Luxburg, U. (2016). Peer grading in a course on algorithms and data structures: Machine learning algorithms do not improve over simple baselines. In J. Haywood, V. Aleven, J. Kay, & I. Roll (Eds.) Proceedings of the 3rd ACM Conference on Learning @ Scale (pp. 369–378). Association for Computing Machinery. https://doi.org/10.1145/2876034.2876036
   Google Scholar
• Sánchez-Prieto, J. C., Gamazo, A., Cruz-Benito, J., Therón, R., & García-Peñalvo, F. J. (2020). AI-driven assessment of students: Current uses and research trends. In P. Zaphiris & A. Ioannou (Eds.). Lecture notes in computer science, Vol.12205. Learning and collaboration technologies: Designing, developing and deploying learning experiences (pp. 292–302). Springer. https://doi.org/10.1007/978-3-030-50513-4_22
   Google Scholar
• Søndergaard, H. (2009). Learning from and with peers: The different roles of student peer reviewing. In P. Brézillon, I. Russell, & J. Labat (Eds.), Proceedings of the Conference on Integrating Technology into Computer Science Education (pp. 31–35). Association for Computing Machinery. https://doi.org/10.1145/1562877.1562893
   Google Scholar
• Staubitz, T. (2020). Gradable team assignments in large scale learning environments [Doctoral dissertation, Universität Potsdam]. Universität Potsdam Campus Repository. https://doi.org/10.25932/publishup-47183
   Google Scholar
• Staubitz, T., Klement, H., Teusner, R., Renz, J., & Meinel, C. (2016). CodeOcean - A versatile platform for practical programming excercises in online environments. In M. Al-Mualla & M. E. Auer (Eds.), Proceedings of the IEEE Global Engineering Education Conference (pp. 314–324). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/EDUCON.2016.7474573
   Google Scholar
• Staubitz, T., & Meinel, C. (2017). Collaboration and teamwork on a MOOC platform a toolset. In C. Urrea, J. Reich, & C. Thille (Eds.), Proceedings of the 4th ACM Conference on Learning @ Scale (pp. 165–168). Association for Computing Machinery. https://doi.org/10.1145/3051457.3053975
   Google Scholar
• Staubitz, T., & Meinel, C. (2019). Graded team assignments in MOOCs: Effects of team composition and further factors on team dropout rates and performance. In J. C. Mitchell, K. Porayska-Pomsta, D. A. Joyner, S. Biderman, T. Mason, & M. Swenson (Eds.), Proceedings of the 6th ACM Conference on Learning @ Scale (pp. 1–10). Association for Computing Machinery. https://doi.org/10.1145/3330430.3333619
   Google Scholar
• Staubitz, T., & Meinel, C. (2018). Team based assignments in MOOCs: Results and observations. In R. Luckin, S. Klemmer, & K. Koedinger (Eds.), Proceedings of the 5th Annual ACM Conference on Learning @ Scale (pp. 1–4). Association for Computing Machinery. https://doi.org/10.1145/3231644.3231705
   Google Scholar
• Staubitz, T., Petrick, D., Bauer, M., Renz, J., & Meinel, C. (2016). Improving the peer assessment experience on MOOC platforms. In J. Haywood, V. Aleven, J. Kay, & I. Roll (Eds.), Proceedings of the 3rd ACM Conference on Learning @ Scale (pp. 389–398). Association for Computing Machinery. https://doi.org/10.1145/2876034.2876043
   Google Scholar
• Staubitz, T., Traifeh, H., Chujfi, S., & Meinel, C. (2020). Have your tickets ready! impede free riding in large scale team assignments. In D. Joyner, R. Kizilcec, & S. Singer (Eds.), Proceedings of the 7th ACM Conference on Learning @ Scale (pp. 349–352). Association for Computing Machinery. https://doi.org/10.1145/3386527.3406744
   Google Scholar
• Suen, H. K. (2014). Peer assessment for massive open online courses (MOOCs). The International Review of Research in Open and Distance Learning, 15(3), 312–327. https://doi.org/10.19173/irrodl.v15i3.1680
   Web of Science ®Google Scholar
• Tritz, J., Michelotti, N., Shultz, G., McKay, T., & Mohapatra, B. (2014, March). Peer evaluation of student generated content. In M. Pistilli, J. Willis, D. Koch, K. Arnold, S. Teasley, & A. Pardo (Eds.), Proceedings of the 4th International Conference on Learning Analytics and Knowledge (pp. 277–278). Association for Computing Machinery. https://doi.org/10.1145/2567574.2567598
   Google Scholar
• Uto, M., Nguyen, D. T., & Ueno, M. (2020). Group optimization to maximize peer assessment accuracy using item response theory and integer programming. IEEE Transactions on Learning Technologies, 13(1), 91–106. https://doi.org/10.1109/TLT.2019.2896966
   Web of Science ®Google Scholar
• Veletsianos, G., & Shepherdson, P. (2016). A systematic analysis and synthesis of the empirical MOOC literature published in 2013-2015. The International Review of Research in Open and Distance Learning, 17(2), 198–221. https://doi.org/10.19173/irrodl.v17i2.2448
   Google Scholar
• Vogelsang, T., & Ruppertz, L. (2015). On the validity of peer grading and a cloud teaching assistant system. In J. Baron, G. Lynch, N. Maziarz, P. Blikstein, A. Merceron, & G. Siemens (Eds.), Proceedings of the 5th International Conference on Learning Analytics and Knowledge (pp. 41–50). Association for Computing Machinery. https://doi.org/10.1145/2723576.2723633
   Google Scholar
• Wang, Y., Fang, H., Jin, Q., & Ma, J. (2019). SSPA: An effective semi-supervised peer assessment method for large scale MOOCs. Interactive Learning Environments. https://doi.org/10.1080/10494820.2019.1648299
   Web of Science ®Google Scholar
• Wen, M., Maki, K., Dow, S. P., Herbsleb, J., & Rose, C. (2017). Supporting virtual team formation through community-wide deliberation. Proceedings of the ACM on Human-Computer Interaction, 1, 1–19. https://doi.org/10.1145/3134744
   Google Scholar
• Wu, W., Tzamos, C., Daskalakis, C., Weinberg, M., & Kaashoek, N. (2015). Game theory based peer grading mechanisms for MOOCs. In G. Kiczales, D. M. Russell, & B. Woolf (Eds.), Proceedings of 2nd ACM Conference on Learning @ Scale (pp. 281–286). Association for Computing Machinery. https://doi.org/10.1145/2724660.2728676
   Google Scholar
• Yousef, A. M., Chatti, M. A., Schroeder, U., & Wosnitza, M. (2015). A usability evaluation of a blended MOOC environment: An experimental case study. The International Review of Research in Open and Distributed Learning, 16(2). https://doi.org/10.19173/irrodl.v16i2.2032
   Web of Science ®Google Scholar
• Zheng, L., Chen, N. S., Cui, P., & Zhang, X. (2019). A systematic review of technology-supported peer assessment research: An activity theory approach. The International Review of Research in Open and Distance Learning, 20(5), 168–191. https://doi.org/10.19173/irrodl.v20i5.4333
   Google Scholar
• Zhu, M., Sari, A. & Bonk, C. (2018). A systematic review of MOOC research methods and topics: Comparing 2014-2016 and 2016-2017. In T. Bastiaens, J. Van Braak, M. Brown, L. Cantoni, M. Castro, R. Christensen, G. Davidson-Shivers, K. DePryck, M. Ebner, M. Fominykh, C. Fulford, S. Hatzipanagos, G. Knezek, K. Kreijns, G. Marks, E. Sointu, E. Korsgaard Sorensen, J. Viteli, J. Voogt, P. Weber, E. Weippl & O. Zawacki-Richter (Eds.), Proceedings of the World Conference on Educational Media and Technology (pp. 1673–1682). Association for the Advancement of Computing in Education. https://www.learntechlib.org/primary/p/184395/
   Google Scholar