Effects of an automated programming assessment system on the learning performances of experienced and novice learners

References

• Anderson, J. R. (1988). Acquisition of cognitive skill. Readings in Cognitive Science, 362–380. https://doi.org/10.1016/b978-1-4832-1446-7.50032-7
   Google Scholar
• Baxter, R. J., Holderness, D. K., & Wood, D. A. (2015). Applying basic Gamification techniques to IT compliance training: Evidence from the lab and Field. Journal of Information Systems, 30(3), 119–133. https://doi.org/10.2308/isys-51341
   Google Scholar
• Beck, J. E., & Gong, Y. (2013). Wheel-Spinning: Students who fail to master a skill. Lecture Notes in Computer Science, 431–440. https://doi.org/10.1007/978-3-642-39112-5_44
   Google Scholar
• Bloom, B. S. (1968). Learning for Mastery. Instruction and Curriculum. Regional Education Laboratory for the Carolinas and Virginia, Topical Papers and Reprints, Number 1. Evaluation comment, 1(2), n2.
   Google Scholar
• Brito, M. A., & de Sá-Soares, F. (2014). Assessment frequency in introductory computer programming disciplines. Computers in Human Behavior, 30, 623–628. https://doi.org/10.1016/j.chb.2013.07.044
   Web of Science ®Google Scholar
• Brooks, R. (1990). Categories of programming knowledge and their application. International Journal of Man-Machine Studies, 33(3), 241–246. https://doi.org/10.1016/s0020-7373(05)80118-x
   Google Scholar
• Brown, N. C. C., & Wilson, G. (2018). Ten quick tips for teaching programming. PLoS Computational Biology, 14(4), e1006023. https://doi.org/10.1371/journal.pcbi.1006023
   PubMed Web of Science ®Google Scholar
• Buitrago Flórez, F., Casallas, R., Hernández, M., Reyes, A., Restrepo, S., & Danies, G. (2017). Changing a generation’s Way of thinking: Teaching computational thinking through programming. Review of Educational Research, 87(4), 834–860. https://doi.org/10.3102/0034654317710096
   Web of Science ®Google Scholar
• Burguillo, J. C. (2010). Using game theory and competition-based learning to stimulate student motivation and performance. Computers & Education, 55(2), 566–575. https://doi.org/10.1016/j.compedu.2010.02.018
   Web of Science ®Google Scholar
• Carnaghan, C., & Webb, A. (2006). Investigating the effects of group response systems on student satisfaction, learning and engagement in accounting education. SSRN Electronic Journal, https://doi.org/10.2139/ssrn.959370
   Google Scholar
• Chen, C., Haduong, P., Brennan, K., Sonnert, G., & Sadler, P. (2018). The effects of first programming language on college students’ computing attitude and achievement: A comparison of graphical and textual languages. Computer Science Education, 29(1), 23–48. https://doi.org/10.1080/08993408.2018.1547564
   Web of Science ®Google Scholar
• Chen, H. M., Chen, W. H., Hsueh, N. L., Lee, C. C., & Li, C. H. (2017). ProgEdu - an automatic assessment platform for programming courses. In Meen, Prior, & Lam (Eds.), Proceeding of the 2017 IEEE international conference on applied system innovation (pp. 173–176). (May 13–17, 2017. Sapporo, Japan). https://doi.org/10.1109/icasi.2017.7988376
   Google Scholar
• Chen, X., Xie, H., Zou, D., & Hwang, G. J. (2020). Application and theory gaps during the rise of Artificial Intelligence in education. Computers and Education: Artificial Intelligence, 1, 100002. https://doi.org/10.1016/j.caeai.2020.100002
   Google Scholar
• Chien, Y. T., Chang, Y. H., & Chang, C. Y. (2016). Do we click in the right way? A meta-analytic review of clicker-integrated instruction. Educational Research Review, 17, 1–18. https://doi.org/10.1016/j.edurev.2015.10.003
   Web of Science ®Google Scholar
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
   Google Scholar
• Daradoumis, T., Marquès Puig, J. M., Arguedas, M., & Calvet Liñan, L. (2019). Analyzing students’ perceptions to improve the design of an automated assessment tool in online distributed programming. Computers & Education, 128, 159–170. https://doi.org/10.1016/j.compedu.2018.09.021
   Web of Science ®Google Scholar
• Douce, C., Livingstone, D., & Orwell, J. (2005). Automatic test-based assessment of programming: A review. Journal on Educational Resources in Computing, 5(3), 4-es. https://doi.org/10.1145/1163405.1163409
   Google Scholar
• Evans, C. (2013). Making sense of assessment feedback in higher education. Review of Educational Research, 83(1), 70–120. https://doi.org/10.3102/0034654312474350
   Web of Science ®Google Scholar
• Fabic, G. V. F., Mitrovic, A., & Neshatian, K. (2018). Investigating the effects of learning activities in a mobile Python tutor for targeting multiple coding skills. Research and Practice in Technology Enhanced Learning, 13(1), https://doi.org/10.1186/s41039-018-0092-x
   PubMedGoogle Scholar
• Fessakis, G., Gouli, E., & Mavroudi, E. (2013). Problem solving by 5–6 years old kindergarten children in a computer programming environment: A case study. Computers & Education, 63, 87–97. https://doi.org/10.1016/j.compedu.2012.11.016
   Web of Science ®Google Scholar
• Feurzeig, W., & Papert, S. A. (2011). Programming-languages as a conceptual framework for teaching mathematics. Interactive Learning Environments, 19(5), https://doi.org/10.1080/10494820903520040
   Web of Science ®Google Scholar
• Flores, R. M., & Rodrigo, M. M. T. (2020). Wheel-Spinning models in a novice programming context. Journal of Educational Computing Research, 58(6), 1101–1120. https://doi.org/10.1177/0735633120906063
   Web of Science ®Google Scholar
• Forsström, S. E., & Kaufmann, O. T. (2018). A literature review exploring the use of programming in mathematics education. International Journal of Learning, Teaching and Educational Research, 17(12), 18–32. https://doi.org/10.26803/ijlter.17.12.2
   Google Scholar
• Gálvez, J., Guzmán, E., & Conejo, R. (2009). A blended E-learning experience in a course of object oriented programming fundamentals. Knowledge-Based Systems, 22(4), 279–286. https://doi.org/10.1016/j.knosys.2009.01.004
   Web of Science ®Google Scholar
• Govender, I., & Grayson, D. J. (2008). Pre-service and in-service teachers’ experiences of learning to program in an object-oriented language. Computers & Education, 51(2), 874–885. https://doi.org/10.1016/j.compedu.2007.09.004
   Web of Science ®Google Scholar
• Grover, S., & Pea, R. (2013). Computational thinking in K–12. Educational Researcher, 42(1), 38–43. https://doi.org/10.3102/0013189x12463051
   Web of Science ®Google Scholar
• Hanus, M. D., & Fox, J. (2015). Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance. Computers & Education, 80, 152–161. https://doi.org/10.1016/j.compedu.2014.08.019
   Web of Science ®Google Scholar
• Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
   Web of Science ®Google Scholar
• Hidi, S. (2006). Interest: A unique motivational variable. Educational Research Review, 1(2), 69–82. https://doi.org/10.1016/j.edurev.2006.09.001
   Google Scholar
• Hislop, G. W. (1999). Anytime, anyplace learning in an online graduate professional degree program. Group Decision and Negotiation, 8(5), 385–390. https://doi.org/10.1023/a:1008609425784
   Web of Science ®Google Scholar
• Hsu, T. C., Chang, S. C., & Hung, Y. T. (2018). How to learn and how to teach computational thinking: Suggestions based on a review of the literature. Computers & Education, 126, 296–310. https://doi.org/10.1016/j.compedu.2018.07.004
   Web of Science ®Google Scholar
• Huang, Y. H., Lin, K. C., Yu, X., & Hung, J. C. (2015). Comparison of different approaches on example-based learning for novice and proficient learners. Human-Centric Computing and Information Sciences, 5(1), 1. https://doi.org/10.1186/s13673-015-0048-8
   Google Scholar
• Hwang, G. J., & Chang, H. F. (2011). A formative assessment-based mobile learning approach to improving the learning attitudes and achievements of students. Computers & Education, 56(4), 1023–1031. https://doi.org/10.1016/j.compedu.2010.12.002
   Web of Science ®Google Scholar
• Hwang, G. J., Xie, H., Wah, B. W., & Gašević, D. (2020). Vision, challenges, roles and research issues of Artificial Intelligence in education. Computers & Education: Artificial Intelligence, 1, 100001. https://doi.org/10.1016/j.caeai.2020.100001
   Google Scholar
• Hwang, G. J., Yang, L. H., & Wang, S. Y. (2013). A concept map-embedded educational computer game for improving students’ learning performance in natural science courses. Computers & Education, 69, 121–130. https://doi.org/10.1016/j.compedu.2013.07.008
   Web of Science ®Google Scholar
• Hwang, W. Y., Wang, C. Y., Hwang, G. J., Huang, Y. M., & Huang, S. (2008). A web-based programming learning environment to support cognitive development. Interacting with Computers, 20(6), 524–534. https://doi.org/10.1016/j.intcom.2008.07.002
   Web of Science ®Google Scholar
• Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010). Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli calling international conference on computing education research, Koli, Finland, Oct 28-31, 2010. https://doi.org/10.1145/1930464.1930480
   Google Scholar
• Insa, D., & Silva, J. (2018). Automatic assessment of Java code. Computer Languages, Systems & Structures, 53, 59–72. https://doi.org/10.1016/j.cl.2018.01.004
   Google Scholar
• Kafai, Y. B., & Burke, Q. (2013). Computer programming goes back to school. Phi Delta Kappan, 95(1), 1. https://doi.org/10.1177/003172171309500111
   Web of Science ®Google Scholar
• Kalelioğlu, F. (2015). A new way of teaching programming skills to K-12 students: Code.org. Computers in Human Behavior, 52, 200–210. https://doi.org/10.1016/j.chb.2015.05.047
   Web of Science ®Google Scholar
• Kalles, D. (2008). Students working for students on programming courses. Computers & Education, 50(1), 91–97. https://doi.org/10.1016/j.compedu.2006.03.003
   Web of Science ®Google Scholar
• Katai, Z., & Toth, L. (2010). Technologically and artistically enhanced multi-sensory computer-programming education. Teaching and Teacher Education, 26(2), 244–251. https://doi.org/10.1016/j.tate.2009.04.012
   Web of Science ®Google Scholar
• Kazakoff, E. R., Sullivan, A., & Bers, M. U. (2012). The effect of a classroom-based intensive robotics and programming workshop on sequencing ability in early childhood. Early Childhood Education Journal, 41(4), 245–255. https://doi.org/10.1007/s10643-012-0554-5
   Google Scholar
• Linn, M. C., & Dalbey, J. (1985). Cognitive consequences of programming instruction: Instruction, access, and ability. Educational Psychologist, 20(4), 191–206. https://doi.org/10.1207/s15326985ep2004_4
   Web of Science ®Google Scholar
• Lye, S. Y., & Koh, J. H. L. (2014). Review on teaching and learning of computational thinking through programming: What is next for K-12? Computers in Human Behavior, 41, 51–61. https://doi.org/10.1016/j.chb.2014.09.012
   Web of Science ®Google Scholar
• Machanick, P. (2007). Teaching Java backwards. Computers & Education, 48(3), 396–408. https://doi.org/10.1016/j.compedu.2005.01.009
   Web of Science ®Google Scholar
• Mioduser, D., Levy, S. T., & Talis, V. (2009). Episodes to scripts to rules: Concrete-abstractions in kindergarten children’s explanations of a robot’s behavior. International Journal of Technology and Design Education, 19(1), 15–36. https://doi.org/10.1007/s10798-007-9040-6
   Web of Science ®Google Scholar
• Nouri, J., Zhang, L., Mannila, L., & Norén, E. (2019). Development of computational thinking, digital competence and 21st century skills when learning programming in K-9. Education Inquiry, 11(1), 1–17. https://doi.org/10.1080/20004508.2019.1627844
   Google Scholar
• Nurulain Mohd Rum, S., & Zolkepli, M. (2018). Metacognitive strategies in teaching and learning computer programming. International Journal of Engineering & Technology, 7(4.38), 788–794. https://doi.org/10.14419/ijet.v7i4.38.27546
   Google Scholar
• Papadakis, S., Kalogiannakis, M., & Zaranis, N. (2016). Developing fundamental programming concepts and computational thinking with ScratchJr in preschool education: A case study. International Journal of Mobile Learning and Organisation, 10(3), 187. https://doi.org/10.1504/ijmlo.2016.077867
   Google Scholar
• Pintrich, P. R., & de Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82(1), 33–40. https://doi.org/10.1037/0022-0663.82.1.33
   Web of Science ®Google Scholar
• Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the motivated strategies for learning questionnaire (MSLQ). MI: National Center for Research to Improve Postsecondary Teaching and Learning (ERIC Document Reproduction Service No. ED 338122).
   Google Scholar
• Rahimi, E., Barendsen, E., & Henze, I. (2018). An instructional model to link designing and conceptual understanding in secondary computer science education. In Proceedings of the 13th workshop in primary and secondary computing education, New York, NY.
   Google Scholar
• Rahmat, M., Shahrani, S., Latih, R., Yatim, N. F. M., Zainal, N. F. A., & Rahman, R. A. (2012). Major problems in basic programming that influence student performance. Procedia - Social and Behavioral Sciences, 59, 287–296. https://doi.org/10.1016/j.sbspro.2012.09.277
   Google Scholar
• Rich, P. J., Browning, S. F., Perkins, M., Shoop, T., Yoshikawa, E., & Belikov, O. M. (2018). Coding in K-8: International Trends in teaching elementary/primary computing. TechTrends, 63(3), 311–329. https://doi.org/10.1007/s11528-018-0295-4
   Web of Science ®Google Scholar
• Robins, A., Rountree, J., & Rountree, N. (2003). Learning and teaching programming: A review and discussion. Computer Science Education, 13(2), 137–172. https://doi.org/10.1076/csed.13.2.137.14200
   Google Scholar
• Román-González, M., Pérez-González, J.-C., & Jiménez-Fernández, C. (2017). Which cognitive abilities underlie computational thinking? Criterion validity of the computational thinking test. Computers in Human Behavior, 72, 678–691. https://doi.org/10.1016/j.chb.2016.08.047
   Web of Science ®Google Scholar
• Rubio Sánchez, M., Kinnunen, P., Pareja Flores, C., & Velázquez Iturbide, Á. (2014). Student perception and usage of an automated programming assessment tool. Computers in Human Behavior, 31, 453–460. https://doi.org/10.1016/j.chb.2013.04.001
   Web of Science ®Google Scholar
• Sabag, N., & Kosolapov, S. (2012). Using instant feedback system and micro exams to enhance active learning. American Journal of Engineering Education (AJEE), 3(2), 115–122. https://doi.org/10.19030/ajee.v3i2.7442
   Google Scholar
• Scherer, R. (2016). Learning from the past-The need for empirical evidence on the transfer effects of computer programming skills. Frontiers in Psychology, 7, 1390. https://doi.org/10.3389/fpsyg.2016.01390
   PubMed Web of Science ®Google Scholar
• Sharan, S. (1980). Cooperative learning in small groups: Recent methods and effects on achievement, attitudes, and ethnic relations. Review of Educational Research, 50(2), 241–271. https://doi.org/10.3102/00346543050002241
   Web of Science ®Google Scholar
• Shi, N., Cui, W., Zhang, P., & Sun, X. (2017). Evaluating the effectiveness roles of variables in the novice programmers learning. Journal of Educational Computing Research, 56(2), 181–201. https://doi.org/10.1177/0735633117707312
   Web of Science ®Google Scholar
• Shute, V. J. (2008). Focus on formative feedback. ETS Research Report Series, 2007(1), i–47. https://doi.org/10.1002/j.2333-8504.2007.tb02053.x
   Google Scholar
• Shute, V. J., Sun, C., & Asbell-Clarke, J. (2017). Demystifying computational thinking. Educational Research Review, 22, 142–158. https://doi.org/10.1016/j.edurev.2017.09.003
   Web of Science ®Google Scholar
• Siddiqui, A., Khan, M., & Akhtar, S. (2008). Supply chain simulator: A scenario-based educational tool to enhance student learning. Computers & Education, 51(1), 252–261. https://doi.org/10.1016/j.compedu.2007.05.008
   Web of Science ®Google Scholar
• Slavin, R. E. (1996). Research on cooperative learning and achievement: What we know, what we need to know. Contemporary Educational Psychology, 21(1), 43–69. https://doi.org/10.1006/ceps.1996.0004
   Web of Science ®Google Scholar
• Umapathy, K., & Ritzhaupt, A. D. (2017). A meta-analysis of pair-programming in computer programming courses. ACM Transactions on Computing Education, 17(4), 1–13. https://doi.org/10.1145/2996201
   Web of Science ®Google Scholar
• Waite, J., Curzon, P., Marsh, W., & Sentance, S. (2020). Difficulties with design: The challenges of teaching design in K-5 programming. Computers & Education, 150. https://doi.org/10.1016/j.compedu.2020.103838
   Web of Science ®Google Scholar
• Wang, L., & Chen, M. (2010). The effects of game strategy and preference-matching on flow experience and programming performance in game-based learning. Innovations in Education and Teaching International, 47(1), 39–52. https://doi.org/10.1080/14703290903525838
   Web of Science ®Google Scholar
• Wang, T., Su, X., Ma, P., Wang, Y., & Wang, K. (2011). Ability-training-oriented automated assessment in introductory programming course. Computers & Education, 56(1), 220–226. https://doi.org/10.1016/j.compedu.2010.08.003
   Web of Science ®Google Scholar
• Wang, Y., Li, H., Feng, Y., Jiang, Y., & Liu, Y. (2012). Assessment of programming language learning based on peer code review model: Implementation and experience report. Computers & Education, 59(2), 412–422. https://doi.org/10.1016/j.compedu.2012.01.007
   Web of Science ®Google Scholar
• Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35. https://doi.org/10.1145/1118178.1118215
   Web of Science ®Google Scholar
• Yang, T. C., Chen, S. Y., & Hwang, G. J. (2015). The influences of a two-tier test strategy on student learning: A lag sequential analysis approach. Computers & Education, 82, 366–377. https://doi.org/10.1016/j.compedu.2014.11.021
   Web of Science ®Google Scholar