72
Views
0
CrossRef citations to date
0
Altmetric
Research Article

AI-assisted evaluation of problem-solving performance using eye movement and handwriting

ORCID Icon
Received 07 May 2023, Accepted 02 Apr 2024, Published online: 09 May 2024

References

  • Akram, U., Ellis, J. G., Cau, G., Hershaw, F., Rajenthran, A., Lowe, M., Trommelen, C., & Drabble, J. (2021). Eye tracking and attentional bias for depressive internet memes in depression. Experimental Brain Research, 239(2), 575–581. https://doi.org/10.1007/s00221-020-06001-8
  • Azevedo, R., Bouchet, F., Duffy, M., Harley, J., Taub, M., Trevors, G., Cloude, E., Dever, D., Wiedbusch, M., Wortha, F., & Cerezo, R. (2022). Lessons learned and future directions of metatutor: Leveraging multichannel data to scaffold self-regulated learning with an intelligent tutoring system. Frontiers in Psychology, 13, 813632. https://doi.org/10.3389/fpsyg.2022.813632
  • Balyan, R., McCarthy, K. S., & McNamara, D. S. (2020). Applying natural language processing and hierarchical machine learning approaches to text difficulty classification. International Journal of Artificial Intelligence in Education, 30(3), 337–370. https://doi.org/10.1007/s40593-020-00201-7
  • Bednarik, R., Eivazi, S., & Vrzakova, H. (2013). A computational approach for prediction of problem-solving behavior using support vector machines and eye-tracking data. In Y. I. Nakano, C. Conati, & T. Bader (Eds.), Eye gaze in intelligent user interfaces: Gaze-based analyses, models and applications (pp. 111–134). Springer.
  • Ben Khedher, A., Frasson, C. (2016). Predicting user learning performance from eye movements during interaction with a serious game. Paper presented at the EdMedia + Innovate Learning 2016, Vancouver, BC, Canada. Retrieved from https://www.learntechlib.org/p/173149
  • Bera, P., Soffer, P., & Parsons, J. (2019). Using eye tracking to expose cognitive processes in understanding conceptual models. MIS Quarterly, 43(4), 1105–1126.
  • Boels, L., Moreno-Esteva, E. G., Bakker, A., & Drijvers, P. (2023). Automated gaze-based identification of students’ strategies in histogram tasks through an interpretable mathematical model and a machine learning algorithm. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00368-9
  • Booth, R. W., & Weger, U. W. (2013). The function of regressions in reading: Backward eye movements allow rereading. Memory & Cognition, 41(1), 82–97. https://doi.org/10.3758/s13421-012-0244-y
  • Brownlee, J. (2019). A gentle introduction to the rectified linear unit (relu). Machine Learning Mastery. Retrieved from https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/
  • Buckingham Shum, S. J., & Luckin, R. (2019). Learning analytics and AI: Politics, pedagogy and practices. British Journal of Educational Technology, 50(6), 2785–2793. https://doi.org/10.1111/bjet.12880
  • Canbek, G., Sagiroglu, S., Temizel, T. T., & Baykal, N. (2017, October 5–8). Binary classification performance measures/metrics: A comprehensive visualized roadmap to gain new insights. Paper presented at the 2017 International Conference on Computer Science and Engineering (UBMK). https://doi.org/10.1109/UBMK.2017.8093539
  • Carpenter, P. A., & Just, M. A. (1978). Eye fixations during mental rotation. In J. W. Senders, D. F. Fisher, & R. A. Monty (Eds.), Eye movements and the higher psychological functions (pp. 115–133). Erlbaum.
  • Carpenter, P. A., & Shah, P. (1998). A model of the perceptual and conceptual processes in graph comprehension. Journal of Experimental Psychology: Applied, 4(2), 75–100. https://doi.org/10.1037/1076-898X.4.2.75
  • Chollet, F, others (2018). Keras. GitHub. Retrieved from https://github.com/fchollet/keras
  • Chung, J., Gulcehre, C., Cho, K., Bengio, Y. (2015). Gated feedback recurrent neural networks. Paper presented at the Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research. https://proceedings.mlr.press/v37/chung15.html
  • Chung, H., & Shin, K-s (2018). Genetic algorithm-optimized long short-term memory network for stock market prediction. Sustainability, 10(10), 3765. https://doi.org/10.3390/su10103765
  • Conati, C., Lallé, S., Rahman, M. A., & Toker, D. (2020). Comparing and combining interaction data and eye-tracking data for the real-time prediction of user cognitive abilities in visualization tasks. ACM Transactions on Interactive Intelligent Systems, 10(2), 1–41. Article 12. https://doi.org/10.1145/3301400
  • Cukurova, M., Kent, C., & Luckin, R. (2019). Artificial intelligence and multimodal data in the service of human decision-making: A case study in debate tutoring. British Journal of Educational Technology, 50(6), 3032–3046. https://doi.org/10.1111/bjet.12829
  • Dessì, D., Fenu, G., Marras, M., & Reforgiato Recupero, D. (2019). Bridging learning analytics and cognitive computing for big data classification in micro-learning video collections. Computers in Human Behavior, 92, 468–477. https://doi.org/10.1016/j.chb.2018.03.004
  • D’Zurilla, T. J., & Goldfried, M. R. (1971). Problem solving and behavior modification. Journal of Abnormal Psychology, 78(1), 107–126. https://doi.org/10.1037/h0031360
  • Eckstein, M. P. (1998). The lower visual search efficiency for conjunctions is due to noise and not serial attentional processing. Psychological Science, 9(2), 111–118. https://doi.org/10.1111/1467-9280.00020
  • Emhardt, S. N., Kok, E. M., Jarodzka, H., Brand-Gruwel, S., Drumm, C., & van Gog, T. (2020). How experts adapt their gaze behavior when modeling a task to novices. Cognitive Science, 44(9), e12893. https://doi.org/10.1111/cogs.12893
  • Ertam, F., Aydın, G. (2017). Data classification with deep learning using Tensorflow. Paper presented at the 2017 International Conference on Computer Science and Engineering (UBMK). https://doi.org/10.1109/UBMK.2017.8093521
  • Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. https://doi.org/10.1016/j.patrec.2005.10.010
  • Feng, J., Lu, S. (2019). (). Performance analysis of various activation functions in artificial neural networks. Paper presented at the Journal of Physics: Conference Series. https://doi.org/10.1088/1742-6596/1237/2/022030
  • Fritz, C. O., Morris, P. E., & Richler, J. J. (2012). Effect size estimates: Current use, calculations, and interpretation. Journal of Experimental Psychology. General, 141(1), 2–18. https://doi.org/10.1037/a0024338
  • Gillan, D. J. (1995). Visual arithmetic, computational graphics, and the spatial metaphor. Human Factors, 37(4), 766–780. https://doi.org/10.1518/001872095778995571
  • Glorot, X., Bordes, A., Bengio, Y. (2011). (). Deep sparse rectifier neural networks. Paper presented at the Proceedings of the fourteenth international conference on artificial intelligence and statistics.
  • Grant, E. R., & Spivey, M. J. (2003). Eye movements and problem solving: Guiding attention guides thought. Psychological Science, 14(5), 462–466. https://doi.org/10.1111/1467-9280.02454
  • Graves, A., Fernández, S., Schmidhuber, J. (2005). Bidirectional LSTM networks for improved phoneme classification and recognition. Paper presented at the International Conference on Artificial Neural Networks.
  • Graves, A., Jaitly, N., Mohamed, A-r (2013). Hybrid speech recognition with deep bidirectional LSTM. Paper presented at the 2013 IEEE workshop on automatic speech recognition and understanding.
  • Grivokostopoulou, F., Perikos, I., & Hatzilygeroudis, I. (2017). An educational system for learning search algorithms and automatically assessing student performance. International Journal of Artificial Intelligence in Education, 27(1), 207–240. https://doi.org/10.1007/s40593-016-0116-x
  • Guo, Y., Freer, D., Deligianni, F., & Yang, G. Z. (2022). Eye-tracking for performance evaluation and workload estimation in space telerobotic training. IEEE Transactions on Human-Machine Systems, 52(1), 1–11. https://doi.org/10.1109/THMS.2021.3107519
  • Harrison, W. J., Mattingley, J. B., & Remington, R. W. (2013). Eye movement targets are released from visual crowding. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 33(7), 2927–2933. https://doi.org/10.1523/jneurosci.4172-12.2013
  • Hegarty, M., Mayer, R. E., & Green, C. E. (1992). Comprehension of arithmetic word problems: Evidence from students’ eye fixation. Journal of Educational Psychology, 84(1), 76–84. https://doi.org/10.1037/0022-0663.84.1.76
  • Hegarty, M., Mayer, R. E., & Monk, C. A. (1995). Comprehension of arithmetic word problems: A comparison of successful and unsuccessful problem solvers. Journal of Educational Psychology, 87(1), 18–32. https://doi.org/10.1037/0022-0663.87.1.18
  • Hochreiter, S. (1998). The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 06(02), 107–116. https://doi.org/10.1142/S0218488598000094
  • Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
  • Hofmaenner, D. A., Herling, A., Klinzing, S., Wegner, S., Lohmeyer, Q., Schuepbach, R. A., & Buehler, P. K. (2020). Use of eye tracking in analyzing distribution of visual attention among critical care nurses in daily professional life: An observational study. Journal of Clinical Monitoring and Computing, 35(6), 1511–1518. https://doi.org/10.1007/s10877-020-00628-2
  • Hossin, M., & Sulaiman, M. N. (2015). A review on evaluation metrics for data classification evaluations. International Journal of Data Mining & Knowledge Management Process, 5(2), 01–11. https://doi.org/10.5121/ijdkp.2015.5201
  • Howard, C., Jordan, P., Di Eugenio, B., & Katz, S. (2015). Shifting the load: A peer dialogue agent that encourages its human collaborator to contribute more to problem solving. International Journal of Artificial Intelligence in Education, 27(1), 101–129. https://doi.org/10.1007/s40593-015-0071-y
  • Jonassen, D. H. (2000). Toward a design theory of problem solving. Educational Technology Research and Development, 48(4), 63–85. https://doi.org/10.1007/BF02300500
  • Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixation to comprehension. Psychological Review, 87(4), 329–354. https://doi.org/10.1037/0033-295X.87.4.329
  • Ketkar, N. (2017). Stochastic gradient descent. In Deep learning with python. (pp. 113–132) Springer.
  • Khaldi, M. (2024). Technological tools for innovative teaching. IGI Global.
  • Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. Paper presented at the The 3rd International Conference for Learning Representations, San Diego.
  • Konrad, R., Padmanaban, N., Buckmaster, J. G., Boyle, K. C., & Wetzstein, G. (2024). Gazegpt: Augmenting human capabilities using gaze-contingent contextual ai for smart eyewear. Retrieved from https://arxiv.org/abs/2401.17217
  • Koochaki, F., & Najafizadeh, L. (2021). A data-driven framework for intention prediction via eye movement with applications to assistive systems. IEEE Transactions on Neural Systems and Rehabilitation Engineering: A Publication of the IEEE Engineering in Medicine and Biology Society, 29, 974–984. https://doi.org/10.1109/TNSRE.2021.3083815
  • LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539
  • Lee, S., Ke, F., & Ryu, J. (2020). Engagement and effectiveness of symbolic and iconic learning support for math problem representation: An eye tracking study. Interactive Learning Environments, 31(3), 1514–1531. https://doi.org/10.1080/10494820.2020.1848877
  • Lin, J. J. H., & Lin, S. S. J. (2014). Tracking eye movements when solving geometry problems with handwriting devices. Journal of Eye Movement Research, 7(1), 1–15. https://doi.org/10.16910/jemr.7.1.2
  • Lin, J. J. H., & Lin, S. S. J. (2016). Integrating eye trackers with handwriting tablets to discover difficulties of solving geometry problems. British Journal of Educational Technology, 49(1), 17–29. https://doi.org/10.1111/bjet.12517
  • Malone, S., Altmeyer, K., Vogel, M., & Brünken, R. (2020). Homogeneous and heterogeneous multiple representations in equation-solving problems: An eye-tracking study. Journal of Computer Assisted Learning, 36(6), 781–798. https://doi.org/10.1111/jcal.12426
  • Mayer, R. E. (2010). Unique contributions of eye-tracking research to the study of learning with graphics. Learning and Instruction, 20(2), 167–171. https://doi.org/10.1016/j.learninstruc.2009.02.012
  • Miller, H. E., Kirkorian, H. L., & Simmering, V. R. (2020). Using eye-tracking to understand relations between visual attention and language in children’s spatial skills. Cognitive Psychology, 117, 101264. https://doi.org/10.1016/j.cogpsych.2019.101264
  • Molenaar, I., Mooij, S. d., Azevedo, R., Bannert, M., Järvelä, S., & Gašević, D. (2023). Measuring self-regulated learning and the role of AI: Five years of research using multimodal multichannel data. Computers in Human Behavior, 139, 107540. https://doi.org/10.1016/j.chb.2022.107540
  • Nwankpa, C., Ijomah, W., Gachagan, A., & Marshall, S. (2018). Activation functions: Comparison of trends in practice and research for deep learning. Retrieved from https://arxiv.org/abs/1811.03378
  • OECD (2016). PISA 2015 Assessment and analytical framework: Science, reading, mathematic and financial literacy. OECD Publishing.
  • Pachman, M., Arguel, A., Lockyer, L., Kennedy, G., & Lodge, J. M. (2016). Eye tracking and early detection of confusion in digital learning environments: Proof of concept. Australasian Journal of Educational Technology, 32(6), 58–71. https://doi.org/10.14742/ajet.3060
  • Radach, R., Kennedy, A., & Rayner, K. (2004). Eye movements and information processing during reading. Psychology Press.
  • Ratwani, R. M., Trafton, J. G., & Boehm-Davis, D. A. (2008). Thinking graphically: Connecting vision and cognition during graph comprehension. Journal of Experimental Psychology. Applied, 14(1), 36–49. https://doi.org/10.1037/1076-898X.14.1.36
  • Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3), 372–422. https://doi.org/10.1037/0033-2909.124.3.372
  • Reeves, T. C. (2000). Alternative assessment approaches for online learning environments in higher education. Journal of Educational Computing Research, 23(1), 101–111. https://doi.org/10.2190/GYMQ-78FA-WMTX-J06C
  • Reeves, T. C. (2002). Keys to successful E-learning: Outcomes, assessment and evaluation. Educational Technology, 42(6), 23–29. Retrieved from http://www.jstor.org/stable/44428789
  • Riegler, A., Aksoy, B., Riener, A., Holzmann, C. (2020). Gaze-based interaction with windshield displays for automated driving: Impact of dwell time and feedback design on task performance and subjective workload. Paper presented at the 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Virtual Event, DC, USA. https://doi.org/10.1145/3409120.3410654
  • Rosé, C. P., McLaughlin, E. A., Liu, R., & Koedinger, K. R. (2019). Explanatory learner models: Why machine learning (alone) is not the answer. British Journal of Educational Technology, 50(6), 2943–2958. https://doi.org/10.1111/bjet.12858
  • Sak, H., Senior, A., & Beaufays, F. (2014). Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition. Retrieved from https://arxiv.org/abs/1402.1128
  • Sharaev, M., Sushchinskaya, S., Bachurina, V., Taranov, G., Burnaev, E., & Arsalidou, M. (2021). Machine learning, eye movements and mathematical problem solving. Journal of Vision, 21(9), 2397–2397. https://doi.org/10.1167/jov.21.9.2397
  • Sharma, K., & Giannakos, M. (2020). Multimodal data capabilities for learning: What can multimodal data tell us about learning? British Journal of Educational Technology, 51(5), 1450–1484. https://doi.org/10.1111/bjet.12993
  • Sharma, K., Giannakos, M., & Dillenbourg, P. (2020). Eye-tracking and artificial intelligence to enhance motivation and learning. Smart Learning Environments, 7(13). https://doi.org/10.1186/s40561-020-00122-x
  • Sharma, S., Sharma, S., & Athaiya, A. (2020). Activation fuctions in neural networks. International Journal of Engineering Applied Sciences and Technology, 04(12), 310–316. https://doi.org/10.33564/IJEAST.2020.v04i12.054
  • Sherstinsky, A. (2020). Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network. Physica D: Nonlinear Phenomena, 404, 132306. https://doi.org/10.1016/j.physd.2019.132306
  • Shute, V., & Underwood, J. (2006). Diagnostic assessment in mathematics problem solving. Technology Instruction Cognition and Learning, 3, 151–166.
  • Sqalli, M. T., Al-Thani, D., Elshazly, M. B., Al-Hijji, M. (2022). A blueprint for an ai & ar-based eye tracking system to train cardiology professionals better interpret electrocardiograms. Paper presented at the Persuasive Technology, Cham.
  • SR-Research (2020a). EyeLink data viewer user’s manual, Version 4.1.1. Retrieved from https://www.sr-research.com/data-viewer/
  • SR-Research (2020b). EyeLink User Manual v1.52. Retrieved from https://www.sr-research.com/eyelink-1000-plus/
  • Strohmaier, A. R., MacKay, K. J., Obersteiner, A., & Reiss, K. M. (2020). Eye-tracking methodology in mathematics education research: A systematic literature review. Educational Studies in Mathematics, 104(2), 147–200. https://doi.org/10.1007/s10649-020-09948-1
  • Sun, J., Liu, Y., Wu, H., Jing, P., & Ji, Y. (2022). A novel deep learning approach for diagnosing Alzheimer’s disease based on eye-tracking data. Frontiers in Human Neuroscience, 16, 972773. https://doi.org/10.3389/fnhum.2022.972773
  • Sundermeyer, M., Schlüter, R., Ney, H. (2012). LSTM neural networks for language modeling. Paper presented at the Thirteenth annual conference of the international speech communication association.
  • Susac, A., Bubic, A., Kaponja, J., Planinic, M., & Palmovic, M. (2014). Eye movements reveal students’ strategies in simple equation solving. International Journal of Science and Mathematics Education, 12(3), 555–577. https://doi.org/10.1007/s10763-014-9514-4
  • Susac, A., Bubic, A., Planinic, M., Movre, M., & Palmovic, M. (2019). Role of diagrams in problem solving: An evaluation of eye-tracking parameters as a measure of visual attention. Physical Review Physics Education Research, 15(1), 013101. https://doi.org/10.1103/PhysRevPhysEducRes.15.013101
  • Tsai, M.-J., & Wu, A.-H. (2021). Visual search patterns, information selection strategies, and information anxiety for online information problem solving. Computers & Education, 172, 104236. https://doi.org/10.1016/j.compedu.2021.104236
  • Tsai, M.-J., Hou, H.-T., Lai, M.-L., Liu, W.-Y., & Yang, F.-Y. (2012). Visual attention for solving multiple-choice science problem: An eye-tracking analysis. Computers & Education, 58(1), 375–385. https://doi.org/10.1016/j.compedu.2011.07.012
  • Unadkat, S., Ciocoiu, M., & Medsker, L. (2001). Recurrent neural networks design and applications. In L. Medsker & L. Jain (Eds.), The CRC press international series on computational intelligence (Vol. 5). CRC Press.
  • Uzzaman, S., & Joordens, S. (2011). The eyes know what you are thinking: Eye movements as an objective measure of mind wandering. Consciousness and Cognition, 20(4), 1882–1886. https://doi.org/10.1016/j.concog.2011.09.010
  • van Merriënboer, J. J. G. (2013). Perspectives on problem solving and instruction. Computers & Education, 64, 153–160. https://doi.org/10.1016/j.compedu.2012.11.025
  • Wang, P. (2019). On defining artificial intelligence. Journal of Artificial General Intelligence, 10(2), 1–37. https://doi.org/10.2478/jagi-2019-0002
  • Worsley, M. (2014). Multimodal learning analytics as a tool for bridging learning theory and complex learning behaviors. Paper presented at the Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge, Istanbul, Turkey. https://doi.org/10.1145/2666633.2666634
  • Wu, C.-J., Liu, C.-Y., Yang, C.-H., & Jian, Y.-C. (2021). Eye-movements reveal children’s deliberative thinking and predict performance on arithmetic word problems. European Journal of Psychology of Education, 36(1), 91–108. https://doi.org/10.1007/s10212-020-00461-w
  • Xue, J., Li, C., Quan, C., Lu, Y., Yue, J., & Zhang, C. (2017). Uncovering the cognitive processes underlying mental rotation: An eye-movement study. Scientific Reports, 7(1), 10076. https://doi.org/10.1038/s41598-017-10683-6
  • Yang, S. J. H., Ogata, H., Matsui, T., & Chen, N.-S. (2021). Human-centered artificial intelligence in education: Seeing the invisible through the visible. Computers and Education: Artificial Intelligence, 2, 100008. https://doi.org/10.1016/j.caeai.2021.100008
  • Yu, Y., Si, X., Hu, C., & Zhang, J. (2019). A review of Recurrent Neural Networks: LSTM cells and network architectures. Neural Computation, 31(7), 1235–1270. https://doi.org/10.1162/neco_a_01199
  • Zhan, P., Man, K., Wind, S. A., & Malone, J. (2022). Cognitive diagnosis modeling incorporating response times and fixation counts: Providing comprehensive feedback and accurate diagnosis. Journal of Educational and Behavioral Statistics, 47(6), 736–776. https://doi.org/10.3102/10769986221111085
  • Zhang, L., Song, N., Wu, G., & Cai, J. (2023). Understanding the cognitive processes of mathematical problem posing: Evidence from eye movements. Educational Studies in Mathematics. https://doi.org/10.1007/s10649-023-10262-9

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.