The comparison of general tips for mathematical problem solving generated by generative AI with those generated by human teachers

ABSTRACT

In designing an intelligent tutoring system, a core area of the application of AI in education, tips from the system or virtual tutors are crucial in helping students solve difficult questions in disciplines like mathematics. Traditionally, the manual design of general tips by teachers is time-consuming and error-prone. Generative AI, like ChatGPT, presents a new channel for designing general tips. This study utilized prompt engineering and Chain of Thought to summarize general tips for given mathematical problems (one geometry problem and one algebra problem) and their solutions. A Turing test was conducted to compare ChatGPT-generated general tips with human-designed ones. Results from 121 human evaluators, each assessing 6 ChatGPT-generated and 6 human-designed general tips for each of two mathematical problems, showed that the average score for ChatGPT-generated tips is less than that of human-designed tips at a statistically significant level (p < 0.05), and Zero-Shot CoT achieved the best score. However, no evaluator could distinguish the tip types exactly. The average precision, recall and F-value of all ChatGPT-generated tips are less than 40%. AI-generated general tips can serve as a valuable reference for teachers to enhance efficiency and students’ mathematical learning.

KEYWORDS:

• General tips
• intelligent tutoring system
• mathematical education
• large language models
• prompt engineering
• turing test

Acknowledgements

This research is supported by the National Education Research Funding Project “Students’ Intelligent Assessment and Tutoring Research Based on Big-data Mining” (Number: BCA220208) granted by National Social Science Foundation, China. The authors extend heartfelt appreciation to all the teachers and students who have participated in this research, as well as gratitude to the reviewers and editors for their invaluable insights and suggestions.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The work was supported by the National Social Science Fund of China [BCA220208].

Notes on contributors

Jiyou Jia

Jiyou Jia, male, professor and department head of Department of Educational Technology, Graduate School of Education, Peking University, director of International Research Centre for Educational Informatization at Peking University. He is mainly engaged in the research of educational technology and the application of artificial intelligence in education.

Tianrui Wang

Tianrui Wang, female, postgraduate student of History of Science and Technology, University of Chinese Academy of Sciences. She is mainly engaged in the research of the history of artificial intelligence and educational technology.

Yuyue Zhang

Yuyue Zhang, female, postgraduate student in Department of Educational Technology, Graduate School of Education, Peking University. She is mainly engaged in the research of educational technology and the application of artificial intelligence in education.

Guangdi Wang

Guangdi Wang, female, postgraduate student in Chinese Institute, Beijing International Studies University. She is mainly engaged in the research of Chinese information processing.