When Young Scholars Cooperate with LLMs in Academic Tasks: The Influence of Individual Differences and Task Complexities

Abstract

As a novel AI-powered conversational system, large language models (LLMs) have the potential to be used in various applications. Recent advances in LLMs like ChatGPT have made LLM-based academic tools possible. However, most of the existing studies on the adoption of LLM for academic tasks were based on theoretical or qualitative analyses, which failed to provide empirical evidence on the effects of LLMs on users’ behaviors. Additionally, although previous work has investigated users’ acceptance of conventional conversational systems, little is known about how scholars evaluate LLMs when they are used for academic tasks. Hence, we conducted an empirical field experiment to assess the performance of 48 early-stage scholars on two core academic activities (paper reading and literature reviews) under varying time constraints. Prior to the tasks, participants underwent different training programs about LLM capabilities and limitations. Then, we built a hierarchy dependency network using the Bayesian network. Statistical regression analyses were further conducted to quantify relationships among influential factors of task performance and users’ attitudes toward the LLMs. It was found that young scholars have upheld relatively high academic integrity when using LLMs for academic tasks, and user-LLM performance varied with the task type and time pressure but not with the type of training we used. Further, scholars’ traits can also affect their performance in academic tasks and attitudes towards the LLMs. This work can inspire the future development of LLM-related user training and guide the optimization of LLMs.

Keywords:

• Large language model
• academic tasks
• Human-AI collaboration
• user attitudes
• empirical study

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 https://openai.com/chatgpt.

2 https://www.claude.co.id/.

3 https://www.google.cn/chrome/index.html.

4 https://openai.com/gpt-4.

Additional information

Funding

This work was supported by Guangzhou Municipal Science and Technology Project (No. 2023A03J0011) and Guangzhou Science and Technology Program City-University Joint Funding Project (No. 2023A03J0001).

Notes on contributors

Jiyao Wang

Jiyao Wang received B.Eng. degree from the Sichuan University in 2021, and M.Sc. degree from the Hong Kong University of Science and Technology in 2022. He is now a Ph.D. student in the Thrust of Robotics and Autonomous Systems at the Hong Kong University of Science and Technology (Guangzhou).

Chunxi Huang

Chunxi Huang received his master’s degree in industrial and systems engineering from Korea Advanced Institute of Science and Technology, Daejeon, South Korea, in 2020. He is currently a PhD candidate at the Hong Kong University of Science and Technology. His research interests include human factors, driver behavior, and traffic safety.

Song Yan

Song Yan received his master’s degree in mechanical engineering from the University of Tokyo, Japan. He is currently a PhD student in Intelligent Transportation Thrust at the Hong Kong University of Science and Technology (Guangzhou). His research interests include automated driving systems, human factors, and driver behavior.

Weiyin Xie

Weiyin Xie received B.Sc. degree and M.Sc. degree from the University of Duisburg-Essen. He is now a Ph.D. student in the Thrust of Robotics and Autonomous Systems at the Hong Kong University of Science and Technology (Guangzhou).

Dengbo He

Dengbo He received M.S. degree from the Shanghai Jiao Tong University in 2016 and PhD degree from the University of Toronto in 2020. He is currently an assistant professor from the Thrust of Intelligent Transpiration, the Hong Kong University of Science and Technology (Guangzhou).