91
Views
0
CrossRef citations to date
0
Altmetric
Research Article

When Young Scholars Cooperate with LLMs in Academic Tasks: The Influence of Individual Differences and Task Complexities

ORCID Icon, , , & ORCID Icon
Received 29 Nov 2023, Accepted 02 May 2024, Published online: 20 May 2024
 

Abstract

As a novel AI-powered conversational system, large language models (LLMs) have the potential to be used in various applications. Recent advances in LLMs like ChatGPT have made LLM-based academic tools possible. However, most of the existing studies on the adoption of LLM for academic tasks were based on theoretical or qualitative analyses, which failed to provide empirical evidence on the effects of LLMs on users’ behaviors. Additionally, although previous work has investigated users’ acceptance of conventional conversational systems, little is known about how scholars evaluate LLMs when they are used for academic tasks. Hence, we conducted an empirical field experiment to assess the performance of 48 early-stage scholars on two core academic activities (paper reading and literature reviews) under varying time constraints. Prior to the tasks, participants underwent different training programs about LLM capabilities and limitations. Then, we built a hierarchy dependency network using the Bayesian network. Statistical regression analyses were further conducted to quantify relationships among influential factors of task performance and users’ attitudes toward the LLMs. It was found that young scholars have upheld relatively high academic integrity when using LLMs for academic tasks, and user-LLM performance varied with the task type and time pressure but not with the type of training we used. Further, scholars’ traits can also affect their performance in academic tasks and attitudes towards the LLMs. This work can inspire the future development of LLM-related user training and guide the optimization of LLMs.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

Additional information

Funding

This work was supported by Guangzhou Municipal Science and Technology Project (No. 2023A03J0011) and Guangzhou Science and Technology Program City-University Joint Funding Project (No. 2023A03J0001).

Notes on contributors

Jiyao Wang

Jiyao Wang received B.Eng. degree from the Sichuan University in 2021, and M.Sc. degree from the Hong Kong University of Science and Technology in 2022. He is now a Ph.D. student in the Thrust of Robotics and Autonomous Systems at the Hong Kong University of Science and Technology (Guangzhou).

Chunxi Huang

Chunxi Huang received his master’s degree in industrial and systems engineering from Korea Advanced Institute of Science and Technology, Daejeon, South Korea, in 2020. He is currently a PhD candidate at the Hong Kong University of Science and Technology. His research interests include human factors, driver behavior, and traffic safety.

Song Yan

Song Yan received his master’s degree in mechanical engineering from the University of Tokyo, Japan. He is currently a PhD student in Intelligent Transportation Thrust at the Hong Kong University of Science and Technology (Guangzhou). His research interests include automated driving systems, human factors, and driver behavior.

Weiyin Xie

Weiyin Xie received B.Sc. degree and M.Sc. degree from the University of Duisburg-Essen. He is now a Ph.D. student in the Thrust of Robotics and Autonomous Systems at the Hong Kong University of Science and Technology (Guangzhou).

Dengbo He

Dengbo He received M.S. degree from the Shanghai Jiao Tong University in 2016 and PhD degree from the University of Toronto in 2020. He is currently an assistant professor from the Thrust of Intelligent Transpiration, the Hong Kong University of Science and Technology (Guangzhou).

Log in via your institution

Log in to Taylor & Francis Online

PDF download + Online access

  • 48 hours access to article PDF & online version
  • Article PDF can be downloaded
  • Article PDF can be printed
USD 61.00 Add to cart

Issue Purchase

  • 30 days online access to complete issue
  • Article PDFs can be downloaded
  • Article PDFs can be printed
USD 306.00 Add to cart

* Local tax will be added as applicable

Related Research

People also read lists articles that other readers of this article have read.

Recommended articles lists articles that we recommend and is powered by our AI driven recommendation engine.

Cited by lists all citing articles based on Crossref citations.
Articles with the Crossref icon will open in a new tab.