Can software improve marker accuracy at detecting contract cheating? A pilot study of the Turnitin authorship investigate alpha

Abstract

Contract cheating happens when students outsource their assessed work to a third party. One approach that has been suggested for improving contract cheating detection is comparing students’ assignment submissions with their previous work, the rationale being that changes in style may indicate a piece of work has been written by somebody else. This approach is time consuming, but recent advances in machine learning and natural language processing suggest that it may be well suited to computerization. We trialed an early alpha version of Turnitin’s Authorship Investigate tool, which compares students’ submissions against their previous work. Twenty-four experienced markers from five units of study were asked to make decisions about the presence of contract cheating in bundles of 20 student assignments, which included 14 legitimate assignments and six purchased from contract cheating sites. We asked markers to determine if each assignment was contract cheating, then provided them with an Authorship Investigate report and let them change their decision. Marker accuracy at detecting contract cheating increased significantly, from 48% to 59% after using the report, with no significant difference in false positives. These findings suggest that software may be an effective component of institutional strategies to address contract cheating.

Keywords:

• Cheating
• contract cheating
• marking
• educational technology
• academic integrity

Acknowledgements

We thank Kevin Dullaghan for his research assistance on this project.

Disclosure statement

Mark Ricksen is employed by Turnitin, and he has been involved in the development of the Authorship Investigate tool.

Funding

This study was funded by Turnitin, who are the developers of the Authorship Investigate tool. One of the authors of this paper is employed by Turnitin. In order to combat reporting bias (Dawson and Dawson 2018), prior to agreeing to this study or any funding, both Turnitin and the academic authors of the paper agreed that the results of the study would be published regardless of whether they showed a positive story. In addition, Turnitin’s funding only supported the direct costs of the research and did not pay for the time of the academic authors. This study received $AU38,000 in financial support from Turnitin, which was used for participant incentives, paying markers, purchasing contract cheating assignments, and research assistance.