Plagiarism: A case study of quality improvement in a taught postgraduate programme

Abstract

Background: Plagiarism is a common issue in education. Software can detect plagiarism but little is known about prevention.

Aims: To identify ways to reduce the incidence of plagiarism in a postgraduate programme.

Methods: From 2006, all student assignments were monitored using plagiarism detection software (Turn It In) to produce percentage text matches for each assignment. In 2007, students were advised software was being used, and that plagiarism would result in penalties. In 2008, students attending a key module took part in an additional interactive seminar on plagiarism. A separate cohort of students did not attend the seminar, allowing comparison between attendees and non-attendees.

Results: Between 2006 and 2007, mean percentage text match values were consistent with a stable process, indicating advice and warnings were ineffective. Control chart analysis revealed that between 2007 and 2008, mean percentage text match changes showed a reduced text match in all nine modules, where students attended the interactive seminar, but none where students did not. This indicated that the interactive seminar had an effect. In 2008, there were no occurrences of plagiarism. Improvements were maintained in 2009.

Conclusions: Advice and warnings against plagiarism were ineffective but a subsequent interactive seminar was effective at reducing plagiarism.

Introduction

Plagiarism has always presented challenges in higher education establishments, but there is evidence that it is on the increase, possibly due to the ever-increasing availability of online material (Park 2003; Smith 2006). Estimates of the frequency of plagiarism vary widely, from less than 5% to over 80%, with variation according to academic discipline, level of study and nationality (Park 2003; Martin 2005; Evans 2006; Jocoy & DiBiase 2006). This article reports on our experience of applying a quality improvement methodology to the problem of plagiarism in the University of Birmingham, Master of Public Health (MPH) programme.

Various electronic tools are available for institutions to check assignments for evidence of plagiarism (Lukashenko et al. 2007). However, it is unclear how to make best use of this software. It can be used systematically to assess all submissions, or only for suspicious assignments. The software can be made available to students to allow them to self-assess (Barrett & Malcolm 2003). Barrett et al. (2003) recommend a systematic approach in order to deter students, to value the work of those who do not plagiarise, and to help those who need to develop their referencing and paraphrasing technique. A number of authors have also suggested that the software should be used in conjunction with educational interventions (Braumoeller & Gaines 2001; Barrett & Malcolm 2003; Park 2003; Martin 2005; Devlin 2006; Jocoy & DiBiase 2006).

Since 2002, all UK Higher Education institutions have been provided with a free plagiarism detection service by the Joint Information Strategy Committee (JISC). JISC's preferred software is Turn It In (iParadigms LLC, available online; Barrett et al. 2003). This software matches text to internet sources and to an archive consisting of all assignments previously submitted to Turn It In anywhere in the world.

The evidence surrounding plagiarism detection software and its application is limited. Martin found a general, though not consistently significant, decline in plagiarism rates over time for business graduate students using Turn It In, when combined with prior warnings, lowering of grades for offenders, and feedback to students (Martin 2005). Two authors observed increased plagiarism rates attributable to improvements in the Turn It In software over time (Hawarth 2006; Culwin, 2008). Jocoy and DiBiase (2006) recorded a decline in plagiarism when using software alongside ‘expectation management’, but this was not statistically significant. Another plagiarism detection system, EVE, has also been shown to reduce plagiarism rates compared to a control group when coupled with verbal and written warnings (Braumoeller & Gaines 2001).

Plagiarism detection software has limitations. Barrett et al. (2003) recommend Turn It In for detecting web plagiarism, but concluded that other resources may be more useful for detecting collusion between students, as ‘shared’ material may not be found online. Rates of plagiarism detected using Turn It In have fallen short of the self-reported rates from students (Jocoy & DiBiase 2006). Several authors reported that determining the level of ‘text match’ warranting further investigation can be challenging (Martin 2005; Evans 2006; Culwin 2008). Evans highlighted that the workload ‘cost’ of detecting lower level plagiarism by manually screening all software outputs may not be justified by the ‘benefit’ of detection of the few offending students (Evans 2006). Overall, it seems that plagiarism-detection software may be a useful tool, but it should be accompanied by other interventions and a supportive institutional culture with a prevention focus.

The University of Birmingham admits approximately 30 students a year to its MPH programme. Students are adults with a number of years of professional experience and come from a wide variety of clinical and non-clinical backgrounds and nationalities. Assessment is by a mixture of written assignments and examinations, with most modules assessed by assignment. Prior to 2006, detection of plagiarism in student assignments relied on the suspicion of academic staff, but from 2006, all assignments were screened using Turn It In software.

As is the experience of other users (outlined above), use of the software increased the detection of plagiarism among students on the MPH course. However, the software also demonstrated that almost all assignments showed a degree of text match to other sources. We therefore, needed a method to determine when the degree of text match was sufficient to warrant further investigation. More fundamentally, the software does not prevent plagiarism, but detects it after it has taken place. Improved detection may deter plagiarism but it was unclear what, if any, additional interventions are needed to reduce the frequency of plagiarism.

We might regard plagiarism as a problem of quality improvement. Assignments should be students’ own work and reflect their own abilities; therefore, the aim is to produce assignments with the minimum degree of text from other sources. If students use some text from other sources it may be legitimate if it is appropriately cited, may reflect poor practice if rephrasing the work of cited authors, or it may be a deliberate attempt to pass others’ work off as their own. Degree of text match therefore indicates degree of deviation from the quality optimum and occurrences of plagiarism as extreme deviation from the optimum.

The science of quality improvement begins with Walter Shewhart, a physicist with Bell Laboratories in the 1920s. Shewhart (1931) realised that the problem of quality is fundamentally a problem of variation and that quality improvement must be guided by an understanding of variation. Shewhart categorised variation according to the most efficient action needed in order to effect improvement. Later, authors referred to these as common cause and special cause variations.

Common cause variation is intrinsic to the underlying process. Many factors contribute to the variation and it is unlikely that we will identify a single factor as having caused the observed variation. Because this variation is produced by the underlying process, if unchanged the process will continue to produce this degree of variation. In an assignment-writing process, all assignments will show a degree of text match. The degree of text match will vary, but the variation will be consistent with variation produced by a common underlying process. Although one student's assignment may show a greater degree of text match than another, it is not possible to attribute this difference to a specific cause. The variation is a product of complex interactions between many factors influencing the way the student writes the assignment. In other words, the variation has no assignable cause, because it is produced by the underlying process. If the process is not changed it will continue to produce the same results with the same degree of variation. In order to reduce this kind of variation, it is necessary to fundamentally change the underlying process.

Special cause variation is not consistent with a common underlying process. It is caused by one or more additional factors affecting the underlying process. If we investigate this kind of variation, we are likely to identify an assignable cause. In quality improvement, when we see this kind of variation we need to investigate, identify the assignable cause and take appropriate action. Special cause variation provides us with an opportunity to identify which factors exert a particularly strong influence on the process. This is important for the observed instance of special cause variation, but it also gives us information about which factors may be the most important influences on variation in our underlying process. In other words, investigating special causes gives us clues about how to improve common cause variation.

Shewhart devised a graphical method, the control chart, to distinguish between common cause and special cause variation. A control chart has three lines, one is the mean and two other lines represent the upper and lower limits of common cause variation. Data points between the lines indicate common cause variation and data points outside of these lines indicate special cause variation (Wheeler 1995). Control chart analysis of quality indicators can therefore guide the user towards the most appropriate action to take in order to effect improvement. The use of control charts to inform quality improvement has been championed by Deming (1986, 1994). They have been applied in education and there are successful examples of the approach in industry (Cornesky-Robert & McCool-Samuel 1994; Wood et al. 2001). They have also been applied in health care and public health (Benneyan, 1998; Williamson & Weatherby Hudson 1999; Mohammed et al. 2001).

We applied a quality improvement methodology to addressing the problem of plagiarism. This meant continuously monitoring the percentage text match of all assignments both to identify potential plagiarism and to assess the extent and variability of text match in assignments produced by our underlying process.

Table

Practice points

Informing students about plagiarism and the use of plagiarism-detecting software is not sufficient to change students’ writing practice. An interactive seminar, involving examples and reflection, with a clear explanation of the consequences, has been shown to produce improvement in both the overall plagiarism scores and incidence of plagiarism. Using control charts of plagiarism scores provides an easy way to interpret individual results and enables comparison over time.

Methods

From academic year 2006, the MPH programme began to screen module assignments using plagiarism detection software (Turn It In). The software allocated each assignment a percentage indicating the percentage of text that matched either to internet sources or to assignments previously submitted to Turn It In. In addition, the software produced a detailed report for each individual assignment, identifying specific areas of text that match to other sources and indicating the sources. Following Turn It In assessment, assignments were grouped by title. Variation in text match for all assignments with the same title was analysed using statistical process control charts (moving range charts – XMR). In keeping with standard practice in the use of statistical process control charts, individual assignments with a percentage text match more than three standard deviations away from the mean were identified as showing special cause variation. In these cases, the detailed Turn It In report for the individual assignment was examined to determine whether plagiarism had taken place. Following standard university policy, if two members of academic staff agreed that plagiarism may have occurred, the student was invited for interview and if appropriate, a penalty mark (usually zero) was imposed for the assignment. Data were therefore available on two outcomes; both the number of individual student assignments where plagiarism was identified and the average text match for each assignment title.

During the 2006 academic year (year starting October 2006), no specific action was taken to reduce the occurrence of plagiarism apart from general information provided in course documentation.

At the start of the 2007 academic year, the introductory lecture included an additional explanation of the need to avoid plagiarism and students were specifically informed that the software was being used. A lecture later in the same academic year explained in some detail how the percentage text match was used to identify possible plagiarism. Furthermore, during the course of the year, all students on the MPH were informed (without identifying individuals) when students had been penalised for plagiarism.

At the start of the 2008 academic year, a new 40-min interactive plagiarism seminar was delivered during teaching time allocated to one of the compulsory MPH modules. This was devised after interviews with some of the students who had been penalised for plagiarism revealed that they remained unclear about what was meant by plagiarism and uncertain how to avoid it. In the seminar, following a brief talk about plagiarism and Turn It In software, students were provided with an anonymised copy of a detailed Turn It In report from a previous student who had been penalised for plagiarism. In small groups, they were asked to decide whether plagiarism had taken place; what (if any) action should be taken; what action should be taken if a similar Turn It In report was subsequently received from the same student; how a student could avoid plagiarism. Following feedback and discussion, it was explained that this was plagiarism; the student had been awarded a mark of zero and required to resubmit the assignment; that repeated plagiarism was automatically regarded as more serious; and there was a brief discussion on how to avoid plagiarism and where to obtain support on assignment writing.

At the start of the 2009 academic year, the plagiarism seminar was run twice, once for MPH students and once for MSc Health Technology Assessment (HTA) students.

The effect of this was that no students attended the plagiarism seminar before submitting assignments in 2006 or 2007; all full time MPH students attended the plagiarism seminar before submitting assignments in 2008; no part time MPH or HTA students had attended the plagiarism seminar before submitting assignments in 2008; all students had attended plagiarism seminar before submitting assignments in 2009.

Year on year plagiarism scores were analysed using statistical process control charts to look for evidence of a change in the process. For each module assignment, mean plagiarism scores before and after introduction of the plagiarism seminar were compared using t-tests with Bonferroni correction for multiple comparisons. The number of occurrences of plagiarism was also determined before and after introduction of the plagiarism seminar.

Results

Description of dataset

Percentage text match scores were available for 14 module assignments. There were 4 years of data for six assignments: Critical Appraisal (CA), Health Economics (HE), Health Information (HI), Health Promotion (HP), Management (M), Public Health in Developing Countries (PHDC) and the project (Project). There were 3 years of data for six assignments: Advanced Statistics (AS), Environment and Health (EH), Methodological Basis of Health Technology Assessment (MBHTA), Public Health Microbiology (Micro), Principles and Practice of HTA (P&P), Sociology and Social Policy (Soc). There were 2 years of data for one assignment, Qualitative Research Methods (QRM).

Percentage text match

There were nine module assignments, where students attended the plagiarism workshop at the start of 2008. Seven showed increases in percentage text match from 2006 to 2007, one showed a small decrease and one had no data for 2006 (Table 1). All nine showed decreases from 2007 to 2008, these decreases were sustained in eight modules in 2009 (one module was discontinued in 2009).

Table 1.  Average percentage text match in assignments from years 2006 to 2009

		Average percentage text match and (N) for each assignment title
		2006		2007		2008		2009
Start of academic year in which students enrolled	Assignment title	Match (%)	N	Match (%)	N	Match (%)	N	Match (%)	N
2008	CA	28.5	39	37.4	47	20.8	35	18.1	47
	EH	24.9	18	25.4	22	10.3	12
	HE	17.2	12	22.7	16	8.1	14	10.3	18
	HI	30.3	41	28.0	45	23.6	35	20.8	53
	HP	23.2	21	23.8	29	14.4	24	10.4	42
	M	10.1	17	13.3	23	7.8	22	2.9	66
	PHDC	26.7	27	41.4	17	12.4	7	11.6	14
	Project	18.6	16	19.4	39	14.6	27	16.1	44
	Soc			29.1	9	10.2	10	12.4	8
2009	AS			14.1	12	18.4	10	18.6	28
	MBHTA			33.1	8	29.7	6	17.8	10
	Micro			25.3	4	16.7	3	7.7	11
	P&P			21.6	8	19.5	18	11.7	16
	QRM					23.8	11	12.4	24

Among the five modules assignments, where students attended the plagiarism workshop at the start of 2009, one had no data for 2007. One showed an increase in text match from 2007 to 2008 and three showed a decrease. From 2008 to 2009, four showed a decrease in text match and one an increase of 0.2% (Table 1).

From 2007 to 2008, control charts show a clear reduction in the percentage text match for most of the module assignments where students had been exposed to the plagiarism seminar. There were reductions in both the mean and the variability. This is illustrated for the HI, CA and P&P assignments (Figures 1–3).

Figure 1. Control chart of percentage text match in CA assignments from 2006 to 2009 (UCL, upper confidence limit (3 SD) and LCL, lower confidence limit (3 SD)).

Figure 2. Control chart of percentage text match in HI assignments from 2006 to 2009 (UCL, upper confidence limit (3 SD) and LCL, lower confidence limit (3SD)).

Figure 3. Control chart of percentage text match in P&P assignments from 2007 to 2009 (UCL, upper confidence limit (3 SD) and LCL, lower confidence limit (3SD)).

By contrast, control charts for module assignments where students had not been exposed to the plagiarism seminar showed common cause variation from 2007 to 2008. In other words, the variation in percentage text match in 2007 and 2008 is consistent with no change in the underlying process. This is illustrated for the P&P module assignment. In Figure 3, the control limits have been calculated from 2007 data and text match scores in 2008 remain consistent with the same process.

Occurrence of plagiarism

In the academic year starting October 2006, 191 assignments were checked and four showed special cause variation. Following further investigation, one case was found to be due to incorrect use of the Turn It In software and the remaining three assignments showed evidence of plagiarism. In 2007, 279 assignments were checked, four were plagiarised. One student plagiarised two further assignments and was required to leave the MPH programme. In 2008, 234 assignments were checked but no occurrences of plagiarism found. In 2009, 381 assignments were checked, revealing two occurrences of plagiarism.

Comparing nine occurrences of plagiarism in 470 assignments (1.9%) from the period before the interactive plagiarism seminar (2006 and 2007) to two in 615 assignments (0.3%) after (2008 and 2009) the difference was statistically significant (Fisher's exact test two-tailed p = 0.013). The result remained statistically significant when the two repeated occurrences by the same student in 2007 were excluded as non-independent events (Fisher's two-tailed p = 0.044).

Discussion

Continuous monitoring of text match scores provided valuable information on the occurrence of plagiarism and on the effectiveness of measures to reduce plagiarism. We found that neither improved detection with advice to avoid plagiarism nor penalties for offenders fundamentally changed the proportion of text match observed in assignments. However, a specific seminar has reduced both text match scores and occurrences of plagiarism. The reduction was maintained in the following year. It seems unlikely that this can be attributed to systematic differences between MPH and HTA students since similar changes in text match were seen in HTA students when they attended the seminar.

We found control charts were an easy way to interpret variation in text match scores. They reduced the number of individual reports that needed to be examined. In every case where special cause variation was observed a cause was identified. In most cases this was plagiarism but in one case it was an error in the way that Turn It In had been used: the assignment matched to an earlier version of the same assignment which had been submitted in the student's workplace (another university). The control charts also made it easy to identify that the plagiarism seminar had fundamentally changed the student process of assignment writing and hence effecting a quality improvement.

There are a number of important observations about the application of statistical process control charts to analyse text match scores. First, there is no fixed threshold text match percentage above which plagiarism should be suspected. Because the degree of text match varies from one type of module assignment to another the threshold must be set in relation to data collected for each specific module assignment. Second, to determine the mean and the limits of common cause variation for a specific module assignment, we must monitor text match scores for all students’ written submissions. We cannot establish the process norms by assessing only suspicious pieces of work. Third, cases of plagiarism may not be sufficiently frequent to distinguish reliably year-to-year changes due to chance from changes in the process of assignment writing. Monitoring the percentage text match is therefore a more sensitive indicator of process change than monitoring numbers of cases of plagiarism. This means that the aim of quality improvement is to reduce text match scores across all submitted work, not simply to focus on the few that are outliers. Improvement is a reduction in both the mean and the degree of variation, which can be clearly seen in Figure 2.

We describe the successful application of a method of quality improvement. Control charts are simply an analytic tool. They tell us about the nature of variation, when to investigate further and they help indicate when improvement has occurred. They do not tell us how to effect improvement. It is instructive that our first attempts to reduce plagiarism had no impact but our plagiarism seminar was effective. Continuous monitoring and analysis was able to distinguish between effective and ineffective interventions. Our seminar followed a format familiar to our students and to our staff who teach CA of research papers. Other formats may be better suited to other students or other institutions.

Declaration of interest: All the authors have no interests to declare.