Assessment criteria and standards of the Geography Dissertation in the UK

Abstract

Undergraduate dissertations may provide up to one third of the weighting of the final year of a degree in Geography and are seen as important indicators of graduates’ independent research ability by prospective employers. This project examines the standards, criteria and assessment of the Geography dissertation in the UK. All Geography Departments in the UK were invited to complete a questionnaire survey that explored a range of aspects related to dissertations including format, assessment criteria and marking procedures. Responses were received from 24 Departments. The findings suggest that there is broad consensus in many areas including product format, study period, assessment criteria, and rigour and transparency in marking procedures. However, in other areas, including credit weighting, procedures followed in the event of a disagreement over marks, and interpretation of assessment criteria to students, there is wide variation in practice. Some suggestions are made to enhance equivalence and consistency in dissertation work.

Introduction

’Dissertations have had a long history in geographical higher education, being widely regarded as the pinnacle of an individual’s undergraduate studies and the prime source of autonomous learning’ (Gold et al., 1991). Our previous investigations into Geography dissertations (Harrison and Whalley, 2006; Harrison and Whalley, 2008) suggest that this is still the case.

Dissertations typically constitute up to one third of the overall weighting for the final year which is Level 6 of the English, Welsh and Northern Irish system, (QAA 2008), and Level 10 of the Scottish system (Scottish Credit and Qualifications Framework 2009) of undergraduate degrees. Dissertation performance may also be used to adjudicate degree awards in borderline cases. They present an opportunity for students to demonstrate their ability to work independently and autonomously. Grades awarded for dissertations are therefore of increasing interest to employers.

The role of dissertations in Geography undergraduate degrees can be set against a background of increasing concern throughout Higher Education about marking reliability, the maintenance of standards, ‘grade inflation’ and accountability. That said, there is increasing evidence of a new assessment culture emerging (Rust, 2007) that includes widespread use of assessment criteria (e.g. Harrison and Whalley, 2008), grade descriptors and formative assessment as well as greater consideration of feedback timing and mechanisms. Rust (2007) argues that there are still poor practices that go unchallenged and a number of studies suggest that there are considerable inconsistencies in marking, weighting and standards (Hand and Clewes, 2000, Pepper et al., 2001), and confusion over terminology (Sadler, 2005). Ambiguities concerning the use, meaning and application of assessment criteria are also evident (Webster et al., 2000). Others (Penny and Grover, 1996, Rust et al., 2003, Woolf, 2004) have commented on the interaction of students with assessment criteria and marking schemes for dissertations, and in particular, their poor conceptual understanding of expectations (Gibbs and Simpson, 2004) and poor matching of assessment grades with tutors (Penny and Grover, 1996). This does not accord with the QAA principle that “students and markers are aware of and understand the assessment criteria and/or schemes that will be used to mark each assessment task” (QAA 2006, p16–17).

This paper reports on a project to review assessment schemes and procedures for undergraduate Geography dissertations in the UK. The project sought to address the following key questions:

1. What assessment criteria are used and how are they established and approved?

2. How are students assisted to interpret assessment criteria? What is the role of supervision in this?

3. What grade descriptors and marking schemes are used?

4. What are the procedures for double marking, anonymous marking and blind marking?

5. What happens in the event of a disagreement between first and second markers?

The project collated baseline information about dissertations, including credit rating, length, format, time available, preparatory work, supervisory arrangements and feedback, and identified good and innovative practice. This paper addresses selected outcomes from the research. A more detailed report will be provided at a later date.

Methods

A questionnaire survey was sent to all Geography Departments in the UK resulting in 24 responses (including one from environmental science). Twenty-two Higher Education Institutions (HEIs) were represented with separate responses from different Faculties / Schools at two HEIs. Responses were evenly split in terms of pre-1992 and post-1992 HEIs. The questionnaire contained a mixture of closed and open questions and invited respondents to submit a range of documents including assessment criteria, marking schemes and grade descriptors. Many respondents additionally submitted copies of Dissertation Handbooks, written procedures in the event of a disagreement over marks, lecture slides and other materials demonstrating good practice.

The nature of the dissertation

At the surveyed HEIs, 79% of students (92% of single honours students) have to prepare a final year dissertation. It is sometimes, but not always, an option for combined honours, joint honours and major-minor students. Alternatives to dissertations for these students include taught modules and work-based options.

Credit-rating and word length

The credit weighting of the dissertation varies from 15 to 40 Credit Accumulation Transfer Scheme (CATS) credits (Figure 1) with the modal weighting at 30 credits (25% of the final year). One HEI permits students to opt for the dissertation as either a single (15 CATS) or double (30 CATS) module.

Figure 1 CATS credit weighting of geography dissertations.

The majority of dissertations (79%) are required to be 10–12,000 words in length (Figure 2). One HEI requires a 4000-word piece of final work, but this excludes a literature review which is submitted separately at an earlier stage.

Figure 2 Median word length of geography dissertations.

The relationship between credit weighting and word length is interesting: overall, there is a positive correlation, but this masks significant variation. The majority of 10,000-word dissertations are awarded 30 or 40 credits but one institution awards just 20 credits for the same. Another institution that provided two questionnaire responses awards 20 credits for a 10,000-word dissertation in one Faculty and 40 credits for the same word length in another Faculty.

Study period and product format

The overwhelming majority of students have at least 10 months to prepare the dissertation (Figure 3), although this may partly reflect the fact that 55% are introduced to preparatory work in their penultimate year.

Figure 3 Dissertation study period.

Most dissertations are submitted as hard copy, but one third also require e-submission for archiving and plagiarism detection. There is some flexibility about the format of submission with some allowing alternative formats or attachments (e.g. audio, visual, field notebook).

Assessed elements

The weighting of the final dissertation product varies from 70 to 100% (Figure 4) and elements making up the remaining portion of the module include interim progress reports, oral presentations, seminars, posters and literature reviews. In addition, several programmes include formative assessed elements (particularly seminars).

Figure 4 The percentage weighting of the final report for the dissertation module.

The nature of assessment

Assessment criteria

A wide range of assessment criteria are in use at the HEIs surveyed (Box 1). In some cases these are presented explicitly as assessment criteria and in others they are embedded into grade descriptors. The criteria submitted address the fundamental requirements for the dissertation, presentation, administrative considerations (such as ethics), evidence of student independence and what we term the ‘X factor’. The latter are the characteristics of a very high quality dissertation that are not easy to define in black and white terms — flair, innovation, creativity and criticality.

Box 1 The range of assessment criteria encountered compiled from all contributing institutions.

Fundamentals of the dissertation

• Evidence of originality and perceptiveness

• Clarity of aims, topic identification

• Evidence of reading, awareness of literature

• Quality of research design and methodology

• Quality of data

• Presentation, analysis, evaluation, synthesis and interpretation of data

• Conceptual awareness, theoretical understanding

• Sustained argument

• Findings and conclusions justified and contextualised in the literature

Presentation

• Standard of presentation, use of English language, structure

• Use of complex academic terminology

• Correct use of referencing conventions

• Coherent integration of illustrative materials

Administrative

• Conduct including engagement with administrative processes

• Assessment of risks and ethical considerations

• Compliance with requirements

Independence

• Ability to work independently

• Exercise of personal initiative and responsibility

• Conduct and competence during practical work

• Cognitive, intellectual, practical and personal skills

• Appropriate and correct use of ICT applications

• Reflective, critically evaluating own performance and personal development

The ‘X Factor’

• Critical ability

• Creative thinking

• Flair, innovation

We were interested to explore how assessment criteria were developed and established. In most cases this occurred through team discussions, working parties and formal approval: at programme review and through consultation with external examiners. Some respondents said that criteria had evolved over time or through shared experiences and tradition. In some cases criteria were generic to the University, though one HEI identified the difficulties inherent in establishing common criteria even across a single department.

Explaining assessment criteria to students

Responses to the survey are encouraging in that the majority of HEIs do something to explain and interpret assessment criteria to students: typically through the provision of detailed text-based or online materials (e.g. a dissertation guide) that explicitly includes explanation of dissertation assessment criteria and expectations, and through the conduct of face-to-face tutorials and classes. These latter include formal, structured supervisory sessions where criteria are discussed, and whole cohort lectures. In some cases, it is assumed that criteria are discussed through the supervisory process, but there is no specific guidance or requirement to do so.

Grade descriptors and marking schemes

Almost all of the assessment criteria presented in the survey responses were couched in the context of standards or grade descriptors. However, in a number of cases it was difficult to extract the assessment criteria from the grade descriptors and the criteria appeared to vary for different grades. It is not clear whether this reflects the confusion around assessment terminology to which Sadler refers (2005) (e.g. criteria, standards, marking schemes) or whether assessment criteria are designed to permit marking flexibility. Although not explicitly stated as such, there appears to be implicit reliance on what Rust et al. (2003) refer to as the ‘connoisseur approach’ - knowledge transfer as a product of the student-tutor relationship.

The degree of numerical breakdown of each grade and the level of detail provided for each varies considerably. In some cases, broad descriptors were developed at institutional level and more detailed interpretations drawn up by departments.

There was very little evidence that any marking schemes were in use to award marks for different weighted sections or attributes of the dissertation. One HEI breaks the marks awarded into four components (40% for content, 40% for argument, 10% for structure / approach and 10% for style) and another has a three-way division (academic context; methodology, data collection and analysis; and interpretations, conclusions and presentation).

Marking procedures

At every HEI surveyed, dissertations are marked by two people. Only one HEI did not involve the supervisor in the marking process (Figure 5). In another case the supervisor was the second marker. In 15 cases (62%), the second marking is completed ‘blind’ (i.e. markers are unaware of grades awarded by the other marker). In seven cases, anonymous marking is undertaken, although comments suggest this is only partially successful because supervisors recognise their students’ work.

Figure 5 The different permutations for marking procedures.

Dealing with disagreements over marks

The first thing to note about dealing with disagreements over marks is that there is considerable variation over what constitutes a disagreement, with anything from 5% to 12% or a difference of class. In most departments, markers are encouraged to engage in discussions before introducing a third marker — and many disagreements are resolved at this stage.

Where the numerical difference is small, the mean mark is often used. Where a disagreement cannot be resolved between first and second markers, it is very common practice to introduce a third marker - often determined by the module leader or Head of Department. Subsequent procedures for resolution include discussions between the three markers, discussing all three marks at internal examination boards, using the median mark, and inviting an external examiner to adjudicate.

Exploring the role of external examiners a little further, we found split opinions on the matter. Some HEIs state, with some conviction, that external examiners are not asked to adjudicate - only to moderate. Others are comfortable in inviting external examiners to adjudicate where necessary. Four HEIs said that externals are commonly invited to comment on dissertations in borderline cases or where there is a particular problem such as suspected plagiarism or a failed degree where passes were achieved elsewhere.

Good practice

The survey respondents themselves identified a number of areas deemed to be good practice. There were two recurring themes:

1. Explanation and interpretation of assessment criteria for students. A variety of ways to achieve this were identified. These included detailed explanation of criteria and expectations in a dissertation guide; learning activities during the penultimate undergraduate year including peer assessment of past dissertations and discussions around criteria; final year whole cohort lectures including detailed discussion of criteria.

2. Marking procedures. Good practice involves following formalised procedures for double marking and blind marking and having rigorous and transparent procedures in the event of a disagreement. There is also good practice in the use of standard mark sheets that require a brief explanation of the mark awarded by each marker and the process of agreeing a final mark where there was disagreement.

Conclusions and recommendations

It is clear from the survey that, although the requirements for dissertations vary, there is broad consensus across the sector in terms of product format, the elements assessed, report weighting, study period, word length and credit-rating. It is a concern that within this overall consensus the credit rating for a comparable length dissertation varies from 20 to 40 CATS credits and that the contribution (weighting) to the degree is very variable.

A consistent range of themes are addressed by assessment criteria, typically research and analytical skills, critical ability, presentation, originality and self-organisation and management. There is encouraging evidence that a range of approaches are being utilised to explain these criteria to students. However, a common survey response stated that ‘students are told in tutorials’. This raises the question of whether students really do understand the expectations. Clearly the level of student understanding of criteria and expectations can be assessed integrally with assessment of the final report - but it would be valuable to assess their level of understanding at an earlier stage.

There is evidence of rigor and transparency in marking procedures and widespread use of grade descriptors, although also evidence of some confusion over terminology. However, there are significant differences in marking procedures, especially the role of the supervisor, the methods for resolution of disagreements, and the role of blind and anonymous marking.

Bearing in mind these conclusions and the good practices identified, we make the following suggestions:

1. Departments should develop rigorous methods to ensure assessment criteria and the expectations of a dissertation are explicitly explained to students, preferably through an action learning approach.

2. Markers should adhere closely to assessment criteria to achieve equivalence and consistency in grading standards;

3. Staff discussions should focus on ensuring clarity over the terms of assessment criteria, standards and marking schemes, as well as marking procedures.

4. Departments should review procedures for dealing with disagreement over marks to ensure they are transparent, consistent and rigorous.

Acknowledgements

We offer our thanks to GEES Subject Centre for providing financial support for this project through the Small Scale Project Fund; to Karen Logan for her assistance with data collection and collation; and to the GEES community for completing questionnaires and submitting details of assessment criteria and procedures.