Abstract
Undergraduate dissertations may provide up to one third of the weighting of the final year of a degree in Geography and are seen as important indicators of graduates’ independent research ability by prospective employers. This project examines the standards, criteria and assessment of the Geography dissertation in the UK. All Geography Departments in the UK were invited to complete a questionnaire survey that explored a range of aspects related to dissertations including format, assessment criteria and marking procedures. Responses were received from 24 Departments. The findings suggest that there is broad consensus in many areas including product format, study period, assessment criteria, and rigour and transparency in marking procedures. However, in other areas, including credit weighting, procedures followed in the event of a disagreement over marks, and interpretation of assessment criteria to students, there is wide variation in practice. Some suggestions are made to enhance equivalence and consistency in dissertation work.
Introduction
’Dissertations have had a long history in geographical higher education, being widely regarded as the pinnacle of an individual’s undergraduate studies and the prime source of autonomous learning’ (CitationGold et al., 1991). Our previous investigations into Geography dissertations (CitationHarrison and Whalley, 2006; Harrison and Whalley, 2008) suggest that this is still the case.
Dissertations typically constitute up to one third of the overall weighting for the final year which is Level 6 of the English, Welsh and Northern Irish system, (CitationQAA 2008), and Level 10 of the Scottish system (CitationScottish Credit and Qualifications Framework 2009) of undergraduate degrees. Dissertation performance may also be used to adjudicate degree awards in borderline cases. They present an opportunity for students to demonstrate their ability to work independently and autonomously. Grades awarded for dissertations are therefore of increasing interest to employers.
The role of dissertations in Geography undergraduate degrees can be set against a background of increasing concern throughout Higher Education about marking reliability, the maintenance of standards, ‘grade inflation’ and accountability. That said, there is increasing evidence of a new assessment culture emerging (CitationRust, 2007) that includes widespread use of assessment criteria (e.g. CitationHarrison and Whalley, 2008), grade descriptors and formative assessment as well as greater consideration of feedback timing and mechanisms. CitationRust (2007) argues that there are still poor practices that go unchallenged and a number of studies suggest that there are considerable inconsistencies in marking, weighting and standards (CitationHand and Clewes, 2000, Pepper et al., 2001), and confusion over terminology (CitationSadler, 2005). Ambiguities concerning the use, meaning and application of assessment criteria are also evident (CitationWebster et al., 2000). Others (CitationPenny and Grover, 1996, Rust et al., 2003, Woolf, 2004) have commented on the interaction of students with assessment criteria and marking schemes for dissertations, and in particular, their poor conceptual understanding of expectations (CitationGibbs and Simpson, 2004) and poor matching of assessment grades with tutors (CitationPenny and Grover, 1996). This does not accord with the QAA principle that “students and markers are aware of and understand the assessment criteria and/or schemes that will be used to mark each assessment task” (CitationQAA 2006, p16–17).
This paper reports on a project to review assessment schemes and procedures for undergraduate Geography dissertations in the UK. The project sought to address the following key questions:
What assessment criteria are used and how are they established and approved?
How are students assisted to interpret assessment criteria? What is the role of supervision in this?
What grade descriptors and marking schemes are used?
What are the procedures for double marking, anonymous marking and blind marking?
What happens in the event of a disagreement between first and second markers?
The project collated baseline information about dissertations, including credit rating, length, format, time available, preparatory work, supervisory arrangements and feedback, and identified good and innovative practice. This paper addresses selected outcomes from the research. A more detailed report will be provided at a later date.
Methods
A questionnaire survey was sent to all Geography Departments in the UK resulting in 24 responses (including one from environmental science). Twenty-two Higher Education Institutions (HEIs) were represented with separate responses from different Faculties / Schools at two HEIs. Responses were evenly split in terms of pre-1992 and post-1992 HEIs. The questionnaire contained a mixture of closed and open questions and invited respondents to submit a range of documents including assessment criteria, marking schemes and grade descriptors. Many respondents additionally submitted copies of Dissertation Handbooks, written procedures in the event of a disagreement over marks, lecture slides and other materials demonstrating good practice.
The nature of the dissertation
At the surveyed HEIs, 79% of students (92% of single honours students) have to prepare a final year dissertation. It is sometimes, but not always, an option for combined honours, joint honours and major-minor students. Alternatives to dissertations for these students include taught modules and work-based options.
Credit-rating and word length
The credit weighting of the dissertation varies from 15 to 40 Credit Accumulation Transfer Scheme (CATS) credits () with the modal weighting at 30 credits (25% of the final year). One HEI permits students to opt for the dissertation as either a single (15 CATS) or double (30 CATS) module.
The majority of dissertations (79%) are required to be 10–12,000 words in length (). One HEI requires a 4000-word piece of final work, but this excludes a literature review which is submitted separately at an earlier stage.
The relationship between credit weighting and word length is interesting: overall, there is a positive correlation, but this masks significant variation. The majority of 10,000-word dissertations are awarded 30 or 40 credits but one institution awards just 20 credits for the same. Another institution that provided two questionnaire responses awards 20 credits for a 10,000-word dissertation in one Faculty and 40 credits for the same word length in another Faculty.
Study period and product format
The overwhelming majority of students have at least 10 months to prepare the dissertation (), although this may partly reflect the fact that 55% are introduced to preparatory work in their penultimate year.
Most dissertations are submitted as hard copy, but one third also require e-submission for archiving and plagiarism detection. There is some flexibility about the format of submission with some allowing alternative formats or attachments (e.g. audio, visual, field notebook).
Assessed elements
The weighting of the final dissertation product varies from 70 to 100% () and elements making up the remaining portion of the module include interim progress reports, oral presentations, seminars, posters and literature reviews. In addition, several programmes include formative assessed elements (particularly seminars).
The nature of assessment
Assessment criteria
A wide range of assessment criteria are in use at the HEIs surveyed (Box 1). In some cases these are presented explicitly as assessment criteria and in others they are embedded into grade descriptors. The criteria submitted address the fundamental requirements for the dissertation, presentation, administrative considerations (such as ethics), evidence of student independence and what we term the ‘X factor’. The latter are the characteristics of a very high quality dissertation that are not easy to define in black and white terms — flair, innovation, creativity and criticality.
Fundamentals of the dissertation
Evidence of originality and perceptiveness
Clarity of aims, topic identification
Evidence of reading, awareness of literature
Quality of research design and methodology
Quality of data
Presentation, analysis, evaluation, synthesis and interpretation of data
Conceptual awareness, theoretical understanding
Sustained argument
Findings and conclusions justified and contextualised in the literature
Presentation
Standard of presentation, use of English language, structure
Use of complex academic terminology
Correct use of referencing conventions
Coherent integration of illustrative materials
Administrative
Conduct including engagement with administrative processes
Assessment of risks and ethical considerations
Compliance with requirements
Independence
Ability to work independently
Exercise of personal initiative and responsibility
Conduct and competence during practical work
Cognitive, intellectual, practical and personal skills
Appropriate and correct use of ICT applications
Reflective, critically evaluating own performance and personal development
The ‘X Factor’
Critical ability
Creative thinking
Flair, innovation
We were interested to explore how assessment criteria were developed and established. In most cases this occurred through team discussions, working parties and formal approval: at programme review and through consultation with external examiners. Some respondents said that criteria had evolved over time or through shared experiences and tradition. In some cases criteria were generic to the University, though one HEI identified the difficulties inherent in establishing common criteria even across a single department.
Explaining assessment criteria to students
Responses to the survey are encouraging in that the majority of HEIs do something to explain and interpret assessment criteria to students: typically through the provision of detailed text-based or online materials (e.g. a dissertation guide) that explicitly includes explanation of dissertation assessment criteria and expectations, and through the conduct of face-to-face tutorials and classes. These latter include formal, structured supervisory sessions where criteria are discussed, and whole cohort lectures. In some cases, it is assumed that criteria are discussed through the supervisory process, but there is no specific guidance or requirement to do so.
Grade descriptors and marking schemes
Almost all of the assessment criteria presented in the survey responses were couched in the context of standards or grade descriptors. However, in a number of cases it was difficult to extract the assessment criteria from the grade descriptors and the criteria appeared to vary for different grades. It is not clear whether this reflects the confusion around assessment terminology to which CitationSadler refers (2005) (e.g. criteria, standards, marking schemes) or whether assessment criteria are designed to permit marking flexibility. Although not explicitly stated as such, there appears to be implicit reliance on what CitationRust et al. (2003) refer to as the ‘connoisseur approach’ - knowledge transfer as a product of the student-tutor relationship.
The degree of numerical breakdown of each grade and the level of detail provided for each varies considerably. In some cases, broad descriptors were developed at institutional level and more detailed interpretations drawn up by departments.
There was very little evidence that any marking schemes were in use to award marks for different weighted sections or attributes of the dissertation. One HEI breaks the marks awarded into four components (40% for content, 40% for argument, 10% for structure / approach and 10% for style) and another has a three-way division (academic context; methodology, data collection and analysis; and interpretations, conclusions and presentation).
Marking procedures
At every HEI surveyed, dissertations are marked by two people. Only one HEI did not involve the supervisor in the marking process (). In another case the supervisor was the second marker. In 15 cases (62%), the second marking is completed ‘blind’ (i.e. markers are unaware of grades awarded by the other marker). In seven cases, anonymous marking is undertaken, although comments suggest this is only partially successful because supervisors recognise their students’ work.
Dealing with disagreements over marks
The first thing to note about dealing with disagreements over marks is that there is considerable variation over what constitutes a disagreement, with anything from 5% to 12% or a difference of class. In most departments, markers are encouraged to engage in discussions before introducing a third marker — and many disagreements are resolved at this stage.
Where the numerical difference is small, the mean mark is often used. Where a disagreement cannot be resolved between first and second markers, it is very common practice to introduce a third marker - often determined by the module leader or Head of Department. Subsequent procedures for resolution include discussions between the three markers, discussing all three marks at internal examination boards, using the median mark, and inviting an external examiner to adjudicate.
Exploring the role of external examiners a little further, we found split opinions on the matter. Some HEIs state, with some conviction, that external examiners are not asked to adjudicate - only to moderate. Others are comfortable in inviting external examiners to adjudicate where necessary. Four HEIs said that externals are commonly invited to comment on dissertations in borderline cases or where there is a particular problem such as suspected plagiarism or a failed degree where passes were achieved elsewhere.
Good practice
The survey respondents themselves identified a number of areas deemed to be good practice. There were two recurring themes:
Explanation and interpretation of assessment criteria for students. A variety of ways to achieve this were identified. These included detailed explanation of criteria and expectations in a dissertation guide; learning activities during the penultimate undergraduate year including peer assessment of past dissertations and discussions around criteria; final year whole cohort lectures including detailed discussion of criteria.
Marking procedures. Good practice involves following formalised procedures for double marking and blind marking and having rigorous and transparent procedures in the event of a disagreement. There is also good practice in the use of standard mark sheets that require a brief explanation of the mark awarded by each marker and the process of agreeing a final mark where there was disagreement.
Conclusions and recommendations
It is clear from the survey that, although the requirements for dissertations vary, there is broad consensus across the sector in terms of product format, the elements assessed, report weighting, study period, word length and credit-rating. It is a concern that within this overall consensus the credit rating for a comparable length dissertation varies from 20 to 40 CATS credits and that the contribution (weighting) to the degree is very variable.
A consistent range of themes are addressed by assessment criteria, typically research and analytical skills, critical ability, presentation, originality and self-organisation and management. There is encouraging evidence that a range of approaches are being utilised to explain these criteria to students. However, a common survey response stated that ‘students are told in tutorials’. This raises the question of whether students really do understand the expectations. Clearly the level of student understanding of criteria and expectations can be assessed integrally with assessment of the final report - but it would be valuable to assess their level of understanding at an earlier stage.
There is evidence of rigor and transparency in marking procedures and widespread use of grade descriptors, although also evidence of some confusion over terminology. However, there are significant differences in marking procedures, especially the role of the supervisor, the methods for resolution of disagreements, and the role of blind and anonymous marking.
Bearing in mind these conclusions and the good practices identified, we make the following suggestions:
Departments should develop rigorous methods to ensure assessment criteria and the expectations of a dissertation are explicitly explained to students, preferably through an action learning approach.
Markers should adhere closely to assessment criteria to achieve equivalence and consistency in grading standards;
Staff discussions should focus on ensuring clarity over the terms of assessment criteria, standards and marking schemes, as well as marking procedures.
Departments should review procedures for dealing with disagreement over marks to ensure they are transparent, consistent and rigorous.
Acknowledgements
We offer our thanks to GEES Subject Centre for providing financial support for this project through the Small Scale Project Fund; to Karen Logan for her assistance with data collection and collation; and to the GEES community for completing questionnaires and submitting details of assessment criteria and procedures.
References
- GibbsG. and SimpsonC. 2004 Conditions under which assessment supports students’ learning, Learning and Teaching in Higher Education 1, 3-31
- GoldJ. R., JenkinsA., LeeR., MonkJ., RileyJ., ShepardI. and UnwinD. 1991 Teaching Geography in Higher Education, Basil Blackwell, Oxford
- HandL. and ClewesD. 2000 Marking the Difference: an investigation of the criteria used for assessing undergraduate dissertations in a business school, Assessment and Evaluation in Higher Education 25, 1, 5-21
- HarrisonM. E. and WhalleyW.B. 2006 Combining student independent learning and peer advice to improve the quality of undergraduate dissertations, Planet 16, 15-18
- HarrisonM. E. and WhalleyW. B. 2008 Undertaking a dissertation from start to finish: the process and product, Journal of Geography in Higher Education, 32, 3, 401-418
- PennyA. J. and GroverC. 1996 An analysis of student grade expectations and marker consistency Assessment and Evaluation in Higher Education, 21, 2, 173-184
- PepperP., WebsterF. and JenkinsA. 2001 Benchmarking in Geography: some implications for assessing dissertations in the undergraduate curriculum, Journal of Geography in Higher Education, 25, 23-35
- Quality Assurance Agency 2006 Code of Practice for the Assurance of Academic Quality and Standards in HE, Section 6 - Assessment of Students, http://www.qaa.ac.uk/academicinfrastructure/codeOfPractice/section6/COP_AOS.pdf Accessed 22 June 2010
- Quality Assurance Agency 2008 The Framework for Higher Education Qualifications in England, Wales and Northern Ireland, August 2008, QAA, Mansfield. http://www.qaa.ac.uk/academicinfrastructure/FHEQ/EWNI08/FHEQ08.pdf Accessed 21 June 2010
- RustC. 2007 Towards a scholarship of assessment, Assessment and Evaluation in Higher Education, 32, 229-237
- RustC., PriceM. and O’DonovanB. 2003 Improving students’ learning by developing their understanding of assessment criteria and processes, Assessment and Evaluation in Higher Education, 28, 146-164
- SadlerD. R. 2005 Interpretations of criteria-based assessment and grading in higher education, Assessment and Evaluation in Higher Education, 30, 175-194
- Scottish Credit and Qualifications Framework 2009 The Scottish Credit and Qualifications Framework http://www.scqf.org.uk/AbouttheFramework/Overview-of-Framework.aspx Accessed 22 June 2010
- WebsterF., PepperD. and JenkinsA. 2000 Assessing the undergraduate dissertation, Assessment and Evaluation in Higher Education, 25, 1, 71-80
- WoolfH. 2004 Assessment criteria: reflections on current practices, Assessment and Evaluation in Higher Education, 29, 4, 479-493