Educator perspectives on the use of alternative assessment methods within taught Masters programmes: an exploratory study using activity theory and Q methodology

Abstract

This article reports on an exploratory two-stage sequential mixed methods research study that investigated the views of university educators on the introduction of assessment methods other than essays, exams and dissertations within taught Masters programmes. In the first stage, interviews were conducted internationally with 45 participants and the data analysed using an activity theoretical framework. The article illustrates how interviewees questioned their existing practice and reworked aspects of it, using new assessment methods as tools to transform their teaching and learning processes. The issues emerging from this analysis informed the design of a second study that used Q methodology to interpret five shared viewpoints on assessment methods among a group of UK-based educators. These viewpoints all saw alternative assessment methods as having advantages over traditional methods in, for example, encouraging student motivation and improving equality of opportunity, although training for academic staff in such methods was seen as necessary. Differences between the five viewpoints related to, for example, whether improving assessment methods requires a shift in how learning is viewed, and the importance of institutional support in encouraging assessment innovation. The five viewpoints are compared and contrasted. Tensions within and across assessment related activity systems are also considered, and recommendations made for further research into stakeholder viewpoints and values to improve collaboration across interacting networks.

Keywords:

• assessment
• postgraduate
• mixed methods
• activity theory
• Q methodology

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. The subject areas included: Arts and Heritage Management, Computer Assisted Translation, Educational Psychology, Future Design, Genetic Counselling, Librarianship and Information Management, Manufacturing, Medical Imaging, Railway Signalling and Telecommunications, Research in Clinical Practice, and others.

2. Aside from the study reported here, which involved activity theory and Q methodology, a compendium of case studies describing the assessment methods was published as a separate output of the project. The project legacy website is available at: https://sites.google.com/a/teams.leedsmet.ac.uk/assimilate-2012/.

3. A final report on the entire project is available on the project’s legacy website, along with two separate reports on the activity theoretical analysis of the interview data and the Q methodological study.

4. The project outputs included an introductory web animation, which outlines to newcomers the stages involved in a Q study. The animation can be viewed at: http://youtu.be/0AejeH6jw2c.

5. In relation to Q and its technique, Watts and Stenner (2012, 10) explain that Stephenson (1953), the originator of Q methodology, used the generic term “R methodology” to refer to all methods of the type involving Pearson’s correlation statistic, r, “which employ tests or traits as variables and operate using a sample of persons”. In contrast to this by-variable approach, Watts and Stenner (2012) describe Stephenson’s change of focus away from R’s concern with “specific bits of people” to a concern with whole persons, by running “by-person” as opposed to by-variable factor analyses. This methodologically significant adaptation was recognised in 1935 by the factor analyst Godfrey Thomson, who proposed the use of the letter Q to signify “any attempt to pursue correlations between persons, rather than correlations between tests or variables, as had been the case in R methodology”.

6. The participants represented a diverse range of subject areas including business and management, dentistry, geography and earth sciences, health and life sciences, clinical pathology, teacher education and training, community and youth studies, electrical and audio engineering, television and film production, computing, agriculture, food and rural development. It should be noted, however, that in a Q study “the sampling goes into the Q set”, while the participants “are chosen to facilitate the expectation of finite diversity” among the variety of possible sorting patterns (Stainton Rogers 1995, 182).

7. For details of the minimal effects of using forced rather than free distributions on the resulting factors, see Brown (1971).

8. In Q studies, centroid factor extraction is the option preferred by followers of Stephenson (Brown 1980). However, as noted in the Q literature (e.g. Schmolck 2014; Stainton Rogers 1995; Webler, Danielson, and Tuler 2009), PCA has become a widely used alternative to centroid. This is reflected in the incorporation of PCA within the dedicated Q software package PQMethod (Schmolck 2002), which was used in the present study.

9. In relation to the flagging of Q sorts to create the factor estimates, Watts and Stenner (2012, 131) argue that some flexibility is acceptable, advising that “you could, for example, use only those Q sorts with factor loadings greater than 0.60 … . Some Q methodologists add a further caveat, insisting perhaps on a factor loading of greater than 0.60 on the relevant factor and factor loadings of less than 0.40 on all the other factors (Jordan, Capdevila, and Johnson 2005)”. The procedure used in the present study adopted a similar approach.

10. In a Q study, the number of individual Q-sorts exemplifying each factor does not necessarily reflect the prevalence of each viewpoint in the wider population (Robbins 2005), and no claims as to generalizability are made here in that regard. In Q methodology, as Newman and Ramlo (2010, 526) explain, “we are not making statistical inferences from samples to populations”.

11. Due to limitations of space, only the factor 3 viewpoint is described here. However, the full profiles of all five viewpoints are available in a separate report on the project legacy website.

12. The factor 3 members’ areas of professional interest were: Teaching, Learning and Assessment (P11), Learning and Philosophy (P20), Teacher Training (P21), Pedagogic Practices (P23), Academic Practice (P25), Education Development (P34), Assessment, Learning and Teaching (P37), and Professional Practice in HE (P39).