The genre regime of research evaluation: contradictory systems of value around academics’ writing

Abstract

This article addresses how academics navigate different kinds of prestige and different systems of value around what ‘counts’ in academic writing, focusing particularly on the impact of the genre regime associated with research evaluation in the UK Research Excellence Framework (REF). It draws on data from an Economic and Social Research Council (ESRC)-funded project working with academics across different disciplines and different institutions in England. We interviewed people about their writing practices several times, exploring their practices, life histories, institutional contexts, and the tools and resources they draw on as they write. Academics’ research writing is framed within explicit institutional and departmental strategies around the numbers and publication venues of research outputs, driven by institutions’ need to succeed in the national competitive research evaluation system. Such institutional strategies do not always map well onto other values systems in which academics have been trained and within which they locate themselves. The articles analyses the interviews we carried out, exploring how academics negotiate tensions between these systems of value and considering the implications of this for what is considered to be important in academic work and, therefore, what it means to be an academic.

Keywords:

• Academic writing
• academic literacies
• research evaluation
• Research Excellence Framework
• accountability
• genre regimes

Introduction

I started writing this article in a room full of other academics, at a departmental writing retreat organised by our research committee. The department was able to justify funding this activity, in a luxury hotel, with food and drink provided throughout the day, because there is a strategic justification for it. Supporting people’s writing is important for the department, which needs to produce a large number of ‘outputs’ (publications) of a quality which will score highly in the REF (Research Excellence Framework, the UK system for rating the research output of universities). Outputs scored at three or even four stars – rated internationally significant or world-leading – boost the chances of our department making a good contribution to the rating of the university, and thereby increasing its income streams. This article explores the effects of the REF in the UK on academics’ writing practices.

It draws on interviews conducted with academics across a range of institutions and disciplines to explore how the values and structures of the system of research evaluation in England emerge in academics’ talk about their research-related writing. It will argue that the systems and structures which frame academics’ writing, particularly the REF and its mediation through institutional and departmental strategies, have the effect of constructing a dominant ‘genre regime’ with its own rules, representations, and preferred practices around academic writing. Academics in different disciplinary and institutional contexts orient to this dominant regime in different ways. For some, the values implicit in the REF structures are consonant with their existing expectations about writing. For others, their disciplinary knowledge traditions may be in tension with certain aspects of the REF, or aspects of the strategies adopted by institutions may push them to make choices about writing that they would not otherwise have made.

Managerialism in universities

The REF and associated institutional strategies are part of a larger shift towards a more managerialist approach in universities (Deem, Hillyard, and Reed 2007). Managerial strategies influencing academic writing can be historically located as part of the shift to a New Public Management orientation in the governance of universities and other public services in the UK and in other Northern European countries (Shepherd 2017; Morphew, Fumasoli and Stensaker 2018), introducing practices and concepts from industry and private enterprise, such as efficiency and effectiveness, into the public sector. More broadly, this is part of a neoliberal perspective which shapes the understanding of the purpose of universities as being primarily to support the socioeconomic development of the country.

Higher education in England has experienced many changes in recent years, particularly in relation to funding and the systems by means of which that funding is differentially distributed. Universities are funded through a range of sources, but the majority of their income comes from a combination of student fees and (state and private) research funding (UUK, 2016). The introduction of student fees in 1998, initially £1000/year but rising to £9000/year in 2010, has been transformational in many respects, fostering increased competition between institutions, repositioning students as consumers (Bunce, Baird, and Jones 2017) and affecting relationships between students and teaching staff (Naidoo and Williams 2017). However, the focus here is on research, particularly the REF, the evaluation system associated with the allocation of governmental research funding for institutions (QR, or quality-related, research funding), how this shapes managerial approaches to writing, and how, in turn, these shape how academics talk about the way different writing choices are valued in their working lives.

The Research Excellence Framework and its effects

Research evaluation exercises have existed for a long time, first introduced in England in 1986 (Stern 2016). From 1992 onwards, the university sector expanded and the Research Assessment Exercise (RAE, 1996 onwards) became increasingly important for both distributing funding and evaluating the research quality of institutions.

The 6–7-year research evaluation cycle is enormously influential in English universities, particularly the research-intensive universities, for whom QR funding and research grants form a significant part of their income stream. Research-intensive universities depend on good ratings in the REF to keep them performing well in university league tables, and so there is also an indirect influence on the funding stream from student fees.

The aim of the research evaluation exercise is to measure and compare the quality of research in academic institutions across the country, in order to ensure the most efficient and effective distribution of QR research funding. Units of Assessment, usually equivalent to Departments, are evaluated using three elements: nominated ‘research outputs’, mostly monographs and journal articles reporting empirical research; an account of the unit’s ‘research environment’; and selected case study accounts of the ‘impact’ of research, beyond academia. Institutions have responded to this high stakes accountability exercise by encouraging individuals to make decisions about their writing designed to achieve the best possible results.

In previous evaluation exercises, institutions selected which people to submit to the REF. The amount of QR funding received is proportional both to the size of the unit of assessment, in numbers of full-time equivalent staff members, and to its overall rating. Hence, the potential for increased funding associated with submitting larger numbers of people must be balanced against the risks associated with submitting someone whose research may not receive a high rating. The impact of such strategic decisions is deeply personal. For an academic working under a contract which includes research, failure to be selected for REF submission could be seen to indicate one’s research is of inadequate quality and therefore lead to problems in gaining promotion, or indeed in continuing in secure employment.

Consequences of the REF on research and on academics have been explored from a range of disciplinary perspectives. In a comparative overview of performance-based research funding systems, Hicks (2012) argues that such systems run the risk of enhancing the control of research elites with a narrow view of research, potentially compromising important values like equity and diversity. This concern is shared by Schäfer (2016), who identifies a wide range of impacts of the REF and RAE including limiting the choice of research topics because of the emphasis on publishing in high impact journals, a concern which will be explored further below. Kelly and Burrows (2012) highlight the recursive nature of research evaluation systems, in which technologies designed initially to measure and value academic work comes to shape and define it as institutions, departments, and individuals make decisions about research in response to the system.

Such systems are not unique to the UK context. Blackmore (2010) analyses how research evaluation has developed in the Australian context, arguing that the criteria of quality, efficiency, performance, and accountability have been counterproductive, particularly when metrics for ‘quality’ more suitable to the hard sciences have been used and inappropriately applied across different disciplines. Ledin and Machin (2016) show how a performance management system in a Swedish university uses statistics and graphics from the Web of Science in a chain of documents to represent and codify research in a standardised way which does not represent process or disciplinary difference and can be used to support a managerialised approach to steering the university. Lillis and Curry (2010) show how knowledge evaluation systems associated with marketisation and internationalisation of global Higher Education shape the writing decisions of multilingual scholars in non-Anglophone centre countries (Hungary, Slovakia, Spain, and Portugal). Pressure to publish in English places additional burdens on multilingual academics and has potentially detrimental effects on knowledge production in local languages.

Some of the more problematic aspects of the REF are changing. A recent independent review (Stern 2016) claimed that while the REF is value for money, drives competition, and fosters high quality research, it has also had some more negative outcomes. These include the high cost of preparing and assessing submissions, and institutions ‘gaming’ the results by hiring staff purely to enhance REF returns, making selective submissions which did not represent the research activity of that Unit of Assessment, or excluding individuals for strategic reasons. There were equality concerns, with more men than women being selected, and black, Asian, non-EU national, and staff with disabilities being under-represented. Concerns were also expressed that academics were being influenced in the design and conduct of their work, avoiding high-risk projects and conducting research likely to lead to a steady stream of REFable papers. Stern’s recommended changes include submitting all research active staff to the REF, rather than selecting; collating outputs at the level of the Unit of Assessment, submitting an average number per full-time member of staff rather than a fixed number per individual; and not enabling outputs to be portable between institutions, to avoid strategic hiring.

These recommendations have largely been addressed in the 2021 REF rules (REF2021, 2017), which is to be welcomed. As this article will demonstrate, the criteria and values associated with the previous REFs have indeed influenced academics’ everyday writing practices. The article will show how academics talk about and negotiate tensions between the values they identify with around writing, for personal and disciplinary reasons, and the systems of value implicit in the research evaluation system, as mediated through the strategies of their institutions and departments.

Method

The project on which this article draws investigated the nature of academic writing in the context of significant changes in English Higher Education.¹ We were particularly interested in sociomaterial aspects of writing (time, space, and networks); new digital platforms and resources; and, most relevant to this article, the influence of managerial systems. The project explored all the different kinds of writing academics do as part of their working lives, following in particular work by Lea and Stierer (2009, 2011) which showed how a broad perspective on lecturers’ everyday professional writing can provide new insights into academic professional identities and practices. This particular paper, though, focuses on the data in the project around research-related writing.

Our theoretical starting point was to approach literacy as a social and material practice (Barton and Hamilton 2000). This entails a holistic perspective, working in detail with people to understand their writing practices, setting these in historical and social context, with attention to ideologies and power relationships, material aspects of writing, and the values and perspectives people express about their writing.

Our core participants were 16 academics across three disciplines, Mathematics, History, and Marketing, representing a STEM (Science, Technology, Engineering, and Mathematics) subject, a traditional humanities discipline and a professional/applied area. Participants came from three universities, of different types. Five core participants came from a large research-intensive institution founded in the nineteenth century. Six came from a medium-sized post-1960s institution, also research-intensive. Five came from a post-1992 university which had been a polytechnic and was more teaching-intensive in nature.

These core participants were from a range of career stages, nationalities, and genders, although men and speakers of English as a first language were in the majority. Each was interviewed three times. First, in a ‘go-along’ interview in their workplaces (Garcia et al. 2012), discussions covered a general understanding of their role and the writing practices within this. Second, a ‘technobiographical’ interview (Barton and Lee 2013) explored the history of the tools and resources people had drawn on in writing over their lifetime. Third, in a ‘day in the life’ interview, people described all the writing they had engaged in on specific recent days. We also recorded people’s writing processes, mostly using a ‘screen-in-screen’ approach (Bhatt and de Roock 2013) to capture both the screens they were writing on and the settings they were writing in.

In the final phase of the project, we interviewed colleagues of our core participants, including other academics, administrative, and managerial staff, testing out our emerging understandings of writing practices in a broader context. We also interviewed colleagues in social science disciplines as part of the piloting process, and carried out autoethnographic work by interviewing one another and keeping records of our own practices. We collected photographs, documents, and writing samples. In total, 70 people were interviewed.

All the interviews and photographic data were uploaded to an ATLAS.ti hermeneutic unit, and coded by the research team. The initial coding scheme was based on the research questions and became more complex and detailed as we worked with the data. The aim of the coding was to be able to track, search, and work with the content of the interviews. Eventually, 515 codes were identified. The main groups of codes located instances in the data relating to different types of boundaries; cultures; digital devices; digital platforms; disciplines; divisions of labour; evaluations; genres; modes; persons; places; practices; resources; spaces; strategies; tools; and worktypes, with some additional codes not in groups, including a code for REF. Most of this coding framework is not directly relevant to the issues discussed in this paper, which focuses specifically on a subset of the dataset identified by specific relevant codes.

The focused data analysis for this article began by identifying the locations in the interviews where participants mentioned REF-related issues. All data coded with REF was extracted from the overall dataset. Data coded with other related topics were also extracted, including management, autonomy, choice, disciplines, and culture (departmental/institutional/national). This data were read multiple times and grouped, additionally annotating manually to identify common themes, patterns, and issues. Particular attention was given to explicit discussions of the influence of institutions, managerial strategy, and the REF, and especially to expressions of evaluation and deontic modality, in order to locate and analyse in particular how participants evaluated their experiences of the research evaluation framework, and discussions of experiences of being constrained or directed. Once the key themes, which form the basis of the subsections in the discussion below, had been identified by grouping the data extracts, quotations from each grouping were selected which were judged to represent and explicate each theme most effectively for inclusion as examples. In the discussion which follows, what it means to conceptualise the REF as a genre regime will be discussed, before moving on to explore the key themes of practices, representations, and rules; identities; values; disciplinary knowledge traditions; and using journal ratings as proxy REF ratings. These themes are interpreted in relation to the concept of genre regime, showing how parallels can be identified with aspects of the idea of language regimes.

REF as genre regime

This special issue as a whole addresses the language regimes governing academic writing. Where some of the other papers in this special issue are writing about language regimes specifically, with an expectation of publication in particular languages (predominantly English) being an important driver of academics’ writing choices, academics in England are for the most part already writing in English. The regimes they face relate to genre choice, rather than language choice.

Working with this data has therefore led me to develop the related concept of genre regimes. The idea of genre regimes is informed by the notion of ‘regimes of language’ as developed, for instance, by the contributors to Kroskrity (2000) as a way to approach the connections between language, ideologies, politics, and identity. Sonntag and Cardinal (2015) take up this notion of language regimes, defining them as the rules, representations, and practices which surround language use in a particular setting. Language regimes incorporate rules about what language can be used where and when, both in terms of representations of language use and in terms of actual practices (Liu 2015). In a similar way, genre regimes incorporate rules, representations, and practices around which genres can and should be used in academic writing to what purposes. The term ‘genre’ is being used in this context to indicate the type of texts participants were writing, as these are labelled and recognised by our participants (e.g. article, monograph, report), with the notion of ‘genre regime’ incorporating both the types of texts which are valued and encouraged and the location of their publication.

Sonntag and Cardinal are particularly interested in the language policies of states. Their discussion revolves around the relationship between state tradition and language policy, using the language regime concept to address how and why states make particular language policy choices. This development of the concept of language regimes has several parallels with the experiences of academics making choices about writing within the institutional frameworks of the contemporary university and research evaluation system. In the discussion below, I will identify some of these parallels. These include the role of rules, representations, and values in academics’ genre choices; how the REF genre regime generates paradigmatic representations of academics’ identities; the relationship between genre choices and questions of power, competition, and the distribution of resources; and how disciplinary ‘knowledge traditions’ can be seen to have a shaping influence on genre choice, in a similar way to the significance of ‘state traditions’ in shaping language policy choices. This broad concept thereby helps to provide analytical purchase on how the REF operates to shape academics’ writing choices and the strategies that people adopt in response to this.

Practices, representations, and rules

Since regimes incorporate practices, representations, and rules, while they may be laid out in the policies of states and institutions, they are experienced, enacted, negotiated, and resisted by individuals. A genre regime comprises both explicit rules within institutional and national systems, and the lived understandings of which genres are most highly valued. Based on our participants’ experience of the REF, the rules of the exercise – or perhaps, better, the way these rules were interpreted by institutions – seem to have led to the development of a genre regime privileging particular types of output (journal articles, and, to a lesser degree, research monographs), published in particular locations.

The rules can be summarised as follows. The evaluation of research publications, or ‘outputs’, still made up the greatest proportion, 65%, of the REF evaluations in 2014 (with 20% being made up by evaluations of impact case studies, and the other 15% based on an account of the quality of the research environment of the unit). REF outputs were assessed by peer review panels against three criteria: originality, significance, and rigour. These criteria were used to evaluate each output as four star (world-leading), three star (internationally excellent), two star (recognised internationally), one star (recognised nationally), or unclassified (falling below the standard of nationally recognised work, or which does not meet the published REF definition of research). Under these regulations, universities and departments have produced strategies to maximise the likelihood of production of three- and four-star outputs. These strategies tend to focus on where academics are encouraged to publish, and in what form.

Universities may be increasingly influenced by the values of the competitive marketplace, but the long-standing underlying value of academic freedom is still a powerful one. Interference with or control of the content of an academic’s research work would therefore still, generally speaking, be perceived as being beyond the pale. It is clear from our data, however, that attempts to influence the genre and location of publication of that research have become the norm. From the strategic perspective of university management, encouraging academics to produce articles in high-impact, peer-reviewed, international journals is seen to maximise the likelihood of good REF scores. And while bibliometrics were used by only a minority of panels in REF 2014, mainly in the sciences where citation counts complemented peer review, there is always the possibility of metrics, including journal-based ones, being introduced more widely in the next REF, though this is controversial and opinions as to the value of this vary greatly (Stern 2016; Wilsdon et al. 2015). There was a high level of awareness among our participants of the perceived importance of publishing in journals considered to be good, ideally with high impact factors. As Rory, a Marketing academic at the teaching-intensive institution, explained to us, ‘I choose what I write within the structure of what you have to do in academia. So I’ll choose what I write about, I won’t choose where I put it. I’ve got to try and find two and three and four star journals.’ (All participants’ names have been anonymised. Interview transcriptions are orthographic and repetitions, backchannelings and fragmented speech have been edited out for clarity of reading. […] indicates that part of the quote has been removed.)

Despite the critiques expressed by many of our participants, the REF genre regime has become a normalised and accepted reality of academic life. This can be seen in people’s frequent use of words like ‘obviously’ or ‘of course’ which presuppose a shared understanding of the importance of REF criteria when explaining their goals. Elizabeth, an historian at the teaching-intensive institution, explained that ‘right now they expect me to be REF-able, obviously’. Natalie, another historian at the larger research-intensive institution, explained that ‘Research is the final aspect of my role. Obviously I’m expected to produce a certain number of articles for the REF within each six-year cycle and give conference presentations and all the usual things associated with research dissemination.’ Describing the expectations of research first and foremost as the production of a number of articles for the REF, rather than, for instance, to push the knowledge of the field forwards or to contribute to the development of the discipline, shows the extent to which this evaluative system has become normalised for her. As an early career researcher, it was particularly striking to see how she described her mentor – a professor – as being behind the times in encouraging her to take time to work on her book rather than to focus on REF-able articles, saying, ‘I think he’s a little out of touch with the REF and the pressure’. This pressure to produce is reflected in Stern’s (2016) critique that the REF process encouraged short-term thinking which focused on maximising the flow of predictable research outputs. Louis, an historian at the same institution, constructed a metaphor of the REF as industrial production line, explaining that the pressure was constantly on him to ‘generat[e] REF widgets’.

Representations of identities

According to Sonntag and Cardinal (2015), citizenship regimes encode paradigmatic representations of citizens’ identities, and social relations between citizens and states. Similarly, the REF genre regime encodes paradigmatic representations of academics’ identities, and social relations between academics and institutions. This can be seen in the introduction of a new identity category: the REF-able academic. Producing outputs acceptable under the genre regime of the REF has become not just part of the job, but a label which can be used to indicate someone’s professional identity. James, an historian at the teaching-intensive institution, explained, ‘You’ve written enough articles, at some point maybe get promoted or do this or do that or you’ll be REF-able, or you won’t be REF-able’. Natalie, the historian quoted above who speaks of the obviousness of REF pressure, feels safe because she is ‘OK for the REF and OK for probation’, having produced ‘three published articles in good quality journals’ on the basis of her PhD work. Safety is achieved through ensuring one becomes REF-able, while to be categorised as not REF-able leaves one in a dangerous and vulnerable position. Therefore, when people’s publications are not in the genres and locations perceived to be a good fit with the REF evaluation criteria, they can be disadvantaged. D. Blue, a historian at the teaching-intensive institution, felt that ‘there are lots of people who actually do good research who can’t even get to the REF, because they don’t do it in the kinds of ways that the REF finds it easy to count.’

Values embedded in institutional monitoring strategies

Language choices are always inherently political, relating to power, the distribution of resources, and competition over these resources, as Sonntag and Cardinal (2015) recognise. And of course, as Kroskrity (2000) argues, language ideologies – perceptions about how language is and should be used – privilege some cultural and social interests over others. Academics’ writing choices have always been political in this sense. Publications in the ‘right’ places and of the ‘right’ kinds have always been one of the key constituents of individual academics’ reputations and career enhancers; see, for instance, Anderson’s (2013) discussion of the explicit choices about publication made by young multilingual scholars in Europe designed to maximise their chances of career success in their disciplinary, national and European contexts. But the importance of the REF, both in relation to the distribution of QR funding and in its broader contribution to league table success and academic reputations, has increased the political significance of publications beyond the individual, in a context in which publications like the Times Higher and QS World University Rankings classify universities across the world competitively against each other. Institutions use the levers they have at their disposal to maximise the perceived quality of individuals’ outputs, to position themselves most advantageously in relation to the distribution of scarce resources.

The academics we interviewed, particularly the heads of department, described a range of institutional strategies designed to foster REF success. These aim to maximise the likelihood that academics will produce three and four-star outputs, particularly by encouraging journal articles in high impact factor international peer-reviewed journals. These strategies are embedded in regular appraisals, probation meetings, and audits of departmental progress. Rose, head of a Marketing department in a research-intensive institution, explained that in addition to the developmental and future-oriented professional development review process in place in their institution, they have an annually audited research appraisal system. People record everything they have submitted, and the outcome, against their personal research plan, in relation to the REF period. Andrew, head of a Maths department at the same institution, described a system where all members of the department were supposed to record their publications on the institutional research repository. The Head of Research and Head of Department would conduct a ‘REF health check’ once or twice a year, assessing the likely ratings of the publications using ‘educated guesses’ based on experience. Such systems make the REF genre regime and its associated rewards and punishments highly salient for academics. Charles, a Marketing lecturer at the same institution, described this in Pavlovian terms: ‘If somebody said to me, “You will get on, keep your job, maybe even get promoted if you bang your desk every morning,” then everybody would be banging their desk every morning.’

This is not to say that heads of department, responsible for mediating institutional requirements in their departments, accepted the REF criteria uncritically. Phil, head of a social science department in a research-intensive university, had ‘mixed feelings’ about assigning potential star ratings to colleagues’ papers. ‘They should not be doing that because I need it. They should be doing that because that is their contribution to the field. Then we take their contribution to the field and use it for departmental purposes. If people are writing to the REF, they are doing the wrong thing.’ This encapsulates the tension between the value attributed to writing which fulfils departmental and institutional strategies, compared with writing driven by contributing to a field or discipline. Chris, a head of department in History at the same institution, made a clear distinction between understanding how work is valued within the genre regime of the REF, and other ways of valuing people’s writing. ‘I’m very sceptical whether there are absolute standards. That’s why I’m sceptical about the REF. But, I do have quite a good sense of what’s going to fly in the REF, and I believe those are two very, very different things.’

Clara, a head of department in History at the other research-intensive institution, explained that the process of reading colleagues’ work has become formalised, within a university-mandated structure for internal assessment of publications. She contrasts the values embedded in the REF criteria with her previous criterion of ‘interestingness’ for evaluating colleagues’ work: ‘Whereas once we probably read our colleagues’ work in the sense of is this interesting, can I learn something from it, I think there’s another dimension to it now, where we’re reading colleagues’ writings in terms of how would this be ranked’.

At the same time, academics are negotiating a range of other evaluation systems. Clara explained that the REF is now accompanied by other competitive systems of accountability, both at the national and at the institutional level: ‘There are the big things like NSS [the National Student Survey], probably TEF [the new Teaching Excellence Framework], REF.’ Similarly, Charles, in Marketing at a research-intensive institution, articulates the pressure to address multiple expectations: ‘How do you manage what you’re doing in terms of the feedback and marking scripts versus the growth in numbers, versus the growth in university expectations for four star agenda?’ Fulfilling requirements to make research publications open access is another hurdle to overcome. ‘Just when you think you’ve got your REF return, just when you think it’s safe to go back in the water, there’s another bloody shark appears and it’s in the shape of the open access.’ Each additional requirement adds another level of complexity and another complication, and another possibility for double-binds. Lisa, a Marketing academic at a research-intensive institution, experiences tension between achieving REF productivity and carrying out activities to support the department: ‘I know I’ll walk in to [my appraisal meeting] and they’ll be like, “So where’s your publications.” I’ll be like, “Well I went to this, this, this, this and this, because you told me I had to for collegiality but that takes away from the time that I can actually sit and write up a four star publication.” ’

The extent to which institutions focus strategically on REF writing varies, depending on the relative importance of QR funding for each institution, and the consequent effects on academics are different. Alex, a historian, works in a research-intensive university which sees itself as an elite institution, with what he describes as a ‘very active central management’. He finds that this management is, to an increasing degree, setting centralised targets which are decided at a higher level of administration than the department. He explains, ‘the department is an important unit in terms of teaching and workload and collegiality and has a good amount of control over what we teach […] But in terms of research, increasingly I think the targets are driven centrally, across a range of arts and humanities disciplines. And then the department is merely the unit which has to enact or deliver the target that’s been decided elsewhere.’ Several points are worthy of note here. First, the goals of research are no longer set internally by the discipline or by the gaps in knowledge perceived by scholars within that discipline, but by central university management, to fulfil university strategy. Second, this is expressed as a compulsion; the department ‘has to’ meet these targets. Thirdly, managerialist language is used (perhaps ironically) by Alex to characterise the job the department has been given, to ‘deliver the target’, with research being represented as a product to be delivered rather than a process of discovery.

However, in the teaching-intensive institution we worked in, where QR income contributed a smaller proportion of the university’s funding, people – especially those from units with strong income streams from other sources such as students or industry consultancies – felt less pressurised by the REF. Susan, based in Computing, explained that her section did not have a strong research culture or lots of publications, but this was not seen as a problem: ‘Well, Computing has the students. We’ll look at Physics and Chemistry for the research papers.’ In an organisational culture where, as another respondent put it, ‘you’ve got quite a lot of people who don’t do any research […] so the expectations are quite low’, the genre regime associated with the REF bears much less weight.

Disciplinary knowledge traditions and genre regimes

‘State traditions’ are identified by Sonntag and Cardinal (2015) as playing an important role in shaping language regimes. Similarly, we can talk about disciplinary ‘knowledge traditions’ which have shaped the regimes governing academics’ research and writing choices. Disciplines orient to knowledge traditions which value particular genres and choices about writing. These are traditions within which academics have been trained and within which their professional identity has been shaped, so they are particularly important in influencing academics’ writing choices. As Kuteeva and Airey (2014) argue, academic disciplines have different knowledge structures which are associated with different patterns of language use. While their focus is on choices about the language of writing (English or Swedish), I argue here that disciplinary knowledge traditions are also associated with different sets of choices about genre and location of publication; that is, disciplinary knowledge traditions have their own genre regimes and associated systems of value for writing which may conflict with those of centralised REF strategies.

The knowledge tradition oriented to, implicitly, by the REF criteria, is closer to the values and traditions of some disciplines than of others, particularly to those of the empirical quantitative sciences. Sonntag and Cardinal (2015) explain that language regimes empower some more than others. Genre regimes do the same thing, empowering those whose existing writing practices fit the demands of the genre regimes most straightforwardly while disempowering others.

In history, for example, many of our respondents talked about the value which had traditionally been placed on the single-authored academic monograph, which could take years of work in archives to produce. While a single monograph could be key to establishing an historian’s contribution and reputation within their discipline, this kind of work does not sit easily with the idea of producing four highly-rated REF-able outputs within a six-year period. While monographs can count as two outputs, this double-weighting still does not necessarily reflect the time which may be required for the production of such historical work.

The official REF 2014 definition of research was a broad one: ‘a process of investigation leading to new insights, effectively shared’ (REF2014, 2011). However, the way this was interpreted within institutional and departmental strategy was narrower. There was a perception that panels would favour research based on original data and sources. This was another problem for the historians. Louis explained that ‘an awful lot of the outputs that have to be made have to be in producing REF units or not […] there are so many distinctions between what is real research, which is entirely based on primary evidence, and what is derivative or secondary, that is really frustrating.’ Note the language of compulsion here, referring to the outputs which ‘have to be’ made and the kind of units they ‘have to be’. This stipulation means that what he calls ‘meta studies’ – overviews of an area which bring a wide range of work together and can produce new knowledge and insights by showing a bigger picture – ‘get no value’. A traditionally respected genre of historical writing loses its worth in this new system. Chris, the head of History at a research-intensive institution introduced above, concurs: ‘Even very sophisticated literature reviews in history are poorly regarded [in the REF] […] I had a couple of colleagues who had very good, very excellent articles that were really sophisticated reviews of the literature. That won’t fly, that will get a one, but it doesn’t mean they’re not fascinating, interesting pieces of work.’ Note, again, the tension between the REF evaluation criteria and the discipline-internal value of ‘interestingness’.

In contrast, while the mathematicians we spoke with were aware of the REF as one of the mechanisms by which their work would be evaluated, for the most part these processes evaluated publications which they would be producing in any case. Edward, for instance, working at a teaching-intensive institution, explained that ‘the target is like, I think, we need to publish at least one article until 2020 by our group but we are doing far more than that. We have published two articles, by just me and my colleague, in top ranking journals, this year and three are under review.’ It was very clear to him what the expectations were of the school and research group about his writing – to ‘target more funding bids and publish in high impact journals […] to get our REF rankings better’ – and this made sense to him, both in terms of the institution’s need for secure funding streams, and in terms of his and his colleagues’ career development. For the most part, there was a good fit between the mathematicians’ disciplinary genre regime and that of the institutional systems and the REF.

Proxy values: journal star ratings

The REF star rating system was ubiquitously embedded in the way people talked about their writing goals and the institutional strategies that they were expected to fulfil. This is despite the fact that, in actuality, people cannot know the rating of their papers in advance of decisions of the peer review panel (or, indeed, even subsequent to these – ratings of individual outputs are not released). Proxies are therefore drawn on to infer the potential likely ranking of outputs, including evaluations of the journal in which articles appeared.

These proxies were more significant in some disciplines than in others. In Marketing, all the academics we spoke to in research-intensive institutions were acutely aware of the need to produce enough three- or four-star outputs. They consistently used the vocabulary of ‘three-star’ and ‘four-star’ papers (or even simply ‘threes and fours’) when discussing their aims. However, the star ratings they were talking about referred both to REF evaluations and to categorisations of journals, often at the same time. Journals in Marketing, and the other management disciplines, are rated from one to four stars by the Chartered Association of Business Schools Academic Journal Guide (CABS 2015). These ratings are allocated through a process which combines peer review from a committee of academics in the field, and analysis of journal metrics and citation data. These star ratings have become the dominant way of describing journals in their area, and academics used these designations in our interviews freely. ‘Crème de la crème for consumer research, top four star’, for instance, was the description used by Lisa, an early career academic, of a journal to which she had submitted.

These star rankings are then used in university business schools to encourage academics to aim for publication in three- or four-star journals, for purposes of probation, promotion, and appraisal. Patricia, a probationary early career academic in a research-intensive institution, explained that the department expected her to get her work published ‘in ABS three and four star journals’ to pass probation. For Emma, in a similar situation, the only explicit requirement for probationary success was that she should publish, but she interpreted this in terms of the journal ranking system: ‘I think if I got a one star they’d probably look for something else. […] Or, if I had a two star and a three in review. If I have a three star I’m probably okay.’

There is no necessary correlation between having one’s research published in a four-star journal, and having one’s research rated at four-star by a panel of peers in the REF process. However, ‘producing a [potentially] four-star article for the REF’ and ‘publishing in a journal rated at four stars by the ABS’ became elided. Diane, a senior academic in Marketing at a research-intensive institution, explained how these had become closely coupled: ‘The idea [of the ABS list] was that if you gave people guidance as to where they should publish, everybody would try to go for the best. That got attached to the REF, Research Excellence Framework. Then that got attached to career progression, and that got attached to market prices for academics as well. […] It’s changed the culture of universities profoundly. It’s changed the writing practice.’

In Marketing, the ABS list tends to privilege US Marketing journals which publish large-scale quantitative or experimental research, while much of the UK field orients to a more qualitative tradition. Lisa explained that even the one four-star publication which she can target – ‘quite a good fit, and it’s a nice four’ - tends to privilege work that is more experimental than her own theory-oriented work. Charles described a recent re-ranking of a top UK marketing journal in apocalyptic terms which capture the list’s all-pervasive influence: ‘The only UK marketing journal that’s of real prevalence […] you know what they’ve done? They’ve taken it from a three to a two. […] So it’s like somebody said to me in an email, “Well, that’s the death of marketing in the UK”.’

Marketing academics are therefore faced with a genre regime in which they have to choose between trying to publish in a four-star journal in their discipline, unlikely because of the nature of their work, or aiming for a four-star journal in which they can publish, but in a different field. Diane talked about this as ‘gaming’ the system, explaining that ‘I can’t get four star marketing because I don’t live in America […] I’m not a positivist, I don’t do modelling. I have no way of engaging with that world. […] Now I target management journals, which is one way of hitting a four star, and also management in certain settings. You can get published in top rated medical journals, so it’s even influenced the setting when researching because you’ve got to play the game otherwise you’re nobody. You get trampled on.’ Successfully publishing in a highly starred journal is described here almost as a matter of academic survival.

This gaming strategy can have detrimental effects in terms of networking and establishing oneself clearly in a field. Lisa, for instance, found herself in something of a double-bind: ‘The school agenda is to go for four star but obviously there is a school agenda and then there’s your research agenda and how you build your academic profile […]. It’s whether or not I go for a four star in [refers to the empirical setting of her research] at the expense of going for a three star in marketing that is going to be read by other consumer researchers that I know and go to conferences with and network with. […] They won’t necessarily be reading [refers to journals focusing on her empirical setting].’ Potential four star papers are privileged in her plans, but at the same time she was constantly aware of the ‘fine balance of what is good for the school and what is good for me, and how do I make them align’. Considerations of strategy are at the forefront of Lisa’s decision-making processes about what and where to publish within this genre regime. The purpose of publishing research, in the way it is described here, becomes primarily to score well in the REF and to fulfil personal and institutional strategic objectives.²

Conclusions

Based on interviews with academics in a range of disciplines and institutions, this article has shown the powerful influence of the genre regime associated with the REF within an increasingly managerialist university system. In an attempt to maximise REF scores, institutions introduce systems which attempt to influence, not the content of academics’ research (at least, not directly), but the type of publication they produce and the location of those publications. These regimes are linked to probation, promotion and appraisal, and replace or overlay existing approaches to the evaluation of colleagues’ work with assessment by (informed guesses at) REF star ratings. The REF has become a normalised, though often criticised, part of academic life, introducing the new identity category of being a ‘REF-able academic’ as defining what it is to be an acceptable academic, in research-intensive institutions in particular.

Heads of Department are placed in the position of mediating between institutional requirements and colleagues. They may recognise conflicts between the value system of the REF and of their discipline, or between the competitive REF system and older values of collegiality. But in order for their department to be as successful as possible, they need to encourage individual academics to publish strategically within this genre regime nevertheless.

The REF has a differential impact on academics, depending on how these strategic genre regimes relate to their established practices. Where there is a good fit between the values implicit in the genre regime of the REF and the genre regime of the disciplinary tradition, as with Maths, there is less dissonance. But where, as in History, the genre regime of the disciplinary knowledge tradition values different types of publications from those encouraged by REF strategies, academics are faced with difficult decisions as to which system of values to privilege. In Marketing, the third of our focal disciplines, the hegemonic use of the ABS journal list as a proxy for REF value potentially reshapes what is valued within the discipline, forcing academics to make choices between locating their work in-discipline, but in lower-ranked journals, or out of discipline in order to score more highly. Success within institutional frameworks comes to be in tension with reputational success within their field.

Of course, we worked with a relatively small number of academics across only three disciplines and three institutions, so these findings cannot be seen as necessarily representative of the whole sector. Having said that, though, the research evaluation process applies across the university sector. While the specific interactions between disciplinary and evaluative genre regimes will be different in different disciplines, and indeed in different institutions, we can be confident that nevertheless, the genre regime of the REF has a significant shaping influence on academics’ writing practices, even if the specifics of how this influence plays out are different in different contexts.

The question was raised earlier as to how evaluative accountability systems like the REF, increasingly widespread in universities across Northern Europe and beyond (Morphew, Fumasoli, and Stensaker 2018), are shaping what it is to be an academic, increasing the pressure on researchers to produce the ‘right’ type of publication in the ‘right’ locations (and in the ‘right’ language, Anderson 2013). One of the key dangers of working within this kind of evaluative system is that of losing the intrinsic rationale for the production of research as contributing to knowledge in a field, replacing it with strategic motivations. The data discussed above have shown how important such strategic motivations have become, as academics negotiate conflicting genre regimes and values systems in the choices they make about the type and location of their publications. This is just one example of how managerialist approaches governing aspects of Higher Education, both in England and in other European countries, are having direct effects not just on the experience of universities as workplaces, but also on the processes of knowledge production of research itself.

Acknowledgements

I would like to thank, firstly, our project participants, without whose generous sharing of their time and experiences this work would not have been possible. Secondly, the other members of the project research team – the Senior Research Associates Dr Sharon McCulloch and Dr Ibrar Bhatt, co-investigators Professor David Barton and Professor Mary Hamilton, and administrator Dee Daglish. This work was a team effort and this paper owes a huge debt to their expertise and thoughtful work. The members of the project advisory group engaged with us at several crucial junctures and their support has been invaluable. All those who have attended project events and conference presentations have also helped us a great deal in developing our ideas.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

Additional information

Funding

This work was supported by the Economic and Social Research Council, under grant number ES/L01159X/1. The data that support the findings of this study are available from the UK Data Service for download to users registered with the UK Data Service at http://reshare.ukdataservice.ac.uk/852710/ or https://dx.doi.org/10.5255/UKDA-SN-852710, reference number 10.5255/UKDA-SN-852710.

This work was supported by the Economic and Social Research Council, under grant number ES/L01159X/1. The data that support the findings of this study are available from the UK Data Service for download to users registered with the UK Data Service at or , reference number 10.5255/UKDA-SN-852710.

Notes

1 The project team included myself as Principal Investigator, Professor David Barton and Professor Mary Hamilton as Co-Investigators and Dr Sharon McCulloch and Dr Ibrar Bhatt as Senior Research Associates. This paper represents my own reflection on one aspect of the project research; see also Barton and McCulloch (2018); McCulloch, Tusting and Hamilton (2017); McCulloch (2017); Tusting and Barton (2016) for more information on the project.

2 See Tourish and Willmott (2015) and Mingers and Willmott (2012) for further development of the negative consequences of the primacy of the ABS rankings in the management disciplines.