Does relevance matter in academic policy research? A comment on Dredge

This section of the journal encourages discussion between several authors on a policy-related topic. The same question may, therefore, be addressed from different theoretical, cultural or spatial perspectives. Dialogues may be applied or highly abstract. The Dialogue in this issue starts with Dianne Dredge's contribution here: http://dx.doi.org/10.1080/19407963.2014.990661

It is often said that simple questions do not always yield simple answers. Does relevance matter in policy research? The question is value-laden, with strong advocates on each side of the argument. Answering it with a resounding ‘yes’ implies to some that academic researchers will relinquish their academic freedom; focus on issues as directed by authorities and businesses; and submit to a neo-liberal approach to education. Answering it with a resounding ‘no’ may be met with the criticism that ‘academics seem all too eager and able to block changes that might lead to universities better addressing the needs of society’ (Dearlove, 2002, p. 257). This discussion will draw on Dredge's contribution, and on relevant literature in management science, to propose an answer of ‘yes, but’.

Academic research has been the subject of extensive criticism, not only from ‘an increasingly vocal section of the media’, as Dredge indicates, but also from within its own ranks. Thomas and Tymon (1982) highlight six common shortcomings of academic research: (i) the common sense nature of findings, (ii) the descriptive nature of research with insufficient focus on action, (iii) the focus on methodological rigour – internal validity – over external validity, (iv) reductionism at the expense of complexity, (v) an over-emphasis on statistical techniques and (vi) the inappropriateness of epistemological assumptions. Starkey and Madan (2001) add that many users do not believe that academic research can benefit them, or that academic research focuses on issues of relevance to them. Moreover, user communities lack awareness of the research. This would imply that although academic research is highly credible (perceived quality of research information in terms of whether it can be believed), it is not generally very usable (the potential or probability for a body of knowledge to be employed (Xiao & Smith, 2006)).

Dredge highlights some of the contextual challenges academics face, which may affect the perception of their work as being irrelevant. A first is the definition of ‘relevance’: it is true that there is no universally accepted definition of this concept. A second is the way in which relevance is framed in universities: very often, the relevance and quality of research is measured by the quantity of publications in ranked journals, which are likely to have a niche readership. A third is the challenge of communication: the recommendations that are outlined by Dredge are indeed very valuable in communicating our research more effectively to different audiences.

The question however remains if better communication goes far enough to address the ‘relevance gap’. Dredge argues that ‘if we cannot define what is relevant academic policy research, then the question of whether or not it matters becomes redundant’; and that ‘relevance doesn't matter, communication does’. But is all academic research by definition relevant, just by virtue of being produced by academics? I would like to argue that although we cannot define ‘relevance’ precisely, there is some indication in the literature on what may constitute ‘relevant research’, and that the term ‘relevant’ does not need to equal an inevitable limitation on academic freedom. Drawing on the management literature, the question should be raised if indeed all we need is to communicate better – is our research just ‘lost in translation’? Or, as Shapiro, Kirjman, and Courtey (2007) argue, is some of it ‘lost before translation’?

The rigour–relevance gap

The question of research relevance encompasses both what knowledge is produced, and how it is produced. Gibbons et al. (1994, quoted in Starkey & Madan, 2001; Thomas, 2012) differentiate between Mode 1 (M1 k) and Mode 2 knowledge (M2 k). M1 k is what we traditionally see as the scientific approach to knowledge creation, with a more theoretical and global focus. M2 k is more concerned with knowledge as it works in practice, and less with disciplinary conventions. It involves practitioners more, focuses on problems in a specific and localised context, and is faster in its dissemination. Starkey and Madan (2001, p. S5) provocatively argue that ‘the Mode 1 approach to research and knowledge production is no longer sustainable. Universities are the last bastion of M1 k in a world where greater accountability and the speed of change in relevant knowledge encourage an M2 k approach’. They argue that although universities produce research that is rigorous, it is not sufficiently relevant for our time.¹

So can this rigour–relevance gap be bridged? Thomas and Tymon (1982) have argued it can, and draw attention to the four stages of scientific enquiry. These are: the perception of a real-world problem situation, broad conceptualization of phenomena, formulation of a scientific model (theory) and the derivation of implications or solutions from the scientific model. Pathologies occur, they argue, when researchers ‘omit the real-world problem from the research cycle, so that the practical relevance or utility of a theory becomes irrelevant in its development and testing’ (p. 346). Perhaps one could add ‘irrelevant in its publication’ here: in an academic context where the number of publications in ranked journals is so important, and where many journals value rigour over practical usability, does it become indeed more appealing to pay less attention to the first step, than to the following three? If relevance of research is, at least at the origin, tied to its focus, then overcoming the gap is more than a question of communication. Indeed, as Hodgkinson and Rousseau (2009, p. 543) argue:

The research-practice gap is due to more than language and style differences between science and practice. The categories scientists and practitioners use to describe the things they focus on are very different. Translation will not be enough to bridge the gap between research findings and potential end users.

The question of what we choose to research is thus important for how relevant our research can be, and to whom. Bartunek (2007, p. 1326) argues that:

The typical way that we as academics are accustomed to having an impact in scholarly writing is through the strength of our logic and our data: we find gaps of some kind, convince others of their importance, then attempt to fill them with our work. Although this works for academic scholarship, it is not all there is to having an impact, and it is not likely to appeal to practitioners, who are not particularly interested, or aided by, filling scholarly holes.

The pressure to ‘publish or perish’ impacts many academics in their choice of research topics. Gulati (2007, p. 778) argues that researchers are pursuing ‘narrowly defined endeavours for highly circumscribed audiences’, often ‘finding themselves essentially saying the same things in slightly different ways, rather than generating original ideas’ – could this be because this strategy is a less risky route to publication in established journals? If that is indeed the case, it could be seen as ironic that a system created to make universities more relevant (by increasing their published outputs) has stifled researchers. Are we indeed more relevant because we are more prolific, or are we mainly publishing more of the same? If the latter is true, then the matter of relevance goes beyond communication – then a more fundamental assessment of our social role as academics is in order.

The question of academic freedom

If not only communicating our research, but also our choice of research topic is important, then the next question is how this affects our freedom to choose our research focus. Academic freedom, defined as the freedom of the professor to teach without external control, and the freedom of the student to learn (Altbach, 2001), is often also applied to research, and the freedom of researchers to pursue their own interests. Academics are historically seen as ‘valuable social critics’ (Altbach, 2001, p. 207) – contemporary interpretations of the relevance of their work, however, are often in the very narrow sense of practical application (Grey, 2001, p. S29). Yet ‘starting from a real-world problem situation’ need not equal researchers being limited to justifying current policies or the advancement of (economic) interests, or producing research that fits very narrowly defined criteria of usability. On the contrary, pursuing relevance as social critics may push researchers towards less well-trodden grounds, or across disciplinary boundaries – towards research areas that may find it harder to find a space in the established journals, with their strict disciplinary gate-keepers.

As Grey (2001, p. S29) argues, universities can offer a context where ‘ideas can be tried, developed and abandoned [ … ], since we cannot know in advance what kinds of knowledge will be useful, and which will not be'. They can offer a freedom to experiment that can allow researchers to transcend current beliefs and assumptions. This critical attitude is vital in academic research and a key component of its value – yet it cannot be used as an excuse to dismiss the issue of societal relevance altogether, and to focus solely on ‘filling scholarly holes’.

Grey (2001, p. S30) therefore argues for a ‘critical distance from relevance to industry [or policy makers]’, however not for ‘disengagement with practice’: the terms of engagement need to be defined. ‘Universities have generally had ambivalent relations with their surrounding societies: both involved and withdrawn, both serving and criticising; both needing and being needed’ (Berdahl, 1990, p. 170). So it is, perhaps, with academic researchers: although not all our research may be immediately usable, it can be relevant, if we keep a critical distance from society, but we do not turn away from it into the ivory tower (even if that is often where the glory of publication awaits).

Why relevance matters

So does relevance matter? I would argue it does. It matters because existing incentive structures around us make it matter. And it also matters if we take our social duty as researchers seriously.

The first argument refers to the fact that a call for greater relevance affects many of us in our professional careers. Definitions of relevant research affect funding sources; impact case studies that may need to be prepared for REF; and considerations of promotion and tenure. It is clear that many funding sources will prefer studies that are seen as relevant in the narrow sense, often inter-disciplinary and with short-term practical impacts – mirroring M2k.

Simultaneously, as this paper has argued, many researchers are under pressure to publish – as much as possible, and in journals where competition is fierce and the pearly gates to publication glory are closely guarded. Often here, it is safer to build on existing discourses, use specific methodologies and stay within the disciplinary boundaries, without a strong focus on usability – mirroring M1k.

The tension between demands is obvious, and balancing both challenging. There are research topics and projects that lend themselves well to the production of M1 and M2 knowledge, and some researchers choose to limit their research interests to these topics because of the incentive system that exists around them, and their generally accepted level of ‘relevance’. Other researchers may choose topics that may not be relevant in this sense, and are not necessarily equally suited to both types of knowledge creation, regardless of how well the research is communicated. They value the creative control over their research topic over potentially greater opportunities for publication and funding. Perhaps it is in this choice, and facing the pressures Dredge has outlined earlier, that we make our academic freedom felt most acutely: to be (relevant) or not to be (relevant).

Besides existing incentive structures, however, there is also the more fundamental question of which role we, as academic researchers, choose to play in society. Research with limited applicability, and of interest only to a very small group of researchers in the same niche, cannot be argued to be equally relevant as research that has a wider applicability and is of interest to a wider group of readers and users. To argue that this schism does not exist, because the definition of relevance is debated, screams of academic arrogance.

Not all academic research that is produced is equally relevant. In the face of the criticism we are already confronted with, that is unpleasant to admit – an inconvenient truth. Yet if we present ourselves as evaluators and validators of knowledge in society, then it is our duty not to try and conceal it. Although academic freedom should allow us to explore societal issues from new and innovative perspectives, and to go beyond concerns of immediate usability, I feel that it is our duty to look beyond the ‘scholarly holes’ mentioned earlier, and to reflect carefully on what Thomas and Tymon (1982) have designated as the first – and most fundamental – step of the research process: the perception of a real-world problem situation. After that, improved communication of our findings, as argued by Dredge earlier, also has to play a role. Although relevance may not be the only criterion to judge research, we ignore it at our peril if we have a role to play beyond the ivory tower.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1. Responses to this paper include Kieser and Leiner (2009), who argue that the rigour–relevance gap is unbridgeable, and Gulati (2007) who claims that the division between rigour and relevance is artificial.