Opinion: Research and How to Promote It in a University

“While the UK Research Assessment Exercise began as a laudable attempt to cut waste and introduce accountability, it has evolved into a self-justifying ritual by which attempted measures of research have themselves become targets…”

A public body known as the University Grants Committee (UGC), responsible for distributing funds for research to British universities, conducted its first ‘research assessment exercise’ (RAE) in 1986. The declared aim was to evaluate the strengths and weaknesses of research in different fields and different universities, in order to decide where money for research would be most effectively spent. A second UK RAE followed in 1989. In 1990 I left the UK for Scandinavia and so missed RAEs three and four. I also missed a change of hands from the UGC to the Higher Education Funding Council for England (HEFCE), the Scottish Funding Council (SFC), the Higher Education Funding Council for Wales (HEFCW) and the Department for Employment and Learning, Northern Ireland (DEL). For UK university funding, one might say that “nothing makes sense except in the light of devolution!”

On returning to the UK in 2005, I found further unexpected new developments in the art of research assessment. One immediate oddity was that my new host department included some of my pre-2005 publications as part of its own submission to the fifth and last RAE, published in 2008 [101]. Other newly recruited staff members found the same. In one extreme case, an individual whose previous papers were submitted to the RAE was sacked 9 months after the RAE report for alleged failure to publish. The practice of buying in RAE credit has been compared with football teams buying in players. However, a transfer fee secures a player who will play, not one whose past goal average enlarges that of his new team. The next comparable research assessment in the UK, successor to the RAE, will be known as a Research Excellence Framework (REF). The REF is scheduled to report in 2013 [102]. Perhaps it will pay closer attention to where research was actually carried out, and by whom [1].

In the last 5 years, with the UGC–RAE now morphed into HEFCE/SFC/HEFCW/DEL–REF, I have had to learn a new and strange vocabulary of ‘performance indicators’, ‘metrics’, ‘indicators of esteem’, ‘units of assessment’, ‘impact’ and ‘impact factor’. I feel, at times, not exactly like the proverbial Martian, but rather more like someone who has lived on Mars for a decade and a half and returns to find planet Earth changed in puzzling ways. Earthlings certainly seem to speak a new language.

One struggles to see sense behind what is now an obsession with assessing and measuring research. Furthermore, the measurements themselves have changed into objectives, rendering them useless as measurements – in accordance with Goodhart’s law [2]. When children stand on tiptoe, pretending to be taller than they are for comparison of measured height with that of their siblings, they all instantly see the joke. But woe betide anyone who points with amusement to an institutional RAE unit of assessment (UoE) walking on stilts.

Here I ask what we scientists do, how we can tell whether it is being done well or badly, and what the assessors might think about these topics. This article, therefore, considers broad questions of research and research strategy. What is research? How can we promote it? How can we be successful at it? What is required to be effective in using resources for research? Twelve specific questions on this theme are listed in Box 1. My aim in posing them is to consider how answers might suggest a strategy for anyone wishing to promote successful and cost-effective research in a university department. I wish to get away from near-ubiquitous but vacuous abstractions, such as ‘performance indicators’ and unqualified ‘impact’. The spirit of this article is, roughly: ‘what, actually, are we talking about here?‘

This article contains my own views, though none are entirely original. I believe that the recommendations with which I conclude (Box 2) are not at all specific to any one country, since they apply at international, national, institutional and departmental levels.

What is research?

Research is investigation and exploration of possibilities, directed at eliminating those that do not correspond with the real world, thereby achieving increasingly accurate descriptions of reality and explanations of our experience of it. In the natural sciences, observation and experiment are the means by which to decide between competing possibilities – ‘nullius in verba’ [103]. In the arts and humanities, this means of checking possibilities may not be quite as clear-cut; however, certain things happened and did not happen in history, for example, and their status can be decided only by means of evidence. In this broad sense, a good historian is a scientist, too.

What does research require?

Research presumes dissatisfaction with existing descriptions of reality and explanations of our experience of it – it rests on the desire to do better than the current consensus. Research, therefore, requires freedom to question received wisdom and some background knowledge of why we think we know what we think we know. In the natural sciences, the means of checking possibilities, in Popperian terms ‘testing hypotheses’ [3,4], must be within the scope of those who take heterodox viewpoints and who can identify problems in existing knowledge that are otherwise unseen, unacknowledged, or glossed over. My own view is that many current funding policies are very bad at this, and seem to have been devised to obstruct research rather than to promote it. The main obstacle seems to be that observation and experiment usually cost money, as well as time. Money is rarely distributed by ‘peer review’ to those who ask awkward questions, since the questions challenge the peers who review. Real, original research does happen and knowledge advances, but it is often by accident, or as a result of individuals departing from the projects for which their funding was originally awarded.

What is ‘research output‘? Can it be measured? If so, how? What are its ‘metrics‘?

‘Research output’ is new knowledge and understanding and it can be assessed in retrospect. It is usually obvious to all, after a sufficient period of time, that knowledge and understanding have advanced and how. At the time of discovery, however, what counts as ‘research output’ is usually and, perhaps always, by definition, controversial. Research funding supplies the resources with which to settle the controversy by reference to observation and experiment. Funding should be available to multiple, independent investigators, even those working on the same problem. ‘Who made the contribution?’ often remains controversial indefinitely. This is a separate question.

‘Can it be measured?’ No. There is no simple, linear scale for any quantity corresponding to ‘research output’ in the sense used here. What would be its units? What arithmetical operations could be carried out on quantities of knowledge and understanding? Did Crick and Watson double or triple our understanding of the mechanism of heredity? What was the impact of Florey and Chain on medicinal chemistry, expressed as a percentage increase?

‘What are the ‘metrics’ of research output?’ Measurements of something, perhaps confusing the process of measurement itself with specific values obtained in a particular case. ‘Metrics’ is to ‘measurements’ rather as the term ‘levels’ is sometimes used in the absence of chemical understanding, in place of quantity, concentration, or activity, without being clear which. ‘Research metrics’ is pseudo-science in the sense of managing to convey the idea of measurability without being clear about what it is that is being measured.

What is ‘research quality’ as used in the phrase ‘quality-related funding‘?

‘Quality’ normally stands in contrast to ‘quantity‘; there is no ‘quantity of quality’, red, green and blue are qualities of light, intensity (brightness) is quantity of light. The two properties are independent of each other.

However, if we define ‘research’ as the mere attempt to increase knowledge, which may fail, then ‘research quality’ might mean the ratio of effective research to total research. Clearly people wish to spend money effectively, not just dissipate resources.

The phrase ‘quality-related’ (QR) of the 2006 UK Department of Further Education and Science (DFES) pre-RAE consultation document [104] is difficult to understand, except in the latter sense of trying to support productive research at the expense of unproductive research. But who argues for ‘quality-unrelated’ funding?

Will research support be distributed effectively if it rewards previous research output?

The 2006 DFES consultation document [104] repeatedly mentions QR-funding as a ‘reward’. I think this is a very bad idea. Research is rewarded by its success, by acceptance of its output despite free and critical questioning and, for human motivation, perhaps by recognition. Whatever motivates individuals is important but lies outside the scope of decision-making on where to invest resources for research. Otherwise, the reward will be claimed as an end in itself, not as a consequence of research output. There is indeed a mistaken view that the output of research is income from grants. Only a fool with money to burn measures the output of a car by the quantity of fuel it consumes.

Let me add that I am sceptical of the claims made by some research councils, for example the UK Biotechnology and Biological Sciences Research Council, that it supports “excellence and promise”, “timeliness and significance” and, most recently, “excellence with impact”. I find ample evidence of lines of influence between committee members, referees and applicants that may be decisive in many cases, while prohibiting risk-taking and innovation. If this pattern is a general one (and I hope it is not), then the act of supporting universities, departments and individuals in proportion to their success at attracting Research Council funds will be quite counterproductive, simply consolidating existing nodes of influence, usually quite conservative in their outcome. In addition, it must be asked how often the individual or laboratory/group credited with a research output is actually the one that formulated the hypothesis, took the necessary risks and first carried out the critical experiments [1].

What are the costs of research that can be provided in order to support research?

Costs of research include salaries for researchers and technicians, equipment and materials for observation and experimental measurement, as well as running and operational costs. We can also consider costs of publication and dissemination of research outputs. Strong and supportive research departments invariably also provide infrastructure, such as stores, staffing and stock control for consumables and materials in common use.

How can support for research be distributed effectively?

By providing able and motivated researchers with the resources they need to do the work that they believe is necessary in order to advance knowledge and understanding.

How has support for research been provided in the past?

There was once little or no public support for research, which was largely carried out by individuals with independent means (e.g., Charles Darwin), patrons (Tycho Brahe), or entrepreneurial flare (Michael Faraday). Universities, as seats of learning, supported research from their own resources and these were gleaned from many sources. Public support from taxation was introduced, usually with specific objectives in mind, for example development of weapons and defense systems in time of war. In the UK, the Department of Scientific Research nevertheless gave independence to researchers as embodied in ‘The Haldane Principle’ [5,6]. The UK research councils now differ in the extent to which they retain freedom for individual researchers; however, increasingly, public funds are diverted to projects perceived to be of practical importance.

How is support for research provided in different countries?

Here, I can only offer anecdotal evidence based on individual experience. It is always important to consider how things are done elsewhere – none of the questions in this article are unique to the UK, nor to the present time.

• USA: in the USA there is substantial support for independent research at the institutional level. There are both publicly supported state universities and wholly independent universities, all of whose members are eligible to apply to public research funding organisations (e.g., the National Science Foundation (NSF), National Institute of Health, Department of Energy, US Department of Agriculture, National Aeronautics and Space Administration) plus private foundations (e.g., the Howard Hughes Medical Institute) and research institutes. The US public purse seems able to stretch to curiosity-driven exploration, apparently as a matter, at least partly, of national self-esteem. The NSF, for example, since the 1950s, has been “tasked with keeping the United States at the leading edge of discovery in a wide range of scientific areas, from astronomy to geology to zoology. NSF’s job is to determine where the frontiers are, identify the leading US pioneers in these fields and provide money and equipment to help them continue” [105].

• Germany: as in the USA, there can be substantial support of independent research at institutional level, plus the autonomous ‘Lehrstuhl’ – professorial chairs with ongoing associated research, staffing and support costs. There is a strong public research council, Deutsche Forschungs Gemeinschaft and the independent Max-Planck-Gesellschaft with its own specialist and admirable Max Planck Institutes.

• Sweden: Sweden seemed to abandon the German model of strong institutional support for research around 1998, as the number of universities increased dramatically and the older, larger, research-led universities came to be regarded as elitist. In Lund, I personally heard the new era ushered in with the words “We are adopting the American model. If you want to do research in this university then you must pay for it yourself”.

• France: it seems that France has little specific university research funding. Research resources tend to be separated from higher education and are provided largely through various public agencies, notably including the Centre Nationale de la Recherche Scientifique, to its own institutes.

• Europe as a whole: pan-European funding has taken the form of a succession of ‘framework programs’ (FP), to which EU member states, and some affiliated non-EU countries, contribute and to which individuals and research teams within them may apply. We are now up to FP 7. Usually the FPs have explicitly been connected with political, strategic and economic objectives and eligibility depends on the applicants’ statement of how they will address these. More recently, the European Research Council (ERC), now in its second annual cycle of funding, has declared objectives more akin to those of the US National Science Foundation. The ERC, established in 2007, aims “to stimulate scientific excellence by supporting and encouraging the very best, truly creative scientists, scholars and engineers to be adventurous and take risks in their research, to go beyond established frontiers of knowledge and the boundaries of disciplines … no thematic or policy-driven research priorities need to be covered. Scientific excellence is the sole evaluation criterion” [106].

• The world: there seem to be few funding organizations with eligibility criteria that are blind to nationality. The Human Frontiers Science Program [107] admits co-applicants from anywhere, but principal applicants must be from participating countries. Private foundations, such as the Gates Foundation [108], may be an exception to this rule.

What are the examples of effective research support?

One of the world’s most effective research laboratories received its 14th Nobel Prize – for Chemistry – in 2009. This is the Medical Research Council Laboratory of Molecular Biology, Cambridge, UK. In the USA, the Lawrence Berkeley Laboratory, Berkeley, California, with two Nobel prizes, must also come high in a ranking based on number of Nobel prizes. These centers of true excellence are affiliated to universities, but are not university departments. In Europe we have such institutions as Centre d‘Etudes pour Recherches Nucleaires and the European Molecular Biology Laboratory.

What is impact?

Hirsch’s “index to quantify an individual’s scientific research output” [7] has quickly been adopted as guide to individuals whose research has had important impact, at least on the literature. Whether the ‘h-index’ can be used to produce an institutional ‘h-value’ is unclear and we should surely guard against the growth of yet another transfer market. The h-index seemed, at first, to be relatively immune to Goodhardt’s Law, since there is not much anyone can do to change it. However, ‘h’ may yet be a measure that ingenuity can convert into a target [8].

Is there a future for dual support?

Support from central sources to institutions

Universities are places where research is carried out and knowledge is both advanced (research) and disseminated (teaching). There is a productive interaction (synergy) between these activities.

This means that funding for research should be provided and will contribute to teaching quality rather than detracting from it. Research is not a diversion from a university’s core activities and funds should be as independent as possible from the peer review and political control that characterize the priorities of the research councils. The tendency towards internal, pre-emptive peer review must be resisted.

Support from research councils & other funding agencies to specific fields, projects & individuals

Public funding for projects through research councils has evolved from direction of resources to specific objectives. These were usually framed with specific applications (e.g., military, medical, agricultural and engineering) in mind. Accountability will probably mean that research council support will continue to be justified in this way. Good basic research still gets done under these headings.

External support was originally intended as a directed ‘top-up’ of a strong baseline of independent research taking place in laboratories supported by central sources. University research must not become ‘risk-averse’, as research council-funded projects have become [5,9].

Conclusion

Knowledge and understanding of the natural world will continue to grow. The geographical location from which it will grow most in the 21st Century is unclear. The burst of new learning that drove and drew strength from the industrial revolution can be attributed to the values of the Enlightenment – to the conviction that the individual may decide for him-, or herself, on the basis of evidence, not on the word of authority. Box 2 contains a translation of these ideals into specific recommendations for a research-led department in a modern university.

A pre-requisite for discovery and innovation is dissent. Freedom is not a luxury, but a necessity – freedom to explore, to think beyond orthodox opinion and to do the apparently crazy experiment that peer review will almost always judge to be a waste of time [5].

While the UK RAE began as a laudable attempt to cut waste and introduce accountability, it seems to have evolved into a self-justifying ritual by which attempted measures of research have themselves become targets – a perfect example of Goodhart’s law. It is as if nothing lies beyond grant income, publications and some vague notion of short-term social impact.

We must return to the values that gave rise to the growth of science and technology [10,11]. If we do not, then future prosperity and an improved quality of life will surely pass to those who embrace these values, even if they do so for the first time.

Box 1. Questions that might usefully be addressed in devising a research strategy.

• What is meant by ‘research‘?

• What does research require?

• What is ‘research output‘? Can it be measured? If so, how? What are its ‘metrics‘?

• What is ‘research quality’ as used in the current phrase ‘quality-related funding‘?

• Will research support be distributed effectively if it rewards previous research output?

• What are the costs that can be provided in order to support research?

• How can support for research be distributed most effectively?

• How has support for research been provided in the past?

• How is support for research provided in different countries?

• What are examples of effective and ineffective research support? Can we learn from these?

• What sort of support can institutions provide for themselves?

• What sort of support can research councils and other external funding agencies provide to specific fields, projects and individuals?

Box 2. Specific recommendations for a university department.

• Invest what you can afford in the talent you already have.

• When appointing, select able and motivated staff.

• Trust individuals to decide what they will work on. No motivated scientist will willingly waste his time and resources.

• Do not trade on recognition, authorship and the promise of future investment. These are the real rewards of research. Individuals must see, rightly, that they can and will obtain these for themselves – and for one another in collaborative work.

• Provide a supportive infrastructure. For example: time, encouragement, and easy access to laboratory equipment, materials and services.

• Promote the view that research success is to the benefit of everyone in your department. Knowledge is an inclusive good, not an exclusive good. No one is in internal competition with anyone else – the is no limit to the things we do not yet understand. Celebrate success in research – it is everyone’s success. Put your department on the map of world science.

• Raise confidence and raise aspirations.

Financial & competing interests disclosure

The author has no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

No writing assistance was utilized in the production of this manuscript.

Additional information

Funding

The author has no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.