The new impact factor has arrived. Who cares?

Last June, the European Journal of General Practice (EJGP) was proud to announce that its impact factor rose to 1.217, an increase of no less than 50%. This results in a steep climb in its rank of primary care journals from position 16/18 to position 11/19. This is good news, as the European Journal of General Practice may expect that a higher impact factor will attract better papers, will increase the status of the journal and will improve its scientific credibility internationally (1). It makes this journal a more attractive option to be considered by authors for whom the impact factor is an important element in their choice of a journal to which they submit their work. Finally yet importantly, it will please the publisher of the European Journal of General Practice, who will eagerly use this milestone in its public relations and marketing campaigns. Yet, this event arouses ambiguous feelings.

Judged by the level of anxiety with which editors and authors look out for this magic number every year, the journal impact factor (JIF) still firmly stands on its high pedestal, in spite of the critiques it has received over the past decades (2). The JIF is based on the assumption that ‘scientific impact’ can be measured as a function of the citations that a publication receives. Hitherto, journals publishing (some) articles that are much cited are supposed to have more ‘impact.’ Yet, the JIF is nothing more than a simple ratio indicating how often papers in a particular journal have been cited in a selected group of journals during a limited period.

The word ‘selected’ is particularly meaningful here and points to an important limitation: only the journals indexed in the Science Citation Index compiled by the private company Thomson Reuters are considered. They represent only a fraction of the total scientific output. In this database, ‘primary healthcare’ represents a small subject category of 19 journals, including one journal on sports medicine. A quick search in Medline—a universally used index of medical journals maintained by an American governmental organization—shows that there are many more journals with either ‘primary care’ or ‘family medicine’ or ‘general practice’ in their name. However, citations in these journals will not affect the JIF; neither will citations in non-primary care journals that are not indexed in the Science Citation Index.

Then, let us have a closer look at the calculation of the JIF. The 2014 JIF of the European Journal of General Practice equals the number of citations in 2014 of papers it published in 2012 and 2013 divided by the number of articles it published in 2012 and 2013. In the denominator, Thomson Reuters counts the number of ‘citable items’ i.e. ‘a collection of items that are likely to influence the scholarly literature’ (original papers, reviews, and proceedings). The company uses a set of criteria to establish ‘citability’ (e.g. presence of a summary, the number of citations, length of the paper etc.) (3). Regardless of whether these criteria effectively select the influential papers, it is obvious that the mere number of papers that are selected is crucial. Variability or lack of clarity in the selection criteria can have a substantial impact on the outcome of the calculation. By contrast, the numerator is much less selective and includes ‘any citation to the journal’, including letters-to-the-editor and editorials (4).

One can easily see how this number can be influenced by authors and editors (5). For instance, if we would cite two papers of the European Journal of General Practice from the previous year in this editorial, it influences the next impact factor: the numerator would increase with two citations while the denominator stays unchanged because an editorial is not counted as ‘citable’. Furthermore, a few highly cited papers can substantially increase the numerator and thus the JIF: if the European Journal of General Practice had published one paper in 2012 and one in 2013, each cited 100 times in 2014, our JIF would now be 4.6 with the same level of citations for the 58 other papers in the two publication years.

Finally, the formula ties the ‘citable period’ down to two years, favouring fast moving research fields such as molecular biology but impairing areas such as clinical medicine and epidemiology, which generally publish data from long-term studies (1). Therefore, JIFs of various subject categories cannot easily be compared.

Thomson Reuters meets some of these critiques by providing additional citation data in the Journal Citation Reports (JCR), including the JIF without self-citations and the five-year impact factor. For example, the European Journal of General Practice 2014 JIF without self-cites is 1.150, and its five-year JIF is 1.169. They also provide data comparing and summarizing all journals in a particular subject category. For instance, the aggregate impact factor of the 19 journals in the category ‘primary health care’ is 1.682, and the median impact factor is 1.295.

Besides the formula, the characteristics of the journal itself influence the JIF. Many journals with a high JIF are American. The British Journal of General Practice had to hand over its lead in the JIF ranking to the American Annals of Family Medicine very soon after the launch of that journal. Also, journals in languages other than English, in general, have lower JIFs. High impact journals also cover the ‘right’ research field: the highest impact factor for family medicine is 5.4, for oncology it is 115.8, a 21-fold difference. Furthermore, they favour to publish reviews, which are cited more often, but by nature seldom cover pioneering research. Finally, journals covering basic science have an advantage over journals covering clinical science: clinical papers will often need to cite fundamental research, but biomedical research articles will less readily cite clinical papers (2).

Therefore, the JIF is far from perfect as an indicator of a journal's impact on science. It probably owes its popularity to the fact that it is just one simple number with an appealing name. Its simplicity may easily deceive us. Moreover, ‘impact’ has many other dimensions than scientific. The ultimate aim of primary care research is to improve outcomes for people living in the community, so why not also consider ‘societal impact.’ More recently this dimension has gained some momentum modelled on the Research Excellence Framework (http://www.ref.ac.uk/) in the UK or the EU's U-Multirank (http://www.beltanenetwork.org/event/impact-of-science-conference/).

The limitations of the JIF would in fact not matter too much if it were just a competitive game among journals motivating authors and editors to do their best. However, despite the drive for innovation in this field, university managers and governmental (funding) bodies still judge researchers and research groups on the impact factor of the journals in which they publish. This modern use of the impact factor is—we cite ‘Nature,’ a very high impact journal—no less than a ‘mortal sin‘ (6). Recently, pleas for alternative ways of assessing the quality of research have been launched, such as the Leiden Manifesto (7).

All this being said, a higher impact factor is good news for any journal because—whichever way you look at it—it means that more researchers are using the journal and find it worthwhile to cite its articles. This is a compliment to the authors and reviewers and everyone who has made this possible. However, in the meantime we all should be aware that this crude and easily countable measure—that may be different next year—is not a comprehensive measure of the ‘real’ impact of a journal. A journal's ambition should go beyond being cited. It should aim to have an impact on clinical practice, health policy and ultimately on society.

The articles in this issue are a fine illustration of the different kinds of possible impacts: e.g. there is a scientific analysis of the diagnostic value of a laboratory test (Willemsen et al.), two studies concerning common clinical problems (Leclercq et al.; Nijrolder et al.), a study on education (Vrdoljak et al.) and finally a review on the quality of healthcare for people with multimorbidity, which concerns healthcare policy (Ricci-Cabello et al.).

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.