Impact assessment publishing – observations and reflections after 7 years of being editor of impact assessment and project appraisal

Introduction

This is my last issue – a special issue on ‘environmental/impact assessment simplification’; subsequently introduced by Fischer et al. (2023) – as editor of Impact Assessment and Project Appraisal (IAPA) after seven years, 39 issues (of which 11 issues with a special focus¹) and 351 published items (including research and professional practice papers, letters, book reviews and editorials). Prior to my time at IAPA, I was editor of the Journal of Environmental Assessment Policy and Management (JEAPM), also for seven years, which means I have continuously edited – what used to be – two of the three main impact assessment (IA) journals (the third being Environmental Impact Assessment Review – EIAR) for a total of 14 years. In this, my last editorial, I will make some personal observations and provide for some reflections on IA publishing and editing, looking at past and present developments, in particular with regards to where IAPA is currently standing.

Impact assessment? What impact assessment?

Before elaborating further, there is a need to establish what IA stands for in the context of IAPA and the community using the journal as a key resource, as the term is currently used in different ways. When doing a scopus² search with ‘impact.assessment’ as the search term for publications’ keywords, the result is about 52,000 hits for the end of February 2023. Whilst up to 1982, scopus lists just 89 documents, with 25 appearing for the year itself, that annual figure stood at just over 3,000 for 2022.

So, are these 52,000 hits overall and the over 3,000 hits for the year 2022 all associated with the IA community represented by IAPA? Not quite, even if IA as an inherently inter- and transdisciplinary subject draws on knowledge generated by a wide range of disciplines. In order to attempt to answer the question of how many papers are associated with the IA community which IAPA is serving, I will subsequently first look at papers associated with ‘environmental impact assessment – EIA’, the IA area with the greatest number of associated publications. Doing a corresponding scopus search, this results in just short of 36,000 hits (in this context, it is important to note that papers dealing with other IA tools also frequently include EIA as a keyword). The largest proportion of these are associated with publications coming from the various strands of the environmental sciences discipline(s), exploring, e.g. interactions between, and associated impacts on, living and non-living environments. In this context, Fischer et al. (2015) estimated that over 90% of all academic professional publications that appear under the search term EIA are connected with that community. Whilst these papers provide the IA community represented by IAPA with important baseline material for their work, their content would only rarely be considered the core daily ‘diet’ for most of those working with the participatory, ex-ante decision support IA tool associated with the 1969 National Environmental Policy Act (NEPA) in the US, the 1985 European EIA Directive and similar legislation from other countries.³ Following on from Fischer et al. (2015) it is safe to state that less than 10% of the 36,000 hits are connected with outputs that have an explicit connection with the participatory decision support tool EIA, which is representative of the community publishing in IAPA.

Other IA tools that have similar remits to EIA include strategic environmental assessment (SEA) and social impact assessment (SIA). For them, scopus searches done at the end of February 2023 resulted in roughly 1,300 hits (of which about 80% associated with IAPA’s IA community) and 1,100 hits (of which about a third associated with that community), respectively. Next to EIA, SEA and SIA, health impact assessment (HIA) and sustainability assessment (SA) represent other widely applied and researched IA tools. The search terms health.impact.assessment and sustainability.assessment resulted in just short of 8,400 and 3,900 hits, respectively. Scanning through the results, though, it is obvious that, again, only a small proportion (less than 5%) of the outputs are connected with anticipatory and participatory decision support assessments, i.e. the type of assessments the IAPA readership is mainly interested in. The term HIA is often used in medical studies when depicting health impacts of, e.g., personal activities or consumption. Furthermore, SIA is a frequent keyword in studies examining impacts of various societal processes. Overall, there are probably up to about 400 HIA and 200 SA publications that can be associated with the IA community represented by IAPA.

Finally, considering that papers usually include around five keywords, some of the hits for EIA, SEA, SIA, HIA and SA are likely to represent more than one count. A safe estimate may therefore be that a maximum of about 8% of the 52,000 IA papers identified through a scopus search represent the IA community IAPA is serving. Translating this into an annual figure results in an estimate of up to about 200 IA English language publications each year (of which around 50 being published in IAPA) over the past decade (i.e. since 2013; with a steady trend over time).

Past and current relevance of IAPA for the IA community

Whilst IA (as defined above) relevant publications have been and are being disseminated through a wide range of outlets, for around 20 years (between roughly 1998/1999 to 2018/2019), three English language journals were seen as the main publishing platforms for the international IA community. These IA dedicated journals include EIAR, IAPA and JEAPM (Fischer and Onyango 2012; Fischer et al. 2015). A scopus search of IA until 2019 shows that next to these three key outlets, a few others journals were also regularly used for publishing IA papers, such as Environmental Science and Policy, Journal of Environmental Planning and Management and Land Use Policy. A couple of other outlets appearing in a top 10 list of journals that have the most hits are considered to be predatory and are therefore associated with scientific non-trustworthiness. Whilst these have been attracting and continue to attract papers, I will not name them here (see Fischer 2022 and further below).

Importantly, IAPA has been the journal of the International Association for Impact Assessment (IAIA) since 1998 (following on from the Bulletin, founded the same year that IAIA was inaugurated in 1981 and Impact Assessment in 1993). There are usually around 1,000 IAIA members formally registered on average each year (membership fees are charged on an annual basis) who have free access to the journal. Furthermore, and importantly, 17 affiliates throughout the world (with several 1,000s more members⁴) also have access to IAPA.

Reflecting further on scopus hits, it is somewhat surprising to find that at the end of February 2023, IAPA was the only remaining international English language peer reviewed journal fully dedicated to IA, managing to maintain a global spread of authors and themes, with about one in four authors coming from various European countries and one in six coming from each, North America, South America, Africa and Asia, and another one in 14 coming from Australia and New Zealand. Reflecting on EIAR, a scopus search shows that since 2019, that journal has experienced substantial growth of published items from around 70–90 a year to 193 in 2022. Over the same period, a reduction of directly IA relevant publications is also observed. Scanning through all 2022 publications results in the identification of about 30 papers in which a direct link with IA (as defined earlier) is made (i.e. 15%). Furthermore, a geographical focus has emerged, with about half of all papers in 2022 being associated with authors from the People’s Republic of China. Finally, with regards to JEAPM, the number of items published has roughly halved between 2018/2019 and 2022. Here, of the 16 items published in 2022, only one can still be said to be distinctly IA relevant. A geographical trend can also be observed, with half the papers being associated with authors from Central and Southern Asian countries.

Personal observations and reflections

Following on from the above, it is clear that IAPA plays an important role for the international IA community interested in the anticipatory and participatory decision support tool IA. In 2023, it was arguably the only globally active, peer reviewed English language journal which had remained fully focused on IA (as defined above). This focus is associated with a conviction of the editorial board and the editor that the ‘value of a journal shows itself in meeting the needs of the community it is serving’ (Fischer 2022). Subsequently, I will put forward some personal observations and reflections which, based on my experiences as an editor, I believe are of particular interest to the IA community, other editors and also publishers. These are organised under five headings: fashions and trends, authors, reviewers, publishers, and IA as a discipline.

Fashions and trends

The emergence of fashionable topics is a routine occurrence in the publishing world. At times, fashions signify trends that are associated with a rethinking of conceptual ideas and the rewriting of paradigms (e.g. in IA from ‘rational’ to ‘post-modern’ to ‘post-factual’ to ‘post post-factual’; see Fischer and González 2021). It is important for academic journal editors to encourage critical debate on fashions and trends, for example, through special issues. Peer-reviewed journals should help to understand upcoming fashions and trends better. Part of this is to, e.g., uncover when topics that are advertised as ‘new’ are anything but (which is arguably the case many a time). For example, on more than one occasion over the past two decades have suggestions been made that because of perceptions that IAs are not effective we need something radically different (see, e.g. Caratti et al. 2004). However, resulting proposals for reform frequently look remarkably similar to what we already have in place with IA.

Importantly, from an editor’s perspective, fashions can come with a flood of very similar papers formulating very similar ideas. In this context, it is an important task of an editor to ensure that an acceptable quantity of papers is finding its way into a journal. Also, balanced debate should be encouraged if ideas are contested or seen as controversial. In addition, it is important to catch out ‘questionable’ fashions, i.e. topics of which the underlying purpose is unclear or where flaws are detectable. An example is ‘bibliometric analyses’. These can provide for some useful insights, but they need to be approached very cautiously. For example, doing a bibliometric analysis of the – above mentioned – over 52,000 scopus hits of IA without clearly distinguishing between the different fields they represent and stand for (i.e. without clearly explaining the context) can easily result in a meaningless exercise, causing confusion rather than clarification.

Trends can also be connected with particular world regions. For example, they can arise from a need of reconciliation between different population groups in a country (or countries). Whilst there is no reason why important issues with a particular regional relevance cannot be of interest to a global audience, explanations are likely needed by the authors that go beyond what would be required if the same paper was to be published in a regional context. In this context, an editor has an important role to play in making sure this is happening.

Authors

Journals exist only because of the authors that publish in them and during my time as editor I have been dealing with many skilled authors to whom I am grateful for choosing to publish in IAPA. About 30% of all papers submitted to IAPA are published, with about half of the submissions being rejected straight away, at times because of an unacceptable quality, but more often than that due to a paper not being of any obvious interest to IAPA’s readership. In this context, it is important that an editor should always apply a balanced and rigorous screening process of submitted papers, not only to ensure that a journal’s identity is kept intact (according to the journal’s aims and scope), but also to avoid other authors from submitting unsuitable papers in the first place. Word spreads quickly when a journal allows literally every submitted paper into the review process. Whilst this tends to lead to attracting greater numbers of submissions, it wastes valuable reviewers’, editors’ and ultimately also authors’ time and effort and is not beneficial to the readership, either.

Another observation to be made under the heading ‘authors’ is that at times authors have what can be called a ‘tunnel vision’ with regards to disciplinary silos, the use of particular terminology and also concerning geographical outlook. When publishing in an international journal, it is important that authors are clear about specific contexts, explaining diverging terminology used in different world regions, showing an awareness of different words being used for the same or very similar concepts and/or practices and also to interpret results in the light of the wider international debate. Reviewers and editors should make authors aware of any possible associated issues and ambiguities.

Finally, in order to encourage authors to submit topical items, an editor should actively encourage calls for special issues and for submissions to particular themes. In this context, contributions to special issues tend to get a wider readership and often also lead to more citations (for more on ‘citations’ and associated metrics see below in ‘Publishers’).

Reviewers

Reviewers play a key role in ensuring the functioning of peer reviewed journals. They are important guarantors of quality. Having ‘good’ reviewers is a key treasure of any editor/journal. Good reviewers are those who agree to conduct reviews based on their knowledge and/or experiences, who do reviews within acceptable time frames and who provide for constructive, fair and valuable feedback to authors. In order for the peer review system to function, anyone publishing should be willing to review papers (at least two per own published paper, preferably in journals that someone has been publishing in). I have come across many excellent reviewers to whom, as an editor I am very grateful. However, I have also come across various ‘see no evil, hear no evil, speak to evil’ types; including those who do not react to invitations to review, others who decline every invitation, and also some who agree, but then do not complete the review. Whilst editors fully appreciate that at times it simply is not possible to do a review, overall for those who publish ‘being busy’ should not and cannot mean ‘never’. In this context, I would like to underline that securing good reviewers has become one of the main challenges for journal editors.

Publishers

IAPA is lucky to have Taylor and Francis as its publisher, as it is in full support of efforts for ensuring high quality and integrity of the journal (Fischer 2022). However, in the wider world of academic professional publishing there have been some worrying developments over the past 20 years, in particular with regards to what is commonly referred to as ‘predatory publishing’ practices. In a nutshell, predatory publishing is based on an exploitative publishing model, where the author pays for publication and where either no or only inadequate checks are in place for quality control and for academic and disciplinary merit (a list of 25 criteria for identifying predatory publishing was prepared by Eriksson and Helgesson 2016).⁵ One sign of predatory publishing is, e.g., the absence of a main editor and the allocation of 100s of special issues to unfettered ‘guest’ editors. Somewhat unsettlingly, there are perturbing signs that even a few of the well-established large publishers are not immune to various practices associated with predatory publishing, for example, proliferation of published items, in particular items that do not seem to be directly relevant with regards to aims and scope of a journal. In this context, the current period of predatory practices has been accompanied by an exponential growth of publications, with about two and a half Million academic papers being published in 2000 and over 7 million just 15 years later (Fire and Guestrin 2019).

Publishing models that are based on outsourcing key services such as editing will invariably be connected with poor quality, even if occasionally good outputs are possible. Importantly, repeatedly publishing (and many unknowingly do so once or twice) in what are considered predatory outlets is likely going to affect the reputation of an author and ultimately it is a waste of time and effort.

An important reason for predatory publishers successfully attracting papers is connected with the criteria routinely used in academic institutions when considering cases for promotion. These include the consideration of publishing metrics. Some predatory outlets have managed to meet metrics such as impact factors⁶ and there is no doubt that the chase of academic publishing metrics helps to explain predatory publishers’ ascendency. Whilst I am not arguing for an abolishment of publishing metrics, a rethink of them is necessary.

What is very likely, though, is that publishing outlets that do not meet accepted academic publishing standards will eventually fail, even if they appear to be successfully catering for the need to meet certain metrics. In this context, last year I saw a post on LinkedIn⁷ where predatory publishing was likened to a ‘Ponzi Scheme’ which will eventually come crashing down.

This does not mean that traditional publishing models are without fault. One accusation made is that they make it difficult for some (in particular new) authors to get published due to ‘old boys’/old girls’ networks’ being in place where an established authorship receives ‘preferential’ treatment (either through bias of editors and/or reviewers). However, for the IA field, this is not something I have observed or experienced much as an editor. Whilst new ideas will usually require a robust justification by authors in order to overcome any possible scepticism of reviewers, I have not seen any real barriers for inexperienced and new authors to publish successfully.

IA as a discipline

Disciplinary grounding is an important ingredient of flourishing journals. Whilst IA (as defined earlier) is not usually considered an academic discipline, it has been able to develop disciplinary building blocks over time. Importantly, there are IA degree programmes in place throughout the world that at times are accredited (in the UK by, e.g., IEMA – Institute for Environmental Management and Assessment; Fischer and Jha-Thakur 2013). Furthermore, there is now a considerable body of IA relevant master and PhD level dissertations available globally (Fischer and Noble 2015). There is also IA associated theory (Pope et al. 2013; Fischer and González 2021) and whilst I have not come across a university department of impact assessment, yet, there are numerous IA research centres based at universities around the world from which IA research is conducted. With the recently inaugurated Academics Section at IAIA, an academic IA ‘society’ has also been formed. Finally, IA related collaboration between practice and academia is well established (see, e.g., Kågström et al. 2023; Bond and Fischer 2022).

One important component of a discipline is the existence of dedicated journals. Apart from those that are English language based and which have already been mentioned above (and amongst which IAPA is currently playing a paramount role), there are also IA journals in other languages, including, e.g. ‘Impact Assessment’ by the Japan Society for Impact Assessment,⁸ ‘Le Valutazioni Ambientali’ of the Italian Environmental Analysts’ Association⁹ and ‘UVP Report’ of the German EIA Association.¹⁰ Furthermore, non-English IA papers are published in other outlets, e.g. in the Brazilian Ambiente e Sociedade¹¹ or the Mexican Gestión y Política Pública.¹² Due to the limited number of dedicated English language IA journals, consideration of IA publications from other languages in the references of paper submissions should be encouraged by reviewers and editors. Also, mainstream English language journals may consider regional special issues in other languages (as long as a trusted and vetted guest editor who speaks that language is available).

Conclusion and outlook

As arguably the most important platform for IA publishing at present, IAPA plays a key role for the academic and wider professional IA community internationally. The journal is evidently thriving and there is every reason to take a positive outlook for the future of IA publishing. It is important to acknowledge, though, that journals change continuously and IAPA is no exception. Due to the very applied nature of the IA discipline, for many years IAPA has allowed for the submission of professional practice papers. These are shorter than research papers (only around 4,000 words rather than 6,000 to 8,000 words), require less in terms of interpretation of results and can include fewer references. They are a clear indication that IAPA is taking the needs of the community it is serving seriously. In addition, four years ago, ‘letters’ were introduced as another publication format. These can range from 1,000 to about 3,500 words and can be either ‘letters to the editor’ in which requests or pleas are made to the discipline, opinion letters or letters that contribute to discussions that have a very clear focus and where they function as pieces of a wider jigsaw (see, e.g., Bice and Fischer 2020). Letters have been favourably received by the IA community and numerous letters have been published over the past few years.

An important question arising from publishing trends occurring over the past two decades is whether there is a need for a code of ethics for publishers and editors. For example, for authors it is not always easy to spot predatory practices and the existence of, e.g., a voluntary accreditation scheme associated with a code of ethics (which should consider a journal’s aims and scope) would add clarity for both, authors and readers. Accreditation is accompanied by continuous reflection and as an editor I would welcome corresponding initiatives. Voluntary accreditation is known and occasionally applied in IA practice (see, e.g., Bond et al. 2017).

Suggestions have been made over the past few years that peer-reviewed academic publishing has become obsolete (see, e.g., Adler et al. 2019). Whilst I cannot speak for other disciplines, for the IA community this is clearly not the case, the reason being experiences with publishing outlets that do not have adequate quality assurance (through peer review) in place and where overall quality of papers is often unsatisfactory.

A thriving discipline should have a good range of dedicated journals representing it. Therefore, and in particular based on recent developments surrounding other IA focused journals, I believe there is currently space for another high-quality peer-reviewed English language IA journal. I would not see this as competition, but as a sign of a thriving discipline. Also, there is clearly scope for dedicated IA journals in other languages.

To end this reflection, I would like to wish the new editor of IAPA a successful time serving the journal. I am very pleased to see Dr Urmila Jha-Thakur in this very important role and I wish her all the very best for her time as editor. There is every reason to be positive about the future of IAPA.

Notes

1. Besides the present issue, these include special issues on ‘displacement, resettlement and livelihoods’ (Vanclay and Kemp 2017); ‘health in impact assessments’ (Fischer and Cave 2018); ‘guidelines for effective impact assessment’ (Fischer and Montaño 2019); two issues on ‘SEA effectiveness’ (Therivel and González 2019); ‘impact assessment for the 21st century’ (Bice and Fischer, 2020); ‘green infrastructure and IA’ (Clement and Fischer 2020); ‘gender-responsive approaches in impact assessment and management’ (Götzmann and Bainton 2021); ‘post-relocation support’ (Downing et al. 2021); and ‘ecological connectivity in environmental assessment’ (Torres et al. 2022).

2. The largest database of peer reviewed literature.

3. Fischer et al. (2023), in this issue, state that 187 of the 195 countries officially recognised by the United Nations have some form of legal requirements for EIA in place.

4. see https://iaia.org/affiliates.php.

5. A list of predatory publishers and journals is available here: https://predatoryreports.org/the-list.

6. Having an impact factor is very important for any journal, including IAPA.

7. ‘the largest professional network on the internet’.

8. https://www.jstage.jst.go.jp/browse/jsia/-char/en.

9. https://www.levalutazioniambientali.it/.

10. https://uvp.de/de.

11. https://www.scielo.br/j/asoc.

12. http://www.gestionypoliticapublica.cide.edu.