Development of a quality assurance review framework for health impact assessments

ABSTRACT

The use of health impact assessments (HIAs) has rapidly increased over the last two decades and has been undertaken to evaluate the health implications of many new projects across the globe. In July 2017, Wales became the first country to legislate for broad approach HIAs, i.e. it considers not only environmental health effects but the social determinants of health, well-being and inequalities. Although the introduction of statutory HIA will ensure the use of HIAs, experts have expressed concerns that legislative measures could lead to poor-quality HIAs and associated reports. This paper outlines the development process and analytical reflections of a quality assurance (QA) review framework for HIAs. The framework has been developed over a 2-year period which allowed the collective intellectual capital behind it to evolve in response to practice-based learning. A standardised form of QA for all types and levels of HIA will provide greater clarity regarding the required criteria for conducting and completing an HIA and ensuring this takes place in a robust, interdisciplinary and inter-sectoral manner. Therefore, this QA review framework should fit into existing HIA practice and complement other tools such as best practice guidance to sustain the global reputation of HIA.

KEYWORDS:

• Quality assurance
• Health Impact Assessment
• framework
• best practice

Background

Over the past two decades, the use of health impact assessment (HIA) has grown rapidly (Harris-Roxas et al. 2012) and this is most likely attributable to the increased combined efforts of the three complementary areas of policy concern and activity: environmental health, the wider determinants of health and health equity, all of which are integral components in HIAs (Harris-Roxas and Harris 2011; Harris-Roxas et al. 2012). The fundamentals of HIAs are concerned with how the health and well-being of a population could be affected by a proposed policy, plan or action and the widely accepted key definition of HIA is that of the Gothenburg Consensus (WHO 1999), which describes HIA as ‘a combination of procedures, methods and tools by which a policy, programme or project may be judged as to its potential effects on the health of a population, and the distribution of those effects within the population’.

The important contributions of HIAs have not gone unnoticed with decision maker and/or policy makers, both of whom have been influenced by the practice in various countries across the world such as New Zealand, Australia and the United States (Dannenberg et al. 2008; Haigh et al. 2013; Bourcier et al. 2015). Generally, government health agencies view HIA as a novel activity rather than as a core capability (Harris-Roxas et al. 2012). Some countries such as Slovakia and Thailand have a statutory requirement for HIA driven by environmental health and planning regulations (Harris-Roxas et al. 2012), whereas regions in Canada and Australia also have requirements for the use of mandatory HIAs (Owen 2017). However, in July 2017, with the passing of the Public Health (Wales) Act (UK Government 2017), Wales became the first country in the world to legislate for broad approach HIAs (Owen 2017). This may be regarded as a novel approach which encompasses not just the environmental impact but the wider determinants of health and also has a focus on reducing inequalities.

Despite emerging legislation on HIAs in some countries, the majority of HIAs are conducted outside legislative or regulatory requirements (Wismar et al. 2007) and consequently HIAs are instigated and carried out in a number of ways. For example, organisations may carry out their own HIAs internally, may engage a partner organisation to complete an HIA on their behalf or may commission an external agency (National Research Council 2011). It is essential that appointed agencies who are commissioned to carry out an HIA have no conflicts of interest, especially as openness, honesty and impartiality are integral components of a useful HIA (Wismar et al. 2007).

It is essential for the credibility and effectiveness of HIA that the process is seen to be applied in a consistent and robust way and to a satisfactory and credible standard. International and national best practice standards exist for conducting and reporting HIA (Quigley et al. 2006; Metcalfe et al. 2009; Chadderton et al. 2012a). Whilst these provide helpful guidance for those carrying out HIAs, these standards do not provide guidance on reviewing both the processes and the final reports of HIAs to ensure that they are fit for purpose and comply with best practice. Although there are a number of frameworks available to critically appraise the quality of strategic environmental assessments (Fischer 2010; Dalal-Clayton and Sadler 2017), very few tools exist to review the quality of impact assessments (IAs) or HIAs. Those tools that are available are usually focused on either the impact and effectiveness of an HIA (Harris-Roxas and Harris 2013) or a specific type of project, such as the review package for HIA reports regarding development projects only (Fredsgaard et al. 2009). Therefore, there is a need for a comprehensive framework which enables the quality of all HIAs to be rated and is suitable for a wider range of HIAs, including those assessing policies, projects, plans, services, developments and programmes.

This article presents the development of a quality assurance (QA) review framework for HIA (Green et al. 2017) and will also discuss the analytical reflections on the development process. The analytical reflections on the process will emphasise and focus on ensuring that the HIA process is not a tick-box exercise and that the quality of HIA is maintained. The framework which has been produced in Wales, and aligned to a Welsh context, has potential applications for any individual or organisation undertaking HIAs globally.

Origins of HIA practice in Wales

The origins of HIA practice in Wales were formulated during the preparation process for devolution which took place in 1999. The strategic policy document ‘Better Health, Better Wales’, published the year before by the Welsh Office (1998), prepared the way for a distinctively Welsh approach to health policy with an aspiration to address social class and geographical inequalities in health within Wales (Williams 2006). To achieve this aspiration, HIA was proposed as a mechanism for action across policy areas (a ‘Health in All Policies’ (HiAP) approach), and to help inform policy decisions to create long-term sustainable health gains. The creation of the Wales Health Impact Assessment Support Unit (WHIASU; https://whiasu.publichealthnetwork.cymru/en/) in 2001 confirmed a strong commitment to a long-term strategy for health improvement and addressing health inequalities.

From the outset, research skills in WHIASU have improved the evidence-gathering components of HIA, and case studies have been used as a way of reflecting on and evaluating different kinds of HIA as well as theorising the contribution of HIA to health knowledge more broadly. The HIA process was first tested internally on energy efficiency schemes and housing regeneration schemes before a third commissioned HIA tested the process with a particular emphasis on community involvement (Chadderton et al. 2012b). The current work of WHIASU, for which HIA practice in Wales is internationally recognised, continues to have a strong focus on participatory methods, health inequalities, a holistic vision of health and well-being and cross-sector engagement; support for community-led and -centred HIAs; and an ability to inform policy at a local and national level (Green et al. 2017).

The QA review framework for HIA (rationale and aims)

The legislative decision to move HIA in Wales from voluntary to statutory (Green 2017) is testament to this international reputation and is the main rationale for the development of the QA review framework; primarily to ensure HIA practice in Wales upholds the values, standards and approaches established over the last 15 years. The framework although designed to align with some of the governmental priorities in Wales contains many aspects and principles which can be applied to the global delivery of HIA practice. The aims of the QA review framework and the international and local implications are detailed in Table 1.

Table 1. Aims of the QA* review framework and the associated international and local context.

Provide a common framework and understanding of what a high-quality HIA looks like. Ensure that the evidence used to inform decisions that affect health and well-being is robust and inclusive. Ensure that people have opportunities to participate in decisions that affect their health and well-being via high-quality HIAs. Support decision-making in respect of policies, projects, plans, planning applications or the commissioning and reconfiguration of services. Ensure that health, well-being and inequalities have been considered in a holistic, systematic and robust manner. Provide resources to support international HIA practitioners and commissioners in their work.	Raise the standard of HIAs carried out in Wales so that they are carried out robustly.
	Support Welsh Government and other designated public bodies to meet any requirements of the Public Health (Wales) Act (2017) and the Well-being of Future Generations (Wales) Act (2015).
	Support Wales-wide commissioners, policymakers and practitioners in meeting their responsibilities to a high standard when they are required to undertake or review HIAs by providing a framework to critically appraise them.

* ‘Quality’ is defined as ‘the standard of something as measured against other things of a similar kind; the degree of excellence of something’ and ‘general excellence and standard level’.

‘Quality assurance’ is defined as ‘the maintenance of a desired level of quality in a service or product, especially by means of attention to every stage of the process of delivery or production’. The definitions are taken from the Oxford English Dictionary.

This QA review framework for HIA was developed over a 2-year period (2015–2017), which allowed the framework and the intelligence behind it to evolve in response to practice-based learning, delivery of training in HIA, a growing focus on well-being and long-term impacts in Welsh Government policy, and collaboration with local and national practitioners and experts in the field.

The process of development which is outlined in Figure 1 has included:

Figure 1. Development protocol for the QA review framework.

• Literature review and synthesis to prepare a draft outline and plan for the work.

• Engagement and discussions with HIA and other IA practitioners, academics and representatives from a wide range of sectors, i.e. environmental health and planning.

• The production of a draft quality review framework based on existing HIA knowledge and practice.

• Testing and reviewing of the draft framework by HIA practitioners, academics and cross-sector practitioners in three engagement workshops plus feedback via email and phone.

• Internal QA and governance processes.

Production of draft framework, workshop review and testing

The draft QA review framework was produced by WHIASU in collaboration with a range of UK-based HIA practitioners and experts to ensure that the included review criteria represented the professional consensus about the content required for a robust HIA. The process was led by the Unit and based on their extensive knowledge, experience and expertise in HIA. The Unit was aided in the development process by a small informal advisory group which evolved from a core of key individuals who were already engaged in discussions about the framework tool. The advisory group consisted of six experienced IA professionals from a range of backgrounds including academia, private planning consultancy and HIA or mental well-being impact practitioners. In total, members of the advisory group have facilitated, led, advised on and quality assured over 500 IAs. One of the major challenging aspects with the advisory group was holding meetings, particularly due to demanding schedules and the geographical dispersion of the members (North and South Wales, London and the South of England). To better facilitate meaningful discussions, communication took place to all via email with feedback gathered face to face, via telephone or email.

The draft framework, and included criteria, was developed using learning from existing QA tools and processes such as the quality mark scheme (IEMA 2011). A literature review was undertaken as part of the draft process and it included both published journal papers and the grey literature. Literature on effectiveness and quality in IA, critical assessment or review tools for IA and HIA frameworks and guidelines was reviewed. It highlighted the fact that whilst there are many HIA (and other IA) guidelines available to carry out HIA, there were very few review resources for IA and specifically HIA and it identified the need for such a QA review framework for HIA. Full details are contained in the final published document (Green et al. 2017). The included draft sections of the framework are documented in Table 2; these draft sections along with their associated criteria were subsequently reviewed in a series of workshops.

Table 2. Overview of sections included in the draft section and the sections included in the final QA review framework following the workshop discussions.

1. Report: Content, structure, accessibility and presentation. 2. Information about the project, policy, plan or proposal. 3. Methodology: Is it an HIA? Has it followed a recognised HIA methodology? 4. Principles and governance: Has it been conducted in a way that meets the principles and ethics of HIA? 5. Evidence: Is the evidence used to identify and assess impacts robust? 6. Appraisal, assessment and the identification of impacts. 7. Conclusions, recommendations and monitoring. 8. Additional criteria for capital and construction development type projects.

1. Information about the project, policy, plan or proposal. 2. Methodology: Is it an HIA? Has it followed a recognised HIA methodology? 3. Evidence: Is the evidence used to identify and assess impacts robust? 4. Appraisal, assessment and the identification of impacts. 5. Recommendations, conclusions and monitoring. 6. Principles and governance: Has it been conducted in a way that meets the principles and ethics of HIA?

Three workshops took place to review the framework criteria and test and evaluate the draft framework. All paperwork and aims were circulated beforehand and the workshops were led and co-ordinated by WHIASU practitioners. Thirty people were invited to participate in the workshops and the development process. These were drawn from a wide range of public health and IA practitioners and specialists. This included the core advisory group who commented on all three iterations and provided clear steer when needed. This included signposting others to include in the stakeholder workshops and suggesting direct amendments to the framework, i.e. changing wording due to how it could be interpreted. One member used the second draft version to review three HIA comprehensive training course reports and provided clear feedback having used it, i.e. the grading was flexible and the matrix was highly usable. Other attendees at the workshops included local and national policy officers (both health and non-health sectors), academics from social science and environmental health departments, an Equality IA lead and practitioners from other sectors such as land use planning. Thirteen participated in the workshops and/or provided feedback remotely – either by email, via telephone interview/discussion or through the structured workshops. Those who did not attend and/or contribute to the workshop discussions came from similar sectors to those who did and reasons for non-contribution/attendance included significant work pressures or a colleague was already contributing on behalf of the team/organisation.

The draft framework was tested against three publicly accessible HIA reports. One of these was chosen by the workshop participants to review at each session. Feedback and questions were raised as the participants worked through the document, and a discussion was held at the end with all comments captured by the facilitators from WHIASU. These workshops were attended by academics, public health practitioners, IA specialists and other sector representatives from Wales and the UK. Some of these individuals had also contributed in the engagement phase to develop the proposed framework criteria. For those who could not attend, there were opportunities to provide oral or written feedback via phone or email (five individuals did so via this method).

The main consensus from the workshops/feedback was that the framework in its initial format was too long and some of the sections duplicated questions and criteria. Therefore, the draft framework was condensed to six sections from eight (Table 2) with Sections 1 and 7 becoming integrated to focus on the effective communication of the findings from the HIA; review summary boxes at the end of each section were removed; and two or three individual criteria were incorporated into one single item with revisions. The final draft framework also included more hyperlinks to other documentation and references in order to condense the final document but retain the necessary information for reviewers. There were discussions in each of the stakeholder groups (and through email feedback) that the appraisal section was the real heart of the review process and therefore the review tool. Thus, this section was brought forward in the final review framework (from Section 6 to 4; Table 2) and given more prominence.

Another discussion from the workshops focused on the grading structure and approach – with the draft approach of G (Good), S (requires Strengthening) and I (Inadequate) being received favourably by all and was preferred to a points based or scoring system which other IA review tools have used (Fredsgaard et al. 2009). It was noted that the act of reviewing and the discussions which take place about the HIA and the report were more important than the grading itself. Several participants referred to the fact that they felt that it was a constructive grading process and the S option allowed for them to ask for clarifications and give clear and coherent feedback about the HIA report to others. One participant from the land use planning sector stated that the reviewing process gave them more confidence in the findings. This also highlighted the need for further training in the QA process and framework – something which WHIASU has developed and tested.

The final debate about the draft review framework centred on the introduction to the framework and the explanatory notes. It was believed that the aims and objectives and what was trying to be achieved needed more clarity from the outset and the language used needed to be more consistent with some terms being expanded on and reference links placed in the documents. The documentation was collated and discussed by the WHIASU team. The framework was amended to reflect the comments made by the participants (where possible) and circulated again for any further comment. The final round of feedback was positive to the changes made.

The full list of criteria for each of the six sections in the final framework and explanatory notes to help assist with completing the review can be found on the WHIASU website (WHIASU 2017).

When to use the QA review framework and expected outcomes

There are a wide range of contexts in which the review framework could (and should) be applied and it is important that we are not prescriptive about its use. However, the small list below details some examples based on practical experience of when the framework would be expected to be relevant:

• Community members who seek an independent assessment of the findings and methods of an HIA.

• Decision makers (e.g. planning officers) who want to be confident in the findings of an HIA related to local development or policy and form an opinion on it.

• Commissioners of an HIA who wish to verify that the HIA they requested has met best practice criteria.

• An HIA practitioner seeking a peer review of an HIA.

• Educators who require criteria on which to base conclusions about HIA.

• Researchers who are investigating the practice and/or theory of HIA.

Table 3 lists the expected outcomes and learning objectives of an individual when they have used the QA review framework to assess a completed HIA.

Table 3. Key objectives and expected outcomes for individuals when using the QA review framework.

The purpose of the review is to carry out a critical analysis of the HIA in the context that it was undertaken.

Each HIA will be unique to a specific context, proposal and decision-making process.

The criteria are to be used as a tool for the process of critical analysis, review and discussion rather than as a scoring system. However, the grading should make providing feedback easier.

The outcome of the review process should be that the reviewer(s) has a better understanding of the HIA, the quality of both the HIA process and the report, and also how much confidence they can place in the findings.

The reviewer(s) can then form an opinion and provide comprehensive, informed and clear feedback.

Discussion and conclusion

This paper has detailed the development of a QA review framework for HIA in Wales, which to our knowledge is one of the first dedicated HIA frameworks of its kind to be produced. The QA review framework fills an important gap in HIA practice by providing an instrument with which to assess all types of HIA and will contribute towards the international field of HIA. Thus, this new framework is also a step in the right direction to ensure that the requirement of review criteria is fulfilled (Harris-Roxas et al. 2012).

HIA criterion is broadly based on the wider determinants of health (Dahlgren and Whitehead 1991; Barton and Grant 2006), and is also one of the most important and effective methods to ensure that countries meet the HiAP approach (HiAP; WHO 2014). For example, HIA has been involved in the decision-making process to evaluate the health implications of many potential new interventions ranging from active travel policies (Mueller et al. 2015) and shared bicycle schemes (Woodcock et al. 2014) through to proposed night time economy frameworks (Ashton et al. 2018). Whilst the introduction of statutory HIA in Wales will ensure that HIAs are carried out, European experts have expressed concerns that legislative measures could lead to poor-quality HIAs and associated reports (Smith et al. 2010; Gruber 2017). Resource restraints within public bodies may result in HIA becoming a ‘tick box’ procedural activity which could be delayed by bureaucracy, rather than the envisioned catalyst to engage populations, academics, policymakers and analysts in a genuine participatory process of strategic planning to improve population health and reduce health inequalities (Krieger et al. 2003). This framework has been developed in part to address this issue and provide an instrument which is flexible enough to appraise different types of HIA, and ensure that the key components of the process are covered to a sufficient standard. Without QA, HIAs may not reach their potential to influence policy and decisions for better health and well-being. It is therefore hoped with the introduction of this described QA review framework that the credibility, impact and practice of HIA will be effectively sustained as HIAs move away from simply being ‘completed’ to being carried out to a high level of quality. It will provide confidence to commissioners and is a clear feedback vehicle.

In addition, a standardised form of QA for HIAs will provide greater clarity regarding the criteria for conducting and completing an HIA and will help ensure that the HIA is conducted in an interdisciplinary and inter-sectoral manner. These criteria were debated and agreed at three engagement workshops involving interdisciplinary peers. Peer engagement and mutual agreement of criteria are commonplace within HIA practice, and the criterion that constitutes a good evaluation of HIA was also agreed following a number of workshops (Parry and Kemm 2005). Unsurprisingly, the broad criteria included in the ‘agreed’ final sections of our QA review framework that would demonstrate a ‘high quality’ HIA are aligned to the proposed criteria to evaluate the impact and effectiveness of HIA (Harris-Roxas and Harris 2013). This is particularly evident in the ‘Principles and Governance’ section of the framework where transparency is one of the key questions. Transparency is an integral element of any HIA, and a well-documented account provides clear details on (i) what procedures were followed, (ii) how evidence was collected, (iii) the relative weighting given to different forms of evidence, (iv) the processes through which conflict and disagreement were managed, (v) the principles that informed the assessment, and (vi) how trade-offs were decided within the HIA (Harris-Roxas and Harris 2013).

One of the most important components of the development process was the series of workshops. At the start of the development process, the lead author outlined the different approaches to scoring IAs (Fredsgaard et al. 2009; IEMA 2011, 2016) and what that meant in practice, how they could be interpreted and how some scoring systems can ‘mask’ deficiencies and lead to HIAs which omit key components but still be deemed adequate or biased (Krieger et al. 2003). The lead author favoured a ‘peer review’ system but wanted to discuss this option at the workshops. The workshops highlighted the ability of the review framework to provide clear constructive feedback and the benefits of a simple flexible scoring system. The open scoring system, in particular, was something which all the participants liked. One of the most significant conclusions from the workshops was that whilst a score is still necessary, the actual act of critically analysing and discussing the HIA is the most important and beneficial aspect of the process. This supports the main objectives of the development of the tool which were to raise the standards of HIA, to ensure that HIA does not become a ‘tick box’ exercise; that there is a process to benchmark HIAs against and a mechanism with which to monitor HIAs and give clear and constructive feedback. This is likely to be achieved as long as it is advocated and applied consistently in Wales, and appropriate theoretical and practical training is provided to support this. It was also identified that the framework could be applied to a broad range of HIAs and could be applied across a number of sectors, and it recognises good practice can come in multiple forms by not being too restrictive nor dictatorial in its approach to HIA.

Although this QA framework has been developed in Wales, the framework has global relevance and this is clearly demonstrated in Table 1, where the international scopes of the aims outweigh the local applications two-to-one. The framework has the potential to be adapted to reflect and account for the legal, policy, economic, social, environmental and cultural context of any country. However, in countries with very different cultural backgrounds, different governmental, political and public health priorities and systems may exist. In these nations, regulatory frameworks regarding HIAs may not exist which could require additional quality queries to be included in the framework. To truly test the universal applicability of this QA framework, future studies should replicate our framework development in other countries (especially non-Western societies).

However, it should also be acknowledged that this developed QA review framework is only one tool to help maintain the creditability and effectiveness of HIA. A number of other complementary methods are also a necessity which include (i) the provision of high-quality training, advice and support; (ii) consistent adherence to best practice guidance; (iii) provision of evidence, resources, case studies and e-learning; (iv) opportunities to share and develop best practice at workshops and conferences; (v) opportunities to monitor and evaluate the effectiveness of HIA; and (vi) the application of review tools and processes. The QA review framework should also fill the practice gap with regard to QA for HIA, and fit alongside the existing tools for QA of strategic environment assessments (Fischer 2010; Dalal-Clayton and Sadler 2017).

In conclusion, this paper has described the development of a QA review framework for HIA which has been developed using current best practice guidance. It is envisaged that this proposed framework will enhance HIA practice both locally in Wales and globally. The framework is a working document and it is expected that the content will be revised and updated in response to further feedback and an evolving policy and research context. Initially, it is proposed that the first review of the QA review framework will take place in 2020.

Acknowledgments

The authors would like to extend their gratitude to Angharad Wooldridge for providing valuable feedback to the revised version of the manuscript.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This study did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.