Identifying and promoting qualitative methods for impact assessment

ABSTRACT

Qualitative methods for impact assessment (IA) represent a broad spectrum of approaches that are important for realising effective IA practice. The purpose of this paper is to identify and promote qualitative methods that are available for use in contemporary and future (next-generation) IA processes. From an extensive literature review, an international survey (145 responses), expert interviews (48 interviewees), and a workshop attended by 27 IA practitioners, 17 qualitative method categories were identified. These were further subdivided into three classes: conventional qualitative methods, highly participatory methods, and mixed methods. Each method is described, and an indication given of how each can be used in IA practice, including the specific stage of the IA process to which they might be applied. Whilst this paper seeks to stimulate practitioners to apply qualitative methods to enrich IA practices, the research also identifies a lack of expertise with social science methods as a significant barrier to the effective use of qualitative methods in IA practice.

KEYWORDS:

• Impact assessment
• qualitative methods
• participatory methods
• mixed methods
• next generation impact assessment

1. Introduction

Many methods for obtaining and analysing data may be utilised at different steps in the impact assessment (IA) process, which include baseline data collection, impact identification and prediction, comparison of alternatives, decision-making, mitigation design and implementation, and post-approval follow-up monitoring and management. In IA practice, researchers have long sought to promote and improve quantitative methods (e.g. Bisset 1978; Beanlands and Duinker 1984), despite the potential for such approaches to mask contentious items and close-down discussions. Notwithstanding that qualitative methods have always been in use in IA, the extent to which they are used, the types of methods being employed, and application of the data in IA have not previously been explicitly examined. Further, there seems to be an underlying perception on the part of some practitioners and decision-makers that social science methods based upon qualitative data are unscientific and subjective, and therefore provide second-rate evidence (Kørnøv and Thissen 2000; Nitz and Brown 2001; Owens et al. 2004; Retief et al. 2013). And yet, subjectivity and value-based decision-making have always been central to IA practice, not only in the assessment of social impacts where it is core (e.g. as acknowledged in numerous chapters within Vanclay and Esteves 2024) but also in the process of determining impact significance, which is fundamental to the theory and practice of IA (e.g. Ehrlich and Ross 2015; Jones and Morrison-Saunders 2016).

The need for appropriately selected and rigorously applied qualitative methods from social science to be more widely used in IA will only increase as governments move towards enshrining next-generation forms of assessment (e.g. Sinclair et al. 2018) in legislation. Following Sinclair et al. (2021), we view such forms of assessment as bringing together process elements that aim to achieve integrated planning and decision-making for sustainability, address policies, and programs as well as projects and cumulative local, regional, and global effects, and establish decision processes that empower the public, recognize uncertainties, and favour precaution.

The purpose of this paper is to respond to this need by identifying and introducing a range of qualitative methods that are available for use in contemporary and future IA processes. The research we report on here is part of a three-year-long program, funded by the Impact Assessment Agency of Canada, specifically seeking to understand and promote effective usage of qualitative methods in IA. Action in this regard was taken by the Agency due to the introduction of the Impact Assessment Act (of Canada) 2019, which broadened the scope of IA in Canada beyond traditional biophysical considerations to include next-generation elements such as a requirement to conduct gender-based analysis and the process features previously identified. A complete account of the research program is provided in Walker et al. (2023), and more specific assistance with the use of the methods identified is provided in this source. In this paper, our focus is specific to the typology of qualitative methods that can contribute to sustainability-oriented IA.

2. Methods

Four methods were used to undertake this research, comprising literature review, surveys, interviews, and workshops with international IA practitioners. Literature review was the initial method employed to identify qualitative methods, resulting in 22 method categories being established. Subsequently, surveys, interviews, and workshops were used to examine, refine, and consolidate these. We outline each of the four research methods in turn before discussing triangulation and the inter-relationships between our methods.

2.1 Literature review

Our literature review was undertaken in several stages. An initial scoping review (following Grant and Booth 2009) provided an indication of the breadth and depth of available literature and informed the selection of appropriate search terms for the structured literature review (SLR). Based upon the scoping review, the decision was made to incorporate search terms from related fields such as environmental planning and natural resource management, to identify qualitative methods that could potentially be applied in IA as well as those that already are. It was also determined that terms related to participatory and community-based methods should be included in the search.

The SLR was based on the following search chain within both Scopus and Google Scholar: (Qualitative OR subjective OR participat* OR community) AND (‘environmental assessment’ OR ‘impact assessment’ OR ‘natural resource management’ OR ‘spatial planning’ OR ‘land planning’ OR ‘land use planning’ OR ‘regional planning’ OR ‘urban planning’ OR ‘environmental planning’). Sources identified were screened for relevance to IA and usefulness of content in terms of understanding the method and its potential application.

In addition to our SLR searches, we also employed the ‘snowball’ methods of Greenhalgh and Peacock (2005) to pursue ‘references of references and electronic citation tracking’ (p. 1065). We also drew upon our ‘existing knowledge and networks’ (Badger et al. 2000, p. 223), including the external Best Practice Advisory Committee (BPAC) that we had established at the outset of the overall research project to guide our work (details of the committee provided in Walker et al. 2023), to inform our searches. This was especially valuable in sourcing grey literature not included in online databases like Google Scholar.

In selecting papers, we chose sources that:

• related to ex-ante assessment of policies, plans, programmes, and projects;

• discussed details of a specific method or technique applied to IA or a closely related field;

• were accessible (open access or available through university databases); and

• were in English.

This resulted in the identification of 423 sources through the scoping review search. An additional 30 sources the BPAC suggested were potentially relevant were also included, resulting in a total of 453 sources screened. Removal of duplicates and application of screening criteria left 135 sources for full-text review.

At this point, an initial list of qualitative methods was generated. A targeted literature review was then conducted on each method in the initial list, which involved both searching for the use of the method in IA and related fields using a similar search chain as for the SLR, as well as searching for the method alone, especially for those that are tightly prescribed (such as fuzzy sets and Q methodology). The targeted searches generated data on the method and its appropriate application; its strengths, limitations, and challenges; and practical considerations. Summaries of each method were then prepared using a standard template.

2.2 International survey

We designed and implemented an international online survey that asked IA professionals about their engagement with the 22 methods identified in our initial literature review and with qualitative research more broadly. To this end, we posed a combination of closed- and open-ended questions as suggested by Bhattacherjee (2012). Those specific to our focus in this paper on identifying qualitative methods used in contemporary IA are reproduced in Box 1. Our online survey by necessity was long, which is counter to the advice of research methods researchers such as Deutskens et al. (2004) or Revilla and Ochoa (2017). Following the advice of Neuman (2014) on how to improve reliability in social research methods, we piloted our survey with the BPAC prior to dissemination, which led to some refinements.

Box 1 Survey questions pertaining to the identification of qualitative methods in IA

• Please indicate the extent to which you use, or engage with, each of the 22 methods [i.e. subsequently itemised] – (closed-question responses: often, sometimes, rarely, or never).

• Please identify any additional qualitative methods [i.e. not in the list of 22 provided] that could be applied within IA.

• Please identify two methods you have used or engaged with that contributed most to the overall IA objectives.

• For each of these two methods, please indicate: a) impact area and IA process steps, b) associated data analysis methods, c) the strengths, weaknesses, and challenges of the methods, and d) practical considerations and tips.

Not knowing the total population of IA professionals who work with qualitative methods, we relied on non-random, convenience and purposive sampling (e.g. after Neuman 2014). Our survey, which was available for two months in 2022 was distributed via: 1) 238 emails sent directly to a list of potential participants known to have expertise in qualitative research in IA, as identified by the research team, the BPAC, and the literature review; 2) nine national and international IA professional associations’ newsletters and/or social media platforms (e.g. International Association for Impact Assessment – IAIA, including the IAIA affiliates and SIAHub); and 3) information cards distributed at the IAIA annual meeting in May 2022. A total of 145 responses were received; of which 80 responded positively to the final survey question enquiring whether they would be willing to participate in a follow-up interview.

2.3 Interviews

A semi-structured interview format was developed (e.g. following Al Balushi 2016) to enable in-depth discussions to emerge. The questions enquired about two specific methods and their use in IA, including details of their data collection and analysis procedures, implementation considerations and tips, strengths and challenges, and appropriate contexts for use in IA (i.e. the interview questions were similar in nature to those in the survey; Box 1). The research team collaboratively drafted an interview guide and four members of the team conducted pilot interviews to test the guide, resulting in some refinements.

Interviewees were initially targeted from the 80 survey respondents mentioned previously, with 46 being identified as approaches that would cover experience with the widest possible range of qualitative methods in the list of 22. Forty of these practitioners subsequently agreed to be interviewed and further eight interviews were also conducted with IA professionals who had not completed the survey, but who were sought out because they had known expertise in otherwise under-represented methods. The interviews were conducted via Zoom or Teams using the transcription or closed caption functions to create initial transcripts. The quality of these transcripts varied considerably, and the audio recordings were used to verify and refine the transcripts.

2.4 Workshops

As a research method, workshops can serve as ‘evaluative, reflexive milestones (formative evaluation)’ (Lang et al. 2012, p. 32) as well as being ‘ways of developing or changing existing programmes or interventions … [and] to explore theories or strategies’ (Ritchie and Lewis 2003, p. 290) arising from research. A workshop was conducted by members of the research team in attendance at the annual conferences of IAIA in 2022 to simultaneously share and test aspects of our overall research program. This workshop, which was attended by 27 IA practitioners (comprising researchers, non-governmental organization representatives, and government/regulatory professionals), provided an opportunity to verify the list of 22 methods that had emerged from the initial literature review and begin to develop a more in-depth understanding of their application in IA. In addition to reflecting and commenting upon their use of these methods, workshop participants were asked to identify any key methods missing from our list. Specific methods suggested by participants were subsequently either found to fit within the existing method categories or were excluded because they were considered research approaches (e.g. ethnography) or technical tools (e.g. Mentimeter, Zoom), rather than specific research methods.

2.5 Data analysis

The qualitative survey data, interview transcripts, and workshop notes were coded together in NVivo 12 using a hybrid deductive-inductive thematic qualitative analysis approach. Deductive codes were initially established for each method based on specific project objectives, such as ‘Strengths & Value,’ ‘Challenges & Limitations,’ ‘Practical Considerations,’ and ‘Contextual Considerations’ (i.e. relevant IA process steps and impact categories). After the data had been coded deductively, an inductive coding process allowed for specific themes to emerge from the data (following the process outlined by Braun and Clarke 2006). For example, 40 discrete sections of data had been deductively coded to ‘Practical Considerations’ for interviews. These data were read thoroughly, and initial second-level codes were inductively developed to cluster the data into meaningful groups. The content of the codes was reviewed again, and related codes merged where related ideas were observed. For instance, upon review, inductive second-level codes ‘respect’ and ‘safe spaces’ were determined to contain similar ideas. These were then merged into a single code ‘creating safe spaces,’ which was identified as an important theme relating to good practice considerations for conducting interviews. This example code structure along with a few samples of coded data are shown in Figure 1.

Figure 1. An example of data coded using the hybrid deductive-inductive coding approach.

The inclusion of data from the multiple methods used and the numerous participants who took part in this research facilitated triangulation, since we were able to ‘learn more by observing from multiple perspectives than by looking from only a single perspective’ (Neuman 2014, p. 166). For example, in establishing the broader themes noted above, we actively sought confirmation from multiple sources of data, such as the coded qualitative data and targeted literature review summary templates. Further triangulation was subsequently conducted during analysis that matched qualitative methods with particular IA needs and considered how different methods might be used in different stages of IA. This is explained further in the results section in relation to Table 2 and Figure 1.

3. Results

While 22 qualitative methods were initially identified for use in IA from the initial literature review (as explained previously), following application of all the methods, the final list was shortened to 17 method categories. Some entries from the original list were excluded because participants perceived them as research approaches (e.g. participatory rural appraisal) or IA processes (e.g. value mapping) rather than specific methods. Others were excluded because the data did not contain sufficient information about how they can be applied to IA (e.g. qualitative modelling) or how they can be applied in novel/innovative ways in contemporary IA (e.g. checklists). Finally, system analysis and network analysis were combined into a single method category in the final list due to their similarities in use and application in IA.

The final list of 17 method categories is presented in alphabetical order in Table 1, along with a brief description and example references for each. Readers are referred to Walker et al. (2023) for more detail, and a more comprehensive reading list, for each method category. Within some of the categories, a variety of specific or unique methods can be identified. For example, deliberative methods involve various discussion-based approaches and the entry for this in the table includes three examples. The decision to group such specific methods into categories was to enable the table (and the subsequent discussion of each category) to serve as a simple reference guide for IA practitioners. Attention is also drawn in the description of methods to those frequently utilised by the survey respondents (with ‘frequently’ being defined as those used by at least 30% of respondents).

Table 1. Qualitative methods used in impact assessment (adapted from Walker et al. 2023, pp. 38–39).

Method category	Description
Deliberative methods	Rely on discussion-based approaches to engage the public in collaborative problem solving and decision-making (Gastil 2009). Deliberative methods are founded on the understanding that ordinary citizens provided with appropriate information and led through a process of deliberation can make sound, well-informed decisions (Wiklund and Viklund 2006). These methods aim for representativeness, including people with diverse backgrounds, viewpoints, and values. Three common deliberative methods include deliberative polling (Fishkin 2021), world cafés (Schieffer et al. 2004), and citizens’ juries (Crosby 1995).
Delphi method	A technique for systematically eliciting advice, and ultimately consensus, from a panel of anonymous experts through iterative rounds of questionnaires (Richey et al. 1985; Egan and Jones 1997). It is particularly useful where an evaluation is required but only incomplete information is available or where the experts are geographically distributed (e.g. Hanna and Noble 2015) or have diverse expertise in relation to the problem. A minimum of two rounds are conducted in Delphi studies (Richey et al. 1985), with additional rounds of questionnaires developed if consensus is not yet reached.
Document analysis	This was the most used method by the survey respondents (83%). It involves systematically identifying, selecting, evaluating, analysing, and synthesizing information about a specific topic from print and electronic documents (Bowen 2009; Savin-Baden and Major 2013). The information is usually organised into major themes, categories, and case examples through thematic or content analysis (Bowen 2009). Common types of document analysis include literature reviews, systematic reviews and meta-analyses, secondary data analysis, archival and historical research, and policy research (Tight 2019).
Focus groups	Identified by 30% of survey respondents as being commonly used in IA, focus groups are facilitator-moderated group discussions that explore participant experiences, perspectives, and opinions about a specific topic (Morgan 1996; Toth 2001). In contrast to research interviews, which elicit individually held knowledge, focus groups are used to understand shared and collectively held knowledge, and the range of perspectives held by certain groups or communities (Creswell 2007). Although a moderator guides the discussion, knowledge is generated through the conversation among focus group participants (typically 6–8) rather than through interaction directly between the moderator and individual participants.
Fuzzy sets	Fuzzy set theory enables the quantitative processing of imprecise qualitative data represented by linguistic variables (Zadeh 1965). This involves transforming qualitative, descriptive data into a form that can be mathematically described and manipulated in a rigorous way that accounts for the subjective nature of the descriptors. Fuzzy set theory is a potentially powerful tool for IA because linguistic variables that are open to different interpretations (e.g. severe/major/minor/negligible) are often used to describe the significance of impacts. Wood et al. (2007) used fuzzy set theory within IA to define degrees of significance of noise and visual impacts of a proposed wind farm development.
Interviews	The second most used method by the survey respondents (61%), interviews allow for in-depth exploration of individuals’ experiences, perspectives, and opinions. Interviews can be unstructured, semi-structured, or highly structured. According to our respondents, interviews in IA are most often semi-structured, combining structure with flexibility by using an established set of questions to guide the conversation, but also allowing for follow-up and exploration of new topics raised by the interviewees. In such interviews, the participant shares knowledge and experiences in their own words and the interviewer probes and re-directs discussion as needed (Morris 2015). Interviews can take place face-to-face, online, or via telephone.
Matrices	Matrices—grids that link system components with project activities—are among the oldest and most familiar methods for IA. Originally focused on environmental systems (Fischer and Davies 1973), their use has expanded to encompass social-environmental systems. Multiple matrix methods can be used in IA (Bisset 1980; Wathern 1984; Shopley et al. 1990). While these often rely primarily on expert judgement to determine the most relevant system components and conduct significance evaluations, participatory applications have been developed to include a wider range of values and perspectives (e.g. Satterfield et al. 2013; Nchanji et al. 2017).
Multi-criteria analysis	Multi-criteria analysis, or multi-criteria decision analysis, is a family of mathematical techniques that support decision-making by assessing and aggregating performance of options (such as alternative development proposals) against multiple, often conflicting, criteria (Dodgson et al. 2009; Geneletti and Ferretti 2015; Te Boveldt et al. 2021). Participatory or qualitative approaches may be used to collect and integrate qualitative data into the analysis. Such approaches are helpful in the weighting step, which involves assessing the relative importance of each criterion and allows for integration of the subjective views and values of different stakeholders, and allows for sensitivity testing (of the sensitivity of conclusions to different value-sets).
Narrative methods	Narrative research involves the telling and interpretation of people’s experiences through storytelling. It helps in ordering and interpreting events and circumstances experienced by individuals, while also revealing broader cultural, social, and institutional contexts in which people live. Narrative research can involve various data collection techniques, such as oral histories, interviews, journal entries, and digital recordings (Ross 1989; Moen 2006). Narrative data can be presented through vignettes (e.g. Vanclay 2015), through detailed contextualizing information interspersed with passages directly from data (e.g. Goater et al. 2012), or, in the case of digital storytelling, through short audio-video compilations (e.g. Gislason et al. 2018).
Q methodology	Developed to enable the systematic study of subjectivity (Brown 1993), Q methodology uses statistical analysis to identify dominant perspectives/discourses around a specific issue by having participants sort and rank a set (called a concourse) of qualitative statements representing a full range of opinions. It can generate insights beyond a polarised description (e.g. jobs versus economy), whereby there are often three, four or five distinct points of view on an issue. It has potential application in IA where the goal is to understand different perspectives or worldviews that may be held by different groups (e.g. Jenkins, 2017).
Qualitative data analysis	Often used by 37% of the survey respondents, qualitative data analysis is a systematic process of organizing and interpreting text and image data. A wide range of data sources and types can be qualitatively analysed (e.g. interview and focus group transcripts, written documents, photographs, and workshop notes). Five generic steps of qualitative data analysis include: 1) organizing and preparing the data; 2) an initial review of the data; 3) data coding; 4) categorizing and theming; and 5) interpretation (Leavy 2017; Creswell and Creswell 2018). These are often managed using computer-assisted data analysis software (e.g. NVivo, ATLAS-ti).
Scenario-based methods	Scenario-based methods integrate qualitative future-oriented scenarios (i.e. plausible or desired pathways by which the future could unfold) into IA analysis. They encompass various techniques that facilitate strategic mid-range and long-term planning when a certain degree of uncertainty exists (e.g. Rounsevell and Metzger 2010; UK Government 2017). These methods are typically used for either risk management (i.e. by testing decisions against desired futures) or for generating creative new ideas (Duinker and Greig 2007). Two main forms of scenario-based methods were identified in this research for use in IA: scenario analysis (Reed et al. 2013) and gaming (UK Government 2017).
Participatory spatial methods	These diverse methods integrate geospatial, rich, and in-depth qualitative information, and can rely on basic materials in situ, printed maps, or digital technologies and global positioning systems. Prominent forms include participatory geographic information systems (PGIS, or PPGIS) (Sieber 2006; González et al. 2008; Alagan and Aladuwaka 2012; Van Riper et al. 2020) and variants, e.g. web-based GIS (Kwan and Ding 2008; Tang and Liu 2016), volunteered geographic information (Brown et al. 2014), geo-visualization and geo-narratives (Kwan and Ding 2008). Other important forms include land use and occupancy mapping (or traditional land use mapping) (Tobias 2014) and community mapping (Chambers 2006; Spaling et al. 2011).
Surveys	Surveys use highly structured questionnaires to examine individuals’ experiences, perspectives, attitudes, or opinions about a specific topic (Neuman 2014). Many of the respondents (38%) indicated they often use qualitative surveys in their IA practice. By obtaining comparable information from respondents, surveys can allow standardised measurement across the answers received. Surveys may be undertaken as a quantitative method when they rely on closed-ended questions and statistical analysis of the data. However, they can also include qualitative open-ended questions, which allow respondents to expand on their answers in their own words.
Systems/network analysis	Systems analysis involves representation and analysis of relationships between systems components relevant to an IA. The analysis may include one or more systems (e.g. ecological, social, economic, institutional). Network analysis is closely related and maps the relationships among stakeholders and/or impact chains relevant to the IA. A broad distinction between the two is that network analysis, is based upon linear chains of causality, while systems analysis also incorporates circular causality and feedback loops (European Commission 1999; Perdicoúlis and Glasson 2006).
Visual methods	These methods collect and analyze visual or audio-visual images as data. Prominent visual methods include photo-elicitation, photovoice, and seasonal calendars. Photo-based methods involve the collection of photographs as data or that use photographs to facilitate data collection through other methods (Vitous and Zarger 2020). Seasonal calendars (also called seasonal activity calendars, community calendars or Indigenous calendars) involve collecting qualitative data that are used to develop diagrams depicting important environmental conditions and community activities, events, issues, and opportunities over an annual cycle (Narayanasamy 2008). Other important visual methods include video narratives (Gislason et al. 2018) and social media image analysis (Chen et al. 2019, 2020).
Workshops	Workshops are facilitated participatory sessions in which participants discuss, brainstorm, and identify solutions for a specific problem (Ørngreen and Levinsen 2017). They are versatile and can integrate a wide range of qualitative and quantitative data gathering techniques, and were one of the more commonly used qualitative methods with 38% of survey respondents indicating they often use them in their IA practice. Workshops typically run longer (three hours to multiple days) and include more participants (10+) than a focus group.

Each qualitative method has unique characteristics that might help determine its suitability to meet particular IA needs (e.g. the extent to which a method is participatory or has the ability to deal with uncertainties) or contextual circumstances faced by a practitioner (e.g. cost and time demands or technical abilities needed to apply a method). It is beyond the scope of this paper to fully analyse and explain these characteristics for each of the 17 qualitative method categories we have identified. However, as a quick reference guide, Table 2 provides some suggestions for which of the methods might best be utilised for various aspects of IA practice. The table as derived by triangulating among data coded to ‘Strengths and Value’ (a deductive subcode established for each method category in NVivo) and the ‘Strengths’ section of literature review synthesis tables that were prepared for each method. Although our triangulation process was not fully comprehensive (i.e. it did not attempt to identify all relevant circumstances present in the data), the table includes the most prominent themes and IA circumstances. Through the qualitative analysis, the research team identified IA stages at which the different qualitative methods have been used or participants identified as relevant (e.g. early development design and planning, screening, and scoping, approval decision-making, and follow-up). These matches between methods and particular steps in IA are presented in Figure 2. The figure is based on findings from the analysis; therefore, it is possible that there are additional IA steps at which IA methods could be applied, but were not identified through this research and thus are absent from Figure 2.

Figure 2. Impact assessment steps for which each qualitative method has been used or suggested as relevant by research participants (adapted from Walker et al. 2023, p. 42).

Table 2. Matching qualitative methods with particular IA needs (adapted from Walker et al. 2023, p. 44).

If an IA seeks to…	The possible methods include…
Gather external expert knowledge on a specific topic	document analysis (particularly academic literature, organizational reports), fuzzy sets, Delphi method, interviews, matrices, multi-criteria analysis
Understand the broad range of public perspectives, concerns, or opinions related to an IA	deliberative methods, document analysis (particularly news media analysis), fuzzy sets, surveys, Q methodology, visual methods (particularly photo preference surveys, photo visioning, and social media image analysis), workshops
Understand local values associated with place and land	document analysis, focus groups, interviews, (participatory) matrices, narrative methods, participatory spatial methods, visual methods
Understand perspectives of specific demographic groups related to specific topics (e.g. such as for gender-based analysis plus [GBA+] processes)	focus groups, (participatory) matrices, scenario-based methods, visual methods, workshops
Gather information to inform community-led IAs	focus groups, narrative methods, participatory spatial methods, surveys, visual methods, workshops
Integrate qualitative and quantitative indicators/assessment criteria and related data	matrices, multi-criteria analysis, fuzzy sets
Identify impact pathways and relationships among impacts and valued components in complex systems	matrices, multi-criteria analysis, systems/network analysis
Understand potential impacts on intangible social, cultural, spiritual, environmental values (e.g. connection to place, social cohesion, mental and emotional well-being, spirituality)	interviews, narrative methods, participatory spatial methods, visual methods
Understand the historical, environmental, social, cultural, economic, and political contexts in which a proposed project is located	document analysis, focus groups, narrative methods, scenario-based methods, visual methods

4. Discussion

Table 1 describes an array of qualitative methods that can contribute to more effective IA by helping to address subjectivity and value-based decision-making in IA practice. The decision to group-specific methods into categories and subsequently display them in alphabetical order was made to permit the table to serve as a simple reference guide for IA practitioners. The 17 method categories are diverse, encompassing a range of conventional qualitative social science methods, innovative participatory methods, and mixed methods that rely on a blend of qualitative and quantitative data collection and analysis techniques. These methods are not mutually exclusive, and combinations of them may be employed simultaneously (for example, visual methods might be used with numerous categories including within interviews, scenario-based methods, multi-criteria analysis, and many others).

Regardless of the suite of methods employed, an important consideration for practitioners is the extent to which they have experience and expertise to enable them to use the methods effectively. A cautionary note in this regard was flagged in the interviews. Several participants commented that there is a general lack of capacity in IA for using social science research methods. For example, one social scientist noted that many people doing IA – even social impact assessment – are trained in the natural sciences and assume that qualitative methods can be done by anyone, even without appropriate training. The interviewee was adamant that this should not be the case. It was further suggested that qualitative data analysis can be particularly problematic in IA where there is a lack of rigour. Further study is needed to clarify the depth and breadth of this problem in IA, but experience, expertise, and rigour are obviously needed for effective application of any research method, whether it is qualitative or quantitative (e.g. Neuman 2014; Leavy 2017; Creswell and Creswell 2018).

Technical expertise is particularly important in the case of mixed methods. Another cautionary note raised in the interviews was that inappropriate quantitative methods are often applied to qualitative data, e.g. ratings of high/medium/low become 3/2/1, and then these get summed, or mean values are calculated, which is not mathematically defensible (as it treats ordinal data as if they are linear). This is one of the reasons that qualitative data are often viewed with suspicion among practitioners. Methods such as fuzzy sets, multi-criteria analysis, and Q methodology offer ways to conduct legitimate mathematical analysis of qualitative data (e.g. Brown 1993; Wood et al. 2007; Geneletti and Ferretti 2015). However, another caution for practitioners is that these methods have very prescriptive and technical requirements and are not appropriate choices unless suitable expertise is available. Additionally, although quantitative and mixed methods provide findings that are highly legible to decision makers (because numerical results provide clarity), they can lose the richness, depth, and nuanced understandings that qualitative methods can provide.

Table 2 and Figure 2 match qualitative methods with, respectively, particular IA needs and discrete IA process steps. Table 2 is not comprehensive but covers highly salient method characteristics and IA circumstances. Together, the table and figure offer a quick reference guide for practitioners on when and how each method can contribute to IA practice, needs, and process. An important observation about the table and figure is that multiple methods can, and perhaps should (for the sake of triangulation and validity), be used to meet each IA need/objective and at each IA step. Another is that the table and figure reinforce the degree to which document analysis is a foundational method for meeting most IA information needs at most IA steps. It was by far the most used method by the survey respondents (83%). These details underscore the importance of addressing the concerns noted earlier about the lack of rigour in qualitative data analysis. When document reviews centre on rich descriptions of social, cultural, and environmental conditions, they need to go beyond quick scans and overviews. They need to be careful, systematic, and focused on yielding valid interpretations of the variables of interest (Bowen 2009).

Given our emphasis on current and future (next generation) IA and addressing subjectivity and value-based decisions, a highly salient consideration for matching methods with IA needs and process steps is the degree to which the methods are participatory (Sinclair et al. 2015; Diduck and Sinclair 2024). As a guide for practitioners, Figure 3 offers a classification of the 17 main method categories outlined in Table 1. We recognise that the classification is rudimentary because of overlaps between classifications, for example, some of the conventional methods (focus groups and workshops) are inherently participatory and some of the mixed methods (e.g. network analysis, multi-criteria analysis, and Q methodology) can be applied in a participatory way.

Figure 3. A basic classification of qualitative methods for IA.

Participatory methods are considered to be more innovative, which perhaps explains why they were perceived to be used less frequently than other methods, although noteworthy examples were cited by the research participants. One such example is community mapping, which emerged from participatory rural appraisal (Chambers 2006) and is likely most relevant to community-based IA in rural contexts, particularly in the global South (e.g. Spaling et al. 2011). Another example is land use and occupancy mapping (or traditional land use mapping) (Tobias 2014). Since this method involves a collection of interview data about traditional use of resources and occupancy of lands by Indigenous peoples, best practice dictates that it be initiated, led, and controlled by Indigenous peoples; applied and used in ways that reflect Indigenous worldviews; and support Indigenous sovereignty and self-representation (e.g. Joly et al. 2018). A third example is the use of participatory geographic information systems in research that was undertaken in the context of IA or environmental planning (Alagan and Aladuwaka 2012; Brown et al. 2014).

5. Conclusions

In this paper, we set out to identify and promote qualitative methods for impact assessment. The 17 categories of methods that have been described were identified from literature and their use within IA by practitioners was confirmed and consolidated through surveys, interviews, and workshops. While there are numerous relationships evident between the 17 methods and others that simply offer an alternative means of delivering a particular IA function, they are individually unique and discrete approaches to collecting and analysing qualitative data. The brief descriptions provided (in summary form in Table 1) as well as what each method offers IA practice (e.g. in summary form in Table 2 and Figure 2) are intended to serve as a useful guide to IA practitioners seeking to enrich their own ‘toolbox’ of IA methods. The hope is that the promotion of these methods will benefit practitioners by serving as something of an introduction and guide to ways of doing IA with which they may not previously have been familiar. In support of this, our full report (Walker et al. 2023) establishes ways that qualitative methods in IA can be strengthened and also offers a guide to implementing the methods identified with IA case examples.

In particular, the hope is that this paper stimulates and enriches current and future IA practice and initiates a shift in the trend we found during our research, namely that practice has been limited to a relatively small number of conventional qualitative social science methods (document analysis, interviews, focus groups, workshops, narrative methods, surveys, and qualitative data analysis). Also, a significant caveat remains regarding the concern expressed by interviewees in relation to the expertise required to properly apply qualitative methods. Social science training is essential in most cases, or the methods will not deliver what is intended.

Ultimately, if practice is to deliver on next-generation IA expectations, then the consideration of the values and perspectives of multiple and diverse stakeholder groups is increasingly important. The methods highlighted in this paper facilitate this shift towards the inclusion of broader next-generation considerations in decision-making. Tables 1 and 2 and Figures 2 and 3 are intended to serve as simple reference guides for IA practitioners wishing to broaden their knowledge and application of qualitative methods. However, they merely represent a snapshot in time for thinking about the 17 qualitative methods identified in this research. New innovations will inevitably emerge, and new applications of qualitative methods for tasks not identified in this research will also emerge over time.

Acknowledgments

We are grateful to the many IA practitioners, researchers, and decision-makers who shared their knowledge of these methods. A nine-member Best Practices Advisory Committee provided direction and feedback throughout the process. The research was supported by four research assistants: Anna Pattison, Ran Wei, Flo Kaempf, and Penelope Sanz.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

Funding for this research was provided by the Impact Assessment Agency of Canada.