Automated content analysis as a tool for research and practice: a case illustration from the Prairie Creek and Nico environmental assessments in the Northwest Territories, Canada

Abstract

Public engagement is essential to the procedural and substantive sustainability of environmental assessment. Public hearings present the lowest barrier to entry for public participation, but these forums face competing political pressures for conducting appropriate public engagement within an expeditious process. Repositories of public hearing testimony provide a source of primary data for examining these public engagement issues during environmental assessments. However, the time and resources required may be prohibitive for conducting the kind of in-depth qualitative analyses that are commonly used. Automated content analysis (ACA) techniques can provide a rapid, replicable, inductive, and systematic way to examine public hearing transcripts, consisting of the critical development and application of computer programming scripts that synthesize evidence from extensive document sets. This case illustration demonstrates the potential utility of ACA, based on the examination of two public hearings, Prairie Creek (EA0809-002; 2008–2011) and Nico (EA0809-004; 2009–2013) conducted in the Mackenzie Valley, Northwest Territories, Canada. Our interpretation of the findings provides an evaluation of ACA methods and situates its potential to inform environmental assessment research and practice across jurisdictions.

Keywords:

• Automated content analysis
• circumpolar region
• environmental assessment
• natural resource development
• public hearings
• public participation

Introduction

Public engagement is essential to the long-term sustainability of environmental assessment (EA) (Abelson et al. 2003), supporting both the procedural and substantive mandates of EA (Green 2014). Procedurally, natural justice affords people who may be adversely affected by proposed developments the right to be heard and have their input considered by EA decision-makers (Morrison-Saunders & Early 2008). Participation in public hearings is one of the lowest barriers to entry for these people, by fostering engagement that requires only the provision of oral testimony (Organization of American States 2001). Substantively, the local knowledge provided by the public can have immense deliberative value, helping to situate and strengthen scientific and technical expertise inputs to the EA process (Corburn 2003). Nevertheless, the promotion of public engagement as a procedural and substantive value can be constrained by the inevitable pressures and tensions that emerge in order to facilitate an expeditious EA process (Morrison-Saunders & Early 2008; Gibson 2012). As such, public engagement practices in EA are inherently political, situated in contextual interactions between the public and other stakeholder groups (Cundill & Rodela 2012).

To address the challenge of understanding engagement through public hearings, researchers and practitioners increasingly draw upon exhaustive records of testimony provided in publically available registries or databases (Kengne et al. 2013; Sainath & Rajan 2014; Sandlos & Keeling 2016). The vast majority of research techniques employed in the analysis of public hearings are qualitative in origin, and in many cases can provide theoretically robust interpretations. However, the time and resources needed to qualitatively examine the full extent of documentation produced during even a single EA is a daunting proposition, which may deter all but the most well-funded projects (Grimmer & Stewart 2013). Moreover, the use of qualitative methods may limit the transferability of findings between public hearing document sets (Malterud 2001). From a practitioner perspective, there is the challenge of developing and implementing accurate and transparent protocols so that information generated through public hearings is appropriately and effectively used (Irvin & Stansbury 2004). All stakeholders have an interest in ensuring that procedures for robust knowledge utilization will support procedural and substantive learning as part of public hearing processes (Crona & Parker 2012). Stakeholders include the proponents of development, regulators, government agencies, civil society, and members of the public.

Automated content analysis (ACA) may provide a new tool to support public engagement best practices by enabling a rapid, replicable, inductive, and systematic approach for summarizing public hearing document sets for EA research and practice. Pioneered for use in search engine optimization, ACA involves synthesizing large volumes of input text into various derived and modeled output formats (Hopkins & King 2010). ACA techniques are becoming more prominent in the fields of political science (Lowe et al. 2011), social media studies (Schwartz & Ungar 2015), and even ecology (Nunez-Mir et al. 2016). In the case of political science, particular emphasis has been placed on applications that address ideological scaling, document classification, and measuring the correspondence between texts (Grimmer & Stewart 2013). In examining public engagement during EA, ACA can address a number of procedural and substantive issues, utilizing approaches specifically tailored to the research or practice context. In short, ACA can increase the accessibility of large volume document sets generated by public hearings, reduce the time and resource costs associated with comprehensive analyses, and facilitate more systematic comparisons between EAs (Laver & Garry 2000).

This case illustration presents the results from ACA conducted on transcripts for two public hearings held during EAs in the Northwest Territories, Canada. There are unique considerations for public engagement in the northern Canadian context, since those with the potential to be adversely affected by development in this region are predominantly members of remote aboriginal communities (Gibson & Klinck 2005). Indeed, it was in this region that the Mackenzie Valley Pipeline Inquiry of the mid-1970s established the gold standard for meaningful consultation with aboriginal Canadians: the Berger Inquiry was based on public hearings in 35 communities, producing tens of thousands of pages of documentation compiled in hundreds of volumes (Gamble 1978). Today, a recent population-weighted (age, region, gender, aboriginal status) random telephone survey of Northwest Territories residents indicates the vast majority (over 80%) have positive feelings about mining, think it benefits the territory, think regulation is working, and would like to see more mining projects go forward (Abacus Data 2016). Thus, our present analysis presents a regional context for EA in which social license is obtainable, yet where the perennial issues around public engagement are nevertheless at play, furnishing insights to apply ACA across a wide variety of EA settings. Procedurally, local knowledge communicated through oral testimony is not always easily captured as an input, especially where political, economic, and social structural barriers impede participation (Witting 2015). Substantively, deliberative practices can potentially marginalize local knowledge in favor of more scientific and technical expertise (Sandlos & Keeling 2016). The case illustration explores these issues as one of the first contributions to the EA literature using ACA techniques, demonstrating the potential utility of this automated form of analysis for research and practice.

Materials and methods

Scope of the case illustration

The public hearing transcripts for this ACA were housed on the Mackenzie Valley Environmental Impact Review Board (MVEIRB) public registry website (www.reviewboard.ca) under the project names Prairie Creek (EA0809-002; 2008–2011) and Nico (EA0809-004; 2009–2013). These two EAs were selected after consulting with staff members of MVEIRB, who identified public hearings that, in conjunction with biophysical impacts, involved a broad slate of potential adverse impacts for affected aboriginal communities. Taking the socio-ecological approach outlined in the Ottawa Charter for Health Promotion, these impacts were considered to concern any and all of ‘the inextricable links between people and their environment’ (World Health Organization 1986). Observed impacts of industrial development in remote northern regions include interference with subsistence lifestyles (due to increased traffic, dust, and pollutants); loss of social cohesion, language, and culture (with the influx of industry); income stratification (unequal gains for employees); stress on infrastructure (housing, health, and social services); and increased social pathologies (substance abuse, violence, and sexual risk-taking) (Bronson & Noble 2006; Davison & Hawe 2012; Government of the Northwest Territories 2015; McGetrick et al. 2015). Given the complexity of these impacts, public engagement in EA is essential for gathering local knowledge and helping to ensure that industry will make the appropriate net contribution to northern aboriginal communities.

Document sample

Prairie Creek (EA0809-002; 2008–2011)

In 2008, Canadian Zinc proposed the Prairie Creek mine as an underground lead, zinc, and silver natural resource development, with water treatment, paste backfill, dense media separation, waste rock, water storage, and worker infrastructure onsite (MVEIRB 2011). The industrial proponent sought to upgrade existing facilities and establish new infrastructure on a previously abandoned mine site within the Nahanni Nature Park Reserve in the Deh Cho region of the Northwest Territories. The Government of Canada had designated the Nahanni Nature Park Reserve as part of the Canadian national park system (but not yet a national park) while they continued to negotiate comprehensive Land Claims with aboriginal groups in the Deh Cho (Notzke 1995; Brown-Leonardi 2012). Aboriginal groups in the Deh Cho region include the Nahanni Butte and Sambaa K’e Dene Bands, the Fort Liard, Fort Simpson, Fort Providence, and Hay River Métis groups, and the Liidlii Kue, Acho Dene Koe, Deh Gah Got’ie, K’atlodeeche, TthedzehK’edeli, Ka’a’gee Tu, Pehdzeh Ki, and West Point First Nations (Aboriginal Affairs and Intergovernmental Relations 2015). MVEIRB held three days of public hearings for the Prairie Creek EA with 66 participants, 23 of whom represented aboriginal communities.

Nico (EA0809-004; 2009–2013)

In 2009, Fortune Minerals proposed the Nico natural resource development as an open-pit cobalt, gold, bismuth, and copper mine with onsite tailings and mine rock disposal, water treatment, concentrate production, effluent and sewage treatment, and worker accommodations (MVEIRB 2013). The proponent sought to establish these mining facilities on land leased within the Tlicho settlement region that had been grandfathered from before the Tlicho Land Claims and Self Government Act (S.C. 2005 c.1). Moreover, the Nico mine would require the eventual construction of a 27-km access road through Tlicho lands by a yet undetermined party (MVEIRB 2013). MVEIRB consulted the Tlicho beneficiaries in all four of the Behchoko, Gameti, Whati, and Wekweeti settlements as interested and affected aboriginal parties. MVEIRB held six days of public hearings for the Nico EA with 144 participants, 87 of whom represented the Tlicho aboriginal communities.

Automated content analyses

The majority of the ACA techniques in this case illustration were conducted with computer scripts using the Python Natural Language Toolkit library, which furnishes programming support for a wide variety of highly complex linguistic analyses (Bird et al. 2009). In order to prepare each of the two EA transcripts for analysis, all stop words (common adjectives and conjunctions like ‘a,’ ‘the,’ or ‘and’) were removed. The transcripts were then parsed into two sets of smaller text files, each file containing all of an individual’s participant testimony during either the Prairie Creek or Nico public hearing. To examine the political context of interactions between aboriginal stakeholders (the public) and other stakeholder groups, the individual files were labeled as belonging to the aboriginal communities, industrial proponents, territorial agencies, federal agencies, or regulators. Three ACA techniques were employed for the case illustration, namely (i) Grade level and testimony proportion, (ii) Correspondence plotting, and (iii) Term frequency–inverse document frequency (TF–IDF) analyses. These techniques were selected as producing text model outputs that spoke to procedural and substantive issues for public engagement, by providing evidence for the relative authority, priority, and salience of the stakeholder groups, the alignment of their testimony, and the corresponding prominence of issue content over the course of either public hearing.

Grade level and testimony proportion

The grade level and testimony proportion ACA techniques provided an approximation of the relative authority, priority, and salience afforded each stakeholder group within the Prairie Creek and Nico public hearings. As a measure of relative authority, average grade level was used to demonstrate the amount of complexity in each stakeholder group’s testimony. This measure was computed by aggregating the grade level for each participant’s testimony to calculate a stakeholder group average, according to the Flesch–Kincaid Grade Level test (Grimmer & Stewart 2013).

where MSL = Mean sentence length; MSW = Mean syllables per word.

Relative priority of each stakeholder group was determined by calculating the proportion of transcript testimony it produced, approximating the amount of access it was accorded in the public hearings overall. This proportion was calculated as a simple ratio of testimony given by a stakeholder group to the total text in each transcript. The salience of each stakeholder group was modeled as the proportion of total lemmatized vocabulary that it utilized over the course of the public hearings. Lemmatization removes the inflection from words and groups them linguistically, so that terms such as ‘operate,’ ‘operating,’ and ‘operation’ are accounted as a single unique vocabulary item (Grimmer & Stewart 2013). Thus, stakeholder groups employing a larger proportion of the total lemmatized vocabulary were considered to be more salient because they occupied more of the conceptual footing for the issues that were raised.

Correspondence plotting

Correspondence plotting as an ACA technique indicated the level of alignment between stakeholder groups in the public hearings. As depicted with the proprietary software program Wordstat (Provalis 2014), a two-dimensional correspondence plot represents the descriptive statistical similarity between stakeholder groups according to the volume and content of their testimony during the public hearings. The correspondence plot axes represent the proportion of statistical correspondence captured in that dimension (Beh 1998). A third, unseen dimension represents a smaller correspondence, and the remaining percentage comprises statistical variability. Aligned stakeholder groups are more proximal to one another in a correspondence plot, while less aligned stakeholder groups are more distal (Grimmer & Stewart 2013).

Term frequency-inverse document frequency

TF–IDF ACA techniques provided a robust measure of the most prominent n-grams (terms or phrases) in the public health transcripts. TF–IDF analysis differs from simple frequency analysis by indicating n-grams of key importance when normalized for their occurrence throughout a document set (Grimmer & Stewart 2013; Peng et al. 2014). TF–IDF has been used by Internet search engines to rank web pages as a measure of their correlation to specified search terms (Peng et al. 2014). It is calculated as the frequency of n-grams in a document multiplied by the log of the ratio of total documents to documents containing the n-gram (Robertson 2004).

where t_i = Frequency of n-grams in a document; N = Total number of documents; n_i = Number of documents with the n-gram.

The 10 n-grams in the Transcripts with the highest TF–IDF scores, their case occurrence, and percent case occurrence indicate the most representative issue content as an outcome in each of the public hearings, which is associated with the political processes between interacting stakeholder groups.

Results

Authority, priority, and salience of the stakeholder groups

In both of the Prairie Creek and Nico transcripts, the grade level of testimony given by the industrial proponents and federal agency stakeholder groups indicated that they spoke with greater authority than aboriginal community members who participated in the public hearings (Table 1). Whereas aboriginal community members communicated their concerns at a grade six to grade eight reading level, the proponents and federal agencies responded to them with a complexity suitable for grades nine to eleven. Notably, the regulators’ testimony was similar to aboriginal participants’ in ranging from a grade seven to eight reading level, indicating a relatively equivalent level of linguistic authority.

Table 1. Number of cases, grade level, number of words, percent of total length, number of unique vocabulary items, and percent of total vocabulary items for each stakeholder group in the Prairie Creek and Nico public hearings in the Northwest Territories, Canada.

	Number cases	Grade level	Number words	Percent length	Number vocab.	Percent vocab.
Prairie Creek
Total	58	–	170,101	100.0	8010	100.0
Proponent	9	10.1	43,307	25.5	4544	56.7
Aboriginal	22	7.8	26,283	15.5	3199	39.9
Territorial	3	9.9	9138	5.4	1796	22.4
Federal	15	10.2	46,993	27.6	4597	57.4
Regulator	9	7.5	44,380	26.1	3168	39.6
Nico
Total	142	–	338,348	100.0	11,540	100.0
Proponent	15	10.8	58,421	17.3	5129	44.5
Aboriginal	87	6.2	154,937	45.8	8408	72.9
Territorial	12	8.1	8892	2.6	1820	15.8
Federal	15	9.5	22,576	6.7	2902	25.2
Regulator	13	7.2	93,522	27.6	5101	44.2

The two transcripts diverged more sharply in terms of the priority accorded to aboriginal community participants in the Prairie Creek vs. Nico public hearings (Table 1). In Prairie Creek, aboriginal participants provided only 15.5% of the total testimony, while in Nico they provided 45.8%. The proponent and federal agency stakeholders assumed a combined priority of 53.1% in the Prairie Creek public hearing, but only 24.0% in Nico. In contrast, the regulator participants had a similar priority of 27.6% in Prairie Creek and 26.1% in Nico. The stakeholder group salience also varied between the two transcripts, for which there was a substantial difference in the amount of lemmatized vocabulary, overall. The Nico public hearing presented 11,540 unique lemmatized vocabulary items, while Prairie Creek had only 8010. Aboriginal participants employed 72.9% of the lemmatized vocabulary in Nico, but only 39.9% in Prairie Creek. In Prairie Creek, the proponents and federal agencies had the greatest salience; in Nico it was the aboriginal participants. The regulators enjoyed a similar level of salience (39.6–44.2%) in both public hearing transcripts.

Alignment of testimony in the public hearings

The correspondence plotting ACA technique indicated two distinct dynamics for stakeholder group alignment in the Prairie Creek vs. Nico public hearings (Figure 1). For Prairie Creek, the correspondence plot depicted 54.8% of the correspondence (29.7% on axis 1, and 25.1% on axis 2), with a third, unseen, dimension (23.3% on axis 3) accounting for a cumulative correspondence of 78.1%. The correspondence plot for the Prairie Creek transcript presents the aboriginal stakeholder group as most similar to the regulators. Otherwise, each of the proponent, and territorial and federal governments occupied statistical positions nearly equidistant from each other, with federal stakeholders most aligned to the aboriginal-regulator cluster in both the axis 1 and 2 dimensions. For Nico, the correspondence plot depicted 67.7% of the correspondence (50.6% on axis 1, and 17.1% on axis 2), with a third, unseen, dimension (16.9% on axis 3) accounting for a cumulative correspondence of 84.6%. The correspondence plot for Nico indicates that while all stakeholder groups were similar in the axis 1 dimension, the aboriginal stakeholder group was especially dissimilar to other stakeholders along axis 2, indicating occupation of a uniquely defining position within that correspondence dimension.

Figure 1. Correspondence plot of the public hearings transcripts across stakeholder groups for the Prairie Creek and Nico natural resource developments in the Northwest Territories, Canada.

Relative prominence of issue content in the transcripts

The relative prominence of issue content in the transcripts also varied between the two Mackenzie Valley public hearings (Table 2). The Prairie Creek public hearing n-grams with high TF–IDF emphasized terminology of regulation (impacts and water quality), operations (paste backfill, road, and management plan), ecosystem resources (wildlife), and contamination (discharge, toxicity, mercury, and tailings). In contrast, the Nico n-grams emphasized terminology of regulation (water quality), operations (access road, closure), ecosystem resources (caribou, wetlands), affected people and places (Tlicho people, Hislop Lake, and Marian River), and information amenable to local knowledge (monitoring, traditional knowledge). Notably, while the range of cases that included each n-gram was similar for both Prairie Creek (11.9–37.3%) and Nico (14.5–36.6%), TF–IDF for Prairie Creek ranged from only 43.2 to 151.6, while for Nico it ranged from 89.3 to 253.5. As well, all of the 10 n-grams with the highest TF–IDF in Nico had higher scores than all but the top three n-grams in Prairie Creek. Because TF–IDF measures the importance of n-grams in characterizing documents (Robertson 2004), higher scores indicate that there was stronger characterization of the identified concepts in the Nico vs. Prairie Creek public hearings.

Table 2. N-grams, term frequency–inverse document frequency (TF–IDF), number of cases, and percent of cases utilizing n-grams across all stakeholder groups during public hearings in the Prairie Creek and Nico environmental assessments in the Northwest Territories, Canada.

N-grams	TF–IDF	Number cases	Percent cases
Prairie Creek
Road	151.6	24	35.8
Water quality	142.6	23	34.3
Discharge	103.1	12	17.9
Toxicity	84.9	8	11.9
Mercury	82.4	11	16.4
Tailings	80.4	17	25.4
Wildlife	77.7	16	23.9
Impacts	75.4	35	37.3
Paste backfill	43.2	11	16.4
Mgt. plan	40.1	9	13.4
Nico
Caribou	253.5	53	36.6
Wetlands	221.1	20	13.8
Closure	185.7	37	25.5
Water quality	175.4	21	14.5
Monitoring	168.5	37	25.5
Trad. knowledge	158.8	31	21.4
Access road	100.0	27	18.6
Tlicho people	94.5	31	21.4
Hislop lake	89.3	42	29.0
Marian river	86.3	25	17.2

Discussion

Implications for public engagement in the Northwest Territories

Natural resource development is accelerating in the Northwest Territories, Canada in response to government policy and the economic value of these resources (Auditor General of Canada 2010). The Northwest Territories Mineral Development Strategy has recommended the institution of an Aboriginal Mining Council to develop plans clearly establishing expectations and procedures for aboriginal communities’ engagement with natural resource developments (Government of Northwest Territories 2014). Drawing on evidence from ACA of the Prairie Creek and Nico public hearings, this case illustration presents a number of relevant implications within the EA statutory framework for the Mackenzie Valley.

Authority was the single metric across the ACA that remained consistent across both of the transcripts. In public hearings, the proponent and federal agency stakeholders consistently communicated at a level of complexity exceeding that of aboriginal participants. In contrast, the regulatory stakeholders were commensurate in their communication in both transcripts. Moreover, the regulators aligned in correspondence with the aboriginal stakeholder group in Prairie Creek, despite the latter’s diminished authority, priority, and salience. Equivalent authority in communication and alignment with potentially marginalized participants are procedural imperatives which the regulatory stakeholder group has demonstrated are amenable to changes in practice (Fitzpatrick et al. 2008). ACA, moreover, can potentially provide a rapid and replicable way to assess how successfully stakeholders implement these imperatives across EAs, by means of evaluating public hearing transcripts.

Variation between Prairie Creek and Nico in the priority and salience of testimony by the aboriginal stakeholder groups may reflect certain contextual realities with perhaps less immediately practicable implications. While the Tlicho Land Claim has been finalized (along with the Inuvialuit, Gwich’in, and Sahtu agreements), the Deh Cho Land Claim is still currently under negotiation (Aboriginal Affairs and Intergovernmental Relations 2015). In the Nico public hearings, the Tlicho beneficiary participants possessed a level of legal empowerment that may have enabled leveraging the public hearings to better reflect their aboriginal world views (Armitage et al. 2008). The need for self-determination has long been recognized by aboriginal peoples worldwide as a prerequisite to meaningful consultation (Gamble 1978; Macintosh 2012; Hanna & Vanclay 2013), and court cases have been pursued in regions without settled Land Claims (such as the Deh Cho) where communities deemed public engagement for natural resource development to be unsatisfactory (Auditor General of Canada 2010). Legal disparities resulting in structural barriers to public engagement may present an largely intractable procedural issue, although certainly one which the proposed Aboriginal Mining Council may wish to address.

Further, contrasting issue content between the Prairie Creek and Nico public hearings parallels a distinction between technical (single loop) and conceptual (double loop) learning in the natural resource management literature (Cundill & Rodela 2012). The most prominent n-grams in Prairie Creek were highly technical in their orientation, while in Nico they were more conceptual (Table 2). Whereas technical learning reflects the adoption of well-established procedures based on scientific evidence and technical expertise to address a problem and achieve a goal, conceptual learning involves a shift in understanding a problem itself, in order to reframe objectives, strategies, and even processes of decision-making (Fitzpatrick et al. 2011). Circumscribing local knowledge with scientific and technical input is potentially a limitation of the substantive value of the Prairie Creek public hearings. For the long-term substantive sustainability of northern EAs, it is important that all stakeholders in EAs across the Northwest Territories prioritize both the technical and conceptual learning frames to promote public engagement.

Future directions for EA research and practice

The case illustration provides a demonstration of the utility of selected ACA techniques in producing a rapid, replicable, inductive, and systematic way to synthesize evidence contained in the extensive documentation for public hearings conducted as part of EAs. Future directions in EA research employing ACA techniques can facilitate both retrospective and prospective studies highlighting lessons learned and opportunities for procedural and substantive innovation. Importantly, public engagement features prominently in emerging paradigms for adaptive governance in natural resource management (Crona & Parker 2012). From the perspective of the socio-ecological approach advocated in the Ottawa Charter (World Health Organization 1986), public engagement can provide local knowledge of the interface and exposure pathways between humans and the environment to support increasingly complex decision-making (Craig & Ruhl 2013). In addition to its capacity as a tool to retrieve and categorize this local knowledge, ACA can provide insight on the problem of conceptualizing and measuring procedural and substantive learning as a key feature of adaptive governance (Crona & Parker 2012). Automated techniques can support a more extensive (and less costly) application of emerging tools and methods, such as comprehensive evaluation frameworks for public engagement contextualized to regional or country settings (Nadeem & Fischer 2011), by mechanically filtering and aggregating a variety of data-sets. As automated techniques become more integrated with other EA research methods, comparative and meta-analytical research may help to identify ACA applications enabling broader generalization across jurisdictions and contexts. Moreover, EA researchers working in this area should strive to foster the advancement of interdisciplinary ACA techniques, drawing upon developments in fields like computer and political science.

There are also a number of current practice implications for the use of ACA across EA settings. Automated content analyses may help support the transparency of public hearing processes, permitting more uniform documentation of evidence synthesis as part of administrative discretion in formulating EA recommendations (Green 2014). As well, each stakeholder group participating in EAs can potentially use ACA to bolster procedural and substantive values in public hearing processes. Regulators can develop ACA protocols specific to their jurisdiction to identify procedural and substantive issues that emerge during public hearings, in order to devise appropriate incentives for stakeholder learning and practice (Irvin & Stansbury 2004). Proponents can utilize ACA to improve public engagement by learning to more effectively respond to the challenges faced and issues raised by members of the public who participate in public hearings (Morrison-Saunders & Early 2008). Government agencies might use ACA to understand and respond to the extent that public hearing testimony aligns with their administrative mandates, independent of the summaries provided by EA regulators. Members of the public, particularly in association with civil society, can leverage ACA techniques to identify opportunities to improve public hearing processes, as well as to demonstrate high levels of interest and concern in particular issues. As ACA techniques are replicable, there is also potential for cumulative insights fostered by the aggregation of findings, enabling stakeholder groups to more effectively communicate and work together.

Limitations of ACA

ACA is inexpensive, empirical, and extensive; however, it is rarely as detailed or reliable as inductive (such as grounded theoretical approaches) or deductive (such as application of an ‘a priori’ coding frame) qualitative content analyses (Budge & Pennings 2007; Lowe et al. 2011). Moreover, meaningful interpretation and the integration of automated text model outputs will require problem-specific validation in every case, most often by carefully situating the results within a relevant context (Grimmer & Stewart 2013). It is therefore important for researchers and practitioners who use ACA to articulate the assumptions and approximations that underpin each technique they choose to employ, and to provide an explanation for the combination of techniques they have selected. Essentially, there can be no substitution for the analysts’ critical thought, which is prone to many of the same potential pitfalls of subjectivity and bias found in all forms of research and practice (Petticrew & Roberts 2003). Importantly, the use of ACA techniques to synthesize the evidence of engagement in public hearings depends upon attaining a level of access and participation that is usually found only in jurisdictions with more advanced EA frameworks (Morrison-Saunders & Early 2008). Where levels of participation in EA public hearings are low, innovative approaches and alternative opportunities for public engagement (such as identifying potential impacts through interactive internet mapping platforms) have demonstrated potential (Fischer et al. 2009) that can further be examined using ACA techniques. Finally, the successful generation of text models correctly situated within the appropriate interpretive context is no guarantee that stakeholder groups’ recommendations derived from these outputs will gain appropriate traction in the implementation of public engagement practice (Rowe & Frewer 2005). However, ACA techniques, in combination with other tools and methods, can clearly contribute to enhancing the policy development cycle for EA research and practice.

Conclusion

This case illustration is among the first to apply ACA techniques to the extensive documentation of EA public hearings, demonstrating how these techniques can support procedural and substantive learning to improve processes of public engagement. Although features of the case illustration were specific to public engagement in the northern Canadian context, broader issues that emerged can inform the development of ACA techniques in other jurisdictions. Public engagement in EA is inherently political, and its successful implementation depends upon identifying opportunities within multi-stakeholder processes that support meaningful integration between local knowledge and a range of scientific evidence and technical expertise. Moreover, the usefulness of ACA techniques will vary by jurisdiction, with regions employing more advanced forms of public engagement in EA providing greater availability of information upon which to develop automated analyses. EA researchers and practitioners who employ ACA techniques may benefit from the rapid, replicable, inductive, and systematic approach to large volumes of testimony that reduces the time and resource requirements associated with analyses that are more qualitative in origin. Nevertheless, there is still a requirement for critical thinking in the meaningful interpretation of ACA for the context in which it is used. Our case illustration provides a useful starting point for encouraging the incorporation of ACA techniques as part of EA research and practice.

Disclosure statement

No potential conflict of interest was reported by the authors.

Funding

This work was supported by the Networks of Centres of Excellence of Canada [ArcticNet ‘Integrating and Translating ArcticNet Science for Sustainable Communities and National and Global Policy and Decision-Making’]; the Polar Knowledge Canada Northern Scientific Training Program [grant number 360150]; and the Canadian Circumpolar Institute Circumpolar/Boreal Alberta Research Program [grant number N011000001].

Acknowledgements

Fieldwork in the Northwest Territories was completed under licensing by the Aurora Research Institute License No. 15242 (2013) and 15409 (2014). Ethical approval for research in the NWT conducted concurrently with this project was obtained from the Human Research Ethics Board 1 at the University of Alberta. Jennifer Ann McGetrick was supported by a Queen Elizabeth II Graduate Scholarship at the University of Alberta (2011–2014).