Framing an independent, integrated and evidence-based evaluation of the state of Australia's biophysical and human environments

Abstract

A new approach was developed for Australia's 2011 national State of the Environment (SoE) report to integrate the assessment of biophysical and human elements of the environment. A Common Assessment and Reporting Framework (CARF) guided design and implementation, responding to jurisdictional complexity, outstanding natural diversity and ecosystem values, high levels of cultural and heritage diversity, and a paucity of national-scale data. The CARF provided a transparent response to the need for an independent, robust and evidence-based national SoE report. We conclude that this framework will be effective for subsequent national SoE assessments and other integrated national-scale assessments in data-poor regions.

Keywords:

• environment report card
• conceptual framework
• integrated performance assessment
• expert elicitation
• consultative process

1. Introduction

Australia is the world's largest island continent and sixth largest country, with jurisdiction and management authority over 27.45 million km², including a landmass of 13.59 million km² and an associated marine zone of 13.86 million km² stretching from the tropics to sub-Antarctic regions (including the Exclusive Economic Zone, Extended Continental Shelf, and Australian Antarctic Territory: Symonds, Alcock, and French 2009). The environments encompassed by this area of land and sea include many tangible and intangible assets and values, including mineral and natural resources, natural and cultural heritage values, and a range of ecosystem services for generation of wealth, lifestyle, well-being, recreation and cultural appreciation (SoEC 2011). The assets and values are also represented by many iconic and globally recognised unique features, including 19 World Heritage sites. The ecosystems and biodiversity are exceptional as they comprise 7‒10% of all known species, including more than 17,000 species of flowering plants and 33,000 known marine species (Steffen et al. 2009; Butler et al. 2010). The geologically ancient landmass is diverse, with cultural traditions that extend back thousands of years (Blewett 2012, 571). This terrestrial and marine domain is now overlain with 200 years of colonial and post-colonial activity and heritage, particularly across the coastal landscapes where the majority of Australians live within the seven largest cities (SoEC 2011).

The Australian Government's Environment Protection and Biodiversity Conservation (EPBC) Act requires a State of the Environment (SoE) report to be prepared every five years on the nation's ecosystems, natural and physical resources, heritage value and quality of places, and relevant social, economic and cultural aspects. The fourth such assessment and report, conducted by an independent committee (SoEC 2011) was built on the foundations provided by earlier national SoE assessments and by state and regional-level environment reports (e.g. GBRMPA 2009; Dobbs et al. 2011). The 2011 SoE report extended the assessment system of earlier reports to encompass more fully the expectations of the EPBC Act and to provide a more comprehensive report. The overall Driving Forces, Pressure, State, Impact, Response (DPSIR) approach adopted for the SoE 2011 report is a performance assessment consistent with modern environmental reporting (Smeets and Weterings 1999), but we incorporated several new aspects, layers of information and a process culminating in summary report cards for each theme. In this respect, the SoE 2011 report was designed to be both technically factual and readily accessible to a wide variety of stakeholders, employing information products at multiple levels in both print and electronic formats. In addition to meeting national objectives, national SoE reports play an important role as source documents for Australia's international reviews and reporting obligations, such as the OECD's environmental performance reviews and Australia's report to the UN Convention on Biological Diversity.

Australia's national SoE process assesses and reports on condition, trends and pressures, but does not make management recommendations nor discuss policy responses/options to any issues that may be described in the report. In that sense, the SoE report is constrained to be assessment, synthesis, evaluation and reporting at the national scale. This provides policy makers with a robustly derived set of issues but does not constrain the types of policy responses that may be invoked. The intention of this approach is to generate an independent perspective on environment performance of the assets and values, to foster a national conversation about issues and their fundamental drivers relevant to these assets/values, and to provide an agreed platform of knowledge that can be used by policy makers to engage with stakeholders about policies and appropriate supporting management strategies to address the issues.

It is difficult to design and apply a single assessment and reporting system that integrates and is equally effective for the biophysical and human elements of highly complex natural ecosystems, built environments and cultural heritage at a national scale. High-level and strategic evaluation of these systems at the national scale requires:

• Identification of metrics that represent the intrinsic attributes of the systems;

• A process for estimating their condition and aggregation to achieve a synthesis and summary of the available information; and

• A reporting modality that is accurate, accessible and timely with direct utility for national-level policy and management initiatives.

In high-value, large and complex natural and urban systems such as that of Australia, careful attention to core ecological and equity principles is needed to guide the content of the process (e.g. Foley et al. 2010), as well as establishing the context and uncertainties. For a low-bias assessment, the use of a small number of well-known system attributes has to be balanced against the use of a greater number of less reliable attributes that represent different aspects of the systems being assessed (to avoid high levels of uncertainty in model structure, sensu Walker et al. 2003, and minimise the potential for Type III error, e.g. Bark et al. 2013). There needs to be an explicit trade-off between detailed information on a limited subset of ecosystem and human attributes (which may have been studied for purposes unrelated to national policy) and lower-resolution information on a broader range of attributes and policy responses that are more relevant to the broader range of issues relevant to an integrated national-scale SoE assessment

In this paper we outline the framework for the integrated assessment system we developed for Australia's 2011 national State of the Environment report, with selected examples of the process detail drawn from the nine assessment themes. With a focus on the processes of consultation, data capture, and synthesis and interpretation, we describe how the national-scale reporting was designed to respond to policy-driven requirements. We also consider a broad array of system-level attributes to provide support for a systematic and integrated approach to environment reporting. This experience is synthesised into a logframe (Team Technologies 2005) that may be of value for guiding future national SoE reports in Australia as well as environmental assessments at other scales and jurisdictions where integration of biophysical and human elements of the environment is required.

2. Conceptual reporting approach

The SoE 2011 report principally comprises information on drivers (key activities and pressures on the environment); a series of theme reports that consider condition (state), pressures, management responses, resilience, risks; and culminates in an outlook for the future (SoEC 2011, 28). This structure, modelled on the Great Barrier Reef Outlook Report (GBRMPA 2009), provides consistency in most themes with previous SoE reports as well as the opportunity to develop a broadly-based environmental outlook. We adopt much of the European checklist approach (Kristensen, Anderson, and Denisov 1999), although the assessment and project management aspects are uncoupled to ensure there is a clear separation between the commissioning government agency (Australian Government Department of Sustainability, Environment, Water, Population and Communities; DSEWPaC—now Department of the Environment) and the independent committee appointed to be responsible for the report content.

A unified assessment and reporting system (the Common Assessment and Reporting Framework ‒ CARF) was established to provide for assessment outcomes that were consistent across all themes irrespective of their environment attributes. Based on performance assessment metrics (sensu Smeets and Weterings 1999), the assessment system provided a single type of finding in each theme, summarised into ‘report card’ format, enabling easy and direct comparisons of the assessment outcomes amongst themes (and, in the future, within themes). Integration was achieved by applying the common assessment and reporting system to all aspects of all themes, including the biophysical and human environments.

A central element of the CARF is the establishment of benchmarks, against which performance grades are assigned. For the natural systems being assessed, intrinsic ‘naturalness’ benchmarks were applied to represent a near-pristine set of the conditions that prevailed at the time of European settlement of Australia (notionally about 1800). In the human-environment constructs, such as the heritage and built environment themes, intrinsic natural attributes are not easily discoverable or estimable. Although these human constructs operate within the context of a natural environment that provides various environmental services (provisioning, regulating, cultural), there are nonetheless few appropriate natural or pristine surrogates that can serve as intrinsic benchmarks for assessment. As a result, perception-based benchmarks for attributes of the environment such as integrity of heritage values, and urban liveability were developed and applied in the heritage and built environment assessments respectively. In the atmosphere theme, target-based benchmarks were applied in relation to human health, for attributes such as levels of ambient air quality that impact human health. In the land theme, soil condition was assessed against the baseline of condition prior to vegetation clearing. All the benchmarks were chosen to be consistent with precedents in national assessments in Australia. The CARF therefore enabled each theme to estimate a measure of current condition quality (inter alia) as ‘distance’ of the current condition from a measured or estimated set of the benchmark conditions relevant to the theme under assessment.

For all themes (Table 1), the central technical issue revolved around development of a CARF that would robustly deal with the institutional and subject-matter complexity and a paucity of available data and information. There were a considerable number of relevant datasets available to some themes, but much of the information either did not relate specifically to the metrics, scale or scope required, or could not be synthesised and made available to the assessment in the required timeframe. In addition to the lack of an adequate knowledge base (or form of quantitative data or surrogates) to resolve the condition, pressures, and trends within the CARF at a national scale, the assessment process was also constrained by the need for the report to be prepared and concluded within a two-year timeframe.

Table 1. Australia's SoE 2011 assessment themes and their main focus.

Theme	Focus
Atmosphere	Climate, greenhouse gases, ambient air quality, indoor air quality, stratospheric ozone.
Inland Water	Inland river, lake, wetland and aquifer environments and ecosystems.
Land	Soil, vegetation and land use systems.
Marine Environment	Marine environments, ecosystems and biodiversity.
Antarctic Environment	Environments, ecosystems and biodiversity of Australia's Antarctic Territory and sub-Antarctic islands.
Biodiversity	Species, environments, native vegetation, ecological communities and ecosystem services.
Heritage	Natural heritage and reserved lands and waters, Indigenous heritage, historic heritage places.
Built Environment	Liveability and efficiency associated with human-made physical structures and the environmental regulating, provisioning and cultural services that support these.
Coasts	Pressures that impact the coast assets and values.

Transparency was a key principle underlying the preparation and reporting system for this SoE, not only to provide for both an appropriate level of accountability in expenditure of public funds, but also to document in the public domain the approach and assumptions underpinning the assessments, the process undertaken and the findings. A high level of transparency was also important for an easily accessible archive (at www.environment.gov.au/soe), so that: (a) the assessment process could be efficiently replicated (and improved where necessary) to permit the five-year scales of change to be estimated at the next SoE assessment in a comparable way; and (b) public-domain contestability was feasible to promote continuous improvement of the assessment system.

3. Design of assessments

The EPBC Act is silent about the process that should be used for SoE assessments. Since the inception of national SoE reporting in Australia (1996), every five years governments have appointed and relied upon a small, independent committee of experts to guide and oversee assessment and production of a report that complies with the requirements of the Act. For the 2011 SoE report, the national environment minister appointed a group of eight experts to the SoE committee to represent each of the main themes, and co-opted a member of DSEWPaC staff with special expertise in Antarctic issues (SoEC 2011, 24). Resources were provided by DSEWPaC, including operating funds and support from dedicated staff, including technical and administrative staff who assisted with research, the logistics for the committee operations, report production and inter- and intra-government liaison. The committee was also supported by commissioned research, case studies, compilations of statistics, analyses drawn where possible from existing data and analytical products of national and state government agencies. While the independence of the assessment process is not mandated in the EPBC Act, independence, transparency, wide consultation and technical robustness provide for a measure of ‘arm's length’ assessment, and this was adopted by the committee as a form of assessment and reporting ‘best practice’ that was particularly of importance for maintaining relevance to community expectations.

The broad approach and design of the SoE process and report were developed by DSEWPaC in conjunction with the committee, including the overall framework and policy-derived principles for the process, the types of products that would be needed, and the extent of consistency with earlier SoE reporting products. The 2011 SoE process resulted in four main products:

1. The main written document, which contains a set of definitive findings in each theme;

2. An ‘in-brief’ written document that essentially provides a summary of the main report;

3. An electronic on-line resource with supporting and additional information not able to be included in the printed report; and

4. A series of stakeholder engagements and presentations following release of the final report.

In addition to presentation in a format consistent with the DPSIR approach, all the products use language and concepts familiar to policy makers and the general public. They were designed to be useful for at least five years in secondary and tertiary education, in local and state government, and as overview material for use in research contexts in the private and public sectors. Key high-impact facts were developed into simple graphics, designed to be accessible for all readers (e.g. the number of places added to the National Heritage List in each year from 2005‒06 to 2010–11: SoEC 2011, 707).

A consistent format for the report card was used to provide a more integrated overview of condition, pressures, trends and confidence (SoEC 2011, 29). The report card presented aggregated and summarised information, using either three or four performance grades, supported by short pieces of text to highlight the main underpinning arguments or evidence. This reporting requirement was also used as the basis for structuring consultation and information capture, although applied differently across themes depending on the availability of data and information, and the type of issues to be addressed in each theme. Each theme assessment involved extensive discussions with government agencies and specific consultation about the assessment approach, direction and data-sources. During the planning and writing periods, theme authors consulted extensively with technical peers and related experts through both individual and workshop sessions to determine the availability of information, to fill knowledge gaps and to determine which data were relevant for reporting.

Uncertainty in the sense of precision and accuracy has been identified, declared and addressed as a core activity in the SoE 2011 process, recognising the importance of this for providing outcomes that are relevant to environmental reporting (Udovyk and Gilek 2013). Precision of the findings (statistical uncertainty, sensu Walker et al. 2003) was established within the consultation process ‒ authors and experts assigned their own estimates of confidence using a confidence structure established in the CARF. The accuracy of the expert opinion (model outcome uncertainty, sensu Walker et al. 2003) was checked by verification against extant technical data where that was practical, a substantial feature in some themes. To minimise the likelihood of substantive inaccuracy, a broad base of experts was consulted within a systematic process using the CARF reporting template that constrains outputs to a maximum of four grades of performance (five nationally-agreed grades in the case of urban air quality) in each metric. In addition, opinion accuracy is traded-off against resolution in the issues by keeping the assessment and reporting focused on a broad base of intrinsic assets and values, thus reducing the risk of decision model failure that could arise from high levels of model outcome uncertainty. Finally, considerable resources were devoted to an independent peer review process to check on the structure, content and findings from each theme.

4. Implementing the assessments

The process of designing and implementing the CARF was developed incrementally, and is summarised here in a scale and theme-independent logframe (Team Technologies 2005) (Table 2). In each theme different forms of data and information were available for analysis, including reports on various aspects of pressure, condition and trends in the environmental assets and values. However, much of the available information consisted of only partial assessments, covering limited topics or areas of the nationally-relevant sets of issues. Therefore, each theme, while consistent with the CARF, took a different path to select, assemble and assess data and information (Table 2), informed mainly by the context of the issues, the actors, the time/resources available for the reporting process, and the need to reach findings consistent with the CARF report card format.

Table 2. The Common Assessment and Reporting Framework (CARF) for Australia's SoE 2011 report: a scale-independent project logframe.

Phase	Step sequence of activities		Objective intended achievements	Policy Driver legislative mandate and requirements	Actors organisations and players involved	Output the desired product	Purpose* attributes of the project serviced	Performance Measure indicative step-wise KPI
Preparatory	1a	Establish purpose for assessment and reporting	Confirm the policy mandate and scope for assessment and reporting, and identify target audiences	Interpret and operationalise the legislative mandate	Australian Environment Minister; DSEWPaC executive; other Australian Government agencies	Whole of government commitment to process	Clients, Relevance, Coordination & Facilitation	Summary statement of purpose; confirmed with key actors
	1b	Identify the set of key stakeholders, and their specific interests	Establish context of the process, specific sectors with an interest in the findings, and specific points of engagement	Ensure that key informants and stakeholders are effectively engaged to build ownership of process and outcomes	SoE community, national and state government agencies, resource users, NGOs, tertiary science/knowledge base	Stakeholders comitted to process	Clients, Relevance, Credibility, Communication, Coordination & Facilitation	Summary statement of stakeholders and their specified interests; confirmed with key actors
	1c	Audit existing data, reports, and benchmarks; secure additional topic reviews where needed	Establish the nature and extent of existing knowledge; fill key gaps in knowledge	Ensure data and knowledge base is current and best available for assessment	SoE community, national and state government agencies, resource users, NGOs, tertiary science/knowledge base	Inventory of data and knowledge, agreed with key actors	Clients, Relevance, Credibility, Communication, Coordination & Facilitation	Summary statement of existing knowledge and sources; confirmed with key actors
Consultation	2a	Set boundaries, be clear about what is in and out of scope; eg spatial and boundaries for the area(s) to be assessed and any sub-divisions that might be needed	Spatially bound the assessment and reporting process and its broad components, for clarity about what areas/assets/values are included/excluded; secure agreement with key stakeholders	Implement the mandate for national environment assessment and reporting consistent with ecosystem based management and reporting	National and state government agencies, NGOs, tertiary science/knowledge base subset (societies, experts, workshop participants)	Bounding the biophysical, issues and governance systems covered by the report	Relevance, Credibility, Communication	Maps, process structure and boundaries agreed with key actors
	2b	Parameters and decision model: identify and agree on typology - aspects to be assessed, the grading scale to be deployed, the baselines/reference points, rules for grading decisions, and form of final reporting	Match the reporting capacity/expectation to the biophysical structure of assets and values; clearly establish the distinction between assessment of the environment vs assessment of sustainability	Operationalise the reference/benchmark framework for assessment decisions relative to policy needs/mandate	Tertiary science/knowledge base subset (experts, workshop participants)	Agreed set of metrics to be covered by the report, and rules governing assessment process decisions	Credibility, Communication	Typology and decision model agreed with key actors
Data capture	3	Prepare information base, assign scores/grades through workshops where specific findings can be tested with peers, iterate workshop outcomes for verification and confirmation	Assign credible and defendable grades to the ecosystem assets and values with a known level of accuracy and confidence	Produce robust assessment decisions in a report that is credible and defendable	Tertiary science/knowledge base subset (experts, workshop participants)	Credible and defendable assessment decisions	Credibility, Relevance	Raw workshop outputs
Synthesis, interpretation	4	Conduct a detailed scoring analysis to support report card output	Synthesise and summarise the assessment findings	Produce assessment decisions in a report that is accessible and effective for the target audiences	Lead agency; supporting facilitator	Credible, accessible and effective assessment report	Credibility, Communication, Coordination & Facilitation	Synthesised workshop outputs
Production	5a	Peer review	Secure quality peer review of findings, independent of theme authors	Produce report that is credible and defendable	DSEWPaC, independent reviewers with qualifications and experience at least equal to those of the theme authors	Report of peer review, response from theme authors	Credibility, Relevance	Reviewer reports, authors response reports, consequent updating of draft report
	5b	Production	Range of accessible products that engage with target audiences	Report that provides strategic decision support and is effective for national-level policy making	Lead agency; supporting facilitator; communications team	Draft SoE report	Communication, Relevance, Coordination & Facilitation	Draft SoE report
Release, outreach	6	Outreach with report card findings and policy implications	Direct engagement with target audiences	Report that provides strategic decision support and is effective for national-level policy making	Australian Environment Minister; national government agencies; NGOs, media team	Issue discussion amongst policy makers and the broader Australian community	Communication, Relevance, Coordination & Facilitation	Level of public engagement
Feedback	7	Formal evaluation of report process and effectiveness	Performance-based feedback on effectiveness, and to guide future improvements for the design and implementation of future national SoE reports	Estimate the effectiveness and impact of the process and the report, relative to the mandate, and contribute to ‘return on investment’ and ‘value for money’ public funds expenditure assessment	Australian Environment Minister; lead agency; other national government agencies	Guidance for incremental improvement of the assessment and reporting process	Communication, Relevance, Coordination & Facilitation, Impact	Public domain report of formal evaluation process
Iteration	8	Repeat comparable process, 5 yearly cycle	Enable adaptive ecosystem-based management through time series analysis of environment condition and trends	Interpret and operationalise the legislative mandate	Australian Environment Minister; DSEWPaC executive; other national government agencies	Whole of government commitment to process	Clients, Relevance, Coordination & Facilitation, Impact	Summary statement of purpose; confirmed with key actors
*Project Attributes and		Clients: ensure that all interested and contributing actors are identified, have the opportunity to input, and have access to outputs
their defining features		Relevance: Maintain the focus of the process and the assessment on the primary environment issues and matters that relate to an objective, robust and independent assessment with accessible outputs
		Coordination/Facilitation: there is an effective set of arrangements with capacity to ensure a high level of coordination is maintained by the process, and facilitation of client consultation is effective
		Credibility: the assessment process is transparent, independent, evidence-based, is conducted to a high technical level of robustness and independently peer reviewed, and issues findings bounded within an appropriate envelope of uncertainty and confidence
		Communication: the interim and final products of the assessments are appropriately framed into accessible language and concepts, and made available to the community and the actors in a programmed manner to enable informed and respectful analysis and public feedback
		Impact: the demand for print and electronic SoE products is tracked, and there is an assessment of citations in the various technical journals and policy development domains that may indicate uptake and the type and level of policy response

The initial consultation tasks involved identification and engagement with an appropriate set of experts to satisfy the dual purposes of securing ownership and engagement with the assessment, and identifying data and information sources. The data and information both suitable and available for assessment were identified and obtained from within government and non-government organisations, and through the experts consulted. Experts were also consulted in some themes to assist with setting the spatial and structural boundaries, with the establishment of a typology for the assessment (such as assisting to establish an assessment structure of parameters, components and metrics), and with development of appropriate surrogates for reference benchmarks. This consultation also helped to ensure that the assessment typologies established links between condition and trends in the environmental assets/values and the management frameworks, through relevant performance measures that could be useful in the development of policy responses. The diversity of approaches used by the assessment themes to implement the consultation, data capture and synthesis aspects is described here using five theme examples.

4.1. Built environment

The built environment essentially refers to human-made constructs, albeit within the context of a natural environment that provides various environmental services to these constructs and the people who live within them. This posed challenges for the assessment process, as these attributes of the built environment are generally neither directly discoverable nor observable in nature itself and there is no natural state to provide a benchmark for making assessments.

After the identification of relevant experts, based on discipline expertise and coverage of the potential issues, the initial consultation process focused on clarifying the scope of the chapter, including attributes of the built environment for which assessments would be made, and identifying data sources that could assist in the assessments. There are considerable data on the economic and social aspects of the built environment, but little data to inform the environmental dimension, particularly at a national level, and so informed expert opinion was needed in order to develop an assessment that conformed to the CARF.

A workshop of the experts was conducted to prepare the assessments. For the assessment of state, a matrix of population size (representing groups of urban areas) by built environment attribute (such as urban amenity, transport, housing, etc.) within the two main components (liveability, urban environmental efficiency) was developed. For each cell of the matrix, the available relevant evidence was considered along with expert opinion to determine both a grade and trend, expressed in terms of an agreed grading scale (SoEC 2011, 821). Environmental attributes were aggregated to determine an overall grade for each population size group, rather than aggregating across population size groups to determine an overall grade for each environmental attribute. Confidence levels were assigned to indicate the extent of consensus in the ratings of the individual cells in the matrix and the consistency of ratings across the environmental attributes in the aggregation process. The comments section of the assessment summary was used to identify key determinants in the grading process, with the narrative and data presented in the chapter itself providing a more comprehensive evidence base for the assessment grades and trends.

While there were differences of opinion among the experts who were involved in the assessment process, the structured approach that was used delivered assessments that considered all of the available evidence and resulted in a balance of opinions of the experts, including a representation of all relevant forms of uncertainty.

4.2. Heritage

Heritage is a subset of the wider natural and cultural environment which is perceived as a valued inheritance to be passed on to future generations. For the purposes of SoE 2011, natural heritage was regarded as those lands which have or should be reserved for conservation purposes. For both Indigenous and historic heritage, SoE similarly addressed not only formally listed places (on statutory and non-statutory registers) but also those places which warrant heritage listing. Assessment of the current state of Australian heritage involves what values have been identified and their current condition.

Evaluation of the condition and trend was undertaken using two interrelated and complementary methodologies: commissioned expert assessment, and a series of expert workshops. For the 2001 and 2006 SoE reports, the condition and integrity of a small stratified sample of places on the Register of the National Estate were assessed (Pearson and Marshall 2011). For SoE 2011 this study was repeated, and extended to cover natural and Indigenous heritage (ERM 2011; Pearson and Marshall 2011; Schnierer, Ellsmore, and Schnierer 2011).

A series of workshops were convened with representatives from peak government and non-government bodies in each heritage sub-theme: natural, cultural and historic. Participating agencies included the Australian Committee for IUCN, the ‘Heads of National Parks’ forum, state heritage officials, Australia ICOMOS and the DSEWPaC Indigenous Advisory Committee. At these workshops, a series of open questions were posed, leading to identification of matters relevant to the assessment of condition and trend of Australia's heritage, as well as individual assessment of the agreed metrics. In the majority of cases, additional data sources or informants were also identified during these workshops. The workshop notes were circulated to participants for verification and published on the SoE 2011 website (http://www.environment.gov.au/soe/2011/report/heritage/index.html). The Australian Heritage Council participated in two workshop discussions ‒ the first to discuss the methodology, priorities and structure of the heritage theme chapter, and the second, an assessment and evaluation similar to that conducted with the other peak heritage bodies.

Considerable structuring of available data and workshop expert opinion was needed to ensure conformity with the CARF. Allocation of grades and data confidence levels was undertaken separately by the heritage theme author, as the assigned grades and confidence levels were also informed by the external condition and integrity reports.

In addition to reporting through the report cards of the CARF, many of the conclusions in the heritage theme chapter are demonstrated through case studies. There was strong consensus expressed in the workshops regarding important issues, circumstances or trends, but often no empirical data. The report card grades or trends are therefore also supported by examples that ‘prove the point’ (e.g. incremental destruction of Indigenous heritage: SoEC 2011, 737).

4.3. Biodiversity

Biodiversity is defined as the variety of life, including the diversity of species and the genetic material that they embody, and the aggregations of species and their interactions with the non-living world that constitute ‘ecosystems’ and landscapes. Successive Australian SoE reports have noted that limited data are available on biodiversity at national scales. While information is collated in various databases at a national scale, the collection of information on biodiversity (such as in field surveys) is performed by state-level government agencies and non-government organisations (e.g. Birds Australia) that may be collections of local groups. Past national SoE reports have collected published and unpublished information on aspects of the pressures on biodiversity and the changes in state that those pressures induce, but this information has rarely been collected with a view to supporting inferences at a national scale.

The assessment of biodiversity in the SoE 2011 report drew primarily on three sources of information:

1. A review of conclusions about drivers, pressures, state, impacts and responses at the state level by each of the state governments in their most recent SoE reports;

2. Assessments of biodiversity made in other chapters of the SoE 2011 report (e.g. the marine chapter conducted an extensive expert consultative process, the inland water chapter drew on recent reviews of biodiversity in Australian rivers, and the land chapter reviewed current information on native vegetation);

3. Key recent reviews of some groups of species at a national level, including a review of representation of terrestrial ecosystems in Australia's protected area system (Taylor et al. 2011), an independent report to the Australian Government about Australia's terrestrial biodiversity (DEWHA 2009), and a national assessment of the state of Australia's birds (Olsen 2008).

Given this information base, which included several previous expert consultative processes, the Biodiversity theme authors made judgements about grades to be applied in the report cards in relation to pressures on biodiversity and state and trends. The levels of consensus among experts and the amount and reliability of information were made explicit in the confidence indicators in the report cards. Where there was limited evidence or consensus, which was a frequent occurrence, specific experts were contacted to determine if relevant information had been overlooked.

This approach did not avoid the problem that in many cases information was inadequate to support unequivocal conclusions, but it did allow identification of uncertainties, where the strong conclusions could be made, and where it was most critical to obtain new information to support strategic decision-making. While most jurisdictions understand the nature and implications of pressures on biodiversity, there was a general pattern of inadequate investment to meet objectives and a failure to achieve desired outcomes for biodiversity. This was recognised as a major issue, which points to the need for critical examination of management effectiveness in relation to biodiversity.

4.4. Marine environment

The available marine data for the national assessment related to a species complement of only a few hundred species and habitats, representing, for example, less than 1% of the known species that occupy Australian marine waters. Further, most of the available species-level datasets were highly spatially biased towards intensively studied shallow water areas, or structurally biased towards data that was required for natural resource management (such as for fisheries) rather than intrinsic ecological attributes of populations or habitats.

An extensive expert consultative process was designed because of the paucity of suitable public domain marine biological data and information. The primary objective of this was to secure a set of low-bias expert judgement assessments of the available marine data and information in relation to the marine assets and values. At least two currently active and field-experienced experts able to attend workshops were identified for each broad discipline/issue area. The consultative process involved establishing an assessment typology to represent the assets and values of the entire marine jurisdiction that was not biased by the extent of available data, and then conducting a series of three assessment workshops. The experts assigned scores/grades to condition, trends, pressures and confidence, and discussed their consensus assignments in the presence of their peers. The outcomes of the assessment process are summarised in the SoE 2011 report (SoEC 2011, 388) and are presented in more detail elsewhere (Ward forthcoming).

Information about each of the metrics (a hierarchical set of parameters, components and indicators) in the assessment was sought from the invited experts. Their grading judgement was assigned using pre-agreed assessment procedures and grading scales (SoEC 2011, 394), and this then provided the primary data for assessment. The opinions and judgements were contested and verified, sometimes extensively, at the workshops, in order to ensure that judgements could be considered as a consensus of the participating experts and be based on the available data and information (Ward 2011, Ward forthcoming).

The data provided by the experts were aggregated into high-level graphical summaries. Summary statistics of the unweighted data and non-parametric statistical tools were used to avoid implicit weighting and complex indices that might bias outputs beyond that established by the explicit architecture of the assessment. All the data were standardised to a single grading scale for the report card, consistent with the CARF (SoEC 2011, 392) and have been also used for more detailed analysis, providing more complex overviews that integrate condition, trend and information quality, to inform the development of integrated policy analysis and responses (Ward forthcoming).

4.5. Antarctic environment

The natural and cultural values of Australia's Antarctic Territory (AAT), the Australian territory of Heard Island and the MacDonald Islands, and the waters surrounding these areas, are managed by the Australian Antarctic Division (AAD) of the Department of the Environment (formerly DSEWPaC). The AAD operates with a strong basis in science, including extensive collaboration with other national and international institutions with Antarctic interests and expertise, and is Australia's principal institution for Antarctic science, policy and management. The data collection, synthesis and analysis for the SoE 2011 report was therefore led by AAD, in cooperation with other relevant government and tertiary education institutions with expertise in Antarctic matters. While not fully independent of government, the process implemented by the AAD was heavily based on established scientific knowledge, was guided by the SoE committee within the common framework for decisions and reporting (the CARF), and involved extensive consultation outside DSEWPaC to establish the issues and verify the information base.

The Antarctic chapter covered various topics that were also addressed in the broader Australia-specific context (inter alia biodiversity, marine, heritage), but it also includes information about the cryosphere and policy and governance issues specific to the management of the AAT and Australia's sub-Antarctic islands. This chapter therefore constitutes a report on the AAT and the sub-Antarctic islands as a single entity. However, not all issues relevant to the AAT and the islands could be dealt with in detail, so topics were selected to be most relevant for the 2011 report and to be representative of a broad range of Antarctic issues.

In preparation for the SoE 2011 report, there was extensive external consultation and collaboration focused on preparing the theme chapter. A number of meetings were held at the AAD's headquarters, also attended by staff from other research organisations, to determine topics to be included in the 2011 report, and the structure of the assessment tables. Consensus was achieved during the expert meetings on grades for condition, trends, pressures and confidence where data, particularly long-term data, were available. Where insufficient data were available to make an assessment, this was indicated appropriately in the tables and the confidence assignments. In support of the chapter development, case studies were commissioned from experts both within and external to AAD. In the discussions of the various topics, the most up-to-date international science literature was considered and the data summaries and presentation of each section were reviewed by repeated iterations of both internal and external experts prior to the final submission of the chapter for external peer review as part of the SoE-wide process.

5. Discussion

Like this assessment, the past cycles of national SoE reporting in Australia have been heavily constrained by limited availability of data/information at the national scale. As a result, past reports are typified by the ‘shopping list’ approach ‒ reporting of issues and examples to provide evidence that reflects the importance of a pre-determined set of themes and issues of the day, filtered through the expertise/experience of the appointed committee and a limited range of consulted experts. While this has provided an acceptable basis for SoE reporting and policy development in the past, as a result of a number of recent, highly fractious, environmental debates there is now a heightened awareness in the Australian community of the need for greater process transparency and accountability in government, and the need to provide for direct links between environment assessments and management responses. This includes a need to use unbiased data/knowledge as well as internally coherent and consistently repeatable structures for making assessments that may be used to influence management strategies in sensitive areas of public policy such as climate, water, heritage and natural resource management. The adoption of the independent evidence-based and highly consultative integrated assessment approach we report here distinguishes the 2011 SoE report from its predecessors, which have mainly relied on selected examples of the presumed issues drawn from information-rich sources. The CARF approach has allowed the SoE 2011 report to move away from the development of a catalogue of what is not known or has not been recorded, to pro-active evaluation and assignment of clear assessment grades based on more defendable, repeatable and transparent data collection processes, supported by explicit estimates of uncertainty that can be applied to the findings.

To provide a mechanism for feedback and assessment of the effectiveness of the SoE process as a whole, a structured feedback process was implemented by DSEWPaC. This resulted in substantial feedback, mostly supportive and constructive, indicating that there has been a wide and positive acceptance in the Australian community. In the public release of the report, the national environment minister declared that the report would serve as a touchstone for subsequent government policy and decisions across all portfolios. Online and download readers of the report were issued with an invitation for voluntary feedback. The web-based survey (conducted through SurveyMonkey ®) indicated that, in the 18 months after the report release, 83% of respondents agreed that the SoE 2011 report had improved their understanding of national environmental issues, 74% agreed they were made aware of new issues, 85% were assisted by the report in their work or study, and the decisions or actions of 68% of readers were influenced by content of the report. Many of the theme chapters are demonstrably influencing national policy ‒ the heritage chapter is directly informing and influencing the preparation of a ‘National Heritage Strategy’ by the Department of the Environment and the Australian Heritage Council. Likewise, the Primary Industries Standing Committee is responding to soil management challenges identified by SoE 2011 (primarily relating to soil carbon stocks, widespread soil acidification and unsustainable rates of soil erosion by water). Internationally, the approach has been adopted for environmental reporting in similarly data-poor marine situations for the purposes of the World Ocean Assessment (Ward 2012).

Ultimately, the most important measure of success will be the extent to which policy responses in the forthcoming decade reflect the issues and findings presented in the report. It is too early to declare the process and report a success in terms of environmental outcomes, even though it is having an important contribution to public debate, and is helping to develop a sound basis for consideration of environment issues within public and policy-development circles. Whether this will translate to effective strategies and actions for improvement in Australia's environment issues remains to be determined, and will unfold with continuing iterations of the national SoE assessment process.

6. Conclusions

SoE 2011 represents a landmark in national environment reporting in Australia. Despite the persistent institutional and information complexities, significant improvements over earlier approaches were achieved. To meet the multiple and sometimes competing objectives, the project design was adaptive and developed incrementally, as summarised here in the CARF logframe. In particular, the CARF was important for improving the planning, implementation and management of SoE activities across multi-year budget cycles, for facilitating effective consultation and engagement with technical experts, and for a clear articulation of the capacity of the project to deliver high-impact outputs that was needed to develop a supportive and engaged base of stakeholders.

For an SoE report that engages with the community, influences policy decisions in government, and is well accepted in the private sector, a well-founded consultative process that operates within a single unified assessment framework has been required. For the SoE 2011 report, the two central purposes for consultation were to develop and benchmark a process that provided a sense of ownership in the stakeholders, and to secure robust and defendable technical information base for the findings of the report. With appropriate design and management of bias/risks, the use of expert judgement worked effectively and met both purposes. We therefore conclude that a basis of expert knowledge drawn from within a well-designed consultation process is both achievable and important for SoE reporting purposes and is effective in delivering credible findings.

The effectiveness and value of the CARF for SoE assessment and reporting will grow with each iteration of SoE assessments that use the same approach. We expect that with repeat cycles of assessments that use this CARF future trends will become more obvious and new information will clarify the condition of many components assessed in SoE 2011. We also expect that such an ongoing implementation of this approach will incrementally increase the effectiveness of management in maintaining the structure, functions and health of the full range of Australia's environments. In this way, the five-year cycle of national SoE assessment and reporting activities may then become accepted as an investment in the future rather than only a cost to the current budget.

Acknowledgements

This manuscript was improved by guidance and input from Nancy Dahl-Tacconi (formerly DSEWPaC). The analysis reported here does not necessarily represent the views or the official position of the Australian Government, or the Australian Government Department of the Environment. The process and outcomes were made possible by the contributions of many consulted experts, reviews and colleagues across many disciplines and organisations, to whom we are grateful (listed at SoEC 2011, 898–903). The work reported here was funded and supported by DSEWPaC, and we recognise the high level of professional support provided by the DSEWPaC SoE Team and other staff, the professional support of Biotext information consultancy (biotext.com.au), the vision of the commissioning Minister for Environment Protection, Heritage and the Arts (Peter Garrett) in presenting us with this challenge, and the support of the presenting Minister for Sustainability, Environment, Water, Population and Communities (Tony Burke) in accepting and delivering the final report to Australia. PTH publishes with the permission of the Chief Executive Officer, Geoscience Australia.