Stakeholder Perceptions of the Environmental Effectiveness of Multi-stakeholder Initiatives: Evidence from the Palm Oil, Soy, Cotton, and Timber Programs

Abstract

The roundtable on sustainable palm oil (RSPO), the round table on responsible soy (RTRS), the better cotton initiative (BCI), and the forest stewardship council (FSC) are examples of multi-stakeholder initiatives (MSIs), established to foster sustainable commodity production. While these programs are promoted as collaborative schemes for natural resource management, the significant gap in understanding of MSIs’ effectiveness presents a major concern over the credibility and legitimacy of such programs. We explore stakeholders’ perceptions of the environmental effectiveness of four MSIs in relation to their impacts in reducing agrochemical use and conserving habitats. We found that stakeholders feel positive about the role of the schemes in advocating environmental sustainability in the commodity sectors, and establishing norms of good practice. However, numerous issues, including the inadequate monitoring and evaluation contribute to perceptions of ineffectiveness and a lack of confidence in the schemes’ ability to drive fundamental transformation in environmental performance.

Keywords:

• Better cotton initiative
• forest stewardship council
• multi-stakeholder initiatives
• private resource governance
• responsible soy
• stakeholders perception
• sustainable palm oil

Introduction

Multi-stakeholder initiatives (MSIs), in which a variety of stakeholders (including corporations and civil society organizations) are integrated into decision-making processes for natural resource management, have become a major feature of the global governance of sustainability (Pattberg 2007; Auld and Cashore 2012; Bennett 2017). There is a large and growing number of MSIs devoted to addressing different sustainable development challenges, from climate change to mining, to agricultural and timber production. Such initiatives have an increasingly significant political role as global rule-setters, and are used by industry to provide assurance to customers and investors of high environmental and social standards, the overall integrity of material flows, as well as norms of participation in decision-making to meet common goals (Schouten and Glasbergen 2011).

A popular and important justification for the growth in the number and scale of MSIs is their perceived level of effectiveness (Auld et al. 2008; Pinkse and Kolk 2012). Although the debate on the effectiveness of MSIs has been largely inconclusive to date, it is assumed by many scholars and practitioners alike, that MSIs are necessary for dealing with the complex challenges of governing sustainability, but also, that they are more effective than traditional state-centric arrangements (Pattberg 2007; Pinkse and Kolk 2012). In the last decade there have been some notable attempts to measure the effectiveness of MSIs through, for example, the development of extensive impact criteria and the use of mapping databases to examine habitat loss and restoration (Mena and Palazzo 2012; Ponte 2014). Nevertheless, there are few impact studies available, in part because of the lack of large-scale and longitudinal data, control groups and baselines, affecting measurement and attribution of impacts (ISEAL Alliance 2009, 2014). Several studies have stressed that the gap in understanding MSIs’ effectiveness is significant and increasingly presents a major question mark over the credibility and legitimacy of these programs (Marx and Cuypers 2010; Mena and Palazzo 2012; WWF 2010; Schouten et al. 2012). With the ever-expanding growth, reach and ambition of MSIs in governing sustainability, ascertaining their effectiveness is a pressing priority for research and policy.

In this paper, we analyze stakeholders’ perceptions of the effectiveness of four globally significant MSIs: the roundtable on sustainable palm oil (RSPO), the round table on responsible soy (RTRS), the better cotton initiative (BCI), and the forest stewardship council (FSC). The research also explores differences between the views of the various stakeholder groups, based on extensive documentary analysis and 82 semi-structured interviews. Assessing stakeholder perceptions in order to better understand the practical nature of schemes is well-established in literature focusing on collaborative water, biodiversity, grazing land, climate, and other natural resource governance schemes (Conley and Moote 2003; Amare et al. 2017; Running et al. 2017). In addition to capturing the experiences of key actors, mapping stakeholders’ perceptions is a useful approach for exploring the subjectivities and levels of confidence of different actors in schemes and how these actors interact, which in turn influences the legitimacy, uptake, and integrity of these programs. It is also valuable in revealing the strengths of these collaborative natural resource programs and areas for possible improvements.

MSIs’ Effectiveness and Stakeholder Perceptions

Multi-stakeholder initiatives have emerged as important platforms for natural resource governance across scales: local, national, regional and global. MSIs provide institutionalized solutions to governing sustainability by serving as mechanisms in which different actors aim to reconcile their varying interests and formulate consensual approaches to addressing a given problem (Pattberg 2007; Bennett 2017). As knowledge and norms brokers (Pattberg 2005), MSIs produce and disseminate knowledge among diverse stakeholders, and set and disseminate standards that help to shape behavior and perceptions of right and wrong actions. As learning networks, MSIs serve as “living laboratories” for experimenting with different institutional configurations, rule-making and implementation procedures for governing sustainable development (Schouten et al. 2012).

Multi-stakeholder initiatives have received mixed and contradictory appraisal in literature to date. Proponents point to MSIs’ ability to improve relations among stakeholders, raise awareness, boost knowledge and elevate “sustainability discourse among all participants in the process” (Gulbrandsen 2004, 77). Others praise MSIs as arenas for the co-production of knowledge which also provide opportunities for stakeholders to draw insights from their different experiences and perspectives to improve natural resource management (Pattberg 2007). Highlighting their emphasis on participation and joint decision-making, commodity roundtables are often portrayed as providing stakeholders with equal positions in negotiations, implying that they offer accountability, transparency, legitimacy and inclusiveness through participation (Ponte 2014). Furthermore, it is suggested that MSIs have the potential to transform markets because of their inclusion of large and influential actors, bringing sustainability issues further into mainstream markets, and allowing the scales of impacts to be large. Critics, however, highlight the power asymmetries in MSIs and argue that these platforms may offer opportunities for corporate actors to legitimize their short term and profit-oriented exploitation of natural resources by co-opting collaborative processes and forestalling more stringent regulation (Bitzer et al. 2008; Mayer and Gereffi 2010; Moog et al. 2015). It has also been noted that MSIs could raise opportunities for conflict and mistrust by exacerbating existing tensions between stakeholder groups (Cheyns 2011, 2014).

While plenty of research on MSIs has focused on the effectiveness of the institutional arrangement, decision-making model, and quality of stakeholder participation, many key writers accept that not much is known about what Gulbrandsen (2004, 78) calls “their actual on-the-ground impact.” Much scholarship accepts that the effectiveness of an institution can be measured according to the institution’s impact on the problem it intends to address (cf: Fuchs and Kalfagianni 2012; Gulbrandsen 2012). However, the definition of effectiveness remains broad and subject to a wide range of formulations and stakeholder perceptions, when applied to MSIs. For example, NGOs may assess effectiveness by comparing the stringency of standards with their own ideals and values, while businesses are naturally more likely to compare with business as usual (the no-regime counterfactual). The following highlights the importance of “perception” in understanding how stakeholders formulate their ideas of impact and effectiveness. These differences in perspectives are consistent with social exchange theory which suggests that individuals’ and groups’ perceptions of impact, benefits and costs of any exchange will be influenced by their different values, experiences and positions in the social structure (Emerson 1976; Liao et al. 2017).

Mapping and comparing stakeholder perceptions have been regular features in diverse resource management scholarship (Conley and Moote 2003). Shackley and Deanwood (2002) observe that stakeholder perception can help uncover wider institutional dynamics which might be hidden from a baseline construction and quantitative measurements. Understanding perceptions of effectiveness can promote learning and feedback which could lead to changes in MSIs’ missions, goals and actions. Varying perceptions could highlight gaps in communication, expectations and tensions among stakeholders. Negative perceptions have been found to hinder the success of initiatives (Shackley and Deanwood 2002) and understanding perceptions could help to enable compromise and reduce conflict (Byrd et al. 2009; Amare, et al. 2017). Fletcher et al. (2003) and Amare et al. (2017) found that a perception of effectiveness is a major predictor of the legitimacy of initiatives and a negative perception can cause stakeholders to withhold support and participation. Mapping stakeholder perceptions of effectiveness and comparing any differences in the perceptions of stakeholders is particularly important in the context of MSIs, given that MSIs are built on the ideals of co-management, co-production of knowledge and inclusiveness of diverse stakeholders. In fact, it is arguable that the success of these schemes is inherently dependent on the perception of stakeholders and how these perceptions are communicated.

Research Context

The MSIs examined here are specific forms of MSIs, known as “sustainability roundtables,” which are private arrangements with the aim of improving the sustainability of global commodity chains. Within such platforms, only private parties (businesses and NGOs) have decision-making power. Governmental agencies and scientists only participate as observing members or advisors. This study covers the RSPO, RTRS, BCI and FSC, specifically in relation to their ability to reduce agrochemical use and conserve habitats. Table 1 provides an overview the four roundtables with the years they were started.

Table 1. Overview of the four sustainability roundtables.

Name	Commodity/sector	Started
Forest Stewardship Council (FSC)	Timber and forest products	1994
Roundtable on Sustainable Palm Oil (RSPO)	Palm oil	2004
Round Table on Responsible Soy (RTRS)	Soy	2006
Better Cotton Initiative (BCI)	Cotton products	2004

The four commodities (palm oil, soy, cotton, and timber) are of great importance in world markets as they are consumed globally. Although the commodities differ significantly, there are several commonalities in their sustainability issues – including the critical issues of agrochemical use and habitat conservation – and in the structure and content of their standards (or “Principles and Criteria”).

The RSPO was formed in 2004 in response to the urgent need for sustainable palm oil (RSPO n.d.). This followed widespread concerns around the environmental and social sustainability of palm oil production in tropical countries (with Indonesia and Malaysia making up 86% of global production). The most urgent environmental issues associated with palm oil include: tropical deforestation, biodiversity loss, use of pesticides and fertilizers, methane emissions from burning and growing oil palm on carbon-rich peat soils, and damage to air and water quality (Laurance et al. 2010; Paoli et al. 2010; Cattau et al. 2016; Vijay et al. 2016). The RSPO has an ambitious aim to “transform markets to make sustainable palm oil the norm” (RSPO 2018). Third-party audits are used as a common compliance mechanism, in which auditors “certify” compliance or identify improvements required for a successful audit (Schouten and Glasbergen 2011). The RSPO’s membership comprises representatives from seven sectors of the palm oil industry, including oil palm growers, palm oil processors and traders, consumer goods manufacturers, retailers, banks and investors, environmental conservation NGOs and social/developmental NGOs. In 2011, nearly 10% of all palm oil was certified, as of late 2012, this increased to 15% (RSPO 2013) and in early 2016, this number stood at 21% (Bissonnette 2016). The RSPO Code of Conduct requires that all supply chain actors (retailers, traders, financial institutions, and processors) actively promote the demand for, purchase and trade of certified sustainable palm oil (CSPO) (Paoli et al. 2010).

Global production of the soybean has increased dramatically, from 26.9 million metric tons in 1961, to 334.6 million metric tons in 2016 (FAOSTAT 2018). Soy is mainly produced in Latin America, and it is associated with deforestation, habit destruction and biodiversity loss, as well as adverse impacts on water resources, unsustainable pesticide use and soil erosion. The RTRS was founded in 2006, to promote the use and growth of responsible production of soy. The RTRS membership consists of “Participating” and “Observing” members, and membership is open to actors in the soy supply chain (including three constituencies of producers (industry, trade, and finance) and NGOs. RTRS decisions are made through the vote of participating members that are equally represented in the three constituencies, and each constituency has a voting power of one third of the total votes. The Executive Board is elected by the General Assembly and composed of the same three constituencies (RTRS 2010). Observing Members do not belong to these three constituencies, and may include individuals or organizations such as regulatory authorities, governmental agencies, consulting and auditing firms, academia and donor organizations (RTRS 2010).

The BCI was founded in 2004 to promote the supply of sustainable cotton for the mainstream cotton market. About 2.5% of the world’s arable land is used for cotton production, and 90% of cotton farmers live in developing countries on farms of less than two hectares (WWF 2014). Impacts of cotton production include wasteful irrigation techniques, ineffective cultivation practices, and unsustainable and unsafe agrochemical use (Mancini et al. 2008). The BCI’s membership consists of producer organizations, civil society, retailer and brands, suppliers and manufacturers, and associate members. All members can attend the General Assembly, at which time a Council is elected and any proposed decisions are voted on (BCI 2018). The Better Cotton assurance system uses a combination of self-assessment, second party credibility checks and third party verification.

Following concerns around deforestation, environmental degradation, and social exclusion (FSC UK 2018a), the FSC was founded in 1994 – by several environmental NGOs, forest companies, and companies that use forest products – to promote environmentally appropriate, socially beneficial, and economically viable management of the world's forests (FSC UK 2018b). Its establishment followed the Rio declaration of the UNCED in 1992 as a response to formal, but nonlegally binding discussions on the promotion of sustainable forest management (SFM) (Marx and Cuypers 2010). In 2016, FSC-certified forests (including natural forests and plantations) produced approximately 16% of global timber by volume, and the FSC covered 10% of global forest-based trade (FSC International 2017). The FSC’s forest certification system involves both Forest Management and Chain of Custody certification, and products certified under the FSC system have a “tick tree” logo (FSC UK 2018c). The FSC has three equally weighted chambers (environmental, economic, and social) and membership within each chamber is also weighted between North and South. Individual chambers represent 33% of votes at FSC general assemblies (Chan and Pattberg 2008). According to Eden (2009, 286), FSC seeks strength and legitimacy through “heterogeneous alliance by representation across various divides, such as lay-expert; industrial–academic; industrial–nongovernmental; and industrial–governmental” groups. This underscores the importance of understanding the perception of the different groups on stakeholders and how such perceptions affect confidence and participation in the schemes.

Sample and Methodology

The study involved qualitative techniques: discourse analysis of 82 semi-structured interviews, documentary analysis, and direct observations in two international sustainable sourcing conferences in London, UK. A qualitative approach is considered to be better suited to adequately capturing nuances, settings, interdependencies, complexities, and contexts (Patton 1990). Given this paper’s focus on perceptions, a qualitative approach allowed for consideration of beliefs, values, feelings, and motivations (Lincoln et al. 1985). The interviewees were grouped according to their sectors and the MSI in which they mostly were involved. Broken down, the sample of participants includes: RSPO (n = 25), RTRS (n = 19), BCI (n = 18) and FSC (n = 20). In relation to their sectors, the stakeholders were grouped according to their supply chain positions, including investors, producers (growers), traders, manufacturers, and retailers. Representatives from the NGO community and the MSI secretariats were also interviewed. All the stakeholders interviewed were involved in standard-setting and implementation of requirements of the MSIs. Table 2 shows the number of stakeholders interviewed across the four MSIs and seven stakeholder groups.

Table 2. Number of stakeholders interviewed across the four MSIs and seven stakeholder groups.

	Palm oil	Soy	Cotton	Timber
	(RSPO)	(RTRS)	(BCI)	(FSC)	TOTAL/Stakeholder group
Secretariat representative	1	1	1	2	5
Producers/Growers	6	5	2	3	16
Traders	4	2	3	2	11
Manufacturers	3	4	4	4	15
Retailers	3	3	3	3	12
NGOs	5	3	5	5	18
Investors	3	1	0	1	5
TOTAL/Commodity	25	19	18	20

Interviewees were chosen based upon their professional backgrounds, published work, associations and experience with the MSI in question. Of the 82 semi-structured interviews conducted, 79 were done by telephone, and three were conducted face-to-face. Most interviews lasted between 30 and 45 min. We asked all participants about their perception of the environmental effectiveness of the four MSIs but concentration was on the schemes in which participants were most involved. Some of the participants, especially NGO representatives and retailers were active in more than one of the four roundtables. The MSI community is fairly close. They hold common conferences, regularly exchange ideas and compare notes and progress. At the same time, various stakeholders have different areas of interest according their sector or position in the value chain. Interviewing a range of stakeholders allowed for examination of different perceptions and subjectivities from multiple perspectives.

Interviewees were asked, against an interview checklist, to describe the extent to which they perceived the MSIs as effective in (1) reducing agrochemical use, and (2) conserving habitats. Participants were also asked if there were “other environmental impacts” that they perceived to be of importance, and were invited to elaborate on their perceptions of contextual issues that influence the effectiveness of MSIs. Interviews were recorded digitally, then transcribed and sorted according to the four MSIs and the stakeholder groups. One of the authors coded the data, indexing large amounts of text to allow it to be retrieved in a variety of ways (Coffey and Atkinson 1996). Content coding was used, focusing on what was discussed in interviews, as well as analytical coding, which provided some interpretation of interviewee statements. Categorization was then undertaken, considering the relationships between different codes, and how the findings were related (Coffey and Atkinson 1996). The classification of perceptions into the various categories of impact perception (“moderate, significant”, etc.) were based primarily on the answer to the direct question inviting interviewers to describe the extent to which they perceived that the MSIs were generally effective in meeting the objectives of reducing agrochemical use and habitat conservation and a score was assigned. As the study examined perceptions of effectiveness, open-ended questions were posed in order to allow interviewees to share politicized, nuanced views; and the researchers could gain a more in-depth understanding of how interviewees prioritized certain issues. Following Denzin and Lincoln (2005) and through discourse analysis, the responses to the questions were categorized and scored as the following:

• Positive (++/100): Significantly positive on-the-ground effects of the MSI are perceived.

• Somewhat positive, (+/50): the MSI is perceived to have moderately positive on-the-ground impacts.

• Not significant: (0): the MSI is perceived to have a neutral effect/no change on the parameter.

• Somewhat negative, (−/−50): signifies the perception of moderately negative on-the-ground impacts.

• Negative (−/−100): the MSI is perceived to have significantly negative on-the-ground impacts.

Stakeholders’ Perceptions of Effectiveness

Interviewees perceived MSIs as having positive environmental effects in relation to reduced agrochemical use and habitat conservation (See Supplementary Figures 1 and 2). However, as Figure 1 shows, there is variation in stakeholders’ perceptions of the effectiveness of the different MSIs in addressing these issues (See Supplementary Figure 3). On average, stakeholders involved in the FSC have very positive perceptions of the FSC’s effectiveness in reducing agrochemical use and conserving habitat. Respondents’ perceptions of the significant positive impacts of the FSC corresponds with the view in literature that FSC is a leading example of product certification as a resource governance mechanism (Pattberg 2007; Auld and Cashore 2012). Despite some critical comments and concerns from respondents, none of the MSIs received a general verdict of negative or significantly negative impact.

Figure 1. Interviewees’ perceptions of MSIs’ environmental effectiveness in reducing agrochemical use and conserving habitat.

Stakeholders involved in the RSPO gave the scheme a relatively high ranking for its effectiveness in reducing agrochemical use, while those involved in the BCI gave it a high ranking for habitat conservation. Stakeholders perceived that the RTRS, RSPO and BCI are all somewhat effective in facilitating the reduction of agrochemical use. An important issue raised by participants was about the “scalability” of positive impacts of MSIs, and whether examples of positive changes made at production are representative of these industries at scale. The following are a few selected quotes regarding reduction in agrochemical use:

“With the RSPO, the use of agrochemicals like fertilizer and weedicides are greatly controlled in any well-managed plantations because it represents about 40% of agriculture cost. We have definitely seen significant impact” (palm oil, producer).

“In India, farmers have saved on average 7500 Indian rupee per Hectare of cotton due to pesticide reduction” (cotton, producer).

“Compared to conventional methods, BCI farmers uses less water and chemicals, so it is definitely benefiting the environment. I think it has good prospects” (cotton, manufacturer).

“We’ve been looking at four years’ worth of data for a BCI project in India, and we are seeing significant savings in pesticides use and fertilizer” (cotton, retailer).

“In China, the FSC can make a change in certifying plantations and improving the way the chemicals are used. I know examples of many South Chinese plantations cannot join because of this issue” (timber, retailer).

“In projects in Pakistan and India, there has been a reduction in chemical pesticides and an increase in the use of farmyard manure” (cotton, NGO).

“There are benefits and we are seeing pesticide use reductions of 70% plus…We’ve trained 150,000 farmers on using pesticides, and considering [the BCI is] only a few years’ old, that's an amazing achievement” (cotton, NGO).

An environmental NGO representative, in relation to RTRS, stated, “The biggest difference has been that farmers are improving management practices; they are measuring, monitoring and sharing knowledge. Getting farmers into the culture of monitoring is an important first step”. Several other participants confirmed this sentiment, with another NGO interviewee suggesting that training was a major factor in explaining the extent of progress made in reducing agrochemical use. However, another NGO representative active in BCI, RSPO and RTRS voiced the concern that a “focus on documentation, paperwork and reporting may distract from action needed on the ground”. In most cases, respondents perceived that MSIs have helped formalize documentation of agrochemical use, thereby promoting good management practices. This view accords with the observations of Kolk et al. (2008) and Utting (2012) both of whom report improvements in producers’ capacity and good management practices as crucial benefits of MSIs. Our analysis revealed, perhaps not surprisingly, that reducing agrochemical use was motivated by both economic and environmental objectives. We observed that many respondents highlighted the economic value of the savings associated with this principle. Two respondents in particular (palm producer and retailer) referred positively to a WWF document which reported that RSPO certification resulted in yearly pesticide cost reductions of $250,000 and herbicide cost reduction of $73,859 for two participating firms (WWF 2012).

With regard to habitat conservation, some interviewees perceived that MSIs, to varying degrees, have made a positive impact. Respondents believed that a crucial route to promoting the conservation of biodiversity and habitat has been through the identification and maintenance of high conservation value (HCV) forest or areas. The following are a few sample quotes:

“Yes, the FSC has made a difference. This is very new to all of us, since we’ve had to follow strictly the maintenance of high conservation value forest (HCVF). We incorporate the HCVF management during forest inventory” (timber, producer).

“I can tell you right away that we would not have invested in documenting high conservation value assessments without RSPO requirements” (palm oil, producer).

“Increasingly, companies will set aside 30–40% of the concession to account for high conservation value areas, including buffer areas, riparian zones and habitat corridors” (palm oil, investor).

“In the past, our understanding of conservation of habitat for biodiversity in the oil palm industry was rather limited. The RSPO has changed this” (palm oil, trader).

“The law requires buffer zones around riparian zones, and through the RTRS, producers are in a process of adopting measures to comply with the legal requirements” (soy, investor).

However, other respondents expressed significant concerns in relation to the schemes’ abilities to address habitat conservation:

“We believe that the principles and criteria are insufficient to guarantee sustainability of palm oil particularly in relation to the expansion and growth of oil palm in high carbon habitats such as peatlands and forest areas” (palm oil and soy, NGO).

“I hoped that the principles and criteria could have been more strict and more clear in what we need to do against deforestation” (soy, retailer).

Despite a general perception of progress, some stakeholders expressed strong concern especially with the continued expansion of oil palm in important areas such as peatlands in Indonesia and tropical forest regions. Many identified important weaknesses and loopholes in RSPO requirements and enforcement, expressing concerns, for example that RSPO rules still allowed peatland forests to be cleared for the expansion of plantations, and allowed these plantations to become certified. Another important issue raised by some stakeholders involved in RSPO was that certification was mostly sought and granted in relation to older plantations that have little virgin forests, aligned with findings by Carlson et al. (2018) who found that RSPO certification on the island of Borneo was mostly adopted in older plantations that contained little remaining forest. Others highlighted the problem of non-compliance with Principles and Criteria and wondered about whether the MSI really has the power and resolve to sanction offending producer companies. This concern resonates with the observation of some critics, that MSIs do little to address the power asymmetry between large multinational corporations and NGOs, but rather serve to hide continued unsustainable natural resource exploitation by large companies (cf. Moog et al. 2015).

Differences in the Perceptions of Groups of Stakeholders

Our analysis also revealed important differences in the perceptions of the effectiveness between the stakeholder groups as shown in Figure 2. Note that in analyzing the differences in the perceptions of effectiveness among the various groups, the last two categories (“moderately negative” and “significantly negative”) were removed since these responses were not identified in the data.

Figure 2. Perceptions of MSIs’ effectiveness in reducing agrochemical use and conserving habitat according to stakeholder group.

It is interesting to note that, on average, retailers and manufacturers, based in Europe and the USA, had the strongest belief that MSIs have had a positive environmental impact, and ranked the schemes with the highest average positive scores for their perceived reduction of agrochemical use and habit conservation. Traders also expressed confidence that the MSIs were delivering these environmental benefits. On average, NGOs expressed the least confidence regarding the effectiveness of these MSIs (with a greater proportion of these respondents expressing that MSIs have had a moderate or no effect). NGOs were particularly concerned that the schemes have not had a positive impact on biodiversity conservation. It could be that the retailers and manufacturers’ positive perceptions of a higher level of effectiveness of MSIs may be explained by their geographical distance from production regions and a lower level of awareness of the environmental problems of MSIs at production points. Another possible explanation is that these buyers may have more trust in, and need to trust in the ability and integrity of certification bodies to verify compliance than other stakeholder groups. The fact that NGOs had a more critical stance was not very surprising. It is common for business stakeholders to emphasize the success of sustainability partnership initiatives where NGOs are pushing for bolder and more radical actions. In addition to ideological differences, one explanation would be that NGO representatives are closer, or have networks closer to production sites and are more active in monitoring impact through interaction with local people, sites and ecologies. We found it interesting that the secretariat representatives perceived a modest impact of MSIs. This may be because much like the NGOs, the secretariat representatives have more “on-the-ground” knowledge of the impact activities. Another explanation might be that they are more aware of the “messiness” of implementation in certain geopolitical contexts, and the lack of empirical data required to verify effectiveness of MSIs in meeting their objectives. The following quotes reflect this skepticism:

“To be honest, we are unable to measure whether there has been an effective reduction in agrochemical use” (RTRS secretariat).

“I am sure that FSC has had significant positive impact worldwide. The problem is that we can’t quantify it” (FSC secretariat).

“As far as impact is concerned, the RSPO realizes that we are not diligently monitoring the impact of certification on the industry as yet” (RSPO secretariat).

“It’s extremely difficult to quantify any changes” (palm oil, investor).

“The quantification, verification and real tangible evidence to be able to make any claims about [the FSC]…this is completely lacking at the moment” (timber, retailer).

“It will take some time to have the evidence [of effectiveness]” (soy and cotton, NGO).

“We then get a problem with trying to work out what's happened due to certification, because the uncertified ones have changed as well. There's a problem trying to find specific evidence of certification” (timber, NGO).

“We may also not get enough data from farmers and some of them are quite subjective…Implementing partners sometimes have difficulty. The lack of control farmers is also important” (BCI secretariat).

More recently, schemes have been claiming they are making significant investments in increasing the capacity of their impact monitoring and measurement teams, reflecting the organizations’ responses to the lack of clear, quantitative impact data. However, we observed that much of the available impact reports published by the schemes themselves seem to still focus more on the viability of the schemes as market instruments in their own right (e.g. market share), rather than providing the environmental and social impacts of the schemes, or dealing with criticism by showing how non-compliances are being addressed.

Discussion and Conclusion

Our results show that key stakeholders perceive that the four initiatives are making some positive contribution to mitigating negative environmental impacts of commodity production. Based on the perceptions of stakeholders, there is evidence that the roundtables may indeed be working to counter the so-called “democratic deficit” in global natural resource governance (Lemos and Agrawal 2006; Schouten et al. 2012) by increasing the participation of multiple stakeholders, including industry actors, in the design and implementation of initiatives’ principles and standards. However, despite some noted progress in establishing and mainstreaming narratives around the need, and opportunities for sustainable production, the mixed reviews from the various groups of stakeholders indicate that these MSIs are still far from achieving the more ambitious goal of driving market transformation toward improved environmental and social performance. Stakeholders perceived that a reason why the MSIs are not having more positive environmental impacts was because the quality of monitoring and evaluation which many stakeholders, from different stakeholder groups, described as “uneven or inconsistent”, remains questionable. Representatives of NGOs and secretariats showed the highest degree of awareness of the weakness of current monitoring and evaluation systems, with some suggesting there was still a long distance to go in establishing reliable indicators and assessment procedures for measuring effectiveness. Furthermore, stakeholders noted that without transparent and effective monitoring and evaluation, it is difficult to understand success and to decide how best to ensure the future development of the standards. A related issue is the quality of the auditing process, which most interviewees downstream of producers identified as lacking in transparency and uniformity. A minority of interviewees were candid enough to admit they are aware of several poor practices that have escaped the scrutiny of audits as shown by these following selected quotes:

“Many of the auditors will be straight out of university and do not understand how the forest business works and how it works in different places. They don’t understand the tricks that forest managers might be carrying out in order to hide their deficiencies and therefore don't detect them. As a result, there are cases where forests certified that should not be” (timber, producer).

“Auditors are in inappropriate roles advocating the assumptions of the standard rather than advocating for good independent auditing and that's bad practice in standard setting” (soy, manufacturer).

“Each certification body has different standards and areas of interest, so there are people who would certify something that I wouldn’t certify” (palm oil, producer).

We posit that improving monitoring and evaluation would require attention to two different, but related challenges. One is the question of which forms of monitoring and evaluation are most suitable in order to enhance the effectiveness of MSIs. The other is about how to enhance the quality and integrity of the auditing process. Unlike environmental management systems such as ISO 14001, which arguably consist of logical steps (of commitment, policy, planning through to implementation, measuring and monitoring and review), the principles of the MSIs are broad statements of good practice that are difficult to measure and do not follow clear processes of implementation. For example, auditors are expected to take into account local social customs as well as ecological factors: a proviso which may be desirable in principle, but which makes consistency difficult to achieve. Moreover, the need to attract the business of companies themselves can result in huge variations in standards across different certification bodies leading to a race to the bottom, as there is no minimum price for an audit.

Some scholars and practitioners argue that there are opportunities for local stakeholders to monitor compliance with MSIs’ requirements. Furthermore, an Impacts Code from ISEAL Alliance encourages standards to strengthen the quality of outcome and impact evaluations by “presenting evaluation results to those who participated in the evaluation and to local stakeholders prior to finalizing the study” (ISEAL Alliance 2014, 18). However, exactly how to balance scientific rigor with the intuitive appeal of such ‘participatory monitoring and evaluation’ proposals remains unclear (Izurieta et al. 2011). Moreover, developing methods that are relevant on a local scale can often be time consuming and costly (ibid.). It is clear that more research is needed to understand how local stakeholders can participate in monitoring processes, and how this data can feed into ongoing adaptive management.

Another major challenge relates to the trade-off between the stringency of MSIs’ requirements and their uptake. In all four cases, some interviewees felt that certain requirements were not stringent enough, while others perceived the standards as too stringent, making this a major point of difference in the views of the participants. Those who backed tougher standards justified their position on the need to enhance the integrity of the schemes, while those who were opposed suggested that stricter requirements would make businesses less inclined to join the voluntary schemes. There was a sense among NGOs that criticism of schemes may lead to stronger processes and more sophisticated standards, thus leading to greater impacts. However, low confidence may lead to stakeholders using other competing standards and reducing actors taking up the standard. One interviewee from the RTRS secretariat captured this sentiment, saying:

“We need to overcome the image of it being difficult…. You need to bear in mind there are other standards covering soy production that are not as strict as RTRS. I think that people are going to go for the easiest standard and many not take into account that easier is not always more sustainable. Being seen as difficult is an important weakness”.

Extant literature suggests there is an inverse relationship between stringency and uptake, and between toughness of standards and compliance (Edwards and Laurance 2012). Gulbrandsen (2012) found that in the forestry sector, less stringent producer-backed standards such as the PEFC have seen much faster growth than the more stringent, and longer established FSC certification system. Klooster (2010), in his examination of the plantation review process of the FSC, highlights a trade-off between the strictness of the standards’ sustainability requirements, their acceptability to corporate participants, and the perceived legitimacy of FSC standards and processes. There is indeed wide agreement among scholars and practitioners alike, that without extensive uptake, certification cannot improve practices in areas of greatest concern (Gullison 2003). MSIs may, therefore, wish to seriously consider making the entry level more attainable to increase initial uptake in priority regions, with the rigor of standards and implementation requirements subsequently increasing over time. Overall, there is a need to carefully think through how to resolve, or least minimize the tension between stringency and uptake in roundtables.

Acknowledgments

The authors would like to sincerely thank all the interviewees who participated in this project, and the anonymous reviewers for their comments.