Supporting local institutions for inclusive green growth: Developing an Evidence Gap Map

Highlights

•	Structured literature search for rigorous studies on local institutions development.
•	Focus on agriculture and rural development for achieving Inclusive Green Growth.
•	Evidence Gap Map visualizes outcomes, revealing key evidence clusters and gaps.
•	Knowledge repository to policy-makers on institutional interventions.
•	Improved research prioritization and prerequisite for meta-analysis.

Abstract

We conduct a structured search of the academic literature that assesses the impact of development interventions that aim to build and strengthen local-level institutions to facilitate Inclusive Green Growth. Inclusive Green Growth extends the standard growth perspective to include welfare enhancements both the poor (‘inclusive’) and for future (‘green’) generations. We restrict our search to studies in the domain of agriculture and poverty alleviation in the developing world. We access ten online databases and various working paper series and focus on summarising evidence from quantitative studies that use rigorous evaluation methods. Together, this yields 158 studies. We then retain 66 studies that contain a credible counterfactual. We visualize the interventions and outcomes in an Evidence Gap Map, highlighting both the available evidence and remaining knowledge gaps. Most studies suggest that strengthening local institutions can improve the delivery and targeting of public services and overall satisfaction with local governance. There are however, clear limitations and knowledge gaps highlighting priorities for future work. Few studies assess impacts on final outcomes such as household income or agricultural productivity and no studies assess inclusive and green outcomes jointly. We discuss the key benefits of a structured literature search and Evidence Gap Map for policy-makers and development practitioners and illustrate how it serves as a knowledge repository and identifies where evidence is lacking, thus setting the agenda for future work.

Keywords:

• Structured literature search
• Inclusive green growth
• Local institutions
• Rigorous evaluations
• Evidence-based policy-making
• Evidence Gap Map

1 Introduction

The past decades have seen a growing recognition of the role of institutions in the development processes. A consensus view has emerged suggesting that institutions rather than geography are the main determinant of growth (or lack thereof, see Acemoglu et al., 2001; Easterly and Levine, 2003; Rodrik et al., 2004; Rodrik, 2006). Besides featuring prominently in academic work, debates over the role of institutions and how to change them have influenced the scope of international development assistance. Views have varied and encompass “big push” and “blue print” approaches (think of the U.N. Millennium Development and Sustainable Development Goals initiatives, see also Sachs, 2005) to “bottom up” and diagnostic approaches incorporating local constraints (Easterly, 2006; Rodrik, 2010). Recently, Inclusive Green Growth (IGG) has become a term central in the in global donor community discourse. Coined by the World Bank (2012), it is referred to as ‘the economics of sustainable development’ as growth that improves the welfare of both current (‘inclusive’) and future (‘green’) generations. The term has become a buzz word for development planning and cooperation and is viewed as a means for achieving the Sustainable Development Goals (SDGs). While IGG typically encompasses a broad range of policy themes, ranging from clean energy development to sustainable urban planning, we focus on the sub-domain of agricultural and rural development. Within this domain, the stimulation of Inclusive Green Growth often entails interventions that build or amend local institutions to internalize (environmental) externalities, support an equitable distribution of benefits and deliver a more optimal provisioning of public goods (World Bank, 2012).

Despite the policy enthusiasm for an institutional focus to achieve inclusive and green growth, the available evidence has been scattered and until recently limited. In addition, generic statements like ‘development interventions should strengthen local institutions’ is of little practical use for policy-makers and development practitioners seeking clear guidelines on most effective interventions in novel project locations. Have such interventions resulted in the desired effect always and everywhere? How can we learn from the cumulative set of relevant studies for guiding more effective development practice? We conduct a structured literature search to identify the available evidence on institutional interventions that aim to foster Inclusive Green Growth in the developing world. We then construct an Evidence Gap Map (EGM) where we identify the set of institutional interventions and outcome (or impact) categories (Snilstveit et al., 2013).

EGMs provide policy-makers with relevant evidence in a transparent way. Evidence Gap Maps uniquely synthesize the available information and facilitating the development of evidence-based policies for policy-makers, development practitioners and researchers alike. In addition, EGMs show where evidence is lacking setting the agenda for future research.

Fig. 1 roughly outlines the type of institutional interventions, distinguishing between contextual, or moderating, factors, intermediate outcomes and final inclusive growth, or green growth outcomes (World Bank, 2012; Bouma and Berkhout, 2015; and papers identified in our structured search). With institutions we refer to “systems of established and prevalent social rules that structure social interactions “(Hodgson, 1988). Following this definition, institutional interventions may be directed at strengthening informal and community type of institutions, like village committees and microcredit groups, or contribute to building or strengthening formal organizations like farmer cooperatives or government organisations, like agricultural extension departments. The institutional interventions considered can be grouped in two types: (i) interventions directed at the distribution of resources (inclusive growth- equity) and (ii) interventions directed at the productivity of resource use (green growth- efficiency). Examples of interventions in the first category include efforts to empower or increase representation of marginalized stakeholders and interventions that secure access for poor households and reduce their vulnerability. Examples from the second category are those that invest in improved access to information, market facilities and property rights, thereby reducing market failures and information costs. Also considered are efforts to strengthen institutions aimed at improving public good delivery and creating incentives for sustainable resource use.

Fig. 1 Theory of change.

Moving from interventions to policy outcomes is not straightforward. For example, training a village committee to become more transparent may enhance local participation in village meetings, but this does not necessarily lead to enhanced public good provision. Similarly, empowerment of marginalized groups may increase participation in meetings, but this does not necessarily imply that they benefit more. Hence, both intermediate and final outcomes should be considered, as interventions may contribute towards improving the quality of the institutional environment in the short-run, but to a final objective of inclusive green growth only in the long-run. Key intermediary outcomes are related to increases in institutional quality and participation, changes in public services and targeting, improved access to finance and changes in human capital.

There are various contextual, or moderating, factors that influence both the choice for the type of intervention needed as well as their eventual impact. This is reflected in both the variety of interventions identified in the Evidence Gap Map as well as in reported impact. Variation is inherent to presenting an overview of the evidence-base on this relatively broad theme. Furthermore, as Hausmann et al. (2008) and Rodrik (2010) emphasize it is unlikely that all potential factors constraining (in this case) IGG are binding simultaneously. The severity of market and governance failures underlying Fig. 1 typically differs from one location to another, pointing to a need for proper institutional diagnostics for selecting the most appropriate intervention.

The studies identified provide detailed information on underlying theories of change, as well as the specific institutional constraints they target. A policy-maker who gains a detailed understanding on the peculiar binding institutional constraints in a new area, can relate to the Evidence Gap Map for identifying the most promising and relevant interventions. We illustrate such a process at the end of this paper, thus operationalizing the concept of institutional diagnostics.

We are not aware of other reviews that look at local institutional interventions to achieve IGG. Closest to our work are two reviews focusing on sub-components of the institutional framework. Mansuri and Rao (2004) reviewed the effectiveness of community-based approaches, finding limited and mixed evidence of the effectiveness. A recent review on land tenure (Lawry et al., 2016) finds limited evidence for the impact of institutional interventions for strengthening property rights and tenure arrangements on improved agricultural productivity. Limited evidence may result from the narrow time-frame at which most interventions are evaluated. As explained above, in this paper we set to gather and disclose evidence from a broader range of interventions aimed at stimulating inclusive green growth in the agricultural production and rural development domain.

The remainder of this paper is organised as follows: Section 2 describes the search strategies employed. We discuss the overview itself in detail in Section 3. In Section 4 we illustrate the value of this approach for institutional diagnostics. On the one hand, it serves to prioritise research based on identified knowledge gaps. On the other hand, the EGM informs policy-makers and development practitioners to identify most promising interventions for targeting specific institutional constraints.

2 Methodology

We conducted a structured literature search in order to identify all relevant studies that both evaluate the impact of a specific institutional arrangement (such as credit provision to smallholder farmers) and attempts to improve the institutional setting (i.e. effectiveness of public service spending) within the realm of agriculture and rural development in developing countries. Our selection criteria (Fig. 2) confined our search to quantitative studies describing an institutional setting or evaluating an agricultural intervention at least at the community level aiming at inclusive green growth outcomes. We only included studies focusing on developing countries and those which contain a credible counterfactual since our target was to uncover causal effects, rather than correlations. For this reason, we included only those studies that make use of randomized control trials, difference in difference, regression discontinuity and propensity score matching. Studies evaluating the impact of institutions using instrumental variables only, often spur considerable debate about the proper identification of causal effects. Such studies are therefore excluded from this study.

Fig. 2 Summary of the structured search procedure.

We developed a list of search terms that describe or relate to inclusive green growth interventions and outcomes. A full list of the search terms included is provided in Table A1 in Appendix A. We used these, both individually and combined with Boolean operators (AND, OR, NOT) and wild card symbols (*) to search for alternative word endings. We applied these search terms to ten academic literature databases (a.o. AgEcon, SCOPUS, and Science Direct) and extended our search to include some key working paper series (a.o. NBER Working papers). The exact search strings varied between academic databases and working paper series. Due to the more detailed nature of the academic databases, the applied search strings included additional limitations on the time-span (studies from the last 3 decades) or excluded irrelevant branches of the literature (Immunology and Microbiology, Physics and Astronomy etc.). The full list of databases covered is provided in Table A2 in Appendix A, together with representative examples of the actual search strings applied (Table A3). Finally, we searched for additional papers by snowballing from the reference lists of several key papers identified (i.e. those most closely matching our selection criteria).

Table A1 Keywords used in structured literature search.

Keywords used to identify the relevant interventions and outcome indicators:	Studies using either one of the following research methods:
Women Empowerment, Inclusive Growth, Natural Resource Management, Institutional Capacity Building, Participatory Technology Development, Co Management, Village Level Development, Farmer Cooperatives, Empowering the Marginalized, Localized Innovation Platforms, Enabling Environment, Infrastructure Development, Agricultural Service Delivery, Public Service Delivery, Resource Access, Decentralization, User Groups, Gender Equality, Technology Transfer, Joint Forestry Management, Participatory Irrigation Management, Micro Credit Groups, Farmer Field Schools, Propensity to Cooperate, Invest in Public goods	Randomized Controlled Trial,
Downward Accountability, Upward Accountability, Enhanced Representation, Equal Institutions, Elite Capture, Information Costs, Trust, Human Capital, Coordination Mechanisms, Property Rights, Legitimacy, Transparency, Democracy, Monitoring and Enforcement, Increased Yields, Deforestation, Soil fertility management, Technology Adoption, Poverty Alleviation, Food Security, Common Pool Resource Management, Agricultural Resilience, Land Restoration, Social Justice	Difference in Difference,
	Regression discontinuity designs,
	Propensity score matching

Table A1 lists the keywords and search terms used to identify the relevant studies. These include the keywords used (first column) and the relevant statistical research methods considered (second column).

Table A2 databases used in structured literature search.

Academic Databases	Working Paper Series
AgEcon, British Library for Development Studies, SCOPUS, Science Direct, Poverty Action Lab, JSTOR, IDEAS, JOLIS, Google Scholar and the 3IE Repository of Impact Evaluations.	NBER Working Papers, The World Bank (Policy Research Working Papers), Harvard Business School, Stanford University, Center for Global Development, European University Institute, Columbia University

Table A2 lists the academic databases (first column) and working paper series (second column) that were scanned to identify the relevant studies.

Table A3 Examples of search strings applied.

The search strings were based on the search terms included in Table A1. In principle, these search strings were applied individually, for all text bodies (not just title and abstract) and all given years. To avoid missing results due to misspelling, appropriate wild card and truncation operators were applied as well. Example are given below:

1. (“Institutional Capacity Building”)

2. (“Participatory Technology”)

3. (“Village Level Development”)

4. (“Micro?credit Groups”)

and so on

However, some academic databases (Table A2) are quite extensive (SCOPUS, Science Direct), therefore it was necessary to refine our results. In such cases, we narrowed down the time frame (1990 till present) and/or searched in specific sub-databases (Agricultural and Biological Sciences) Finally, we combined the keywords to identify the relevant interventions and outcome indicators (Table A1) resulting from specific methodological approaches. An example is provided below:

1. (“Inclusive Growth”) AND (“Random?ed Contro* Trial)

? stands for either s or z

* stands for either Control or Controlled

Fig. 2 summarizes the steps implemented in the search and screening process. Applying our search terms to the academic databases and working paper series yielded a total of 57972 records. These were subsequently screened (title and abstract) for further relevance (interventions and developing country setting) narrowing it down to 1865 potentially relevant studies. The full texts in this set were further screened for eligibility in two stages.

The first part of this eligibility screening identified all quantitative studies that described institutional interventions referring to smallholder agricultural populations either at the individual or community level in developing countries. We thereby follow the broad definition of institutions (formal and informal) and include studies that either set to change the institutional setting, or evaluate the impact of a specific existing institutional arrangement. Examples include: setting up village councils; novel arrangements to spur public goods supply and services delivery; farmer cooperatives; institutional arrangements set to overcome failures of financial markets; cash transfers, but only when explicitly linked to a change in a specific institutional setting (like a public works scheme); different types of extension and training (but only when training was organised in a clearly specified institutional arrangement) and institutional arrangements to stimulate efficient resource use (commodity certification, payment for environmental services, input voucher schemes). Moreover, studies that assessed the impact on institutions, but without an intervention set to build or change institutions, were excluded. We only retained those studies documenting outcomes within the realm of inclusive green growth. In addition, it was necessary for the paper to include a credible counterfactual. Therefore, in the second part of the eligibility screening, we scanned the full text detail focusing on the methodological and the results sections. We included only experimental and quasi-experimental studies using rigorous econometric technique (RCT, DiD, RDD and PSM), maximizing the likelihood that the included studies identify true causal effects. Some duplication was removed (working papers and journal publications of the same intervention). Finally, as a robustness check, we asked several peers to double check the final list for completeness. All together we retained 66 studies meeting all these criteria.

We subsequently carried out two further steps. First, each study was annotated by the key intervention(s) described and investigated, the outcome indicators assessed (both intermediate and final) as well as the country in which the study took place. This wielded a great diversity of interventions and impact indicators. We suppressed some of this diversity, primarily for ease of presentation, and categorized the interventions in eight dissimilar categories (Table A4 – Appendix A): i) those that strengthen local institutions, ii) establish producer cooperatives, iii) improve public service provision, iv) empower marginalized groups, v) transfer cash or assets, vi) provide access to financial services, vii) information provision, training and extension services viii) and those creating incentives for better resource use. Similarly, we categorized the intermediate (Table A5 – Appendix A) and final inclusive green growth indicators (Table A6 – Appendix A).

Table A4 Aggregation of interventions into eight distinct categories.

Build or improve local institutions	Establish or improve producer cooperatives	Enhance public service provision	Empower local marginalized groups	Cash or asset transfer	Provide access to financial services	Agricultural extension and skills	Incentives for efficient resource use
– Community Driven Development (CDD)	– Stimulate membership farmer cooperative	– Digitisation of public service provision	Stipulate participation vulnerable groups	– Cash or asset transfer program (including food aid)	– Group based financial services (microfinance etc.)	– Business & entrepreneurial training	– Commodity certification
– Enhanced participatory processes (deliberation, project selection, etc.)		– Stimulate competition in public procurement	Self-targeting disadvantaged groups		– Credit provision	– Agricultural extension: Farmer Field Schools	– Input vouchers
– Introduce democratic practices (of project or council selection)		– Address scale (of irrigation systems)			– Insurance provision	– Agricultural extension: telephone/computer etc.	– Subsidize input use
		– Public works program			– Bank account provision	– Agricultural extension (methods unspecified)	– Payment for Ecosystem Services (PES)
		– Decentralization of governance				– Agricultural marketing activities (e.g. High value crops)
		– Use social networks (for public service provision)

Table A5 Intermediate outcome indicators.

Enhanced local institutions and participation	Improved public service delivery	Better targeted public services	Improved use of financial services
– Trust (experimental measures)	– Public service provision	– Type and location of projects selected	– Use of and access to (in)formal financial services
– Willingness to contribute to public goods (experimental measures)	– Effective public spending: lower costs	– Effective public spending: greater reach
– Other social capital measures or indicators	– Greater security and safety	– Access to rural services
– Performance of local institution, changes in participation (groups) and time spent

Table A6 Final outcome indicators.

Inclusive Growth		Green Growth
Changes in poverty and vulnerability	Changes in household income	Productivity of agricultural land use	Reduced deforestation
– Vulnerability to income shocks	– Household income	– Agricultural productivity	– Forestry practices & (de)forestation
– Household income of vulnerable groups	– Household expenditures: consumption or investment	– Agricultural technology uptake or adoption
– Household expenditures of vulnerable groups	– Economic perception	– Knowledge on improved agricultural practices
– Poverty gap or poverty headcount	- commercialisation and income diversification	– Input use or uptake
- Food security (status)		– Input use, uptake or knowledge on pesticides
		– Environmental awareness

In all these cases, the process was driven by the diversity encountered in the studies, and less by an a priori establishment of categories. Arguably this resulted in a narrow operationalization of IGG, particularly for green growth indicators. We captured the latter through only two sets of indicators encountered, namely i) enhanced productivity of land use and ii) reduced deforestation. This choice is motivated by the frequent call to raise land use productivity on existing crop lands, in order to prevent the conversion of remaining pristine areas into agricultural area (FAO, 2009; PBL, 2012). However, it remains a narrow operationalization of green growth, even within the realm of agriculture, but foremost signals that only few studies assessed impact on green growth indicators.

Second, we assessed the statistical rigour of each paper following a risk of bias assessment tool¹ (EPHPP, 2016). Using these guidelines, we assessed how studies addressed (i) Selection Bias, (ii) Allocation Bias and (iii) Confounders,² (iv), Withdrawals and Dropouts (v). We subsequently allocated for each category a score (1 = weak; 2 = moderate; 3 = strong) and summed the scores over the four categories. A study with a final score of ten or greater was labelled: “strong confidence”. A study with a score of six or lower was labelled “weaker confidence, with multiple issues”. The remaining studies were labelled: “good confidence, with minor issues.

Not surprisingly, given our focus on statistically rigorous impact evaluations, the vast majority of studies were labelled as statistically solid (strong confidence). We identified two studies with weak confidence, due to unclarities and/or lack of control for various biases. These studies failed to report important elements of the intervention implementation process and are therefore not discussed in further detail. Eight studies had some minor issues but were classified as ‘good with minor issues’. The list of remaining studies is up to date up till June 2016 and a full annotated list of studies is included in Appendix B. The list includes describes intervention categories, intermediate and final output categories and the statistical appraisal.

3 Results

We identify 66 studies that meet our inclusion criteria. Below we organise the studies by intervention type and outcome group. Fig. 3 provides the classification, where we list the interventions row-wise and the intermediary and final outcomes column-wise. Each study is assigned to an intervention – outcomes cell. Where multiple interventions or multiple output indicators are included a single study may appear multiple times.

Fig. 3 Evidence Gap Map.

Most interventions target outcomes related to income. Evidence, however is not spread out evenly across the interventions and outcomes. One larger cluster of studies straddles the axis between institutions building and participation and governance quality. Another focuses on financial inclusion. A third cluster sits around agricultural interventions and its impact on productivity. We should emphasize that such clusters are not suggestive of greater evidence for a (positive) impact of an intervention on outcome indicators. Rather, they suggest that these relations have been investigated with greater frequency, irrespective of the actual impact documented.

There are at the same time, some key limitations and gaps. For instance, while the promotion of producer cooperatives has been common in development practice, only few rigorous studies focus on this specifically. There are also few studies that assess interventions on empowering marginal groups and few studies attempt to assess the eventual impact on poverty.

The Evidence Gap Map contains all studies retained, irrespective of whether they assess intermediate outcomes, final outcomes, or both. Most studies either report on intermediate outcomes, such as increased participation, improved access to financial services or enhanced public service delivery, or on final outcomes, such as improved agricultural productivity, increased household income and reduced poverty and household vulnerability. A variety of studies thus assess attempts to build local institutions, but most of the evidence stays clear of final outcome indicators.

To reflect on the common assumption that strengthening local institutions is beneficial to IGG, we identify the subset of studies that measured impact of both intermediate and final outcomes and can thus provide insights into such a causal mechanism. We identify fourteen studies that report on both intermediate and final outcomes (Fig. 4). Of these, most report on either poverty levels or agricultural productivity outcomes, but none report on inclusive and green outcomes jointly. Altogether this makes it difficult to substantiate the hypothesis that interventions targeting local institutions spur IGG.

Fig. 4 Studies reporting final outcomes¹.

¹The total number of studies displayed in Fig. 4 exceeds fourteen since some studies evaluate multiple combinations of intermediate and final outcomes. The outcome indicators result from different interventions, information on which has been suppressed in this figure for clarity

Looking at global spread, most of the interventions evaluated (37) describe and assess interventions in Africa, followed by Asia (31) and Latin America (7). Fig. 5 displays the distribution of studies across continents. There are no large differences between Asia and Africa, except for the fact that all studies on producer cooperates are carried out in Africa. The latter reflects the dominance of producer cooperatives in primarily Ethiopian development programs. Most of the studies from Latin America describe cash transfer interventions, reflecting the more common use of this intervention on this continent.³

Fig. 5 Distribution of interventions evaluated across continents¹.

¹The total number of studies displayed in Fig. 5 exceeds the overall number of 66 studies identified since some studies evaluate multiple interventions, or combinations thereof.

Below we discuss the contents of the overview in greater detail by intervention type and relate these to outcomes groups (or lack there of).

3.1 Building and improving local institutions

3.1.1 Rationale

There is a substantial body of evidence on the impact of interventions aimed at building and improving local institutions. We identify nineteen studies that investigate the impact of these type of interventions on intermediary outcomes such as the functioning of local institutions and public service delivery. Seven studies examine the impact on income and poverty, as well as agricultural productivity. Even though the exact design of the interventions and studies differ, the dominant feature is to enhance the participation and voice of all social groups in community decision-making. The rationale is that better representation leads to the delivery of public goods and services that is of greater benefit to all members of the local community, rather than serving the interest of a few. It is thus a mechanism to stem elite capture. All interventions aim to form local development councils, or to improve the functioning of existing ones. Often this includes democratization and inclusion of vulnerable groups. Development councils are often tasked with overseeing the allocation of a block grant, to be spent on local public goods, food aid distribution or selecting recipients for cash grants.

3.1.2 Findings

Most studies find a positive effect on the functioning of local institutions, the efficacy of local governance and the effectiveness by which development budgets are spent. Examples include improved perceptions of local governance effectiveness (examples include Beath et al., 2012b; Casey et al., 2012; Fearon et al., 2011; Nguyen and Rieger, 2014). In most cases, the willingness to make private contributions to public goods increases as well as the trust placed in others. One exception is Nguyen and Rieger (2014) who find that while the willingness to contribute to public goods increases, trust in co-habitants decreases. Notably, another study does not find any change in measures from behavioural games (Avdeenko and Gilligan, 2015). Sometimes, villagers vote with their feet, and spend more time on participatory meetings and gatherings (Labonne and Chase, 2011).

While the exact change in behaviour appears to be somewhat ambiguous, rather robust evidence emerges that public service delivery can become more effective and often better targeted. For example, in Bangladesh, Madajewicz et al. (2014) find that councils with equal representation from various groups, provide arsenic free drinking water to significantly more households than councils or external consultants. In Afghanistan (Beath et al., 2013a) democratically elected councils raise the odds that the neediest households receive food aid. In Indonesia, council decisions on infrastructure projects are less prone to corruption, when a broad representation of villagers participate in audit meetings.

Even though some of these outcomes are modest, they are both statistically and economically significant. Implemented projects and service delivery become more beneficial to the wider community and less tailored towards the interests of the elite. In some instances, even the overall cost of public service delivery decreases (Olken, 2007). Thus, incidences of elite capture and/or corruption appear to reduce. Indeed, in one study (Beath et al., 2012a, p. 14Beath et al., 2012aBeath et al., 2012a, p. 14) the authors point out: “elite influence over allocation decisions by councillors (…) is perceived by villagers as malevolent capture rather than benevolent control."

While improved local institutions are expected to lead to improvements in economic indicators, such evidence has not yet been established unambiguously. Eight studies assess the impact on inclusive or green (economic) growth indicators. Only three studies (Beath et al., 2012a; Casey et al., 2012; Fearon et al., 2011) report on changes in household income or consumption and intermediate institutional outcomes. Only one of these (Casey et al., 2012) finds a positive short-run economic impact. The other five studies only report on final outcomes. Of these Voss (2008) is the only to report the impact on the poorest households, noting this group to benefit relatively more, but only so in the poorest communities.

Some evidence that institutional interventions could lead to green growth, or green growth trajectories comes from a few studies on projects aimed at stimulating forest co-management. Even though these appear to be relative effective, with declining rates of deforestation, project participation is skewed towards specific groups like larger, younger and female headed in Malawi (Mazunda and Shively, 2015). If and how vulnerable households can, or should, be included in such programs remains an open question.

A key caveat, however, for any of the studies is that little is known about whether such changes are durable and prolonged. Even though these institutions are reasonably expected to contribute to IGG in the long run, few studies are conducted at a sufficiently long time-scale to asses such impact.

3.2 Establish or improve producer cooperatives

3.2.1 Rationale

Even though the promotion and development of producer cooperatives has been common throughout recent development history, there are surprisingly few studies analysing their impact rigorously. We identified nine studies, of which only four are classified as providing evidence with strong confidence. By and large all studies are relatively similar featuring training on improved planting material or cropping systems. On the marketing side, cooperatives aim to integrate the cooperative better into markets and/or value chains, through aggregation across producers, enhanced quality control, or both. Finally, producer cooperatives may wield increased negotiation power, which could result in a more beneficial price setting. One intervention is relatively novel, whereby participating households buy into the cooperative through shares. The more shares are purchased the greater the revenue from the proceeds.

3.2.2 Findings

Compared with the rather detailed analyses on various institutional indicators as reported in the previous section, the studies identified here (Abebaw and Haile, 2013; Fischer and Qaim, 2012; Matchaya and Perotin, 2013; Tilahun et al., 2016) provide only superficial insight into changes in institutional characteristics. A core theme are selection effects. Newly formed cooperatives bind specific groups. More often than not, these are groups that already are better off to start with. Often male-headed, better educated and already in the possession of more assets. None of the studies ventures deeper into changes and impact on local institutions. It thus remains to be seen how the formation of such, potentially influential, producer cooperatives impacts other groups in local communities, either through changes in local public service or good provision, or through latent behavioural changes such as trust amongst villagers (either outside or within the cooperative).

All studies find a noticeable impact of the formation of producer cooperative on household income. Finally, the formation of cooperatives could accomplish green growth objectives through a more intensified land use. Here, the impact is less firmly established, with most studies only assessing changes in input use such as fertilizer and pesticides. None of the statistically most rigorous studies investigates impacts on downstream outcomes such incomes or crop yields.

3.3 Enhance public service provision

3.3.1 Rationale

We categorized nine studies as interventions that enhance public service provision. The common denominator is the aim of exploring and assessing novel methods of providing current public services, with the same level of output, at lower costs and less corruption. These include analyses on other ways to organise agricultural extension (Banerjee et al., 2015c; BenYishay and Mobarak, 2015), provide food aid (Banerjee et al., 2015c), manage forestry schemes (Somanathan et al., 2009) or implement public works programs (Banerjee et al., 2014; Adimassu and Kessler, 2015).

Given the diversity of studies considered, the mechanisms differ as well. In all instances, the interventions are channelled through local institutions, either existing local institutions are used to make top-down interventions more effective, or local institutions are incentivized to deliver more effective outcomes. First, the digitization of fund transfers in a public works program in India (Banerjee et al., 2014) is likely to reduce the potential for local elite capture and corruption. Second, in Indonesia (Banerjee et al., 2015c) stimulating competitive bids by private suppliers to supply and distribute food aid to the neediest, and subsequent control by villages themselves is expected to reduce corruption. Other interventions explore novel ways to organise agricultural extension, for instance by setting up a telephone helpdesk for farmers (Cole and Fernando, 2012), or smartly distributing information through farmer social networks (BenYishay and Mobarak, 2015). Another uses social networks to disburse information on a new weather insurance scheme (Cai et al., 2015). Finally, one study assesses the performance of communities in managing forest schemes in relation to state-managed forests (Somanathan et al., 2009).

3.3.2 Findings

Surprisingly, none of the studies assesses the impact on local institutions, for instance through behavioural experimental measures such as trust. By contrast, all of the studies assess intermediary impact by means of the increase in effectiveness of service provision, that is whether the same services can be delivered at the same costs. Digitization in the public works program in India delivers the same output at lower cost (Banerjee et al., 2014), stimulating competitive bidding in the food aid program in Indonesia reduces costs (Banerjee et al., 2015c) and management costs in community managed forests in India are lower than state managed forests (Somanathan et al., 2009). The telephone helpdesk in agricultural extension (Cole and Fernando, 2012) is able to reach a large group of farmers at a low cost and with an impact on changes in land use.

Few of these studies investigate the subsequent impact on household income, on the presumption that existing public services remain unaffected, rather being supplied at lower cost. A few, mainly the ones exploring new ways of delivering extension, assess impact in cropping decisions and productivity of land use. The studies report positive and significant changes, ranging from increased fertilizer use, increased knowledge on effective pesticide use and increased technology adoption, but no study reported actual changes on crop yield.

3.4 Empower local marginalized groups

3.4.1 Rationale

We found five studies that explicitly aim to get the voices of the most vulnerable represented, making development processes more inclusive and potentially longer-lasting. Three studies describing such effects have been discussed in Section 3.1 (Casey et al., 2012; Beath et al., 2013a; Madajewicz et al., 2014) where seats in the local councils are reserved for poor and vulnerable groups. Here, we focus on the remaining two studies which consider different mechanisms for empowering the poorest of the poor.

First, one study investigates the most effective way to transfer cash grants to the most vulnerable households (Alatas et al., 2016). Recipients are typically identified through screening by project staff on observable assets. It is, however, hypothesized that self-application may be a more efficient means as applicants consider all assets, some of which are unobservable to project staff. Statistical confidence in the results of this study is, however, lower than the others. Second, one study from India assesses the impact when only women can stand election for council leader (Chattopadhyay and Duflo, 2004), on the supposition that women will better represent the interest of the wider community.

3.4.2 Findings

The novel design of the cash grant scheme, making use of self-application (Alatas et al., 2016), leads to superior outcomes, both through lower program costs and better targeting of grants. The impact of the other intervention is ambiguous and sobering. Women council leaders (Chattopadhyay and Duflo, 2004) use local resources differently and spend more on drinking water projects, but noticeably less on infrastructure and education. Whether such changes are of overall benefit to local communities is debatable. Only the cash grant study (Alatas et al., 2016) assesses impact on poverty and estimates a substantial reduction in poverty gaps compared with a top-down screening.

3.5 Cash and asset transfers

3.5.1 Rationale

We retained six studies that describe interventions on cash or assets transfers, many of which assess impact in Latin America, reflecting its more common use on that continent. Cash or asset transfers serve to cushion consumption fluctuations in poor and vulnerable households, or to provide these households with a head start to enhance their economic productivity. Contrary to the previous section, in which studies aimed to raise the voice of vulnerable and marginalized groups, these interventions primarily aim to supply economic resources to these groups. A major challenge, however, is to identify the households for which the transfer would be most beneficial.

We emphasize that, as per our search strategy, we retained only eight studies that are explicitly linked to local institutions. Arguably, the body of rigorous statistical evidence on these interventions is considerably larger (e.g. Fiszbein and Schady, 2009). Overall, the wider evidence on cash and asset transfer suggests these to be relatively effective, a finding from which the six studies retained here do not digress.

3.5.2 Findings

Three studies investigate methods to improve the selection mechanisms through local institutional structures. These only assess the impact on the efficiency of targeting, and provided no further insights into the functioning or changes in local institutions. Two of these have been discussed earlier (Beath et al., 2013a; Alatas et al., 2016), with increases in effectiveness over top-down screening. A third study adds a further perspective, by combining both council selection and self-targeting (Banerjee et al., 2015b). The reasoning is that councils as well have imperfect information on those most in need of transfer. It turns out that distributing cards in villages with information on the program (distributing rice to the neediest) and criteria for eligibility, greatly improves the targeting over council selection. While the efficiency of the interventions appears to improve, little is known on the eventual impact on income indicators. Only one study reports on reductions in poverty gaps (Alatas et al., 2016). Another study uses participatory wealth ranking to distribute livestock to the neediest households (Banerjee et al., 2015a), observing significant changes in consumption.

3.6 Provide access to financial services

3.6.1 Rationale

Fourteen studies reported on the impact on the provision of (rural) financial services. These include the provision of bank accounts, (weather-indexed) insurance, group-based savings schemes or loans, often provided in conjunction with rural extension schemes. Most studies assess the effect of the new financial institutional arrangements on production decisions and income or consumption. Only, a few delve into specific institutional characteristics. One study describes a group-based savings scheme, where members develop and enforce own rules for saving loan taking and repayment (Beaman et al., 2014a). Within group trust and norms are expected to reduce defaulting. And another intervention, again assesses whether selecting recipients for a grant by a local council (Beaman et al., 2014b) could improve targeting of a credit scheme.

3.6.2 Findings

Few studies assess the impact on a detailed institutional level, the majority only reporting on significant changes on product take up. All studies report institutional changes at the most formal level, namely the take-up of financial services due to the development of the financial institution under study. Many studies proceed to report changes on income and poverty levels. Indeed, group-based saving with internal rule enforcement has a significant impact on consumption smoothing, with lower levels of food insecurity (Beaman et al., 2014a). Local councils are also more effective in selecting the most promising local recipients for credit (Beaman et al., 2014b). Even though these studies do not reflect the impact of institutional interventions per se, the evidence illustrates how local institutions can be used to make interventions more effective.

3.7 Agricultural extension and skills

3.7.1 Rationale

A total number of fifteen studies investigate the relationship between the provision of agricultural extension or skills training and changes in income and agricultural productivity. Knowledge on new and improved (agricultural) technologies, and the provision thereof, is of great importance for increasing (agricultural) productivity. Its provision, and associated processes of social learning are of a public good nature. Adapting new technologies to local conditions entails experimentation, for which the first experimenters are often not rewarded by other adopters. A number of studies illustrate how institutional interventions can partially internalize such learning and experimentation costs. For instance, by forming producer cooperatives (Fischer and Qaim, 2012; Matchaya and Perotin, 2013), or by making smart use of local institutions, that is farmers’ social networks (BenYishay and Mobarak, 2015). Finally, the prohibitive costs of providing large-scale extension can be mitigated by deploying a phone-based helpdesk (Cole and Fernando, 2012).

3.7.2 Findings

Only three studies report on intermediate outcomes (Cole and Fernando, 2012; Fischer and Qaim, 2012; Matchaya and Perotin, 2013), the key findings of which have already been discussed above. We reiterate that these provide, however little, evidence of how local institutions (cooperatives) can be formed or used (social networks) to increase the effectiveness of extension. A more detailed impact on local institutions also through time, remains to be investigated.

Extension channelled through social networks (BenYishay and Mobarak, 2015) leads to significant increases in adoption of a new technology. Most of the other studies retained, provide evidence of selected agricultural technologies on farm household income and agricultural productivity. These studies provide, as per our search criteria, the best available statistical evidence of their impact. However, no uniform results emerge, rather they reflect the multitude of improved agricultural technologies available, and their impact varying across different settings.

3.8 Incentives for efficient resource use

3.8.1 Rationale

Five studies provide insights into the effect and impact of changing incentives, mainly for rural smallholder producers. These include the subsidized provision of inputs producers, product certification or the use of Payment for Ecosystem Services (PES). In each of the three types of interventions, the individual incentives change. Subsidies make inputs less costly (Awotide et al., 2013; Carter et al., 2014), and certification makes output more rewarding (Van Rijsbergen et al., 2016). Either way producers are expected to increase productivity. In PES systems (Hegde and Bull, 2011; Arriagada et al., 2012) farmers are remunerated financially for the supply of ecosystem services, such as maintaining wooded areas.

3.8.2 Findings

Only one study on a PES scheme (Hegde and Bull, 2011) provides evidence on the impact of local participation. The study finds that, similar to studies on producer cooperatives, relatively wealthier and male-headed households were likely to join the PES programme. None of the other studies provide insights into the impact on local institutions. Subsidized inputs do indeed increase input use (Carter et al., 2014; Awotide et al., 2013) and may lower poverty levels (Awotide et al., 2013). The two studies on PES suggest that income of participating households may increase (Hegde and Bull, 2011), while the forest cover could increase (Arriagada et al., 2012).

4 Discussion

By design Evidence Gap Maps reveal both evidence clusters and evidence gaps. While the latter should inspire researchers in conducting more rigorous evaluations (Section 4.2), the former should spur ways to learn (more) from the existing evidence (Sections 4.1 and 4.3). In this section, we discuss both ways forward by highlighting three key benefits, for policy-makers and researchers alike. First (1) Policy-makers benefit from this knowledge repository by gaining insights into the types of interventions that have been tried and tested and are available for use in other settings. Despite the revealed diversity in interventions, in equally diverse countries, these studies contain valuable information on the potential to apply such interventions elsewhere. Insights on the mechanisms described in the various studies (and Section 3), combined with information from potential project locations, guide policy-makers in matching the right intervention with the right location. We thus provide an operationalization of the concept of institutional diagnostics (Section 4.1). The EGM also serves as a fact-check that actual evidence for some commonly perceived wisdoms in development practice remains scarce. Indeed, the EGM allows researchers to identify key evidence gaps (Section 4.2). Finally (Section 4.3), the EGM uncovers areas with sufficient numbers of studies to make additional statistical meta-analysis worthwhile. But, it should also inspire additional interdisciplinary research to unravel the exact chain of events through which impact comes about. We discuss these three contributions below.

4.1 Guiding institutional diagnostics

Hausmann et al. (2008) and Rodrik (2010) emphasize that it is unlikely that all potential factors constraining (in our case) Inclusive Green Growth are binding simultaneously. Rather, some market and governance failures are more pressing than others and the challenge to the diagnostician, in a particular location, is to identify the right intervention targeting those peculiar constraints. The EGM, and the detailed evidence-based information it stores, are an important building-block in this quest and we illustrate so by using a framework developed by Bates and Glennerster (2017). It serves to build an understanding on where and when an intervention could be used effectively in another setting. The framework revolves around four steps, respectively, (1) detailing the (institutional) mechanism underlying the intervention, (2), assessing the local constraints and conditions, (3) reflecting on the (range) of effect sizes and finally (4) see whether the intervention requires adaptation to the particular context. By considering a few of the studies identified in the EGM we use the first two steps to operationalize institutional diagnostics. Step 3 builds on insights from a statistical meta-analysis (see Section 4.3 below) and Step 4 considers practical policy arrangements. A detailed reflection on the latter two is outside of the scope of this study.

First, the interventions identified build on universal human responses. Such responses are likely to be very similar in the wake of similar constraints, or the relieve thereof. For instance, when given a chance, elite groups divert more public resources for their own benefit. Many of the papers on building institutions and empowering marginalized groups set to change this disconnect between actual and desired governance. Successful interventions introduce forms of (social) punishment for corruption or elite grab (e.g. Olken, 2007) thereby placing more scrutiny on public spending. Or, interventions amend local decision-making structures through ballots or reserving seats in councils for vulnerable groups (Olken, 2010; Beath et al., 2012a). As described in Section 3, these (slight) changes in the governance structure incentivize more and better public goods delivery to marginalized groups. Even though the intervention may need slight adaptations in different settings, these responses will be similar when similar constraints bind. Another set of studies exploits existing social networks, on the understanding that people generally place more trust in close kinship and friends. Again, there is no reason to believe that this key premise will vary greatly across different settings, but rather that it is a generic characteristic of human social interaction (Apicella et al., 2012). The fact that people are more likely to share both information and risks with people closer to them in their social network is a useful characteristic for making public services, like agricultural extension or information provision more effective (BenYishay and Mobarak, 2015; Cai et al., 2015), but again interventions may need tweaking to local conditions.

Second, it remains important to gain an understanding on the locally binding constraints. In new settings, this may proceed through some form of Rapid Rural Appraisal (RRA) combined with local expert knowledge. Such a RRA may indeed reveal that local councils are present, but that many inhabitants feel these do not represent the broader local society. Then, the Evidence Gap Map presents the set options to intervene, whereby local conditions may further lead to adjustments in intervention designs. Ballots may be introduced to vote on spending of local public resources, meetings can be organized to review public investments and curb incidences of corruption. In an extreme case a RRA may reveal the near absence of functional councils. Then the studies on developing elected councils in Afghanistan (Beath et al., 2012a, 2013a, b) provide guidance. And even though social networks are ubiquitous, a choice to use a social network-based intervention (say for agricultural extension) still needs to be informed by an understanding of the local context. For instance, a village with distinct sub-groups that see little social interaction (like castes in India or otherwise different ethnic groups) may be mirrored in a set of unconnected social networks and relative great barriers to information flow on agricultural technologies between groups. Then social-network based interventions, targeting each of the various networks, may provide a means to more efficiently deliver agricultural extension to all groups. Hence, the process of selecting an intervention always needs to be informed by detailed information on the local setting. Simply proposing an intervention because it proved successful elsewhere is bound to be counterproductive, a reasoning intuitively similar to (statistical) concerns on external validity (see Section 4.3).

4.2 Making sense of the evidence, or lack thereof

The EGM illustrates that studies are not evenly spread across the full intervention-output matrix. Clusters emerge for some intervention – output combinations, in contrast with some noticeable evidence gaps. More precisely, a clustering emerges along the diagonal from the upper left to the lower right. In part this reflects common logic. Studies aimed at building local institutions more often assess changes in these institutions; studies aimed at providing financial services often assess the take up of financial services; etc. However, important areas of research are left blank that should warrant closer attention from researchers. Moreover, they should come as a warning to policy-makers as it shows that actual evidence for some commonly perceived wisdoms in development practice remain scarce.

We set out to assess if ‘better local institutions’ contribute to IGG. We highlight above that such a causal chain occurs in some locations, but not everywhere. Moreover, Fig. 3 highlights that few studies investigate the full chain from intervention to intermediate outcomes and then to final outcomes. These findings are in line with other studies (Mansuri and Rao, 2004; Lawry et al., 2016) possibly because of narrow time frame of most project evaluations. This is not necessarily a bad thing. One can argue that positive changes in intermediate outcomes, such as a more efficient and better targeted supply of public goods to local citizens, next to better representation, are commendable targets by themselves. But it does present a challenge to policy-makers seeking to promote IGG. At least in the short run that is, since no studies investigate impact over a time frame longer than 3 years. After all, it is plausible that it takes time for such interventions to ‘institutionalize’.

With regards to final outcomes, specifically considering the inclusiveness of growth, the EGM shows that most studies assess mean income effects, and only few investigate the effect of interventions on different income groups. A mean effect could obfuscate differences in impact across subgroups in the sample. Income effects could even be negative for some. One of the clearest examples is given by Voss (2008), who finds positive economic impact of a CDD in the poorest communities under investigation, as opposed to insignificant and sometimes negative impact in wealthier communities. In other words, reporting a mean positive income change does not provide a guarantee for inclusive growth, a claim that can only be supported through additional analysis.

Despite the broad search procedure set to identify and include studies assessing impact on Green Growth, only very few rigorous studies were identified. Moreover, there proved to be only limited variation in the output indicators assessed. This led to our narrow operationalization of green growth through crop yields and deforestation. Only two studies report on the use of pesticides, one of which reports only on the perception of its impact. Altogether this points to some key knowledge gaps, in line with Ferraro et al. (2011) and Mansuri and Rao (2004).

At the same time, the impact on intermediate outcomes goes unnoticed in many of the other studies, for instance, amongst interventions aimed at agricultural extension or incentivizing efficient resource use. These studies were retained in our EGM as they rely on, or describe a specific substantial institutional component. In some cases, new institutional structures are set up, such as group-based saving schemes or farmer field schools. In others, the interventions make smart use of existing institutional structures. These include the idea to disseminate agricultural technologies through social networks (BenYishay and Mobarak, 2015; Beaman et al., 2015) or the idea that local councils are better placed (better than project administrators) to identify recipients for food aid or credit schemes (e.g. Beath et al., 2013a; Banerjee et al., 2015b). These studies show an increased effectiveness of the project, either a through better targeting, or through lower costs for a given effect.

Yet potentially important changes warrant closer attention. These interventions are likely to redraw the local pre-existing institutional picture, either in a positive or in a negative way. New social connections could be formed, and trust or willingness to cooperate may increase. This is most obvious when considering the studies on farmer cooperatives, which are typically formed by specific groups in local societies. The discussion highlights that these selection effects are sizeable and benefits of the intervention accrue mostly to specific farmers, typically not the most vulnerable. If and how vulnerable groups can (or should) be included in cooperatives remains an open question.

Despite these knowledge gaps (foremost signalling important areas for additional research) the EGM also reveals areas in which relatively greater amounts of information is available, which forms a sound base for additional meta-analyses.

4.3 The EGM as a sound base for a systematic review

Even though studies on similar interventions sometimes point to similar outcomes, one should then be careful in projecting impact to areas not covered in these studies. First, impact from interventions is likely to be heterogeneous and context-dependent (Deaton, 2010; Pritchett and Sandefur, 2013). Indeed, heterogeneity in program impact is a logical implication of Rodrik’s argument on binding constraints. Not all constraints bind IGG by the same amount across different contexts. Consider, for instance, our observation that many studies aimed at local institution building, through Community Driven Decision-making (CDD), report positive outcomes at an intermediate level. In many instances, positive impacts at local public service delivery are documented, albeit assessed at a variety of indicators. A simple and tempting stance would be to conclude that a lack of local participatory decision-making constrains IGG across multiple contexts. But the picture revealed by the EGM is in fact complicated. For Sierra Leone, Casey et al. (2012) observe an increase in public goods provision and positive economic impacts. Further down along the West African coast, in Liberia, Fearon et al. (2011) find increases in trust and willingness to contribute to public goods, but no effects on economic outcomes. In contrast, in Morocco, Chi Nguyen and Rieger (2014) do document increases in public good contributions but they observe reductions in trust (and they do not assess economic indicators). Moreover, not all studies on CDDs reported significant changes. For instance, Avdeenko and Gilligan (2015) did not observe statistically significant changes in behavioural outcomes. This could be due to a lack of statistical power that obfuscates a true non-zero impact (either positive or negative).

Heterogeneity and low statistical power are reasons why tallying of evaluations (i.e. taking multiple positive impact assessments as proof for a generalized effect) is undesirable. Other arguments include selection and publication bias (see for a more comprehensive discussion: Vivalt, 2015): the studies retained, and the countries in which these were carried out, are not random draws. They are often conducted in places where stakeholders have an interest in showing impact. Given the resources involved, RCTs are often conducted in relatively stable countries (Blair et al., 2013). Fig. 5 lends some credence to such an argument. Finally, publication bias and specification searching may lead to an underreporting of insignificant or negative findings (Brodeur et al., 2016), something that pervades all studies included in our database.

Even though these arguments limit direct inference on a generalized effect from a stock of evaluations the structured (and replicable) literature search underlying this paper (Section 2) and the ranges of impact described (Section 3) open up possibilities for additional statistical analysis (e.g. Waddington et al., 2012) in a systematic review. Such a review is outside of the scope of this study, but can deliver confidence intervals of impact across a diverse range of settings. Such statistically robust estimates also allow for more precise estimates of project costs and benefits and may thus guide policy-makers in the ultimate choice of using an intervention in a new setting.

5 Conclusion

It is widely acknowledged that institutions serve a crucial role in supporting Inclusive Green Growth. However, the policy enthusiasm has outstripped the available evidence and it remains unclear what can be learned from the interventions that seek to build or strengthen institutions. Have these interventions resulted in the desired effects and how can we learn from this cumulative set of studies for guiding more effective development practice?

To address these questions, we implemented a structured literature search and used this to construct an Evidence Gap Map (EGM). It synthesizes information on building local institutions and highlights where evidence is available and where it is still lacking. We identify eight types of interventions focussing on building local councils, improving producer cooperatives, enhancing service provision, empowering marginalized groups, cash transfers, providing access to financial and extension services and incentives to promote efficient resources use. We find that on the whole improving local institutions, or improving existing ones can improve the delivery and targeting of public services and overall satisfaction with local governance. However, this insight is based on a diverse set of interventions evaluated in a wide range of conditions. A subset of studies looks at outcomes directly related to IGG and some report positive impacts on household income and agricultural productivity. The evidence, however, is not evenly spread across all potential interventions. The EGM reveals major knowledge gaps that should guide the future research agenda.

The challenge for the institutional diagnostician is to unlock this cumulative body of information effectively. As Rodrik stresses: ‘we need a systematized way of choosing among them for the context at hand’ (2010, p. 43) since (in Rodrik’s words (2008)): “not all constraints bind equally”. Thus, the challenge to the diagnostician is to identify the right intervention targeting those peculiar constraints that are most binding in a particular location. Further, EGMs can serve as a knowledge repository as it collates the best available evidence on impact of (novel) interventions, and the underlying theories or mechanisms of change, across a variety of settings. This, combined with knowledge on local constraints, guides development practitioners in identifying the most appropriate interventions for other settings. In addition, systematized literature search provides a sound base for conducting a more statistically oriented systematic literature review. Such a review could yield precise bandwidths of the magnitude of impact across a variety of settings.

Acknowledgements

We thank the editors and two referees for helpful comments. Thanks to Filip de Blois (image editor at the Netherlands Environmental Assessment Agency) for the design of figures. PBL Netherlands Environmental Assessment Agency received funding from the Netherlands Directorate-General for International Cooperation (DGIS) for conducting this research. We further acknowledge financial support from the Netherlands Organisation for Scientific Research [N.W.O. grant #451-14-001] We declare no competing interests. The funders had no role in the study design, data collection, data analysis, data interpretation, or writing.

Notes

1 We have followed the Effective Public Health Practice Project (EPHPP) framework (http://www.ephpp.ca/tools.html). It presents a systemized method to test the specific assumptions underlying claims of causality in statistical studies. As the name suggests, it has initially been developed to assess health interventions, but the framework is sufficiently general to apply it in other settings also and matches closely with other risk of bias tools. See e.g. Waddington et al. (2012).

2 It proved to be impossible to assess studies on the categories Data Collection Methods, Blinding, Analysis and Intervention Integrity as outlined in the EPHPP framework. Either the relevant information was not provided, or the variation was minimal across the studies, or the category was less relevant to interventions in the social sciences (e.g. double blinding).

3 For instance, two famous and long running cash transfer programs are Oportunidades in Mexico and Bolsa Familia in Brazil.

Appendix A

Literature search procedure

Appendix B

List of papers identified in structured search

Table

Authors and Year	Title	Country	Appraisal	Intervention(s)	Intermediate outcome(s)	Final outcome(s) Inclusive Growth)	Final outcome(s) Green Growth
Abate et al. (2014)	Impact of Agricultural Cooperatives on Smallholders Technical Efficiency: Empirical Evidence from Ethiopia.	Ethiopia	Good confidence, with minor issues	Establish or improve producer cooperatives			More productive (agricultural) land use
Abebaw and Haile (2013)	The Impact of Cooperatives on Agricultural Technology Adoption. Evidence from Ethiopia	Ethiopia	Strong confidence	Establish or improve producer cooperatives	Enhanced local institutions and participation		More productive (agricultural) land use
Adimassu and Kessler (2015)	Impact of the Productive Safety Net Program on Farmers Investments in Sustainable Land Management in the Central Rift Valley in Ethiopia.	Ethiopia	Strong confidence	Enhance public service provision			More productive (agricultural) land use
Alatas et al. (2016)	Self Targeting: Evidence from a Field Experiment in Indonesia	Indonesia	Good confidence, with minor issues	Empower local marginalized groups	Improved public service delivery	Reduced poverty and vulnerability
				Cash or asset transfer	Better targeted public services
Ali and Abdulai (2010)	The Adoption of Genetically Modified Cotton and Poverty Reduction in Pakistan	Pakistan	Strong confidence	Provide access to financial services			More productive (agricultural) land use
Arcand and Wagner (2016)	Does CDD Improve Inclusiveness in Peasant Organizations? Evidence from Senegal	Senegal	Strong confidence	Build or improve local institutions	Enhanced local institutions and participation
Arriagada et al. (2012)	Do PES Reduce Deforestation? A Farm Level Evaluation from Costa Rica	Costa Rica	Good confidence, with minor issues	Incentives for efficient resource use			Reduced deforestation
Ashraf et al. (2009)	Finding Missing Markets (And a Disturbing Epilogue): Evidence from an Export Crop Adoption and Market Intervention in Kenya	Kenya	Strong confidence	Provide access to financial services			More productive (agricultural) land use
				Agricultural extension and skills
Avdeenko and Gilligan (2015)	International Interventions to Build social Capital. Evidence from a Field Experiment in Sudan	Sudan	Strong confidence	Build or improve local institutions	Enhanced local institutions and participation
Awotide et al. (2013)	The Impact of Seed Vouchers on Poverty Reduction Among Smallholder Rice Farmers in Nigeria	Nigeria	Strong confidence	Incentives for efficient resource use		Reduced poverty and vulnerability
						Improved household income
Banerjee et al. (2014)	Can E-Governance Reduce Capture of Public Programs? Experimental Evidence from a Financial Reform of India's Employment Guarantee	India	Strong confidence	Enhance public service provision	Improved public service delivery
Banerjee et al. (2015b)	Contracting out the Last Mile of Service Delivery: Subsidized Food distribution in Indonesia	Indonesia	Strong confidence	Enhance public service provision	Improved public service delivery
Banerjee et al. (2015b)	The power of transparency: information, identification cards and food subsidy programs in Indonesia	Indonesia	Strong confidence	Cash or asset transfer	Better targeted public services	Reduced poverty and vulnerability
Banerjee et al. (2015a)	A Multi-Faceted Program Causes Lasting Progress for the Very Poor. Evidence from 6 Countries.	Ethiopia	Strong confidence	Cash or asset transfer		Reduced poverty and vulnerability
		Ghana
		Honduras
		India
		Pakistan
		Peru
Beaman et al. (2014a)	Saving for a (not so) Rainy Day: A Randomized Evaluation of Savings Groups in Mali	Mali	Strong confidence	Provide access to financial services	Improved use of financial services	Reduced poverty and vulnerability
Beaman et al. (2015)	Can Network – Theory Based Targeting Increase Technology Adoption?	Malawi	Strong confidence	Enhance public service provision			More productive (agricultural) land use
				Agricultural extension and skills
Beaman et al. (2014b)	Selection into Credit Markets. Evidence from Agriculture in Mali	Mali	Strong confidence	Build or improve local institutions		Improved household income
				Provide access to financial services
Beath et al. (2012b)	Winning Hearts and Minds? Evidence from a Field Experiment in Afghanistan.	Afghanistan	Strong confidence	Build or improve local institutions	Better targeted public services
Beath et al. (2012a)	Direct Democracy and Resource Allocation. Experimental Evidence from Afghanistan	Afghanistan	Strong confidence	Build or improve local institutions	Better targeted public services	Improved household income
Beath et al. (2013a)	Do Elected Councils Improve Governance? Experimental Evidence on Local Institutions in Afghanistan.	Afghanistan	Strong confidence	Build or improve local institutions	Better targeted public services
				Cash or asset transfer
Beath et al. (2013b)	Empowering Women Through Development Aid: Evidence from a Field Experiment in Afghanistan	Afghanistan	Strong confidence	Build or improve local institutions	Enhanced local institutions and participation
				Empower local marginalized groups
BenYishay and Mobarak (2015)	Social Learning and Incentives for Experimentation and Communication	Malawi	Strong confidence	Enhance public service provision			More productive (agricultural) land use
				Agricultural extension and skills
Bernard et al. (2008)	Impact of Cooperatives in Smallholders Commercialization Behavior. Evidence from Ethiopia.	Ethiopia	Strong confidence	Establish or improve producer cooperatives		Improved household income
Bhamoriya and Gandhi (2010)	Adaptiveness in Water Management Institutions in India: Nature and Impact	India	Weak confidence, with multiple issues	Build or improve local institutions	Enhanced local institutions and participation
Brune et al. (2016b)	Savings Defaults and Payment Delays for Cash Transfers: Field Experimental Evidence from Malawi	Malawi	Strong confidence	Cash or asset transfer	Improved use of financial services
				Provide access to financial services
Brune et al. (2016a)	Facilitating Savings for Agriculture. Field Experimental Evidence from Malawi	Malawi	Strong confidence	Provide access to financial services	Improved use of financial services
Burke (2013)	Selling Low and Buying High: An Arbitrage Puzzle in Kenyan Villages	Kenya	Strong confidence	Provide access to financial services	Improved use of financial services
Carter et al. (2014)	Savings, Subsidies and the Persistence of Technology Adoption: Field Experimental Evidence from Mozambique	Mozambique	Strong confidence	Incentives for efficient resource use			More productive (agricultural) land use
Casey et al. (2012)	Reshaping Institutions: Evidence and Aid Impacts Using A Pre-Analysis Plan	Sierra Leone	Strong confidence	Build or improve local institutions	Enhanced local institutions and participation	Improved household income
				Empower local marginalized groups
Cavatassi et al. (2011)	How do Agricultural Programmes Alter Crop Production? Evidence from Ecuador.	Ecuador	Strong confidence	Agricultural extension and skills		Improved household income	More productive (agricultural) land use
Chattopadhyay and Duflo (2004)	Women as Policy Makers: Evidence from a Randomized Experiment in India	India	Strong confidence	Empower local marginalized groups	Better targeted public services
Cole and Fernando (2012)	The Value of Advice: Evidence from Mobile Phone-Based Agricultural Extension	India	Strong confidence	Enhance public service provision	Improved public service delivery		More productive (agricultural) land use
				Agricultural extension and skills
Cole et al. (2013)	How does risk management Influence Production Decisions? Evidence from a Field Experiment	India	Strong confidence	Provide access to financial services		Improved household income
Cole et al. (2014)	Dynamics of Demand for Index Insurance. Evidence from a Long-Run Field Experiment	India	Strong confidence	Provide access to financial services	Improved use of financial services
Cunguara and Moder (2011)	Is Agricultural Extension Helping The Poor? Evidence from Rural Mozambique.	Mozambique	Strong confidence	Agricultural extension and skills		Improved household income
Dillon (2011)	Do Differences in the Scale of Irrigation Projects Generate Different Impacts on Poverty and Production	Mali	Good confidence, with minor issues	Enhance public service provision		Improved household income	More productive (agricultural) land use
Dupas et al. (2015)	The Effect of Savings Accounts on Interpersonal Financial Relationships. Evidence from a Field Experiment in Rural Kenya	Kenya	Strong confidence	Provide access to financial services	Improved use of financial services
Fearon et al. (2011)	Democratic Institutions and Collective Action Capacity: Results from a Field Experiment in Post-conflict Liberia	Liberia	Strong confidence	Build or improve local institutions	Enhanced local institutions and participation	Improved household income
Fischer and Qaim (2012)	Linking Smallholders to Markets. Determinants and Impacts of Farmer Collective Action in Kenya	Kenya	Strong confidence	Establish or improve producer cooperatives	Enhanced local institutions and participation	Improved household income
				Agricultural extension and skills
Francesconi and Heerink (2011)	Ethiopian Agricultural Cooperatives in an Era of Global Commodity Exchange. Does Organizational form Matter?	Ethiopia	Weak confidence, with multiple issues	Establish or improve producer cooperatives		Improved household income
Giné and Yang (2009)	Insurance, Credit and Technology Adoption. Field Experiment Evidence from Malawi	Malawi	Strong confidence	Provide access to financial services	Improved use of financial services
Guo et al. (2015)	Farmer Field School and Farmer Knowledge Acquisition in rice Production. Experimental Evaluation in China	China	Strong confidence	Agricultural extension and skills		Improved household income	More productive (agricultural) land use
Hegde and Bull (2011)	Performance of Agro Forestry Based PES Project in Mozambique. A Household Level Analysis	Mozambique	Strong confidence	Incentives for efficient resource use	Enhanced local institutions and participation	Improved household income
Humphreys et al. (2015)	Social engineering in the Tropics: A Grassroots Democratization Experiment in the Congo	DR Congo	Strong confidence	Build or improve local institutions	Enhanced local institutions and participation
Cai et al. (2015)	Social Networks and the Decision to Insure	China	Strong confidence	Provide access to financial services	Improved use of financial services
Karlan et al. (2012)	Agricultural Decisions after Relaxing Credit and Risk Constraints	Ghana	Strong confidence	Provide access to financial services		Improved household income
Labonne and Chase (2011)	Do CDD projects enhance social capital? Evidence from the Philippines.	Philippines	Strong confidence	Build or improve local institutions	Enhanced local institutions and participation
Macours et al. (2012)	Transfers, Diversification and Household Risk Strategies: Experimental evidence with lessons for climate change adaptation	Nicaragua	Strong confidence	Cash or asset transfer		Improved household income
Madajewicz et al. (2014)	Community Participation in Decision – Making. Evidence from an Experiment in Providing Safe Drinking Water in Bangladesh	Bangladesh	Strong confidence	Build or improve local institutions Empower local marginalized groups	Better targeted public services
Matchaya and Perotin (2013)	The Impact of Cooperative Patronage. The Case of NASFAM in Kasungu District	Malawi	Good confidence, with minor issues	Establish or improve producer cooperatives	Enhanced local institutions and participation	Improved household income	More productive (agricultural) land use
				Agricultural extension and skills
Mazunda and Shively (2015)	Measuring the Forest and Income Impacts of Forest User Group Participation Under Malawi’s Forest Co-Management Program	Malawi	Strong confidence	Build or improve local institutions	Enhanced local institutions and participation		Reduced deforestation
Mojo et al. (2015)	Social and Environmental Impacts of Agricultural Cooperatives. Evidence from Ethiopia	Ethiopia	Good confidence, with minor issues	Establish or improve producer cooperatives			More productive (agricultural) land use
Nagaraj et al. (2013)	Technological and Institutional Interventions in Enhancing Livelihood of Farmers in Semi-Arid Tropics Areas. Experience of ICRISAT HOPE Project	India	Strong confidence	Agricultural extension and skills			More productive (agricultural) land use
Nguyen and Rieger (2014)	Community Driven Development and Social Capital. Evidence from Morocco	Morocco	Strong confidence	Build or improve local institutions	Enhanced local institutions and participation
Nkonya et al. (2012)	Initial Impact of Integrated Agricultural Research for Development in East and Central Africa	DR Congo	Strong confidence	Build or improve local institutions	Better targeted public services	Improved household income
		Rwanda
		Uganda
Olken (2007)	Monitoring Corruption. Evidence from a Field Experiment in Indonesia	Indonesia	Strong confidence	Build or improve local institutions	Better targeted public services
Olken (2010)	Direct Democracy and Local Public Goods. Evidence from a Field Experiment in Indonesia	Indonesia	Strong confidence	Build or improve local institutions	Improved public service delivery
Schaner (2015)	The Persistent Power of Behavioral Change. Long Run Impacts of Temporary Savings Subsidies for the Poor	Kenya	Strong confidence	Provide access to financial services		Improved household income
Somanathan et al. (2009)	Decentralization for Cost – Effective Conservation	India	Strong confidence	Enhance public service provision	Improved public service delivery
Takahashi and Barrett (2013)	The System of Rice Intensification and Its Impacts on Household Income and Child Schooling. Evidence from Rural Indonesia	Indonesia	Strong confidence	Agricultural extension and skills		Improved household income	More productive (agricultural) land use
Tilahun et al. (2016)	Impact of Membership in Frankincense Cooperative Firms on Rural Income and Poverty in Tigray, Northern Ethiopia	Ethiopia	Strong confidence	Establish or improve producer cooperatives	Enhanced local institutions and participation	Improved household income
Togbé et al. (2015)	Effect of Participatory Research on Farmer's Knowledge and Practice of IPM. The Case of Cotton in Benin	Benin	Good confidence, with minor issues	Agricultural extension and skills			More productive (agricultural) land use
Van Rijsbergen et al. (2016)	The Ambivalent Impact of Coffee Certification on Farmers Welfare. A Matched Panel Approach For Cooperatives in Central Kenya	Kenya	Strong confidence	Incentives for efficient resource use		Improved household income
Verhofstadt and Maertens (2014)	Can Agricultural Cooperatives Reduce Poverty? Heterogeneous Impact of Cooperative Membership on Farmers' Welfare in Rwanda	Rwanda	Good confidence, with minor issues	Establish or improve producer cooperatives		Reduced poverty and vulnerability Improved household income
Voss (2008)	Impact evaluation of the Second Phase of the KDP in Indonesia	Indonesia	Strong confidence	Build or improve local institutions		Reduced poverty and vulnerability Improved household income
Wright et al. (2016)	Decentralization and Deforestation: The Local Politics of Environmental Policy	Bolivia	Strong confidence	Enhance public service provision			Reduced deforestation
		Peru