Contribution analysis of a Bolivian innovation grant fund: mixing methods to verify relevance, efficiency and effectiveness

ABSTRACT

We used contribution analysis to verify the key assumption in the intervention logic of an innovation fund in Bolivia directed to economic farmer organisations to develop value-added activities. We focussed the research on three sub-components of the intervention logic: relevance of the farmer groups for local economic development, effectiveness of the fund in strengthening these group, and efficiency of the grant allocation mechanism. We used a case-based comparative analysis to assess effectiveness: improved market access for members, strengthened organisational capacities and the capacity to pay organisational costs. We showed that the grants to already well-endowed organisations were particularly unsuccessful.

KEYWORDS:

• Bolivia
• contribution analysis
• matching grants
• cooperatives
• impact evaluation

1. Introduction

Innovation grants help to reduce the risk faced by innovators when starting a new business process (Kessler 2013). Grant systems specifically targeted to economic farmer organisations are part of agricultural innovation policies in several countries (Berdegué 2001; Toro and Espinosa 2003). Evidence on the effectiveness of this type of grant systems on local economic development is still scarce. Several factors explain this relative neglect of impact evaluation (Ton et al. 2015). First, attributing outcomes to grants alone is challenging, as the business plans emerge from a complex process of decision-making in which the grant covers only (part of) the investments needed, while organisational capabilities or marketing strategies are shaped by other actors and factors. At most, it is possible to assess if the grant is a nonredundant contributory factor within this causal configuration (Ton, Vellema, and de Ruyter de Wildt 2011; Mackie 1965; Stern et al. 2012). Second, the economic benefits of grant-funded investments often materialise only after some years of starting-up and gradual market penetration. Generally, impact evaluations can measure only the changes in short-term outcomes, not on outcomes that only materialise long after the support period. Third, even when changes in key outcomes can be measured, the scale and activities of farmer groups vary a lot. This diversity results in a large variance in performance indicators, such as turnover, membership and patrimony. This implies the need for large samples to detect statistically significant effects (McKenzie 2012), whereas the limited number of relevant actors that exists in a country or region may function as a hard cap on sample size. Together, this constraints experimental or quasi-experimental designs to measure the impact of grants on economic farmer organisations. Alternative ways to make use of smaller datasets are needed to verify the assumption that these funds generate development impact (Ton 2015a).

Contribution analysis has been developed by John Mayne to design monitoring and evaluation systems for complex interventions where experimental and quasi-experimental designs are impossible (Mayne 2001, 2011, 2012). Despite substantial attention to contribution analysis in the field of evaluation methodology, few studies have applied it in practice (Dybdal, Bohni Nielsen, and Lemire 2010). We used this approach to assess the relevance, effectiveness and efficiency of an innovation fund in Bolivia, Fondo para el Desarrollo de Organizaciones Economómicas Campesinas (FONDOECAS), directed to economic farmer organisations that develop value-added activities and collectively market their products. Our research is, to our knowledge, the first academic paper in which contribution analysis is used to assess the effectiveness of a real-world agricultural support intervention.

The paper proceeds as follows. First, we discuss the intervention logic and theory of change of FONDOECAS and select three key assumptions on which we focus the research: relevance of the farmer groups for local economic development, effectiveness of the fund in strengthening these group, and efficiency of the grant allocation mechanism. Second, we present the research methods used to verify each of these causal links. The design includes time-series data collection, in-depth qualitative interviews used for process-tracing and a novel method, qualitative comparative analysis (Ragin 2008, 2009), to explore for (combinations of) starting conditions that predict success or failure of the grant. Third, we use the results to develop an evidence-based contribution story for the FONDOECAS grant fund, and finish with a reflection on the methodology used.

2. Contribution analysis

We used contribution analysis (Befani and Mayne 2014; Mayne 2001, 2011, 2012) as our approach to impact evaluation. Contribution analysis is a form of theory-driven evaluation intended to verify and build the intervention’s ‘contribution story’, a narrative of how an intervention contributes to development. Mayne’s principal motivation to develop contribution analysis was to find a systematic way to collect and use monitoring information to reflect on the relevance and effectiveness of policy interventions, even when it is impossible to attribute the societal outcomes unambiguously to these policy interventions (Mayne 2012). Contribution analysis checks multiple causal links along the intervention logic through a combination of research questions. The monitoring and evaluation system needs to ‘build a compelling case with evidence from which it is reasonable to conclude with confidence that the intervention has made a contribution and why’ (Mayne 2012). Contribution analysis consists in a six-step process to strengthen the evidence-base of the contribution story.

These steps (see Box 1) describe an iterative process of building and refining the intervention logic. It identifies the key assumptions of impact that need to be verified and bolstered, gathers evidence to verify these, and reflects on the results. It is an exercise in which deductive and inductive research paradigms meet, and three essentially different types of research are combined: theory generating, theory testing and theory modifying (Rohlfinger 2012). The mix of methods may include surveys for net-effect estimates on outcomes that are directly influenced by the intervention, while using monitoring data, expert panels or other information sources to reflect on the contribution to outcomes that are outside the span of direct influence (Ton 2012).

Box 1. Key steps in contribution analysis

Step 1. Set out the cause–effect issue to be addressed

Acknowledge the causal problem for the intervention in question Scope the problem: determine the specific causal question being addressed; determine the level of confidence needed in answering the question Explore the nature and extent of the contribution expected from the intervention Determine the other key factors that might influence the realisation of the results Assess the plausibility of the expected contribution given the intervention size and reach.

Step 2. Develop the postulated theory of change and risks to it, including other influencing factors

From intervention documents, interviews and relevant prior research, develop the postulated theory of change of the intervention, including identifying the assumptions and risks for the causal links in the theory of change Identify the roles other key influencing factors may play in the theory of change Determine how contested the postulated theory of change is to better understand the strength of evidence needed.

Step 3. Gather the existing evidence on the theory of change

Gather the evidence that exists from previous measurement, past evaluations and relevant research to assess the likelihood: (1) of the expected results, assumptions and risk being realised; (2) of each of the causal links in the results chain occurring; and (3) of the other influencing factors making a significant difference.

Step 4. Assemble and assess the contribution claim, and challenges to it

Set out the contribution ‘story’ on the likelihood that the intervention ‘worked’: the causal claim based on the analysis of logic and evidence so far. Assess the strengths and weaknesses in the postulated theory of change in light of the available evidence, and the relevance of the other influencing factors – which links seemreasonable and which look weak and need more evidence If needed, refine or update the theory of change.

Step 5. Gather new evidence from the implementation of the intervention

With a focus on the identified weaknesses, gather data on the theory of change results that occurred, the assumptions and risks associated with the causal links and the other identifiedinfluencing factors

Step 6. Revise and strengthen the contribution story

Build a more credible contribution claim based on the new data gathered Reassess its strengths and weaknesses, that is, the extent to which the results, assumptions/risks and other influencing factors occurred Conclude on the strength of the theory of change and the role played by other influencing factors, and hence on the contribution claim If the evidence is still weak, revisit Step 5.

Source: Befani and Mayne (2014, 20). Reproduced with permission of the authors.

2.1. Step 1: set out the cause–effect issue to be addressed

We started with defining the boundaries of the research. The grant fund wanted credible data on its performance in order to be scaled up or replicated as an effective institutional arrangement to empower economic farmer organisations in markets. The theory of change was implicit in proposal to funders (Condori 2005; Ton 2005). The originators elaborated on its additionality to other grant systems that were functioning or had functioned before. For example, Bolivia had piloted a grant fund called LIL/INDÍGENA (LIL stands for ‘Learning and Innovation Loans’) with World Bank funds between 2001 and 2005. An evaluation of the project showed that village organisations had been successful in generating interest and ideas for economic initiatives, but also that the groups that emerged from these processes often lacked the capacities needed to implement and manage the projects. Economic farmer organisations, formally constituted as associations, proved better suited to handle these business plans, especially organisations that had existed for some years (Llorenti et al. 2005). The evaluation of LIL/INDÍGENA also highlighted that organisations which submitted proposals to the fund experienced long administrative delays (usually several years), which resulted in several business opportunities no longer existing at the time the grants were finally approved. CIOEC and AOPEB’s knowledge on the trajectories of the applicants could prevent investments in organisations that only existed in name, which were reactivated only when soliciting external support but had no real economic life of their own. In earlier grant funds, the selection of ‘real’ organisations with ‘sound’ business plans had been a recurring challenge (Hartwich, Alexaki, and Baptista 2007; Ton 2007). FONDOECAS hoped to develop a grant allocation system that could be replicated within other agricultural development programmes (FONDOECAS 2010). We identified three interlinked key assumptions that guided our research design, and we labelled them in Figure 1 as ‘relevance’, ‘efficiency’ and ‘effectiveness’.

• Relevance – the first assumption related to the expected impact on local economic development of the supported collective marketing groups – the development impact in the intervention logic.

• Efficiency – the second assumption was that the lean system of grant allocation would be efficient in selecting feasible business proposals to viable organisations. The feasibility analysis by the technical committee of experts was seen as the key institutional arrangement to efficiently target the grants.

• Effectiveness – the third assumption was that the grants would result in investments that facilitated business plans that would have a positive impact on the group’s organisational capacities, market access of members, and the ability to pay for the costs of collective action.

Figure 1. FONDOECAS’ intervention logic

2.2. Step 2: develop the intervention logic and identify the key assumptions

We started by reflecting with the involved stakeholders (CIOEC, AOPEB, donors) on the impact pathway implicit in the rationale of the grant fund, in order to focus the evaluation questions. The intervention chart in Figure 1 shows the intervention logic. FONDOECAS impact pathway is relatively straightforward. The main activities of FONDOECAS are the channelling of funds to feasible business plans. The grants need to help farmer groups to seize business opportunities. The grants are targeted to organisations with specific quality attributes. In the case of FONDOECAS, these quality attributes are codified in the formal eligibility requirements: a legal status; a documented participatory consultation process having approved the business proposal; annual financial statements for the last two years; and being affiliated to one of the two national representative organisations CIOEC or AOPEB. The proposal needed to provide information to assess the feasibility of the business plan: raw materials and complementary processing equipment must be available, the group must show a credible marketing strategy, the break-even point needs to be reached in a time span within which the organisation is able to pay for the start-up costs, beneficiaries need to cofinance the investment as a guarantee of their commitment, and the organisational structure should be such that the business operations could be handled effectively. A technical committee assessed the feasibility of the submitted business plans by reviewing these supporting factors. Figure 1 depicts the grant-supported business plans as ‘immediate outcomes’, the technical and economic feasibility as ‘supporting factors’, the changed business practices as ‘intermediate outcomes’ and the changes in business performance as ‘ultimate outcomes’. By generating or expanding activities to process agricultural products, both income and organizational capabilities were expected to be enhanced. Together, these would make it possible to improve service delivery and contribute to local economic development.

2.3. Step 3: gather the existing evidence on the intervention logic

Before designing the data collection methods to verify the key assumptions in the intervention logic, we reviewed the existing evidence-base.

2.3.1. Relevance

The key assumption of relevance is that local economic development benefits from economic farmer organisations that have better service provisioning to their members. To understand the (contested) relevance of FONDOECAS, it is important to consider the rural political constellation in Bolivia at the time that the impact evaluation was designed. Between 2008 and 2011, a fierce political-ideological debate started on the role of economic farmer organisations versus traditional village organisations. Economic farmer organisations were regarded by some as ‘neoliberal’ and threatening the cohesion of (preferred) communal organisations. The Morales government had introduced several preferential policies, like municipal cofunding of local business initiatives, accessible only to the traditional village organisations (sindicatos, ayllus) and not to economic farmer organisations with a more selective membership (Ton, Gouet, and Gonzalez 2013; Antequera Guerra 2013).

The academic literature is largely supportive to the assumption that independent economic farmer organisations are important for local economic development and intrinsically different than the traditional village organisations in rural areas in Bolivia. For example, Bernard, De Janvry, and Sadoulet (2010) show that the influence of village organisations negatively affects the performance of economic groups. This is in line with Woolcock (1998) who argues that the homogeneity and ‘closure’ characterising ethnic communities may at some point stifle members’ personal and business development. Many development programmes in Bolivia emphasise that farmers organisations need to be independent in their economic activities from the traditional village organisations in order to survive competition (Swen and Both 1999; Healy 2001; Flores et al. 2007; Bebbington 1996; Llorenti et al. 2005; Toornstra 2000). Others emphasise that close coordination with these traditional village organisations is necessary to constrain processes of social differentiation in rural villages (Muñoz et al. 2005; Laguna, Cáceres, and Carimentrand 2006; Laguna 2011).

2.3.2. Efficiency

Evidence on the targeting efficiency of grant systems is scarce. Unlike in the case of microfinancing institutions (http://www.themix.org) and impact investors (http://iris.thegiin.org) comparable benchmarking information on grant systems is unavailable. At the start of the research some internal monitoring information was available. Since FONDOECAS started with pilot projects in 2006, the project coordinator had used an Excel sheet to register and report some basic metrics on grant allocation and contracting. This information showed that several organisations were still in the investment phase. In addition, the analysis showed that many organisations that had been disapproved by the technical committee had resubmitted a proposal in following rounds.

2.3.3. Effectiveness

In Bolivia, several scholars and practitioners have studied economic farmer groups that received donor support and describe and analyse the process and dynamics of change that this generated inside these organisations (Bebbington, Quisbert, and Trujillo 1996; Bebbington 1996, 1997, 1998; Swen and Both 1999; Toornstra 2000; Flores et al. 2007; Ton and Bijman 2006). Generally, these studies are favourable towards grant support to these organisations. An external evaluation of FONDOECAS pilot phase (Prudencio 2010) presented anecdotic evidence that the grant contributed to economic performance and organisational strengthening.

2.4. Step 4: assemble and assess the contribution claim

2.4.1. Relevance

The conviction of the initiators and funders of FONDOECAS that economic farmer organisations were better suited for empowering smallholder farmers in markets than the traditional village organisations was contested in an ideological debate without a credible evidence-base. CIOEC hoped that the impact evaluation could generate a piece of evidence to convince these opposing political forces that funding of economic farmer organisations was legitimate and effective to reach public goals.

2.4.2. Efficiency

In comparison to earlier grant funds, better information to assess FONDOECAS expected to have a much leaner grant-allocation mechanism in order to limit the time between proposal and implementation of the business plan, while assuring that the grants ended up in feasible business proposals. FONDOECAS had as it main eligibility requirements for beneficiaries that these existed as a legal entity and had a minimum of two years of collective marketing experience. CIOEC and AOPEB, the two national apex organisations that managed FONDOECAS, could be expected to have sufficient information that would prevent grant allocation to organisations with formal existence but without real activities and organisational life. This was expected to result in better information to assess the feasibility of the business proposals. It also implied the assumption that the external technical committee would be able to discern between feasible and unfeasible business plans based on this information. The combination of a practical protocol to describe the business opportunity and an external committee to assess this information was considered a key feature of the grant, which could attract major donors or be replicated in other (public) grant funds.

2.4.3. Effectiveness

The grants were targeted explicitly to new value-added activities complementary to the collective marketing of unprocessed raw material. Three intermediate outcome areas were expected to result from this grant support. First, the new business activities would imply new agreements on logistics, internal transactions and decision-making. Second, the written business plan and the investments made with the grant would increase the ‘bankability’ of the organisation. This objective was prominent in the 2010–2103 strategic plan for FONDOECAS (FONDOECAS 2010), which emphasised the importance of linking the farmer organisations to credit institutions and larger public grant programmes, such as the Banco de Desarrollo Productivo. Third, the grant could strengthen the capacity of the organisations to pay the organisational expenses. Collective action needs some financial resources and time investments. The organisations need some cash income to pay their organisational activities and, when relevant, staff to operate the new economic activity. The complementary economic activities were expected to generate organisational income even when, in the initial years, the scale of the processing activities would be small in relation to the members supply of raw material.

2.5. Step 5: gather new evidence to verify key assumptions

Based on the former analysis, we developed the intervention logic and methodological design of this research, reported in Ton (2012), to collect evidence to verify the three key assumptions.

2.5.1. Relevance

As a check whether economic farmer organisations were indeed considered by the rural population themselves as an important instrument for local economic development, we opted for a survey with Likert scale statements. Data was collected piggybacking on a household survey made by the consultancy firm AnálisisReal-Latinoamérica – AR-LAT. The survey could be conducted at low cost, as the results could be combined with data from a previous survey on 20 municipalities in 2008, in view of AR-LAT’s interest in elaborating an atlas on local economic development potential in Bolivia. AR-LAT had used a peculiar sampling strategy for these surveys (Aramayo 2008). It used purposively stratified sampling in three categories of respondents: households units, economic units and local government units. The economic units were selected in clusters of villages within the municipality, in areas with good agricultural production conditions, based on information provided by the local government. Economic farmer organisations are generally active in areas with higher agricultural potential. The survey resulted in a database of 1945 observations from 39 municipalities (Aramayo 2013). Without necessarily being representative of all producers in the municipalities, the data permits comparisons between households having a member in an economic farmer organisation and households that have not.

2.5.2. Efficiency

We focused the data collection in order to assess the efficiency of the technical committee to select business proposals that have a high change of being successful. During the period 2007–2013, a total of 10 different experts had taken part in the committee. Five out of the 10 experts were more consistently involved in these evaluations, of which three continuously so in the whole period 2007–2013. We could compare these valuations with another dataset, the results of the monitoring missions of the FONDOECAS M&E staff, who visited 61 organisations during 2011 and 2013 to check on progress of the business-plan implementation. Most of these organisations were visited one year after having had their grant approved. The selection of these organisations was not random. Some organisations were visited because of persistent delays in implementation. FONDOECAS’ monitoring officer reported on progress in four areas: organisation, production, marketing strategy and financial investments. Each area was monitored on four proxy-indicators of progress.

2.5.3. Effectiveness

To assess the effectiveness of FONDOECAS grants, we designed research to monitor changes on four outcome areas, the intermediate outcomes in the intervention logic, depicted in Figure 1: organisational strengthening, increased market access for members, improved capacity to pay organisational expenses and new access to financial service providers. In each of these outcome areas, we registered changes between 2010 and 2013. The proxy-indicators to measure these changes may well change irrespective of the grant support; organisations will change in time, irrespectively of receiving a grant. Therefore, it was necessary to have a design that can inform about the changes that would most likely have taken place without the FONDOECAS grant. Most impact evaluations use a quasi-experimental design to get information on this ‘counterfactual’. The evaluation literature propose various quasi-experimental designs (Khandker, Koolwal, and Samad 2009; Bamberger et al. 2004; Shadish, Cook, and Campbell 2002), with randomised control trial, regression discontinuity and matched difference-in-difference as the most likely candidates for use in our research. First, we explored the feasibility of random assignment of grants, a requirement in a randomised control trial. This proved impossible, as it was considered as seriously compromising the legitimacy of the fund and negatively affect CIOEC’s image as an inclusive national apex organisation. As a second option, we explored the possibility of a regression-discontinuity design, selecting a comparison group based on the evaluation scores of the organisations that submitted proposals but were not awarded a grant. However, a review of the administrative information on the 2006–2010 period showed that most of the rejected applications were due to administrative reasons, not the quality of the business proposal. They never reached the Technical Committee for the scoring of the feasibility of proposals. Moreover, the few organisations that did get through this administrative selection and were rejected by the FONDOECAS evaluation committee qualified in their second or third attempt, having resolved the issues related to their business plan. Thus, a regression discontinuity design using the threshold scores of the evaluation committee proved impossible.

A panel study, comparing a grant beneficiary with a comparison group in a difference-in-difference design, seemed a feasible option to generate information for counterfactual thinking. However, the variance in most performance indicators was very high (see Table 1), which makes that the registered differences between treatment and comparison groups would lack statistical significance even when we could measure them in all organisations that existed in Bolivia. Even when we would include all 200 CIOEC members in our sample, detectable effect-size in group sales would still lie far above expectations.

Table 1. Effects that could be detected with a large sample of 200 organisations based on variance at baseline

Indicator	Mean	Standard deviation	Detectable effect size*
Sample size			n = 200
Organisational capacities (TCC score)	22	9.0	3.58
Annual group sales	US$ 186,135	US$ 300,634	US$ 119,700
– Sales of unprocessed products	US$ 278,134	US$ 386,523	US$ 153,890
– Sales of processed products	US$ 36,642	US$ 60,381	US$ 24,040

* p < 0.05; statistical power = 0.8.

Therefore, we refrained from a quasi-experimental design and decided for a smaller sample in which we applied case-based qualitative comparative analysis (Ragin 2008; Befani and Mayne 2014; Rihoux and Lobe 2009; Byrne and Ragin 2009). In this case-based analysis, each case was analysed through an approach inspired by process tracing (Beach and Pedersen 2013; Collier 2011; Perri 6 2006; Befani and Stedman-Bryce 2016). We followed 30 beneficiary organisations between 2010 and 2014 in order to find traces of a process that would support the assumption that the grant indeed contributed to organisational strengthening, members’ access to markets or capacity to pay organisational expenses. To be considered as a success, the grant had to be a clear INUS-condition (Mackie 1965) – a nonredundant factor in a configuration of factors that caused these intended outcomes. We also opted to follow a group of 20 unsupported organisations, also members of CIOEC, to feed counterfactual thinking. However, we soon experienced problems of ‘contagion’. Several of these unsupported organisations had become grant solicitors in late 2010 and in early 2011, and three organisations became grant beneficiary during the research period. We also had many cases of ‘attrition’: seven of the unsupported organisations in the sample did not exist anymore, or had severe internal problems that made it impossible to collect information. Six of them proved to be nongovernmental organisations or social-oriented groups, not economic farmer organisations. Coupled with some logistic problems which prevented some organisations to be contacted, we ended up, end 2013, with 26 grant beneficiaries and only five organisations that, in 2013, had not (yet) received a grant. In this case-based analysis, we analysed the available information in order to find ‘real events’ that show that the grant-supported business plans had been implemented and generated effects related with the three outcome areas. Summaries of the case studies are provided as supplemental data. The main information sources were the two interview reports that had resulted from the semi-structured interview in 2011 and 2013. Each interview report provides a ‘thick description’ (Geertz 1973) of the status of the economic and social situation of the group, the progress in the implementation of the grant supported business plan, and the organisational capabilities of the organisation to contain ten agency dilemmas, challenges inherent to collective marketing (Ton 2010; Ton et al. 2014). Next to this qualitative information, we summarised the information on organisational in a one-page summary sheet, called ‘organisational radiography’, with two assessment questions. The first question captured the ‘presence’ of each of the agency dilemmas in the practice of the organisation, and the second the ‘effectiveness of the organisational solution’. We used this information to derive a quantitative measure (TCC). In the final evaluation, both information sources are used to trace the effects of the grant and assess its success. The differences in TCC scores of the same organisation are triangulated with the qualitative information on the real dynamics and change processes documented in the two in-depth qualitative interview reports, and the time-series data on sales, membership and group income spending.

Being aware of the subjective and normative element in the final assessment of success and failure of the grant to reach the intended outcomes by the researcher–evaluator, we used a peer-review process to improve the validity of the judgement. Two researchers, the author of this paper (GT) and the local researcher (LF), assessed each case independently. These two relatively independent assessments of the change processes in each of the organisations were compared and reconciled. Most differences were due to differences in access to information (GT had more detailed knowledge about the time-series data) or substantive knowledge about the cases (LF had conducted all the interviews and knew more about the context and process of the changes). Table 2 shows the Cohen’s Kappa scores, as an indicator of inter-rater agreement. According to the common interpretation of the kappa-score (Landis and Koch 1977) the inter-rater agreement was ‘moderate’ for the outcomes enhanced market access and improved organisational capacities, and ‘fair’ for the increased capacity to pay organisational expenses. The latter low kappa-score was due to initial different interpretations of ‘success’ on this outcome. During reconciliation, it was agreed that the capacity to pay expenses was evaluated as being increased when the level of sales had increased due to the grant, even though in most organisations the total amount of expenses or member income had not changed.

Table 2. Evaluation of the contributionary role of the grant in three outcome areas

	Positive outcome market access			Positive outcome organisational strengthened			Positive outcome capacity to pay organisational expenses
	LF original	GT original	Reconciled	LF original	GT original	Reconciled	LF original	GT original	GT + LF Reconciled
AAAT	0	1	0	1	0	1	0	1	1
ADAPICRUZ	1	1	1	1	0	1	0	1	1
AGAYAP	0	0	0	0	0	0	0	0	0
AMAGA	0	0	0	0	1	1	0	1	1
AMDESOY	0	0	0	0	0	0	0	1	1
AOCEMM	1	1	1	1	1	1	1	1	1
APAM MIZQUE	0	0	0	0	0	0	0	0	0
APCA	.	1	1	.	0	0	.	0	0
APROAMOL	0	1	0	1	1	1	0	1	1
APROQUIRC	1	1	1	1	1	1	0	1	1
APSU	0	0	0	0	0	0	0	0	0
ARAO	1	1	1	1	1	1	0	1	1
ASAFOP	0	0	0	1	0	1	0	1	1
ASOCOM	0	0	0	0	0	0	0	0	0
CECAOT	1	0	0	1	0	0	0	0	0
CELCCAR	0	0	0	0	1	0	0	0	0
CEMUR	0	0	0	0	0	0	0	1	1
CEPLACH	1	0	0	0	0	0	1	1	1
CIAPEC	0	0	0	0	1	0	0	0	0
COAINE	0	0	0	0	0	0	0	0	0
COMART	0	0	0	1	1	1	0	1	0
COPROQUINACC	0	0	0	0	0	0	0	0	0
INCA PALLAY	0	0	0	1	1	1	0	0	0
OMCSA	0	0	0	0	0	0	1	1	1
ORLIPA	0	1	0	0	0	0	0	0	0
SOPPROQUI	0	0	0	0	0	0	0	0	0
Success rate	6/25	8/26	5/26	10/25	9/25	10/26	3/25	13/26	12/26
Cohen’s kappa			0.48			0.41			0.22

2.6. Step 6: revise and strengthen the contribution story

2.6.1. Relevance

The survey yielded valuable information to assess the relevance of the support to economic farmer organisations (Table 3). Our direct question, ‘Do you consider the association/OECA an important instrument for economic development?’, was answered positively by two-thirds of the respondents. Surprisingly, there are no significant differences in opinion about this statement between respondents that had and those did not have membership in an economic farmer organisation. Sixty-five per cent of the households that are members of an association consider this organisation to be more supportive to them than the traditional village organisation, the rural syndicate and/or the ayllu. Most of these members also consider that economic farmer organisations are among the best-functioning farmer organisations in the area. On the question whether they would recommend their neighbours becoming a member, nonmembers are a bit more negative than the households with a member in an economic farmer organisation. However, the majority of them agreed with the statement that they would recommend others to become a member.

Table 3. Survey on the relevance of economic farmer organisations compared with traditional village organisations

	No-one member (N=1007)					Someone member (N=938)
	Likert scale^a					Likert scale^a
Statements	No opinion(%)	— (%)	–(%)	+(%)	++(%)	No opinion (%)	—(%)	–(%)	+(%)	++(%)
Relevance
The association/OECA is an important instrument for economic development in the area	22	10	30	47	14	17	7	27	55	12
The association/OECA is one of the farmer organisations in the area that functions best**	93	100	0	0	0	17	7	27	55	12
The association/OECA supports me more than the sindicato or ayllu	100					21	10	25	52	13
Not member, but someone in the household would be interested to become member of an association/OECA	27	0	30	70	0	79	0	100	0	0
I would recommend my neighbours to become member of the association/OECA**	42	18	25	55	2	31	14	22	55	10
Member characteristics
The association/OECA consists of a group of families that have different characteristics than the average in the area	25	7	57	26	10	25	8	57	24	11
The members of the association/OECA have more land or working capital than average in the area	32	8	55	26	11	28	9	56	22	12
The association/OECA have better houses than average*	30	9	64	19	8	30	7	71	18	4
The members of the association/OECA have more political influence than average	33	10	53	28	10	38	12	51	31	7

a. – = Strongly disagree; - = Disagree; + = Agree; ++ = Strongly agree

* Significant differences between groups, p<0.05; **Significant differences between groups, p < 0.01.

When we compare respondents that had an opinion on the commitment of both village authorities and leaders of farmer associations, we see that, overall, the opinion is slightly more positive about leadership in economic farmer organisations (p < 0.05, using a paired sample t-test). The survey data is supportive of the key assumption in the FONDOECAS intervention logic that farmer organisations are an important component of the social capital in rural areas and are relevant for local economic development. Interesting, Table 4 shows that only a minority considered the association as a means to gain access to markets. More prominent was the association’s function in improving production and accessing outside support. Moreover, they are considered valuable aspects of social life, with 55 per cent of the responding households that did not participate in an economic farmer organisation considering them positively in this respect. This shows that even though their identity as economic farmer organisations (CIOEC-Bolivia 2010a, 2010b) is framed around their pivotal role in creating access to markets, they are only considered instrumental in doing so by a minority of the households. Nevertheless, the willingness to engage in some sort of collective marketing is high, which supports the assumption that grant support to facilitate the emergence and development of economic farmer organisations is indeed a relevant activity.

Table 4. Opinion of farmers on the main functions of economic farmer organisations

	No-one member (N=1007)					Someone member (N=938)
	Likert scale^a					Likert scale^a
	No opinion (%)	— (%)	–(%)	+ (%)	++(%)	No opinion (%)	—(%)	– (%)	+(%)	++ (%)
Social performance
The leaders of the association/OECA have ample commitment to the village**	23	14	36	50	0	19	9	30	58	4
The leaders of the association/OECA have ample commitment to their members	100					17	0	33	52	14
The leaders of the sindicato/ayllu have ample commitment to the village	20	11	34	56	0	17	11	37	52	0
The leaders of the sindicato/ayllu have ample commitment to the members of the association**	74	0	100	0	0	20	0	42	48	10
The association/OECA has a good relation with the sindicato/ayllu	48	0	53	47	0	40	0	47	53	0
Functions
The association/OECA is a mean that helps me to get support (aid)	100					0^b	0	39	54	8
The association/OECA is a mean that helps to access markets	100					0^b	0	60	30	10
The association/OECA is a mean that helps me in the social life of the village	19	3	42	46	9	18	6	36	47	11
The association/OECA is a mean that helps me to improve production	100					19	0	26	64	11
I like to sell my products in an organised way together with others	28	7	27	59	7	27	7	25	61	8

a. – = Strongly disagree; - = Disagree; + = Agree; ++ = Strongly agree

b. We cannot discard a data entry error on this statement. ‘No opinion’ had been coded as ‘disagree’.

* Significant differences between groups, p<0.05.

**Significant differences between groups p<0.01.

2.6.2. Efficiency

Overall, the monitoring information in the inspection reports of the FONDOECAS M&E officer (Table 5) indicated that organisational progress in business plans lagged. The progress indicators for these organisations showed a similar pattern. The involvement of members as suppliers or operators in the businesses was far lower than expected, and only sporadically were specialised persons operating the business. In addition, progress in the area of marketing was less advanced than expected, especially on the issue of quality certification. Progress in financial investment and in production was relatively good, with 72 and 67 per cent, respectively (Table 5).

Table 5. Monitoring scores on business plan implementation

		All visited (N = 61)		Matched with evaluator scores (N = 50)		Matched with case studies (N = 13)
Area	Aspect	Average progress	Stand. dev.	Average progress	Standard deviation	Average progress	Standard deviation
Organisation	Increasing involvement of members	0.16	0.36	0.16	0.36	0.15	0.38
	Members know how to manage production	0.48	0.52	0.48	0.52	0.31	0.48
	There is professional staff available	0.36	0.48	0.4	0.48	0.54	0.52
	Business plan benefits all members	0.56	0.52	0.64	0.48	0.46	0.52
	Subtotal	39%	25%	42%	24%	37%	32%
Production	Primary production	0.80	0.40	0.84	0.36	0.77	0.44
	Procurement from members	0.64	0.48	0.68	0.48	0.85	0.38
	Processing of product	0.52	0.52	0.56	0.52	0.77	0.44
	Collective marketing of product	0.92	0.24	0.96	0.2	1.00	0.00
	Subtotal	72%	26%	76%	24%	85%	16%
Marketing	Finished product	0.68	0.48	0.76	0.44	0.85	0.38
	Ready market	0.52	0.52	0.48	0.52	0.69	0.48
	Quality certification of product	0.24	0.44	0.28	0.44	0.23	0.44
	Dedicated buyer for product	0.52	0.52	0.52	0.52	0.38	0.51
	Subtotal	48%	24%	50%	21%	54%	20%
Financial	Equipment in place	0.88	0.32	0.92	0.32	1.00	0.00
investment	Machinery in place	0.52	0.52	0.52	0.52	0.69	0.48
	Infrastructure in place	0.84	0.36	0.88	0.36	0.69	0.48
	Working capital available	0.44	0.52	0.48	0.52	0.46	0.52
	Subtotal	67%	29%	69%	27%	71%	22%

We could pair 50 of the 61 monitoring observations with the average feasibility score of the committee, 13 of these were part of our sample of comparative cases studies. The unpaired observations differed in name or referred to the same organisation having been visited twice. We used the 50 paired observations to test the correlation between feasibility scores of the technical committee evaluation and progress reported by FONDOECAS monitoring staff. We tested with a multivariate analysis of covariance (MANCOVA). The regression model can be written as

$Y_{ij}=\left[y_{i1},y_{i2},y_{i3},y_{i4}\right]=\alpha +\beta X_i+{\epsilon }_i,$

with

• y_i1 = progress in organisational commitment of organisation i,

• y_i2 = progress in production and logistics of organisation i,

• y_i3 = progress in marketing strategy of organisation i,

• y_i4 = progress in infrastructural investments of organisation i,

• X_i = feasibility score of organisation i,

• ε_i = error term in regression of organisation i,

The model is statistically significant (F[4, 50] = 3.11, p = 0.02, Pillai’s trace, p = 0.02, partial eta-squared = 0.22). The feasibility scores predict the progress of implementation of the grant-supported business plan fairly well (see Table 6). The partial eta-squared is considered as a large effect according to Cohen’s rules of thumb (Cohen 1988) in multivariate analysis. Testing the Pearson correlations between the feasibility score and the monitoring information, we see that the feasibility score predicts the progress in financial investments in the business plan (F[1, 50] = 7.54, p < 0.01), but is negatively related with progress in production and logistics (F[1, 50] = 4.16, p < 0.05). When we revised the data, we detected that the negative correlation was heavily influenced by the low scores of two dairy organisations, which had invested the grant in processing equipment but never managed to get their business operational. These are the only organisations in the dataset that had received zero points for both progress in production and in marketing. The negative correlation between high evaluation scores and progress in the area of production disappears when we exclude these two outliers (F[1, 48] = 0.03, p = 0.87). The monitored progress in financial investments realised with the grant remains the only area that has a statistically significant correlation with the committee’s feasibility scores (F[1, 48] = 5.01, p < 0.05).

Table 6. Multivariate analysis with average feasibility score as predictor of field monitoring results (N = 50)

Source	Area	Pearson correlation	F	Sig.	Partial eta-squared
Feasibility score	Organisation	0.049	0.115	0.736	0.002
	Production	−0.282	4.155	0.047	0.080
	Marketing	−0.011	0.006	0.939	0.000
	Financial investments	0.368	7.539	0.008	0.136

These result show that the evaluators’ score on feasibility of the proposed business plan indeed predicted progress in implementation (financial investment). However, there is no positive correlation with the more substantial outcomes in the areas organisation, production and marketing. This suggests that the committee proved not very efficient at differentiating between feasible and less feasible business plans.

2.6.3. Effectiveness

2.6.3.1. Did the grants improve organisational capacities?

To verify the assumption that grants are effective in improving organisational capabilities, we collected panel data in two rounds, 2011 and 2013. In each round, we conducted in-depth interviews on the dynamics within each organisation surrounding 10 agency dilemmas that are common in collective marketing arrangements (Ton 2010; Ton, Vellema, and de Ruyter de Wildt 2011; Ton 2015b; Ton et al. 2014). We summarised this qualitative information in a summary sheet, called an ‘organisational radiography’ (Ton 2015b).

We expected that as a result of the investment in value-added production, in addition to their current bulking activities, the number of relevant agency dilemmas would increase. To our surprise, we found a decrease. Whereas in 2011 the grant beneficiaries had an average of 7.0 agency dilemmas, in 2013 this was only 5.8. The average change was significantly different from zero (paired sample test p = 0.05). However, the interviewed board members indicated that most of these changes were considered to be unrelated to the grant. They changed as a result of a wide range of other factors and actors, such as market dynamics, context-specific political dynamics, or support from other agencies.

In Table 7, we distil only the organisations that indicated that the change in their capacity to handle an agency dilemma was indeed related to the grant-supported business plan. Eight different organisations mentioned a grant-related change in one or more agency dilemmas. Five of these six organisations reported that they had fewer problems with quality assurance (T₂). This is in line with FONDOECAS’ focus on supporting added value through processing, and paying attention to the niche markets of public-procurement programmes. Processing creates the need to improve the quality of the products supplied by members and develop rules and regulations that are effective in doing so.

Table 7. Self-assessed changes in organisational capacities to contain agency dilemmas in collective marketing related with the grants

Agency dilemma	Outcome	Agency dilemma is relevant	Change is unrelated to grant support	Change is related to grant support	Name of the organisation that reported a relation with the grant-supported business plan
T1 – ‘Regulating member supply’	More problems	4	4	0
	No change	4	4	0
	Fewer problems	3	1	2	AOCEMM, ARAO
T2 – ‘Quality assurance systems’	More problems	4	3	1	ADAPICRUZ
	No change	13	9	4	COMART, AAAT, ASAFOP, CEMUR
	Fewer problems	9	4	5	AOCEMM, ARAO, APROQUIRC, APROAMOL, CEPLACH
T3 – ‘Coping with working capital constraints’	More problems	3	2	1	AMAGA
	No change	18	16	2	ARAO, CEPLACH
	Fewer problems	4	3	1	ASPASA
T4 – ‘Anticipating side-selling’	More problems	6	6	0
	No change	10	9	1	ADAPICRUZ
	Fewer problems	0	0	0
T5 – ‘Ways to use profits’	More problems	3	2	1	AMAGA
	No change	15	15	0
	Fewer problems	2	2	0
T6 – ‘Differentiating services to members and nonmembers’	More problems	2	1	1	AMAGA
	No change	10	8	2	AOCEMM, ADAPICRUZ
	Fewer problems	1	1	0
T7 – ‘Decision-making on activities that benefit only a subgroup’	More problems	2	1	1	AMAGA
	No change	8	4	4	AMLECO, ASAFOP, CIAPEC, ORLIPA
	Fewer problems	2	1	1	AOCEMM
T8 – ‘Task delegation and supervision of professional staff’	More problems	4	4	0
	No change	14	14	0
	Fewer problems	5	5	0
T9 – ‘Liability in contracts and loans	More problems	1	1	0
	No change	10	8	2	ASPASA, ARAO
	Fewer problems	1	0	1	COMART
T10 – ‘Managing political aspirations’	More problems	8	8	0
	No change	11	11	0
	Fewer problems	2	1	1	COMART

Source: Based on 2013 interview data. Valid N = 28.

Other agency dilemmas that changed by the grant-supported activities were the issue of working capital constraints (T₃) by ASPASA, the issue of differentiating services between members and nonmembers (T₇) by AOCEMM, the issue of distribution of legal responsibilities in contracts and loans (T₉) and the issue of managing political aspirations (T₁₀) by COMART. Two organisations indicated that the grant had a relation with an area where they experienced more problems. ADAPICRUZ mentioned that there were more problems with quality assurance due to the implemented business plan. They were in the process of organic certification of their honey products and experienced that the requirements in this market forced them to be more demanding. In AMAGA, problems around various agency dilemmas had increased between 2011 and 2013, after their dairy processing plant started to operate.

Apart from the relevance, we assessed the strength of the organisational solution to each dilemma (Table 8). For each tension, we assessed if the solution resolved the tensions around the agency dilemma or that the organisation was still trying to find a solution. We used the information on both questions – relevance and strength – to compute a tension containment capacity (TCC) score. We describe the method in detail in another paper (Ton 2015b).

Table 8. Registered changes in organisational capacities between 2011 and 2013

Name of organisation		Relevant agency dilemmas (out of 10) 2011	Relevant core tensions 2011	TCC score 2011	Relevant agency dilemmas (out of 10) 2013	Relevant core tensions 2013	TCC score 2013	∆TCC score	∆TCC from 2011 level (%)	Significant change in TCC	Grant is related to agency dilemma	Reconciled verdict: grant contributed to organisational strengthening
GRANTEES	Mean	7.0	4.2	28	5.8	3.6	22	−6.6		Positive: 6		Yes: 10
	(SD)	(1.7)	(0.97)	(9.6)	(2.6)	(1.6)	(10,0)	(11.3)		Neutral: 6 Negative: 14		No: 16
AAAT		6	4	27	5	3	24	−3	−11	±	T2	Yes
ADAPICRUZ		7	5	42	9	4	22	−20	−48	–	T2, T4, T6	Yes
AGAYAP		8	4	27	0	0	0	−27	−100	–		No
AMAGA		5	3	18	7	4	20	2	11	±	T3, T5, T6, T7	Yes
AMDESOY		5	2	12	3	1	6	−6	−50	–		No
AOCEMM		10	5	36	9	5	31	−5	−14	±	T1, T2, T6, T7	Yes
APAM MIZQUE		5	3	17	9	5	31	14	82	+		No
APCA		8	5	39	6	5	26	−13	−33	–		No
APROAMOL		3	3	12	5	4	18	6	50	+	T2	Yes
APROQUIRC		6	4	22	4	3	21	−1	−5	±	T2	Yes
APSU		8	5	20	3	2	15	−5	−25	–		No
ARAO		6	2	12	7	3	18	6	50	+	T1, T2, T3, T9	Yes
ASAFOP		8	4	36	5	4	27	−9	−25	–	T2, T7	Yes
ASOCOM		6	4	28	3	2	12	−16	−57	–		No
CECAOT		8	5	36	8	5	33	−3	−8	±		No
CELCCAR		8	5	39	9	5	26	−13	−33	–		No
CEMUR		8	5	27	7	3	16	−11	−41	–	T2	No
CEPLACH		4	3	24	4	3	17	−7	−29	–	T2, T3	No
CIAPEC		8	5	33	5	4	20	−13	−39	–	T7	No
COAINE		6	4	30	7	5	39	9	30	+		No
COMART		8	4	27	9	5	31	4	15	+	T2, T9, T10	Yes
COPROQUINACC-T		6	4	19	8	5	27	8	42	+		No
INCA PALLAY		10	5	39	6	5	36	−3	−8	±		Yes
OMCSA		8	5	33	0	0	0	−33	−100	–		No
ORLIPA		8	5	32	6	3	15	−17	−53	–	T7	No
SOPPROQUI		9	5	45	8	5	30	−15	−33	–		No
NONGRANTEES	Mean	8.0	4.6	28	7.4	4.4	26	−2.4		Positive: 0
	(SD)	(1.2)	(0.54)	(5.6)	(2.1)	(0.89)	(5.2)	(6.8)		Neutral: 4 Negative: 1
AMLECO		8	4	21	5	3	19	−2	−10	±	T7	Loan 2012
APME		6	5	31	7	5	31	0	0	±		Grant 2012
ASOPROF		9	4	27	9	5	30	3	11	±		No grant
ASPASA		9	5	27	6	4	28	1	4	±	T3, T9	Loan 2012
FENCA		8	5	36	10	5	22	−14	−39	–		No grant

Table 8 shows that the average change (∆TCC) in grant beneficiaries is −21 per cent, with a change of 5.9 points less than the average TCC score of 27.7 in 2011 (p < 0.01). Most organisations appear weaker in 2013 than in 2011. Acknowledging the limits in accuracy of the TCC construct (Ton 2015b), we applied a margin of 15 per cent up or down to classify the change between 2011 and 2013 as ‘the same’. Six grant recipients increased their tension containment capacities, six stayed the same and 14 had a lower score.

2.6.3.2. Did the grants improve access to markets?

The average group sales presented a tendency of growth in most of the organisations studied. The differences in growth rate per organisation vary greatly (see Table 9). The strongest growth of sales occurred in the quinoa exporters (COPROQUINACC, ASPASA, SOPROQUI, CECAOT and APROQUIRC), especially in the 2011–2012 period, when quinoa prices more than doubled. We see that the strong growth in the coffee cooperatives between 2008 and 2010 period slowed down in the period 2011–2012. This change is due to the implosion of the international coffee price during that period. In addition, the growth rates of the honey processor ADAPICRUZ levelled out in 2011–2012. Growth rates were extremely high for AMDESOY, a small soy processing enterprise, more than doubling their sales each year. However, the absolute value of their sales remained still relatively low.

Table 9. Registered group sales in the period 2008–2012

		Annual group sales (US$)					Average yearly growth rate (%)
Sector	Name	2008	2009	2010	2011	2012	Benef.	Comp.
Agriculture	ASOPROF	–^a	–^a	–^a	1,000,000	1,700,000		70
	OMCSA	–^a	2500	2971	3800	1500	1
	FENCA	–^a	–^a	142,857	142,857	0		−50
Cattle	AGAYAP	0	0	0	17,423	14,286	−18
	APCA	–^a	–^a	45,714	51,429	41,429	−3
	ORLIPA	0	2286	3071	5857	10,214	66
Coffee	CELCCAR	285,714	285,714	372,143	443,571	501,429	16
	CIAPEC	400,000	600,000	802,857	1,302,857	1,331,428	37
	COAINE	608,000	714,143	1,409,571	1,388,571	1,116,857	23
Dairy	AMAGA	3214	3143	4600	3551	4371	11
	AMLECO	–^a	–^a	339,000	355,714	367,143		4
	CEPLACH	11,429	11,429	11,429	10,000	4286	−18
Handicrafts	AAAT	28,571	42,857	64,324	61,830	45,792	18
	APSU	–^a	–^a	18,571	18,571	8857	−26
	ARAO	60,532	71,420	83,322	88,923	97,793	13
	COMART	214,286	228,571	207,143	192,857	171,429	−5
	INCA PALLAY	69,202	82,262	72,748	89,034	97,309	10
Honey	ADAPICRUZ	36,286	114,857	145,429	197,714	205,714	71
	AOCEMM	47,143	45,181	31,991	45,994	55,015	8
	APAM MIZQUE	10,257	11,643	8786	17,357	20,857	27
	APME	–^a	30,649	34,987	37,653	45,050		14
	APROAMOL	0	0	2934	2649	2220	−13
Processed foods	AMDESOY	–^a	1102	3749	4412	10,286	130
	ASAFOP	47,143	51,429	0	28,571	38,571	−19
	CEMUR	171,429	114,286	142,857	204,224	217,448	10
Stone mining	ASOCOM	185,714	167,143	204,286	214,286	242,857	8
Quinoa	APROQUIRC	457,143	514,286	571,429	720,000	1,300,000	33
	ASPASA	0	0	100,000	200,000	500,000		125
	CECAOT	142,857	178,571	172,000	186,286	795,714	89
	COPROQUINACC	128,571	135,714	200,000	205,714	457,217	45
	SOPPROQUI	600,000	500,000	571,429	1,400,000	1,600,714	39
Average annual growth rate							21	33
Standard deviation							(36)	(67)

^aNo data available.

The average growth rate of total group sales seems a good proxy for commercial health of the organisations. However, it is a poor indicator for judging the effectiveness of the FONDOECAS grants, especially when the grant investments had been directed to new business activities which were complementary to existing, traditional economic activities. Therefore, in Table 10, we make a distinction between sales of processed and unprocessed products. We indicate the year and sales group in which the grant was invested and indicate if the interviewed board members considered the registered change in these group sales related to the grant investment.

Table 10. Group sales (turnover) for unprocessed and processed products during 2008–2012.^a

			Group sales unprocessed products in USD^b						Group sales processed products in USD^b						Reconciled outcome
Sector	Name	Grant contract year	2008	2009	2010	2011	2012	Grant related	2008	2009	2010	2011	2012	Grant related	Improved market access of members	Increased income to pay organisations’ costs
Agriculture	ASOPROF	No grant				1,000,000	1,700,000					0	0		No grant
	FENCA	No grant			142,857	142,857	0				0	0	0		No grant
	OMCSA	2009/2011		0	0	0	0			1000	1471	2300	0		No	Yes
Cattle	AGAYAP	2008	0	0	0	17,423	14,286		0	0	0	0	0		No	No
	APCA	2009^c			0	0	0				40,000	42,857	31,429	Much	Yes	No
	ORLIPA	2009^c	0	1429	1571	4286	4286	Much	0	857	1286	1429	5714	Much	No	No
Coffee^b	CELCCAR	2009	285,714	285,714	371,429	442,857	500,000	Much	0	0	714	714	1429	Much	No	No
	CIAPEC	2009^c	400,000	600,000	800,000	1,300,000	1,328,571	No	0	0	2856	2856	2856	Much	No	No
	COAINE	2009	590,000	695,000	1,390,000	1,371,429	1,097,143	No	0	2000	2429	0	1143	Much	No	No
Dairy	AMAGA	2008	0	0	0	0	0		3214	3143	4600	3551	4371	Much	No	Yes
	AMLECO	No grant^c			0	0	0				0	5714	10,000	Much	No grant
	CEPLACH	2008	0	0	0	0	0		11,429	11,429	11,429	10,000	4,286	Little	No	Yes
Handicrafts	AAAT	2007/2009	0	0	0	0	0		28,571	42,857	64,324	61,830	45,792	Much	No	Yes
	APSU	2009			0	0	0				18,571	18,571	8857	No	No	No
	ARAO	2011/2012	0	0	0	0	0		60,532	71,420	83,322	88,923	97,793	Much	Yes	Yes
	COMART	2009/2011	0	0	0	0	0		214,286	228,571	207,143	192,857	171,429	Much	No	No
	INCAPALLAY	2007	0	0	0	0	0		69,202	82,262	72,748	89,034	97,309	Little	No	No
Honey	ADAPICRUZ	2009	22,857	100,000	125,714	168,571	171,429	No	12,857	14,286	17,143	25,714	30,857	Much	Yes	Yes
	AOCEMM	2008^c		44,381	31,673	44,880	54,915	Much		0	0	0	0		Yes	Yes
	APAM IZQUE	2010	6286	8000	4286	5143	3429	No	3714	3429	4286	12,000	17,143	No	No	No
	APME	(2012)		30,004	34,380	37,275	44,286	No		0	0	0	0		No grant
	APROAMOL	2010	0	0	2857	2571	2143	Much	0	0	0	0	0		No	Yes
Proc. foods	AMDESOY	2009/2011		0	0	0	0			1102	3749	4412	10,286	Much^d	No	Yes
	ASAFOP	2009	0	0	0	714	0	No	47,143	51,429	0	27,857	38,571	Much	No	Yes
	CEMUR	2009^c	0	0	0	0	0		171,429	114,286	142,857	191,175	196,983	Much	No	Yes
Stones	ASOCOM^d	2008	0	0	0	0	0		185,714	167,143	204,286	214,286	242,857	Much	No	No
Quinoa^b	APROQUIRC	2007	457,143	514,286	571,429	714,286	1,300,000	Much^e	0	0	0	5714	0	No	Yes	Yes
	ASPASA	No grant^c	0	0	100,000	200,000	500,000	Much^e	0	0	0	0	0		No grant
	CECAOT	2009/2012	142,857	178,571	171,429	180,000	757,143	No	0	0	571	571	28,571	Little	No	No
	COPROQUINACC	2007	128,571	135,714	200,000	205,714	457,217	No	0	0	0	0	0		No	No
	SOPPROQUI	2008	600,000	500,000	571,429	1,400,000	1,600,000	No	0	0	0	0	286	No	No	No

^a Empty cells indicate missing information; zero indicates no sales in the category; shaded boxes indicate the year when the first grant was received.

^bCoffee beans and quinoa prepared for bulk export are considered as unprocessed. Raw honey is also considered as being unprocessed, except when innovatively packaged.

^c These organisations received a FONDOECAS loan in 2012.

^dThe equipment (compressor) was never used, being too heavy to carry. The answer is inconsistent.

^e The self-assessed relationship refers to FONDOECAS credit facility not the grant.

Table 11. Implementation and outcomes of grant supported business plans

	Number of organisations	Success rate (%)
Beneficiaries	29	100
Grant implementers in 2012	23	79
Business plans in operation in 2012	19	66
Grants contributed to organisational capacities	10	35
Grants contributed to market access of members	5	17
Grants increased capacity to pay organisational expenses	12	41

As described earlier, for each case we ‘traced’ the process of change and evaluated if the grant could be considered a contributory factor to improved market access of members. Market performance is influenced by more factors than FONDOECAS. Therefore, there might be some discrepancy between the trend in sales of an organisation and our evaluation of the successfulness of the grant in improving market access for members. The columns at the right in Table 10 present the reconciled verdict of the researchers based on data and qualitative information about the change process in each organisation (see online supplementary material for further details). In six cases, the ‘final verdict’ differed markedly from the tendency of growth and the self-assessed contribution of the grant to this growth. On these organisations, the two researchers coincided that the grant could not be considered as a contributory factor. For example, the three coffee organisations registered an increase in sales of processed products (roasted coffee). However, the amount of coffee used in this processing was negligible when compared with the amount of coffee sourced from their members for export. And, in two of them (COAINE and CIAPEC) the equipment bought with the FONDOECAS grant was in fact never used in processing for external clients. Furthermore, three organisations active in food processing bought their inputs from nonmembers in the local market. They gained market access as micro-enterprises, but did not create market access for their members. Only for five grant recipients did the grant indeed improve market access for members (APCA, ARAO, ADAPICRUZ, AOCEMM and APROQUIRC).

Our explorative analysis to find predictors of success of failure of the grant using fuzzy-set qualitative comparative analysis (Ton 2015c) showed that successful outcomes – organisations in which the grant was considered a contributory factor to increased market access of members – were found only among organisations that sourced their raw material from members. The analysis showed also that, small organisations proved more likely to be unsuccessful. In addition, the old, large and strong organisations that already had high organisational capacities when awarded the grant proved particularly unsuccessful. The interview reports pointed to a causal mechanism that could explain this pattern in the data. In many of these organisations, the grant had been used for complementary processing, a secondary economic activity alongside their primary activities in bulking and exports (for example, quinoa, coffee). All these organisations indicated the limited amount of the grant (USD 10,000) as a reason for the failure of the business plan. The grant had resulted in underscaled investments; the supported business activities could not get the production volume needed for a commercially viable market launch.

2.6.3.3. Did the grants increase the capacity to pay organisational expenses?

The evaluations were more positive considering the grant’s contribution to the capacity to pay organisational expenses, like office supplies, expenses related with communication and travel, or personnel. Annual organisational expenses are modest, with an average of US$ 2411 (S.E. US$ 448) in 2012. ARAO, a handicraft organisation, registered the highest amount with US$ 8962, because they need to rent shop space. On average, payments to hired staff were 46 per cent (S.E. 6%) of total organisational costs. Small nonsourcing microenterprises that sell processed food (AMDESOY, ASAFOP, CEPLACH) had lower costs because members do most of the work.

Table 10 shows that for 12 organisations the grant contributed to an increased ability to pay organisational costs, representing 41 per cent of the grant recipients. Almost two-thirds of the implemented business plans were successful in generating group income. The explorative analysis of predictors of success and failure, using qualitative comparative analysis (Ton 2015c), shows that especially the organisations that sourced their raw material form members were unsuccessful in generating group income. Nonsourcing organisations have the advantage that they can use all benefits to pay organisational costs and salaries and do not have to negotiate the collective interests of the group with the short term interest of the members in a higher price of the raw material that is sourced from them. When the increased capacity to pay organisational expenses would be the main objective of the fund, targeting the grants to nonsourcing organisations would likely improve the effectiveness of the fund. In addition, the grant had proven particularly unsuccessful for the larger and stronger organisations that already had high organisational capacities and a large scale of operations at the moment of receiving the FONDOECAS grant. Income generated with the new activities only represented a small percentage of total income available to pay organisational expenses.

3. Discussion

3.1. FONDOECAS’ contribution story

FONDOECAS emerged as a pilot-project to address the specific problems of economic farmer organisations in collective marketing, in view of scaling up and replication. The fund provided grants (around US$ 10,000) to business plans that were evaluated on feasibility by a committee of external experts. The donor community supported FONDOECAS because they consider stronger economic farmer organisations as relevant actors for local economic development. This relevance legitimates the use of public money to support them. To make these groups stronger, the grant fund needs to have a mechanism to select the best business proposals. And, these grants need to be effective in strengthening the groups’ organisational and economic performance. We used contribution analysis to reflect on these assumptions.

The research shows that the implementation rate of grant-supported business plans in the sample is 66 per cent (see Table 11). One third of the grants were not properly invested in necessary technology or infrastructure. FONDOECAS improved organisational capacities in 10 of the 29 organisations. Half of the implemented business plans resulted in an improved capacity to manage collective marketing, foremost being the capacity to resolve the issue of quality assurance. Although we cannot compare this with the outcomes of other grant funds, and a certain percentage of failure is inherent to innovative business plans, FONDOECAS has certainly been less effective than was initially expected by its initiators. The results on the capacity to generate income to pay organisational expenses were relatively good. Almost two-thirds of the implemented business plans were successful in generating group income, representing 41 per cent of the grant recipients. This is even more evident for the objective to increase market access for member products. Only 17 per cent of the grants were successful in this respect. Nevertheless, most economic farmer organisations registered a high growth of group sales between 2008 and 2013. Overall, they have become commercially stronger, especially due to higher agricultural prices and preferential procurement by public nutrition programmes. However, the contribution of FONDOECAS to this growth is modest.

Support to economic farmer organisations is, nevertheless, relevant, even when innovation grants are clearly not the panacea. The household survey showed that there is majority support for economic farmer organisations among the rural population. Households that were participating in economic farmer organisations considered them even more supportive than the village organisations. Interestingly, their role in creating market access is reported as being the least evident benefit. There was more agreement on the statement that they are a means to support production, access outside support, and as a component of social life. There are, however, prospects for growth, considering the willingness of two-thirds of the farmers to market their products collectively.

The research to verify the assumption of efficiency in selecting the more feasible business proposals suggests that the technical committee has not been very efficient. Although we showed a positive correlation between feasibility scores and progress in implementation of investments, there seems no predictive power for progress in the organisation, production and marketing related aspects of the business plan. FONDOECAS may have to redesign the grant allocation mechanism in order to improve efficiency. Some steps have been taken to do so. In 2012, FONDOECAS reduced the number of member in the technical committee from five to three but retained the most critical external reviewers. In 2013, FONDOECAS decided to make grant allocation conditional on the information in field visit report by the M&E officer to organisations with a provisionally approved grant proposal. In addition, when based on the available information one external reviewer scored the feasibility as insufficient (lower than 4.5 on a scale of 10), the average score was not decisive and he or she had vetoing power. In 2015, CIOEC and AOPEB decided to intensify the use of their decentralised structures and technical support team to backstop the implementation of the grant proposal in order to increase the efficiency and effectiveness of the grants. It is, however, too early to evaluate if these changes in grant governance resulted improvements.

The research shows that grant funds that want to empower smallholder farmers in markets through collective marketing activities should target their grants, and distinguish between farmer organisations that source and those that do not source their raw material from members. Both types respond differently to investment support in value adding activities. Market access for members is only possible with sourcing organisations, and larger organisations are more likely to be successful in using the grant to do so. Instead, organisations that source their raw material from spot markets are more likely to be successful in using the grant to raise group income to pay for organisational expenses. The research shows that the grants proved particularly unsuccessful for the larger and stronger organisations that already had high organisational capacities and a large scale of operations at the moment of receiving the FONDOECAS grant. Loans not grants may be more appropriate and effective instruments for strengthening these larger organisations.

3.2. Contribution analysis as framework for impact evaluation

The approach to impact evaluation used in this research, contribution analysis, has wider relevance. The shift in development cooperation ‘from aid to trade’, with an increasing number of grant funds directed to companies, comes with an increasing need to verify relevance, effectiveness and efficiency of grants for business development (Kessler 2013). There is a need to reflect on the effectiveness of support interventions that, like FONDOECAS, want to trigger complex change processes and work in a restricted population of firms or organisations. Quasi-experimental designs are often not possible or too vulnerable for unanticipated changes in context or implementation. However, counterfactual thinking can be supported with other information than data from a comparison group.

Counterfactual thinking implies a structured and transparent way of discarding alternative explanations for an effect (Vellema et al. 2013; Yin 2013). This requires reflection on the question, what would have happened without the intervention? In our research, this implied a case-by-case evaluation of the contributory role of the grant in generating certain outcomes, based on a close reading of the dynamics within each organisation, looking for ‘traces’ of grant-induced change and alternative explanations. To do so, we collected time-series data and in-depth qualitative information, and evaluated a grant as successful, when the changes in indicators were supported by qualitative descriptions of the change processes induced by the grant. This process tracing resulted in a dataset on success or failure of the grant for each of the organisations in the sample. Qualitative comparative analysis proved useful as exploratory research method to explore for plausible predictors of success, and resulted in actionable working hypothesis that were used to guide the redesign of the intervention in order to improve the success rate.

We showed that through a creative mix of methods and data-analytical approaches, using fund specific monitoring data, comparative cases studies on beneficiaries and surveys, it is possible to critically review the key assumptions in the rationale behind developing interventions. Contribution analysis, the verification and bolstering of the contribution story, helps implementing agencies to reflect on expectations and learn to refine the intervention and targeting mechanism.

Acknowledgements

I thank Lithzy Flores who made a tremendous effort by travelling to the remote places to conduct most of the interviews. In addition, I would like to thank Freddy Ticona, Oscar Chambi and Richard Arguedas, who helped me to organise the research and provided monitoring information for my analysis. I owe special recognition to Professor Erwin Bulte at Wageningen University for the validity checks on my analysis.

Disclosure statement

The author discloses that he has been working between 2000 and 2004 for CIOEC-Bolivia. In 2005, as a consultant for ICCO, he helped CIOEC and AOPEB to design the FONDOECAS grant system

Supplemental data

Supplemental data for this article can be accessed here.

Additional information

Funding

This work was supported by The Netherlands’ Ministry of Economic Affairs [KB-11-004-003 and BO-0-010-129]. The fieldwork was cofunded by the Dutch Interchurch Development Organizations, ICCO.

Notes on contributors

Giel Ton

Giel Ton works as senior researcher in Wageningen Economic Research. He holds a PhD from Wageningen University. Since 2006, he has been working as a senior researcher on projects relating to business training, contract farming, export promotion and certification schemes. Before, he worked in Nicaragua (1989–1996) on participative agricultural innovation and in Bolivia (1999–2004) as policy advisor to empower farmer organisations in markets.