Evaluating the impact of adaptation interventions on vulnerability and livelihood resilience

ABSTRACT

Robust evaluation of adaptation interventions is necessary to monitor adaptation projects and ensure broader accountability in adaptation responses. Yet, to date very few frameworks are formulated with a robust impact assessment in mind. This study uses the Before-After-Control-Impact (BACI) research design to develop the BACI Adaptation Impact Evaluation Framework. The framework compares participating and non-participating households across time and was applied to panel data of 291 households, combined with ethnographic data, to evaluate the impacts of one of the Global Climate Change Alliance's (GCCA+) flagship adaptation and resilience projects in Tanzania. The results illustrate various benefits of the project, including strengthening social networks, providing education and diversifying information sources among the participating households. However, evidence of unintended consequences and maladaptation also exist, particularly among poorer non-participating households. We conclude that equitable adaptation requires longer projects that better target poorer households, engage with a broader array of climatic events, support the transfer of knowledge into action, and are more responsive to emergent trade-offs, ensuring that unintended impacts are minimized. The paper demonstrates how the application of the BACI Adaptation Impact Evaluation Framework provides a robust tool to assess the impacts of adaptation interventions.

KEYWORDS:

• Climate change adaptation
• maladaptation
• BACI research design
• evaluation framework
• equitable adaptation

1. Introduction

Development intervention, framed around climate change adaptation and resilience, increasingly tries to tackle the lack of global action in addressing climate change impacts in the global South (Bahadur et al., 2010; Frankenberger et al., 2014). Such intervention is implemented through projects, by development or other organizations (e.g. humanitarian, environmental conservation), and incorporates policies and activities that aim to reduce vulnerability and build resilience to different shocks and stresses, including climate change (see Christian Aid, 2012; Concern Worldwide, 2019). Resilience is viewed by those development actors as a helpful conceptual tool to understand how people can manage the effects of shocks and stresses that affect their livelihood outcomes (Bahadur et al., 2013).

Designing and evaluating adaptation and resilience building projects aspiring to change vulnerability and livelihood resilience in a positive direction is however complex, and more evidence from intervention and learning is required. Resilience has been recognized as being specific to scale and shock and highly dynamic, and thus difficult to measure (Constas, Frankenberger, Hoddinott, Mock, et al., 2014; Cumming et al., 2005). At the same time vulnerability assessments require knowledge on climatic and socio-economic pressures that affect people’s lives, their sensitivity and adaptive capacity (IPCC, 2014). Globally, efforts have been made to assess various forms of vulnerability and adaptation to climate change impacts, especially in developing country contexts (see Omerkhil et al., 2020; Pandey et al., 2016). Similarly, many conceptual frameworks concerned with operationalizing vulnerability and resilience, and more broadly, climate change impacts, have been put forward by academics, NGOs and major development agencies (see Oxfam, 2016; Sekhri et al., 2020; UNDP, 2014). Yet, their practical utility to elicit the impact of projects on reducing vulnerability and building livelihood resilience, in a given context, has rarely been tested (cf. Béné et al., 2017). Development practitioners usually lack the resources to collect high intensity data to ensure robust impact assessments, or if data is available it often remains unexamined due to the lack of capacities, such as technical expertize (GIZ, 2015). Consequently, development actors and governments fall short in understanding their programmatic contributions to reducing vulnerability and building livelihood resilience of the vulnerable people they deem to support. This is necessary not only to monitor implemented interventions but also to ensure their broader accountability in national and global adaptation responses.

The present study develops the BACI Adaptation Impact Evaluation Framework, which allows a robust project impact evaluation by adopting the Before-After-Control-Impact (BACI) research design. This approach compares changes in indicators of vulnerability and resilience before and after an intervention and for households who participated in the intervention and those who did not. By using the non-participant group as a control, it is possible to attribute factors of change directly to the intervention, thereby accurately evaluating the impacts of an intervention. The Before-After-Control-Impact (BACI) approach, also called difference-in-differences (DiD), is more robust than the Before-After (BA) approach alone (Bos et al., 2017). With this in mind, this study sets to fulfil two specific research objectives:

1. Describe the methodology that allows operationalizing the BACI Adaptation Impact Evaluation Framework.

2. Operationalize the framework to evaluate one of the European Union’s Global Climate Change Alliance (GCCA+) flagship adaptation and resilience projects in Tanzania (2015–2019).

Most specifically, this study focuses on the assessment of the GCCA+ project impacts on vulnerability and on enhancing households’ livelihood resilience through building buffer capacity, self-organization and capacity for learning. The paper provides two major contributions to the adaptation literature. First, the operationalization of the BACI design in a study on adaptation and resilience, which combines survey evidence with ethnography, facilitates the understanding of intended and unintended outcomes. Second, it offers new insights for designing effective development interventions aimed at reducing vulnerability and building livelihood resilience. Next, the paper presents the theoretical background, followed by the project impact evaluation method. The results illustrate the findings from the BACI analysis, showing how the intervention impacted the mitigation of vulnerability and the enhancement of livelihood resilience. Finally, the discussion compares the evidence to other studies on adaptation and resilience-building interventions and concludes with recommendations for development practice.

2. Theoretical background

This section introduces the related concepts of vulnerability and resilience and explores how they have been operationalized for adaptation and development. Specifically, we provide a brief overview of where the debate on assessing vulnerability and resilience currently stands in the development discourse and position our study in this literature. In the second part of this section, we present the theoretical foundation for our framework, before focusing on its analytical power for evaluating the impact of adaptation and resilience interventions.

2.1. Theoretical foundations and assessment of vulnerability and resilience for adaptation and development

The relationship between vulnerability, adaptation and resilience is historically complex (Cutter et al., 2008; Gallopín, 2006; Vogel et al., 2007). Adger (2006) refers to vulnerability as the ‘state of susceptibility to harm from exposure to stresses associated with environmental and social change and from the absence of capacity to adapt’ (p. 268). Adaptation is often regarded as a process (cf. Adger et al., 2005) that results in an adjustment in response to actual or expected climate stimuli and its effects (IPCC, 2014). Many academics define resilience as the capacity of a system to absorb disturbance without losing its basic structure and function and the capacity for self-organization and learning (Folke, 2006; Nelson et al., 2007; Speranza et al., 2014). Some authors suggested that vulnerability and resilience are antonyms (Folke et al., 2002), others saw their relationships whereby one is a subset of the other (Cutter et al., 2008). Nelson et al. (2007) consider adaptation as an actor-centred process of negotiation and decisions, and resilience as being concerned with the implications of these processes on the whole system. Studies also attempted to quantify vulnerability assessments, in the context of adapting to climate change, to produce policy-relevant recommendations (Gerlitz et al., 2017; Hahn et al., 2009). They often turned to the vulnerability definition that came from the Intergovernmental Panel on Climate Change (IPCC), which considers vulnerability to be a function of the system exposure, its sensitivity and adaptive capacity (McCarthy et al., 2001) and are based on climate vulnerability indices constructed on weighted factors grouped into key components.

More recently, however, resilience has been used as a synonym for adaptation and vulnerability reduction, and so, to mean taking action to reduce risk (Schipper & Langston, 2015). This is especially prominent in the development discourse where there is widespread adoption of resilience among NGO and donor agency programmatic pillars. For instance, in 2016, Oxfam set out its vision on resilient and sustainable development, that is ‘development that does not cause or increase risks, stresses and volatility for people living in poverty, and which makes progress towards a just world despite shocks, stresses and uncertainty’ (Oxfam, 2016, p. 6). Despite various critics (Cannon & Müller-Mahn, 2010; Manyena, 2006), resilience gradually became an overarching concept and a long-term goal for the process of adaptation.

Similarly to vulnerability, assessing resilience proved complex for academics and development practitioners. Concerns relate to the lack of clear and coherent definitions, multiple scales and multilevel interactions, specificity to shocks and stresses, identification of common yet context-appropriate indicators, to name a few (Constas, Frankenberger, and Hoddinott, 2014). While substantive progress has been made to formulate and actively pursue the agenda to inform resilience assessment, for instance by the Resilience Measurement Technical Working Group (Constas, Frankenberger, Hoddinott, Mock, et al., 2014), complexities persist especially when it comes to its practical operationalization (Constas et al., 2016; Williams, 2016). Recent research increasingly points out the need to develop robust monitoring, evaluating and learning frameworks for assessing and documenting the impact of resilience interventions (Béné et al., 2017). In the present development landscape, where many development actors use the principles of resilience in their programmes, evaluating resilience has become a key component of assessing projects’ success. The ability to evaluate projects for resilience through robust and consistent mechanisms, is needed in order to enhance the accountability of donor funding flows for projects, and as a way of assessing progress towards achieving resilience, as defined by various global policy frameworks, for instance the UN Sustainable Development Goals or the Sendai Framework for Disaster Risk Reduction 2015–2030 (Schipper & Langston, 2015). It is also an opportunity for initiating institutional learning about what works and what doesn’t work, and ultimately to implement more effective and equitable programmes of intervention. Still, very few frameworks for evaluating adaptation and resilience interventions are formulated with an operational, robust impact assessment in mind (see GIZ, 2015).

2.2. Theoretical foundations and operationalization of the BACI Adaptation Impact Evaluation Framework

For our BACI Adaptation Impact Evaluation Framework we draw from the theoretical heritage of vulnerability, adaptation and resilience research (Adger, 2006; Folke et al., 2002) and focus on livelihood resilience, particularly the conceptual framework of livelihood resilience developed by Speranza et al. (2014). Speranza et al. (2014) created their livelihood resilience framework to conceptualize how people’s livelihood practices contribute to maintaining individuals’ capacity to respond to adverse conditions and to influence societal structures and processes that support, or constrain, maintaining livelihoods, especially during shocks and stresses. Consequently, livelihood resilience is defined as ‘the capacity of livelihoods to cushion stresses and disturbances while maintaining or improving essential properties and functions’ and is determined ‘by actors’ assets and strategies to maintain and increase assets, to self-organize and to learn’ (Speranza et al., 2014, p. 111). We adapt this framework from Speranza et al. (2014) and create the BACI Adaptation Impact Evaluation Framework (Figure 1) that conceptualizes how to empirically assess the contribution of adaptation interventions to households’ capacity (buffer capacity) and the societal processes (self-organization and capacity for learning) that support positive livelihood resilience outcomes.

Figure 1. The BACI Adaptation Impact Evaluation Framework for evaluating the impact of adaptation interventions on reducing vulnerability and building livelihood resilience (livelihood resilience approach was adapted from Speranza et al., 2014).

Buffer capacity refers to ‘the capacity to cushion change and to use emerging opportunities to achieve better livelihoods outcomes such as reduced poverty’ (Speranza et al., 2014, p. 112), and depends upon endowments, entitlements and their diversity. Endowments represent people’s livelihood assets, referred to as ‘capitals’ in the Sustainable Livelihoods approach (Chambers & Conway, 1992) and include a mix of human (skills, knowledge), natural (land, trees, crops), financial (income, savings), social (networks, reciprocity) and physical (livestock, infrastructure) capitals (DFID, 1999). Entitlements represent goods and services that a person can gain access to ‘using the totality of rights and opportunities that he or she faces’ (Sen, 1984, p. 497 cited in Devereux, 2001). Livelihood resilience theory assumes that the accumulation and exchange of capitals provide a safety-net against shocks and stresses. Studies have shown that diverse assets and services (e.g. crops, financial and agricultural services) and strategies (e.g. alternative income-generating activities) improve resilience by spreading risk (Sallu et al., 2010; Sina et al., 2019; Speranza, 2013). To operationalize buffer capacity, it should therefore include locally-relevant indicators of assets, access and diversity, as it was achieved in previous studies (Matter et al., 2021; Pandey et al., 2017; Quandt, 2018). In the framework, we omit the human and social capital from buffer capacity due to significant overlap with the dimensions of livelihood resilience – self-organization and capacity for learning.

Self-organization can be understood as either the process of emergence/re-creation of society through a dialectic of social structures (top-down processes) and individual actions (bottom-up processes), or the ability of social actors to determine their own rules that affect them in a collective process of social cooperation (Speranza et al., 2014). Internal community interactions regulating these processes are vital for self-organization. Trust, norms and networks, which constitute key features of social capital, can improve the efficiency of society by facilitating cooperation and coordinated action (Putnam, 1993). This is fundamental for risk management. Previous studies have shown that many adaptive decisions, made at local levels, derive from participation in social networks and rely on good relations with others (Adger, 2003). Development practitioners who introduce a new set of rules and networks, alongside existing community dynamics and negotiations, influence components of self-organization. Outcomes, in terms of changes to household’s livelihood resilience, are likely to hinge on whether the process leads to increased trust and cooperation, or the opposite (Bahadur et al., 2013). Indicators reflecting participation in local groups and networks, cooperation and self-reliance are relevant to describe this dimension of livelihood resilience (Speranza et al., 2014).

Capacity for learning, the last dimension of livelihood resilience in the framework, describes the acquisition of new knowledge and skills, together with the capacity to act on them (Speranza et al., 2014). Ensor and Harvey (2015) argued that ‘learning processes can contribute to adaptive capacity by providing a way to alter practices or decision-making norms in the face of uncertainty’ (p. 512). Committing to learning and having the ability to identify valuable knowledge, through exploring various sources of information and considering new ideas and practices through experimentation, is the core in adapting livelihoods to ever changing conditions (Marschke & Berkes, 2006; Speranza et al., 2014). Development practitioners have an important role to play in facilitating the introduction of new knowledge and skills through adaptation and resilience projects. At the same time, any behavioural changes and acceptance of new knowledge are contingent on many factors and require supportive systems that foster processes of learning (Armitage et al., 2008).

Recent research shows that adaptation and resilience-building interventions do not adequately address the root causes of vulnerability (Eriksen et al., 2021). Although vulnerability is implicit in the original framework from Speranza et al. (2014), we position our analysis of livelihood resilience alongside the vulnerability context of climatic and socio-economic pressures that affect people’s lives and their sensitivity (Suckall et al., 2014). These pressures represent the type of disruption households experience from different events such as crop failure from drought, poor prices for produce and illnesses. Temporality, uncertainty, events, and changes plus other contextual factors such as history and policies, constitute the unique local context that situates livelihoods and projects. In this framework, the unit of analysis is the household, a level where the impacts of intervention are predominantly felt (Quandt, 2018). As a result of an intervention, the livelihood resilience of households may be enhanced, and their vulnerability reduced. Other outcomes might include unintended consequences such as increased vulnerability or even maladaptation (Barnett & O’Neill, 2010; Eriksen et al., 2021). The aim is to measure changes in vulnerability and livelihood resilience capacities between ex-ante (before) and ex-post (after) an intervention, and for households who participated in the intervention, and those who did not. By measuring change in vulnerability and resilience capacities from time T1 to T2, the framework goes beyond assessing a ‘snapshot’ of vulnerability and livelihood resilience in a fixed moment in time (cf. Quandt, 2018). Resilience, as an emergent property can only be manifested through the interaction of the system over time, thus a ‘snapshot’ of a few factors cannot accurately represent resilience (Tyler et al., 2016). Moreover, the framework is robust to the so-called ‘history threat’ that occurs when other confounding factors or events, which also could affect the outcomes, take place between the before and after measurements (Robson et al., 2001, p. 20). A study design measuring indicators before and after intervention, but without the non-participant comparison group, could be susceptible to this type of validity issue (cf. Jha et al., 2017). The Before-After-Control-Impact (BACI) or difference-in-differences (DiD) approach, presented in this study, aims to control for these contextual changes and constitutes a robust method to evaluate the impact of intervention on vulnerability and livelihood resilience (Bos et al., 2017; Smokorowski & Randall, 2017).

3. Materials and methods

3.1. Study area and the GCCA+ intervention context

The research was conducted among agricultural communities of the East Usambara Mountains located in Tanga Region, in north-eastern Tanzania. Due to proximity to the Indian Ocean, these highlands receive relatively high annual rainfall with approximately 1600–2300 mm, depending on location (Hamilton, 1989). The area has two main rainy seasons: the long rains of masika (March-May) and short rains of vuli (October-December). Long-term climatic records from the East Usambaras show that rainfall on the plateau is variable from year to year (Yanda & Munishi, 2007). In 1989, Hamilton and Macfadyen (1989) showed a tendency to a greater number of exceptionally dry years and a smaller number of exceptionally wet years, from the 1960s onwards. Rains have become less reliable and less predictable in terms of onset, duration and termination, which creates problems for the East Usambara communities relying on rain-fed agriculture. Climate projections for the area are also inconclusive. Some models suggest an increase in annual precipitation in this part of Tanzania (Luhunga et al., 2018), others project a drying trend for the region (Conway et al., 2017). Even the latest regional convection-permitting climate modelling, illustrates high levels of uncertainty in future projections of rainfall for the region (Chapman et al., 2020).

The European Union funded GCCA+ project (2015–2019) entitled ‘Integrated Approaches for Climate Change Adaptation in the East Usambara Mountains’ formed the basis for this study. It was implemented by two non-governmental organizations – Engineering for Human Development (ONGAWA) and the Tanzania Forest Conservation Group (TFCG) – and Muheza District Council to support the highland communities to adapt to climatic changes and to strengthen livelihood resilience. Two research institutions contributed to the design of monitoring and evaluation and conducted research to stimulate learning on resilience and adaptation. The project covered eight rural villages with a population of 10,992 people in 2853 households (ONGAWA, 2013). The dominant tribe in the area is the Wasambaa, but other ethnicities such as Zigua and Bondei are also present due to historical in-migration to the region (Powell et al., 2013). Average household income in the period 2016–2019 was 355 USD per annum (own data), which signals abject rural poverty even before income distribution to all household members (Aikaeli, 2010). The villagers depend mostly on farming and grow various food and cash crops including maize, cassava, beans, yams, banana, sugarcane, tea, vegetables and spices (Reyes et al., 2005). Job opportunities outside agriculture are scarce (small businesses, motorbike taxi, construction) and even employment in the nearby tea estates is vulnerable to low rainfall limiting the tea yield. The project introduced various capacity building and income-generating activities and technologies aiming to support local livelihoods under climate change (see Table 1). Some of the implemented activities sought to reduce households’ vulnerability and improve buffer capacity through spice tree nurseries, livestock hand-outs, farmer field schools and extension for uptake of climate-smart agriculture (CSA) technologies, and income-generating forest-based enterprises (butterfly farming, beekeeping, ecotourism). Others supported the functioning of village structures and networks overseeing forest, land and water management (including water supply technologies) and various self-help and development groups nurturing collaboration and self-reliance (ONGAWA, 2013). The training that was provided and the knowledge platforms that were developed were designed to enhance capacities to learn and to experiment, contributing further to building livelihood resilience in the East Usambara Mountains.

Table 1. Summary of the GCCA+ East Usambara project activities and corresponding vulnerability and livelihood resilience indicators for which the project impact was assessed.

Description of project activity	Indicator of vulnerability and livelihood resilience	Theoretical link of project activity with indicator *^, **	Measurement
Exposure
Climate smart agriculture (CSA)+	Climatic pressures	CSA technologies decrease exposure to climatic pressures (Khatri-chhetri et al., 2017)	Climatic pressures experienced (yes/no, crop failure from drought, crop pests and diseases, soil erosion, livestock diseases)
CSA Financial education Business education Searching for markets Income generating activities	Socio-economic pressures	Education, diversified income-generating activities and access to markets decrease exposure to socio-economic pressures (Sina et al., 2019; Thulstrup, 2015)	Socio-economic pressures experienced (yes/no, poor price for produce, illness/family problems, land shortage, transport problems, lack of information, lack of tools)
Sensitivity
Improved water supply++ Management of water resources Environmental by-laws regulating agricultural activities near water	Water shortage	Improved water infrastructure and water management decrease sensitivity to water shortages (Thulstrup, 2015)	Months HH has insufficient water during the year (total number of months)
CSA Income generating activities	Food shortage	Improved and diversified farm technologies and income activities decrease sensitivity to food shortages (Matter et al., 2021; Speranza et al., 2008)	Months HH has insufficient food during the year (total number of months)
Buffer capacity
Training on tree nurseries Promoting agroforestry (including spice trees)	Tree ownership+++	More household resources (trees) increase buffer capacity (Quandt et al., 2017)	Trees owned by HH (total number of fruit and spice trees)
Training on dairy livestock husbandry Livestock hand-outs	Livestock ownership+++	More household resources (livestock) increase buffer capacity (Weldegebriel & Amphune, 2017)	Livestock owned by HH (total number of livestock)
CSA technologies	Crop diversification	Increased diversity of crops increases buffer capacity (Speranza, 2013)	Crops grown by HH (total number of different crops grown)
Income generating activities: butterfly farming, dairy cattle, eco-tourism, beekeeping, income from CSA and small businesses	Total income	Diversified household resources (income) increase buffer capacity (Sina et al., 2019)	Sum of all income sources (agriculture, salaried job, own business, daily labour, dairy cattle, forest enterprises)
Access to inputs, access to extension (agricultural and livestock), access to agricultural storage facility	Agricultural services+++	Better access to agricultural services increases buffer capacity (Speranza, 2010)	Agricultural services HH has access to (total number of services)
Rural microfinance groups Access to bank loans	Financial services+++	Better access to financial services increases buffer capacity (Ulrich et al., 2012; Weldegebriel & Amphune, 2017)	Financial services HH has access to (total number of services)
Economic groups Access to market: spices, milk, butterflies, honey and eco-tourism	Market services+++	Better access to market services increases buffer capacity (Speranza, 2010)	Market services HH has access to (total number of services)
Self-organization
Establishment of new community groups and village institutions (as above)	Networks	Participation in networks supports self-organization (Marschke & Berkes, 2006; Speranza et al., 2014)	Professional groups and village institutions HH members are part of (total number of groups)
Cooperation through group work	Cooperation	Cooperation increases self-organization (Speranza et al., 2014)	Reciprocal livelihood activities (total number of productive activities done with neighbours)
Supporting self-reliance within family and neighbours	Self-reliance	Availability of own/local sources of support increases self-organization (Speranza et al., 2014)	Own/local sources of support HH has access to (total number of local sources of help)
Capacity for learning
Knowledge platforms (places/people) for discussing issues and opportunities	Information sources	Availability of alternative sources of knowledge increases capacity for learning (Speranza et al., 2014)	Information sources HH uses (total number of information sources)
Provision of education to established groups and institutions	Commitment to learning	Committing to education increases capacity for learning (Speranza et al., 2014)	Trainings HH participated in during the project (total number of trainings received)
CSA technologies	Experimentation	Experimentation increases capacity for learning (Marschke & Berkes, 2006; Speranza et al., 2014)	Sustainable agricultural technologies HH practices on their own farm (total number of CSA technologies practiced)

+CSA refers to integrated land management framework with three main objectives: increase productivity (intensification), enhance resilience to climate change (adaptation), reduce greenhouse gas emissions (mitigation). See Figure A2 for breakdown of CSA technologies promoted by the GCCA+ in the East Usambara.

++Improved water supply refers to installation of 31 new public water points (small gravity technology).

+++ Livelihood activities and services already existent in the project villages that were intended to scale up by the intervention.

*Decreased exposure = reduced vulnerability; decreased sensitivity = reduced vulnerability.

**Increased buffer capacity, self-organization, capacity for learning = increased livelihood resilience.

3.2. Data collection

We used quantitative household survey and qualitative insights from focus groups, interviews and ethnographic notes from observations of project activities to help interpret the quantitative findings and to provide additional context. Structured survey information was collected in person in October 2016 and September 2019 from the same households, which provided panel data of 291 randomly selected households before and after the intervention. Panel data (the same person/household over time) is considered a powerful tool for estimating policy or project effects (Wooldridge, 2012, pp. 465–468). The sample comprised of both project participants and non-participants. The project participants in the intervention were selected based on specific criteria such as ‘spice producer’ or ‘suitable land holder’ for some groups, e.g. economic groups and butterfly-farmer groups respectively, and general criteria such as ‘at least 50% from poor/vulnerable background’, ‘50% females’ for most groups. In our survey a ‘household’ was defined by the Swahili word kaya that implies family members that have a common residency (Randall et al., 2011). The survey was divided into six sections covering the following topics: basic households’ characteristics, households’ vulnerability, buffering capacities and assets, agricultural practices, social capital and self-organization, capacity for learning and project participation. The questions were close-ended with a mix of nominal, multiple-choice, and yes/no questions (Creswell, 2003). For instance, the participants were asked how many months they experienced water shortage in the last 12 months (nominal) or to indicate which agricultural services they have access to (multiple choice). This research was conducted according to the principles of academic excellence and integrity. The rights and dignity of the research subjects were prioritized on all occasions, including obtaining informed consent, data protection, anonymity and confidentiality and the right to withdraw from the study at any time.

Qualitative focus groups (FG) and face-to-face interviews were carried out in intervals from December 2016 to April 2019. The aim of the focus groups (n = 8) conducted in each intervention village in 2016 was to investigate the villages vulnerability context. Further interviews (n = 60) were undertaken between March 2018 and April 2019. The respondents were selected through convenience and targeted sampling (Newing et al., 2011). These interviews were semi-structured and unstructured, and while some topics related to the participation in the project were pre-prepared, there was also scope for the interviewees to express their own opinions (Bernard, 1995). We also carried out interviews with households who did not participate in the project (n = 12) and those who dropped out after some time (n = 10) due to lack of time or not seeing the benefits. Ethnographic notes and ‘jottings’ produced throughout the fieldwork provided additional analytical depth and context (Bernard, 1995, p. 181). All interviews were conducted in locally relevant languages (Kiswahili or Kisambaa), recorded on a voice recorder, and then transcribed into English by two qualified translators.

3.3. Indicators and measurements

It is crucial to select context appropriate indicators when operationalizing a method or framework (Jones & Tanner, 2017). We constructed 12 simple indicators of vulnerability and 13 simple indicators of livelihood resilience based on the respondent’s survey answers. The indicators reflect different dimensions of locally relevant characteristics of vulnerability and livelihood resilience in relation to the activities that the GCCA+ project implemented. Table 1 provides evidence of theoretical literature, which, together with the expertize of researchers and practitioners in the field, informed the choice of indicators. Each indicator measures one outcome used in the difference-in-differences regression models described below, against which the project impact was evaluated.

3.4. Data analysis

3.4.1. Project impact evaluation method

We assessed the project impact using several analytical methods. For the quantitative analysis we adopted the difference-in-differences (DiD) method (Angrist & Pischke, 2008) to allow for comparisons over time between participant and non-participant households (HHs). The DiD method measures the effect of a treatment or intervention by comparing the treatment group (the participant HHs), before and after the intervention, with a similar control group (the non-participant HHs) that did not receive the intervention. The intervention effect is the difference between the participant HHs and the non-participant HHs, across time. This is possible because the non-participant comparison group effectively ‘differences-out’ contemporaneous confounding factors (Antonakis et al., 2010). We define participant HHs as those with at least one household member participating in the project and non-participant HHs as those whose household members did not participate in any of the project activities. DiD assumes that, in the absence of the intervention (counterfactual outcome scenario), the difference between those two groups would be relatively stable over time and that their trends would follow parallel paths. Groups that do not exhibit those characteristics are not suitably matched analytical groups (Wing et al., 2018). Figure 2 demonstrates a simple two-period and two-group DiD design, as adopted in this study.

Figure 2. Estimating intervention effect using difference-in-differences. Reprinted from Antonakis et al. (2010, p. 1109).

The statistical analysis was performed in Stata 16.1 software. We obtained differences in outcomes between the participant and non-participant HHs before and after the intervention (in T1 and T2) and the ‘participation by time’ interaction terms (DiD estimators) by computing difference-in-differences regression models for continuous outcomes of livelihood resilience and sensitivity and logit regression models for binary outcomes of exposure. Participation (dummy variable where 0 = non-participant HHs and 1 = participant HHs), time (dummy variable where 0 = pre-project period and 1 = post-project period) and the interaction of ‘participation by time’ were the model fixed effects. Covariates representing HHs socio-demographic characteristics (see section ‘Characteristics of participating and non-participating households’ below) were added to the models to mitigate for potential imbalances between participant and non-participant HHs at baseline. Robust standard errors were used in all models. Our model is summarized in the simplified equation below: $Y={\beta }_0+{\beta }_1\ast \mathrm{Participation}+{\beta }_2\ast \mathrm{Time}+{\beta }_3\ast (\mathrm{Participation}\times \mathrm{Time})+\mathrm{covariates}+\mathrm{error}\mathrm{term}$

Our primary interest – the coefficient β₃ – captures how the mean change in outcomes from before to after the intervention was different for the participant and non-participant HHs, thus reflecting the project impact (DiD estimator). In the probability metric of the logit models of exposure, the project impact was obtained from additionally calculating the marginal effects. For variables of vulnerability, where higher absolute values indicate higher vulnerability, positive and significant DiD scores indicate greater increase in vulnerability in the participant group, in comparison to non-participant group, therefore negative project impact. Inversely, for indicators of livelihoods resilience, where higher absolute values indicate more resilience, positive and significant DiD scores indicate positive change for the participant group in comparison to the non-participant group and thus positive project impact. When the mean difference in outcomes from before to after was greater for the non-participant than the participant HHs, and/or the DiD estimator was not statistically significant, there was no project impact.

We also analysed trends in rainfall data sourced from Marikitanda Tea Research Centre and the East Usambara Tea Company (EUTCO) to determine whether annual rainfall has changed over time using the Mann-Kendall trend test.

Qualitative interviews, focus groups and field notes supported the quantitative analysis. They were coded in NVivo 11 software using thematic analysis (Boyatzis, 1998). Codes and themes were developed along the research process and represented developed theories about the project impact on participant and non-participant HHs. We frequently compared quantitative and qualitative data to triangulate our findings. Developed memos also provided relevant quotes for this work.

3.4.2. Limitations of the framework, data and design

To confidently attribute changes in vulnerability and livelihood resilience to the project activities, the participant and non-participant HHs should be comparable in the pre-project period, with regards to the outcome indicators and the covariate distributions of the household characteristics (Sills et al., 2017). Trends in outcomes can be plotted if sufficient data is available (>2 time periods). In this study we lack such data therefore baseline trends could not be properly evaluated. Three out of 25 of our outcome indicators (‘getting poor price for produce’, ‘information sources’ and ‘markets’) were statistically different at baseline for the participant and non-participant HHs (p < .01, p < .05, p < .05 respectively), with the participant HHs having higher means (for indicators of resilience) or lower proportions (for indicators of vulnerability). This could suggest that the participant and non-participant HHs have already been on different pathways with regards to those indicators. Furthermore, the difference-in-differences methodology is sensitive to ‘spillovers’ from higher level interventions to the households’ level variables. Such ‘spillovers’ might reduce the ability to separate the project impact from other factors. In this study we use qualitative understanding to triangulate our findings. Finally, 4 out of 13 of the baseline HHs covariates (‘HH size’, ‘wealth ranking’, ‘HH head years of schooling’ and ‘ownership of mobile phone’) showed statistical differences between the participant and non-participant HHs, exposing the participant selection bias. These covariates were included in all difference-in-differences models as controls.

4. Results

4.1. Characteristics of participating and non-participating households

Descriptive statistics of socio-demographic characteristics of the participant (hereinafter P) and non-participant (hereinafter NP) HHs pre-project are presented in Table 2. Among the random sample of HHs surveyed across the project villages (10.2% of all HHs), 207 HHs participated in the project and 84 HHs did not. The results show that P HHs were bigger, have HHs heads that are more educated and have more access to mobile phones. P HHs were also less poor than NP HHs despite the project intention to target poor and more vulnerable HHs. All other socio-demographic characteristics were comparable (Table 2).

Table 2. Comparison of the socio-demographic characteristics of the participant and non-participant households at baseline.

Variable	Participant HHs (N = 207)	Non-participant HHs (N = 84)	p-value
	Mean / Proportion
HH size	4.69 (2.02)	4.11 (2.25)	0.034**
Working age adults	2.00 (1.11)	1.90 (1.14)	0.491
Male	53.2% (110)	50.0% (42)	0.627
Female	46.8% (97)	50.0% (42)
Male-headed HHs	78.3% (162)	71.4% (60)	0.214
Female-headed HHs	21.7% (45)	28.6% (24)
Poor HHs	41.1% (85)	52.4% (44)	0.049**
Average HHs	47.3% (98)	44.0 (37)
Better off HHs	11.6% (24)	3.6% (3)
Land owned (acres)	3.14 (3.24)	2.66 (2.15)	0.214
Years of schooling (HH head)	6.17 (3.37)	5.19 (3.04)	0.021**
From the village	71.01% (147)	69.05% (58)	0.739
Has electricity	22.71% (47)	21.43% (18)	0.813
Has mobile phone	73.91% (153)	55.95% (47)	0.003***
Has transport	13.53% (28)	9.52% (8)	0.347
Has satisfactory house+	88.89% (184)	82.14% (69)	0.122
Using improved water source++	28.50% (59)	20.24% (17)	0.146

p-values estimated by two sample T-test (for means) and Pearson chi2 (for proportions) *** p < .01; ** p < .05; * p < .1.

Reported in brackets – standard deviation for variables expressed as means and No. of respondent HHs for variables showing proportions.

+Walls and roof made of solid materials such as wooden poles (with mud), bricks and iron sheet roof.

++Piped water, borehole or protected stream.

4.2. Vulnerability context and mitigation

During the project implementation period, the climatic conditions and related livelihoods pressures changed. At the start of the project, in 2016/2017 the East Usambaras, like most of the region, experienced severe drought caused by the El Niño–Southern Oscillation (ENSO). On the other hand, in the second half of the project in 2018/2019 the rainfall was particularly abundant (Figure 3). Our analysis of annual rainfall recorded daily in the East Usambara Mountains between 1969 and 2019 confirmed considerable annual variation, however there was no evidence of long-term decline or increase in rainfall in the area (Mann-Kendall trend test – monthly averages used, tau = −0.0204, Prob > |z| = 0.451).

Figure 3. Trends in annual rainfall in the East Usambara between 1969 and 2019 with fitted trend line. The period when the project was implemented is enlarged (averages by month). Data sourced from Marikitanda Tea Research Centre (continuous line) and the East Usambara Tea Company (EUTCO) (dotted line). Data for 1972 is missing.

Regression models demonstrated that all indicators of climatic pressures improved during the project. However, the observed changes were not related to the project activities – there was no statistical difference between P and NP HHs (Table 3). With regards to socio-economic pressures the project had small positive impact on mitigating ‘lack of tools and equipment’ with P HHs having lower probability of reporting those issues. Notably two indicators of socio-economic pressures worsen over time – ‘getting poor price for produce’ and ‘land shortages’ with P HHs being less likely of having land shortages than NP HHs, although this effect also cannot be assigned to the project activity. Poor prices for produce may relate to the seasonal variation in production that may have been influenced by the increase in rainfall in the second half of the project. This is also demonstrated by the indicators of sensitivity discussed next. Reported number of months with food and water shortages decreased for both groups on average from over 4.5 to 1.5 months (food shortage) and from 2.3 to 1 month (water shortage) from before to after the project. Again, the differences between P and NP HHs on those indicators across time were not significant and thus not attributable to the project.

Table 3. Comparison of variables of exposure and sensitivity before and after the intervention for participants and non-participants households.

	Participant HHs (N = 207)		Non-participant HHs (N = 84)		Diffrence-in-diffrences model (robust s.e.)	Project impact
	Before	After	Before	After
Climatic pressures+					Marginal effects
Crop pests and diseases	35.2% (73)	16.9% (35)	38.0% (32)	11.9% (10)	0.073 (0.076)	No impact
Crop failure from drought	89.3% (185)	56.0% (116)	88.0% (74)	45.2% (38)	0.083 (0.076)	No impact
Soil erosion	30.4% (63)	13.0% (27)	33.3% (28)	8.3% (7)	0.089 (0.073)	No impact
Livestock diseases	16.4% (34)	6.7% (14)	14.3% (12)	3.5% (3)	0.027 (0.053)	No impact
Socio-economic pressures+
Getting poor price for produce	40.5% (84)	53.1% (110)	46.4% (39)	54.7% (46)	0.042 (0.089)	No impact
Illness and/or family problems	54.1% (112)	39.1% (81)	57.1% (48)	47.6% (40)	−0.051 (0.089)	No impact
Land shortage	29.9% (62)	30.4% (63)	27.3% (23)	33.3% (28)	−0.044 (0.083)	No impact
Transportation problems	38.1% (79)	10.1% (21)	39.2% (33)	5.9% (5)	0.060 (0.070)	No impact
Lack of information and/or understanding	24.6% (51)	14.9% (31)	27.3% (23)	13.0% (11)	0.050 (0.071)	No impact
Lack of tools and/or equipment	28.9% (60)	9.6% (20)	20.2% (17)	14.2% (12)	−0.139 (0.067)*	Positive impact
Sensitivity++					DiD scores
Food shortage	5.01 (2.98)	1.53 (1.79)	4.63 (3.17)	1.63 (1.73)	−0.433 (0.460)	No impact
Water shortage	2.28 (2.23)	1.09 (1.81)	2.28 (2.64)	0.93 (1.87)	0.159 (0.421)	No impact

All models were adjusted for socio-demographic differences between P and NP HHs.

p-values *** p < .01; ** p < .05; * p < .1.

+ Indicators of climatic and socio-economic pressures (binary dependent variables) estimated with logistic regression (proportions and marginal effects reported).

++ Indicators of sensitivity (continues dependent variables) estimated with linear regression (means and DiD scores reported).

4.3. Enhancing livelihood resilience

Table 4 summarises the results of the project impact on different dimensions of livelihood resilience. The DiD estimators were calculated by difference-in-differences linear regression models and adjusted for socio-demographic differences between P and NP HHs presented in Section 4.1. Qualitative enquiry supported the interpretation of the results throughout.

Table 4. Project impact on livelihood resilience before and after the intervention for participants and non-participants households.

Indicators of livelihood resilience	Participant HHs (n = 207)		Non-participant HHs (n = 84)		Difference-in-differences model
	Before	After	Before	After	Coefficients Participation x Time (DiD scores) (robust s.e.)	Project impact
Buffer capacity	Means (SD)+
Tree ownership	30.41 (74.38)	35.70 (86.21)	20.08 (33.83)	31.50 (90.53)	−5.944 (13.374)	No impact
Livestock ownership	1.17 (2.81)	1.65 (3.22)	0.82 (1.62)	1.14 (2.79)	0.092 (0.447)	No impact
Crop diversification	5.62 (2.75)	6.43 (3.15)	5.48 (2.85)	4.66 (2.97)	1.660 (0.497)***	Positive impact
Total income (USD)	315.26 (551.0)	460.13 (684.53)	255.80 (390.84)	391.48 (553.31)	−5.456 (91.113)	No impact
Agricultural services	0.86 (1.01)	1.02 (0.87)	0.85 (0.98)	0.38 (0.63)	0.581 (0.160)***	Positive impact
Financial services	0.83 (0.72)	1.15 (0.79)	0.80 (0.71)	0.50 (0.63)	0.607 (0.123)***	Positive impact
Market services	0.47 (0.56)	0.72 (0.67)	0.30 (0.51)	0.75 (0.55)	−0.162 (0.103)	No impact
Self-organization
Networks	0.65 (1.06)	2.89 (2.54)	0.42 (0.74)	0.46 (0.79)	2.155 (0.220)***	Positive impact
Cooperation	0.35 (0.63)	0.22 (0.75)	0.28 (0.50)	0.22 (0.44)	−0.063 (0.098)	No impact
Self-reliance	0.84 (0.95)	1.37 (0.50)	0.79 (0.87)	1.39 (0.65)	−0.082 (0.144)	No impact
Capacity for learning
Information sources	2.40 (1.53)	5.29 (3.51)	1.83 (1.28)	1.93 (2.23)	2.694 (0.386)***	Positive impact
Commitment to learning	0.09 (0.29)	2.85 (2.89)	0.07 (0.25)	0.11 (0.32)	3.467 (0.300)***	Positive impact
Experimentation	3.41 (2.30)	2.66 (1.60)	2.95 (2.35)	1.45 (1.60)	0.787 (0.357)**	Positive impact

All indicators of livelihood resilience estimated with linear regression adjusted for socio-demographic differences between P and NP HHs.

p-values *** p < .01; ** p < .05; * p < .1.

+Unadjusted means reported.

4.3.1. Buffer capacity

Regression analysis suggests that the implementation of the project led to an improvement in indicators of ‘crop diversification’, ‘agricultural services’ and ‘financial services’ among P HHs compared to NP HHs, which indicates a positive project impact. However, further analysis supported by qualitative data suggests a possible unintended outcome for the NP HHs. In parallel, the project oversaw the introduction of village-level by-laws that prohibit growing certain crops such as yams and vegetables, near water courses, for environmental conservation reasons. The number of NP HHs vegetable cultivators dropped from 26.1% before the project to 9.5% after, while only about 3% of P HHs gave them up (Figure 4). Similarly, amid those by-laws NP HHs did not expand planting yams, in contrast to P HHs where the number of yam cultivators increased by about 20%. It is likely that while the project aimed to reduce cultivation near the water for conservation purposes overall, it disproportionally affected those households that did not participate, and were not provided with alternatives. P HHs, which were more powerful and better positioned in the village, likely were less intimidated by the new by-laws and may have found a better way not to obey them. In our interviews, poorer NP HHs seemed to be the most disappointed with those by-laws:

I cannot accept such solutions. With one yam you can feed all your children (…) They grow fast [yams] and we also sell them seasonally. (ID60, women)

Figure 4. Crops grown by participant and non-participant households before and after the intervention.

A female focus group participant also explained that vegetable cultivation helps women to earn income during drought:

When it is dry like now, we plant vegetables [in farms near water to allow bucket irrigation] and sell them to fellow villagers because not many people have vegetables on their farms [that are not near water]. (FG2, women)

Similarly, increased access to agricultural services for the P HHs, through the farmer’s field schools, may have had a negative knock-on effect on the NP HHs that struggled to access extension services during the programme operation period. Indeed, testimony from NP farmers suggest that district staff became too occupied with the project activities, which resulted in decreased delivery of everyday services to other villagers:

I used to call XX [extension officer] for advice but he is too busy now with shamba darasa [farmer field school] (…). (ID12, man)

Regression results also show that, overall, the project had little impact on ‘total income’ earned, ‘tree and livestock ownership’ and ‘market services’, despite the introduction of income-generating activities (e.g. butterfly farming, beekeeping and ecotourism), dairy cattle hand-outs and searching for new markets. Interviews with farmers indicated that agricultural income did not significantly increase, after the project, due to poor uptake of climate-smart technologies, long maturing period of introduced cash crops, and poor performance of drought-resistant seeds due to heavy rainfall. Other income generating activities such as butterfly farming were also compromised by a nationwide ban on exporting live animals (including butterfly pupae) issued by the Tanzanian government in 2016. Detailed breakdown of different income sources before and after the project is presented in the Appendix in Figure A1. On the other hand, there was a high interest in possession of spice tree seedlings (indicator of ‘trees ownership’) in both groups over time, followed by an increased access to ‘market services‘ (Table 4). Although not attributed to the project, this suggests a growing desire for planting spices as a chosen adaptation strategy among the villagers:

When I think about the future, I see that the spices can take us to the next level. Nothing else. (ID34, man)

4.3.2. Self-organization

The project’s positive influence on self-organization manifested in increased number of networks that more than doubled for the P HHs compared to NP HHs (Table 4). This is likely to be due to the many development groups and village institutions the P HHs engaged with throughout the project. On the other hand, small means of the indicator of ‘cooperation’, which captures the help HHs offer to each other in relation to productive livelihood activities, show that cooperation was low in the intervention villages; and this was independent of the project. During our focus groups the residents explained:

Most families work individually here. But people show good collaboration for funerals, weddings, births and to attend sick persons. (FG3, women)

Focus groups attendees also suggested that people might increase support to each other during drought:

I know that my fellows who are more blessed than me will give me some work. In the year like this [drought] we sometimes work for cassava [provide labour in exchange for food]. Others will give you food even for free. (FG4, woman)

Inversely, the improvement over time in the indicator of ‘self-reliance’ was significant, however no difference between P and NP HHs indicates no project impact on this indicator either.

4.3.3. Capacity for learning

The results show a positive project impact on the capacity for learning dimension of livelihood resilience. Overall, during the project, P HHs received almost 3.5 times more training than NP HHs and increased their use of different information sources on average over 2.5 times. Regression analysis also suggests that despite downward trend the project positively influenced agricultural experimentation, with P HHs reporting practicing more climate-smart technologies than NP HHs (Table 4). Interestingly, during interviews, majority of participating farmers reported struggles with practicing many of the introduced technologies, for instance constructing terraces on their farms:

This is a very hard work and requires many men to accomplish. You need to have money to hire people to help you. (ID 23, man)

The survey and interviews also provide evidence that certain technologies promoted by the project, particularly those focusing on soil conservation in food production such as contour farming, retaining grasses or mulching, may have been considered redundant by some farmers and show low uptake amid increasing spice cultivation (compare Figure A2 in the Appendix). A testimony from one village leader confirms this:

Farmers now calculate how much return they can get from 1 acre of land. Spice production gives them better return than cultivation of maize or cassava. From the money earned they can easily buy food. (ID7, man)

5. Discussion

Having shown how the BACI Adaptation Impact Evaluation Framework can be operationalized to evaluate the impact of adaptation interventions, we now move to discuss the key findings from its application during this study. We make inferences with other studies on adaptation and resilience-building interventions, reflect on our learning, and offer recommendations for development practice.

5.1. Vulnerability context and mitigation

In East Usambara, the vulnerability context changed with the change of rainfall from 2016 to 2019, and the outcomes for households’ vulnerability proved largely independent from the project activities. One reason is that the project predominantly focused on mitigating drought, and its impacts likely became less relevant when climatic conditions changed from drought to heavy rain. For instance, Gaworek-Michalczenia et al. (2019) noted that drought or disease-resistant seeds do not perform well in wetter conditions in the East Usambara Mountains. On the other hand, the observed outcomes of socio-economic pressures for P and NP HHs likely reflect the existing differences between them: for instance, NP HHs with fewer assets, education, and mobile phone access might have more difficulty in securing enough land. However, those differences in vulnerability between P and NP HHs were mostly not significant and should be interpreted with caution. Our findings support previous studies that highlight the necessity of thorough investigation of a broad range of shocks and stressors and their effects on livelihoods to inform adaptation and resilience intervention (Béné et al., 2015; Frankenberger et al., 2014). Current projection models are not clear cut about the direction of precipitation change in this region of Tanzania (Chapman et al., 2020). While drought remains an important livelihood risk, increases in extreme rainfall are expected, especially in the highlands (Luhunga et al., 2018). Development practitioners must therefore ensure a risk-informed project design, and be prepared to address various climate scenarios to avoid gearing up for short-term solutions and heading towards maladaptive outcomes in the future (Magnan et al., 2016). At the same time, there are challenges that development practitioners face when working with rain-fed agriculture-dependent communities in areas with a highly uncertain and variable climate. While longer projects may have more opportunities, for instance, to test different seed varieties ‘on the ground’ in various weather conditions, it is clear that the ‘best practice’ guides used in designing similar interventions do not offer enough guidance on how to make approaches like climate-smart agriculture more effective under such conditions (see URT, 2017). Those gaps are limiting the impact of investments on reducing vulnerability and building resilience to climate change.

5.2. Enhancing livelihood resilience

Our analysis illuminated various challenges for development assistance to design programmes that enhance livelihood resilience. Positive impact of the project was most visible on the capacity for learning dimension of livelihood resilience, particularly the indicators of ‘commitment to learning’ (training that HHs received) and diversified ‘information sources’ (knowledge platforms HHs use). The project committed to fostering learning including the development of ‘know-how’ practical skills and other education, involving training on good governance, entrepreneurship, or natural resource management, and created links between those platforms. There is also a possibility that ‘information sources’ were partially increased through other factors, not related to the project. On the other hand, primary activities aiming to build households’ buffering capacities, such as income-generating activities, dairy livestock keeping, climate-smart agricultural technologies, or market services, demonstrated limited impact at the end of the project. Frankenberger et al. (2014) noted that short funding cycles of development projects do not allow the time for addressing adaptive and transformative capacities related to the underlying drivers of risk and vulnerability, such as human capital (skills, health, education), good governance, infrastructure (markets, roads, communications systems), policies and regulations that provide enabling environment for bigger systemic changes (Béné et al., 2015). Our findings suggest that projects can potentially be successful in transmitting new knowledge related to those capacities during the intervention period. However, the transmission of knowledge alone is not sufficient for a meaningful modification of the capacity base necessary for resilience building. Speranza et al. (2014) argued that learning in resilient systems should come with the ability to translate new knowledge into action. Acceptance and subsequent conversion of new knowledge into action is contingent on many factors, including the broader systemic processes that foster social learning (Rist et al., 2006), existing ‘generic‘ capacities (Williams et al., 2015), shared vision (Wenger, 1998) and many others. In the East Usambara Mountains some of the introduced ‘new knowledge’, including various climate-smart agricultural technologies, seemed disconnected from the residents’ existing capacities and their shared adaptation vision. This was evident in P and NP HHs high uptake of spice tree seedlings and low experimentation with technologies in food production, which also required additional funds. The growing importance of spice production in the area requires continued support with markets and creating enabling conditions allowing households to earn sustainable income from the sale of spices. This in turn requires sustainable funding mechanisms, and longer-term intervention consisted of multiple, complementary, and integrated activities, instead of short-term and fragmented projects demonstrating transient impacts.

Nurturing community self-organization is another livelihood resilience building strategy (Speranza et al., 2014). The project was mindful of community self-organization and achieved positive impact in expanding social networks, which constitute an essential part of social capital, upon which people draw in pursuit of their livelihoods (DFID, 1999). However, the expansion of community networks did not go in tandem with the increase in households’ cooperation, which was reported as low for activities other than funerals, births, weddings, and attendance of the sick. Slightly better cooperation was reported in the dry year (2016) than the wet year (2019), for both groups, which signal that there was some work-related cooperation among neighbours during drought. Those results were nevertheless independent of the programme activities. Bahadur et al. (2013) noted that it is somehow naïve to assume that project-based interventions, limited in time and resources, can fully accommodate the complexity of fostering community networks and cooperation. It is like ignoring that multiple values, agendas and interests are present in ‘communities’, which are not made up of homogenous groups of people (Agrawal & Gibson, 1999). This part of building livelihood resilience requires developing long-term relationships of trust, negotiating differences and building on shared vision with communities (Bahadur et al., 2013); and this is difficult to achieve in 3–5 year-long projects. At the same time projects should recognize that certain adaptation or resilience-building activities may themselves erode trust and create unintended, and often negative externalities for livelihoods, which we discuss next.

During the project tensions arose between conservation policies and livelihood enhancement goals, due to their disproportionate negative effect on the NP HHs. Negative impacts included reduced access to district extension staff, who became less available while working on the project, and decreased crop options due to conservation-related by-laws. The latter is also linked to elimination of an important livelihood option in times of drought (yam and vegetable cultivation). While protecting key resources is part of learning, and attention to major ecological aspects of resilience is necessary to achieve long-term sustainability (Folke & Gunderson, 2010), trade-offs between livelihoods and natural resources and across scales (household-community-landscape) are inevitable and need to be considered at design stage to avoid unintended outcomes. Bamberger et al. (2016) argued that ignoring unintended outcomes in development projects, especially those affecting the most vulnerable groups, leads to breaching equity objectives and allows more powerful citizens to reap the benefits or even worsen the situation of the poor. It is especially likely if the levels and quality of participation and representation are low, with those who participate not adequately representative of the full extent of the community and/or merely consulted (Botes & Van Rensburg, 2000). This runs the risk of many important situations and lived experiences relating to the presence of the intervention going unnoticed (Morell, 2010). This also resonates with what has long been discussed in the livelihoods and human-oriented resilience literature: what looks like resilience building from one perspective or at one level might not be when observed from another level or perspective; thus tools to achieve resilience are not value neutral (Béné et al., 2014; Brown, 2014; Marschke & Berkes, 2006). Recognizing this is a first step for more reflective and ultimately more equitable adaptation and resilience intervention.

6. Conclusions

The BACI Adaptation Impact Evaluation Framework expanded the livelihood resilience framework of Speranza et al. (2014) to incorporate a BACI research design, which allows to accurately evaluate the impact of adaptation interventions on vulnerability and livelihood resilience of households. It is relatively easy to operationalize yet a robust tool that can be used to assess vulnerability and resilience in different contexts. Some knowledge of inferential statistics and longitudinal data sets with at least two time points is, however, required to successfully operationalize this framework. We envisage that this study will be relevant to the international community, including donors and development practitioners, involved in delivering and evaluating adaptation and resilience programmes supporting livelihoods. This framework and associated methodology can generate robust evidence on the impacts of resilience-building initiatives that can inform learning within projects and inform future intervention design. This is needed to ensure more effective and equitable adaptation and resilience-building intervention, and avoidance of unintended outcomes, which then can guide recommendations for setting policy priorities.

Our focus on several key outcomes in the GCCA+ project, enabled us to accurately evaluate which dimensions of livelihood resilience and vulnerability the project influenced and in what ways. Education and hands-on training in local groups visibly strengthened the capacity for learning and networks for the participating households. Yet the project impact on most buffering capacities (e.g. income, markets) remained low. We trace this result to the traditional funding mechanisms, where the short-term cycles of projects do not allow for effective transfer of knowledge into action. Funding should therefore prioritize long-term in situ projects at local and regional levels to increase ongoing support for the introduced resilience-building activities, while communities still learn how to derive sustainable benefits from them. We also produced evidence of unintended consequences and maladaptation, particularly among poorer non-participating households. This shows that trade-offs between livelihoods and environmental policies, which are frequently unrecognized in adaptation interventions, are real and should be considered carefully by development practitioners. Finally, the results show that in communities where livelihoods are largely dependent on rain-fed agriculture, variable weather plays a significant role in determining the vulnerability context of livelihoods. Hence, projects intending to reduce vulnerability must consider climatic variability and apply risk-informed project design to avoid maladaptive outcomes. At the same time, there is a need for more research and practical guidance on how such intervention might look in areas with highly uncertain climate scenarios.

Acknowledgements

The authors are indebted to the implementing team from TFCG, ONGAWA and Muheza District Council for executing the household survey and their support while in the field. Our gratitude must also be extended to the research assistants Kajiru and Salimu and the communities of the East Usambara.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

The research was funded by the Economic and Social Research Council’s White Rose Collaborative Doctoral Studentship programme in the UK [grant ES/J500215/1] with contributions from the EU-funded GCCA+ Tanzania programme and the Centre for Climate Change Economics and Policy (CCCEP) funded by the UK Economic and Social Research Council [grant ES/K006576/1]. Time of Sallu has also been supported by Biotechnology and Biological Sciences Research Council as part of the Global Challenges Research Fund AFRICAP programme [grant BB/P027784/1]. This research was conducted under COSTECH research permit numbers: 2018-80-NA-2017-340 and 2019-100-NA-2017-46.

Notes on contributors

M.F. Gaworek-Michalczenia

M.F. Gaworek-Michalczenia is a postgraduate researcher at the Sustainability Research Institute of the School of Earth and Environment at the University of Leeds in the United Kingdom. She is also a member of the ESRC Centre for Climate Change Economics and Policy (CCCEP) and an Associate of the Priestley International Centre for Climate. Her work focuses on interdisciplinary research on climate change adaptation and the implications of climate compatible development projects on vulnerability and livelihood resilience of communities in developing countries. She holds MSc in Anthropology, Environment and Development from University College London, 2015 and MSc in Environmental Biology, 2008 from the Jagiellonian University in Krakow. Prior to undertaking her PhD, she worked for the Polish Academy of Sciences (2007–2008), the Zoological Society of London (2010–2013) and has carried out anthropological research on conservation and development issues in Guyana (2015) (https://www.cccep.ac.uk/profile/marta-garowek-michalczenia/).

S.M. Sallu

S.M. Sallu is an Associate Professor in Environment and Development at the University of Leeds where she specialises in interdisciplinary scholarship and teaching on human-environment interactions and more specifically on rural livelihoods, environmental change, marginalisation and natural resource governance. Her main interest lies in the theoretical and empirical intersections of political ecology, complex systems science and environmental justice. Susannah and her research team have particular regional expertise in Africa and work together in participatory ways with communities and a wide range of academic and non-academic partners to develop a better understanding of:

• dynamics, resilience and vulnerability of social-ecological systems, landscapes, and livelihoods

• impacts on human well-being and livelihoods of differing modes of conservation and ecosystem services governance and management, and climate change

• climate change adaptation and development

• climate compatible development intervention and policy.

M. Di Gregorio

M. Di Gregorio is an environmental social scientist and Associate Professor in Environmental Politics and Governance at the Sustainability Research Institute of the School of Earth and Environment at the University of Leeds. Her research examines contentious environmental politics and development and is cantred on land-use and climate change politics and policy, and natural resource governance more broadly. Much of her work relates to the agricultural-forest interface in developing countries, policy processes on reducing emissions from deforestation and forest degradation (REDD+) and synergies between climate change mitigation and adaptation in the land- use sector. She is an Associate at the Centre for Climate Change Economics and Policy and at the Priestley International Centre for Climate.

N. Doggart

N. Doggart is a conservation practitioner and research scientist with an interest in solutions to the environmental and social challenges associated with tropical deforestation. Through her work with the Tanzania Forest Conservation Group (www.tfcg.org), she has provided technical advice on conservation and development projects supporting participatory forest management, climate change adaptation and piloting REDD+, sustainable charcoal, sustainable timber and payments for water ecosystem services. She has also participated in tropical forest biodiversity research projects in Tanzania, Guatemala, Belize and Indonesia.

J. Mbogo

J. Mbogo is a conservation and development practitioner and a project monitoring and evaluation specialist. She graduated in 2008 from the University of Dar es Salaam in Tanzania with a BSc in Geography and Environmental Studies. She provided monitoring and evaluation expertise for a variety of non-governmental organisations on different multistakeholder development projects in East Africa. Some of these included: the Global Climate Change Alliance's climate change adaptation and resilience project in the East Usambara Mountains – worked for the Tanzania Forest Conservation Group (2016-2019), the USAID Tanzania Staple Value Chain Project (NAFAKA) – worked for WWF (2011-2015). She has also participated in research projects conducted by the University of Dar es Salaam and the International Institute of Tropical Agriculture (IITA) on forest restoration and integrated water, forest and land resources management.

Appendix

Figure A1. Breakdown of different income sources of participant and non-participant households before and after the intervention.

Figure A2. Climate-smart agricultural technologies practiced on private farms by participant and non-participant households before and after the intervention.