ABSTRACT
Impact investors and development finance institutions are starting to proactively examine gendered impacts to ensure their investments progress the opportunities available to women instead of reproducing existing inequalities. In October 2020, we trialled the use of the Qualitative Impact Protocol (QuIP) method to measure empowerment changes created by an investment into a cotton company in Uganda. The QuIP method is a qualitative approach to impact evaluation which assesses whether an investment, is achieving its intended impact. We found that the method worked with the impact investing operating model, required minimal input from the investee company, reduced response bias, and addressed contribution without the need for a baseline. The trial generated lessons on investee selection, geographical scope and blindfolding which can improve the use of this method for impact investing. This trial has confirmed the value of a method that other investors can now consider when measuring the gendered impact of their work.
1. Introduction
Literature on conceptualising economic empowerment is still largely based on Amartya Sen’s (Citation1985, Citation1999) concept of development as the freedom to make choices. So, developing out of poverty is to increase the freedom you have, not the number of resources you have. Kabeer (Citation1999) used this capability approach as the basis for her definition of empowerment as a multi-dimensional process of improving access, agency, and wellbeing outcomes. The definition of women’s economic empowerment (WEE) we use in this paper is based on Kabeer’s work; economic empowerment is the process of changing men and women’s ability, as an individual and a member of a household and community, to participate in, contribute to, and benefit from economic opportunities.
The debate around measuring economic empowerment has developed and expanded since 1999, with the current emphasis on going beyond advancement indicators like ‘securing a loan’, or proxies like ‘spending patterns’ (Buvinic Citation2017). The discourse grapples with three substantial challenges at its core. Capturing the subjective dimension of empowerment relies heavily on self-reporting, which brings a set of biases that are expensive and challenging to overcome. There is an acknowledgment that empowerment is context-specific, and therefore, measurement of empowerment has to attempt to use localised proxies, outcomes, and goals. Then there is the complexity of measuring a change that is multi-causal and takes place at an individual, household, and community level.
Currently, the most commonly used solutions to measuring economic empowerment are survey-based indices, psychometric tests (for example, stress levels), capturing changes in decision making power, participatory research, and varying semi-structured interviews with qualitative coding. However, recent studies have exposed how these solutions are all challenged by the fact that empowerment is immensely contextual (Martínez-Restrepo and Ramos-Jaimes Citation2016).
The challenge is exacerbated when trying to measure the WEE generated by investments. Impact investors operate on a commercial basis, and therefore, typically have small impact measurement budgets (relative to direct aid or research organisations). Most of the commonly used methods cited above are costly. Additionally, the engagement with the investee businesses needs to be managed in order to maintain the commercial nature of that relationship. It is important to impact investors to keep the burden of engagement on these investee companies low. They are often under-resourced, and investors try to preserve the commercial nature of the relationship, as opposed to the relationship between funder and grantee. Even before we get to monitoring or assessing impact, there is already investee fatigue from the expectations most impact investors place on early-stage businesses to meet the international standards expected from the impact investors themselves.
A bigger challenge is how the model of impact investing limits what research designs are feasible. For example, baselines for quasi-experimental or panel research are rarely possible because companies for the most part are already operating prior to the investment. This means that by the time capital is committed, the chance to make pre-treatment measurements is complicated and testing the parallel trend assumption is almost impossible. Getting around this by moving research ahead of the deal-making process is not advisable due to the unpredictable length of the deal process and the risk that the deal falls away and the resources are wasted. All the usual challenges of applying experimental (and quasi-experimental) research methods to complex market-based interventions also apply to impact investing (Befani, Ramalingam, and Stern Citation2015).
This article explains how an impact investor, AgDevCo, trialed a specific qualitative contribution approach to measuring WEE. The first section provides background to the investor and the method used. The second section gives detail on this specific piece of research and the methodology. The findings are shared in the third section, and the article finishes with a discussion of how well the method worked in measuring WEE in the context of impact investing.
2. Background on the research
In the context of these challenges, AgDevCo has been trying out different solutions to assess its impact with enough confidence so they can use the findings for accountability commitments and internal learning.
AgDevCo is an impact investor supporting the growth of sustainable and impactful agribusinesses in sub-Saharan Africa. The company provides more than just capital; they are long-term partners with a focus on capacity building and technical support. AgDevCo’s primary aim is building successful African agribusinesses through long-term investment and support to deliver positive impact at scale. Agricultural development is the primary purpose of its investments, but AgDevCo believes that gender equality and women’s empowerment are significant factors in the commercial success of its investments and the quality of the development impact those investments deliver. AgDevCo’s intention is to make investments which, at a minimum, do no harm and safeguard both women and men. All of their investments are required to meet mandatory safeguarding criteria. Where the opportunity exists, they go beyond these minimum requirements to create beneficial outcomes which empower women.
AgDevCo measures the changes in impact and gender equality that its investments and technical support create through annual investee reporting and impact assessments. Annually, AgDevCo looks at the key themes, potential risks, and data gaps in its portfolio to select research topics for impact assessments, although these only cover a small sample (10–15%) of the portfolio. With small budgets to navigate, AgDevCo has tried to measure women’s economic empowerment with Women’s Empowerment in Agriculture Index (WEAI) surveys, asking about decision-making power in structured interviews and focus group discussions and network analysis, with varying results. Most importantly, WEAI surveys primarily captured changes in women’s advancement but did not capture changes in empowerment such as confidence, social capital, and wellbeing. In 2020, AgDevCo trialled using the QuIP method to capture these changes.
2.1. The quip methodology
The Qualitative Impact Protocol (QuIP) is a qualitative approach to impact evaluation which assesses whether a project, intervention, or investment, is achieving its intended impact. QuIP seeks to provide a reality check for organisations and investors to understand how and why change is occurring from the perspective of intended beneficiaries. Independent researchers collect narrative causal data from stakeholders through semi-structured individual interviews and focus group discussions. The questionnaire guide covers predetermined ‘domains’ (specific areas of a person’s life, e.g. farming, health, or wellbeing) relevant to the theory of change of the intervention being researched and based on the areas of beneficiaries’ lives or livelihoods where the programme expects to have achieved some impact. In each domain, researchers pose open-ended questions asking respondents to reflect broadly on whether anything has changed during a pre-defined recall period and to share what they perceive to be the main factors influencing or driving any reported changes. In order to mitigate potential response biases, such as confirmation or pro-project bias, researchers and respondents are usually both ‘blindfolded’ to some extent. Where possible, interviewers do not know the identity of the organisation being evaluated or the specifics of programme implementation.
Once the data have been collected, an unblindfolded analyst qualitatively codes causal claims within the transcripts and then analyses the trends and patterns across the dataset to understand the relationships between causal factors and to reveal whether there are any differences across different groups of respondents. The stories, experiences, and perceptions of respondents can then be compared to the programme’s theory of change, alongside other forms of evidence or monitoring data.
QuIP coding is done in a very structured and systematic way using set guidelines and conventions. Part of this process involves discussions with other researchers for feedback. The software Bath SDR, the non-profit consultancy that promotes standards in applying the QuIP method, uses to code QuIP data is designed to make qualitative findings more transparent, allowing for frequent reviews and quality assurance.
The QuIP was originally developed through a three-year action research project, led by researchers at the University of Bath. It has since been applied in a wide range of countries, contexts, and thematic areas, including WEE (Copestake, Morsink, and Remnant Citation2019). There have been a few QuIP studies specifically focused on WEE, including a recent evaluation of a WEE programme which combined group-based microfinance with agricultural and financial literacy training in rural Ghana. However, more generally, many QuIP studies have included domains related to economic empowerment, such as income, budgeting and financial planning, spending, asset ownership, and saving. In fact, most QuIP questionnaires have included sections on household and community relationships, wellbeing, and aspirations for the future. These questions often elicit detailed and nuanced narratives about gender dynamics at home and in the wider community, in terms of roles and responsibilities, decision making, and leadership.
One particular advantage of using this open-ended approach to research empowerment is that it allows respondents to share what is most important to them and increases the likelihood of uncovering any unintended outcomes, whether positive or negative. For example, the Ghana QuIP study found that women involved in the WEE programme were reporting empowerment-related outcomes which had not been anticipated or included in the theory of change. This style of interviewing, particularly when blindfolding is used, also enables respondents to report unexpected drivers of change, therefore, providing a much broader context of how a particular programme intervention may have influenced change alongside other factors. This is pertinent to understanding the causal pathways to empowerment as there are often multiple, complex, and interrelating factors influencing change. Another benefit of the qualitative aspect of this methodology is that the data can provide in-depth insights about changes in attitudes, as well as behaviours, which is especially important for what can be a very nuanced and gradual area of change.
3. Case study: Gulu Agriculture Development Company (GADC)
From 1986 to 2006, the war in northern Uganda hampered growth in the region. The conflict caused displacement, theft, and poor health, which significantly reduced production. Within this context, Gulu Agricultural Development Company (GADC) revived an old cotton ginnery in Gulu in 2009 and now processes cotton and sesame bought from small-scale farmers.
Whilst the region has had high growth rates since the end of the war, this has not been accompanied by a significant increase in opportunities for formal employment, especially not for women. Women are primarily stuck in low-productivity subsistence agriculture and the informal sector. GADC works to change the opportunities available, and in 2019 employed 450 men and 270 women from the local region.
AgDevCo provides GADC with a revolving working capital facility (alongside other lenders) and technical assistance to support GADC to employ men and women from Northern Uganda as well as provide small-scale farmers with access to secure premium markets and extension services. AgDevCo has worked with GADC to improve its human resources policies, maternity leave packages, financial management, and health and safety practices. AgDevCo also trains some of GADC’s employees on how to deliver training on good agricultural practices to small-scale farmers.
In 2020, AgDevCo decided to use the QuIP method to test a key hypothesis which underlies their portfolio theory of change: when rural women from low-income households get formal jobs for the first time, their economic empowerment is improved (the formal employment hypothesis) (Said-Allsopp and Tallontire Citation2014).
Data collection was carried out between November and December 2020 in Kitgum and Gulu state, Uganda. Special precautions were made to minimise the risk of exposure to or spreading of Covid-19. Information was gathered directly from women employed by GADC through open-ended semi-structured interviews and focus group discussions. The focus of the research was to understand their perceptions of what had changed in their lives over the last three years. We specifically looked at five domains of economic empowerment assumed in AgDevCo’s formal employment hypothesis.
These domains were:
human capital (professional development, knowledge, skills, and confidence in the workplace);
social capital (relationships with managers and co-workers);
financial capital (income, spending, saving, and resilience);
household relationships (including how decisions and labour are shared); and
wellbeing.
At the end of the questionnaire, respondents were also given an opportunity to share stories of change specifically related to the Covid-19 pandemic.
Interviews were conducted by a team of local researchers who were not aware that the research had been commissioned by AgDevCo, nor that the company involved was GADC. This ‘blindfolding’ is used where appropriate and relevant, to help reduce pro-investment or confirmation bias. In this case, only the enumerators were blindfolded. The Lead Researcher, who was managing the research training, quality assurance and logistics, was not blindfolded because they needed to interact with AgDevCo and GADC to organise the research.
The QuIP methodology typically uses a combination of purposive and then random sampling; in this case, snowball sampling was also used to help locate focus group participants. According to GADC’s current database, there are 104 formal female employees and 260 casual female workers. The QuIP does not intend to be representative of this whole population but seeks to conduct a ‘deep dive’ into the experiences and perceptions of some of these women to gain a better understanding of the causal processes at work in their lives and livelihoods. The research team conducted 36 individual interviews with female formal employees (Field Officers, Buying Agents, and Are Coordinators). This may limit how applicable the findings are to casual workers and the research team also held 4 focus group discussions with women and men formally employed by GADC in Gulu and Kitgum. A standard ‘single’ QuIP usually includes 24 respondents, but this was increased to 36 in this case to cover both geographical areas in greater depth (see and ). These districts were chosen (and the West Nile region was rejected) based on how long GADC had been operating in these areas.
Table 1. Individual interview sample.
Table 2. Focus group sample.
The individual interviews focused on women only, as this was the key population of interest. The focus group discussions were split evenly between men and women. These discussions provided an opportunity to delve deeper into the stories of change shared and to triangulate the findings from the individual interviews. The two focus groups were conducted with men to understand any differences in their experiences of how formal employment has affected their lives and livelihoods.
The intention was to split the sample evenly between Gulu and Kitgum head offices. However, the researchers could not interview all the participants on the list provided for Gulu state, so the numbers were made up in Kitgum (see ). As planned, the researchers facilitated one male and one female focus group per district (see ). There were ten participants present at each focus group discussion.
To ensure confidentiality, verbal consent was obtained from participants before interviewing them. Data from all respondents were anonymised using unique identification numbers assigned to each interview. Participants were invited to a meeting to validate the preliminary analysis and their feedback was incorporated in the final analysis. Respondents were interviewed at convenient times and locations of their choice and their privacy during the interviews was respected. Interviews were conducted in the local dialect spoken by the participants. The main risk was the exposure of information to the employers of the participants. This was overcome by anonymisation of data, restricted access to the raw data to only the data collectors and analysts and presentation of findings in aggregate form.
4. Findings
The research generated ‘stories of change’; information on what respondents think are the main changes in their lives and what they view to be the drivers of those changes. Through this research, we found that formal employment at GADC does contribute towards women’s empowerment. Female employees reported having more money, improved financial sustainability, and the capacity to manage their own budget. Formal employment instilled a sense of pride in the women, and the training GADC provided was particularly responsible for making them feel more confident and more connected to other people (especially colleagues). summarises the findings against each of the five domains of economic empowerment which the research focused on. The QuIP method was able to capture what the employees said had changed in their lives and the drivers that they believe caused those changes. For example, we saw that the domains of empowerment where change was the most prevalent (mentioned by everyone except one person) were confidence and income.
Table 3. Findings by domain.
To obtain this understanding of what changes are closest to universal through a semi-structured classic survey would involve a lot of questions and it would be difficult to avoid leading questions and the bias these create. With the QuIP methodology, you do not get the magnitude of changes, such as revenue, as we would in a statistical inference-based design. However, the QuIP research provided a richer understanding of why change happened and the detail of that change.
The research found it was the specifics of the role at GADC that changed women’s confidence. The women in Field Officer roles had to train large groups of cotton farmers whom they did not necessarily know. Over time, this activity improved women’s communication skills, which reduced their fear of speaking in public, and prompted them to share their opinions more often. The QuIP method picked up the specific causal mechanisms which increased the women’s confidence. This is useful because now we know not to assume that formal employment in any role will increase confidence, we know that it is the common tasks and responsibilities of a role that influence how and what change happens. The QuIP method overcame the context challenge of measuring women’s economic empowerment. The respondent chooses what to talk about in response to questions which can transcend most contexts, for example, ‘tell me about changes in your relationships in the past three years’.
As with a lot of qualitative methods, the QuIP method proved to be effective in unintended consequences. For example, we learned that GADC provides ad-hoc training and support on relationship advice, which the women said had a large influence on how their relationships with other members of their family had changed. The QuIP method was useful for identifying confounding variables, which a quantitative study would be unlikely to control for or capture. A lot of the respondents mentioned training they received from African Women Rising when talking about relationship changes and village savings and loans associations (VSLA) played a key role in the changes in income and savings. In this way, the QuIP method provided a rounded picture of the changes for these women. Similar to ‘most significant change’ approaches, the QuIP method finds out what matters most to people, which is hard to achieve in an index-based survey. Using the QuIP method, this research showed that capacity building (from training, information, and experience) was a strong element in all the stories of change and a large part of the causal paths towards the five different dimensions of empowerment we focused on. AgDevCo can use this insight to increase and encourage capacity building efforts at other companies in their portfolio.
The QuIP method showed us the different causal factors to which women attributed the changes in their lives. It did not, however, measure how each different investor in GADC contributed to these changes. The GADC employees interface directly with GADC, not its investors, and therefore, report the causal mechanisms as between their experience with the company and the rest of their lives. The QuIP method would need to be paired with another method in order to measure the different contribution of different investors to the changes women spoke about.
The QuIP method did not overcome the self-reporting bias challenge of measuring empowerment. The respondents will have been susceptible to social desirability or selective recall bias. A future option for AgDevCo would be to triangulate the QuIP findings with quantitative data collected through the investee, such as income changes, or insights through key informant interviews with community observers, such as religious group leaders, VSLA managers, or local government and medical officials.
The study cannot provide us with information on how far the findings can be generalised or on the precise magnitude of stated changes. That is not the intention of the QuIP method. Findings are useful to understand the key causal processes (which drivers are influencing change) and which outcomes are being observed. This information provides a ‘reality check’ for the existing theory of change. AgDevCo can use these data to reflect on their future investments. For example, the team may wish to continue or to develop activities which are reported as driving positive outcomes. Conversely, if there are unintended negative consequences, adjustments to programme delivery may need to be considered. Biradovola et al have developed very short (5 questions) surveys for measuring agency by applying machine learning statistics to identify the subset of close-ended questions most predictive of agency as collected by commonly used measurements (Biradavola, Cooper, and Jayachandran Citation2021). Pairing this type of short survey with the QuIP in the future could assist AgDevCo to understand the extent to which the changes observed through the QuIP are experienced by a wider population. To determine the size of changes in the future, AgDevCo could pair the QuIP method with quick and cheap quantitative data collection, such as phone surveys.
5. Discussion
AgDevCo used this small QuIP study as a test to see how well the methods work in the context of assessing the impact of impact investments. Overall, AgDevCo considered the trial a success, but there were some valuable lessons which can be taken forward to improve the quality of similar studies in the future.
There were several practical advantages of using the QuIP. As discussed above, minimising the burden on investees is key for impact investors, and this approach required very little investee input. All that was required was approval to carry out the discussions with their employees, a meeting to find out how the study could be useful for them, and compiling the names, numbers, and locations of its employees (though this is not always easy, as noted below). This is a lot less input than if we used an alternative qualitative method, such as process tracing or outcome mapping.
The method was also good value for money in comparison to other qualitative methods AgDevCo has tried, not least because the QuIP method provides a rigorous contribution analysis, which is not common in qualitative methods. The capacity and location of Bath SDR’s QuIP enumerators are also an advantage for AgDevCo. AgDevCo works in nine African countries, and often research companies take time to find enumerators in the right location with all the required skills. Bath SDR has trained enumerators in many of AgDevCo’s countries, which made things move very quickly and gave the investor confidence.
AgDevCo collects quantitative data from investees on the number of employees and their wages on an annual basis, so methods like this work to complement our rich quantitative data on economic advancement.
When interviewing stakeholders in an investee’s supply chain, it is challenging to overcome the bias from stakeholders’ contractual relationships with the investee. It is also important to avoid the discomfort stakeholders can feel when asked to discuss topics which could be perceived as a risk to their contractual relationship with the investee. Therefore, the blindfolding used in the QuIP was valuable for AgDevCo.
AgDevCo learnt a lot from this QuIP method pilot that will inform useful changes when they use the method again. The key lesson was around investee selection. AgDevCo invests in early-stage agricultural businesses, which commonly do not have fully formalised management information systems. It was a time burden on the business to collect the necessary data for the study. Next time, AgDevCo could work with a more administratively developed company that has this information already, so as to minimise the time burden on them and the delay to the study.
The main cost driver for the study was transport. The respondents were spread out in two regions of Northern Uganda, so the enumerators had to travel far between interviews and were limited on the number of interviews they could do a day. It would be more cost efficient to select an investee which has employees that all live in a small geographical area. These savings could be used to increase the sample size or add men to the sample.
Finally, by the end of the data collection, some of the enumerators were able to guess who had organised the study because the investee was mentioned by every respondent. To prevent this in the future, AgDevCo could interview the employees from two or more different investee companies that all live within one geographic area. This would dilute the ubiquity of one investee and strengthen the blindfolding.
These advantages make QuIP a good choice of method for impact investors. However, the impact investing industry currently focuses mainly on quantitative metrics. There is a lot of focus on establishing a lexicon and standardised metrics. Investors are working on getting the basic elements of impact measurement in place in a standardised way. Therefore, they would need to have additional resources and capacity to go beyond these industry-wide initiatives to appreciate the value of qualitative research using the QuIP and other methods.
6. Conclusion
This exercise was both a success and a learning curve for AgDevCo. This means that AgDevCo will use the QuIP research method again on different investments and has learned what changes to make to maximise the value of such future research. While this trial does not offer conclusive evidence that this method will be useful for all impact investors, it does show that there is value in using qualitative contribution approaches to understand the impact created by investments, especially on employees. This is significant for impact investors applying a gender lens to their portfolio. Much work has been done to develop mainly quantitative metrics for women’s advancement that are useful for gender-lens investing. This pilot shows that there is value in augmenting these metrics with qualitative assessments, and that it is feasible to use the QuIP method to do this. The results show that the QuIP method works in the context of investments, which is not the case for all research methods. Other methodologies available to conduct similar assessments could include outcome harvesting, the Most Significant Change Approach, or SenseMaker.
It would be useful to do future research to test how well the QuIP method works when paired with larger scale quantitative data collection or capture lessons from where this has already been done. This would provide investors with rich information on how and why changes happen as well as the data to understand to what extent that information can be generalised across a population of beneficiaries. There is the possibility that the improvements suggested off the back of this trial do not tackle the challenges faced. Therefore, AgDevCo should test the method with these improvements to take this work further.
The QuIP method as used by AgDevCo relied on in-person interaction, which could be a short-coming in the context of the Covid-19 pandemic. Some technology-based research companies have tried to apply the QuIP method over the phone to make it more suitable to mobility restrictions. More work could be done to see if these remote data collection methodologies can produce data with the same level of depth. Remote data collection often needs to be short, to keep people’s attention, and it is harder to develop rapport with respondents when not with them in-person.
Most importantly, understanding how gender lens investments affect women’s economic empowerment is crucial to learning what works and what does not – which is how improvements can be made.
Disclosure statement
Mollie Liesner works for AgDevCo Limited and Rebekah Avard works for Bath Social & Development Research Ltd. No potential conflict of interest was reported by Moses Mukuru.
References
- Befani, B., B. Ramalingam, and E. Stern. 2015. “Introduction – Towards Systemic Approaches to Evaluation and Impact.” IDS Bulletin 46 (1): 1–6.
- Biradavola, M., J. Cooper, and S. Jayachandran. 2021. A Five-Question Women’s Agency Index Created Using Machine Learning and Qualitative Interviews. Chennai: Institute for Financial Management and Research.
- Buvinic, M. 2017. Measuring Women’s Economic Empowerment: Overview. Data 2X. https://data2x.org/resource-center/measuring-womens-economic-empowerment-overview/.
- Copestake, J., M. Morsink, and F. Remnant. 2019. Attributing Development Impact: The Qualitative Impact Protocol Casebook. Rugby: Practical Action Publishing.
- Kabeer, N. 1999. “Resources, Agency, Achievements: Reflections on the Measurement of Women’s Empowerment.” Development and Change 30: 435–464. https://onlinelibrary.wiley.com/doi/abs/https://doi.org/10.1111/1467-7660.00125.
- Martínez-Restrepo, S., and L. Ramos-Jaimes, eds. 2017. Measuring Women’s Economic Empowerment: Critical Lessons from South America. Springfield, VA: IDRC.
- Said-Allsopp, M., and A. Tallontire. 2014. “Pathways to Empowerment: Dynamics of Women’s Participation in Global Value Chains.” Journal of Cleaner Production 107: 114–121.
- Sen, A. K. 1985. Commodities and Capabilities. Amsterdam: North Holland.
- Sen, A. 1999. Development as Freedom. New York: Alfred Knopf.