Microfinance trials on trial

ABSTRACT

Microfinance has attracted great attention, stimulated, initially, by its association with the award of the Nobel Prize for Peace, based on its transformative potential in addressing poverty. However, work based on randomised trials associated with the award of the Memorial Prize in Economic Sciences, put the brakes on. As health economists, and given their promotion as a ‘gold standard’ method borrowed from medicine, we take a particular perspective on the microfinance trials. We question whether full account had been taken of methodological developments in the health arena that were in place before the microfinance trials were embarked upon. This may help explain the outcomes of research, subsequent to the trials, casting doubt on their initial results, but also aid calls for even greater attention to be paid to developments in health evaluations – many drawing from social science more broadly – to better explain what works for whom in which circumstances.

JEL CLASSIFICATION:

• C93
• G21
• I18

Introduction

The use of randomised controlled trials (RCTs) in development economics and, particularly in the microfinance arena, has been lauded as the best way of assessing impact. Since the first randomised evaluation of microlending took place (Banerjee, Karlan, et al., 2015), studies have proliferated, evaluating not only the impact of different microcredit contracts but also several other related financial products and credit-plus programmes (Ashraf et al., 2010; A. Banerjee & Duflo, 2011, 2015b; Field et al., 2012, 2013; Giné & Karlan, 2014; Jan et al., 2011; D. Karlan & Zinman, 2010, 2011; D. S. Karlan & Zinman, 2008; Kim et al., 2007; Pronyk et al., 2006; Ssewamala et al., 2010).

The growth of RCTs was, in part, a reaction to concerns around the quality of initial microcredit impact evaluations, specifically, in disentangling causation from correlation (Deaton, 2010; Roodman & Morduch, 2013). Also noted was a need for more rigorous studies that are suitably designed to correspond to their particular setting (Armendariz & Morduch, 2010; Hulme, 2000). The proliferation of RCTs reflected a growing trend in other areas of development economics research where evaluations of social interventions seek to replicate methods previously thought to be largely the domain of the medical field. Indeed, RCTs were promoted as the ‘cleanest way’ to evaluate the effectiveness of social and development interventions (A. Banerjee & Duflo, 2011), based on the belief they offer the best way to identify what is responsible for an observed effect (the attribution problem) and for making sure the observed effect is not dependent on characteristics of the borrowers (i.e. selection biases). As a result, many disciplines place the RCT at the top of their hierarchies of evidence (Bédécarrats et al., 2020). Based on such assertions, subsequent high-profile microcredit RCTs that found modest impacts have been interpreted as casting doubt on its transformative effect. Modestly positive effects were found across different indicators, such as stabilising income, increasing profits of pre-existing business and increasing happiness. Importantly, the impact of microcredit is not the same for all borrowers.

In this paper, we question whether the emphasis on RCTs over these past 15 years has been worth it. This has been stimulated by two things. First, at the time RCT evidence was beginning to emerge, as health economists, we were puzzled by the attention given to such evidence in this field. This is because, at that time, medicine itself had moved on, and indeed had been doing so for a while (Grossman & Mackenzie, 2005). The UK’s Medical Research Council had published its MRC Framework for Developing and Evaluating Complex Interventions, subsequently updated twice (Campbell et al., 2000; Craig et al., 2008; Skivington et al., 2021), challenging the status of RCTs as some sort of gold standard for the evaluation of complex interventions. We would argue that microfinance is best evaluated through a ‘complexity’ lens, as an ‘event’ in a complex system displaying feedback loops, non-linear effects, adaptation and ultimately, outcomes that cannot be fully explained by any one component, or even the sum of many individual components (Höhn et al., 2023). In this context, the MRC Framework encourages the use of an array of methods to explore the wide range of potential outcomes and the pathways by which they are achieved. Second, subsequent research, using methods promoted by the MRC, indicates that the RCTs may have underestimated the impact of microfinance (Breza & Kinnan, 2021; Dahal & Fiala, 2020).

Although other commentators have called into question RCTs versus other statistical and econometric approaches – note the famous debate in development (and health) over evidence with respect to deworming in schools (Allen & Parker, 2016; Miguel & Kremer, 2004) – we add to this literature by reviewing the major microcredit trials from the perspective of the MRC Guidelines along with the latest set of MRC guidance on global health randomised trials more generally (MRC Clinical Trials Series, 2017), asking ‘what would be done in medicine?’ and ‘did the microfinance trials follow this?’. Despite the ‘groundhog’ qualities of again questioning the role of randomised trials in development, this is important given that borrowing methods from medicine was seen as a major justification for the trials and in influencing the debate as to what are the best ways forward in evidencing microfinance. It also raises important ethical questions. Given that experimenting on patients and communities is expensive and ethically questionable, it is important to ask whether everything was done in the preparation, and conduct of the microfinance RCTs to ensure that the human rights of study participants were not compromised and that the research represented value for money. Finally, following the latest iteration of the MRC Framework, we suggest alternative methodologies for evaluating complex, community-based initiatives, such as microcredit.

It is important to stress that we are not challenging the emphasis on robust evaluation embodied in the call for RCTs of microfinance. It is arguably unethical to implement interventions that have not been evaluated rigorously and the call for RCTs of microfinance represents an attempt to address that particular ethical issue. However, in this paper, we challenge whether the RCT was the best solution to that problem, given the complex nature of microfinance as an intervention and given other developments in evaluation over the past 15 years or so.

Recent evidence and review of the trials

New evidence has emerged recently, from rigorous systematic reviewing and observational studies, that the RCTs may have underestimated the impact of microcredit (Breza & Kinnan, 2021; Dahal & Fiala, 2020). This evidence shows that all eight trials were underpowered (Dahal & Fiala, 2020). Thus, although effect sizes are large, they are often insignificant, a result that is altered (for business profits, business revenue and household assets) when analysing pooled data (Dahal & Fiala, 2020). A large quasi-experiment based on credit supply withdrawals in Andhra Pradesh in 2010, subsequent to the microfinance crisis there, showed that microfinance is associated with both business growth and expansion in aggregate demand in the economy (Breza & Kinnan, 2021). Interestingly, the use of systematic reviews in microfinance evaluations was called for by health researchers McHugh et al. (2017), whilst consideration of the use of such observational (quasi-experimental) studies has been recommended, since 2000, by the MRC Framework in the sphere of evaluating complex interventions, especially that of community-based public health initiatives, which, as we have indicated, share many characteristics with microfinance.

The extension of evaluations of microfinance to include potential effects on health and well-being poses further questions about the appropriateness of RCTs as the sole or preferred evaluation method. Community-level interventions of this type often have large population-level health effects but small individual-level health effects. This makes it difficult to detect such effects in RCTs without extremely large sample sizes, and it increases the likelihood that systematic reviews limited to such studies will find no (statistically) significant findings even if effects that are important in public health terms were observed (Fischer et al., 2013; Threlfall et al., 2015). In these circumstances, the use of a variety of sources of evidence alongside theory provides a better basis for judgements about whether an intervention has ‘worked’ than sole reliance on one or more RCTs.

Pre-dating the MRC Framework, a claimed reason for considering observational studies as an alternative to RCTs of community-based initiatives is partly to do with feasibility (Craig et al., 2012). The conduct of a randomised trial in many such situations is not possible. It is practical in more clinical settings where it is easier, for example, to ensure the capture of all relevant patient groups. But, more fundamentally, the case for observational studies also rests on the issue of generalisability. The advantage of observational studies is that they can examine impacts in more realistic settings (Craig et al., 2012). They do not require the setup of randomised experiments, which, by definition, control aspects of the delivery of the intervention, for example, pathways followed by users of microfinance or participant selection, but in doing so achieve ‘internal validity’ at the expense of being less realistic and giving a less valid picture of what would happen if the intervention were put into practice. Generalisability is the first criterion against which we assess the body of RCTs – see Table 1. Of course, randomised trials, too, can be set up pragmatically in order to be more generalisable than often characterised, but, as evidenced by the microfinance trials, this is often more challenging in social than clinical settings (Hawe & Shiell, 2004).

Table 1. Where might the microfinance trials have gone wrong?

Key factors in preparing for a trial in ‘medicine’	What would medicine say?	Augsburg et al. (2015)	Tarozzi et al. (2015)	Banerjee, Karlan, et al. (2015)	Angelucci et al. (2015)	Attanasio et al. (2015)	Crepon et al. (2015)	D. Karlan and Zinman (2011)
Generalisability Maybe make this representativeness and move it down the list) (MRC Trials) (MRC Complex Interventions)	Usually, a representative patient group can be captured in the clinical setting. If inclusion criteria are too restrictive, … . with little by way of cross-contamination. Quasi-experimental studies are also recommended, especially for assessing longer-term impacts in the community.	X	X	X (again recruited ‘at the margin’, people of lower interest to provider)	X	Y (for those pre-disposed)	Y (for those pre-disposed, which is perhaps what we want to know about?)	X (only generalisable to marginal borrowers)
Community engagement (MRC Trials) (MRC Complex Interventions)	Investigators should seek to establish relationships with communities from which study subjects will be drawn to discuss various aspects of study design, implementation and communication	? (Worked with provider but not community)	? (Discussed with provider but not community)	? (Discussed with provider but not community)	X	? (Partly – as targeted loans at those who had expressed an interest and explained probabilistic access)	? (Discussed with provider but not community)	? (Worked with provider on credit-scoring algorithm, but not with community)
Uptake and drop-out (MRC Complex Interventions)	Uptake, drop-out, and thus capacity to achieve required samples sizes, would be assessed in a prior feasibility study.	Y (pilot mentioned but not reported on. Good reporting of [low] attrition)	? (Reasonable uptake in experimental group but different panels of ‘before’ and ‘after’ groups, but no feasibility study)	? (Ditto, used estimates from provider which proved optimistic)	X	X	Y (reported on a pilot)	? (Tried to mitigate with credit scoring, but still no pilot as such)
Consent (MRC Trials) (MRC Complex Interventions)	Consent would be a requirement, ethically and practically, and, again, would be assessed in a prior feasibility study.	Y (not explicitly mentioned but individuals knew they only had a probability of receipt of a loan)	X	X	X	Y (interested people could sign up)	X	X
Equipoise (MRC Trials, but don’t cover it explicitly. More now about ensuring benefits outweigh the risks)	It is unethical to conduct an RCT without being in equipoise. (MRC – consider alternative designs)	Y (seems to have been maintained as indicated by analyses of control group access to other sources of loans)	X (clear uptake in control group)	X (clear uptake in control group)	X	X (microfinance already available)	Y (in sense that microcredit did not exist in areas covered by the study)	Y in sense that uptake in control group was low)
Exclusion	If, for some reason, people are excluded from RCTs in medicine, it is challenging to leak into the intervention group as access is controlled by clinicians.	Y (ditto)	Y	Y (in sense of being recognised by study team)	Y		Y (covered this issue well, showing borrowers were those pre-disposed)	Y (seemed to manage this as indicated by low uptake in control group)
Comparators (MRC Trials)	In medicine, the comparator against which a new intervention is assessed, is normally best practice. This helps with ethics, too, as we do not know if the new treatment is better and also, for those not getting the new treatment, they are receiving best (known) practice.	? (unclear as with all studies, but good on ‘what would otherwise happen?’)	?	X (acknowledged that this is messy and was not anticipated)	?	Y (strong on comparing group with individual lending, but comparisons are unclear wrt no lending group)	Y (but only in sense that ‘control’ was largely areas where there was no microcredit)	? (other borrowing is documented, but authors unsure of data and control ‘picture’ is unclear)

The remaining criteria, although presented separately, are related to each other, and also to the issue of generalisability. They come not only from the MRC Framework on complex interventions but also that on randomised trials more generally (MRC Clinical Trials Series, 2017). The assessments we have made come from reviewing seven of the eight trials reviewed by Dahal and Fiala (2020) – those of D. Karlan and Zinman (2011), Banerjee, Duflo, et al. (2015), Augsburg et al. (2015), Tarozzi et al. (2015), Angelucci et al. (2015), Attanasio et al. (2015), Crepon et al. (2015) – as there was one which is unpublished (Fiala, 2013). Details of our assessment of each study against these criteria are documented in Table 1. Overall, our argument is that failure to account for such aspects of study design and conduct reflects a naïve adoption of RCTs from medicine as a gold standard. This is due not only to the differing context in which such trials are possible in the medical field but also reflects the fact that medicine has moved on methodologically – and, indeed, as evidenced by the MRC Framework, had done so by the time the microcredit trials were beginning.

For example, much work in medicine, often accompanied by substantial amounts of research funds, goes into community engagement and assessing the feasibility of trials. We would contend that this has been problematic, in several ways, with respect to the microcredit trials. The uptake of microcredit has not been as expected in many ‘intervention’ areas (Dahal & Fiala, 2020). In a way, this is a result in and of itself, and perhaps shows how important it is to get one’s research question tightly specified; is it the case that many studies have actually evaluated making microcredit more available rather than the uptake of microcredit itself? Nevertheless, in the case of the latter, this dampens the estimated effect of microcredit, especially when the unit of randomisation is the village or some other geographic area. In medicine, the patient or population group to be randomised is often more-easily identifiable, captured and controlled. Drop-out, either at recruitment stage or follow-up is, therefore, harder. However, it can happen, and whether it occurs and to what extent would be assessed in a feasibility study. Another way of dealing with this is through establishment of consent to participate in said experiment. Of course, in medical experiments this would be a prerequisite from an ethical perspective too. It also relies on equipoise whereby both patient and clinician are genuinely uncertain as to which treatment option (usually current practice versus some new form of care) is better. Otherwise, it would be unethical to randomise to a control group from which a beneficial or at least a preferred option is withheld. One would have to question, from details provided in the published literature, the degree to which such key factors were accounted for in the microfinance trials. It is worth noting here that the use of a ‘X’ or ‘?’ in Table 1 does not mean that such factors were not accounted for, but rather are not reported in the published paper (X) or is unclear (?). The proliferation of such symbols in Table 1 does, however, help explain the lack of participation and also raises ethical questions about whether such trials should have been undertaken. In medical feasibility studies, any future trial would be abandoned if uptake by, or consent of, patients was insufficient.

Related to these issues, it is also not clear whether many such studies accounted for the negative psychological impact of not being permitted access to microcredit and how this has impacted on leakage of study recruits, initially allocated to control groups, into intervention groups. Again, this challenge is related to the lack of equipoise and, also, to people’s rights. In this context, if we are unclear exactly what it is we are uncertain about, we may be unethically restricting the right to access credit, and communities will react to that, which may then lead to further restrictions on whom to include in a trial. Equipoise is another issue which has attracted recent attention, both in the field of public health, but also amongst economists (Abramowicz & Szafarz, 2020; McCartney et al., 2021). Its lack of consideration in study design can lead to trials being conducted in very particular circumstances (e.g. amongst those who have just missed out on being extended a loan). The trial takes place but, taking us full circle, is then severely limited in terms of generalisability. Will policy-makers and loan providers pay much attention to studies of people already rejected, by standard conventions, for a loan? Comparators, too, may also not be of the sort expected by a reader. In medicine, the control group is normally allocated to current (known) best practice. Do we know what this was in the case of the microcredit RCTs? Likely, study participants accessed other forms of (micro) loan which would further dampen the ability to find differences between groups, taking us on another full circle back to thinking through the rationale for a controlled experiment in such contexts. The quality of such alternatives would likely also be variable at best, which, again, raises ethical concerns over involving people in studies in which some alternatives may be sub-standard. To reiterate, in medical research most of these issues are dealt with via prior, and often quite extensive, feasibility studies of the sort which have never been undertaken for microcredit trials.

Where now?

It is important to recognise that the above critique does not necessarily apply to all randomised trials within the microfinance sphere. Trials of initiatives attempting to build business skills development or health initiatives into (or on top of) existing microfinance schemes are unlikely to be subject to the same flaws (Bulte et al., 2017; Hamad et al., 2011). Indeed, even for those reviewed, some of the aspects that we have highlighted may have been covered but not reported. Furthermore, we are obviously not the only authors to have suggested reasons as to why the microfinance trials may have underestimated impact, having potentially missed important aspects of context across geographic settings and evidence from other disciplinary approaches, such as ethnography (Bédécarrats et al., 2020; Kabeer, 2018; Kar, 2020).

Nevertheless, given that our focus (and, hopefully, contribution) has been on the limitations of not fully adopting recommendations from medicine, what would we recommend? First, we would commend the use of systematic review techniques (Dahal & Fiala, 2020) that first called for by McHugh et al. (2017). Second, an obvious recommendation from above is for more feasibility studies to be conducted in advance of proposed randomised trials. Amongst other things, through providing evidence on intervention uptake and drop-out rates, more accurate estimates of required sample sizes can be made should a proposed trial still go ahead.

Such feasibility studies might also point to other potential research designs more suited to the target populations of microcredit and microfinance programmes. Randomised trials are often silent about how and why an effect has occurred (commonly referred to as a ‘black box’ approach to evaluation). Initially, MRC recommended process evaluations, using qualitative methods to explain why and how interventions worked. This was an important step forward, but even these studies still took place within the experimental model. The problem is that the experimental model on which randomised trials are based tell us what works on average when a singular intervention (or initiative) has been imposed on a population (or group of patients), with little or no regard to context or the mechanisms which lead to outcomes, especially different outcomes for different groups. From a public policy perspective, knowing if something works on average might not be problematic; such data can be used to assess aggregate effects and aid judgments about allocating limited resources so as to achieve the greatest good for the greatest number. But, such results do not then reflect the needs of clinicians, and other policy-makers, to know what works for whom in which circumstances, so that measures can be taken to mitigate bad effects and help practitioners identify aspects of delivery that will help ensure the good effects are observed for as many as possible. Reflection of such decision-making needs has now developed, in health research, into recommending realist approaches, originally propounded by Pawson and Tilley (1997). Presumably, process evaluations and realist approaches have never been used alongside, or instead of, microcredit trials. More recently, the MRC Framework has been extended to include realist approaches which allow us to unpack the role of context and different sources of variability in outcomes (Skivington et al., 2021). Going in the opposite direction to the randomistas, health researchers are borrowing financial diaries methods from microcredit research to gather in-depth information on people’s financial lives in relatively deprived settings and relating this to their health trajectories (Biosca et al., 2024; Ibrahim et al., 2021). These directions of travel are indicative of recent reflections, even within economics, advocating employment of more mindful combinations of different forms of evidence (A. V. Banerjee, 2020), and the need to, still, keep reinforcing such messages within the field of alleviating health inequalities (Kelly-Irving et al., 2022).

Conclusions

Just as the miracle of microcredit is something that requires a balanced assessment, so, too, has been the wholesale adoption of RCT evidence as some sort of gold standard against which to assess its impact. Of course, the trials were a reaction to concerns about the perceived quality of initial microcredit impact evaluations and the ethical, validity and value for money implications of poorly executed research. But the same standards apply to the trials themselves. Although seen as raising the standard of evaluation in development to that in medicine, just as the trials were being embarked upon, the medical field had moved on. This was related not only to the practicality of conducting RCTs in complex, community-based settings but also to the relevance and generalisability of such study designs. Failure to recognise the limitations of RCTs has led to under-estimated (and, perhaps, missed) impacts as well as a lack of understanding about how impact occurs, raising major questions about their expense and – through recruitment of participants to badly designed studies through which access, for some, to needed services is restricted – their ethical basis.

With medicine moving on yet again, in part towards more realist approaches for evaluating complex community-based interventions of the sort that microcredit comprises, we are in danger once again, of falling behind in our approaches to establishing answers to the real questions, not of whether microcredit works on average, but rather of whether, and how, it works (and does not work) for particular people in particular contexts. Perhaps the latest MRC Framework, summarised by Skivington et al. (2021), offers the best selection of methods to be employed in the next phase of microcredit impact research.

Disclosure statement

No potential conflict of interest was reported by the author(s).