The varied perspectives of organisational effectiveness: What’s at stake for early childhood development programmes in Rwanda?

ABSTRACT

Within global health and development, dissatisfaction with nongovernmental organisations’ effectiveness (NGOs) is an increasingly pervasive aspect of programming. Today, the international community no longer accepts that NGOs are doing what they claim. This change in expectations has emphasised the importance of measuring organisational effectiveness for improved health and development impact. Using New Institutionalism as a theoretical framework, we investigated how institutional norms and expectations influence the adoption of structures and processes by NGOs, and Early Childhood Development (ECD) programming effectiveness in Rwanda – since little research connects these concepts. We employed qualitative methods: 45 in-depth interviews and 6 focus group discussions. Findings revealed a misalignment of ‘organizational effectiveness’ across scales, from global to local. Findings stress that, effectiveness, though an expectation of the institutional environment, may not be a valid construct for NGOs, generating implications for ECD programming. Findings also indicate measurement of global health interventions generally and the notion of effectiveness specifically can yield adverse implications for ECD programming. These findings are relevant for researchers and practitioners trying to better understand organisational effectiveness for ECD programmes because they suggest that effectiveness is socially constructed and measured differently across the different scales.

KEYWORDS:

• International nongovernmental organisations
• organisational effectiveness
• new institutionalism
• governance
• Rwanda

Sustainable Development Goals:

• SDG 3: good health and well-being
• SDG 4: quality education

Introduction

Global NGOs in health and development have proliferated in the last three decades. This is due to the rise in complex emergencies, growth of civil society and, from a neoliberal perspective, a need for alternatives to state regimes as a way of limiting government influence and increasing NGO legitimacy (Karns et al., 2015; O’Sullivan et al., 2016; Teegen et al., 2004). Moreover, NGOs have become critical to global health governance. We define governance as the interplay between organisations that engage with health and the embedding of normative expectations within institutions aiming to improve the health of citizens, worldwide (Youde, 2018; Zürn, 2018). NGOs are seen as crucial for the delivery of health interventions on the ground, policymaking, and the development of a global health research and practice agenda more broadly (Delisle et al., 2005; Doyle & Patel, 2008). This is particularly true as it relates to promoting needed early childhood development (ECD) interventions.

Global health and global governance scholars agree that the proliferation of NGOs has greatly expanded the ability of affected populations to receive the necessary attention. Yet, the challenges this proliferation has presented within global health governance are significant (Delisle et al., 2005.; Sidiropoulos et al., 2021; Youde, 2018) These challenges include coordination among other global governors (Moore et al., 2003); management of competing loyalties between beneficiaries, donors and organisational interests (Krause, 2014; Stoddard, 2003); accountability; and arguments around effectiveness: what it is, what it entails, and how it is measured (Panda, 2007). The literature debates suggest that, although largely unintentional, NGOs may be more concerned with their continued existence as organisations – and the need to be considered legitimate – rather than the effectiveness of their programmes.

Research in New Institutionalism (NI), a theory from organisational sociology, proposes that NGOs comply with normative environmental demands to secure legitimacy (DiMaggio & Powell, 1983; Meyer & Rowan, 1977). Evident in previous studies is that NGOs are embedded in environments that maintain cultural conventions and shape organisational goals and practices (Hasselbladh & Kallinikos, 2000). NI posits that NGOs are constituents of a wider, much more amorphous cultural domain than the originally envisioned self-contained and well-demarcated sector (Greenwood et al., 2017; Meyer & Rowan, 1977; Pope et al., 2018). This observation is important because it suggests that cultural forces are diffuse and penetrate organisations, leading to isomorphic pressures in which organisational leadership searches for trends of ‘best practices’ – or those things that organisations must do to be considered legitimate – all of which are primary tenets of NI (Pope et al., 2018).

Adoption of best practices is a result of mimetic isomorphism, as NGOs gain ‘legitimacy’, or that which is good and proper within an institutional environment (Suchman, 1995), typically by imitating other organisations (Claeye & Jackson, 2012; DiMaggio & Powell, 1983). Since legitimacy is specific to different institutional environments, the cultural forces that are considered good and proper are entirely contextual. Because NGOs do not operate in a vacuum, they are not immune to the cultural forces and face pressures to, as Pope et al. note ‘formalize their structures, professionalize their ranks, specify their goals, and measure and manage their resources’ (p. 1302). One such norm is the measurement of global health interventions, typically conducted through implementation-based frameworks for monitoring and evaluation (M&E) (Binnendijk, 2000; Xue et al., 2013; WHO, 2018). M&E systems were initially adopted by NGOs to capture transparency in programme design and demonstrate enhanced effectiveness of interventions for donors, a process which has become a normative expectation (Boehmer & Zaytsev, 2019; Kusek & Rist, 2004). Adoption of M&E systems has led to a project-based model where the achievement of interventions is linked to the planned work, as outlined in a logic model (Xue et al., 2013). Moreover, implementation-based M&E systems are used by NGOs to justify budgetary requests (Kusek & Rist, 2004).

Adherence to such cultural forces as M &E signals to funders that the organisation is committed to both progress and addressing issues of legitimacy (Abrahamson & Fairchild, 1999; Pope et al., 2018). This argument by some about expectations is crucial because it suggests that institutionalised norms shape not only how NGOs implement programmes, but also the personnel they choose to hire, and the governance structures they adopt. In contrast, some scholars argue that the adoption of cultural forces are performative so that outside actors confer legitimacy, but informal norms drive day-to-day practice of NGOs (Nee & Ingram, 1998). These informal norms can include how and why projects are chosen, designed, and implemented, as these norms exist outside formal governance structures. By implication, organisational legitimacy is about both formal governance structures and informal norms that are embedded in the different relationships throughout organisational networks. This is particularly true in Rwanda because some argue the technocratic approach to development ultimately politicises and institutionalises the expected behaviours of NGOs (Roberts, 2013).

By using New Institutionalism as a theoretical framework, this study explored how NGOs engage with formal and informal constraints on behaviour, and how they link their work from global to local scales, using Rwanda as a case study. New Institutionalism is well-suited for understanding the complex relationship between institutions and organisations: a relationship that ultimately shapes the implementation of ECD in Rwanda because of its emphasis on the institutionalisation of norms. For the purposes of this study, institutions are considered formal and informal norms that shape social relationships (Nee & Ingram, 1998). Early childhood development refers to the ‘sensory-motor, social emotional, and cognitive-language development changes … a child undergoes during their early years of life from conception to 6 years, as well as support that caregivers need to provide childcare’ (NECDP, 2018).

Rwanda and early childhood development programming

We chose Rwanda as the setting for this research for three reasons: the stable institutional environment, good progress on key health determinants, and the establishment of the ECD network. The Government of Rwanda is deeply committed to transition from an agriculture-based economy to one that is more knowledge-based and market-oriented (Kaberuka, 2000; Williams, 2017). At the core of this commitment is both the need for an improved private sector and an investment in the people of Rwanda through the development of economic and human capital. Therefore, post-genocide Rwanda has become a technocratic state that prioritises knowledge and expertise, diversification of the economy, and investment in human capital. Rwanda now has a strong focus on incorporating the global development agenda, like the Sustainable Development Goals (SDGs), into national priorities. This is in large part because these goals are clear, measurable and tend to focus on what Rwandans need in their daily lives (Roberts, 2013).

Over the last three decades, Rwanda has shown significant progress in many different domains. These include governance, human development, human rights and disaster management (UN, 2019). Some of the biggest gains are linked to health. For example, there has been a 77% reduction in maternal mortality; a 70% reduction in under five child mortality; and 93% of children 12-23 months are fully vaccinated- all of which are very relevant for ECD (PRB, 2018; WHO, 2015). Despite these advances, Rwanda remains one of the poorest and least-developed countries in the world, ranking 165 of 191 countries on the human development index (Abbott et al., 2017; UNDP, 2022). Additionally, 39% of the population of 12 million live in poverty (NISR, 2017; UNDP, 2018) and 38% of children experience stunting (NECDP, 2018 ). The Government of Rwanda understands that stunting undermines biopsychosocial development over the trajectory of a child’s life course. The Government has acknowledged that it will be unable to achieve its long-term development goals without reducing the stunting rate because of its adverse impact on human capital. Effective ECD interventions are especially important in Rwanda because of its collective memory of both suffering and rebuilding. Of note, 50% of the population is below the age of 18 and 15% are under the age of five (UNICEF, 2017). Moreover, with a fertility rate of 4.2 births per woman, that percentage of children is likely to increase (UNFPA, 2020 ). Indeed, in recognition of the stunting rate, the Ministry of Education (MINEDUC) and the Ministry of Gender and Family Promotion (MIGEPROF), began instituting ECD programmes throughout the country. This led to the creation of the National Early Childhood Development Program (NECDP) in 2017, a network that brings together various actors from different sectors to address ECD needs from a holistic perspective including elements of positive parenting, water, sanitation and hygiene (WASH), malnutrition, child protection, and education.

Throughout the early childhood development network in Rwanda, several structures and processes are considered best practices. These include formal governance structures as outlined by the Rwanda Governance Board, normative expectations on programme implementation for NGOs, as well as behavioural expectations, including interorganisational partnerships that engage in capacity-building activities between international NGOs (INGOs) and local NGOs (NECDP, 2018; RGB, 2016). Post-COVID, a 2022 household study conducted in Rwanda underscored the twin challenges of behaviour change and poverty limiting food security for families. The authors found that further behavioural changes are needed in families and communities to reduce child stunting; and that such changes are possible (Birungi et al., 2023 ), amplifying the importance of Rwanda’s ECD network and practices and the urgency of addressing child health in this setting.

Although the literature on NGO effectiveness is vast, it is also inconclusive, particularly as it relates to the relationship between legitimacy and effectiveness (Boehmer & Zaytsev, 2019; Kusek & Rist, 2004; Xue et al., 2013). The purpose of this research was to explore the relationship between legitimised behaviour (best practices) and effectiveness of ECD programmes as understood by actors across scales, through empirical research undertaken in Rwanda. This research investigated whether the design and behaviour of NGOs contribute to programme effectiveness, while also reflecting on the implications of these findings for theory and practice. For the purposes of this study, we define evaluation as an assessment of a project or policy that examines the design, implementation, and results of said project or policy (Kusek & Rist, 2004; OECD, 2002).

Methods

We employed a case study research design to explore NGO effectiveness in Rwanda, using qualitative methods for their ability to capture the perspectives of ECD stakeholders. These included in-depth interviews and focus group discussions (FGDs) (Finch & Lewis, 2003; Legard et al., 2003). The research focused on two international NGOs. Selection criteria included: well-established institutions with headquarters in the Global North, and country offices in Kigali; both are regarded as authorities on ECD in Rwanda; and work in different districts in Rwanda. These INGOs differ in hierarchical structures, missions, development priorities/sectors, and length of time in Rwanda. By comparing these two differing organisations, we were able to trace the links between institutionalisation, implementation, and effectiveness, and explore the theoretical assumptions of New Institutionalism in context. It was important to choose organisations that originated in the Global North, because it enabled charting how organisations that operate at different geographical scales navigate decision making and isomorphic pressures across these scales: local to national and global. Moreover, both organisations have long been considered legitimate actors at the global scale and maintain a strong influence at the national scale, with both perspectives having a bearing on the research concern.

In total, we conducted 45 in-depth semi-structured interviews. Of these, 21 were key informants (KIs) and the remaining 24 were staff employed at five NGOs (referred to as participants to distinguish leadership from staff). Two INGOs and three local NGO implementing partner KIs included government officials, programme officers from INGOs and country representatives from donor IGOs. Participants included employees from INGOs and local NGOs not part of organisational leadership. We used a purposive sampling strategy based on individuals’ affiliation with the ECD project (Ritchie et al., 2003). The selection criteria for KIs and participants were based on their relevant role within the organisation, i.e. knowledgeable staff working on projects related to ECD, leadership, donor IGOs, etc. In addition to in-depth interviews, six focus group discussions (FGDs) with community-based volunteers (CBVs) and families that participated in ECD projects were conducted. FGDs, facilitated in Kinyarwanda by two trained research assistants, afforded us access to groups of people whose voices and experiences are not typically captured when studying organisations: a strength of this study. All interviews and FGDs took place in one of three districts in Rwanda: Gasabo, Musanze and Huye. The districts in which the two INGOs implement ECD projects were selected because they represent urban, peri-urban, and rural communities, allowing for variation in data sources. All data were transcribed verbatim and de-identified. All participant names were assigned pseudonyms. Thematic analysis was undertaken using a coding matrix and NVivo. Following preliminary theme identification, we conducted supplementary, cyclical coding, applying a code-re-code strategy to identify, then verify, analytical themes and sub-themes.

Findings and discussion

To better understand the relationship between the need to be considered legitimate actors and the effectiveness of programming, we explored how organisations and individuals within those organisations define effectiveness: or rather, how effectiveness was understood in Rwanda. Notably, across interviews, there was a lack of consensus on how effectiveness was measured.

Theme 1: Effectiveness as process not outcome

In Rwanda, M& E outputs and outcomes are detailed through contractual agreements between organisations, as well as in applications to the Rwanda Governance Board (RGB, 2016). This standard procedure suggests that there are institutional expectations, or behaviours that are considered good and proper (Suchman, 1995), for the incorporation of outputs and outcomes when considering organisational effectiveness. More specifically, Alexandre (Male, 39, Rwandan, KI, government organisation) suggests that the government of Rwanda sets the standard for all planned activities.

‘The government is very much involved in this [evaluation] practice because we need to make sure that what is planned meets the priority of our country- or our institution- and that what was planned has been actually implemented as it was planned’.

Alexandre's perspective highlights the critical role of governmental oversight in ensuring that organisational activities align with national priorities and are executed as intended, thereby linking the legitimacy and effectiveness of the organisation within the Rwandan context.

When asked how effectiveness is measured, Keza stated (Female, 35, Rwandan, Participant, INGO):

‘Effectiveness is measured through monitoring and evaluation. Quality benchmarks are preemptively decided, and success is then measured on whether or not outcomes meet those benchmarks’.

Patrick (Male, 44, American, KI, INGO) describes the measurement of effectiveness to be outputs, or the number of children engaged, attendance and parenting lessons. Patrick was careful to stress:

‘we are not gauging, are they smarter than they were when they came or do they have better social skills than when they came. This is not our mandate’.

Notable in Keza and Patrick’s responses is that the process for measuring effectiveness is gauging whether benchmarks (quantifiable outputs) identified prior to the start of a project have been met post-implementation, suggesting reliance on rationalised behaviours rather than programmatic effectiveness.

Similarly, Jean (Male, Rwandan, 36, Participant, INGO), too, posits that effectiveness must include cost:

‘So, you have resources that you are putting on the table and you have activities that you are putting together. Those two things, plus the resources that you are using in terms of human resources, which should generate anticipated results. If you look at the costs that generate this outcome, there’s what you’ve planned and what you’ve achieved. At the end of the day, you draw a conclusion that you have achieved something or not. So, there’s cost effectiveness that has to come into play but there’s also trying to measure all the ingredients that you put in … to generate a specific result’.

Jean finishes his thought in a reflective way, suggesting that effectiveness is about using resources to achieve a specific result, providing this example:

‘So, If I can save 200 children instead of the 100 that I had assumed, but at the end of the day I achieve the same result of changing the behavior that I want in parents or children or achieve the same running outcomes – if it’s literacy that I’m looking at, for example –, then I have to measure effectiveness from those perspectives’.

Jean’s remark offers a slight contradiction from his previous statement – indicative of the conflict individuals within organisations may experience when defining effectiveness. While effectiveness as a process means adherence to the approved project plan, there is also a level of behaviour change noted by participants that need to be considered for a programme to be deemed effective, while still defaulting to quantifiable measures of outputs rather than outcomes.

This analytical theme highlights that, for some individuals in organisations, the effectiveness of programming is not about outcomes, but rather, meeting the outputs identified in the original project plan: a plan that is an expectation of the institutional environment. Arthur (Male, Rwandan, Participant, Local NGO) suggests that:

‘They [INGOs] come down to evaluate what is happening. If what we have planned is what has been done. If not we’re to be held accountable’.

The implication here, rooted in these participant responses is that effectiveness for INGOs is about adherence to project plans and cultural pressures linked to legitimacy. This is consonant with previous literature that links effectiveness to NGO decision-making processes, dependence on funds, and institutional expectations (Belush, 2018; Ebrahim, 2005).

In contrast, Joseph (Male, 47, Zambian, KI, IGO) takes issue with that process:

‘the drive to show results before you even design and implement the programs by the international NGOs, I find that a bit problematic’.

His response is striking, given his position in an IGO purportedly responsible for setting this standard.

Michael (Male, 37, Rwandan, Participant, Local NGO) suggests that the expectation to quantify work is a norm defined across scales – by donors, the government, and partner organisations alike:

‘People want numbers to prove to the high officials that they have been doing something. But if you go in those numbers, you see finally that there is no impact. That’s the most challenge I have seen in many organizations’.

Michael’s argument is echoed by Rebecca (Female, 36, American, KI, INGO) who states that ‘everyone is fighting for this big dollar money’.

Significantly, Michael and Rebecca’s responses imply that the results-driven, project-based model does not necessarily lead to a positive impact on ECD programmes. From his statement, we see that some are critical of this method of measuring effectiveness and yet Michael recognises it as an expectation of the normative environment. From a NI standpoint, Michael’s view suggests that individuals and organisations are navigating the conflict between doing what was planned and expected through the process, which confers legitimacy while wrestling with possible other effective approaches to ECD programme planning.

To measure effectiveness as part of a process, one of the INGOs conducts monthly and annual monitoring assessments of their local implementing partners to review proposed action plans, functioning of the leadership within the organisation, reporting mechanisms (financial and administrative), and personnel information (salary, contract, etc.). For local organisations that are partnering with multiple INGOs, this process starts to become burdensome, particularly as it relates to reporting and timelines. Janvier (Male, 55, Rwandan, KI, Local NGO) argues:

‘The first challenge we meet is the different evaluation formats. For example, each donor has his own format of report … I think another challenge is, you see programs are for three years. Three years is not big. It will be better if the programs are five years. Three years is not enough to reach good results’.

Janvier's statement underscores the challenges local organisations face with diverse reporting requirements and constrained timelines from multiple international partners, suggesting that extended programme durations could enhance the achievement of meaningful outcomes. However, to measure meaningful outcomes would require a culture shift in evaluation.

‘We don’t have a culture of impact evaluation. I think it would be a great thing for us, but our expectations are for evaluations after projects’. (Janvier, Male, 55, Rwandan, KI, Local NGO)

Janvier's insight points to a reliance on standardised evaluation methods, considered a best practice, but also identifies a missed opportunity in not incorporating impact evaluation. Notably, it is not only INGOs that adhere to this standardised process of evaluation. Alexandre (Male, 39, Rwandan, KI, government organisation) suggests a similar process is applied by the Rwandan government:

‘We have … staffs who do field visits and monitoring. We focus on collecting learnings and from them we are able to say that what we are doing is the right way. But also, through the consultative meetings we get feedback from different stakeholders, local leaders, church leaders or other people. They give us feedback that we take into consideration for the future planning’.

Most significantly, and perhaps unique to Rwanda, is the emphasis given to the level of engagement expected with the Rwandan government concerning whether programmes are effective:

‘Every NGO is strong if that collaboration with government from national level to district level, to local level [is present] … I think the quality is the involvement of the government each step of the project’. (Keza, Female, 35, Rwandan, Participant, INGO)

This level of engagement from the Rwandan government is significant because it suggests that pressures the organisations (and individuals within) feel are not simply a result of the funding model and donor expectations. Rather, the government also enforces those pressures. Indeed, the Rwandan Government has adopted M&E systems to monitor and evaluate activities by donors and NGOs, which requires organisations to present yearly action reports demonstrating how national priority indicators are being met (Boehmer & Zaytsev, 2019; RGB, 2016). Xue et al. (2013) suggest that these systems result in good governance by states and NGOs. However, others argue that they can lead to ineffectiveness because not all indicators are easily measured (Boehmer & Zaytsev, 2019). Moreover, methods for measurement are often quantitative, thereby acknowledging change in indicators in the aggregate but not at the individual or community levels, including qualitative dimensions (Görgens & Kusek, 2009).

Of importance, this analytical theme reveals that effectiveness, while an expectation of global health networks more broadly and projects specifically, is more aligned with the performative aspects of organisational processes and best practices, rather than long-term project results. This suggests that INGOs not only continue to perpetuate institutional norms but prioritise the institutionalised processes during programme implementation. As compared to local NGOs, effectiveness measured as a process is indicative of continued isomorphic pressures at national and global scales. While some actors linked effectiveness to organisational processes, others at different scales measured effectiveness as outcomes, as part of theme 2.

Theme 2: Effectiveness as outcome not process

In the previous theme, we saw how effectiveness is measured as a product of normative organisational processes that are a result of isomorphic pressures, rather than long-term results. In theme 2, we will see a parallel perception of effectiveness held by respondents that contrasts with theme 1. In this theme, respondents viewed projects as effective if they had been implemented in a way that promoted community use of the knowledge and skills gained through their participation in the project. Therefore, effectiveness as an outcome here transitions away from quantifiable measures and towards knowledge acquisition.

As Parfait (Male, 34, Rwandan, Participant, Local NGO) emphasises, ‘success’ in ECD programming can be viewed by participants as behaviour change among children:

‘To measure the success of this curriculum, it cannot be done in a short term but in a long term, in observing how children behave, the change in the family and also to go in the schools to see how children perform’.

Building on Parfait’s observations, Kelesa (Female, 54, Rwandan, Participant, Local NGO) provides examples of behaviour changes among parents and children:

‘Their parents got to know how to treat their children. Secondly parents got the time to work. Thirdly, it prevented the children to not keep wandering in the neighborhood and violence and hygiene also. And another thing, it opened up for children for timid children they are now open and they are in kindergarten’.

From these statements, we see that the understanding of effectiveness shifts away from the normative results-oriented approach to represent those changes observed within communities. Similarly, this pivot is echoed by Neema (Female, 53, Rwandan, Participant, Local NGO) who cites examples resulting from ECD programming, spanning children, caregivers and parents:

‘before ECD, some children weren’t accepted because they have not as much of knowledge but ECD prepared them to start in primary … children who went in ECD, they are different from children without ECD because they have basic knowledge. It is continuous … Another thing, caregivers may know which kid is sick if the parents didn’t figure it out and inform parents for their child’s treatment … Nutrition also got improved because parents are taught about cooking and at ECD there is a kitchen garden; they [ECD programs] prevent malnutrition’.

Michael (Male, 37, Rwandan, Participant, Local NGO) reflects similar observations on effectiveness, with examples drawn from hygiene, nutrition, and childrearing:

In general, the changes are people have improved their hygiene, the nutrition for the whole family, and the children, the communication in the household. The fathers started being involved in raising their children not only to provide food but being one of their friends, playing with their children, helping the female do family chores. Most of the families who participated in the interventions, not all 100%, but most of them like 80% are really showing the improvement’.

Important in these observations is that effectiveness varies across the different geographical scales and within the different levels of organisations themselves, suggesting greater complexity and differing perspectives. For example, Parfait (Male, 34, Rwandan, Participant, Local NGO) suggests that for local leaders, effectiveness is multifaceted:

‘Local leaders show us that what we are doing is helping them about reduction of conflicts in families, how parents are caring of their children, about hygiene. We said that we have a problem of stunting so they say ‘where you have worked, you gave parenting sessions’, stunting is on a lower level’.

Implicit in Parfait’s account is that leaders at the local level, including district officials, acknowledge that the ECD curriculum is holistic in its approach. Notably, though, effectiveness at this scale is still measured by the reduction in the number of children experiencing stunting, or quantitative targets. However, as we continue to get into communities and families, these perceptions start to shift. Parfait’s narrative captures this shift:

‘[Effectiveness] is about what parents say. When you are discussing with them, they say that ‘before there was fighting in our family, we didn’t talk about singing and dancing with our children but after sessions we think about that’. … So through all those testimonies, we see that what we are doing is perfect’.

While most FGDs with CBVs and families generally agree that dynamics within households have greatly improved, family members also suggest that ECD programmes are far from perfect:

‘They taught us a lot of things but sometimes we don’t have means to implement what we learned and that is very sad’. (FGD, family participants)

From this statement, we see a contradiction between the perspectives of those implementing ECD programmes and those participating in them. While effectiveness is measured as outcomes, such as engagement between family members, harmony in the household, etc., some families stated that what they learned was great, but they still lacked the tools to implement the ideals of ECD in their homes, suggesting that perhaps the programmes were not as successful or sustainable as intended. Importantly, this point illustrates effectiveness as a process rather than outcome. For example, if processes included addressing access, rather than just teaching skills, then outcomes could be more sustainable. This rivals the age-old tale of teaching someone to fish but not giving them a fishing pole.

Although in the narratives presented above, participants and KIs suggest that effectiveness is captured in the perceived changes in community and family behaviour, they do not provide specific examples of what that means for these communities and families. There is, therefore, a disconnect across the different scales and levels within these organisations. For example, it is that from CBVs and families that most clearly elaborate on the impact of these ECD programmes. As evidenced below, parents that participated in ECD indicated greater affection towards their children, through exposure to such programmes:

‘It helped us as parents because it taught us to love more our children, to create friendships with our children. Before we thought that it was difficult to stay with children but now, we are friends with children, we have fun with children, we make each other happy’. (FGD, family participants)

Similarly, another group of parents linked cultural attitudes with changes in their engagement with children:

‘No parents learn how to raise up their children, but we learned the importance of what we have been doing all these years. We got trained and we were taught that playing with your child is very important as they learn from it; feed a child has steps; pronouncing words is necessary for your children to grow up properly; and so many other tips that we never took seriously before’. (FGD, family participants)

One focus group discussion with CBVs indicated that the programmes are effective because of the knowledge that they as volunteers and ‘teachers’ gained, as reflected by one participant here:

‘I think it was a very good idea as they taught us to take good care of our children by stimulating their brain which helps them in growing, so they really did something good’.

Another FGD participant stated:

‘I also learnt that when you don’t take good care of your child earlier and they reach five years, they will have growth issues all along their life’.

Similar sentiments were echoed across the FGDs for CBVs affiliated with other organisations, illustrated as:

‘When a program like this comes and it concerns young children from the village … though we knew that children that have access to education are from town, they are the ones to access school from the age of three years, but now even our children also have access to this kind of education’.

From this narrative, the participant links effectiveness with increased access to educational opportunities for their children. Even though they live in peri-urban and rural sectors, ECD programmes give children from these areas a chance at early education. A CBV participant characterises the benefits of this access as follows:

‘The benefits of this are that now we have the knowledge to show parents that when they wake up, they bath them [the children] though they never used to do so … Now they have learned to speak languages, have learned to call us teacher though before they used to fear talking to us. That shows that there have been great improvements because there is no project that had done this before’. (Focus group discussion, CBVs)

This sense of pride in themselves as CBVs and the gains children are making is reflected by an FGD participant working with an implementing partner:

‘[the organization] made us confident, and we are very sure that our children will go far and have a great life ahead of them simply because we did everything necessary. Before, a parent thought that children should not go to school because their parents didn’t, but now those mentalities have changed, and we are very happy with that’.

It’s not only CBVs that have a sense of pride in the children with whom they work, but parents also feel that by engaging in ECD, children are set up for success throughout their life course.

‘Our children are now open minded; they are very smart. They know a lot of languages and have great discipline thanks to these ECDs. They are really different from those who do not come here’. (FGD, family participants)

Embedded in these experiences, CBVs and families perceive the programmes as effective because they have expanded how families understand human behaviour, what is important to attain different behavioural milestones, and how that shapes a child throughout the life course, not because these are linked to organisational structures or processes. This interpretation of effectiveness is important because it implies that with knowledge acquisition comes behaviour change, yet behaviour change is not measured by these organisations.

Discussion and implications

By using NI, combined with empirical research, we find that NGOs, through mimetic pressures, adopt a set of best practices in line with norms embedded in the institutional environment. These best practices are related to governance structures and organisational processes, such as measuring effectiveness. These findings are significant in demonstrating that, although effectiveness can be measured as both process and outcome, those within organisations that maintain the most authority ultimately determine how effectiveness is defined within the organisation in this study. For the INGOs, this means a numerical measurement (e.g. number of children in ECD; a number of districts and sectors implementing ECD).

Whereas, for those actively engaged in the communities, it reflects a shift in behaviour and thinking (e.g. engaging in play; increased father engagement). Of significance, the different understandings of effectiveness are important because they suggest that effectiveness is socially constructed and measured differently across different scales and actors. What is considered effective at the global and national scales, where behaviours become institutionalised and legitimised, is not how effectiveness is measured at the local scale. This suggests that effectiveness may not be a valid construct for NGOs, despite being an expectation of the institutional environment, which may be relevant in other settings and types of programming.

As a framework, New Institutionalism proved insightful for this study because of its emphasis on the relationship between institutional norms and organisations. Dissatisfaction with nongovernmental organisations is an increasingly prominent feature of global governance, with greater scrutiny of NGOs and attention to measuring organisational effectiveness. Findings here illustrate that, although institutionalised norms are adopted by NGOs across local and global scales, once they are implemented in practice, they do not necessarily reflect what communities need as they define ‘need’ in this setting. Moreover, such norms and practices, once adopted, ultimately shape the effectiveness of programming, with implications for early childhood development programmes, especially among children in low-income settings like Rwanda.

Apparent in the interpretations of effectiveness as both process and outcome is a tension between an organisation’s need to be considered legitimate (engaging in best practices) and the effectiveness of their projects. Participants who argued effectiveness was measured in how the programme was implemented, characterised the importance of projects as the formal and informal processes prominent in the network. These processes included partnerships between organisations, engagement with government officials, identification of districts and beneficiaries, use of CBVs, etc. Those that understood effectiveness to be more closely linked with behaviour change and knowledge acquisition provided examples of how acknowledgement of the importance of ECD at the national level has contributed to positive development among children and families at the community level. However, those that viewed effectiveness as an outcome rather than process – primarily CBVs and staff from local NGOs – were the two populations affiliated with the organisations that had less influence on ECD programme decision-making.

Of significance to global health and development objectives, neither of these understandings of effectiveness measured long-term impact. While there were anecdotal examples of beneficial work happening at the local level in the data, families and communities still face cases of stunting, which some argue was the primary rationale for establishing the ECD network. Findings indicate there are concerns with both the measurement of global health interventions generally and the notion of effectiveness more specifically. From a New Institutional perspective, these challenges stem from isomorphic pressures that result in bureaucratic structures adopted by NGOs (Barnett & Finnemore, 2004). In Rwanda, these dynamics make measuring the effectiveness of ECD programmes particularly challenging because the institutionalisation of bureaucratic structures complicates decision-making and localisation, which seem to be a precondition for measuring effectiveness. Innovative in this research is that we do not focus primarily on funding mechanisms as structures. While financial interactions are central to NGO-funder relations, an analysis based primarily on funding misses other crucial forms of resource exchange, namely reputation and status (Ebrahim, 2005). Although understanding funding mechanisms is important, these mechanisms are embedded in the larger process of institutionalisation occurring within the ECD network in Rwanda.

Study limitations

As a first limitation, while the findings may be indicative of a larger pattern within global health networks, they are not generalisable across settings – in line with qualitative inquiries. An additional limitation was the potential risk of researcher bias, or steering research towards a desired outcome. While difficult to avoid completely, the comparative nature of the project permitted us to triangulate independent sources to ‘corroborate, elaborate or illuminate the research problem’, thus increasing not only the rigour of the design but also the validity and reliability of results, while minimising the potential for social desirability bias (Decrop, 1999, p. 158). Third, in Rwanda, there is a cultural norm that individuals are expected to answer questions according to their perceived expectations, introducing some potential risk of social desirability bias into the data (Decrop, 1999; Shah, 2019). Moreover, this meant there was little criticism of the organisation from those who did not hold a leadership role. To mitigate the risk of social desirability, questions were phrased in ways which elicited or guided participants towards their personal views. To further minimise the risk of these biases, questions were primarily open-ended, precluding participants from simply agreeing or disagreeing.

Future research and recommendations

Regarding organisational and ECD programme effectiveness, academics and practitioners are left with more questions than answers. Therefore, future research on effectiveness could focus on organisations that have trended more towards participatory monitoring and evaluation. To understand the relationship between legitimacy and effectiveness, research could examine the intraorganisational motivations for this trend, and whether leadership matters for this process. Further research on effectiveness should also consider how training in M&E among specific roles within organisations might shape perceptions of output or impact. Because effectiveness is measured as output rather than impact (Kusek & Rist, 2004), it is difficult to draw a causal inference between the isomorphic tendencies of organisations and their effectiveness on the ground, suggesting a need to rethink or expand New Institutionalism. This is particularly true since NI has not fully grappled with power differentials among and within organisations in highly institutionalised, technocratic states, such as Rwanda; and appears insensitive to the underlying dynamics of international development models of partnerships. However, while some scholars (Arvidson, 2018) have moved beyond isomorphism as a justification for the adoption of structures and processes at the organisational level, these pressures are still prevalent in Rwanda and ultimately shape the effectiveness of ECD programmes. Therefore, it is necessary to reimagine effectiveness, currently a results-based management technique often tied to funding, to one that truly measures impact. This will require a shift in funding, timeline and goals of projects. Hence, organisations should trend away from implementation-based M&E, and more towards impact evaluation. This may mean that organisations present impact differently from what was intended in the original project plan. Demonstrating impacts differently from what was intended can offer valuable lessons and opportunities for organisations to refine their strategies, improve their practices, and better align their efforts with the needs of their communities and stakeholders. Measuring impact requires a longitudinal understanding of how programmes shape behaviour at the individual and community levels. If organisations want to understand effectiveness through the lens of behaviour change, they must consider investing resources in research that focuses on the long-term.

Conclusion

Significant in this research is that, by using in-depth interviews and FGDs, we illustrate through real-world data from an under-studied hard-to-reach population in Rwanda that there are multiple narratives for effectiveness. Implicit in our findings is that measurement of prescribed indicators may determine effectiveness in the aggregate but fail to acknowledge how communities and individuals understand the effectiveness and their ability to benefit from these programmes. This is problematic given the high need for development, including the urgent need to curtail child stunting in many settings. Research has long reinforced that early childhood development programmes are essential for a healthy society more broadly (Zonji, 2018). At present, the global community is off track to attain key SDG targets, including those for good health and wellbeing (goal 3) and quality education (goal 4). Therefore, these debates over what constitutes effectiveness for different stakeholders require us to consider what is at stake for child health and development when they do not agree. The implication here is the loss of developmental potential for children, which ultimately has social, emotional, educational, and economic consequences throughout the life course (Binagwaho et al., 2016). Interventions aimed at increasing the developmental potential of children are not only good investments for states but imperative for reducing the impact of biopsychosocial risk factors, while also increasing resilience and preventing intergenerational poverty among populations, whether in Rwanda or other settings (Binagwaho et al., 2016; Walker et al., 2011).

Acknowledgements

The authors thank all our participants and Key Informants for their time and for sharing their views with us. The authors thank our Rwandan colleagues for their support throughout this process.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by University of Massachusetts Boston.