Sustainability of donor programs: evaluating and informing the transition of a large HIV prevention program in India to local ownership

Background

Sustainability is the holy grail of many development projects, yet there is limited evidence about strategies that effectively support transition of programs from donor funding to national governments. The first phase of Avahan, the India AIDS Initiative supported by the Bill and Melinda Gates Foundation (2003–2009), aimed to demonstrate an HIV/AIDS prevention program at scale, primarily targeted at high-risk groups. During the second phase (2009–2013), this large-scale program will be transitioned to its natural owners: the Government of India and local communities. This paper describes the evaluation design for the Avahan transition strategy.

Methods/Design

A detailed logic model for the transition was developed. The Avahan transition strategy focuses on three activities: 1 enhancing capacities among communities, non-governmental organizations (NGOs), and government entities, in line with India's national AIDS control strategy; 2 aligning technical and managerial aspects of Avahan programs with government norms and standards; and 3 promoting and sustaining commitment to services for most-at-risk populations. It is anticipated that programs will then transfer smoothly to government and community ownership, become institutionalized within the government system, and support a sustained HIV/AIDS response.The research design evaluates the implementation and effectiveness of 1 activities undertaken by the program; 2 intermediate effects including the process of institutionalization and the extent to which key Avahan organizational procedures and behaviors are integrated into government systems; and 3 overarching effects namely the impact of the transition process on the sustained delivery of HIV/AIDS prevention services to high-risk groups. Both qualitative and quantitative research approaches are employed so that the evaluation will both assess outcomes and explain why they have occurred.

Conclusions

It is unusual for donor-supported projects in low- and middle-income countries to carefully plan transition processes, and prospectively evaluate these. This evaluation is designed so as to both inform decision making throughout the transition process and answer larger questions about the transition and sustainability of donor programs.

• sustainability
• evaluation
• HIV/AIDS

There has been much debate about the future funding requirements of HIV/AIDS control programs in low- and-middle-income countries 1 2 3 . With severe budgetary pressures in many industrialized countries, the flow of funds available to support the continued scale-up of international health programs is in doubt. Many development agencies are considering how best to deploy their resources efficiently and some are already planning to ‘graduate’ some countries from their list of aid recipients 1. In this light, there is considerable interest in the question of how best to plan and implement the transition of donor-funded programs to local ownership, particularly in contexts where recipient governments can afford to take over the funding of such programs.

Since 2003, the Bill and Melinda Gates Foundation (BMGF) has committed over US$400 million to address the spread of HIV/AIDS in India through Avahan, the India AIDS Initiative. At its inception, Avahan represented a unique effort to establish prevention strategies with most-at-risk populations (MARPs), principally female sex workers (FSWs), men who have sex with men (MSM), transgenders, male clients of sex workers, and truckers,1 to saturation scale (over 80% coverage in designated geographies) 2 3. Working across the six states in India with highest HIV/AIDS prevalence, 350,000 FSWs and 100,000 high-risk MSM had been contacted by Avahan outreach workers by December 2008 3. Avahan was put in place as a stand-alone program through a series of cascading contracts from the BMGF to a range of international and some national non-governmental organizations (NGOs) that are responsible for running programs in different states. These State Lead Partners, as they are called, in turn contract with smaller NGOs to provide services for high-risk groups (known as targeted interventions (TIs))4. In addition to MARP prevention programs across six different states, Avahan has also supported a program for clients of sex workers, including one at sex workers solicitation sites in the four southern states and another for long-distance truckers who deliver prevention activities at designated intervention points on main national highways across the country.

From the early years of the program, it was clear that ownership of Avahan would need to be transferred to country stakeholders and the project institutionalized within the broader Indian government response 5. Through the National AIDS Control Program (NACP) Phase 3 (2007–2012), the Indian government pledged increased financial resources to support scaled up prevention programs, thus paving the way for a transition of previously donor-supported programs. Avahan has pursued three different goals during its history. While the first phase of Avahan (2003–2009) focused on goal 1: ‘To build and operate an HIV prevention program at scale in target geographies of India and document epidemic impact,’ the second phase of Avahan (2009–2013) focuses on goal 2: to transition the program to its ‘natural’ owners, notably the Government of India (GOI), NGOs and community groups, and potentially other development partners (see Fig. 1). The third goal, not addressed here, concerns fostering and disseminating learning from Avahan within India and worldwide.

Fig. 1.  Avahan and Evaluation timeline.

The assessment described here is designed to serve three main purposes. First, it is intended to provide timely information to stakeholders in Avahan and the Indian government about the implementation of the transition process and its early effects. The transition is being implemented as a phased sequence of handovers over a four-year period, as such, evidence from earlier phases of transition can inform later ‘phases,’ enabling stakeholders to adapt strategies for the transition and to strengthen implementation processes if necessary. Second, the research should provide an independent assessment of how successful Avahan has been in achieving its goal of an effective transition from a donor-funded project to local ownership and management. Third, the assessment should contribute to global learning about effective strategies to transition a stand-alone donor-run program into an existing health system.

Avahan transition strategy

The goal of the transition strategy is to ensure a ‘sustained HIV response through an effective transition.’ This goal thus emphasizes both the need for ‘an effective transition’ and also the broader purpose which is to sustain impact.

The BMGF program team has been actively planning and refining the transition strategy since 2007, and the strategy has been adapted over time as the challenges and implementation issues have become clearer. The strategy identifies three main stakeholders to whom HIV/AIDS prevention activities will be transferred. The most significant of these is the GOI. A 2009 Memorandum of Cooperation between the National AIDS Control Organization and the BMGF that builds on an earlier 2006 Memorandum of Understanding sets out clear agreements regarding the transition process, with 10% of TIs to be transferred to government by April 2009, a further 20% by April 2011, and the remaining 70% by April 2012. The BMGF program team has worked intensively to promote and ensure continuation of government commitment to prevention among high-risk groups. By sharing its own experience of prevention among most-at-risk groups, and through technical support programs, Avahan has also sought to enhance government capacity to manage the HIV prevention program at multiple levels. Finally, Avahan has sought to prepare the TIs that it supports for the financial and managerial handover through, for example, aligning interventions with government guidelines and ensuring that necessary management skills are transferred to the NGOs responsible for administering the programs.

Community groups or community-based organizations (CBO), made up of the affected communities, are also seen as natural owners of the interventions. They play a key role not only in delivering services but also in sustaining the demand for services and holding the government accountable to its commitment to provide such services. Transition activities related to community groups include both organizational development activities to build the capacity of such groups and develop networks between them and structural interventions that aim to shape the environment through, for example, strengthening links with and promoting understanding among the police force. Finally, it is acknowledged that other potential stakeholders, such as other donors or other local institutions, may also play a role in the transition process.

It appears that several critical dimensions of transition are already in place. For example, NACP III includes a stronger focus on scaled prevention, in comparison to previous phases of the NACP. The HIV/AIDS prevention strategies such as TIs, prevention of mother-to-child transmission, voluntary counseling and testing, condom promotion, sexually transmitted infection (STI) management, and blood safety account for 67.2% of the NACP III estimated resource needs, and the government is clearly committed to saturated coverage of MARPs. In addition, BMGF along with other development partners,2 is currently making substantial investments in capacity enhancement at national, state, NGO, and community levels. For example, BMGF is currently supporting the training of trainers, the development of training tools and materials, and support to the National AIDS Control Organization's Information Education and Communications program to ease the transition. The first 10% of Avahan-supported TIs were transferred to government during 2009, a further 20% transitioned this year 2011, and the remaining TIs will transition in 2012 (Fig. 1).

Previous evaluations of transition

Much of the research in the development field that has addressed the phasing out of donor support to a project or a whole country (donor exit) takes the form of small-scale, retrospective case studies based on short-term consultancy work,which draw the bulk of their data from discussions with stakeholders 6 7 8 9 . The studies focus in particular on how communication and transition planning affect the overall success of the transition.

In high-income country contexts, and particularly within the health promotion field, there have been a number of studies with stronger theoretical and conceptual foundations that have sought to identify what factors have promoted sustainability, once external funding for a program has been withdrawn. These studies have typically identified three core dimensions of sustainability, all of which are relevant to the Avahan transition 10 11 12 :

• Community–continued capacity of a community to develop and deliver services, particularly relevant when the initial program worked via a community structure.

• Continuation of health programs–continuation of program activities within an organization (or by another organization).

• Maintenance of health benefits–continued health benefits for individuals after the initial program-funding ends

Literature on sustainability planning identifies processes that help lead to such sustainability. These include routinization that is typically viewed as the extent to which the program has been integrated into existing organizational systems and practices, and institutionalization that considers the role of institutional standards, and the extent to which innovation and learning not only gets adopted and sustained, but is also reflected in institutional standards and norms that govern multiple organizations within the broader health system 11 13 14. It may also be important to assess whether and how the transfer of the stand-alone program into the broader health system triggers further adaptation and change throughout the system 15. If the health system is viewed as a complex adaptive system 16 17, then the transfer of a stand-alone program into the health system will stimulate interactions between the program and the broader system that it is embedded within. This dynamic interaction is argued to be an important dimension of sustainability that can potentially trigger system-wide changes over time 12.

Much of the empirical work on program sustainability is retrospective. For example, a review of sustainability studies identified 19 studies that sought to assess the sustainability of health programs after their completion 11. The studies were typically implemented 1–5 years after the cessation of program funding or the formal ‘completion’ of the program and sought to asssess which if any program components or effects continued. Some studies have sought to assess the extent of sustainability planning. For example, Sridharan et al analyzed strategic plans to identify strategies that aimed to promote sustainability 18. In one of the few developing country papers, Hanh et al. developed a framework to assess and predict the likely sustainability of different dengue control projects in Vietnam 19. One longitudinal study sought to predict sustainability and then assess actual sustainability against predictions 20. However, none of the studies identified so far have prospectively sought to analyze sustainability and to guide processes so as to promote program sustainability.

Building on case studies and conceptual work by Yin 21, scales to assess the degree of ‘routinization’ and ‘institutionalization’ have been developed 13 22; however, these are quite context specific and require careful adaptation to different contexts. In general, there appears to be a dearth of empirical work on routinization and institutionalization 23.

Methods and design

A transition logic model

A logic model (Fig. 2) was developed through an iterative process involving review of Avahan documentation, relevant literature, and interactions with government and Avahan staff involved in the design and implementation of the transition.

Fig. 2.  Basic elements of logic model for transition.

The headings at the top of the figure represent a logical progression, from the activities conducted under the second phase of Avahan, to the immediate proximal impacts of these activities (defined as a state of ‘transition preparedness’), to the institutionalization of Avahan activities within the government system and finally the achievement of the transition goal (‘impact is sustained through an effective transition’). Achievement of the goal in turn contributes to India's national AIDS control goals and also to the overall purpose of Avahan in terms of maintaining or improving trends in reduction of new HIV infections among MARPs and the general population in India.

The first column of the figure draws on discussion of the Avahan transition strategy above to identify five main clusters of activities in preparation for transition. Three of these sets of activities relate to supporting and adding to the capacities of various entities, namely:

• Supporting government capacity – activities include 1 enhancing the technical and managerial skills of government staff members through training and mentoring; 2 supporting quasi-government HIV/AIDS prevention structures and the systems necessary for those structures to operate effectively, and 3 supporting the development and production of training materials as well as government norms and guidelines.

• Supporting NGO capacity – including the provision of capacity development support to implementing partners (NGOs/CBOs) to prepare them for transition and to enable them to take over some of the analytical and management functions previously conducted by Avahan contractors, in the post-transition period.

• Supporting community capacity – including support to community organizations both through strengthening management and governance structures and through building networks of CBOs.

The final two activities under the first column concern:

• Alignment of interventions – alignment of the technical, managerial, and cost elements of Avahan programs with government norms so as to facilitate transition. For example, Avahan programs have often operated their own STI clinics, and these services have sometimes provided a broader range of primary-care services 24. Under government guidelines, NGOs are encouraged to establish public–private partnerships with private health care providers for STI services and/or develop linkages with government STI services.

• Sustaining and monitoring commitment–government commitment to high levels of service coverage for TIs for high-risk groups is critical to the successful transition of Avahan programs. Securing and maintaining such commitment is inherently political and thus may be difficult to plan for. While government commitments are already documented in the current NACP and in the 2009 Memorandum of Cooperation with the BMGF, NACO and the development partners, including BMGF, jointly monitor these commitments, so as to help ensure that they are sustained.

These packages of activities are intended to ensure that all organizations involved in program planning, management, delivery, and oversight, both among Avahan partners and their counterparts within the government who will take over these functions, reach a state of ‘transition preparedness,’ so that Avahan programs can be transitioned into government systems with minimal disruption. The constructs within the second ‘transition preparedness’ column relate very directly to the five clusters of activities in the first column and indicate that transition preparedness has been achieved with respect to each of these activities.

The third column of the transition logic model reflects the processes that need to occur during and after the transition for the final goal (sustained impact) to occur. We have called this set of processes ‘institutionalization’: by institutionalization, we mean that the key elements of the Avahan program are integrated into the organizational procedures and behaviors of government agencies and other key implementing partners. Drawing on the dimensions of institutionalization discovered during the literature review, three different levels of institutionalization are identified. At the most basic level, routinization needs to occur, that is, key selected practices associated with Avahan-supported TIs need to be adapted to better fit government systems, adopted, and implemented on a routine basis. Second, representing a higher degree of institutionalization, select Avahan practices need to be reflected in government norms, standard operating procedures, guidelines, and policies. To some degree, this process of institutionalization has already occurred through the development of the current NACP and the norms and guidelines associated with this phase of the program. However, this process of institutionalization needs to continue and be consolidated, particularly at the state level.

The final dimension of institutionalization concerns the extent to which the transitioning of Avahan into the broader government health system has provoked dynamic changes within that system. For example, the transition process should lead to greater use of government STI services by MARPs and increased uptake of government counseling, and testing services. To the extent that these high-risk populations have been empowered by community mobilization and are accustomed to services that treat them with respect, they may provide a strong voice within the government health system that calls for improved standards of care for MARPs. Furthermore, there has been substantial concern in the literature about how stand-alone programs may drain government of scarce expertise 25 26. Although these concerns are particularly acute in Sub-Saharan African countries facing severe and generalized epidemics, the integration of Avahan programs into government may still release such scarce technical skills, previously employed by Avahan, and make them available to provide broad support across government programs. Integration of the Avahan program into the government health system may thus give rise to a variety of possible feedback loops and unanticipated consequences of dynamic interactions between actors.

The final column reflects both the Avahan Transition goal (‘sustained HIV response through an effective transition’) and the broader outcomes that the achievement of this goal is meant to lead to, including improved coverage of HIV/AIDS prevention services and a sustained impact on the HIV/AIDS epidemic in India.

Finally, the arrow at the bottom of the figure illustrates that learnings from early transition waves will inform the strategies and practices for later phases of the Avahan transition.

Management and governance

The research protocol was submitted to the Johns Hopkins School of Public Health Institutional Review Board (IRB), where it was exempted and also to the YRG Care IRB in Chennai, India where it was approved. The protocol was also reviewed by the WHO-led Evaluation Advisory Group of Avahan.

Research objectives and questions

Research objectives and questions have been guided by the logic model presented above. The research seeks to address both shorter term questions regarding the implementation of different activities and their proximal effects as well as higher level, and longer term questions regarding the overall achievement of Avahan goals.

Our overarching research question is:

• Has Avahan successfully transferred its program to government and other stakeholders in a manner that sustains its effects?

Lower level research questions include:

• Did Avahan implement all the elements of its transition strategy, as set out in plans originally agreed with NACO and the states? If not, why not?

• How effective are the different transition elements in achieving transition preparedness?

• What are the elements necessary for effective transition and are any not addressed?

• How effectively have the elements of the Avahan transition strategy translated into institutionalization of the program?

Research design

In developing the overall study design, the researchers sought to not only address the research questions identified above but also to craft the study so that it produced early findings that could inform later rounds of transition programming as well as broader overarching conclusions. Furthermore, a research study to assess the readiness of CBOs to transition has already been initiated separately, and a multidonor process to track the implementation of NACP III and government's commitment to the plan is in place. This study will not replicate the other work, but rather will draw on relevant findings from the other assessments to inform the overall evaluation of the transition process. Accordingly, five main ‘work packages’ (WP) were identified. These are depicted in Fig. 3 in relationship to the transition logic model.

Fig. 3.  Overview of the Five Main Work Packages associated with the Transition Assessment.

WP1 government capacity assessment (addressing research questions 2 and 3)

This substudy seeks to assess the effectiveness of Avahan's support to government capacity. The assessment will focus only on capacity with respect to HIV/AIDS prevention functions where Avahan has provided additional support to NACO and select State AIDS Control Societies (SACS), and as it relates to the transition. It will assess the extent to which Avahan support has contributed to ensuring the availability of staff with appropriate skills and training, in light of their job responsibilities, and with access to relevant norms, guidelines, and job support tools at national and state levels. WP1 plans to employ a structured survey of all staff in relevant units, semi-structured interviews with selected managers, and an administrative record review.

WP2 NGO and TI transition preparedness (addressing research questions 2, 3, and 4)

This WP assesses the extent to which Avahan TIs are aligned with government norms prior to transition and how well prepared Avahan funded NGOs and CBOs are for the transition. Based on reviews of government norms and standards, a standardized checklist was developed that identifies the key issues in transition alignment (such as the composition of the TI team, adjustments to budgets and reporting formats, and adherence to guidelines on STI syndromic management). Structured surveys employing the checklist are planned to be administered at all of the TIs just prior to their transitions in 2011 and in 2012. Review of administrative records is used to validate the verbal responses received from informants. Simple indicators of transition readiness will then be developed so as to measure transition readiness across various dimensions (such as NGO capacity and alignment of program elements such as costs, STI services). It is anticipated that given the phasing of the transition process, evidence from this assessment can be used to inform subsequent rounds of the transition.

WP3 longitudinal case studies of TIs (addressing research questions 4 and 5)

Using a longitudinal case study design, a series of case studies of select TIs will be studied to explore in detail how transition preparedness affects transition and institutionalization. The case studies employ both qualitative and quantitative data to describe and analyze these processes and will provide both the details of transitions as well as a picture of the transition process over time. The case studies will also look for any unanticipated consequences of the transition, including any dynamic changes in the system provoked by the transition. It is anticipated that this small-scale qualitative work may also help inform the development of WP4.

WP4 institutionalization assessment (addressing research questions 1 and 5)

This WP seeks to examine the routinization of Avahan processes among TIs post-transition and more broadly the adoption and institutionalization of Avahan learnings within the government. In a preliminary step, the Delphi method 27 was used with Avahan staff and partners to identify key features of the Avahan approach that should be institutionalized. The study will then assess if key practices associated with Avahan TIs are adopted and implemented on a routine basis, and whether Avahan practices are reflected in government norms, standard operating procedures, guidelines, and policies post-transition. Structured questionnaires will be implemented to assess the uptake of Avahan learnings and their routinization in NGO and CBO practices. Institutionalization is planned to be examined using semi-structured interviews among NACO and SACS staff and an administrative record review.

WP5 summative evaluation (research question 1)

The summative evaluation will seek to synthesize findings from different elements of the study, and from relevant studies and assessments conducted by others (notably on community preparedness for transition, and government commitment to implementation of NACP III) so as to consider the entire set of links illustrated in the logic model for transition. Drawing on existing data on service coverage and health impacts, this analysis will assess how differing degrees of transition preparedness, government capacity and commitment, and program institutionalization have contributed to sustaining effective services and program outcomes.

Research implementation

The evaluation design described here is being implemented by a team of independent evaluators; however, the team clearly needs to work closely with those implementing Avahan, and for many dimensions of the evaluation, the team is reliant on existing sources of secondary data. Early experience with implementation of the research has highlighted a number of challenges:

• Secondary data is not always available in consistent formats, there are differences across states, but also before transition (collected through the Avahan monitoring system) and after transition (collected by the government system). These differences make it very difficult to detect real trends in service delivery for example.

• Although the research is intended to serve the needs of program implementers and inform their actions, program implementers have heavy burdens, particularly at the time of transition, and the evaluation can sometimes be seen as intrusive and yet another burden in an already busy schedule.

• The research can be viewed as politically sensitive. Although the evaluation is primarily concerned with the transition process and the effects of the transition, if not carefully framed, it can appear as an assessment of government performance (post-transition).

Discussion

Strengths and limitations

There are a number of aspects of the current study design, which we believe are quite innovative. First, to the best of our knowledge, this is the first prospective analysis of a transition and institutionalization process that seeks to determine the effects of these processes on sustainability. Second, the fact that this is a mixed-method study enables us to triangulate between different data sources and use the rich detail available in the qualitative research to inform the design and analysis of quantitative research components. Third, the study has been designed in a phased manner, so that early practical lessons from the research can be used to inform later rounds of the transition process.

Avahan is a complex and large-scale program. The study protocol seeks to strike a balance between its comprehensiveness and focus. It is difficult at this point in time to know whether the right balance has been struck. There are clearly elements of the Avahan transition process that our logic model and research design do not play close attention to (for example, efforts to change structural issues such as attitudes toward FSWs, high-risk MSM, and transgenders) and yet could be critical for transition success. Conversely, although we are conceptually clear as to how different components of the overall study design relate, we believe that there will be substantial challenges in synthesizing data collected through different WPs and understanding the broader chain of connections between transition preparedness, program institutionalization, and sustainability outcomes.

Contribution

Although this study is tailored to one particular, albeit very large program, we believe that both its methods and findings are likely to be relevant to other programs and contexts. Although substantial effort is frequently invested in the design of a new program, there is rarely a comparable investment in ensuring the sustainability of program benefits. In the current financially constrained context, carefully planned program transitions may be the key to ensuring allocative efficiency and the sustainability of benefits. This study design illustrates one possible option to help strengthen transition processes.

Conflict of interest and funding

Work on this assessment has been funded by the Bill and Melinda Gates Foundation that also supports the implementation of the Avahan program.

Acknowledgements

This research was funded by the Bill and Melinda Gates Foundation. The views expressed herein are those of the authors and do not necessarily reflect the official policy or position of the Bill and Melinda Gates Foundation. The authors would like to thank staff of the Indian government, Avahan, and the Bill and Melinda Gates Foundation who all gave generously their time to ensure that programatic aspects of Avahan and its transition were fully understood. All authors participated in the conceptualization of the study. SB developed the first draft of the paper. SS, SO, NT, and JK critically reviewed the paper and provided edits and additional text. All authors read and approved the final manuscript.

Notes

¹Intravenous drug users were also covered by the program in two northeastern states.

²Including the World Bank, the United Kingdom, Department for International Development, the United Nations Program on HIV/AIDS, the United Nations Children's Fund, the United Nations Development Program, and the United States Agency for International Development.