Effects of the Frontiers Prevention Project in Ecuador on sexual behaviours and sexually transmitted infections amongst men who have sex with men and female sex workers: challenges on evaluating complex interventions

Abstract

In this study, the authors evaluate the impact of the Frontiers Prevention Project (FPP), a community-based strategy for HIV prevention amongst female sex workers (FSW) and men who have sex with men (MSM) in Ecuador. The FPP impact evaluation methodology consisted of a community trial with randomised intervention/comparison assignment and pre-post surveys in six cities in Ecuador (2003–2007). The results suggest prevention strategies involving key populations in their design and implementation can contribute to abating risk behaviours in Ecuador. However, the FPP experience also shows that implementing and evaluating large-scale HIV prevention programmes still present important challenges.

Keywords:

• HIV prevention
• Latin America
• Ecuador
• female sex workers
• men who have sex with men

Introduction

Conducting impact evaluations of HIV interventions is complex, particularly when these interventions take place in highly politicised environments with diverse pools of social players and heterogeneous prevention programme implementation. Even as the global financial crisis affects development financing, an important share of financial resources is still channelled to combat the HIV epidemic. In order to reinforce this continued support, there is a sense of urgency to acquire clear evidence on the most effective methods for using those resources (Padian et al. 2011).

Given the urgent need for relevant evidence and the necessity of working within poorly controlled conditions that can limit the power and quality of evaluations, it is necessary to understand the extent to which valid conclusions, relevant to real-world conditions, can be reached. Due to the contamination of comparison observations, the design of an intervention itself can cause an impact evaluation to have unexpected results, as we will describe herein.

This study analyses an HIV prevention intervention amongst men who have sex with men (MSM) and female sex workers (FSW) in Ecuador. It uses an intent-to-treat (ITT) approach and thus assumes potential contamination as an element of implementation in the real world while seeking to control it through a variable that accounts for level of exposure to intervention activities. The impact of the intervention on each of these groups is discussed below. The study also considers and highlights the difficulties in evaluating an HIV-prevention programme, including limited coordination amongst key players and the lack of an explicit process to align incentives. While for the FPP in Ecuador the evaluation process was agreed with the implementing partners of the FPP, the evaluation team (comprising external evaluators and evaluation teams at the implementing organisation) sought to document effects attributable to the intervention, while the implementing team sought to obtain visible results, both for financiers and the programme's target population. For example, the successful formation of a sex workers network (RedTrabSex) in Ecuador led to the spread of prevention and support initiatives across the country, including comparison cities and thus implementing an un-anticipated intervention that would affect the evaluation.

Challenges of impact evaluations of HIV-prevention programmes

Examples from different studies across the globe have shown that large-scale interventions to prevent HIV infection face different challenges that can compromise the implementation of the intervention, as well as its evaluation (Ross 2010). Thus, the measurement of weak outcomes from interventions may be attributable to the aspects of the study design and programme implementation (Padian et al. 2010).

A critical component of an impact evaluation is identifying a comparison group as representative as possible of the study population (Duflo and Kremer 2005) to draw conclusions about what the outcomes would have been if the programme not been implemented (Baker 2000).

In general, randomisation is considered the most robust strategy to create comparable intervention and comparison groups (Baker 2000). Nonetheless, the need for a pure control group comes with important challenges. For example, if a programme is not able to offer the same intervention to everyone in the selected population due to financial constraints, then it may be relatively easy to randomise the selection of the population to be covered. However, if sufficient financial resources are or become available, the control population will be lost if the program is considered standard-of-care and a true control unethical. This is the population/program equivalent of clinical equipoise that governs the use of placebo controls in clinical trials. Similarly, in the context of HIV prevention, the need to quantify the effectiveness of prevention strategies through an impact evaluation does not justify withholding prevention activities known to be effective from a specific group. The development of an evaluation methodology should not be used as the criterion to decide whether to invest in prevention activities.

Another fundamental aspect in the evaluation of the impact of HIV prevention programmes is heterogeneity in the intensity of preventive services rendered both in the intervention and comparison groups (Padian et al. 2010). In practice, community intervention evaluations take place within geographical and social constraints, making it difficult for factors taking place on a large scale to be controlled (Grassly et al. 2001). For example, the extended length of an impact evaluation increases the probability of other preventive strategies being designed and implemented, including at the comparison groups that may mask the effects of the programme under evaluation. Addressing this challenge by conducting the evaluation over a short time span, however, is inadequate because changes in rates of HIV transmission cannot be detected over short time periods (Hayes et al. 2010).

An additional consideration arises for evaluation strategies that include community participation in programme design. Because the communities are heterogeneous, the resulting programme designs are heterogeneous. However, homogenising interventions is complex and thus different versions of the programme are evaluated, thereby affecting the ability to identify single effects.

Impact indicators may present additional challenges for evaluations of HIV prevention activities. The standard approach to select appropriate indicators is to choose those prioritised according to the programme objectives (Bertrand, Magnani, and Rutenberg 1996). In terms of HIV prevention that would imply focusing on behavioural change, specifically risk behaviours (Coates, Richter, and Caceres 2008) as well as biomarkers. However, when the baseline level of a key indicator is already high, measuring the changes brought about by the intervention is difficult. For instance, assuming that even in the absence of a specific intervention condom use has a positive trend with a high baseline per cent of condom use, measuring an impact on this indicator may not be feasible. When baseline measures are already high, behaviour maintenance can be measured but may require an unfeasibly large sample size.

Epidemiological context of HIV in Ecuador

In 2010, new HIV cases reached 2.7 million globally, with countries in Latin America (LA) contributing 100,000 new infections (Choi et al. 2004). HIV prevalence among adults LA is estimated at 0.4 per cent (Choi et al. 2004), and the number of HIV infections amongst women began to increase. Countries in Latin America (LA) tend to have concentrated HIV epidemics; epidemics in which there is a low prevalence in the general population and a high prevalence among key populations with more risky sexual or drug-injecting practices (Calleja et al. 2002).

In the case of Ecuador, HIV prevalence in the adult population is 0.4 per cent. It is estimated that in 2009 there were 39,000 individuals of 15 years of age or older infected with HIV, 11,000 of whom were women (UNAIDS 2010).

In a studied group of LA countries, higher prevalence of HIV has been identified amongst MSM (Bastos et al. 2008), with anal sex being the most common mode of transmission (Cáceres, Pecheny, and Terto 2002). According to a meta-analysis of 38 studies, MSM have a 33-fold higher probability of HIV infection compared to the general population of reproductive age in LA (Baral et al. 2007). Nonetheless, the social stigma towards homosexuality in the region has hindered the development and implementation of effective prevention strategies (Caceres 2002; Geibel et al. 2010). Despite being the group most affected by this epidemic, the amount of directed resources is disproportionately lower for MSM when considering the incidence amongst this population (Caceres 2004; Izazola-Licea et al. 2004; Baral et al. 2007).

FSW are also disproportionately exposed to sexually transmitted infections (STI), including HIV. FSW have more sexual partners than the general population, lack access to healthcare services, have difficulty accessing condoms, use condoms inconsistently (Ghys, Jenkins, and Pisani 2001) and practice anal sex more often than the general population (Schwandt et al. 2006; Priddy et al. 2011; Tucker et al. 2012). In addition, gender roles, violence and the exploitation and criminalisation of sex work contribute to the practice of unprotected and forced sex, increasing the risk of contracting STIs, including HIV (Panchanadeswaran et al. 2008; Okal et al. 2011). According to UNAIDS, HIV prevention programmes in LA have reached 76 per cent of FSW (UNAIDS 2010). However, there is scant evidence of the effectiveness of these programmes.

In Ecuador, the few studies that have included MSM as part of the regional studies in South America have identified important variations in HIV prevalence within this population. The highest prevalence (23–28%) has been recorded in Ecuador's largest city, Guayaquil (Bastos et al. 2008) whereas in Quito, the capital city, reported prevalence amongst MSM oscillates between 11 and 17 per cent (Cáceres, Pecheny, and Terto 2002; Herbst et al. 2005; Montano et al. 2005). Some studies, however, indicate that HIV prevalence amongst FSW has remained relatively low in Ecuador (Bastos et al. 2008), at approximately 0.5 per cent in cities such as Quito (Hierholzer et al. 2002; Montano et al. 2005). According to these studies, there are also important variations within the same country; HIV prevalence amongst FSW in Guayaquil has been documented at 2.1% (Montano et al. 2005; Bautista et al. 2008).

Although the number of impact evaluations of HIV prevention interventions amongst MSM and FSW is increasing (Johnson et al. 2002; Herbst et al. 2005; Shahmanesh et al. 2008), evidence from LA countries is scarce (Johnson et al. 2008; Huedo-Medina et al. 2010), and the existing literature regarding the state of the HIV/AIDS epidemic amongst these groups is still limited (Bautista et al. 2006). For Ecuador in particular, evidence of impact of HIV prevention interventions amongst MSM and FSW is lacking.

The Frontiers Prevention Project (FPP) in Ecuador

The FPP was a multinational initiative coordinated by the International HIV/AIDS Alliance (IHAA) and funded by the Bill and Melinda Gates Foundation. The initiative emerged as a strategy to contribute to the battle against HIV infection in low-prevalence countries with concentrated epidemics amongst key populations (KPs), defined as those groups that are crucial to the epidemic due to their high propensity both to infection and transmission (namely, MSM and FSW) and to pandemic growth (Gutierrez et al. 2010). The FPP included an evaluation strategy designed to demonstrate the effect of an intervention package on the reduction of high-risk sexual behaviours and STIs amongst KPs.

The theory of change behind FPP was grounded on the assumption that empowering KPs in HIV prevention initiatives will contribute to reducing high-risk behaviours and, consequently, the incidence of STIs. The reduction of high-risk behaviours and STIs is expected to lower the HIV infection rates amongst KPs, which, in turn, is expected to lower the STI incidence amongst the population at large. To promote empowerment amongst KPs, however, it is necessary to work in environments that favour prevention – an aspect which involves local advocacy – and render fundamental goods and services available for targeted interventions (Gutierrez et al. 2010).

In addition to these premises, the FPP design revolved around two fundamental axes: (a) the implementation of an intervention package in geographic zones that allow for ‘immersing’ the sites in prevention activities. That is, geographic sites are the focus for FPP interventions; and (b) the involvement of KP members in the design process, not only to ensure the inclusion of the elements that are essential to each population but also to facilitate uptake of the interventions.

Project interventions were implemented from December 2003 to December 2006 through non-governmental organisations (NGOs) and community-based organisations (CBOs) coordinated by Kimirina Corporation. Implementation of the intervention package was aligned to three intermediate outcomes: KP empowerment for HIV prevention; promotion of a favourable environment; and prevention and care service coverage. Numerous activities were performed under the next categories:

•	Individually focused health promotion: (a) KP facilitator training and peer education; (b) awareness/education meetings; (c) organisation of mass events; and (d) information, education and communication campaigns and materials.
•	Ensuring access, scaling-up, targeting and improving service and commodity delivery: This was done with special emphasis on condom accessibility and improving voluntary counselling and testing services: (a) condom accessibility; (b) the formation of KP reference and counter-reference networks focused on counselling, voluntary tests and sexually transmitted diseases (STD)/HIV care; and (c) technical staff training for HIV prevention and care.
•	Mobilisation of KP Communities: (a) participatory site assessments in partnership with KP; (b) leadership training workshops; (c) promotion in defence of human rights; (d) establishment of solid KP networks; and (e) provision of safe spaces for KPs to meet and work together.
•	Advocacy, policy change and community awareness: (a) information/awareness workshops for authorities and service providers; (b) the promotion of KP participation in decision-making positions; and (c) workshops on public policy advocacy.
•	Capacity building of NGOs and CBOs to effectively implement quality prevention interventions: This includes strengthening HIV organisational capacities to different social actors at the local or national level: (a) training on technical, financial and/or organisational processes; and (b) strengthening the interaction, cooperation and partnership among NGOs, CBOs and/or other agencies.

Participatory site assessment was the first intervention implemented as part as the FPP; it was designed to mobilise KPs and to inform the design of subsequent FPP interventions. In Ecuador, 2333 people participated in PSA, mainly KP, in the FPP intervention sites (six cities) (International HIV/AIDS Alliance 2003).

The FPP included work with 40 NGOs and/or CBOs and helped to mobilise new groups of MSM, FSW, transgender people and people living with HIV in the six cities; it is estimated that by the end of 2007, the number of people reached through HIV prevention activities, in general, in Ecuador was 12,808 whereas the number of individuals reached through stigma and discrimination reduction initiatives was 50,976. (International HIV/AIDS Alliance 2010) Within FPP, successful advocacy work achieved positive changes. During the FPP duration, a large number of CBOs appeared (Public Health Ministry of The Republic of Ecuador 2005) – for example, the formation of a Sex Workers Network (RedTrabSex).

Main challenges of FPP assessment in Ecuador

The FPP evaluation methodology was designed in conjunction with the FPP itself to obtain an initial agreement about the relevance of and approach to the evaluation. However, upon intervention of additional players as a consequence of the action model proposed by the IHAA (community participation), the evaluation team did not align its goals with the new actors. This lack of coordination led to the disjointed implementation of the FPP and to little or no tracking of preventive interventions. In particular, an emphasis of the prevention activities targeting FSW, with less emphasis on strategies targeting MSM, was decided.

In addition, while the need to maintain comparison sites, sites without intervention activities, was communicated to all partners, this was not supposed to tackle community activities. In 2005, a Female Sex Worker Network in Ecuador was formed with support of the FPP. Members of the network actively participated in education activities and leadership training for HIV prevention and care, expanding its activities thoroughout the country. The Sex Worker Network currently includes sub-networks in 13 of the 24 provinces in Ecuador. These factors resulted in heterogeneous implementation and a significant degree of contamination amongst comparison groups. While in terms of prevention activities the formation of the network is a positive effort, it represented a major challenge for the evaluation. The analysis presented hereafter reflects these limitations, as well as the approach taken in their evaluation.

Methodology

The FPP evaluation methodology consisted of a community trial with randomised assignment to intervention and comparison groups and repeated cross-sectional surveys. An important aspect to attend to is that this is not a individual panel study, but a site panel study. We will use the terms pre-intervention or baseline to refer to data gathered from the 2003 survey and post-intervention or follow-up to refer to the 2007 survey.

Based on existing STI and HIV data and the experience of the responsible NGO, eight cities were selected from a group of cities in Ecuador deemed able to assimilate the prevention activities. Of the eight cities selected, two proved unsuitable for randomisation and were excluded from the final evaluation component. The remaining six cities were randomised to the intervention and comparison groups. The three intervention cities were provided with additional resources for the aforementioned FPP prevention activities focused on KP and expanded advocacy actions, while the three comparison cities received the standard national HIV programme initiatives. The latter (and, to a lesser degree, the former) benefited from HIV prevention efforts implemented by other organisations, particularly national initiatives from the Ministry of Public Health. At no time did FPP seek to limit any actions taken by other parties.

To estimate the study population size, mapping exercises were conducted in each city with the participation of local MSM and FSW population members and NGOs and CBOs, which were involved in the identification of KP high-concentration sites (mainly social and/or working environments) at each city and measurement of their sizes in terms of the number of MSM and FSW identified.

Two cross-sectional and anonymous surveys to MSM and FSW were carried out, one in 2003 and another in 2007. The resulting information was used to fit a panel data analysis. The sample size estimate was based on the minimum number of subjects required to detect a difference in condom use at last intercourse between the intervention and comparison cities at follow-up, considering the design effects of cluster selection (cities). A 10-percentage-point difference in condom use was calculated with a 2.2 design effect.

The study was approved by the Ethics and Research Committee at the National Institute of Public Health of Mexico (abbreviated to INSP in Spanish), the Ethics Committee of the Ecuador National Health Board and the International HIV/AIDS Alliance.

Instrument

The study questionnaire was developed by an international, multidisciplinary and inter-institutional team in conjunction with local researchers and key actors in Ecuador. The questionnaire was designed to explore the high-risk sexual behaviours of KPs, particularly in reference to partners, types of sexual practices and condom use. Additionally, the questionnaire collected data on socio-demographic variables, information and knowledge on HIV/AIDS and STIs and attitudes towards individuals living with HIV. Focus groups and in-depth interviews were conducted previously with FSW and MSM to ensure that the language in the questionnaire was suitable for the study population.

For the 2007 survey, the questionnaire was adjusted to include specific questions on exposure to intervention activities. The questions were applied at both the intervention and comparison sites with the purpose of detecting potential sample contamination and obtaining a proxy of prevention activities implemented by other actors, including those of the national programme. It was not the intention to use these variables as key outcomes, but more as information that could facilitate the interpretation of results.

Data collection

To achieve an optimal response rate, both survey interviews were conducted by KPs trained in interview techniques. Data collection under the pre-intervention survey was carried out in mid-2003 through anonymous face-to-face interviews at sites frequented by MSM and FSW. Each site in which surveys were administered was classified under predefined codes. For example if an FSW was surveyed on a street, the code of ‘street’ was used regardless of which street it was, and the same approach was performed for those working in clubs and for the different types of concentration sites. Separate mappings were done for MSM and FSW populations. At each site, the interviews were held with MSM and FSW previously identified by the interviewers. Further details on FPP baseline data collection are available in previous publications (Gutierrez et al. 2006a, 2006b).

For the post-intervention survey in 2007, a new mapping exercise was carried out to update the list of MSM/FSW concentration sites while respecting the original categories of site types used in the 2003 survey. Again, anonymous interviews were conducted with KPs face-to-face at the meeting sites. During both baseline and follow-up, capillary blood samples were taken for assays on serological antibodies to herpes simplex virus type 2 (HSV-2) (using Focus Diagnostics HerpesSelect 2™) and syphilis (using bioMerieux Trepanostik™). The HIV antibody assays were performed only at follow-up.

Participants were offered an explanation of the study objective and, subsequent to their voluntary acceptance to participate, were asked to sign a written consent, guaranteeing their anonymity at all times. General characteristics of participants by group and time of measurement are found in Table 1.

Table 1. Socio-demographic characteristics of a sample of men who have sex with men (MSM) and female sex workers (FSW) in Ecuador. Baseline (2003) and follow-up (2007).^a

	MSM (n = 2400)				FSW (n = 2662)
	Baseline (%)		Follow-up (%)		Baseline (%)		Follow-up (%)
	Comparison	FPP	Comparison	FPP	Comparison	FPP	Comparison	FPP
	(n = 479)	(n = 1248)	(n = 211)	(n = 462)	(n = 774)	(n = 752)	(n = 562)	(n = 574)
Age (mean)	26.8	24.8	27.8	25.9^†	28.3	27.2	28.8	28.2
Marital status:
Married/in union	21.8	19.0	31.7	18^†	45.3	46.8	48.8	52.1
Divorced/separated/widowed	3.7	0.7^†	2.8	3.9^‡	4.7	5.3	6.4	5.9
Single	74.5	80.3	65.4	78.1^†	50.0	47.9	44.8	42.0
Has at least one child	31.5	21.7	31.3	23.6	87.5	85.4	89.0	83.6^†
Group of self-identification (MSM only):
Bisexual (active and/or passive)	48.4	43.5	48.8	41.1	N/A	N/A	N/A	N/A
Masculine gay	21.5	31.9^†	20.4	31.4^†	N/A	N/A	N/A	N/A
Heterosexual	8.4	3.6^†	4.7	3.0	N/A	N/A	N/A	N/A
Other	21.8	21	26.1	24.5	N/A	N/A	N/A	N/A
Self-identifies as a sex worker (FSW only)	N/A	N/A	N/A	N/A	84.6	85.0	82.4	83.9
Family is aware of sexual preference (MSM only)	86.5	88.6	82.9	85.5	N/A	N/A	N/A	N/A
Family is aware of sex work (FSW only)	N/A	N/A	N/A	N/A	57.0	59.3	51.6	59.2
Participates in support group	11.1	6.5	10.0	11.0	10.5	8.1	7.3^‡	11.0
Socioeconomic level:
1^st quartile	18.8	24.8	44.5^‡	16.5^†‡	20.5	33.4^†	24.4	27.5
2^nd quartile	27.2	22.5	22.7	26.2	21.6	24.8	24.4	23.9
3^rd quartile	27.0	25.2	21.3	26.8	27.8	22.2^†	24.4^‡	25.4
4^th quartile	26.9	27.4	11.4^‡	30.5^†	30.1	19.7^†	26.9	23.2
Notes: ^aAdjusting by cluster sampling.
^bAssessed only at follow-up.
^† p ≤ 0.05 for the difference between comparison and FPP group within each assessment.
^‡ p ≤ 0.05 for the difference between baseline and follow-up within each group.

Data management

The questionnaires were entered into a database using a double-capture system to ensure the quality of the information. Data analysis was performed with Stata 10.0 (Stata Corp, College Station TX). The databases corresponding to each measurement were integrated into two databases: one for MSM and another for FSW. To correct for any group underrepresentation at follow-up, the final MSM database was weighed considering the distribution of categories in the variable of self-identification of sexual preferences, according to the reported baseline distribution.

While assignment to intervention and comparison was done by city, an alternative representation of the clusters was proposed since the reduced number of cities limited the power of the estimations. Considering that most of the heterogeneity within city is due to the specific type of work site or meeting site, the analysis implemented herein considers instead that a cluster is conformed by each KP concentration site used for the recruitment of participants. Thus, each type of site per city represents the sampling unit. This approach provides a sufficient number of clusters to perform estimations and is consistent with the fact that each site type identifies a distinct profile of the KP. While it is true that assignment was done at the city level, this approach allows to capture the important heterogeneity within cities that is related to specificities of the sub-groups of the key populations. The validity of the analysis depends upon the degree of true independence of the sub-clusters from each other and is compromised by the fact that all of the sub-clusters in a city were, by necessity, randomised together to intervention/comparison, but under the circumstances it is the most robust analytic approach we can propose. Intracluster correlation coefficients (ICC) were estimated for each dependent variable using both the city and the sites as clusters. In general, ICCs tend to be small; however, ICCs obtained for the sub-cluster are larger. This is presented in Table 2. No geographic information is provided for the virtual clusters, as a cluster may be constituted for similar but different sites in a given city.

Table 2. Socio-demographic caracteristics of a sample of men who have sex with men (MSM) and female sex workers (FSW) in Ecuador. Baseline (2003) and follow-up (2007).^a

			Baseline (%)		Follow-up (%)
MSM (n = 2400)	ρ1	ρ2	Comparison (n = 479)	FPP (n = 1248)	Comparison (n = 211)	FPP (n = 462)
Condom use with male partner	0.01	0.01	37.0	33.0	49.4^‡	76.2^†‡
Condom use with female partner	0.00	0.00	29.9	23.6	40.5	58.1^‡
Syphilis seroprevalence	0.00	0.01	7.0	7.8	10.9	8.2
HSV-2 seroprevalence	0.06	0.07	51.6	45.1	92.9^‡	76.6^†‡
HIV seroprevalence^b	N/A	N/A	N/A	N/A	11.8	8.7
			Comparison	FPP	Comparison	FPP
FSW (n = 2662)	ρ1	ρ2	(n = 774)	(n = 752)	(n = 562)	(n = 574)
Consistent condom use with clients	0.07	0.14	84.5	80.7	98.6^‡	97.0^‡
Condom use with regular partner	0.00	0.00	7.6	5.2	10.8	13.6^‡
Syphilis seroprevalence	0.00	0.02	2.5	3.5	6.8^‡	7.0^‡
HSV-2 seroprevalence	0.03	0.04	87.3	81.0	94.8^‡	89.7
HIV seroprevalence^b	N/A	N/A	N/A	N/A	6.6	3.7
Notes: ^aAdjusting by cluster sampling.
^bAssessed only at follow-up.
^† p ≤ 0.05 for the difference between comparison and FPP group within each assessment.
^‡ p ≤ 0.05 for the difference between baseline and follow-up within each group.
ρ1 = Intracluster correlation coefficient for the 2003 survey with original clusters (cities).
ρ2 = Intracluster correlation coefficient for the 2003 survey with sub-clusters (sites) used in the final analysis.

Statistical analysis

Statistical analysis included the following dependent variables: condom use, HSV-2 seroprevalence, HIV and syphilis diagnosis (0 = No or seronegative; 1 = Yes or seropositive). With regard to condom use amongst MSM, reference is made to the last sexual intercourse according to partner gender, based on available information on the last three sexual relations. Participants having had intercourse with a woman in any of their three most recent sexual relations were included in the model for condom use with a female partner, with priority being assigned to the last relation reported. The same procedure was followed for condom use with a male partner.

In the case of FSW, consistent condom use with clients was assessed using information on condom use with the last three clients. Condom use with a regular partner was also assessed.

Statistical analysis also included the following independent variables: age (continuous); marital status (married/in union, divorced/separated or single); whether the subject had any children (0 = no and 1 = yes); whether the subject's family was aware of his sexual preference (in the case of MSM) or sex work (in the case of FSW) (0 = no and 1 = yes) and participation in any support group (0 = no and 1 = yes). To adjust for differences in socioeconomic status, an index was developed using information on possession of properties and goods, savings and sources of economic income. The index was constructed using principal component analysis with polychoric correlation matrix (Kolenikov and Angeles 2004) and was divided into quartiles, with the first one representing the 25% of participants with the lowest socioeconomic level. The index was established separately for each population (MSM and FSW) and each measurement round.

As the interventions were implemented heterogeneously throughout the cities (in terms of both intensity and coverage), an exposure control variable was established based on the average proportion of participants in each city who reported having attended a condom use demonstration and having been contacted by a peer educator. This variable was used for adjustment purposes only.

Descriptive statistics were obtained regarding socio-demographic characteristics and STI prevalence. To estimate intervention effects, logistic regression models were adjusted on all the outcome variables and including an interaction term between time (pre- and post-intervention) and group (FPP and comparison) and in all cases adjusting by clustering. The odds ratios for the interaction were calculated with the Stata lincom command for linear coefficient combinations. This procedure was adopted to obtain not only the net effect of the intervention at follow-up, but also any differences between the FPP and comparison cities at baseline as well as the effect of time in each group (Ai and Norton 2003; Norton, Wang, and Ai 2004). Only in the case of HIV prevalence did the analysis consider a logistic regression model without the interaction term (timeXgroup) but controlling for the same variables. This was done since HIV sero-positivity data were not collected during baseline.

Analyses were limited to the follow-up participant sample showing the highest baseline participation probability. For this exercise, the minimum participant age at follow-up was estimated by combining the minimum age registered at baseline and the time elapsed between measurements (minimum age + four years). The statistical analyses were done at the individual level adjusting by survey design effects (sites) and with consideration given to the weights mentioned above.

For FSW, multivariate models were adjusted to estimate risk ratios by the formula proposed by Zhang and Yu (1998), in which odds ratios tend to overestimate or underestimate risk ratios when the incidence of an output variable is frequent.

Results

MSM results

At baseline, information was gathered from a total of 2026 MSM in the six cities: 1419 from FPP and 607 from comparison cities. In 2007, 1677 MSM were surveyed. In total, 2506, from both 2003 and 2007, surveys provided complete information for all of the variables and around 4 per cent (n = 94) were excluded from the analysis, as they did not meet the specified age criterion (minimum age reported at pre-intervention + four years). The final analytical sample considered 2400 MSM: 1727 that participated in the 2003 measurement (1248 in the FPP and 479 in the comparison cities) and 673 from the 2007 survey (462 in the FPP and 211 in the comparison cities).

As can be shown for the numbers, reaching MSM for the data collection in the comparison sites was a challenge; numbers were low during pre-intervention survey and even lower for the post-intervention survey. This could affect the validity of the results; to minimise the potential bias that reflects limitations in identifying and recruiting members of the study populations that could be related to the intervention itself, pre-intervention data were used to balance participants’ characteristics between the two groups. Any potential bias was minimised with analyses comparing the differences in changes over time by site (differences of differences). The reported results were compared with alternative analyses, such as fixed and random effects, and obtained consistent results overall (analyses not shown).

Socio-demographic characteristics of MSM

At the baseline survey, participants’ average age was 25 years. The majority reported being single (78.7%), and 24 per cent reported having at least one child. Nearly half of the participants self-identified as bisexual (insertive, receptive or both) (44.8%). The majority (88%) stated that their families were aware of their sexual preference. Most participants indicated that their last sexual partner had been male, although 25 per cent reported having a female partner at last intercourse. Table 1 shows participants’ characteristics by group (comparison and intervention) and survey time. Overall, no difference can be observed in socio-demographic characteristics between the two groups at the time of both surveys, except for marital status and group of self-identification. Detailed information on MSM sexual behaviours and knowledge at baseline is available in a previous publication (Gutierrez et al. 2006a).

Condom use amongst MSM

In general, condom use was reported for a low percentage of participants. At baseline, only 34 per cent of MSM indicated using a condom at last intercourse with a male partner and barely 25 per cent at last intercourse with a female partner.

At follow-up, condom use increased overall twofold compared to the baseline count. Specifically, 68 per cent of those who had had intercourse with a male partner reported condom use, and 50 per cent of those who had had intercourse with a female partner reported condom use. Table 2 shows condom use at both measurements by comparison and intervention group.

Seroprevalence for syphilis, HSV-2 and HIV amongst MSM

At pre-intervention measurement, approximately 8 per cent of MSM were syphilis-seropositive, and 47 per cent were HSV-2 seropositive, indicating that seroprevalence was six times higher for HSV-2 than for syphilis. Co-infection of both was found in 6 per cent of all participants.

At follow-up, the 9 per cent of participants were seropositive for syphilis and 81 per cent (almost double the baseline results) were seropositive for HSV-2. HIV overall seroprevalence was 9 per cent. HIV prevalence within the group of HSV-2 and syphilis co-infected individuals at post-intervention (7.2% of participants) was 26 per cent versus 8 per cent of those who were not co-infected.

Tendency over time and FPP effects on the outcome variables amongst MSM

Table 3 shows the results of the multivariate model. At baseline, participants from the comparison and intervention cities showed no difference in terms of self-reported condom use and STI seroprevalence. On analysing the effect of time on both groups, it was observed that, condom use increased (with male and female partners) both in comparison and intervention cities, consistent with decreasing syphilis seroprevalence but in contrast to the increasing seroprevalence of HSV-2.

Table 3. FPP effects on condom use and STIs in a sample of men who have sex with men (MSM) and female sex workers (FSW) in Ecuador (2003–2007).*

		MSM^§				FSW^£
	Condom use with male partner^a	Condom use with female partner^a	Syphilis	HSV-2	HIV	Consistent condom use with clients^b	Condom use with regular partner^c	Syphilis	HSV-2	HIV
(sample size)	(n = 2008)	(n = 1001)	(n = 2400)	(n = 2400)	(n = 673)	(n = 2662)	(n = 1518)	(n = 2662)	(n = 2317)	(n = 1153)
Differences at baseline (FPP = 1)	0.84	0.67*	1.06	0.84		0.96	0.71	1.59	0.93
	(0.56–1.27)	(0.42–1.05)	(0.59–1.94)	(0.46–1.53)		(0.89–1.01)	(0.41–1.20)	(0.89–2.81)	(0.87–0.98)
Time trend (follow-up = 1)
Comparison	3.69	3.70	0.21	58.71	N/A	1.20	1.28	3.48	1.15	N/A
	(1.46–9.30)	(1.27–10.7)	(0.08–0.54)	(23.9–143.7)		(1.15–1.21)	(0.47–3.19)	(1.28–8.46)	(1.06–1.18)
FPP	12.68	10.63	0.06	28.1		1.20	2.31	2.42	1.14
	(4.59–35.1)	(1.1–103.6)	(0.02–0.19)	(9.9–79.8)		(1.12–1.21)	(0.82–5.47)	(0.84–6.38)	(1.03–1.18)
FPP effect	2.89	1.91	0.32	0.40	0.84	0.98	1.29	1.08	0.93	0.75
	(1.34–6.28)	(0.47–7.8)	(0.14–0.72)	(0.11–1.5)	(0.31–2.31)	(0.93–1.00)	(0.85–1.90)	(0.70–1.67)	(0.85–0.99)	(0.42–1.3)
Notes: *Multivariated models adjusting by age, marital status, have any child, family awareness about sexual preferences or sex work, participates in support group, socioeconomic level and average of exposure to intervention activities by city and considering the design effect of cluster sampling; FPP effects estimated by an interaction term (groupXtime) and odds ratio calculated using lincom command in Stata; significative values in bold; HIV models do not include the interaction term thus FPP = 1.
§ = Results are odds ratios.
£ = Results are risk ratios: RR = RM/[(1–P0) + (P0*RM)].
a = Refers to the last intercourse according to sexual partner gender.
b = Refers to last 3 clients.
c = Refers to last intercourse among those who have regular partner.

At follow-up, FPP participants were three times more likely to report condom use at last intercourse with a male partner than those in the comparison group. Additionally, at follow-up, participants from the FPP cities had reduced odds of having a positive syphilis diagnosis (OR: 0.32; 95% CI: 0.14–0.72).

FSW results

During the pre-intervention measurement, information was obtained from 2093 FSW in the six cities (1063 in FPP cities and 1030 in comparison cities), while in 2007, 1760 were interviewed. Data from 2786 participants with no missing information across variables, including those who took part both in the 2003 and 2007 surveys were used for analysis. Taking into account the analysis’ age restriction (minimum reported age in baseline + 4 years), information was excluded from 124 participants.

The final analytical sample comprised 2662 FSW, 1526 of whom took the 2003 survey (752 in FPP cities and 774 in comparison cities) and 1136 of whom took the 2007 survey (574 in FPP cities and 562 in comparison cities).

Socio-demographic characteristics of FSW

In the pre-intervention survey, the mean age amongst participants was 27.8 years and only 50 per cent of them reported reaching an educational level of middle or technical studies. Half of the participants described themselves as single (49%), while 46 per cent of participants were married or in civil unions, and 86 per cent reported having at least one child. Of all participants, 85 per cent self-identified as a female sex worker, and the mean number of years in that role was five. In general, there were no observable differences amongst participants in the comparison and intervention groups. Participant characteristics by group and time of measurement are found in Table 1. More detailed information about different aspects of FSW activities, clients and partners is available in a previous report (Gutierrez et al. 2006b).

Condom use amongst FSW

Reported condom use with clients was high. Using available information regarding their last three clients, condom use amongst FSW was reported as 82 per cent at baseline. However, only 6 per cent reported condom use at last sexual intercourse with a regular partner.

At follow-up, consistent condom use with clients was reported at 97.8 per cent for all surveyed sex workers, while use with regular partners was reported at levels two-fold higher than baseline, at 12.2 per cent. Table 1 shows condom use rates with clients and regular partners for comparison and intervention groups.

Seroprevalence for syphilis, HSV-2 and HIV amongst FSW

Syphilis was detected in 3 per cent of FSW at baseline, while a large share of participants (84%) were HSV-2 seropositive. At baseline, co-infection for both conditions was identified in almost 3 per cent of all participants.

During follow-up, the prevalence of syphilis amongst FSW was 7 per cent, or twice that reported at baseline, while HSV-2 seroprevalence reached 92.4 per cent. HIV seroprevalence reached 5 per cent in the six cities without a significant difference at follow-up between the group with syphilis/HSV-2 co-infection (HIV prevalence: 4.8%) and the group without co-infection (HIV prevalence: 5.1%). In Table 2, STI prevalence is depicted in terms of comparison and intervention group for both measurements.

Tendency over time and FPP effects on the outcome variables amongst FSW

According to multivariate analysis (Table 3), there were no significant differences at baseline for any output variables between comparison and intervention groups. Time effect analysis for each group, however, indicates a significant increase in condom use with clients and in HSV-2 seroprevalence in comparison, as well as in FPP cities. Syphilis seroprevalence showed a significant increase over time only in comparison cities (relative risk (RR) = 3.4).

No significant effects were identified in condom use or STI seroprevalence in response to FPP intervention, except for HSV-2, which displays a lower prevalence in FPP cities at follow-up, albeit with a marginal effect (Table 3).

FPP effects on a sub-sample of FSW: secondary analysis

Exposure to prevention activities, including demonstrations of correct condom use and contact with peer educators, was high in both MSM and FSW comparison groups, but particularly so for FSW. As can be observed in Figure 1, exposure in City 2 was particularly high, suggesting it may be an inappropriate comparator. For this reason, a new analysis was performed in a sub-sample of FSW that excluded participants from this particular city (Table 4).

Table 4. FPP effects on a sub-sample of female sex workers (FSW) in Ecuador, excluding participants from one comparison city due to potential contamination in the data. Results are risk ratios (CI 95%).*

FSM
	Consistent condom use with clients^a	Condom use with regular partner^b	Syphilis	HSV-2	HIV
(sample size)	(n = 2187)	(n = 1243)	(n = 2187)	(n = 1897)	(n = 908)
Differences between groups	0.97	0.62	1.35	0.96
at baseline (FPP = 1)	(0.91–1.03)	(0.34–1.08)	(0.72–2.49)	(0.89–1.01)
Time trend (follow-up = 1)
Comparison	1.20	2.00	0.88	1.20	N/A
	(1.09–1.22)	(0.54–5.83)	(0.28–2.69)	(1.18–1.21)
FPP	1.15	5.92	0.22	1.21
	(0.67–1.21)	(1.40–12.10)	(0.04–1.11)	(1.19–1.21)
FPP effect	0.91	2.16	0.35	1.05	0.91
	(0.62–1.00)	(1.13–3.65)	(0.16–0.74)	(1.00–1.08)	(0.34–2.21)
Notes: *Risk ratios calculated using the formula RR = RM/[(1–P0) + (P0*RM)]. Adjusted by age, marital status, have any child, family awareness about sexual work, participates in support group, socioeconomic level and average of exposure to intervention activities by city and considering the design effect of cluster sampling; FPP effects estimated by an interaction term (groupXtime); significative values in bold; HIV model does not include the interaction term and thus FPP = 1.
^aConsistent condom use with clients refers to last three clients.
^bCondom use with regular partner refers to last intercourse among those who have regular partner.

Figure 1. Average of exposure to specific intervention activities among MSM and FSW by city.

In contrast with the full-sample analysis, results from the secondary analysis with the aforementioned FSW sub-sample do, in fact, show some positive effects of the FPP intervention (Table 3). During follow-up, participants from the remaining cities were twice as likely to report condom use in their last sexual intercourse with their regular partners (95% CI: 1.1–3.6%). The effect of the battery of intervention activities on syphilis seroprevalence was also apparent, as FSW from intervention cities were less likely to report positive diagnosis compared to participants from the comparison cities (RR: 0.35, 95% CI: 0.16–0.74%).

Discussion

This study presents the results from the impact evaluation of the FPP, an HIV prevention community strategy amongst MSM and FSW in Ecuador. The analysis suggests a positive impact of the FPP for both increased condom use and reduced syphilis prevalence amongst both MSM and FSW. The full intervention package appears to have contributed to modifying risk behaviours amongst KP in Ecuador. These results are consistent with previously reported findings in India, where the FPP was shown to be associated with a higher likelihood of condom use and lower STI seroprevalence in MSM and FSW (Gutierrez et al. 2010).

As a parallel contribution and a complement to previous publication of the present study (Gutierrez et al. 2006a, 2006b), the results of this study portrays a profile of MSM and FSW in Ecuador in more detail than ever before reported. Some relevant elements of each population should be highlighted. In relation to MSM, a sizable percentage of these participants have sexual relations with women as well, regardless of how they identify themselves in terms of their sexual preference. Even more relevant is the fact that condom use with female partners is significantly low.

On the other hand, it is a very positive finding to note a high level of consistent condom use among FSW and their clients. However, condom use with regular partners is extremely low. Improving negotiation skills for condom use with regular partners still represents a challenging task for any prevention effort. MSM and FSW are confirmed once again as a highly vulnerable group but, simultaneously, as one that is susceptible to innovative, intense and comprehensive preventive strategies.

Financing and efforts to promote HIV prevention programmes must be focused on strategies with demonstrated impact and must ensure that unproven prevention programmes include an assessment of impact (Holmes et al. 2010). However, evaluation of large-scale programme implementation remains methodologically complex, as reflected by the complications described in this study.

First and foremost, this study sought to analyse the impact of a set of additional activities beyond standard ones implemented by the national HIV prevention programme in Ecuador. Performing an evaluation in a context in which other prevention activities might be implemented by other players increases the likelihood for such activities to impact the selected comparison groups. For example, a brief literature review found that in addition to the consolidation of the aforementioned Sex Worker Network in Ecuador, in the time span of the FPP (2003–2006), other large-scale HIV prevention initiatives were also implemented. The UNIVIDA Project developed by CARE International between 2005 and 2010 in 11 provinces in Ecuador and the activities by The Global Fund to Fight AIDS, Tuberculosis and Malaria deserve special mention.

Under the conditions described above, the output variables under examination in this study were no different for the FSW sample between intervention and comparison groups; in both condom use improved dramatically compared to baseline. A simple analysis to explore exposure to two prevention activities (condom use demonstration and contact by peer educator) identified an important amount of prevention activity in comparison cities. In fact, control cities had levels of exposure to these two prevention activities as good as intervention cities. Thus, what we are essentially evaluating is the potential benefits of prevention activities when they are done in a participatory manner with community, which is the key element in the FPP, in comparison to national standard initiatives that are individual or group-based. However, it is likely that such prevention activities in comparison cities may have reduced the measured effect of the FPP particularly among the FSW population.

In addition, given that the FPP is a strategy whose coverage and acceptance relies on community involvement, prevention activities were implemented heterogeneously, thus representing directed prevention efforts at a different scale. A major complication especially for the evaluation of the intervention's effectiveness was the fact that no quantitative data was collected about the number of KP members reached with prevention activities at each city. This lack of records on the precise exposure levels to implemented FPP activities made it impossible to identify which elements in the prevention strategy were more successful in producing impact. Subsequent studies should address this challenge and must identify tools that allow access to an information system about strategy implementation and can generate objective data about spread, intensity and even quality of programmes (Laga et al. 2012).

A central aspect in evaluating the FPP impact amongst FSW is that increased condom use with clients was not demonstrated. During baseline measurements, condom use with clients was reported by an already high proportion of participants, thus questioning the feasibility of demonstrating a significant increase in use. Nonetheless, the fact that condom use did not decrease from baseline high levels can be regarded as a positive outcome.

Although a statistically significant effect was not shown for HIV and HSV-2 seroprevalence, data still suggest a protective trend due to FPP interventions both in MSM and FSW. While the prevalence of HSV-2 cases increased over time, overall, the increase was less for intervention groups. It must also be noted that according to the core principle of this programme, the purpose of saturating sites with intervention activities is to dampen the rapid increase in STI prevalence amongst the overall population. The strikingly large proportion of participants infected with HSV-2, amongst these populations, is a relevant fact anyhow.

Due to all these factors, the initial impression of FPP effectiveness in particular for the FSW population was somewhat disheartening, at least for the expected statistical impact. In reality, there is some sort of trade-off between statistical significance and relevance for decision-making; although there is an accepted necessity to document the effectiveness of HIV prevention programmes with statistically robust methodologies (Laga et al. 2012), even non-significant results can be useful for informing decision-making when evidence is extremely scarce.

Other limitations of this study should be mentioned. The important reduction in the MSM sample during follow-up (one-third of baseline observations) and in the overall evaluation might indicate that the follow-up survey team was not sufficiently successful in identifying members of this population. Regardless of the cause, it reduced the power of the evaluation and represents a mayor limitation as this reduced the generalisability of the findings. Also, since this was not an individual panel study and as data were captured on a completely anonymous basis, it cannot be assured that information was obtained from the same participants in both measurements, posing a potential for bias.

Lastly, regarding randomisation, it is important to highlight that while random assignment of the intervention was done at the city level; the adjustment of the standard errors for the design effect was done considering a set of virtual clusters (or sub-clusters within cities), not the cities. This approach was taken for two reasons: (i) to allow for the multivariate analysis that would not be feasible with only six clusters and (ii) to recognise heterogeneities within key populations that might be related to relevant behaviours. The virtual clusters refer to meeting or working site categories that reflect specific profiles of MSM or FSW within each city. Although this fact challenges the definition of a pure randomisation exercise, intervention and control cities were balanced as shown by the comparison of participants’ socio-demographic and behavioural variables. Also, estimation of the intracluster correlation coefficient both at the city level and the virtual clusters proved to be small and somehow similar. In combination, these elements give some evidence of the success of randomisation; and thus, we do consider this study as a randomised community cluster, although not in the most orthodox form.

The fight against HIV requires identifying which strategies have demonstrated impact and which ones do not. In the LA region, evidence on the impact of HIV prevention interventions is almost nonexistent. This study represents a pioneering effort by reporting results of effectiveness from an intervention package for MSM and FSW in Ecuador. Altogether, the FPP results suggest that a prevention strategy that includes KP in its design and implementation is able to achieve community-level participation and may lead to reductions in risk behaviours amongst MSM and FSW and, in the long run, contribute to preventing new HIV infections. KPs are central to the HIV response and cannot be left out (International HIV/AIDS Alliance 2010).

More importantly, the experience at evaluating the FPP illustrates some practical lessons to improve future efforts of HIV prevention and to guarantee more robust and adequate evaluations. This evaluation required a project-specific design capable of adapting to environmental shifts. An evaluation must be rigorous but it is not an independent task within an unvarying social framework. Thus, it is necessary to integrate monitoring to inform management and impact evaluation and to coordinate actions between the programmatic and evaluation teams regarding financing, programme design, implementation and assessment (Hayes et al. 2010; Laga et al. 2012). All of these items require close relations and collaboration amongst researchers, implementers, local players and the KP. It is also important to find effective ways to measure exposure to prevention activities at the community and individual levels in addition to the records obtained from implementing organisations. These features may ensure the quality of impact evaluations of potentially successful strategies in HIV prevention.

Acknowledgements

We thank all participants from Ecuador for their valuable information. We would also like to thank the International HIV/AIDS Alliance and their local implementing partners Kimirina. The FPP was supported by a grant from the Bill and Melinda Gates Foundation.