Implementation Science Protocol: evaluating evidence-informed interventions to improve care for people with HIV seen in Ryan White HIV/AIDS program settings

ABSTRACT

In 2017, the Health Resources and Services Administration’s HIV/AIDS Bureau funded an Evaluation Center (EC) and a Coordinating Center for Technical Assistance (CCTA) to oversee the rapid implementation of 11 evidence-informed interventions at 26 HIV care and treatment providers across the U.S. This initiative aims to address persistent gaps in HIV-related health outcomes emerging from social determinants of health that negatively impact access to and retention in care. The EC adapted the Conceptual Model of Implementation Research to develop a Hybrid Type III, multi-site mixed-methods evaluation, described in this paper. The results of the evaluation will describe strategies associated with uptake, implementation outcomes, as well as HIV-related health outcomes for clients engaged in the evidence-informed interventions. This approach will allow us to understand in detail the processes that sites undergo to implement these important intervention strategies for high priority populations.

KEYWORDS:

• HIV
• Proctor Model
• mixed methods
• Hybrid Type III

Introduction

State of the HIV epidemic

Advances in HIV medicine, coupled with innovative interventions to successfully engage people with HIV in care, are important tools to end the HIV epidemic. However, intervention implementation gaps have resulted in persistent health disparities including differential rates in HIV diagnosis, linkage to and retention in care, and viral suppression (Centers for Disease Control and Prevention, 2019b). These disparities particularly affect populations including Black men who have sex with men (BMSM), transgender women, individuals who have co-occurring mental health and/or substance use issues, and those with a history of trauma. To close these gaps in HIV care, strategies are needed to accelerate the dissemination, uptake, and evaluation of effective interventions.

In 2017, the Health Resources and Services Administration’s (HRSA) HIV/AIDS Bureau launched the “Using Evidence Informed Interventions to Improve Health Outcomes for People Living with HIV” Initiative (E2i). E2i aims to address the disparities in access to and retention in effective care and treatment for people with HIV served by HRSA’s Ryan White HIV/AIDS Program (RWHAP) by funding the implementation of evidence-informed interventions specifically for underserved groups. This paper describes the evaluation of E2i, based on an established framework, the Conceptual Model of Implementation Research (also known as “The Proctor Model”) as it is currently being conducted by the Evaluation Center (EC), located at the University of California, San Francisco’s Center for AIDS Prevention Studies (CAPS).

Background

Implementation Science and the RWHAP

The RWHAP provides direct medical and support services to more than half of all people with diagnosed HIV in the U.S. by funding HIV service providers as the payor of last resort. The RWHAP funds: eligible metropolitan areas (Part A), states and territories (Part B), early intervention services and capacity development at the local level (Part C), and specialty medical care for families, women, infants, children and youth with HIV (Part D). RWHAP Part F funds clinical training, technical assistance and demonstration projects focused on evaluating the adaptation, implementation, utilization, cost, and health-related outcomes of innovative treatment models while promoting the dissemination and replication of successful interventions in real world settings, such as the clinics and hospitals receiving Ryan White Parts A-D funding (Health Resources & Services Administration, 2019). Currently, the disproportionate burden of HIV on Black MSM and transgender women, and the lack of effective interventions for them, are central challenges to improving outcomes (Melendez & Pinto, 2009; Mizuno et al., 2015; Rosenberg et al., 2014; Singh et al., 2017). Furthermore, many people with HIV face greater need for behavioral health services, and for services that address prevalent experiences of traumatization (Health Resources & Services Administration, 2015; Mugavero et al., 2006). Thus, HRSA funded the E2i Initiative to support the adaptation, implementation, and evaluation of evidence-informed interventions for these high priority populations.

Transgender women

Transgender women are disproportionately vulnerable to HIV (Nemoto et al., 2004). A recent Centers for Disease Control and Prevention (CDC) analysis demonstrated that transgender people had the highest rates of HIV diagnoses among all groups tested (Centers for Disease Control and Prevention, 2015). Current efforts to provide effective access, care, and treatment to transgender women with HIV are not as successful as with other populations. Transgender women with HIV are less likely to be receiving antiretroviral therapy (ART) compared to cisgender men and women (Melendez & Pinto, 2009), and transgender women on ART face greater challenges to adherence and have fewer positive interactions with healthcare professionals (Sevelius et al., 2010). Addressing these challenges is critical to altering the pattern of HIV-related disparities that lead to disproportionately poorer health outcomes.

Black men who have sex with men

In 2017, MSM represented 70% of all HIV diagnoses in the US, and Black MSM have higher HIV prevalence rates and lower viral suppression than all other racial/ethnic groups (Centers for Disease Control and Prevention, 2019a). Compared with other MSM, BMSM have higher rates of undiagnosed HIV infection (Millett et al., 2011), are less likely to be linked to care within 90 days (DeGroote et al., 2016), and are less likely to be on and adhere to ART (Rubin et al., 2010). To reduce disparities, evidence-informed interventions are needed to address multiple barriers preventing Black MSM from successfully engaging in HIV healthcare (Maulsby et al., 2013).

Behavioral health integration into primary medical care

People with HIV have high rates of mental health issues; approximately 50% of adults treated for HIV also have symptoms of a psychiatric disorder and 13% having co-occurring mental illness and substance-use disorder (Higa et al., 2012). People with HIV who experience depression are more likely to have higher viral loads, more anxiety, and are more likely to have a problem with substance use (Bengtson et al., 2019; Garey et al., 2015; O’Cleirigh et al., 2015). These rates of mental illness present a serious challenge in implementing behavioral health interventions; many patients with HIV also have complex mental health issues (Gaynes et al., 2008). Current mental illness, especially depression and substance abuse, predict lower ART adherence, a greater likelihood of failing ART, and increased mortality (Substance Abuse and Mental Health Services Administration and Health Resources and Services Administration, 2016). Successfully engaging people with mental health issues in HIV care requires evidence-informed approaches to fully integrate behavioral health interventions into primary medical care.

Identifying and addressing trauma

According to the HRSA Center for Integrated Health Solutions, people with HIV are as much as twenty times more likely to have experienced trauma than the general population (Substance Abuse and Mental Health Services Administration and Health Resources and Services Administration, 2016). Being part of a marginalized community may increase the risk of exposure to potentially traumatic events. In order to improve healthcare engagement among people with HIV who have experienced trauma, evidence-informed interventions must be implemented that address the mental health sequelae of trauma, diminished ART adherence, more frequent opportunistic infections, and higher risk of HIV-related mortality (Schafer et al., 2012).

E2i initiative Description and evaluation

E2i is a four-year initiative funding 26 RWHAP sites implementing 11 evidence-informed interventions in the four focus areas highlighted above (Keuroghlian et al., 2018). Sites differ in the services they offer (clinical and non-clinical), the larger context in which those services are provided (university hospitals, federally qualified health centers, community-based organizations), and where they are located throughout the U.S. (rural and urban settings). The Fenway Institute serves as E2i’s Coordinating Center for Technical Assistance (CCTA), in partnership with AIDS United, to provide intervention training, toolkit development, technical assistance, and intervention funding to the sites. UCSF CAPS is the initiative’s EC and is developing and executing a Hybrid Type III implementation study (Curran et al., 2012) and multi-site evaluation. In this paper, we describe the EC’s comprehensive evaluation, based on the Proctor Model.

Methods

Comprehensive evaluation Plan

The Proctor Model (Proctor et al., 2013) provides a framework for understanding the process and success of intervention implementation and how specific strategies of that process impact outcomes. The model offers a framework for the simultaneous evaluation of intervention strategies, implementation strategies, and three categories of outcomes. For the E2i Initiative, the EC designed a multi-method approach to assess each of Proctor’s primary concepts, including intervention and implementation strategies and all three levels of outcomes. Below, we describe the procedures, measures, and planned analyses for the E2i comprehensive evaluation and show how they correspond to key components of the Proctor Model.

Procedures and measures

From Figure 1, Intervention Strategies correspond to the core elements of the 11 evidence-informed interventions implemented by the E2i sites (Table 1). Implementation Strategies are the methods used to enhance the adoption, implementation, and sustainability of an intervention; Implementation Outcomes are the effects of deliberate and purposive actions to implement a new intervention; Service Outcomes are the ideal qualities of health care; and, Client Outcomes are the direct impacts on the participants (Proctor et al., 2013). We designed the evaluation to utilize multiple data sources, measured at different levels of the organization and at multiple time-points throughout the implementation process. Data sources include the E2i site leadership, the intervention frontline staff, intervention participant characteristics, and reflections of the CCTA and EC staff on the intervention implementation process. The evaluation was designed to minimize data collection burden on intervention staff and requires no collection directly from participants (see Table 2 for sources and timing by Proctor concept).

Figure 1. E2i Adaptation of the Conceptual Model of Implementation Research (“The Proctor Model”).

Table 1. Evidence-Informed Interventions.

Focus Area	Intervention	Description
Transgender Women	Healthy Divas (Sevelius et al., 2019; Wilson et al., 2009)	Intervention developed to address the multiple and unique barriers to engagement in HIV care experienced by transgender women with HIV by integrating two complementary theoretical models: the Gender Affirmation Model and the Model of Health Care Empowerment.
	T.W.E.E.T. Transgender Women Engagement and Entry To Care Project (Rebchook et al., 2017)	Aims to identify newly diagnosed transgender women of color and link them to care, identify HIV-positive transgender women of color who are currently out of care and link them to care, enroll identified clients into the TL-Teach Back Intervention, and identify and utilize Peer Leaders.
Black Men who have Sex with Men	PEERS MI Motivational Interviewing with Peer Outreach (Naar-King et al., 2009; Naar-King & Suarez, 2011)	Intervention to increase motivation to engage in treatment and improve retention in primary care.
	Project Connect and Retention through Enhanced Contacts (Mugavero, 2008, 2016)	Intervention where recently diagnosed patients have a scheduled orientation visit within 5 days and screened for referrals to substance abuse, mental health, and other ancillary services, as well as facilitating rapid institution of prophylactic medications when necessary.
	TXTXT: Text Messaging Intervention to Improve Antiretroviral Adherence among HIV-Positive Youth (Garofalo et al., 2016)	An individual-level intervention using 2-way text messages to increase medication adherence
Behavioral Health Integration	Buprenorphine Treatment (Altice et al., 2011)	Buprenorphine opioid abuse treatment
	Collaborative Care Management (Pyne et al., 2011; Unützer et al., 2002)	A model of depression care that requires collaboration between the patient's primary care physician, care manager and consulting psychiatrist
	Screening, Brief Intervention, and Referral to Treatment (Babor et al., 2007; Dawson-Rose et al., 2017; Kaner et al., 2009)	Practice in which a health care provider screens for problematic use of alcohol and/or illicit drugs, provides some brief counseling, and refers to treatment as needed.
Trauma-Informed Care	Cognitive Processing Therapy (Resick et al., 2017; Resick et al., 2015)	A type of cognitive-behavioral therapy that focuses on repairing the damage done by a traumatic event
	Trauma-Informed Approach/Coordinated HIV Assistance and Navigation for Growth and Empowerment (Jain et al., 2016)	Systems level change program utilizing methods of care to ensure that barriers including mental health conditions, substance use, history of trauma, low health literacy, and lack of support services, can be addressed
	Seeking Safety (Empson et al., 2017; Najavits et al., 1998)	Cognitive-behavioral group therapy program designed to address co-occurring substance use and PTSD

Table 2. Proctor Model Concept by Data Source.

	Data Source
Proctor Model Concept	Document Review	Session Observation	Organizational Assessment	Intervention Exposure	Medical Records	Costing
Timing	Baseline and every 6 months	Twice a year	Baseline and every 6 months	On-going	Every 6 months	Annual
Intervention Strategies	Evidence-informed interventions
Implementation Strategies	Systems Environment Organizational Group/Learning Supervision Individual	Systems Environment Organizational Group/Learning Supervision Individual	Organizational Group/Learning Supervision Individual
Implementation Outcomes	Feasibility Fidelity Penetration Acceptability Sustainability Uptake Appropriateness	Feasibility Fidelity Penetration Acceptability Sustainability Uptake Appropriateness	Feasibility Fidelity Penetration Acceptability Sustainability Uptake Appropriateness	Fidelity Uptake		Cost
Service Outcomes				Equity Timeliness	Effectiveness Equity	Efficiency
Client Outcomes					Symptomatology

Secondary document review

We are using secondary document review to describe Intervention Strategies. Initially, we conducted a review of initial program documents generated prior to intervention implementation. The CCTA consulted with the original intervention developers to prepare one logic model per intervention and to confirm each intervention’s core elements. The CCTA then refined the logic models in discussions with the developers and the individual sites so that each site had a tailored logic model. The CCTA and the developers then worked with the sites to formalize comprehensive implementation plans. The results of this baseline document review allowed the EC to describe the sites’ intervention strategies and serves as the rubric to assess implementation strategies and implementation outcomes.

For assessment of the implementation strategies and outcomes, we are using additional Secondary Document Review to describe sites’ implementation progress, intervention uptake, and mid-course changes. Throughout the implementation period the EC regularly collects secondary documents for analysis including: training materials, monthly site monitoring call notes, quarterly reports, action plans, site visit reports, and any other project-related documents. In this secondary document review, we will be assessing the sites’ larger contexts and describing the content and reasons for intervention adaptations.

Document review measures

Using a template, we extract relevant information from each source document on an on-going basis. Every five months, the EC conducts a within-site content analysis of data abstracted and produces an implementation snapshot. We will use the three implementation snapshots (Months 8, 13, 18) to facilitate a longitudinal analysis of implementation progress. We will identify common implementation themes across sites within a focus area and will identify notable similarities and differences across focus areas.

Learning session observations

Sites engage in bi-annual Learning Sessions (LS) comprised of E2i leadership and front-line staff of all sites to support intervention implementation. With origins in continuous quality improvement efforts (Headrick et al., 1996), these LS are meetings bringing together key implementers to participate in a two-day collaborative learning process, at times grouped according by focus area and other times by commonalities across sites (intervention, setting, etc.). LS provide the EC an opportunity to observe how site representatives interact with, learn from, and teach one another. The formal aspects of the LS observations center on assessing implementation progress, challenges, and processes. We specifically attend to information concerning how the sites use the LS process to change their implementation plans and shift their implementation strategies. Observations conducted during the LS will be structured to also capture site perspectives on the eight implementation outcomes.

Learning session observation measures

Using our own observation tool, members of the EC team take notes during the LS which are then used to generate field notes. These field notes are systematically analyzed following typical conventions of coding (Ryan, 2006), theme identification, and interpretation all focused on understanding implementation experiences.

Organizational assessment

The EC asks each site’s Project Director to complete an Organizational Assessment online containing items to assess implementation strategies as well as each of the eight implementation outcomes. The baseline assessment took place during the formative phase, prior to implementation, and administration continues every six months thereafter through the funding period.

Organizational assessment measures

The assessment asks site directors about the culture of the organization, staffing patterns, placement of the intervention relative to other services, anticipated changes to workflows, and other elements relevant for implementation capacity. Items are modified on subsequent administrations to capture change since the prior administration. The assessment also contains the Checklist for Assessing Readiness for Implementation for Evidence-Based Practices (Barwick, 2011). This assessment will allow a comparison of the site’s capacity and readiness to implement the intervention to the specific strategies and desired outcomes, measuring all eight implementation outcomes.

Intervention exposure

In order to be able to describe participants and to understand their receipt of intervention services, sites complete a single enrollment form for each intervention participant. Sites record demographic information and assign a Unique E2i Identifier Number to each participant. This identifier is used on all patient/client forms to link enrollment, intervention exposure, and medical records. Measurement of intervention exposure is completed upon each encounter with E2i participants and contributes to our assessment of the degree to which the program was implemented (uptake) as intended (fidelity) as well as any adaptations to intervention delivery (frequency, dosage, duration, delivery method). We will be able to determine the time elapsed between enrollment and first exposure to the intervention (timeliness) and determine that the intervention is targeted to those who it would best serve (effectiveness) and without differences in delivery (equity) based on participant characteristics.

Intervention exposure measures

A single enrollment form documents the participant’s year of birth, sex assigned at birth, current gender identity, racial/ethnic identity, date of entry into care, and date of E2i intervention enrollment. Intervention Exposure data are collected contemporaneously with participation in the intervention. These include the date of participation, category of staff administering the intervention, interaction modality, types of activities administered, and any activity outcomes. Using the documents and results from the baseline Secondary Document Review, we developed a grid of the Core Elements and noted common and unique activities and outcomes of each of the 11 interventions. These form the response options for this measure.

Medical record data

We collect information about participants’ HIV healthcare utilization for 12 month periods before and after enrollment. Sites use a combination of Ryan White Service Report data or abstracted medical record data to measure linkage to and retention in care, ART prescriptions, and viral suppression. Before submitting these health data to the EC, all records are fully de-identified, using only the participants’ Unique E2i Identifier Numbers. These clinical indicators will assess Proctor’s outcome symptomatology.

Medical record measures

Information from medical data include: HIV care visit dates, CD4 and viral load counts, and dates of ART prescriptions. These data permit us to measure linkage (primary care visit), treatment (ART prescription), retention (at least one visit in each of two 6-month periods, >60 days apart), and viral suppression (viral load < 200).

Costing data

The EC will conduct annual cost assessments (cost) to measure the financial and human capital resources directed toward intervention implementation, including total monetary expenses, in-kind donations, and personnel hours for intervention adaptation and implementation. This analysis will also allow us to assess the cost per person of implementing the intervention (efficiency).

Costing data measures

The goal is to describe the true intervention implementation costs regardless of funding source. Examples of expenses included in these assessments are those related to: personnel effort to develop intervention protocols; electronic medical record modifications necessary to implement the intervention; provider and staff trainings; routine performance monitoring and feedback that are integral to an intervention model (e.g., reviewing clinic record data to perform panel management); and personnel and care-related expenses that are above and beyond levels required before intervention implementation. The costs of intervention implementation do not include expenses and personnel dedicated to the development and implementation of evaluation and routine costs of care that exist independently of the intervention.

Analyses

In this mixed-methods study, we are using qualitative methods to document intervention strategies, quantitative methods to assess service and client outcomes, and mixed methods for a comprehensive understanding of implementation strategies and outcomes.

Analyses of qualitative data, including all documents and LS field notes, consist of organization and examination following the principles of thematic and Framework Analysis (Ritchie & Spencer, 1993). These techniques are useful for analysis of qualitative data when some a priori domains are defined based on the research questions of interest; in this case, the Proctor domains (Powell et al., 2015). Once a priori domains are defined, initial coding of the data consists of reviewing source documents and identifying sections of the text that correspond to the a priori domains and developing new domains as needed. To organize and sort data, all materials are entered into Dedoose (Version 5.0.11) a web-based qualitative data analysis platform. Excerpts associated with key codes are summarized and tabled for comparison and theme identification to inform the evaluation of intervention and implementation strategies, in addition to implementation outcomes.

As noted above, preliminary analyses of quantitative data occur on an ongoing basis. The quality of the quantitative data is monitored throughout the initiative. A combination of monthly and specialized data quality reports, identifying missing and potentially inaccurate data values, are generated and provided to the sites. Using these reports, we work with the site data teams to correct errors and prevent them from recurring. Regular technical assistance, training, and documentation is provided to keep sites informed on best practices for data collection.

We conduct analyses of the Organizational Assessment at the end of each assessment period and disseminate information on enrollment and intervention exposure for each intervention and site monthly. Additionally, we disseminate results from medical chart abstraction every six months and information on the costs of each intervention annually. Analyses of our quantitative data will employ linear and non-linear mixed models to evaluate the quantitative relationship between components of the Proctor Model measured at the organizational and individual levels while controlling for repeated measures. For each intervention, our primary analyses will identify implementation strategies, implementation outcomes, and service outcomes that predict our primary client outcome, viral suppression.

Overall interpretation of our results will consist of an integration of qualitative and quantitative findings, using Convergence Model Triangulation (Creswell & Plano Clark, 2011). Each method is given equal weight in terms of importance and contribution. In both the qualitative and quantitative approaches, our multi-level data collection is on-going and examines consistency and differences across methods. This approach will permit us to compare the results of both methods for assessing implementation strategies and implementation outcomes, as well as connect the results from the assessment of implementation strategies to the service and client outcome data.

Discussion

This implementation science approach to evaluation, using the Proctor Model, will allow us to understand in-depth how sites are implementing these important evidence-informed interventions for high priority populations. Because we are focusing on implementation strategies explicitly, we hope to have results to inform accelerated progress in closing gaps between knowing what works and applying it broadly in HIV care settings. Other organizations considering future adoption of these, or similar, evidence-informed interventions can learn from the implementation experiences of the sites in this Initiative. The EC’s findings may help guide future organizations to select appropriate intervention strategies, avoid common implementation pitfalls, and maximize their opportunity for implementation success, thereby improving outcomes for people with HIV.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Health Resources and Services Administration: [Grant Number U90HA31099].