An evaluation of an employment assistance program focused on people living with HIV in Toronto, Canada

ABSTRACT

Unemployment is more common among people living with HIV (PLWH) compared to the general population. PLWH who are employed have better physical and mental health outcomes compared to unemployed PLWH. The main objective of this mixed-methods study was to conduct a program evaluation of Employment Action (EACT), a community-based program that assists PLWH in Toronto, Ontario, Canada to maintain meaningful employment. We extracted quantitative data from two HIV services databases used by EACT, and collected qualitative data from 12 individuals who had been placed into paid employment through EACT. From 131 clients included in the analysis, 38.1% (n = 50) maintained their job for at least 6 weeks within the first year of enrollment in the EACT program. Gender, ethnicity, age, and first language did not predict employment maintenance. Our interviews highlighted the barriers and facilitators to effective service delivery. Key recommendations include implementing skills training, embedding PLWH as EACT staff, and following up with clients once they gain employment. Investment in social programs such as EACT are essential for strengthening their data collection capacity, active outreach to service users, and sufficient planning for the evaluation phase prior to program implementation.

KEYWORDS:

• HIV
• AIDS
• employment
• social determinants of health
• program evaluation
• mixed-methods

Introduction

Employment is a key social determinant of health (Vancea & Utzet, 2017). Being employed contributes to income security, improves accessibility to stable healthcare, and influences self-esteem and social connectedness. Among people living with HIV (PLWH), employed individuals are shown to have better physical and mental health outcomes compared to those who are unemployed (Cooper et al., 2017; Rueda et al., 2012). A key benefit of employment for PLWH in Canada is an increased likelihood of accessing employment-based drug benefits, as Canada lacks universal pharmacare (Rachlis et al., 2018; Yoong et al., 2018). For example, a study of PLWH in Ontario, Canada found those with employment-based drug coverage were more likely to have a suppressed viral load when compared to PLWH who relied on a public health insurance formulary (Rachlis et al., 2018). Despite the benefits associated with employment for PLWH, the rate of unemployment among the 60,000 Canadians living with HIV (45–65%) is much higher when compared to the rate in the general population (5–10%) (Longobardo & Oliva-Moreno, 2018). This may be due to the unique barriers PWLH face when attempting to obtain adequate employment, such as stigma, discrimination, and mental health conditions linked to their HIV status (Perri et al., 2021).

One of the very few employment programs in Canada that are tailored to meet the needs of PLWH is Employment ACTion (EACT), which is run by the AIDS Committee of Toronto (ACT) (AIDS Committee of Toronto, 2021). EACT is funded by the Ontario Ministry of Community and Social Services, with in-kind support from ACT. EACT is designed to provide a range of supports to job-seeking service users who are also accessing Ontario Disability Support Program (ODSP), a social assistance program providing income and employment supports to eligible Ontario residents living with disabilities (Ontario Disability Support Program, 2020). Developed in 1999 as a collaboration between ACT and the Toronto People with AIDS Foundation (PWA), EACT had worked exclusively with people living with HIV and recently expanded their program eligibility to anyone receiving income supports from ODSP, regardless of HIV status.

The goal of EACT is to assist clients with attaining and maintaining meaningful employment by offering a range of services that include career planning, resumes and cover letter customization, interview skills coaching, provision of job search resources, and referral to health and employment resources in the community. Examples of experiences gained through involvement with EACT include participating in paid employment opportunities and internships to build Canadian work experience. The client-centered and interdisciplinary approach provided by EACT aligns with the integrated employment service models described in the literature, which draws on multiple structural interventions (e.g., housing and health-care integrated programs) in response to the inter-related challenges faced by people living with chronic illnesses (Conyers et al., 2021).

This study sought to evaluate the effectiveness of the EACT program and identify the program’s key facilitators and barriers. This study was organized in partnership between ACT and the Upstream Lab, a research group focused on addressing social determinants of health based in Toronto. It is part of a larger community-based research initiative in Toronto and Ottawa to address unemployment among PWLH in Canada by developing partnership between community leaders and researchers.

Methods

We approached this study as a program evaluation, with the perspective of both “insiders” (i.e., people involved with the design and implementation of the program, or as clinicians who have had patients make use of the program) and as “outsider” (i.e., academics and researchers). We employed a mixed-methods design comprised of quantitative analysis of HIV services data and qualitative interviews (Shorten & Smith, 2017). Specifically, we adopted a sequential approach in which quantitative HIV services data were first analyzed to measure the effectiveness of the EACT program in supporting their clients to maintain employment, followed by qualitative interviews with the EACT clients to contextualize the barriers and facilitators to effective program delivery (Ivankova et al., 2006).

This research study was approved by research ethics boards at Unity Health Toronto, Ottawa Health Sciences Network, and the Bruyère Institute.

Quantitative data and analysis

Quantitative data was provided to the study team by EACT staff. This data was extracted from two databases: the Ontario Community-Based AIDS Service and Evaluation (OCASE), a client records management tool managed by the Ontario HIV Treatment Network (OHTN), and an internal database used to report outcomes to the Ministry of Community and Social Services, Government of Ontario. The OCASE database contained information on demographics, employment goals, and session case notes. The internal database contained dates which indicated whether clients had been employed for 6 and 13 weeks (milestones related to program funding), hourly wage, and total hours employed. Data from April 1, 2014 to March 31, 2019 were extracted. The earlier cut-off date was chosen as it corresponds to the time when ACT replaced its intake forms with the online standardized forms developed by the OHTN and Ministry of Health to be used by all AIDS service organizations in Ontario. Prior to this date, ACT used agency-specific intake forms with open text fields that were not suitable for analysis.

To be included in the analysis, the clients must have had a recorded enrollment date and had been enrolled between April 1, 2014 and April 1, 2018. Individuals who had previously entered the program before April 1, 2014 were excluded to ensure that every client included in the analysis were only exposed to the program once. Similarly, if the client entered the EACT program more than once between 2014–2018, only the first entry was used in the analysis.

The main outcome of interest was a successful job placement, which is defined by the authors as maintaining employment for at least 6 weeks within 1 year of enrollment in the employment assistance program offered by EACT. We chose the 6-week employment milestone since the actual start date of employment was not available in the reporting form to the Ministry of Community and Social Services and any employment that lasted less than 6 weeks was not recorded in the provided database.

Descriptive analysis was conducted for all clients who met the inclusion criteria on the following variables: ethnicity, gender, age, sexual orientation, immigration status, first language, housing situation, education level, medication usage, and length of HIV diagnosis. All the variables, except for age and length of HIV diagnosis, were comprised of many response options. Except for gender, the responses were aggregated into a smaller number of categories to reduce the number of empty cells or cells with small values (Appendix 1).

Bivariate analysis with pairwise deletion was used to compare the characteristics of clients who maintained employment to those who did not on four variables: gender, ethnicity, age, and first language. Bivariate analysis was not conducted in the other variables due to a large proportion of missing values among the participants (>10%). Outcome difference across genders was tested using Fisher’s exact test, given the small values in each cell. Chi-squared test of goodness fit was used to conduct the bivariate analysis on the other categorical variables (e.g., ethnicity and first language). Age was the only continuous variable and the difference in means was evaluated using Student’s t-test. All analyses were performed using R (version 3.62).

Qualitative data and analysis

To participate in the interview, individuals had to be between the ages of 18–64 and had to have been successfully placed into employment through EACT, defined as having been involved in any paid employment for at least 6 weeks following the initiation of employment. At the time of the interview, some participants were no longer employed. To recruit participants, posters were put up at ACT and email invitations were sent to past clients of EACT. Recruitment took place from February to October 2019.

In-person interviews were conducted using an interview guide (Appendix 2) by a member of the study team (ACN). The guide was developed in collaboration with various stakeholders including AIDS service agencies representatives, HIV-focused clinicians, and those with lived experiences. Following informed consent, the interview explored perceptions of the facilitators and barriers to effective program delivery by EACT along with recommendations to improve the program effectiveness. Recruitment continued until saturation was reached in themes related to these areas. Interview data were analyzed by two members of the study team (MP, ACN) following Braun and Clarke’s approach to thematic analysis (Braun & Clarke, 2006). Transcripts were professionally transcribed. Team members (MP, ACN) reviewed the data, systematically coded the transcripts using NVivo 11, and developed an initial coding framework where the codes were collated into broad themes. The coding framework used in this analysis was expanded and refined in an iterative fashion through multiple meetings (Braun & Clarke, 2006).

Results

Quantitative results

There were 502 active EACT clients between April 1, 2014 and March 31, 2019. Among them, 23.4% (n = 61) did not have a program start date, 62.5% (n = 163) were enrolled in the program prior to April 1, 2014, and 14.2% (n = 37) were enrolled less than 1 year of the study cut-off date, resulting in 241 clients who met the inclusion criteria. From those, 45.6% (n = 110) did not have any information on their employment outcome, leaving 131 clients with sufficient data to be included in the analysis. Among them, 38.1% (n = 50) were able to maintain their job for at least 6 weeks within the first year of enrollment in the employment assistance program. Comparing the characteristics of participants with (n = 131) and without (n = 110) information on their employment outcomes, we found that people of color, participants who identified as heterosexual, and older participants were more likely to have missing outcome data (Appendix 3).

Table 1 presents the results of both descriptive and bivariate analyses. Missing outcome data across the co-variables ranged from 0.8% in gender to 63.4% in education level. Most participants who reported on gender identified as male (87.7%) and the overall average age was 41.2 (SD: 9.4). Most participants also identified as LGBTQ2S + (89.2%). The ethnicity distribution was evenly split between White and non-White (53.8% and 46.2%, respectively). 46.9% of participants reported English as their first language. In terms of immigration status, over 80% had permanent residence or citizenship. 23.7% of the clients had at least some post-secondary education and a small number (6.6%) did not have stable housing. Among those with the available information (n = 54), the average length of time since HIV diagnosis was 10.9 years (SD: 7.7). Data on anti-retroviral medication was available for about two-thirds of the clients (n = 87). Among them, about 72% were on anti-retroviral medication.

Table 1. Results from descriptive and bivariate analyses.

Variables	Groups	Retained Job (N: 50)	Did Not Retain Job (N: 81)	Total N (%)	Proportion after excluding the missing value (%)	Statistics^a
Sociodemographic Variables
Gender^b	Female	5	10	15 (11.5%)	11.5%	p-value = 0.8654
	Male	45	69	114 (87.0%)	87.7%
	Trans woman	0	1	1 (0.8%)	0.8%
	Missing	0	1	1 (0.8%)	-
Ethnicity^c	Person of color	27	28	55 (42%)	46.2%	Chi-squared: 2.6447
	White	22	42	64 (48.9%)	53.8%	Df: 1
	Missing	1	11	12 (9.2%)		p-value = 0.1039
Sexual Orientation		29	45	74 (56.5%)	89.2%	N/A
	Heterosexual	4	5	9 (6.9%)	10.8%
	Missing	17	31	48 (36.6%)	-
Immigration Status	Citizen or permanent resident	17	36	53 (40.5%)	82.8%	N/A
	Other	5	6	11 (8.4%)	17.2%
	Missing	28	39	67 (51.1%)	-
Housing Situation	Stable housing	36	50	86 (65.6%)	90.5%	N/A
	Unstable housing	1	8	9 (6. 9%)	9.5%
	Missing	13	23	36 (27.5%)	-
Education Level	K-12	4	6	10 (7.6%)	20.8%	N/A
	Post-Secondary	12	26	38 (29.0%)	79.2%
	Missing	34	49	83 (63.4%)	-
Age (Continuous)^d	Mean (SD)	40.7 (9.34)	41.4 (9.51)	41.2 (9.4)	-	t = 0.40383 df = 105.58
	Missing	0	1	1 (0.8%)	-	p-value = 0.6872
First Language^c	English	24	36	60 (45.8%)	46.9%	Chi-squared = 0.83825
	Non-English	24	44	68 (51.9%)	53.1%	df = 1
	Missing	2	1	3 (2.3%)	-	p-value = 0.3599
Health-related Variables
Length of Diagnosis (Continuous)	Mean (SD)	12.8 (7.54)	9.53 (7.27)	10.9 (7.5)	-	N/A
	Missing	24	43	67 (51.1%)	-
On Medication	No	8	16	24 (18.3%)	27.6%	N/A
	Yes	25	38	63 (48.1%)	72.4%
	Missing	17	27	44 (33.6%)	-

^aStatistical test was only conducted if missing values were less than 10% and missing values were excluded from the test; ^bstatistical test used was Fisher’s exact test; ^cstatistical test used was Chi-square test; ^dstatistical test used was Student’s t-test.

The bivariate analysis revealed no significant difference in gender, ethnicity, age, and first language between individuals who successfully maintained employment and those who did not. We were unable to statistically test differences in the remaining variables due to extensive missing data (>10%).

Qualitative results

Twelve individuals participated in in-depth qualitative interviews, ranging from 15 to 75 min. Themes emerged within three key areas: facilitators to effective program delivery, barriers to effective program delivery, and recommendations for future employment-based interventions. Quotes illustrating these themes are presented in Table 2.

Table 2. Example quotes from the qualitative interviews.

Quote Number	Themes	Descriptions	Quotes
1	Facilitators to effective program delivery	A participant shared her thoughts surrounding the benefits of EACT and the associated case-workers	“He has been very, very, very, very helpful cause well he helped me a lot with regards to interviewing techniques and job coaching and resumes, anything and all of that so the experiences were very, very good.” [Female, 40–55]
2	Facilitators to effective program delivery	A participant described an example of employment-related assistance he received from the EACT staff	“Oh yes, I have my resume they, I sent to my case worker and he found the other people who help to you know improve it, yeah.” [Male, 40–55]
3	Facilitators to effective program delivery	A participant described an example of non-employment related assistance she received from the EACT staff	“I think it was a blessing that I really found out about Employment ACTion because they’ve really helped me to you know to go and to achieve my goals and he also helped me with suggestions with regards to housing and when to start looking for housing and different areas to look for and who to connect to.” [Female, 40–55]
4	Facilitators to effective program delivery	A participant explained how the EACT staff was helpful in overcoming personal barriers in attaining employment.	“My first counselor there he was had really good rapport … It was less about whether I was capable of working but it was more the mental issues that I was having like difficulty being around people like if it was up to me I wouldn’t … I said I need you to push me cause I won’t do it otherwise.” [Male, 40–55]
5	Facilitators to effective program delivery	A participant explained how the EACT staff was helpful in overcoming personal barriers in attaining employment.	“I think they gave me the confidence to do it you know and then through that I was able to find a job.” [Male, 26–39]
6	Barriers to effective program delivery	A participant expressed their disappointment when they were advised to use general job searching strategies.	“I mean like are you nuts like you know and sending me to job fair? I mean I can find jobs fair everywhere like I went a couple of them like it is a charade.” [Male 40–55]
7	Barriers to effective program delivery	Barriers to effective program delivery	Barriers to effective program delivery
8	Barriers to effective program delivery	Barriers to effective program delivery	Barriers to effective program delivery
9	Recommendations for future employment services	A participant explained the lack of the PLWH working at EACT.	“I went there and the guy was like oh, we have people that can help and I asked them are they living, person who lived with, with living and he said no, but they know about it and no, it won’t work.” [Male 26–39]
10	Recommendations for future employment services	A participant explained how having a PLWH as EACT staff could be helpful for the clients.	“I think that’s the beautiful thing about ACT so I don’t have to tell them before they already know [your HIV status] because that’s why they are there; right, and if I am correct like 97, 95 percent of the people that work there [have HIV] so I don’t have, that part, there’s no idea in my mind that okay, oh, he’s judging me, he’s judging me so that part is not there.” [Male, 40–55]

Facilitators to effective program delivery

Overwhelmingly, participants made positive comments about the EACT program. Key themes that emerged included the program’s focus on job search strategy planning, assistance in attaining employment, as well as general support and relationship building with caseworkers (Table 2, Quote 1). In terms of job search strategy planning, participants described how actions such as providing connections to job fairs, disseminating information on how to tailor or construct resumes and cover letters (Table 2, Quote 2), and training on interview and networking skills were all extremely helpful in attaining employment. This was particularly relevant for our participant population, as none were born in Canada, and all had limited Canadian-based employment experience.

In multiple instances, participants described how caseworkers connected them to available educational opportunities, which further assisted them in attaining employment. Furthermore, the participants explained how the program and the associated job search strategy planning aided in the achievement of non-employment related goals such as housing (Table 2, Quote 3).

In building relationships with caseworkers, participants described that they were able to address underlying mental health and confidence concerns that had previously inhibited their engagement with employment opportunities. As EACT program staff provided essential support to the clients, many participants were able to overcome personal barriers (Table 2, Quotes 4 and 5).

Barriers to effective program delivery

Barriers to effective program delivery included a mismatch between participants’ expectations and the EACT’s service model, as well as staff turnover. In addition, participants highlighted a structural barrier that limited the impact of employment program like EACT on the hiring policies of employers. Participants noted that they expected counselors to look through job postings and highlight potential and appropriate jobs; however, they were instead advised to review the job postings independently. EACT employed a capacity building approach to assisting their clients, where they provided guidance with the job search process and the work of finding job rested with the clients. Some participants wanted more extensive assistance with tasks like resume writing and job searching and expected a more specialized advice for their fields. Participants were disappointed when they were advised to use general job searching strategies like attending general job fairs when looking for a job in a specific industry (Table 2, Quote 6).

Participants noted that there were staffing turnover issues at EACT, which impacted their relationship building with the employment counselors. Given the personal and sensitive nature of participants’ experiences with their employment (e.g., stigma regarding their HIV status), it was difficult to change employment counselors when there was staff turnover (Table 2, Quote 7).

Lastly, participants highlighted how employment programs such as EACT had limited influence on employers’ hiring policies. Participants described that, although EACT assisted in gaining interviews with organizations, issues such as pre-conceived notions of HIV remained a barrier to employment success (Table 2, Quote 8). This limitation is common amongst employment assistance providers as employers ultimately decide on the hiring process.

Recommendations for future employment services

Finally, the participants provided recommendations for the development and implementation of future employment services for PLWH. Participants focused on the importance of technical training and education attainment in the preparation for employment. Participants explained that their lack of skills or gap in knowledge presented significant barriers to attaining and maintaining employment in current competitive circumstances. Some mentioned completing programs that address these gaps by emphasizing factors such as computer skills.

Additional recommendations included the importance of incorporating individuals with HIV as a part of program staff (Table 2, Quote 9), as it will assist in motivating them to gain employment. Often, discussing outcomes as a result of employment with an individual who has experienced similar barriers could facilitate engagement in employment opportunities (Table 2, Quote 10).

A final recommendation which was overwhelmingly outlined by the participants included the importance of follow-up by the service providers following attainment of employment. This was described as a potentially helpful way to ensure that participants remain integrated within the program and continue to have access to supports should they be needed.

Discussion

We found that 38.1% of EACT clients were able to maintain employment for at least 6 weeks within the first year of enrollment in the program. There was no significant difference in the employment rate across gender, ethnicity, age, and first language. Evaluation data on employment programs in Canada is limited, making it difficult to compare EACT’s performance against other programs. A 2015 survey conducted by the Ontario Ministry of Community and Social Services indicated that about half of the participants found a job within 6 months of receiving supports from employment programs funded by the Ontario Labour Market Agreement for Persons with Disabilities (Ministry of Community and Social Services, 2016). Among the income support recipients of ODSP, however, only 11% were employed (Stapleton et al., 2011). Therefore, the EACT program that supports PLWH receiving income supports from ODSP provides a promising solution to help improve the employment rate.

Our analysis contributes to existing literature surrounding factors to be considered when developing employment-based interventions for PLWH. Core findings from the qualitative interviews with EACT program clients corroborate existing findings outlined within employment-based literature. For example, Pinto et al. (2018) identified features of successful employment-based interventions within health and social settings, such as having an interdisciplinary support team, providing patient centered care, and engaging directly with potential employers. Similarly, Maulsby et al. (2020) outlined how improving job search strategy skills through training surrounding interview and resume preparation have been consistently described as essential among PLWH. Recommendations surrounding the integration of individuals with lived experiences as staff within employment-based programs for PLWH relate closely to experiences of stigmatization faced by these groups. According to Perri et al. (2021), the role of stigma within employment institutions represents a major barrier to attaining and maintaining employment among PLWH. This evidence supports the insights provided by clients in the EACT program and demonstrate an existing gap between evidence and practice pertaining to HIV-specific health and social services.

A main strength of this study is the sequential approach of the research, whereby the quantitative findings informed key areas to explore through qualitative data garnered from participants who had successfully gained employment through EACT. Similarly, the use of qualitative interviews allowed the research team to gain a more comprehensive understanding of the barriers and facilitators of the EACT program while also gaining perspectives on what strategies future HIV-specific employment programs must integrate. Despite these strengths, the single-site location of this study presents a significant limitation, particularly in relation to rural and remote regions. However, we believe that broader findings, particularly surrounding what strategies are deemed effective among PLWH, can be applied to other employment programs in different geographical contexts. Further, all interview participants were born outside of Canada and may have different opinions about the program than those who were born in Canada (Maurer & Chapman, 2013; Ng et al., 2005). Similarly, our data represents the perspectives of those who received employment through the EACT program negating to consider the perspectives of those who did not find employment through the program. These perspectives would also contribute to understanding the strengths and limitations of the EACT program. An additional limitation of this study included the use of the 6-week employment milestone as the main outcome of interest. The database used did not record clients who attained employment but quit or were terminated before the 6-week mark. As a result, the selected outcome of interest may have yielded an underestimated number of clients who effectively received an employment placement following engagement with the EACT program.

Our quantitative analysis is also limited by the large number of missing data, which is a common challenge in program evaluation (Halstead, 2019). Post-hoc correspondence with the EACT staff provided some context to the missing data on certain outcome variables. For example, some clients decided to return to school prior to seeking employment, and EACT staff had connected them to relevant resources on education rather than employment. Therefore, future evaluations of the EACT program can consider inclusion of more data sources to ensure a broader understanding of program effectiveness. In addition, future evaluations must focus on meaningfully engaging PLWH to ensure appropriate and effective data collection, analysis, and dissemination.

Our findings have implications for future work in the area of employment and HIV surrounding factors that must be considered for future program development. This evaluation emphasizes the importance of tailored employment-based programs that integrate peer workers for PLWH. As evidenced by this work, HIV-specific employment services are continually needed for PLWH and uniquely beneficial for these groups. Tailored employment programs for PLWH not only have the potential to address internalized and institutionally driven HIV-related stigma, but also hold potential for the strengthening of community and connection to other HIV-related services. Moving forward, other jurisdictions in similar and differing geographical contexts must aim to scale up employment-based programs for these groups. As outlined above, integrating cross-collaboration between sectors such as the health care systems may improve the effectiveness of employment programs for PLWH. Additional considerations for future employment-based programs include the role of gender in relation to HIV and employment status. Given the demographic nature of this evaluation, gendered aspects and needs of individuals who do not identify as cis men are not highlighted. Yet, these have been reported as incredibly important for ensuring program effectiveness and minimizing unintentional harms (Cheuk et al., 2019; Fearon et al., 2019; Martinez, 2019; Poteat et al., 2019).

Conclusion

This sequential mixed-method study aimed at evaluating the EACT program and determining its effectiveness, barriers, and facilitators. We further sought to describe recommendations of features that should be considered in the implementation of future HIV-specific employment programs. As described above, integrating job search strategy training, HIV-specific education, and peers with lived experience of HIV within employment programs are essential for effective employment attainment. Furthermore, it is important as a society to invest in HIV-focused programs like EACT to strengthen their data collection capacity, active outreach to service users, and sufficient planning for evaluation prior to program implementation.

Availability of data and materials

Data sharing is not available for this study. Data is held in a secure network at St. Michael’s Hospital and is only accessible to the study team members named in the protocol who have been approved by the Research Ethics Board.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This study was made possible by funding from the Canadian Institutes of Health Research (CIHR) (Grant Number: 201710CDP). Dr. Ann Burchell holds a Canada Research Chair in Sexually Transmitted Infection Prevention (Tier 2) and receives further support from a Non-Clinician Salary Award, Department of Family and Community Medicine, Temerty Faculty of Medicine, University of Toronto. Dr. Andrew Pinto is supported as a Clinician Scientist by the Department of Family and Community Medicine, Faculty of Medicine, University of Toronto, the Department of Family and Community Medicine, St. Michael’s Hospital, and the Li Ka Shing Knowledge Institute, St. Michael’s Hospital. Dr. Pinto is also supported by a fellowship from the Physicians’ Services Incorporated Foundation and as the Associate Director for Clinical Research at the University of Toronto Practice-Based Research Network (UTOPIAN).