Effectiveness of online training for improving knowledge, attitudes, and confidence of alcohol and other drug workers in relation to co-occurring mental health conditions

Abstract

There is increasing recognition that the high rates of co-occurring mental disorders among clients of substance use treatment services require alcohol and other drugs (AOD) workers to be skilled in their identification, management, and treatment. Despite advances in e-learning, to our knowledge, based on our review of the literature, there are no training programs with the capacity to build foundational knowledge, attitudes, and confidence in mental health, essential to the provision of evidence-based practice. The current study evaluated an evidence-based online program developed to address this gap, in a pre–post-design. One-hundred-fifteen Australian AOD workers completed pre- and post-training online surveys. Participants rated level of agreement with statements assessing comorbidity-related knowledge, attitudes, and confidence, which were reassessed immediately post-training. The overwhelming majority of participants found the training useful (94%), were satisfied (97%), and reported having gained knowledge that would enable them to work more effectively with comorbidity-related issues (95%); 89% reported having used skills gained in clinical practice. The starkest improvements were observed in relation to improved confidence and knowledge following training completion. Findings illustrate that online training programs, such as this could be used to improve the capacity of the current workforce, as well as the next generation of clinicians, to respond to comorbid mental health conditions.

Keywords:

• Computer-assisted instruction
• comorbidity
• evidence-based practice
• mental health
• substance-related disorders
• workforce education

Introduction

Workforce capacity to address co-occurring mental and substance use conditions is an international concern. In 2019, an estimated one-in-three, or 36.9% of people worldwide with a substance use disorder experienced at least one co-occurring mental disorder (Harris et al., 2019), yet treatment need remains high. Of particular concern, more than half (51.4%) of US adults who experienced co-occurring mental and substance use disorders did not receive substance use or mental health services (Substance Abuse and Mental Health Services Administration, 2020).

Outside the US, co-occurring mental disorders are also common among clients of alcohol and other drugs (AOD) treatment services, with estimates indicating that between 50 and 76% of clients of Australian AOD treatment services meet diagnostic criteria for at least one co-occurring mental disorder (Kingston et al., 2017). Clients experiencing co-occurring mental and substance use disorders typically present to treatment with a more complex and severe clinical profile compared to those with a substance use disorder alone, including poorer general physical and mental health, greater substance use severity, increased risk of homelessness, poorer social and occupational functioning, greater difficulties in interpersonal and family relationships, and increased risk of suicide (Burns et al., 2005; Kelly & Daley, 2013; Schäfer & Najavits, 2007).

Evidence from numerous reviews, policy documents, and AOD workforce surveys, however, suggests that many practitioners do not feel equipped with the essential skills to identify symptoms of mental health conditions and respond accordingly (Merkes et al., 2010; NSW Ministry of Health, 2015). A 2014 systematic assessment of 256 mental health and AOD programs across the US using the Dual Diagnosis Capability in Addiction Treatment (DDCAT) and Dual Diagnosis Capability in Mental Health Treatment (DDCMHT) indexes showed that only 18% of AOD programs and 9% of mental health programs met criteria for dual diagnosis capability (McGovern et al., 2014). Further, findings from 2019 to 2020 Australian National AOD Workforce Survey conducted on over 1500 AOD workers found that along with managing trauma (64%), workers were most concerned about receiving training on working with clients with co-occurring mental health issues (62%), and responding to multiple and complex needs (55%) (Skinner et al., 2020). While the past few decades have seen an increase in the amount of evidence relating to the management and treatment of co-occurring mental and substance use disorders, there remains a substantial evidence-to-practice gap, which in some areas of health has been cited as 17 years (Morris et al., 2011). The need to improve the translation of research into practice, therefore, remains a priority.

Evidence has highlighted the important role of effective workplace training in improving evidence-based practice by targeting the knowledge, attitudes, and confidence of clinicians (Ayu et al., 2015; Calder et al., 2017). For the AOD workforce, training has traditionally been provided in face-to-face workshops, yet key challenges associated with time and travel required for in-person sessions, the diversity of the AOD workforce, high running costs of providing (and attending) large-scale training, and the lack of qualified supervisors and trainers, remain (Fairburn et al., 2017; Henggeler et al., 2013; Miller et al., 2004; Roche et al., 2008; Stewart et al., 2012). Rapid improvements in technology over the past two decades have provided the opportunity to utilize internet-based approaches to overcome some of these barriers.

Online or e-learning refers to education that is provided over the internet (Means et al., 2009). Online learning offers several advantages over traditional face-to-face models, including the capacity to deliver consistent training with a high degree of fidelity (Martino, 2010); providing access to a large number of people, making it more cost-effective to deliver and attend—especially important to those from rural and remote areas (Calder et al., 2017; Singh & Reyes-Portillo, 2020); providing a more sustainable method of training in terms of ability to update information; the flexibility to be used where and when most convenient for the user, in an interactive environment incorporating multimedia components (Calder et al., 2017; Singh & Reyes-Portillo, 2020); and the ability to take into account different learning styles and preferences (Carroll & Rounsaville, 2007), while allowing users to set their own learning pace.

Three recent reviews have been conducted examining evidence-based online training programs for AOD workers (Calder et al., 2017), behavioral healthcare providers (Jackson et al., 2018), and public health clinicians (Singh & Reyes-Portillo, 2020). While all three had difficulties drawing definitive conclusions due to the heterogeneity of training, methodological approaches, and poor-quality of studies, all found promising support for the use of e-learning, particularly in terms of promoting and facilitating uptake of evidence into clinical practice in a sustainable, cost-effective way.

While there was variation in program content, recipients, and dosage, the majority of e-learning programs developed for AOD workers to date have almost exclusively focused on specific treatment methods (such as cognitive behavioral therapy, CBT; motivational interviewing; dialectical behavioral therapy) for single co-occurring disorders (e.g. co-occurring borderline personality disorders) (Jackson et al., 2018; Larson et al., 2013; Martino et al., 2011). To our knowledge, a study by Covell et al. (2011) is the only online training program to feature online comorbidity training modules with a broad mental health focus; however, outcome measures examined were limited to uptake and did not capture changes in knowledge, attitudes or confidence (Covell et al., 2011).

To improve the ability of Australian AOD workers to identify, manage and treat co-occurring mental health conditions the Australian Government Department of Health funded the development of an evidence-based online training program for AOD workers, based on the national Guidelines on the management of co-occurring alcohol and other drug and mental health conditions in alcohol and other drug treatment settings, 2nd Edition (hereafter referred to as ‘the Comorbidity Guidelines’) (Marel et al., 2016). The Comorbidity Guidelines online training was developed to facilitate the translation of the guidelines and support their uptake into clinical practice. Specifically, the online training program aimed to increase AOD workers’ knowledge and awareness of co-occurring mental health conditions in AOD treatment settings, improve the confidence and skills of AOD workers to respond to comorbidity, and increase the uptake of evidence-based care.

The current study aimed to examine:

1. Australian AOD workers’ perceptions of the usefulness and relevance of the Comorbidity Guidelines online training to clinical practice, and factors associated with these perceptions; and

2. The impact of the online training program on Australian AOD workers’ knowledge, attitudes, and confidence, in relation to the management and treatment of co-occurring mental health conditions; and factors related to change over time.

Methods

Participants

All new training program registrants between 29 November 2017 and 3 May 2019 were invited to take part in the evaluation. During this period, 1415 people registered for the training program. The study received ethical approval from UNSW Sydney (HC 17059) and was subsequently transferred to the University of Sydney ethics committee (HC 2018/872).

Procedure

Training program

The training program was based on content from the Comorbidity Guidelines, which was developed in collaboration with a panel of experts and discussion forum members (involving clinicians, consumers, carers and families, addiction medicine specialists, academic researchers, AOD workers, doctors, nurses, psychologists, psychiatrists, social workers, and other healthcare providers), and a series of comprehensive reviews of the literature. The program incorporates several key components identified as important by AOD workers including interactive content, clearly-defined key learnings, job-relevant content, multimedia components (images, videos, animations, case studies), assessments at the end of each module to reinforce learnings, and flexible self-paced learning (Curran et al., 2015; Larson et al., 2009; Shafer et al., 2004). Ten clinicians focus tested the program in June 2017, and their feedback informed program modification. The program was made available in November 2017 and officially launched in February 2018.

The program, available at www.comorbidityguidelines.org.au, consists of 11 content modules and one introductory module: (i) what is comorbidity; (ii) guiding principles; (iii) classification of disorders; (iv) holistic health care; (v) identifying comorbidity; (vi) risk assessments; (vii) care coordination; (viii) approaches to comorbidity; (ix) managing and treating specific disorders; (x) worker self-care; (xi) specific population groups. From beginning to end, the program takes ∼10 h to complete and combines interactive content with multimedia components (including animations, videos, and case studies). The program was fully narrated providing trainees with the option of listening to audio for each slide. Successful completion of all end-of-module assessments is required to complete the program.

Pre- and post-questionnaires

All participants completed a baseline questionnaire pre-training, and a further survey immediately post-training, after completing all 11 modules.

Pre- and post- self-report questionnaires were based on a previous evaluation measure developed by the authors (Mills et al., 2012), which gathered key demographic information about participants and the organisation where they were currently employed. All questionnaires asked participants to indicate the degree to which they agreed with a series of 26 statements assessing knowledge, attitudes, and confidence to address comorbidity in practice on a five-point Likert scale (see Table 2).

In addition to these statements, the post-training questionnaire asked participants to rate on a five-point Likert scale their perceptions of the program including overall satisfaction with the training, the degree to which they found the training relevant and useful, how often they had referred to content to assist with their clinical decision making, whether they felt the training had led to any specific changes in their clinical practice or impacted on client outcomes.

Analysis

All tests were conducted using PASW Statistics 25. Due to small numbers in the ‘strongly disagree’ and ‘strongly agree’ response categories (<1.0%) for the majority of questions assessing knowledge, attitudes, and confidence, strongly disagree/disagree and strongly agree/agree were combined to form a three-point Likert scale for the purpose of analysis (strongly disagree/disagree, unsure, agree/strongly agree). Wilcoxon signed-rank tests were used to investigate changes between pre- and post-training scores relating to knowledge, attitudes, and confidence to address comorbidity. Kruskal-Wallis tests were used to investigate whether baseline participant characteristics were associated with participants’ perceptions of the program as well as any changes in knowledge, attitudes, and confidence to address comorbidity between baseline and post-training.

Baseline variables that were examined included age (categorized 17–25, >26 years), gender, the highest level of education completed, geographic location of service, state/jurisdiction, service sector, current occupation, whether participants were students or workers, experience with comorbidity-related issues, whether participants had undertaken previous comorbidity-related training and mental health training in past 12-months.

Baseline factors associated with study retention were examined using a binomial regression, with included variables that were hypothesized to potentially impact retention. These included age, sex, the highest level of education, geographic location of service, current occupation, years of experience with AOD, mental health or comorbidity-related issues, state, whether participants were students or workers, and service sector.

Factors associated with training completion were examined using a series of multinomial logistic regressions to identify whether they were broadly representative of those who had registered for online training. Variables included were those hypothesised to have a potential impact on completion, including Australian jurisdiction, occupation type, and service sector.

An alpha level of 0.05 was set for significance for all tests.

Results

Participant characteristics

One-hundred and fifteen participants provided informed consent and completed the baseline and follow-up survey, administered immediately post-training. The mean age of participants at baseline was 43.0 years (SD 12.6) and 74.8% were female (Table 1). The majority had completed some tertiary training, either a university degree (69.6%) or vocational training (29.6%).

Table 1. Baseline characteristics of participants.

	n = 115
Demographics
% Female	74.8
% Male	23.5
% Other	1.7
Mean age (SD)	43.0 (12.6)
% Completed university education	69.6
% Completed vocational training	29.6
% Completed secondary school or less	0.9
Current occupation
% Caseworkers, counsellors, social workers	41.7
% Medical and other allied health professionals	23.5
% Other AOD workers	20.9
% Students	13.9
Service location
% New South Wales	33.3
% South Australia	16.2
% Queensland	14.4
% Victoria	13.5
% Western Australia	10.8
% Tasmania	7.2
% Northern Territory	2.7
% Australian Capital Territory	1.8
Experience with AOD and mental health
% More than 15 years’ experience  working with AOD-related issues	16.0
% More than 15 years’ experience  working with mental health-related issues	16.8
% More than 15 years’ experience  working with comorbidity-related issues	13.8
% Undertaken training in mental health	67.9
% Completed courses in comorbidity	79.1

Participants had been working in their current service for a median of 1.0 year (range 0.0–27.0). One-third of participants reported having more than 10-years’ experience working with issues related to AOD (33.0%) and mental health (34.7%), with just under one-third reporting similar experiences regarding comorbid AOD and mental health (28.7%). A range of occupations was represented, most commonly caseworkers, counsellors, and social workers (41.7%); medical and allied health professionals (25.3%); and other AOD workers (including Aboriginal health workers, community support workers, consumer advocates, drug court diversion clinicians, teachers and educators, grief and trauma workers, health promotion workers, nurse educators, occupational therapists, peer support workers, prison officers, social workers, trainers, youth workers; 20.9%) and students (13.9%).

Service characteristics

Participants worked in services across all Australian jurisdictions, predominantly New South Wales (33.3%) and South Australia (16.2%), followed by Queensland (14.4%), Victoria (13.5%), Western Australia (10.8%), Tasmania (7.2%), the Northern Territory (2.7%), and the Australian Capital Territory (1.8%). The majority of services were located in major urban areas (population between 100,000 and 1 million or more, 57.9%), with 37.7% from other urban or country areas (population between 1000 and 99,999), and 4.4% from a small country, rural (population between 200 and 999), or remote areas (population < 200).

Participants worked in a range of service types, including outpatient counselling (61.6%), residential rehabilitation (32.3%), inpatient and outpatient detoxification (25.3 and 18.2%, respectively), substitution therapies (18.2%), and other services (32.3%; such as mental health services, community support/outreach, integrated AOD, and mental health services, case management, forensic services, education, youth services, health promotion, crisis care/management, medical services, needle and syringe programs).

Those who completed the training were broadly representative of the total registered sample; two factors associated with training completion were the Australian jurisdiction of those registered (p < .0001) and the service sector (p < .0001). Between group comparisons for factors associated with training completion are presented in the Supplementary Materials (Table S1). Of note, registrants from Victoria were significantly less likely to complete the online training than those from other Australian jurisdictions. Registrants from the non-government sector were significantly more likely to complete the online training than those from the government or private sectors.

Knowledge, attitudes, and confidence

As illustrated in Table 2, responses to these questions at baseline indicated that more than two-thirds of the sample felt confident in their ability to identify and manage symptoms of common mental health conditions, talk to clients about comorbidity and traumatic events, where and how to refer clients who need treatment for comorbidity, and where to access screeners and assessments. More than half felt confident in their ability to treat comorbidity.

Table 2. Participant knowledge, attitudes, and confidence at baseline and post-training.

	Baseline			Post-training
	% Strongly disagree /disagree	% Unsure	% Agree /strongly agree	% Strongly disagree /disagree	% Unsure	% Agree /strongly agree
Knowledge
Comorbidity treatment should involve consideration of the whole person, including psychological, physical, and sociodemographic needs	0.0	1.7	98.3	0.0	2.6	97.4
Mental health symptoms need to be monitored throughout treatment*	2.6	2.6	94.7	0.0	1.7	98.3
Clients with AOD problems often have co-occurring mental health conditions	3.5	1.7	94.7	0.0	2.6	97.4
All AOD clients should be routinely screened for mental health conditions*	2.6	5.2	92.1	0.0	4.3	95.6
Clients with co-occurring mental health conditions have special treatment needs*	7.0	13.0	80.0	2.6	4.3	93.1
Clients with co-occurring mental health conditions often require long term treatment	4.3	12.2	83.5	2.6	5.2	92.2
Clients with mental health conditions should be referred to mental health services	16.5	19.1	64.3	15.6	13.0	71.3
Clients with co-occurring mental health conditions have poorer treatment outcomes that those with AOD use disorders alone	30.5	24.3	45.3	32.1	11.3	56.6
Mental health conditions may resolve once the client has stopped drinking/using*	33.9	18.3	47.8	18.3	6.1	75.7
Attitudes
Clients with co-occurring mental health conditions are dangerous	89.6	7.0	3.4	88.7	7.8	3.5
It is my responsibility to follow-up clients once they have been referred to other services*	19.1	20.9	60.0	3.5	7.8	88.7
Clients with co-occurring mental health conditions are more difficult to treat that those with AOD use disorders alone	19.2	19.1	61.7	23.5	10.4	66.1
Clients should only be excluded from an AOD service if they are in a crisis situation (e.g. actively suicidal or psychotic)*	36.6	20.0	43.5	28.7	13.9	57.4
Once I have referred a client to another service they are no longer my responsibility*	71.3	20.0	8.7	87.0	7.0	6.0
I do not feel safe with clients who have co-occurring mental health conditions	93.9	4.3	1.7	89.6	5.2	5.2
AOD workers should not have to deal with mental health conditions as well as the AOD use disorder	93.9	4.3	1.7	88.7	3.5	7.8
Clients with other mental health conditions (such as depression, anxiety, personality disorders) should be excluded from AOD treatment services	95.6	1.7	2.6	93.1	4.3	2.6
Clients with severe mental health conditions (such as schizophrenia and bipolar disorder) should be excluded from AOD treatment services	89.6	8.7	1.8	90.4	5.2	4.3
Confidence
I am confident in my ability to identify symptoms of common mental health conditions (e.g. depression, anxiety)*	4.3	8.7	87.0	0.9	7.0	92.2
I am confident in my ability to talk to clients about co-occurring mental health conditions*	7.9	10.4	81.8	3.5	2.6	93.9
I am confident in my ability to talk to clients about traumatic events that may have happened in their lives (e.g. physical or sexual abuse)	5.2	13.0	81.8	4.4	8.7	86.9
I know where to refer clients who need treatment for co-occurring mental health conditions*	6.1	17.4	76.5	2.6	8.7	88.7
I know how to refer clients who need treatment for co-occurring mental health conditions*	6.1	18.3	75.6	2.6	7.8	89.6
I know where to access screeners and assessments for mental health conditions*	11.3	16.5	72.2	1.7	4.3	93.9
I am confident in my ability to manage the symptoms of co-occurring AOD and mental health conditions*	13.9	20.9	65.3	4.3	12.2	83.5
I am confident in my ability to treat co-occurring AOD and mental health conditions*	15.6	28.7	55.6	7.9	14.8	77.4

*Significant change in score from baseline to post-training (p < .05).

Perceptions of the training program

As shown in Table 3, the vast majority of participants who had completed training found it quite a bit/extremely useful (38.3 and 55.7%, respectively) and were mostly/very satisfied with the training program (39.1 and 58.3%, respectively). While no participants indicated that the training was not useful, two (1.7%) reported being quite dissatisfied. The majority (88.7%) had used the training program content to assist with clinical decision making, with 37.4% indicating they had done so often/almost always.

Table 3. Perceptions of the training program.

	Post-training N = 115
	% Not at all/slightly	% Moderately	% Quite a bit/extremely
How useful have you found the online training program	0.0	6.1	94.0
How satisfied are you with the online training program	1.7	0.9	97.4
	% Strongly disagree/disagree	% Unsure	% Agree/strongly agree
The online training program has enabled me to respond to comorbidity related issues with greater confidence	1.7	4.3	94.0
I gained skills or knowledge from online training program that enabled me to work more effectively with comorbidity issues	0.9	4.3	94.8
The online training program effectively illustrated links between the theory of responding to comorbidity and the practical aspects of responding	0.9	3.5	95.6
The information/materials provided in the online training program improved the quality of comorbidity related responses in my workplace	1.7	17.4	80.9
All in all, the online training program improved my responses to comorbidity in my workplace	0.0	7.8	92.2
The online training program addressed practical constraints of responding to comorbidity	2.6	7.8	89.6
The online training program effectively incorporated relevant workplace issues	0.9	7.0	92.2
The content of the online training program was appropriate for my current work needs	0.0	8.7	91.3
The online training program encouraged me to pursue further learning ‘on-the-job’	2.6	8.7	88.7
The online training program was consistent with my job requirements	0.0	7.8	92.2
The online training program was too removed from my experiences at my workplace to be useful	61.7	14.8	23.4
I have used some of the things I learnt from the online training program in my work	4.4	8.7	87.0

Just under half (44.3%) of participants indicated that the training had led to specific changes in their clinical practice. More than one-third (36.7%) of specific changes in clinical practice related to improved recognition of mental health symptoms underlying or contributing to AOD use. Moreover, 58.3% reported that their use of the Guidelines had improved client outcomes. The overwhelming majority of participants who completed the post-training survey reported that the training improved their ability to respond to comorbidity in their workplace. No baseline factors were significantly associated with perceptions of the program.

Knowledge, attitudes, and confidence in addressing comorbidity

Table 2 illustrates the degree to which participants agreed with statements assessing their knowledge, attitudes, and confidence to address comorbidity at baseline and post-training.

Change in knowledge following training completion

Four of the nine comorbidity-related knowledge statements demonstrated a significant improvement from pre- to post-training. The largest shifts in levels of agreement were evident in relation to the statements ‘Mental health conditions may resolve once the client has stopped drinking/using,’ which increased from 47.8 to 75.7% between baseline and post-training (Z = −4.584, p < 0.001), and ‘Clients with co-occurring mental health conditions have special treatment needs’ over the same time period (80.0–93.1%; Z = −2.359, p 0.018). The proportion of participants who agreed/strongly agreed with the statements ‘Mental health symptoms need to be monitored throughout treatment’ (Z = −3.732, p < 0.001) and ‘All AOD clients should be routinely screened for mental health conditions’ (Z = −2.895, p 0.004), also significantly increased between baseline and post-training (94.7–98.3%; 92.1–95.6%, respectively).

Change in attitudes following training completion

Three of the nine comorbidity-related attitude statements demonstrated a significant improvement from pre- to post-training. The proportion of those who agreed/strongly agreed with the statement ‘It is my responsibility to follow-up clients once they have been referred to other services’ increased from 60.0 to 88.7% between baseline and post-training (Z = −6.428, p < 0.001). Similarly, participants demonstrated significant shifts in levels of agreement with the statement ‘Clients should only be excluded from an AOD service if they are in a crisis situation (e.g. actively suicidal or psychotic)’ between baseline and post-training, from 43.5 to 57.4% (Z = −3.124, p 0.002). The proportion of participants who disagreed/strongly disagreed with the statement ‘Once I have referred a client to another service, they are no longer my responsibility’ increased significantly between baseline and post-training, from 71.3 to 87.0% (Z = −2.617, p 0.009).

Change in confidence following training completion

As illustrated in Table 2, seven of the eight statements related to confidence in addressing comorbidity demonstrated a significant improvement from pre- to post-training. The most substantial shifts in confidence between baseline and post-training were evident in relation to the statements ‘I am confident in my ability to treat co-occurring AOD and mental health conditions’ (55.6–77.4%; Z = −4.863, p < 0.001), ‘I know where to access screeners and assessments for mental health conditions’ (72.2–93.9%, Z = 4.826, p < 0.001), and ‘I am confident in my ability to manage the symptoms of co-occurring AOD and mental health conditions’ (65.3–83.5%, Z = 4.496, p < 0.001). Participants also demonstrated significant shifts in confidence related to knowing where to refer clients who need treatment for co-occurring mental health conditions (76.5–88.7%, Z = −3.702, p < 0.001), and how to refer clients who need treatment for co-occurring mental health conditions (75.6–89.6%, −4.000, p < 0.001). The degree to which participants agreed/strongly agreed that they felt confident in their ability to talk to clients about co-occurring mental health conditions significantly increased from 81.8 to 93.9% between baseline and post-training (Z = −3.203, p 0.001). Confidence in identifying symptoms of common mental health conditions also significantly increased from 87.0 to 92.2% between baseline and post-training (Z = −2.018, p 0.044).

Factors associated with changes in knowledge, attitudes, confidence/skills

There were two baseline covariates that were significantly associated with shifts in knowledge and confidence between baseline and post-training: being aged 17–25 years and having completed previous training in comorbidity (Table 4). Participants aged 17–25 years showed greater increases in the level of agreement with the statement ‘Clients with co-occurring mental health conditions have special treatment needs’ compared to those aged >25 years [H(1) = 6.58, p = 0.01]. Those who had not completed previous comorbidity-related training at baseline demonstrated greater increases in confidence in relation to knowing where to refer clients who need treatment for co-occurring mental health conditions [H(1) = 5.77, p = 0.016], compared to those who had completed previous training. Similarly, those who had completed previous training demonstrated greater increases in confidence in relation to knowing how to refer clients who need treatment for co-occurring mental health conditions [H(1) = 8.05, p = 0.005].

Table 4. Factors associated with a change in knowledge, attitudes, and confidence between baseline and post-training.

	Baseline			Post-training
	% Strongly disagree/disagree	% Unsure	% Agree/strongly agree	% Strongly disagree/disagree	% Unsure	% Agree/strongly agree
Knowledge
Clients with co-occurring mental health conditions have special treatment needs
Aged 17–25 years	18.2	27.3	54.5	9.1	9.1	81.9
Aged 26+	5.8	11.7	82.5	1.9	3.9	94.1
Confidence
I know where to refer clients who need treatment for co-occurring mental health conditions
Not completed previous comorbidity training	11.1	33.3	55.6	0.0	18.5	81.5
Completed previous comorbidity training	4.5	12.5	83.0	3.4	5.7	90.9
I know how to refer clients who need treatment for co-occurring mental health conditions
Not completed previous comorbidity training	11.1	33.3	55.6	0.0	14.8	85.2
Completed previous comorbidity training	4.5	13.6	81.9	3.4	5.7	91.0

Discussion

To our knowledge, this study is the first to evaluate the effectiveness of an evidence-based online training program for improving the knowledge, attitudes, and confidence of AOD workers in relation to co-occurring mental health conditions. Perhaps one of the most important findings is that despite high levels of knowledge and confidence among participants at baseline, there were significant improvements in levels of comorbidity-related knowledge and confidence in several domains over the study period. The lack of significant change for some variables within the knowledge and attitude domains may reflect a ceiling effect regarding the high levels of willingness to work with and treat those with comorbidity, that may have been evident pre-training.

The vast majority of participants (94%) found the training useful, and 97% were satisfied with the program. Of particular note, 95% reported gaining skills or knowledge that enabled them to work more effectively with comorbidity-related issues, and 89% had referred to training content in clinical practice. While there were some significant shifts in relation to knowledge and attitudes (4/9 and 3/9 statements, respectively), there were more notable differences observed in relation to confidence (7/8 statements). Although improvements in confidence without concurrent shifts in knowledge and attitudes may at first seem concerning, it is not necessarily surprising given that the sample was on the whole, highly educated and very experienced in working with AOD, mental health, and comorbidity-related issues. Further, although few covariates were significantly associated with shifts in knowledge and confidence between baseline and post-training, it is possible that other barriers to the training program exist that were not measured in the current study. Taken together, these findings demonstrate that the online training program was effective in increasing levels of knowledge, attitudes, and confidence among AOD workers, and may be applicable and appropriate to a diverse range of students and workers across their careers.

It should also be noted that the training had little impact on one attitude statement, ‘Clients with co-occurring mental health conditions are dangerous,’ with which 89.6% of participants strongly disagreed/disagreed, and 3.4% agreed/strongly agreed. There was little change post-training, which may indicate that more specialized training to address stigmatized attitudes regarding comorbidity may be required for some people.

Findings should be viewed within the context of the following limitations. Firstly, although participants encompassed AOD workers from a variety of service types, organisations, roles, and sectors across Australia, they were not recruited at random, and findings may not be generalisable across the sector. However, in terms of professional qualifications, the ratio of urban to rural participants, and level of experience working with AOD-related issues, the cohort was comparable to recent surveys of the Australian AOD workforce and international systematic reviews (Calder et al., 2017; Skinner et al., 2020). Secondly, as participants were recruited as part of the online training program registration process, it is possible that response bias exists, whereby those who had an interest in or self-identified a need for, further training was drawn to enroll in the program and consented to participate in the evaluation. Consistent with other studies conducted online, response rates for the online survey were low (Fan & Yan, 2010). However, the baseline sample was generally representative of the broader population of 1415 training registrants enrolled during the study period. Our routine program monitoring of the broader registrant pool identified that the only factors associated with training completion were the Australian jurisdiction of those registered and service sector. Those from Victoria were significantly less likely to complete the program than all other jurisdictions, excluding Queensland. This may be due to high levels of mental health literacy among AOD staff in Victoria, as Victoria has been leading the way in delivering comorbidity treatment for many years, and is home to numerous comorbidity initiatives (e.g. Victorian Dual Diagnosis Initiative). There were no significant differences between those from Victoria and those from other jurisdictions in terms of perceptions of the program’s utility and satisfaction. Those from non-government organisations were significantly less likely to complete the program than those from government or private organisations. These findings echo an evaluation of the first edition of the Comorbidity Guidelines, which assessed the perceived relevance and utility of the resource on clinical practice (Mills et al., 2012 ).

The study was also limited to those who had completed the training program. Participants who had consented to the study but had not completed the online training program were emailed a non-complete survey four weeks after their last login. These findings have not been reported here due to a very low response rate (n = 12). Previous research has identified higher completion rates among those undertaking training with a supported component, such as supervision or engagement interventions, even if these are conducted online (Singh & Reyes-Portillo, 2020). Support reinforced key learnings, provided an avenue for discussion, review of skills, and critical feedback (Singh & Reyes-Portillo, 2020). Online training that can encourage the application of learning beyond the online ‘classroom,’ whether in discussion with peers or supervisors, may have a greater likelihood of maintaining engagement and knowledge. And finally, although the current evaluation asked participants whether the use of the Guidelines or online training had led to changes in client outcomes or changes in their practice, it was not possible to measure participant or practice-level changes.

While advances continue to be made by way of research examining the effectiveness of interventions, it is the translation of that research into practice and improving the capacity of the AOD workforce and healthcare workers to intervene, that ultimately has the potential to improve outcomes and quality of life for people with co-occurring mental and substance use disorders. Two ways of improving the capacity of AOD workers include: (i) the development of a National Minimum Qualifications Framework for the AOD workforce, such as those that currently exist in Victoria and the Australian Capital Territory; and (ii) that such a framework would require the core competency of training in the recognition, management, and treatment of co-occurring disorders, within the capacity of an AOD worker’s professional role. Having access to up-to-date, evidence-based guidance is critical to the provision of such training, in professional development for the existing workforce, as well as in undergraduate levels when a person is undertaking their vocational or tertiary training. Online learning has played an increasing role in continuing professional development over the last decade, but the need for high-quality, evidence-based, self-paced learning options has been accelerated by the challenges posed by the COVID-19 pandemic (Seymour-Walsh et al., 2020). It has also heightened the debate regarding the relative benefits of face-to-face, online and other forms of learning. It is critical that educators and researchers continue to optimise online learning options to ensure the best learning outcomes.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was funded by the Australian Government Department of Health and supported by Australian National Health and Medical Research Council (NHMRC) Fellowships to Christina Marel, Maree Teesson and Katherine Mills.