Increasing resiliency and reducing mental illness stigma in post-secondary students: A meta-analytic evaluation of the inquiring mind program

Abstract

Objective: Worsening student mental health, along with more complex mental illness presentation and increased access to campus mental health services, has led to a mental health “crisis” on campuses. One way to address student mental health needs may be through mental health programs which have been found to increase resiliency and help-seeking, and reduce stigma. Participants: The effectiveness of The Inquiring Mind (TIM), a mental health promotion and mental illness stigma reduction program, was examined in 810 students from 16 Canadian post-secondary institutions. Methods and Results: Using a meta-analytic approach, TIM improved resiliency and decreased stigmatizing attitudes from pre to post, with medium effect sizes (d > .50). Analyses with those that completed the follow-up (about one-third of the sample) showed that effects were mostly retained at three months. Other outcomes also point to the program’s effectiveness. Conclusion: TIM appears to be an effective program for post-secondary students. However, additional research, including randomized control trials, is needed to address study limitations.

Keywords:

• Mental health
• mental illness
• post-secondary students
• resiliency
• stigma
• stigma reduction

Post-secondary student mental health has gained increased attention in recent years. Particular attention has been paid to the “mental health crisis” that is currently affecting post-secondary institutions.^1–4 This concern is consistent with the National College Health Assessment (NCHA) data that have pointed to an increase in the prevalence of distress and mental health concerns in post-secondary students. Most recently, the 2019 Canadian NCHA data showed that more than half of students surveyed felt so depressed that it was difficult to function at least once and nearly 70% had felt overwhelming anxiety, within the last 12 months.⁵ Perhaps more concerning, over 16% of students had seriously considered suicide at least once in the past year, an increase of 3.4% from 2016 to 2019.^5,6 This dramatic rise in suicidal thoughts is consistent with the overall picture of worse student mental health that is evident across many health indicators.^5–8 Similarly, American NCHA data points to a comparable situation, as 41% of students reported being so depressed that it was difficult to function and 62% reported overwhelming anxiety in the previous year.⁹ Between 2007 and 2017, rates of post-secondary students with a mental health diagnosis rose from 19% to 34%.⁸

Despite the statistical trend of worsening mental health on post-secondary campuses in North America and the “crisis” referenced above, post-secondary students have been shown to have similar rates of mental illnesses as their non-student counterparts in population-based data.^10,11 Some research has found that when compared to same-aged peers, post-secondary students experience lower rates of mood and anxiety disorders and suicidality. It may be that post-secondary students are somehow different than their age cohort, or it may be that enrollment in post-secondary education could be a protective factor. Thus, the mental health “crisis” may be the convergence of several simultaneous factors on campuses, including an increase in counseling center utilization and an increase in the severity of presenting concerns, which require more mental health resources than historically required.^12,13 More students may also feel safer and more comfortable to seek help now that many campuses have addressed negative and stigmatizing attitudes toward mental health (e.g., programs; see below).¹⁴

Although the rates of mental health concerns are similar to non-student peers, post-secondary students experience a number of unique factors that increase their vulnerability to mental health concerns. A large body of research has demonstrated that late adolescence to early adulthood is when a person is at greatest risk of developing a mental illness.^15–17 Research has also showed that post-secondary students may experience higher levels of stress when compared to their same-aged peers.^11,18 Researchers have identified several areas that appear to contribute significantly to students’ stress and poor mental health, including increased academic pressure, financial burden, technology, changing demographics with more diverse students needing different supports, the stress of transitioning from high school to post-secondary, concerns regarding personal safety on campus, and homesickness.^19–23

There are many significant and negative consequences of poor mental health for post-secondary students. Anxiety and depression have been correlated with lower academic performance, reduced ability to focus, worse sleep, poorer physical health, more missed classes, and an increased chance of dropping out.^{19,22,24–26} Students with poorer mental health are also at an increased risk of substance use problems.^27,28

Another factor that affects student mental health is the stigma of mental illness. Generally, (mental illness) stigma can be conceptualized as a multi-component and multi-level process that involves labeling of human differences that lead to stereotypes and prejudice that lead to status loss and unequal outcomes (i.e., discrimination).²⁹ Additionally, stigma exists at the intrapersonal level, interpersonal level, and structural level.^30,31 At the intrapersonal level is self-stigma, or internalization of the negative attitudes, prejudice, and discrimination that leads to negative impacts on self-esteem and self-worth.³² At the interpersonal level is public stigma, or the stereotypes and prejudice that is perceived in the public or endorsed by individuals, as well as the actions that negatively impact the stigmatized group.³³ Finally, structural stigma is composed of the policies and practices of organizations that restrict the rights of stigmatized groups.^31,34 As a consequence of mental illness stigma, those with mental illnesses experience prejudice and discrimination in health care,^30,31 within the workplace,³⁵ and in housing,³⁶ for example.

Given the negative impact on people with mental illnesses, reducing the stigma of mental illnesses has become an important priority for many Canadian post-secondary campuses, especially for those who have implemented mental health frameworks.^22,37 Studies that have examined public stigma’s function for young adults and post-secondary populations have found that stigma significantly affects the willingness to seek mental health care.^38–41 As such, reduced public stigma on campus should lead to increased help-seeking behaviors and increased demand and accessing of resources for those that require it.^14,42 Programs that address and reduce public stigma, and improve mental health awareness and literacy, are critical for early identification of students who may be at risk for mental health concerns.²² Some data suggests promising trends in this regard over the past decade, with students reporting lower rates of perceived and personal stigma (i.e., the two components of public stigma) on their campuses.⁸ Despite immense work in this area, Linden et al.²² still recommended that campuses continue their public stigma reduction initiatives, particularly because help-seeking rates are still low in this population.^43–45

Post-secondary institutions can take many different actions to address mental health concerns. Recently, post-secondary institutions have implemented holistic multi-level approaches to improve post-secondary student mental health.^22,37,46 At an organizational level, post-secondary institutions can examine policies through a mental health lens,⁴⁷ or generally think about how health promotion is embedded within an institution’s structures.⁴⁶ At a programmatic level, post-secondary institutions could examine and implement different types of mental health services to best suit their population. Some promising approaches include stepped-care,^48,49 along with walk-in clinics.⁵⁰ Finally, at the individual (student) level, post-secondary institutions can provide resources that teach coping skills, increase resiliency, improve mental health literacy, as well as continue to reduce the stigma of mental illnesses on campuses. A general push for this type of individual “upstream” programming for students may, over time, increase earlier help-seeking and reduce demand for on campus counseling services.

Programs directed at post-secondary students are often offered in a workshop format. The range of mental health programs varies greatly, but general trends show that brief sessions focused on teaching and practicing skills are a resource-efficient and effective way to improve student mental health.^51–55 Research has also found that the effects of mental health programs (i.e., stigma reduction, increased resiliency) persist over time. For example, students have continued to report more positive coping strategies and less depressive symptoms three months after participating in a session targeted at improving student mental wellness.⁵⁴ A recent meta-analysis of mental health programs in post-secondary contexts found that the greatest effectiveness was when students were given the chance to practice the skills during the actual program facilitation.⁵²

The Inquiring Mind

Recent research has demonstrated the effectiveness of two anti-stigma and mental health promotion programs in Canada called The Working Mind (TWM),^56,57 along with the Road to Mental Readiness (R2MR) for First Responders (now called The Working Mind for First Responders).^58–61 These programs, originally adapted from the Canadian Department of National Defence’s R2MR,^62,63 demonstrated success at reducing public stigma, increasing resiliency skills, and improving other mental health-related outcomes for both the general workplace employees and for first responders (e.g., police, fire, paramedics, nurses).

Given the positive results of TWM programs,^56,57,61 The Inquiring Mind (TIM) was an adaptation developed specifically for Canadian post-secondary students. TIM uses the same core concepts as its predecessors,⁶⁴ but adaptations were made in consultation with an advisory group, which included students, student service professionals, and the researchers who had developed the original TWM programs. Through the 6-month period that the advisory group worked on this project, the members first reviewed existing content (i.e., materials from the TWM program) and gave their feedback on how to revise, adapt, and develop these materials be relevant and appealing for post-secondary students to form the TIM program. For example, although contact-based video segments were a core component of the previous programs as well, new videos were filmed with students who shared their experiences with stigma, mental health concerns, and help-seeking behaviors. Revisions to the program were made and brought to the advisory committee to review. Additional revisions were made from this round of feedback. Afterwards, the resulting TIM program was piloted with several student groups and feedback obtained through focus groups at the end of the session. After feedback from the pilot tests were incorporated, the program was implemented and evaluated at the 16 sites for the current study.

The final version of the program was divided into six modules designed to be delivered in a total of approximately 4 hours from facilitators who have completed the TIM train-the-trainer (i.e., 3-day workshop). The TIM program provides students with current data regarding the prevalence of mental health concerns on campus in order to normalize mental illness and help-seeking and uses contact-based education to counter stigmatizing attitudes toward those with mental health concerns (i.e., public stigma). There is ample research to support the use of contact-based education (i.e., those with lived experience of mental illnesses talking about their experiences), including video-based experiences, as an effective stigma reduction approach.^33,65,66 Although not a primary focus of the program, internalized or self-stigma is also briefly discussed to help people understand the impact that this can have on someone’s willingness to seek help.

A cornerstone of the program is the Mental Health Continuum Model developed by the Department of National Defence.⁶⁴ This model conceptualizes mental health in phases ranging from green (healthy), yellow (reacting), orange (injured), and red (ill) across various domains of function (e.g., emotions, thinking and attitudes, physical). Critical to this model is the absence of diagnostic labels, highlighting that everyone can experience mental wellness and poor mental health throughout their life. It emphasizes that one’s mental health is dynamic, and individuals can frequently move back and forth along the continuum. One might find themselves moving toward orange and red phases more frequently, but with good supports and skills building, they can make a positive shift toward the green phase. Research has demonstrated the Mental Health Continuum Model as an effective mental health self-assessment tool.^67,68 As well, this model reduces stigma by providing people with simplified and a more accessible way to talk about mental health in particular by “facilitating more openness and dialogue, as it provided a common language” (p 35S).⁵⁹

In TIM, coping and resiliency are emphasized through four evidence-based skills: mental rehearsal, SMART goal setting, deep (diaphragmatic) breathing, and positive self-talk.⁶⁹ Participants practice each skill as they are taught in order to increase skill retention. Other programs that have focused on skill building in post-secondary students have also demonstrated positive mental health outcomes and improved resilience.^70–72 Additionally, at the end of the program, students work through several interactive examples in small groups to integrate the program’s content.

The current study

The current study evaluated the effectiveness of TIM program at different post-secondary institutions across Canada, utilizing a pre-post (follow-up) design with specific emphasis on the pre-post outcomes that are shared with the related TWM programs: stigma reduction and increased resiliency immediately before and after the workshop. Consistent with previous findings from TWM programs,^56,57,61 it was expected that TIM would reduce mental illness public stigma and increase resiliency skills from before the workshop to immediately after in post-secondary student participants. An additional follow-up survey was also sent to participants three months after the workshop to evaluate several other outcomes. These outcomes included the retention of stigma and resiliency overtime, and changes in the Brief Resilience Scale from pre-workshop to follow-up. Additional outcomes also include if participants are more likely to talk about or address mental health concerns in their daily lives, including understanding mental health problems on their campus, asking peers how they are doing, planning to seek help themselves (if needed), and talking openly about mental health issues from pre to follow-up.

Method

Participants

Data was collected from students across 16 different post-secondary institutions in three Canadian provinces: 11 in Alberta, four in Nova Scotia, and one in Newfoundland. There were eight universities, seven colleges, and one institute, with institution sample sizes ranging from 13 to 137 participants. Students self-selected to participate in the workshops with a maximum of 25 participants per workshop. In other words, post-secondary institutions held open workshop that students could sign-up for and attend. A total of 810 participants completed the pre-workshop questionnaire. About 90% (n = 726) of the pre-workshop questionnaires were matched to a post-workshop questionnaire (the remaining were unable to be matched with the participant identifiers), and about 33% (n = 266) of these participants completed a follow-up survey. Those pre-workshop questions that were unable to be matched to at least a post-workshop questionnaire (n = 84) were not used in the analyses. No other exclusion criteria were used.

Regarding demographics of the participants, the majority identified as female (76%), were between 17–25 years of age (74%; M = 23.83, SD = 6.81; range = 17–59), and indicated that they were single (76%; another 20% married or common-law). Mental health (1 = poor to 5 = excellent) varied for participants, but the majority indicated fair, good, or very good mental health on the workshop day (86%) and during the past month (80%). The participants’ programs of study were diverse, with many students specifying programs in the sciences (n = 127; e.g., engineering, biology, health sciences, behavioral sciences). There were also many students who specified programs in nursing (n = 83), education (n = 79; e.g., primary, secondary, child and youth care), psychology (n = 73), business (n = 66; e.g., accounting, management, administrative assistant), and a variety of arts/humanities/fine arts (n = 69; e.g., sociology, history, communications, native American studies, music, drama). Other specific programs included social services (n = 41), social work (n = 35), corrections/law and security (n = 30), addictions community outreach (n = 15), continuing care assistant (n = 14), physical education/kinesiology (n = 10), medicine/public health/midwifery (n = 18), and hairstyling (n = 9).

Materials

Opening Minds Scale for Workplace Attitudes

The Opening Minds Scale for Workplace Attitudes (OMS-WA) was created for workplace environments to assess attitudes and behavioral intentions toward individuals with mental illnesses, and it has been previously used in TWM programs to assess participants public stigma toward those with mental illnesses in the workplace.^56,57,61 For the current TIM program, the OMS-WA was adapted to fit the post-secondary environment (e.g., language includes students/instructor rather than employees/employers) and includes one additional item on dating that was removed from the scale for TWM due to common dating policies in workplaces. The adapted OMS-WA is a 23-item 5-point agreement scale that encompasses five dimensions of stigma: (1) desire for social distance (e.g., I would be upset if someone with a mental illness always sat next to me in class); (2) dangerousness and unpredictability; (3) negative attitudes about mental illness on campus (e.g., You can’t rely on someone with a mental illness); (4) negative attitudes about helping those with a mental illness; and (5) beliefs on responsibility for having a mental illness (e.g., People with a mental illness could snap out of it if they wanted to). Items were reverse coded as necessary so that higher scores indicated higher stigma. A mean score was calculated for the entire scale, as well as each dimension, for each timepoint. The full OMS-WA has been shown to have excellent internal reliability in the previous TWM programs (e.g., Cronbach’s α > .90), and the subscales showed acceptable internal reliability except for (4), which only has four items and was approaching the acceptable range for most timepoints (.58 ≤ α ≤ .69).^57,61 In the current study, the OMS-WA had good to excellent Cronbach’s alphas (.88 ≤ α ≤ .92) at all three timepoints, and all five dimensions also had acceptable to good values (.70 ≤ α ≤ .89; see Table 1).

Table 1. Internal consistency/scale reliability of measures: Cronbach’s alpha.

Measure	# items	Cronbach’s alpha
		Pre	Post	Follow-up
Total adapted OMS-WA scale	23	.916	.923	.878
Social distance /avoidance	6	.871	.890	.886
Dangerousness/unpredictably	5	.742	.772	.746
Work-related beliefs / Competency	5	.810	.798	.821
Helping Behavior	4	.752	.804	.829
Responsibility for Illness	3	.704	.730	.776
Resiliency skill scale	5	.810	.881	.860
Brief resiliency scale	6	.857	–	.857

Note: α > .9 is considered excellent, .8 < α < .9 is good, and, .7 < α < .8 is acceptable.

Resiliency skills measure

The resiliency skills measure is a 5-item 5-point agreement scale that was developed to assess one of the main program outcomes for the TWM.^56,57,61 This measure assessed each participant’s perceptions on their skill and ability to recover from difficult situations (e.g., I have the skills to cope with traumatic events or adverse situations). Higher mean scores reflected greater perceived resiliency. This measure has shown good internal consistency in previous programs (α > .81)^56,57,61 and the current study (α > .81; Table 1), as well as the ability to detect change overtime in intervention studies.^56,57,61

Brief Resilience Scale

The Brief Resilience Scale (BRS)⁷³ was used to measure participants’ general tendencies to “bounce back” or recover from stress (e.g., It does not take me long to recover from a stressful event), as opposed to the resiliency skills measure that was developed to assess skills gains from the program. The BRS measure used a similar 5-point agreement scale to other outcomes, with items reverse coded as necessary so that higher mean scores indicated higher resiliency. It was shown to be reliable and unidimensional in its creation,⁷³ and its Cronbach’s α at both timepoints for the current study was good (α = .86; Table 1). It should be noted that the BRS was not evaluated at post-workshop (and only at pre and follow-up) as “general tendencies” and experiences with recovery due to stress should not be meaningfully affected after a 4-hour workshop but may have shifted after 3 months.

Ability or readiness to engage questions

Participants were asked four questions (5-point agreement scale) regarding the extent to which they are able to talk about or address mental health issues on campus: (1) I understand how mental health problems present on campus; (2) I plan to seek help for my mental health problems, when needed; (3) When I am concerned, I ask my peers how they are doing; and (4) I talk about mental health issues as freely as physical health issues. Participants were reminded to answer each question based on how competent or ready they are to engage, and not how much they would like to do each behavior to ensure ability and readiness were being captured. These measures were developed for and have been used previously in the evaluation of TWM in the same manner (i.e., at pre and follow-up).^56,57,61 Similar to the BRS, these questions would require time for the participants to engage in these behaviors; therefore, they were also only assessed pre-workshop and at follow-up.

Additional questions

Participants were also asked a series of open-ended questions regarding their thoughts about the program (e.g., how relevant the content was, what they liked least/most); this data was used to inform future program development and for quality assurance and not analyzed for the current study. Brief information about the participants’ mental health and demographics, including age, gender, marital status, program of study, and experience with mental health, was also collected.

Procedure

For each post-secondary campus in the study, a similar non-randomized pre-post-follow-up design was utilized. The relevant outcomes were assessed via questionnaires before the program materials were delivered (pre-workshop; all outcomes and demographics), immediately after the material was delivered (post-workshop; OMS-WA [stigma], resiliency skills), and approximately three months following the workshop (follow-up; all outcomes). Completion of the questionnaires at all timepoints was voluntary and participants who responded to the follow-up online survey request (via email) three months later were remunerated with a $10 coffee card. To maintain anonymity in matching questionnaires across time, participants created a unique four-part identifier (e.g., last digit of birth year) on each questionnaire they completed. The University of Calgary Conjoint Faculties Research Ethics Board approved this research project.

Analytic approach

The analysis was informed by previous analyses of the TWM programs,^57,61 and was completed using STATA Version 12.⁷⁴ To show the two primary pre-post outcomes by campus, the ‘metan’ command was used. This determined the effect measure of pre to post-workshop change and used a forest plot to visualize the individual campus effect.⁷⁴ A random effects meta-analysis was chosen a priori, although Q statistics and I² were used to assess homogeneity of campus results. A pooled data set (N = 726) was then used in a random intercept linear mixed model approach to analyze the two pre-post program outcomes (stigma and resiliency skills) with campus modeled as a random effect, and change studied as a difference score (i.e., Δ = pre - post). The adapted OMS-WA measure (stigma) was analyzed as a whole, as well as each dimension separately. This approach supported the modeling of individual characteristics (gender, age, self-rated mental health at baseline) as independent variables (i.e., variables influencing the difference score), which were entered individually and only reported if significant.

Changes from post-workshop to follow-up for the adapted OMS-WA (stigma) and the resiliency skills measure, as well changes from pre-workshop to follow-up for the BRS (resiliency) and the four questions about readiness/ability to engage, were analyzed using the same method as the primary pre-post outcomes described above (i.e., random intercept linear mixed model analysis) with the subsample with follow-up responses (N = 266).

Results

To examine normality of the data, histograms and QQ plots were generated. Aside from a few minor outliers, which were retained, the data was approximately normal, with some skewness (e.g., sample tended to be lower in stigma). The initial analyses exploring individual campus effect (i.e., “metan” function and forest plots) showed the 16 schools had statistical similar program effects pre-post workshop as indicated by the adapted OMS-WA (stigma) measure (Heterogeneity χ²(15) = 10.04, p = .817) and the resiliency skills measure (Heterogeneity χ²(15) = 18.02, p = .261). TIM program’s overall combined effect size for stigma reduction was SMD = .50, with individual campus effect sizes ranging from small (.11) to large (.81; SMD); one campus having a small effect size in the opposite direction as the rest (−.11). The test of SMD = 0 (pooled effect) was significant at the 95% confidence interval (z = 9.43, p < .001), indicating the overall effect size was significantly greater than zero. The TIM program’s overall combined effect size for the resiliency skills measure was SMD = .54, with individual campus effect sizes ranging from small (.11) to large (1.07). The test of SMD = 0 (pooled effect) was also significant at the 95% confidence interval (z = 5.24, p < .001).

Pre-post outcomes

For those questionnaires that were matched in the pooled sample dataset (N = 726), the mean scores on the adapted OMS-WA (stigma) measure were 1.70 (SD = 0.50) at pre-workshop and 1.47 (SD = 0.45) at post-workshop. The mean decrease in stigma scores was 0.23 (SD = 0.29) scale points. As shown in Table 2, the mixed-model analysis showed that this reduction in stigma was statistically significant, as was the reduction in each dimension (ps < .001). No individual differences were significant for this measure. On the resiliency skills measure, the mean at pre-workshop was 3.37 (SD = 0.69), and 3.74 (SD = 0.70) at post-workshop, for an average increase of 0.37 (SD = 0.61) resiliency skills scale points. The mixed-model analyses for this measure indicated that this improvement was also statistically significant (p < .001) and that gender was a significant individual difference with males improving less than females (p = .001).

Table 2. Random intercept mixed model regression: Adapted OMS-WA (and subscales) and resiliency skills (with gender as predictor) pre to Post-Workshop change (Δ= pre – post).

Measure	Coef.	SE	z	p	95% CI
Total OMS-WA Scale	.218	.018	12.38	<.001*	[.183, .252]
Social distance /Avoidance	.173	.021	8.07	<.001*	[.131, .215]
Dangerousness/Unpredictability	.422	.026	16.31	<.001*	[.371, .473]
Work-related beliefs /Competency	.228	.033	6.97	<.001*	[.164, .292]
Helping Behaviour^a	.131	.024	5.39	<.001*	[.083, .178]
Responsibility for Illness^a	.125	.015	8.16	<.001*	[.095, .155]
Resiliency Skills	−.338	.041	−8.32	<.001*	[−.418, −.259]
Male	.185	.055	3.37	.001*	[.077, .293]
Female (constant)	−.383	.042	−9.09	<.001*	[−.465, −.300]

N = 716 – 726.

a Evaluated at α_critical = .01 (Bonferroni correction).

Follow-up outcomes

A total of 266 follow-up surveys were matched to corresponding pre- and post-workshops; however, any campus with fewer than 10 follow-up completions were removed from the analyses due to inadequate sample size, which removed 21 participants (N = 245). The six remaining campuses that were involved in the follow-up analyses were all from Alberta. An analysis of sample characteristics was conducted to see if participants who were able to be matched at follow-up were significantly different those who could not. It was found that there was a significantly smaller proportion of men in the complete matched group (non-matched: 28% male; matched: 15% male; χ² = 14.68, p < .001). No significant differences were observed in age, marital status, self-rated mental health at baseline, baseline adapted OMS-WA (stigma), and baseline resiliency skills.

Stigma reduction and resiliency skills retention

Table 3 shows the results for the post to follow-up change for the adapted OMS-WA (stigma) measure, its five dimensions, and the resiliency skills measure. There was a significant increase in the overall stigma measure and four of its five subscales (responsibility for illness subscale was not significantly different) and a significant decrease in resiliency skills from post to follow-up (i.e., regression toward baseline for both measures). Investigating further, an analysis from pre-workshop to follow-up was conducted on the adapted OMS-WA measure. This analysis showed that the pre to follow-up change was significant, as the follow-up scores were lower than baseline (coefficient = .105, SE = .040, z = 2.63, p = .009). The average mean stigma score at follow-up (M = 1.53, SD = 0.42) was higher than at post-test (M = 1.42, SD = 0.41), but still lower than baseline (M = 1.65, SD = 0.47). The same pattern was seen with the resiliency skills measure. Resiliency skills were lower from post to follow-up, but still higher than baseline (Pre: M = 3.32, SD = 0.71; Post: M = 3.83, SD = 0.64; Follow: M = 3.64, SD = 0.67; coefficient = −.307, SE = .054, z = −5.72, p < .001).

Table 3. Random intercept mixed model regression: Post to follow-up change for adapted OMS-WA and resiliency skills (Δ= post – follow-up). pre to follow-up change for the brief resiliency measure and readiness statements (Δ= pre – follow-up), N = 245.

Measure	Coeff.	SE	z	p	95% CI
OMS-WA Total Measure (post-follow)	−.111	.021	−5.16	<.001*	[−.153, −.069]
Social distance/avoidance^a	−.103	.030	−3.41	.001*	[−.162, −.044]
Dangerousness/unpredictability^a	−.131	.036	−3.61	<.001*	[−.203, −.060]
Campus-related beliefs/Competency^a	−.078	.027	−2.93	.003*	[−.131, −.026]
Helping Behaviour^a	−.218	.082	−2.67	.007*	[−.378, −.058]
Responsibility for Illness^a	−.018	.023	−0.80	.421	[−.063, .026]
Resiliency Skills Measure (post-follow)	.201	.038	5.24	<.001*	[.126, .276]
Brief Resiliency Measure (pre-follow)	−.142	.045	−3.14	.002*	[−.230, −.053]
“I understand how mental health problems present on campus.” (pre-follow)	−.613	.060	−10.26	<.001*	[−.731, −.496]
“I plan to seek help for my mental health problems, when needed.” (pre-follow)	−.354	.059	−6.03	<.001*	[−.470, −.239]
“When I am concerned, I ask my peers how they are doing.” (pre-follow)	−.053	.054	−0.98	.329	[−.160, .053]
“I talk about mental health issues as freely as physical health issues.” (pre-follow)	−.497	.093	−5.32	<.001*	[−.680, −.314]

Note. Only 6 institutions (1–4, 12, 13 from Table 1) were used in the follow-up analyses (requirement of n > 10 at follow-up).

a Evaluated at α_critical = .01 (Bonferroni correction).

Brief resiliency outcome

The analysis for the brief resiliency outcome (BRS) from pre-workshop to follow-up can also be seen in Table 3. This general resiliency measure had a statistically significant improvement from pre to follow-up (Pre: M = 3.06, SD = 0.77; Follow-up: M = 3.21, SD = 0.77).

Ability and readiness to engage outcome

For the perceived ability/readiness to engage questions that were asked at pre-workshop and follow-up, there was a significant improvement from pre to follow-up on three of the four statements (see Table 3). Mean agreement scores for these questions are as follows: Q1: baseline = 3.45 (SD = 0.88), follow-up = 4.04 (SD = 0.66); Q2: baseline = 3.73 (SD = 0.94), follow-up = 4.08 (SD = 0.82); Q3: baseline = 4.21 (SD = 0.68), follow-up = 4.26 (SD = 0.77); Q4: baseline = 3.43 (SD = 1.13), follow-up = 3.93 (SD = 1.06).

Discussion

This study examined the effectiveness of The Inquiring Mind (TIM) to decrease stigma and increase resiliency in post-secondary students. The results show that there were clear significant pre-post changes across our measures of stigma and resiliency skills. There was some regression toward baseline at the 3-month follow-up timepoint, but these were still significant and positive changes. These follow-up results should be viewed as preliminary given the high attrition rate at the follow-up timepoint. Overall, the findings suggest that the program yields positive changes for participants’ stigma regardless of age, gender, marital status, baseline mental health, or baseline scores for the assessment measures. There was a similar pattern for resiliency skills pre-post, except that males showed significantly less improvement in resiliency skills than females. Previous research has found resiliency to be higher in male university students than females, suggesting that females might benefit more from resiliency training.⁷⁵ In general, these findings are consistent with previous research that has shown the benefits of a relatively brief mental health intervention for post-secondary students.⁷⁶ These results further support the integration of such programs into campus planning.

TIM appeared to show relatively better results than the two TWM programs,^57,61 as the effect sizes for stigma reduction and resiliency skills in TIM were .50 and .54, respectively. In contrast, the stigma reduction effect sizes for TWM and TWM for First Responders were .38 and .25, respectively, while the resiliency skills effect sizes were .50 and .32, respectively. These results were obtained even though the TIM participants’ pretest stigma scores were lower (i.e., less stigmatizing) than in the other two samples. Further, the current results demonstrated larger effect sizes despite smaller sample sizes and less power. Resiliency skills scores for the current sample were lower at pretest but increased from pre to post than the other two programs.

It is unclear why outcome differences emerged across the three similar programs. On one hand, these differences may be due to demographic differences across the recipients of these three programs. For example, the current sample is younger, has a higher proportion of females to males, and is more educated than the other two samples. Angermeyer and Dietrich’s⁷⁷ review of 62 studies found that these demographic characteristics were related to less stigmatizing attitudes. On the other hand, program content differences may be driving the outcome differences instead of participant differences. As indicated earlier, although all three programs have the same core content (i.e., video-based contact, the Mental Health Continuum Model, the “Big 4” skills) and use similar program formats (i.e., workshop style, train-the-trainer format), there are content differences across the three programs associated with each target group. For example, there is more of an emphasis on the stress response (i.e., autonomic vs. sympathetic nervous systems) in TWM for First Responders than both TWM and TIM. It is also possible that program outcome differences should be attributed to a combination of demographic and program related factors discussed above. One final note is that there appears to be more variation in the effect sizes among the sites in our sample for both stigma and resiliency skills outcomes than in those of the Dobson et al.⁵⁷ and Szeto et al.⁶¹ samples. It is unclear why this was the case, but one possibility is that there may be more variation in adherence to program fidelity across the current sites than in the TWM sites. Training for trainers in the TWM programs did emphasized program fidelity and adherence to the training manual more so than in TIM.

TIM achieved relatively superior results to the TWM programs and also greater stigma reduction than other mental illness stigma reduction studies. In their meta-analysis of mental illness anti-stigma interventions, Corrigan et al.³³ found that the mean effect size (d) for contact interventions was .28 (see also Griffiths et al.⁷⁸). Moreover, these authors found that video-based contact interventions had a mean effect size of .16, while in-person interventions achieved an effect size of .52. Although TIM uses video-based contact as a key stigma reduction intervention, the effect size was similar to Corrigan et al.³³ findings for in-person contact interventions. It is possible that TIM’s video-based contact worked in conjunction with other components of the program to optimize stigma reduction. For example, TIM has knowledge/educational/mental health literacy components, which also effectively reduce stigma.^33,78,79 In addition to creating a common language and facilitating mental health discussions, The Mental Health Continuum Model may also reduce stigma through its exposition of mental health and illness as a dynamic continuum rather than dichotomous categories. Research has found that continuum beliefs about mental illnesses are associated with reduced stigma.^80,81 As opposed to a single method, it may be that multiple program components act in synergy to optimize stigma reduction. Knaak et al.⁸² found that multiple components (e.g., multiple contact experiences, focus on recovery) made for the most effective stigma reduction programs for health care providers in their meta-analysis.

Finally, the 3-month follow-up data collected in the current study demonstrated the positive and significant impact of TIM on stigma and resiliency, as measured by the OMS-WA and resiliency skills. There was a significant increase in stigma and decrease in resiliency from post to follow-up. Despite this regression toward baseline, the follow-up scores remained significantly improved from scores at pre-workshop. Additionally, three of the four statements regarding knowledge, readiness, and intentions on campus showed significant improvement at the follow-up assessment. Overall, there was significant medium-term impact of TIM. It may be that short “booster” sessions could be implemented to mitigate some of the regression currently observed following the post-workshop time point and beyond, as a recent study has found that stigma reduction regressed to almost baseline levels at 6- and 12-month follow-ups.⁵⁸ With all this said, there was high attrition at the follow-up timepoint so these results should be viewed cautiously.

Study strengths and limitations

A significant strength of this study was its sample size and the inclusion of multiple post-secondary schools across Canada, which allowed for comparisons among campuses to demonstrate the impact of the program regardless of institutional differences. Additionally, surveys were conducted at three timepoints, which allowed for follow-up analyses. Follow-up evaluations are not often included in program evaluations but are needed to better understand how changes in skills and attitudes persist or change following the program delivery.

As this study used an open trial design, alternative explanations exist for the current effects. Randomized control trials would reinforce the current methods. A recent randomized cluster trial with the TWM program demonstrated reductions in stigma and increases in resiliency skills from pre to post with the gains maintained at the 3-month follow-up.⁵⁶ Experimental designs with the TIM program would increase the confidence in the current positive results and address gaps in the current stigma reduction literature.⁸³ Additionally, the stigma and resiliency skills measures (i.e., OMS-WA and resiliency skills) have not been formally validated yet. These two measures, however, have been used extensively and have demonstrated good reliability and responsiveness to multiple programs across different groups and samples.

Other limitations regarding the sample and the facilitation of the study were also present. Firstly, the ethnicity of the participants was not collected for this study. Since there are known differences in mental health experiences across ethnicities (e.g., access to mental health, quality of care),^84,85 the current study cannot control for any differences or explore how ethnicity may impact the experiences of the participants who took the program. Another limitation of this study is the potential self-selection bias of the students who completed the workshop. Program participation was voluntary, and it is possible that the average student who voluntarily attends a 4-hour mental health workshop may not represent the average post-secondary student. For example, program participants were overwhelmingly female. As well, there was high attrition at the follow-up timepoint that was gender-biased, with more female respondents completing follow-up. Therefore, although the follow-up results were promising, they should be viewed with caution. Finally, students completed this program at different times in the academic year. Researchers who have examined the patterns of student depressive symptoms and stress have found that it is highest in December.⁸⁶ Seasonal variations may have implications for program evaluations, as students may report greater benefit from resiliency training during times of high stress.⁵¹

Future directions

Given the focus on program effectiveness (as opposed to moderating factors) in the current study, it would be advantageous to see TIM evaluated at more institutions and ideally in different settings with a wider variety of students. For example, some current institutions implemented TIM as mandatory training for students working in residence. Comparisons between obligatory and voluntary participants in future research could demonstrate differences that inform future programming. Other studies should also explore potential differences in program response among student sub-groups (e.g., different academic programs, LGBTQ + community, commuter vs. resident students). This type of research would inform both the program developers and implementers to whether the program needs contextualization for different contexts or subgroups, or under which conditions the program is most effective. This information could potentially enhance program effectiveness, especially given the diverse nature of post-secondary institutions and students who attend them. Along a similar vein, future research should examine site or post-secondary institution characteristics and how that may impact effectiveness. It is possible that, for example, differences in institution size or availability of mental health resources may impact uptake and effectiveness of the program. Although this was not a research question in the present study and the Heterogeneity χ² indicated low heterogeneity across sites for stigma and resilience outcomes, it would be useful to understand if such factors exist and use them to increase program effectiveness.

Related, qualitative information from participants and facilitators suggests that some students have unique concerns that are not captured in this version of TIM. Based on that feedback, there is ongoing work on the development of versions of TIM for both international and graduate students. Further, and predicated on the importance of early detection and treatment of mental health concerns, a youth version of the program has been developed and piloted in secondary schools across Canada. Finally, as we look to increase uptake and applicability it is critical to consider how we can integrate this programming into electronic platforms, with 91% of university students using smartphones and preliminary evidence showing that interventions delivered virtually are effective in improving depression, anxiety, and wellbeing.⁸⁷ Similarly, research that examines the impact beyond program outcome, such as impact on academic and university-level outcomes, could increase implementation and uptake of mental health programming across campuses if indeed the effects were positive. As well, reducing self-stigma as an outcome seems to be important and is largely absent from the post-secondary mental health literature. Research has demonstrated self-stigma’s relationship with service use, help seeking, and mental health outcomes.^88–90 Exploring ways to reduce self-stigma would seem to complement efforts to reduce public stigma and may increase the impact of interventions.

Acknowledgments

The authors are grateful to the post-secondary sites and their staff for participating in the pilot evaluation of The Inquiring Mind program.

Conflict of interest disclosure

The authors confirm that the research presented in this article met the ethical guidelines, including adherence to the legal requirements, of Canada and received approval from the Conjoint Faculty Research Ethics Board at the University of Calgary.

Additional information

Funding

Development of The Inquiring Mind was funded by a grant from Student and Enrollment Services, University of Calgary and the Mental Health Commission of Canada. Funding for the evaluation of The Inquiring Mind was provided by the Mental Health Commission of Canada.