Evaluating recovery colleges: a co-created scoping review

Abstract

Background

Recovery Colleges (RCs) are education-based centres providing information, networking, and skills development for managing mental health, well-being, and daily living. A central principle is co-creation involving people with lived experience of mental health/illness and/or addictions (MHA). Identified gaps are RCs evaluations and information about whether such evaluations are co-created.

Aims

We describe a co-created scoping review of how RCs are evaluated in the published and grey literature. Also assessed were: the frameworks, designs, and analyses used; the themes/outcomes reported; the trustworthiness of the evidence; and whether the evaluations are co-created.

Methods

We followed Arksey and O’Malley’s methodology with one important modification: “Consultation” was re-conceptualised as “co-creator engagement” and was the first, foundational step rather than the last, optional one.

Results

Seventy-nine percent of the 43 included evaluations were peer-reviewed, 21% grey literature. These evaluations represented 33 RCs located in the UK (58%), Australia (15%), Canada (9%), Ireland (9%), the USA (6%), and Italy (3%).

Conclusion

Our findings depict a developing field that is exploring a mix of evaluative approaches. However, few evaluations appeared to be co-created. Although most studies referenced co-design/co-production, few described how much or how meaningfully people with lived experience were involved in the evaluation.

Keywords:

• Recovery college
• evaluation
• scoping review
• co-creation

Introduction

Personal recovery in mental health and addictions (MHA) is defined as living a purposeful, meaningful life despite the presence of mental distress (Slade, 2010). While recovery-oriented practices existed in European healthcare settings as early as 200 years ago, their appearance in North America at the national policy level did not occur until the late 1980s in the U.S. (Substance Abuse and Mental Health Services Administration [SAMHSA], 2004) and the early 2000s in Canada (Kirby & Keon, 2006). This introduction was due to persistent consumer/survivor/ex-patient advocacy (Morrow & Weisser, 2012) and epidemiological evidence that substantial symptom reduction could occur in mental illnesses previously thought to be “incurable” (Harding & Zahniser, 1994).

The key conceptual shift introduced by recovery is that the focus is the individual not the symptoms. The concept of recovery “…involves making sense of, and finding meaning in, what has happened; becoming an expert in your own self-care; building a new sense of self and purpose in life; discovering your own resourcefulness and possibilities and using these, and the resources available to you, to pursue your aspirations and goals” (Perkins et al., 2012, p. 2). In this context, social inclusion can be a significant goal for many people working toward recovery (Mental Health Commission of Canada, 2015).

However, despite decades of efforts to incorporate a recovery approach into the mental health care system, people with lived experience of mental health/illness and/or addictions (MHA) continue to confront inequitable social inclusion, including high rates of un- and under-employment and low rates of educational achievement (Whitley et al., 2019). In response, recovery colleges (RCs) were developed and first implemented in 2009 in the United Kingdom. They have since been established in Australia, Canada, Hong Kong, Ireland, Japan, and the United States (Perkins et al., 2018). Drawing on educational theories such as transformative and constructivist learning (Hoban, 2015), RCs provide education and skills development courses to help manage and navigate daily living.

A critical principle is that RCs are co-created by people with lived experience of MHA and people with other forms of relevant expertise (e.g. mental health professionals, administrators, and researchers). Co-creation encompasses a wide range of activities including co-design, co-production, collaborative implementation, and mutual decision-making and planning. It can be defined as multiple stakeholders with various perspectives and areas of expertise, including lived experience, collaboratively identifying gaps and subsequently producing, developing, implementing, and evaluating programs (Brandsen et al., 2018). Co-created programming produces a culture of equity and inclusion where everyone’s assets, strengths, resources, and networks are celebrated and leveraged in pursuit of a common goal (Lewis et al., 2017). Co-creation allows for the right to self-determination and choice to be exercised by all stakeholders. It is thus central to providing those with lived experiences the opportunity to gain skills, confidence, hope, and resilience through the shift from “patient” to “student.” By re-centering the voices of lived experience, the diverse nature of consumer needs and understandings of neurodiversity become visible (Happell et al., 2003).

There is promising evidence that RCs increase positive recovery outcomes such as connection, hope, meaning, and empowerment (Thériault et al., 2020). However, researchers also note the need for more evaluation. A recent systematized search found almost no robust evaluative research on how RCs work and what outcomes they produce (Toney et al., 2018). This is a significant concern given the rapid global expansion of RCs.

There are complexities to developing evaluation measures for RCs. A fundamental issue is the complex nature of recovery that encompasses one’s attitude, values, feelings and goals (Mead et al., 2000). As such, recovery and the intended impacts of RCs include both objectively measurable (e.g. re-hospitalization, employment, return to school) and subjectively defined outcomes (e.g. increased confidence, improved resilience).

Our response is to conduct a scoping review to map out what work has been reported and identify important knowledge gaps (Arksey & O’Malley, 2005). This would serve as a first step for establishing an evaluation framework and designing corresponding methodologies, data collection, and analysis tools. However, given the centrality of co-creation to all aspects of RCs, it is crucial that this principle is also applied within the context of a meaningful and rigorous review.

Our purpose is to describe the process and results of a co-created scoping review designed to answer the question: How have RCs been evaluated in the published and grey literature?

This review is part of a larger grant that aims to combine the review results with qualitative research findings to identify recovery-oriented processes and outcomes that matter most to RC participants and use this information to co-create how RC evaluation measures will be selected (Lin et al., 2022).

Method

Overview

We followed Arksey and O’Malley’s (2005) methodology with one important modification. “Consultation,” originally an optional last stage, became our first, foundational step. We re-conceptualized this as “co-creator engagement.” The purpose was to ensure that RC student/facilitator participation was integrated throughout the review process.

The PRISMA extension for scoping reviews (PRISMA-ScR) was used to ensure our process and results met the current standards for reporting on knowledge synthesis research (Tricco et al., 2018). Covidence, a systematic review management software program, was used to import citations, screen titles and abstracts, upload references, and for data abstraction (https://www.covidence.org/).

Step 1: Co-creator engagement

Although we describe co-creator engagement as a “step,” it was imbedded throughout the scoping review. To ensure the inclusion of the RC student/facilitator perspective, individuals with the relevant experiences were recruited and eventually constituted two-thirds of the team. To create a common knowledge base, education sessions were a standing agenda item, with the sessions addressing the voiced needs of the team members. Topics included presentations by team members on their own lived experiences with mental health challenges; introductions to scoping review methods, evaluation, qualitative research, epistemology and ontology; the philosophy and history of RCs; and a walkthrough of a complex analytic publication by a researcher team member.

Processes were established to address the power differentials inherent in co-production teams. Every meeting began with a “check-in” where members were encouraged to share what had happened in their lives since the previous meeting. The purpose of this style of communication is to establish that all are valuable contributors with lives outside of the project, equalize power relations across the different roles and types of expertise, and provide unique perspectives to the team. Sharing personal daily experiences and individual challenges creates a culture of trust essential to fostering reciprocity and mutual respect (Peters et al., 2013).

As the team developed, we added a recorded “check-out” at the end of each meeting. The purposes were to share reflections on how the meeting went and generate a potential database on co-production processes. A structure was also developed so that members uncomfortable with any aspect of the meetings were encouraged to consider the options of raising them to the group or discussing them privately with either the team or project leaders.

The team routinely revisited issues such as whether the scoping review questions and search terms were adequate; whether the data abstraction plans and templates needed revision; and how the results should be analyzed, interpreted, and communicated.

Step 2: Identifying the research question

Our research question was: how have RCs been evaluated in the published and grey literature?

Related questions included what theoretical frameworks, approaches, and designs were chosen; how trustworthy was the evidence provided; what kind of impact did the evaluations have; and, most importantly, how were people with lived experience involved in the evaluation?

Step 3: Identifying relevant studies

The librarian (TR) designed a comprehensive search strategy based on consultation with the team. A preliminary sample of abstracts were test-reviewed and used to refine the strategy. To identity relevant scholarly literature, TR then translated and executed the finalized strategy in the following bibliographic databases: Medline, Embase, APA PsycInfo, Cumulative Index of Nursing and Allied Health Literature (CINAHL), Education Resources Information Center (ERIC), Applied Social Sciences Index and Abstracts (ASSIA), and the International Bibliography of the Social Sciences (IBSS). A “wide angle” approach was used to ensure broad coverage and include all relevant disciplines and articles regardless of study design or article type (Levac et al., 2010). Publication year limits were from the date of each database’s inception to 18–19 January 2021. No language limits were applied. (See full search strategy, Supplementary Material A.) In addition, national and international RC networks were consulted and reference lists of review articles handsearched to ensure that evaluation studies were not missed. Each database search was rerun on 17 March 2022 to capture articles published in the 14 months since the original searches were conducted.

To identify relevant grey literature, team experts and national and international RC networks were consulted to develop a list of known RCs and recovery-oriented organizations around the world. TR handsearched each program or organization’s website to locate evaluation reports, feedback materials, or documents that included robust metrics. TR also conducted multiple Google searches using keywords such as “recovery college” AND “evaluation” and advanced search operators (e.g. filetype:pdf) to identify any additional relevant documents. All identified materials were presented to the team in an Excel spreadsheet in preparation for screening. During the location of the full texts associated with this initial list, a further two grey literature documents were found and added to screening. Figure 1 shows the PRISMA-ScR flowcharts for our peer-reviewed and grey literature searches.

Figure 1. PRISMA flow diagrams: peer-reviewed and grey literature.

Stage 4: Study selection

A two-step selection process was performed. First, titles and abstracts were screened; then full-text screening followed for those articles or reports that met criteria. The team discussed potential inclusion and exclusion criteria which were then applied to a random selection of four articles and four reports. Team discussions led to modifications of the criteria and refinement of the team’s understanding of what these criteria meant. These discussions continued until the team was satisfied with both the criteria and the screening process.

Five inclusion and one exclusion criteria were chosen (Table 1). Criterion 2 was identified as especially complex and difficult to apply. The team discussed extensively questions such as “how do we know they are a ‘real’ Recovery College as opposed to an organization that simply uses the label?” or “what characteristics should they have before they count as a true Recovery College?” Since RC fidelity guidelines are neither mandated nor universally accepted and since it was not within our scope to dictate a definition, we opted to include all studies that assessed an organization that either called itself a Recovery College or whose main focus was clearly educational rather than therapeutic.

Table 1. Inclusion and exclusion criteria for abstract screening template.

Inclusion criteria 1. The article or report must be about a primary study (as opposed to secondary analyses such as literature reviews). 2. It must explicitly mention “Recovery College” or an equivalent term that showed that education was the main approach to supporting or improving recovery. 3. Students must include people with lived experience. 4. The article or report must be related to mental health and/or the social determinants of health. 5. It must provide details about the evaluation methods and findings. Exclusion criterion 6. The article or report is only about clinical treatment or interventions.

Each screening step was performed by two independent reviewers. General concerns about how to apply the criteria were discussed and resolved in the group’s weekly meetings. For reviewer differences regarding specific article/reports, a designated team member who was not a screener (SS) made the consensus decision.

Step 5: Charting the data

The 43 articles and reports that passed the screening process then underwent full abstraction. The development, piloting, and modification of the abstraction template followed the same method used for the inclusion/exclusion criteria. A draft template was co-created and piloted on two peer-reviewed articles and one grey literature report and then discussed and modified by the team. Each article or report was abstracted by two independent reviewers with general concerns about how to apply the template discussed and resolved by the group and specific differences between reviewers resolved by a third team member (SS) who did not perform any abstractions.

The final abstraction template covered two main domains: descriptive and evaluation information. (For the full template, see Supplementary Material B.)

Descriptive information included the RC’s name, its geographic location, the year it was founded, and who was eligible to attend. The team also judged whether any aspect of the RC or the evaluation itself was co-produced or co-created. “Co-production” and “co-creation,” like the definition used for “Recovery College,” were liberally defined. Essentially, any mentions of words such as “co-create,” “co-produce,” “co-facilitate,” or “co-design” were coded as “yes” while the absence of such words was coded as “information not provided.” If abstractors were unsure, they used an “uncertain” code. For all “yes” and “uncertain” codes, the abstractors were provided supporting information from the study in the template.

Information about the evaluation included what theory or framework was used, whether qualitative or quantitative approaches were taken, the study and analysis design, and what outcomes were measured or themes identified. In keeping with Arksey and O’Malley (2005), we did not assess study quality. However, we felt that addressing the “trustworthiness” of the results was important. Consequently, for qualitative studies, we noted whether the authors described processes supporting the credibility of the results or reflection by the evaluators. For quantitative studies, we recorded whether conflict of interest statements or evidence of instrument reliability or validity was provided. Also, for quantitative evaluations, we noted whether influences other than RC attendance were considered – for example, analyzing covariates or control/comparison groups. Finally, we were interested in any reported impacts or consequences of the evaluation.

Step 6: Collating and summarizing the results

Pairs of team members collated and summarized the abstraction results for the descriptive information, evaluation frameworks, and co-production/co-design sections while the lead author (EL) reviewed all of these summaries. Occasionally, the original articles or reports were re-examined to verify that the important details had been correctly captured. These summaries were then integrated into a complete final draft by the lead author (EL) and then reviewed, discussed, and modified by the full team.

Results

Descriptive information (Table 2)

Of the 43 evaluations we reviewed, 34 (79%) were peer-reviewed, and nine (21%) were grey literature. These latter included six reports (Anglia Ruskin University, 2014; Barton & Williams, 2015; Kaminskiy & Moore, 2015; Kenny et al., 2020; Taylor et al., 2017; Wels, 2016), one PowerPoint presentation (Arbour et al., 2019), one conference poster (Mayo Recovery College, n.d.), and one white paper (Canadian Mental Association Calgary, 2020). The earliest evaluation in our search was published in 2008, with numbers beginning to increase in 2014 and reaching a peak in 2018 and 2019 (Figure 2). The numbers decreased by a half to two-thirds in 2020 and 2021.

Figure 2. Number of peer-reviewed and grey literature evaluations by year.

Table 2. Descriptive information of RCs provided in peer-reviewed and grey literature.

Study	Publication year	RC Name	Location	Year founded	Label used	Eligible participants in addition to PWLE
Bourne et al.	2018	Sussex Recovery College^a	UK	2013	RC	Family Professionals
Burhouse et al.	2015	Pop-up Recovery Colleges, Gloucestershire and Herefordshire	UK	Not Stated	RC	Family Professionals
Cameron et al.	2018	The Imagine project	UK	2017 (data collection is from 2015 to 2016.	RC	Family
Critchley et al.	2019	Starting from Scratch	UK	2016	RC	–
Cronin et al.	2021	South Eastern Sydney Recovery and Wellbeing College	Australia	2014	RC	–
Dunn et al.	2008	Recovery Centre	United States	Not Stated	Recovery Center	–
Durbin et al.	2021	Supporting Transitions and Recovery Learning Centre (STAR)^b	Canada	2014	Recovery Education Centre	–
Ebrahim et al.	2018	Not Stated (recovery college in Northern England)	UK	2014	RC	–
Frayn et al.	2016	Discovery Centre (Langdon Secure Hospital)	UK	2012	Discovery Centre	Family Professionals
Hall et al.	2018	Mind Recovery College	Australia	2013	RC	Family Professionals
Harper & McKeown	2018	Recovery College	UK	Not Stated	RC	–
Hopkins, Foster et al.	2018	discovery college (South East Melbourne)	Australia	2014 started, established in 2016	Discovery College	Family Professionals
Hopkins, Pedwell et al.	2018
Hopkins et al.	2022
Kay & Edgley	2019	Health and Wellbeing College	UK	2016	RC	Family Professionals Public
Khan et al.	2020	Supporting Transitions and Recovery Learning Centre (STAR)^2	Canada	(2014)	Recovery Education Centre	–
King et al.	2019	(survey of multiple RCs)	International	Various	RC	–
Lucchi et al.	2018	Programma Formazione, Opportunita per la Recovery (FOR)	Italy	2013	RC	Family Professionals Public
Meddings et al.	2014	Sussex Recovery College^1	UK	(2013)	RC	Family Professionals
Meddings et al.	2015
Meddings et al.	2019
Muir-Cochrane et al.	2019	pilot RC, Southern Adelaide Local Health Network	Australia	2016	RC	–
Newman-Taylor et al.	2016	Recovery College in South of England	UK	Not Stated	RC	Family Professionals
Nurser et al.	2017	Recovery College East	UK	2013	RC	Family Professionals
Peer et al.	2018	Recovery Centre (VA Medical Center)	United States	Not Stated	Recovery Education Center; Recovery Center	–
Reid et al.	2019	The Supporting Transitions and Recovery Learning Centre (STAR)^2	Canada	(2014)	Recovery Education Centre	–
Sommer et al.	2018	South Eastern Sydney Recovery College	Australia	2014	RC	Family Professionals
Sommer et al.	2019
Stevens et al.	2018	Make Your Mark (Sussex Recovery College)	UK	2015	RC	–
Sutton et al.	2019	Recovery, Health and Wellbeing Academy (within the Greater Manchester Mental Health NHS Foundation Trust)	UK	2013	Recovery Academy	–
Thompson et al.	2021	Sussex Recovery College^1	UK	(2013)	RC	–
Wilson et al.	2019	South East Essex Recovery College	Canada	2017	RC	–
Windsor et al.	2017	Devon Recovery Learning Community	UK	Not Stated	RC	–
Zabel et al.	2016	Recovery Health and Wellbeing Academy	UK	2013	Recovery Academy	Family Professionals
Grey literature
Anglia Ruskin University	2014	Mid Essex Recovery College	UK	2013	RC	Family Professionals Public
Arbour et al.	2019	Ontario Shores’ Recovery College	Canada	2016	RC	–
Barton & Williams	2015	“The Exchange”—Barnsley Recovery College	UK	2015	RC	–
Canadian Mental Health Association	2020	CMHA Calgary Recovery College	Canada	Not Stated	RC	–
Kaminskiy & Moore	2015	South Essex Recovery College (SERC)	UK	2014	RC	Family Professionals Public
Kenny et al.	2020	Dublin North, North East Recovery College (DNNERC)	Ireland	2016	RC	Family Professionals Public
Mayo Recovery College	n.d.	Mayo Recovery College	Ireland	2013	RC	Family Public
Taylor et al.	2017	Mid West ARIES (Advancing Recovery in Ireland Education Service) Project	Ireland	2016	Recovery Education Project	Family Professionals Public
Wels	2016	Greenwich Recovery College	UK	2014	RC	Family Professionals

^aSame RC was evaluated by Bourne et al., 2018; Meddings et al., 2014, 2015, 2019; and Thompkins et al.

^bSame RC was evaluated by Durbin et al., 2021; Khan et al., 2020; and Reid et al., 2019.

Because some RCs were evaluated more than once, these 43 papers and reports represented 33 individual RCs. The majority were from the UK (19, 58%), followed by Australia (5, 15%), Canada and Ireland (3 RCs each, 9% each), the USA (2, 6%), and Italy (1, 3%). Additionally, one evaluation surveyed multiple countries (King & Meddings, 2019). The majority of the individual RCs were initiated between 2013 and 2016 (23, 70%) with one beginning in 2012 (Frayn et al., 2016), two in 2017 (Cameron et al., 2018; Wilson et al., 2019), and the remaining seven providing no information (Burhouse et al., 2015; Canadian Mental Association Calgary, 2020; Dunn et al., 2008; Harper & McKeown, 2018; Newman-Taylor et al., 2016; Peer et al., 2018; Windsor et al., 2017).

Twenty-six RCs (79%) used the label “Recovery College.” The remaining seven used a variety of terms including “Recovery Academy,” “Recovery Education Center,” “Recovery Education Project,” “Recovery Center,” “Discovery College,” or “Discovery Centre.”

Involving people with lived experience, since it was an inclusion criterion, was endorsed by all of the RCs represented in Table 2. Nineteen RCs also described other groups who were eligible to attend. The most commonly mentioned were family members (100% of the 19 RCs), health and other professionals (17, 89%), and the public (7, 37%).

Table 3. Evaluation information provided in peer-reviewed and grey literature.

Study	RC name	Theory or evaluation framework	Purpose/outcome	Design	Sample	Data collection methods^a	Analysis	Outcomes Themes
Bourne et al., 2018	Sussex Recovery College	–	Assess impact of RC attendance on MH service use	Quant. Pre-post	436 RC students who used MH services	Archival data	Descriptive stats	System-level Occupied bed days Admissions Admissions under mental health act Days on community treatment order Community contact
Burhouse et al., 2015	Pop-up Recovery Colleges, Gloucestershire and Herefordshire	–	Assess sense of hope and recovery	Qual.	50 RC students	Focus groups Recovery journals	Thematic analysis	Student-level Self-defined goals Self-reported goal outcomes
				Quant. Pre-post		Registration info Questionnaires Instruments: Manhattan Recovery Measure	Not stated	RC/course-level Confidence Goal acquisition Hope Sense of recovery
Cameron et al., 2018	The Imagine project	Social learning theory	Evaluate design and development of a resilience course	Qual.	9 RC students 4 RC tutors	1-On-1interviews Course materials	Thematic analysis	RC/course-level Prior experience and design Co-delivery Course methods and environment
Critchley et al., 2019	Starting from Scratch	BASIC Ph (Lahad et al., 2013)	Evaluate dramatherapy courses	Qual.	RC students attending dramatherapy courses	Not described	Not described	6 Themes: Psycho-education Dramatherapist Dramatherapy Social resilience and peer support Experts with experience and coproduction Arts wing of the RC
				Quant. Pre-post		Basic Ph Model	Coding of group poems in terms of individual coping skills and resiliency	RC/course-level Pre-post bar graphs of group poem coding
Cronin et al., 2021	South Eastern Sydney Recovery and Wellbeing College	–	Assess impact of RC attendance on health service use	Quant. Pre-post Covariates: multivariate stats	184 RC students	Student RC enrolment forms Linked medical records	Calculation of costs using units of activity Binomial regression	System-level Community mental health attendance MH inpatient admissions, length of stay Involuntary hospital length of stay Emergency department use Costs
Dunn et al., 2008	Recovery Centre	–	Assess health, mental health, and subjective and role functioning	Quant. Baseline, 6- and 12-months Covariates: Comparison group Multivariate stats	97 RC students; 81 comparators	1-On-1 interviews Instruments: SCL-90 SF-36 Basis 32 Tennessee Self Concept Scale Brief Quality of Life Interview Empowerment Scale PVRQ RAQ-7	Independent t-tests Chi-square Repeated measures ANCOVA	RC/course-level Mental wellbeing Symptoms Health and mental health Functioning, quality of life Empowerment Personal beliefs Attitudes
Durbin et al., 2021	Supporting Transitions and Recovery Learning Centre (STAR)	Realist Eval. Framework	Assess effectiveness of RC compared to usual care on recovery outcomes	Quant. Baseline, 6- and 12-months Covariates: Age-gender matched conttrols	184 Participants (92 RC students, 92 matched controls)	Instruments: Rogers Making Decisions Empowerment Scale QoLi-20 INSPIRE QPR-15 SF-12 EQ-5D-5L Pearlin Mastery Scale Modified Residential Time-Line Follow-Back Inventory	Descriptive stats t-test Fisher’s exact test Linear mixed effect models	RC/course-level Confidence Goal acquisition Mental wellbeing Perceived Empowerment Disease-specific quality of life Health-related quality of live Quality of recovery-supporting care Recovery Physical Health
Ebrahim et al., 2018	(Recovery college in Northern England)	Shepherd et al. (2014): 6 domains for evaluating recovery-oriented services: quality of recovery-supporting care, achievement of recovery goals, subjective measures of personal recovery, achievement of socially valued goals, quality of life, service use	Evaluate effectiveness of RC	Qual.	89 RC students	Open-ended survey items Course feedback forms	Thematic analysis	System-level Self-reported service use RC/course-level Goal acquisition Confidence Well-being
				Quant. Pre-post	56 RC students	Instruments: Short WEMWBS	Kruskal-Wallis test Paired t-tests	RC/course-level Mental well-being Personal resources
Frayn et al., 2016	Discovery Centre (Langdon Secure Hospital)	–	Process, outcomes, and challenges of developing the RC	Qual.	8 RC students & facilitators	1-On-1 interviews Quotes from feedback forms	Not stated	RC/course level Satisfaction RC Course attendance Formal course qualifications
				Quant.	Male patients with forensic needs	Feedback forms	Descriptive stats
Hall et al., 2018	Mind Recovery College	The evaluation framework was designed in collaboration with people with lived experience but a specific type of framework was not described.	Examine RC implementation processes; Measure intermediate outcomes	Qual.	51 Participants from 4 groups: RC students Families and carers Staff Community stakeholders	RC documents and materials 1-On-1 interviews Focus groups	Framework analysis Content analysis Thematic analysis	System-level Self-reported volunteer or paid RC positions RC/course-level RC process
				Quant.		On-line survey Instruments: MASS DREEM	Descriptive stats	System-level Adherence of RC to recovery-oriented principles RC/course-level Satisfaction
Harper & McKeown, 2018	Recovery College	–	Explore student motivations, facilitators, and barriers for RC attendance	Qual.	4 RC students	1-On-1 interviews	Thematic analysis	RC/course-level 4 Themes (“making the effort, being ‘too unwell,” friendly environment, glad I came’)
Hopkins et al., 2018	discovery college (South East Melbourne)	–	Explore experiences in establishing RC in a clinical service	Qual.	12 Service managers and RC staff	1-on-1 interviews (10 participants) Written answers to interview (2 participants)	Thematic analysis	RC/course-level 4 Main theme clusters establishing RC; organisational context; nature of RC; service transformation)
Hopkins et al., 2018			Explore reasons for enrolling, RC experience, and attitude change	Qual.	64 RC students	Open-ended questions (online)	Thematic analysis Grounded theory	RC/course-level Improve knowledge Improve self-knowledge Learn something new Satisfaction
				Quant. Pre-post		Online survey	Paired t-tests	RC/course-level Attitudes towards education
Hopkins et al., 2022		–	Feasibility of extending RC to adult inpatient setting	Qual.	7 Staff	1-On-1 interviews	Thematic analysis	RC/course-level Positive effects of expanding RC
				Quant.	Patients, staff, consumers: 23 completed feedback forms 12 online surveys	Written course feedback forms Online feedback survey	Descriptive stats	RC/course-level Satisfaction Achievement of learning objectives
Kay & Edgley, 2019	Health and Wellbeing College	–	Evaluate health outcomes and cost efficiencies	Qual.	176 RC students who were users of mental health care services	Not stated	Not stated	Student-level Selected case studies and quotations
				Quant. Pre-post		Administrative mental health care data Instruments: PAM WEMWBS	Descriptive stats	System-level Costs Employment RC/course-level Mental wellbeing Patient activation
Khan et al., 2020	Supporting Transitions and Recovery Learning Centre	Realist informed interview guide	Investigate enablers for RC participation	Qual.	Adults transitioning from homelessness	1-On-1 interviews	Thematic analysis	RC/course-level Barriers: Homelessness Enablers: Welcoming, respectful environment Seamless access Co-production
King & Meddings, 2019	survey of RCs	–	Outline of the commonalities and differences among RCs	Qual.	RCs in 22 countries	Open-ended survey questions	Themes identified separately by authors and discussed.	RC/course-level 7 Themes (recovery, culture change, coproduction, lived experience, education and learning, inclusivity, sustainability)
				Quant.		Survey	Descriptive stats	System-level Costs RC/course-level Rationale for developing RC Operational model Adjustments to RC model
Lucchi et al., 2018	Programma Formazione, Opportunita per la Recovery (FOR)	–	Evaluation of first year of RC	Qual.	34 RC students 20 RC staff	Focus groups	Not stated	RC/course-level RC development phase Staff commitment
				Quant.	34 RC students	Questionnaire developed by RC	Descriptive stats	Student-level Satisfaction Goal acquisition Expectations Quality of life
Meddings et al., 2014	Sussex Recovery College	–	Perspectives on RC and what makes RC effective	Qual.	40 RC students	1-on-1 interviews	Thematic analysis	RC/course-level 9 Categories: Learning from other students Co-production Safe environment New knowledge Social opportunity Daily structure Choice Curriculum Supporting learning
				Quant.		Course feedback forms	Descriptive stats	RC/course-level Satisfaction Confidence Likelihood of recommending RC to others Knowledge/skill improvements
Meddings et al., 2015		–	Evaluation of RC (student outcomes, potential efficiencies)	Qual.	35 RC students	Site admin records 1-on-1 interviews Feedback forms	Not stated	RC/course-level Quotes provided. Most common goals: Confidence Knowledge Meet others
				Quant. Pre-post		Questionnaire Instruments: QPR MANSA WEMWBS	Descriptive stats	RC/course-level Satisfaction Mental wellbeing Goal acquisition Learning outcome attainment Quality of life Resource use
Meddings et al., 2019		–	Evaluation if RC students are representative of larger community	Quant.	All registered RC students (exact n not stated)	Census data Admin data re mental health service use RC registration data	Descriptive stats Chi-square	System-level Equity of access by potentially marginalized groups
Muir-Cochrane et al., 2019	Pilot RC, Southern Adelaide Local Health Network	–	Exploring experiences and perceptions of those involved with RC	Qual.	RC students, facilitators, care coordinators	1-on-1 interviews Focus groups	Thematic analysis	Student-level Three main themes: (i) hope (ii) identity (iii) RC as a transition space
Newman-Taylor et al., 2016	Recovery College in South of England	–	Impact of RC	Qual.	11 RC students	Group interviews	Thematic analysis	Student-level 1 Main theme ‘connecting with others differently; 3 Subthemes: Reflection on stuckness Relationships and change Widening horizons
Nurser et al., 2017	Recovery College East	–	Changes in recovery and internalized stigma	Qual.	58 RC students	Comments in self-reported questionnaires	Not stated	Illustrative quotes
				Quant. Pre-post		Self-reported questionnaire Instruments: QPR ISMI	Paired t-tests	RC/course-level Recovery-related progress Internalized stigma
Peer et al., 2018	Recovery Centre (VA Medical Center)	Framework based on 3 principles of recovery and 2 barriers was used to organize presence/absence of processes and procedures as well as outcomes to measure each process/procedure.	Evaluate ease of access, zero-exclusion, other recovery-oriented procedures	Qual.	781 Veterans referred to RC	Focus groups	Not stated	RC/course-level Implementation considerations 2 Barriers: uneven access to MH programs coordinating with outside unit
				Quant. Comparison of feasibility measures across two referral sources of veterans		Program administrative data	descriptive stats t-tests F-tests Chi-square	RC/course-level Referral rate Attendance Scheduling
Reid et al., 2019	The Supporting Transitions and Recovery Learning Centre (STAR)	Realist framework	Explore participant experiences and outcomes of participation in RC	Qual.	20 RC students	1-on-1 interviews	Thematic analysis	Student-level Supporting recovery themes Judgement-free environment Supportive relationships Mutuality, role modeling De-construction of self-stigma Reclaiming power Personal outcomes themes Improved physical/-mental health Self-esteem, confidence Empowerment, control Personal responsibility
Sommer et al., 2018	South Eastern Sydney Recovery College	–	Exploring experiences regarding participation in RC	Qual.	19 Consumers, carers, MH clinicians, educators	Consumer and staff focus groups	Thematic analysis	Student-level Connection with others Hope for future Lived experience Changing attitudes and systems
Sommer et al., 2019		Goal Attainment Scale (GAS) framework	Evaluate goal attainment for all students completing at least two learning plans	Quant. Pre- and up to 3 follow-up points Covariates: Multivariate modeling	64 RC students	Student enrollment forms (admin data) Instruments: GAS	Multinomial logit regression models	RC/course-level Goal acquisition Kinds of goals set by students Characteristics of goal setting
Stevens et al., 2018	Make Your Mark (Sussex Recovery College)	–	Assess change in wellbeing and arts participation	Qual. 3-, 6-, and 9-months post	11 RC course facilitators/ artists 28 RC students	1-on-1 interviews Focus groups Anonymous course feedback forms	Thematic analysis Fishers exact test (for a priori themes)	RC/course-level 3 Themes: transforming student lives development as trainers overcoming hurdles.
				Quant. Pre-post	Between 14 and 28 RC students, depending on tool	Instruments: WEMWBS CHOICE	Paired t-tests Wilcoxon signed ranks test Exploratory pattern matching analysis of qual and quant findings	RC/course-level Satisfaction Mental wellbeing Pre-post participation in arts
Sutton et al., 2019	Recovery, Health and Wellbeing Academy (within the Greater Manchester Mental Health NHS Foundation Trust)	–	Health economic impact of course attendance	Quant. Pre-post	108 RC students (47 attendees, 61 non-attendees)	Instruments: Modified CSSRI-EU	Chi-square Mixed ANOVA tests	System-level Employment status Job seeking Primary income source Healthcare use
Thompson et al., 2021	Sussex Recovery College	The Framework Method	Outcomes, experiences, perspectives of RC students 1-year post-course attendance	Qual.	15 RC students	1-On-1 interviews	Thematic analysis Data analyzed using Framework Method and inductive processes	Student-level 3 Themes: Ethos of recovery and equality Springboard to opportunities Intrapersonal changes
Wilson et al., 2019	South East Essex Recovery College	–	Student experience of RC	Qual.	17 RC students	Focus groups Free-text responses to quantitative questionnaire	Thematic analysis Grouping of free-text responses (method not stated)	RC/course-level themes Course length and follow-up Student-level themes Enjoyment Mental health benefits Social benefits Communication and organization
			RC impact on wellbeing and social inclusion	Quant. Baseline, 3- and 6-month follow-up	19–25 RC students	Instruments: Short WEMWBS SIS	Paired t-tests	RC/course-level Satisfaction Confidence Mental wellbeing Social inclusion Motivation and attitudes Relationships Skill development
Windsor et al., 2017	Devon Recovery Learning Community	–	RC process and outcomes	Qual.	5 Facilitators 5 Students	Focus groups (separate for facilitators and students)	Thematic analysis	RC/course-level Process Themes (i.e. why the college “works,” etc.) Outcome Themes: Confidence; Wellbeing; Increased motivation; Increased social interaction; Link to new resources; Changed work behaviour, Empowerment; Changed perceptions
Zabel et al., 2016	Recovery Health and Wellbeing Academy	–	Explore RC experiences and impact	Qual.	21 RC students	Focus groups	Thematic analysis	Student-level 4 Main themes: RC ethos; Personal and organisational impact; Co-production; Barriers
Grey literature
Anglia Ruskin University, 2014	Mid Essex Recovery College	–	Explore student experiences	Qual.	47 RC students	1-On-1 interviews Focus groups	Thematic analysis	Student-level Echoed quantitative findings: increased confidence, self-worth, coping skills and positive thinking social inclusion benefits plans to attend mainstream courses significance for social inclusion, including reduced isolation and increased social activity. Of particular significance, almost all the participants had already started or were planning to start mainstream courses as a move on from or in addition to courses at the Recovery College.
			Assess wellbeing, social inclusion outcomes	Quant. Pre-post		Instruments: WEMWBS SIS	Paired t-test Wilcoxin rank Spearman’s Rho	RC/course-level Satisfaction Confidence Mental wellbeing Social inclusion
Arbour et al., 2019	Ontario Shores’ Recovery College	–	Explore participation, attendance, knowledge	Qual.	RC students (exact n not stated)	Not stated	Not stated	Illustrative quotes
			Overview of impact of RC participation	Quant.			Descriptive stats	RC/course-level Satisfaction Confidence Mental wellbeing Goal acquisition
Barton & Williams, 2015	“The Exchange”—Barnsley Recovery College	Four strand evaluation framework (general data analysis; return on investment, social return on investment, individual stories) developed by RC	Return on investment of RC Costs of care	Quant. Pre-post	50 RC students	Administrative data: patient records and financial information	Descriptive stats	System-level Cost of health care six months before and after RC attendance
Canadian Mental Health Association, 2020	CMHA Calgary Recovery College	–	Explore perceptions and progress	(Qual.?)	RC students (exact n not stated)	Not stated	Not stated	(Many student quotes but sources not stated)
				Quant.		Bespoke instrument (PROMS)	Descriptive stats	Student-level Skill or information acquisition Connections Feelings of hopefulness
Kaminskiy & Moore, 2015	South Essex Recovery College (SERC)	4-Part evaluation informed by Moore’s framework. And ImROC practice guidelines	Impacts and experiences of course attendance; Areas for RC development; Importance of co-production and peer facilitators	Qual.	12 RC students 2 Peer facilitators	Focus groups Group interviews Open-ended survey questions	Thematic analysis	RC/course-level Barriers and enablers
				Quant. Pre-post	18 RC students	Instruments: QPR	Repeated measures t-tests	RC/course-level Satisfaction Goal acquisition Skill acquisition
Kenny et al., 2020	Dublin North, North East Recovery College (DNNERC)	Proof of concept evaluation based on three parts: course evaluations; focus group research, and college evaluations	proof of concept evaluation	Qual.	16 RC students	Course evaluation sections Focus groups Qualitative sections of college evaluations	Not stated	RC/course-level Factors contributing to learning: Safe group learning; Peer support Student-level Factors hindering learning: Own fears; Course time
				Quant. Pre-post (group goals)	40 RC students	Course evaluations Quantitative sections of college evaluations Instruments: CHIME	Descriptive stats	RC/course-level Confidence Group goals CHIME factors: Connection with others Hope Identity Meaningful participation
Mayo Recovery College, n.d.	Mayo Recovery College	–	Their RC experiences; What else they wanted to include; Most important aspect of RC	Qual.	RC students (exact n not stated)	Anonymous submissions to Recovery Educator	Not stated	Student-level Testimonial letters Examples of responses to the questions, “What else would you like to be included?” and “what did you enjoy most about the courses?”
Taylor et al., 2017	Mid West ARIES (Advancing Recovery in Ireland Education Service) Project	CHIME Framework	Self-reported recovery outcomes	Qual.	RC students Facilitators Medical and other health professional university students (exact n not stated)	Open-ended survey questions	Thematic analysis of random sample of 8 feedback forms	Student-level 20 Themes; key themes: Safety, Personal experiences, Equality, Connection, Support, Managing recovery
				Quant.		Self-report questionnaire based on CHIME	Descriptive stats	Student-level Self-reported recovery
Wels, 2016	Greenwich Recovery College	–	Evaluate RC short- and medium-term effectiveness; Evaluate financial effectiveness and impact on students; Identify main improvements and future main elements	Quant.	First wave of RC students (75% response rate, but exact n not stated)	RC administrative documents Student survey	Descriptive stats	System-level Actual vs budgeted costs Tutor feedback Student-level Confidence Goal acquisition Perceived safety/respect Learning

For description of abbreviations and sources of standardized instruments, see the following list:

Standardized Instruments: explanation of abbreviations and sources.

Basic PH: Belief, Affect, Social, Imagination, Cognitive, Physiological (Lahad et al., 2013).

Basis 32: Behavior and Symptom Identification Scale (Eisen et al., 1994).

Brief Quality of Life Interview (Lehman et al., 1994).

CHIME framework: connectedness, hope and optimism about the future, identity, meaning in life, empowerment (Leamy et al., 2011).

CHOICE: CHoice of Outcome In Cbt for psychosEs Scale (Greenwood et al., 2010).

CSSRI-EU: Client Socio-Demographic and Service Receipt Inventory – European version (Chisholm et al., 2000).

DREEM: Developing Recovery Enhancing Environments Measure (Dinnis et al., 2007).

Empowerment Scale (Rogers et al., 1997).

EQ-5D-5L (EuroQol Office, n.d.).

GAS: Goal attainment scale (Kiresuk & Sherman, 1968).

INSPIRE: source of measures of Quality of Recovery-Supporting Care (Williams et al., 2015).

ISMI: Internalized Stigma of mental illness scale (Boyd et al., 2014).

Manhattan Recovery Measure (Davies, 2011).

MANSA: Manchester short assessment of quality of life (Ridgway & Press, 2004).

MASS: Mind Australia Satisfaction Survey (Ridgway & Press, 2004).

Modified Residential Time-Line Follow-Back (Tsemberis et al., 2007).

PAM: Patient Activation Measure (Hibbard et al., 2004).

Pearlin Mastery Scale (Pearlin & Schooler, 1978).

PVRQ: Personal Vision of Recovery Questionnaire (Ensfield et al., 1999).

QoLi-20: Quality of Life Index-20 Item (Uttaro & Lehman, 1999).

QPR, QPR-15: Questionnaire about the Process of Recovery (Neil et al., 2009).

RAQ-7: Recovery Attitudes Questionnaire (Borkin et al., 2000).

Rogers Making Decisions Empowerment Scale (Rogers et al., 1997).

SCL-90: Symptom Checklist-90 (Derogatis & Lazarus, 1994).

SF-12, SF-36: Short Form Survey (12 and 36 item versions) (J. Ware et al., 1995; J. E. Ware et al., 1995).

SIS: Social Inclusion Scale (Secker et al., 2009).

Tennessee Self Concept Scale (Fitts & Warren, 1996).

WEMWBS: Warwick-Edinburgh Mental Wellbeing Scale (Tennant et al., 2007).

Table 4. Co-production information provided in peer-reviewed and grey literature.

Study	RC name	“Co-production” mentioned vis-à-vis RC	Evaluation
			Co-produced?	If yes, how?	If uncertain, why?
Bourne et al., 2018	Sussex Recovery College	Yes	Yes	Each stage discussed and developed at monthly RC meetings.	–
Burhouse et al., 2015	Pop-up Recovery Colleges, Gloucestershire and Herefordshire	Yes	Information not provided	–	–
Cameron et al., 2018	The Imagine project	Yes	Information not provided	–	–
Critchley et al., 2019	Starting from Scratch	Yes	Information not provided	–	–
Cronin et al., 2021	South Eastern Sydney Recovery and Wellbeing College	Uncertain	Uncertain	–	Collaboration process but unclear if persons with lived experience involved
Dunn et al., 2008	Recovery Centre	Information not provided	Information not provided	–
Durbin et al., 2021	Supporting Transitions and Recovery Learning Centre (STAR)	Information not provided	Uncertain	–	Service-user input mentioned
Ebrahim et al., 2018	Not Stated (recovery college in Northern England)	Yes	Information not provided	–	–
Frayn et al., 2016	Discovery Centre (Langdon Secure Hospital)	Yes	Information not provided	–	–
Hall et al., 2018	Mind Recovery College	Yes	Yes	Detailed information on how evaluation was co-designed and co-implemented.	–
Harper & McKeown, 2018	Recovery College	Uncertain	Yes	working group of seven former RC students collaborating with research team regarding aims, design and procedures	–
Hopkins et al., 2018	Discovery college (South East Melbourne)	Yes	Information not provided	–	–
Hopkins et al., 2018		Yes	Information not provided	–	–
Hopkins et al., 2022		Yes	Information not provided	–	–
Kay & Edgley, 2019	Health and Wellbeing College	Yes	Information not provided	–	–
Khan et al., 2020	Supporting Transitions and Recovery Learning Centre	Yes	Information not provided	–	–
King & Meddings, 2019	survey of RCs	Uncertain	Information not provided	–	–
Lucchi et al., 2018	Programma Formazione, Opportunita per la Recovery (FOR)	Yes	Information not provided	–	–
Meddings et al., 2014	Sussex Recovery College	Yes	Information not provided	–	–
Meddings et al., 2015		Yes	Yes	Team of potential students, trainers, service providers co-produced methodology	–
Meddings et al., 2019		Uncertain	Information not provided	–	–
Muir-Cochrane et al., 2019	Pilot RC, Southern Adelaide Local Health Network	Yes	Information not provided	–	–
Newman-Taylor et al., 2016	Recovery College in South of England	Yes	Information not provided	–	–
Nurser et al., 2017	Recovery College East	Yes	Information not provided	–	–
Peer et al., 2018	Recovery Centre (VA Medical Center)	Information not provided	Information not provided	–	–
Reid et al., 2019	The Supporting Transitions and Recovery Learning Centre (STAR)	Yes	Information not provided	–	–
Sommer et al., 2018	South Eastern Sydney Recovery College	Yes	Information not provided	–	–
Sommer et al., 2019		Yes	Information not provided	–	–
Stevens et al., 2018	Make Your Mark (Sussex Recovery College)	Yes	Uncertain	–	Parts of the evaluation may have been co-developed/co-facilitated. Consultation on one tool. Focus group co-facilitated by peer expert.
Sutton et al., 2019	Recovery, Health and Wellbeing Academy (within the Greater Manchester Mental Health NHS Foundation Trust)	Uncertain	Uncertain	–	Consultation re language and practical issues
Thompson et al., 2021	Sussex Recovery College	Yes	Yes	Lived Experience Advisor recruited and paid to shape, develop, and pilot the evaluation	–
Wilson et al., 2019	South East Essex Recovery College	Yes	Information not provided	–	–
Windsor et al., 2017	Devon Recovery Learning Community	Uncertain	Information not provided	–	–
Zabel et al., 2016	Recovery Health and Wellbeing Academy	Yes	Yes	semi-structured interview created collaboratively	–
Grey literature
Anglia Ruskin University, 2014	Mid Essex Recovery College	Yes	Information not provided	–	–
Arbour et al., 2019	Ontario Shores’ Recovery College	Yes	Information not provided	–	–
Barton & Williams, 2015	“The Exchange”—Barnsley Recovery College	Information not provided	Information not provided	–	–
Canadian Mental Health Association, 2020	CMHA Calgary Recovery College	Yes	Information not provided	–	–
Kaminskiy & Moore, 2015	South Essex Recovery College (SERC)	Yes	Information not provided	–	–
Kenny et al., 2020	Dublin North, North East Recovery College (DNNERC)	Yes	Information not provided	–	–
Mayo Recovery College, n.d.	Mayo Recovery College	Yes	Information not provided	–	–
Taylor et al., 2017	Mid West ARIES (Advancing Recovery in Ireland Education Service) Project	Yes	Uncertain	–	CHIME approach (co-created elsewhere) used but no information on how chosen for this study
Wels, 2016	Greenwich Recovery College	Yes	Information not provided	–	–

Evaluation frameworks (Table 3)

The definition of “evaluation framework” varies but can be conceptualized as “both a planning process and written product” that guides the conduct of evaluation (Markiewicz & Patrick, 2016, p. 1). Based on this definition, we identified 14 evaluations (33%) as using a framework.

Four employed frameworks that they identified as rooted in RC principles. Taylor et al. (2017) used the CHIME (Connectedness, Hope and Optimism, Identity, Meaning, Empowerment) framework (Leamy et al., 2011) while three studies were steered by ImROC guidance on quality and outcomes assessment for RCs (Shepherd et al., 2014). Ebrahim et al. (2018) drew on a framework, designed in partnership with ImROC, covering six areas of evaluation: quality of recovery-supporting care, achievement of recovery goals, subjective measures of personal recovery, achievement of socially valued goals, quality of life, and service use. Sommer et al. (2019) adopted a framework based on the Goal Attainment Scale (GAS)—an instrument and process outlined in the ImROC guidance (Kiresuk & Sherman, 1968). Kaminskiy and Moore (2015) used a 4-part evaluation framework informed by Moore’s process evaluation of complex interventions (2015) and the ImROC practice guidelines for choosing outcome domains.

Although not specifically informed by RC frameworks, two further papers noted that their evaluation was based on recovery concepts. Cameron et al. (2018) drew from social learning theory within the context of developing communities of practice (Wenger-Trayner & Wenger-Trayner, 2014), and Critchley et al. (2019) used the BASIC Ph Model of resilience (Lahad et al., 2013) in their evaluation of their RC’s dramatherapy program.

Three papers (Durbin et al., 2021; Khan et al., 2020; Reid et al., 2019), evaluating the same RC, used realist evaluation to develop a mixed-methods quasi-experimental design to assess the impact of RC participation on individuals with mental health challenges who had also experienced homelessness (Durbin et al., 2021). Another evaluation (Thompson et al., 2021) used the Framework Method (Ritchie & Lewis, 2003) to guide an inductive qualitative evaluation.

Finally, four evaluations used “bespoke” frameworks – that is, ones that they developed for their own purposes. This included one framework for evaluating financial and social return on investment (Barton & Williams, 2015) and one based on the principles of recovery (Peer et al., 2018). Two other evaluations described collaborative mixed-methods frameworks. Hall et al. (2018) used a co-produced design comprised of document review, interviews, surveys, and focus groups while Kenny et al. (2020) conducted a cross-sectional evaluation of outcomes experienced by RC members.

What evaluation approaches and designs were chosen?

There were 33 evaluations that used qualitative methods and 31 that used quantitative methods. Nearly half used both (49%, 16 peer-reviewed, 5 grey literature). Twenty-eight percent used qualitative methods only (11 peer-reviewed, 1 grey literature), and 23% used quantitative methods only (eight peer-reviewed, two grey literature). Two other evaluations included student quotations illustrating their quantitative findings but gave no description of how these were obtained (Canadian Mental Association Calgary, 2020; Critchley et al., 2019).

Sample size, data sources, data analysis

Sample sizes varied widely between four (Harper & McKeown, 2018) and 781 participants (Peer et al., 2018). While the qualitative evaluations tended to have smaller numbers, some analyzed open-ended questions embedded in quantitative surveys resulting in larger than usual qualitative samples (e.g. 89 students, Ebrahim et al., 2018).

For the 33 qualitative studies, the most common data sources were free-text information gathered from answers to open-ended survey questions, course feedback comments, or other forms of written communication such as solicited anonymous letters (48%, 12 peer-review, 4 grey literature). One-on-one interviews and focus groups were the next most common (45% and 39%, respectively). Only two studies used group interviews (Kaminskiy & Moore, 2015; Newman-Taylor et al., 2016). Thirty-six percent used two or more qualitative data collection methods.

The qualitative studies largely used thematic analysis (23 studies, 70%) with only one using grounded theory (Hopkins et al., 2018). Four used additional analytic methods. Hall et al. (2018) used their bespoke evaluation framework in addition to thematic analysis for their 1-on-1 interviews and focus groups and content analysis for their secondary analysis of RC documents. Stevens et al. (2018) used Fisher’s exact test, a quantitative method, to examine the relationship between themes identified from a qualitative interview and subsequent self-reported improvement. Thompson et al. (2021) used the Framework Method (Ritchie & Lewis, 2003) to support their thematic analysis, and Wilson et al. (2019) grouped responses to free-text answers to survey questions although methodological details were not provided.

Data sources for the 31 quantitative studies included administrative/archival databases, student surveys, and course feedback forms. Twenty-nine percent analyzed administrative or archival data such as student registration information, linked health service records, or financial/cost data while 71% conducted RC student surveys. Thirteen percent used course feedback information, and 19% used two or more of these sources of data.

Sixty-one percent (15 peer-reviewed, 4 grey literature) used a pre-post design. Of these, three (Dunn et al., 2008; Durbin et al., 2021; Sommer et al., 2019) also measured at least one additional time point (e.g. 1 year later).

In analyzing quantitative data, 48% (nine peer-reviewed, six grey literature) reported descriptive statistics only such as frequencies, unweighted percentages, or means. Forty-five percent (12 peer-reviewed, 2 grey literature) evaluated whether the differences they found were statistically meaningful using tests such as the Wilcoxon signed, Fisher’s exact, or paired-sample t-test. Four of these studies also controlled for covariates via multivariate modeling techniques (Cronin et al., 2021; Dunn et al., 2008; Durbin et al., 2021; Sommer et al., 2019), and three studies (Dunn et al., 2008; Durbin et al., 2021; Peer et al., 2018) included comparison groups.

Themes, outcomes

The purposes for which the evaluations were done fell roughly into three categories: to explore or evaluate impacts at the system-level, the RC or course-level, or the individual student level. Of the 43 evaluations, 26% (eight peer-reviewed, three grey literature) covered more than one category.

Ten evaluations (23%) gathered information about system-level outcomes with the most common being health care use and costs. Pre-post changes for RC students were examined by five studies using linked administrative health or financial data (Barton & Williams, 2015; Bourne et al., 2018; Cronin et al., 2021; Kay & Edgley, 2019; Sutton et al., 2019). Wels (2016) also evaluated the RC’s performance against the standards defined in their health system service level agreement.

Three studies considered impacts outside of health care. Sutton et al. (2019) also considered the impact of attending RC courses on employment status, primary income source, and job seeking behaviour. Hall et al. (2018) reported qualitative findings that some students became involved in RC volunteer or paid positions. Finally, Meddings et al. (2019) used census information to evaluate access by comparing diversity of the RC student body to that of the local community.

Not surprisingly, the majority of the 43 studies (28, 65%) focused on the RC/course level. Sixteen used primarily qualitative methods to explore aspects of RC development or implementation including facilitators and barriers, course design processes, and RC adherence to recovery principles (Cameron et al., 2018; Frayn et al., 2016; Hall et al., 2018; Harper & McKeown, 2018; Hopkins et al., 2022; Hopkins et al., 2018; Hopkins et al., 2018; Kaminskiy & Moore, 2015; Kenny et al., 2020; Khan et al., 2020; King & Meddings, 2019; Lucchi et al., 2018; Meddings et al., 2014; Peer et al., 2018; Wilson et al., 2019; Windsor et al., 2017). Some of the themes found in these studies included the shift from treatment to education, the value of co-facilitation, the utility of lived experiences of mental ill-health, redressing power imbalances, and social inclusion.

Twenty (47%) examined the impact of RC attendance particularly related to specific courses (Anglia Ruskin University, 2014; Arbour et al., 2019; Burhouse et al., 2015; Critchley et al., 2019; Dunn et al., 2008; Durbin et al., 2021; Ebrahim et al., 2018; Frayn et al., 2016; Hall et al., 2018; Hopkins et al., 2022; Hopkins et al., 2018; Kaminskiy & Moore, 2015; Kay & Edgley, 2019; Kenny et al., 2020; Meddings et al., 2015; Nurser et al., 2017; Sommer et al., 2019; Stevens et al., 2018; Wilson et al., 2019; Windsor et al., 2017). The majority of these studies used quantitative methods to describe cross-sectional results or assess pre-post changes. The most commonly measured outcomes were satisfaction (10 studies); well-being (6 studies); and recovery, gains in knowledge and skills, and achievement of the student’s learning goals (5 studies each). There were 28 standardized instruments used in these quantitative studies (see end of Table 3), but no universally adopted ones. The most frequently administered were the Warwick-Edinburgh Mental Well-Being Scale (WEMWBS) which measures well-being and psychological distress (Tennant et al., 2007) and the Questionnaire about the Process of Recovery (QPR) (Neil et al., 2009).

Some of these studies also included qualitative components which yielded a number of themes including many of the themes described above as well as personal transformation, increased motivation and recovery management skills, and a sense of hope and purpose.

Finally, 16 evaluations (37%) used qualitative methods to further elicit student perceptions and experiences. In addition to the themes reported by the RC/course-level studies, students mentioned topics such as the RC as a transformational space (Muir-Cochrane et al., 2019), appreciating the value of education (Anglia Ruskin University, 2014), addressing their own fears (Kenny et al., 2020), and the opportunity to widen their horizons (Newman-Taylor et al., 2016).

How trustworthy was the evidence provided?

While quality assessment of these studies was out-of-scope, we did abstract several characteristics related to the trustworthiness of the evaluation results. For qualitative studies, we noted whether the authors mentioned reflexivity or strategies to address credibility, two important processes in qualitative research to identify influences or biases in the results (Santiago-Delefosse et al., 2016). For quantitative studies, we noted where potential biases might be introduced by not considering non-RC factors, not disclosing conflict of interest, or not considering instrument reliability and validity.

Only one qualitative evaluation described reflection by the authors (Stevens et al., 2018) while 12 of the 33 studies (36%) mentioned strategies supporting credibility. For the quantitative studies, as described earlier, the potential influence of non-RC factors was addressed by five studies through including covariates (Cronin et al., 2021; Sommer et al., 2019), comparison groups (Dunn et al., 2008; Peer et al., 2018), or both (Durbin et al., 2021). Two quantitative studies included conflict-of-interest statements in the body of their publications (Bourne et al., 2018; Durbin et al., 2021) although others likely included them in journal sections not retrieved by our search process. Finally, 14 of the 31 quantitative studies either specifically mentioned the reliability and/or validity of their survey instruments or chose established questionnaires.

What kind of impact did the evaluations have?

Only four RCs (12%) reported consequences resulting from their evaluations. Impact on courses was reported by three studies in terms of modifying course duration and order (Cameron et al., 2018), refining curriculum committee processes (Hall et al., 2018), or improving registration (Arbour et al., 2019). Another RC used their evaluation results to improve their outreach efforts, particularly for underrepresented groups (Meddings et al., 2019), to change the RC’s training methods and governance, and to create processes that would support routine outcome measurement (Meddings et al., 2014). Finally, one RC reported using their results to support funding applications as well as to establish external partnerships (Hall et al., 2018).

How were people with lived experience involved? (Table 4)

Because of the importance of co-production and co-design in RCs, it is not surprising that the majority of the 43 evaluations (79% overall; 76% peer-reviewed, 89% grey literature) described some aspect of their RCs using terms such as “co-produced,” “co-developed,” “co-designed,” or “co-facilitated.”

However, only six studies (14%) provided specific information describing co-development or co-production of their evaluations with two describing fairly extensive processes. Hall et al. (2018) note their “deliberate strategy to ensure that the perspectives of people with lived experience was [sic] invited and valued through all aspects of the project” (p. 10). This included involving people with lived experience in the co-design and implementation of the evaluation framework, methods, and data collection materials and having them participate in a focus group to digest and distill the results. The study by Thompson et al. (2021) recruited and hired a previous RC student to serve as a Lived Experience Advisor. This individual helped shape their evaluation approach, developed and pilot-tested the data collection materials, and co-authored the publication.

Three other studies also reported the participation of people with lived experience in the research process or the evaluation working group. Meddings et al. (2015) noted that all methodological decisions were co-produced via consultation with both students and peer trainers as well as mental health professionals. In the study by Bourne et al. (2018), all stages were reviewed and discussed in monthly meetings with RC students, trainers, managers, researchers, and service providers. Harper and McKeown (2018) also described collaboration between a working group of former students with the research team in developing both broad and specific aspects of the evaluation.

Finally, Zabel et al. (2016) used an semi-structured interview guide that they described as collaboratively created.

The remaining evaluations either provided no (32, 74%) or limited information (5, 12%). For example, no mention was made of stakeholder collaboration, input, or consultation (Cronin et al., 2021; Stevens et al., 2018; Sutton et al., 2019). One study (Durbin et al., 2021) mentioned service user input but provided no details, and another (Taylor et al., 2017) reported using an evaluation approach that had been co-created elsewhere but not how the decision was made to adopt it.

Summary and discussion

The literature covered by this scoping review demonstrates a growing appreciation of the need for evaluation in RCs. The number of publications and reports almost quadrupled between 2017 and 2018/2019. There was a recent decrease in 2020 and 2021 possibly due to the impact of the global pandemic.

One-third used a theoretical framework. These frameworks tended to be recovery-based (e.g. CHIME, ImROC) rather than using traditional evaluation approaches such as Kirkpatrick (1994) or Moore et al. (2009). Almost half used both qualitative and quantitative methods, reflecting a combination of intentions to both explore and assess. Within these methods, a number of studies (36% for qualitative studies, 19% for quantitative studies) used more than one data-gathering method. Several evaluations took advantage of already gathered, secondary data (e.g. open-ended questions from quantitative surveys, administrative health data), either alone or together with primary data. For quantitative evaluations that assessed RC/course-level impact, almost half provided descriptive statistics only. However, a promising 45% tested for meaningful pre-post differences with a few using more complex multivariate or comparison group analyses to account for non-RC influences.

These 43 studies investigated a variety of outcomes or experiences at the system, RC/course, and student levels. Over one-quarter examined at least two levels. System-level evaluations focussed primarily on health care and related costs although a few studies included wider impacts such as volunteering, paid employment, and participation in further education. RC/course-level evaluations focused on both implementation processes and pre-post student outcomes. At the student-level, the results yielded information on the personal impact of attending RCs and of the underlying principles of inclusion and co-development that the RCs embodied.

Our findings depict a field that is developing, growing, and consequently exploring a mix of evaluative approaches. However, we also identified a number of areas that would benefit from further consideration.

The most striking is the small number of evaluations (14%) that appeared to be co-developed or co-produced. As a pillar of the RC model, co-production should be a guiding principle for all RC activities. While nearly every study referenced recovery and co-design/co-production in their introductions and descriptions of their RC, many did not provide clear information about how these principles were applied to their evaluations. In particular, very few described how much or how meaningfully people with lived experience were involved. There may be many explanations for this scarcity of information including that it was provided elsewhere (e.g. an internal report) or that the contributing individuals chose not to disclose their personal histories in published form. However, co-production is such a central principle that our review suggests that this is a sizeable information gap in the published literature that deserves considerably greater attention.

The second point is that the variety of study approaches, consistent with the developmental nature of RC evaluation, encompasses a mix of top-down and bottom-up methods. There is an inherent tension between the use of standardized tools and more individualized approaches that measure success by the individual’s own goals. The use of standardized tools is often consistent with the expectations and even requirements of external accountability audiences such as government ministries or funding agencies (Shepherd et al., 2014). The perceived advantage is that such tools allow us to compare and/or combine data with other RCs using the same instruments. Comparing or creating opportunities for larger sample sizes can generate information about gaps or strengths beyond the individual RC doing the evaluation. On the other hand, evaluations geared towards measuring success as defined by an individual’s own goals are better for creating a more complete picture of the human experience. These client-centered methods can provide in-depth information about the experience and impact of the individual RC on its participants, which is the driving value in recovery. In addition, evaluations that are for participants are more likely to be meaningful to RC participants and others with lived experience (Bloom, 2010).

Both approaches have strengths and weaknesses, and the variability in the studies we reviewed suggests that RCs are very much aware of them. We propose that the choice of which approach or mix of approaches should be tailored to the RC’s purpose (e.g. comparability with other RCs, in-depth exploration of their own students). Regardless of the purpose, our results suggest two important considerations. First, procedures establishing the trustworthiness of the evaluation results should be described. Second, as mentioned earlier, there should be close attention to how individuals with lived experience are actively and meaningfully involved in the evaluation. In particular, clearer descriptions of whether and how co-creation was used to choose methods, analyze findings, and prepare the final manuscript are critical even if they are worded broadly (e.g. individuals with lived experiences were actively involved vs. person XX was an individual with lived experience) to respect individual preferences about disclosing their backgrounds.

Finally, as the field develops, more attention is needed toward synthesizing findings, either via theoretical frameworks or by integrating quantitative and qualitative analytic methods. For example, Stevens et al. (2018) used a creative combination of quantitative and qualitative tools to evaluate an RC arts course. They conducted a pattern-matching analysis where they used Fisher’s exact test to determine whether the presence of certain themes identified at a 3-month follow-up interview were associated with being categorized as having improved wellbeing three or six months later. They synthesized these findings with pre-post test scores from standardized instruments, deductive qualitative analyses of post-course focus groups with facilitators, and written student feedback. In this way, they sought to balance the tension across evaluation priorities to provide a nuanced exploration of program outcomes.

Strengths and limitations

As with all scoping reviews, this project is not without limitations. The lack of a universally accepted definition of Recovery College and associated fidelity criteria means that there is variability in how this term is used. Consequently, we chose to broadly include studies using “Recovery College” or an equivalent term indicating that education was the main approach to supporting or improving recovery for students. An additional limitation is that only English-language literature was reviewed. Future directions should address scholarship presented in different languages. Also, due to limited resources, we were unable to update our grey literature search in 2022; therefore, grey literature published after January 2021 was not included.

Nonetheless, this project offers a significant and unique contribution to the scholarship landscape. To our knowledge, this is the first peer-reviewed publication examining the methods and outcomes used to evaluate Recovery Colleges. This report also offers unique insight into the ways in which the methods and outcomes are chosen. Given that co-design is central to the Recovery College model, an important future direction would be to engage various stakeholders to co-design an evaluation strategy for Recovery Colleges that is relevant and impactful to students.

Acknowledgements

The support and contributions of Sanjeev Sockalingam are gratefully acknowledged.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Canadian Institutes of Health Research under SPOR Catalyst Grant [# 424837].