Best-practices for co-design processes involving people with severe mental illness for eMental health interventions: a qualitative multi-method approach

Abstract

The lack of adoption of eMental Health technologies by people with severe mental illness (SMI) might be explained by a mismatch between technology design and users’ skills, context and preferences. Co-design can optimize this fit, but populations labelled as ‘vulnerable’ are often excluded or misrepresented. The goal of this study is to gain insight into best-practices for co-design with people with SMI. A qualitative, multi-method approach was used, consisting of a systematic scoping review of 21 included studies, 25 co-design expert surveys and six participant interviews. The results delivered 23 best-practices divided into four overarching aspects of co-design, namely: (1) activities to carry out before the start of a co-design study; (2) fruitful collaboration of the co-design team; (3) bespoke approach within co-design to accommodate the skills and abilities of SMI participants; and (4) mitigation of challenges surrounding power balance. The best-practices may help researchers and designers offer the SMI population a more specialized approach for co-design, which can cause the innovative output of eMH projects to be more effective and better adopted. Throughout the co-design process, more attention should be paid to the personal and clinical benefits of participation for the participants themselves.

Keywords:

• co-design
• user-centred design
• severe mental illness
• eMental health
• eHealth
• best-practices

Introduction

Globally, around 1% of the population has a severe mental illness (SMI) such as bipolar disorder, schizophrenia and major depressive disorder (Dekkers et al. 2022; Naslund et al. 2015a). In order to be classified as SMI, these diagnoses must have persisted for at least two years, during which care was required from various coordinated disciplines to treat disability in functioning (Delespaul and de consensusgroep EPA 2013; Wiersma 2006). People with SMI’s access to care is reduced due to treatment barriers, such as long waiting times (Rekenkamer 2020), costly therapy and stigma (Batra et al. 2017). This reduced access to care has negative effects on an individual’s health outcome (Reichert and Jacobs 2018) and quality of life (Rekenkamer 2020). Because of barriers to mental healthcare or limited staff in inpatient clinics, patients can benefit from innovative alternatives to in-person treatment to self-manage- and support their wellbeing (Rotondi et al. 2017). Websites, serious games and mobile applications have the potential to offer an effective, accessible and scalable way to ensure access to- and quality of care (Gaebel et al. 2016; Jonathan et al. 2019; Torous et al. 2021; Ben-Zeev et al. 2019), as well as to address the disparities in mental healthcare (Biagianti, Hidalgo-Mazzei, and Meyer 2017). The use of technology to support mental health, wellbeing and healthcare is also referred to as eMental Health (eMH) (Gemert-Pijnen et al. 2018).

eMH interventions are increasingly used by the SMI population (Naslund et al. 2015b). They see eMH as feasible and acceptable (Naslund et al. 2015b; Batra et al. 2017). Examples of eMH solutions deemed (cost-)effective are a computerized cognitive behavioural therapy (CBT) program for anxiety and depressive disorders (Andrews et al. 2010), a virtual reality (VR) CBT intervention for people with psychosis (Pot-Kolder et al. 2020), and a smartphone application that challenges maladaptive beliefs in SMI related domains (Ben-Zeev et al. 2021). In general, research shows that clinical outcomes of eMH interventions are better than waiting list conditions and comparable to in-person interventions (Richards et al. 2020). Thus, eMH may provide individuals with SMI access to treatment while on waiting lists for in-person treatment or as a blended care approach. However, despite high levels of acceptability and evidence, the use of eMH in practice falls behind expectations with regard to uptake (Vis et al. 2018; Schreiweis et al. 2019).

A mismatch between the eMH intervention and the user’s skills, abilities, context and preferences might explain this lack of adoption (Gemert-Pijnen et al. 2018). SMI-related cognitive deficits – such as difficulty with abstract reasoning, decreased attention span, low literacy levels and reduced working memory – can harm navigating and interpreting eMH systems (Rotondi et al. 2017). Ill-fitted designs that do not accommodate the abilities of the target group result in poor usability and thus low uptake (Rotondi et al. 2017; Biagianti, Hidalgo-Mazzei, and Meyer 2017). Rotontdi et al. (2015) found that more user-friendly and effective eMH designs for people with SMI were seen in designs that required a reduced cognitive effort through easy-to-understand text, minimal use of text, content grouping and memory aids (Rotondi et al. 2015). By improving technology design, eMH may show higher rates of adoption, resulting in more impact on clinical practice. To optimize the match between the technology and the consumer, the users’ needs and abilities need to be taken into account.

One way to include user preferences in eMH interventions is to develop the technology together with the end-user through co-design (Biagianti, Hidalgo-Mazzei, and Meyer 2017). Co-design facilitates a bottom-up approach to technology design that involves end-users as equal partners (Orlowski et al. 2016). This specific branch of user-centred approach advocates for the collaborative effort between different stakeholders throughout the development process (Hardy, Wojdecka, Jonathan, et al. 2018; Sanders and Stappers 2008). Since this approach allows for the perspectives of end-users to be incorporated in the design of eMH technology from the start, a better fit between user, technology and context can be achieved (Gemert-Pijnen et al. 2018).

However, despite the benefits of co-design, populations labelled as ‘vulnerable’ are often excluded or misrepresented (Moll et al. 2020). While the active role of people with SMI in co-design is developing, their roles and responsibilities in the process are not yet clearly defined (Groot, Haveman, and Abma 2022). Similarly, researchers require a new set of knowledge and skills due to a role shift from translators to facilitators (Sanders and Stappers 2008). Ideally, co-design is best executed by highly skilled and experienced researchers, as otherwise the approach may be less successful (Orlowski et al. 2016). This especially holds true for working with people with SMI. Researchers need to apply an array of tools and methods while accounting for the level of skills of people with SMI, they also need to be sensitive to the challenges within psychiatry - such as power imbalance (Groot, Haveman, and Abma 2022; Moll et al. 2020), stigma and participant distress (Mulvale et al. 2016). Then, there are practical hurdles in the form of strict research criteria such as specific budget distributions and limited timeframes (Conder, Milner, and Mirfin-Veitch 2011), along with resource scarcity that make the collaboration seem less feasible to researchers (Moll et al. 2020; Biagianti, Hidalgo-Mazzei, and Meyer 2017). To support researchers in shaping co-design processes, thus developing better eMH interventions and improving their effectiveness, there is an urgent need for more knowledge on how to best involve people with SMI in co-design (Moore et al. 2019; Moll et al. 2020; Biagianti, Hidalgo-Mazzei, and Meyer 2017; Orlowski et al. 2016; Ramirez and Coskun 2020).

The aim of this study is to gain insight into the best-practices for co-design with people with SMI. An overview of best practices to guide co-design is created through the findings of a qualitative multi-method approach that include a scoping review, co-design expert survey and client interviews. The following research questions (RQ) will be addressed:

RQ 1: What relevant experiences, tips, best-practices and lessons learned are reported in scientific literature by e(Mental) Health studies that co-designed with people with SMI?

RQ 2: What are considered best-practices and key recommendations for co-designing with people with SMI by people with experience in conducting co-design with vulnerable target groups?

RQ 3: What are the experiences and preferences of people with SMI that participated in a co-design project on how to be included?

Materials and methods

The qualitative multi-methods followed the three pillars of evidence-based medicine, namely, (1) scientific literature, (2) practitioners’ expertise and (3) client values (Masic, Miokovic, and Muhamedagic 2008). A scoping review was used to gain insight into the lessons learned of eMH co-design studies targeting people with SMI. These lessons functioned as the foundation for the best-practices. The following two methods were used to adapt the initial list. An online survey rendered insight into the best-practices according to experts in co-design with vulnerable groups. Lastly, semi-structured interviews were held with people with SMI who participated in the co-design of an app to gain insight into their opinions and preferences for co-design involvement.

Scoping review

A scoping review was deemed appropriate as the topic under review was broad (Peters et al. 2015). This review followed two guidelines for scoping reviews (Arksey and O'Malley 2005; Peters et al. 2015). Additionally, the PRISMA extension for scoping reviews was used (Tricco et al. 2018).

Sources and search strategy

Scopus, PubMed and three health design journal archives (i.e. CoDesign, Design for Health and The Journal of Health Design) were searched with the same search string in February of 2021. The search string was based on a scoping review about co-design with people with dementia (Wang et al. 2019). The search string for this study focussed on (1) psychiatric patients and (2) co-design, leading to the search string: ‘(“psychiatr*” OR “mental healthcare” OR “mental illness”) AND (“participatory development” OR “co-creation” OR “co-design” OR “participatory design” OR “co-production” OR “generative design” OR “Scandinavian design” OR “participatory research”)’.

Eligibility criteria and selection process

The only source characteristic used as eligibility criteria was to only include peer-reviewed articles. No quality appraisals were conducted. After removing duplicates, a single round of snowballing was applied to the primary included sources to look for additional titles. One researcher (SES) conducted most of the review. However, during each stage of the screening process, two researchers (HK, TD) were continuously available to discuss any doubts surrounding an article’s inclusion or exclusion. Different in- and exclusion criteria were used for title, abstract and full-text screening to allow for an increasingly narrowed focus.

• During the title screening, one researcher (SES) included all titles that concerned co-design;

• During the abstract screening studies that solely involved non-SMI participants were excluded. Additionally, studies not written in English or Dutch were excluded. The abstract screening was performed independently by two researchers (TD, SES) with an even split. Beforehand, adequate inter-reviewer reliability of $\kappa$ = 0.84 was reached during a pilot screening of five abstracts (McHugh 2012);

• During the full-text screening, one researcher (SES) included studies that concerned the co-design of eHealth, and peer-reviewed studies. Studies were excluded if (a) the abstract or full-text article could not be obtained, (b) the preferences of psychiatric inpatients or people with SMI were only surrogated through other participants – such as caregivers, family members, friends or staff members – as those cases had no direct participation of people with SMI, (c) studies did not carry co-design out themselves, thus non-primary studies (e.g. systematic reviews and editorials) and

• the diagnosis or severity of the participants’ mental illness were not mentioned.

Scoping review analysis

The main categories for the data extraction form were study aim, technology, target group, goal, design framework, SMI involvement phase, methods, tools, environmental setting, organizational setting, and lessons learned. An overview and substantiation of the (sub-)categories can be found in Supplementary Appendix I (see Supplemental data). Relevant data was copied verbatim from manuscripts. Once this process was completed, the extracted data in the ‘lessons learned’ category underwent an iterative, inductive coding process according to the Grounded Theory approach (Boeije 2010). One researcher (SES) familiarized themselves with the content, by repeatedly reading the data. Best-practices were formulated based on the codes resulting from the inductive coding process. This process was discussed with two other researchers (HK, TD) until consensus was reached.

Survey with experts

Ethical approval for the survey was obtained by the ethics committee of the Behavioural Management and Social Sciences (BMS) faculty at the University of Twente in the Netherlands (#210121).

Participants and setting

The survey target group were adults (≥18 years) who conducted co-design with vulnerable participants in relation to eHealth. Supplementary Appendix II includes the survey where vulnerable target groups are defined as ‘people in (socially) vulnerable situations who, as part of this vulnerability and a lack of resources, have an increased risk or susceptibility to adverse (mental) health outcomes. For example, due to poverty, low literacy, poor housing, an immigrant background, frailty, or (severe) mental illness. This includes related terms such as “complex” target groups and “difficult-to-reach” target groups’. The use of this terminology is consistent with the jargon used in mental healthcare. It is used throughout the text to distinguish between participants in and outside the target group. Further, the scope of the survey was broadened to vulnerable groups to learn from co-design experiences with other populations that face similar challenges as the SMI population. It was deemed appropriate as the SMI population is a subsection of the vulnerable target group.

The respondents were recruited through convenience sampling within the networks of the development team of a self-control training application for the SMI population (Dekkers et al. 2022). In addition, the opportunity for snowball sampling was encouraged as the respondents were asked to further distribute the survey within their network.

Materials and procedure

The web-based survey was administered through ‘Qualtrics’ during March and April of 2021. The survey consisted of three parts (Supplementary Appendix II). First, an introduction page with the aim and scope of the research was shown, followed by an active online informed consent. Second, respondents were presented with background questions on their birth year, gender, nationality, employment role, employer, experience level, and target population. Third, a set of open-ended questions was presented that covered the most common design challenges faced by researchers in healthcare. These challenges were categorized into ‘challenges in practice’, ‘challenges in project management’ and ‘miscellaneous or generic challenges’, based on Groeneveld et al. (Groeneveld et al. 2018). Respondents were asked to reflect on the challenges they faced and to offer ways to overcome them. As this study was conducted during the COVID-19 pandemic, a specific section was added to cover remote co-design research.

Analysis

The survey data was deductively coded according to the coding scheme from the scoping review. The best-practices that were mentioned by the respondents were added under the relevant scoping review codes. Respondent data that was not covered by any codes were grouped under ‘miscellaneous’. The miscellaneous category underwent the same inductive coding process as the scoping review. New themes that emerged were added to the coding scheme.

Interviews with SMI participants

Group interviews were held after a series of co-design workshops on an eHealth development project for a self-control training smartphone application for the SMI population. Ethical approval for the interviews was granted by the ethics committee of the BMS faculty, domain Humanities and Social Sciences, at the University of Twente in the Netherlands (#210814).

Participants & setting

The interviews took place straight after the final workshops. In total, five workshops were held in June and July of 2021 at two different facilities (Dekkers et al. 2022). Each workshop lasted around 90 minutes. Three workshops were held at an inpatient forensic psychiatric facility where people with SMI are treated who have committed an offence. The other two workshops were held at an outpatient mental healthcare facility. The recruitment for the workshops was carried out in collaboration with care providers of both facilities through convenience sampling. Before the interviews were conducted, the workshop participants were informed about the purpose of the evaluation questions and asked for their willingness to participate. Participants (≥18 years) were deemed eligible if they had a SMI diagnosis that currently required care or had required care in the past, thus people with lived experience.

Materials and procedure

After concluding the final co-creation workshop, the group of participants were asked several interview questions as an evaluation of the co-design process. The interview topics offered insight into the opinions and preferences of participants regarding participation, workshop activities, communication and continued involvement (Supplementary Appendix III). A semi-structured approach was chosen to maintain the informal environment that was created during the co-design workshops preceding the interviews. The two group interviews were recorded and lasted 8 and 13 minutes.

Analysis

The recordings of the group interviews were transcribed verbatim, anonymized and analyzed using a deductive approach, using the merged codes from the scoping review and survey study. Relevant quotes from participants were processed with the same approach as the survey analysis. If the input mentioned by interviewees fully supported any existing best-practice, no alterations were made. Whereas, if the data offered new insight into an existing best-practice, they were adapted.

Results

Literature screening

1224 records were identified. After the removal of duplicates, 950 studies remained for the screening process. A final selection of 21 studies from the years 2008–2021 were included, after the application of the inclusion- and exclusion criteria. Most full-text articles were excluded as they did not concern health technology. Figure 1 displays the study selection process.

Figure 1. PRISMA flow diagram of scoping review..

Summary of included literature

Supplementary Appendix IV shows the study characteristics of the included records. Nine studies developed eHealth for people with psychosis (Berry et al. 2020; Sin et al. 2019; Hardy, Wojdecka, West, et al. 2018; Realpe et al. 2020; Nakarada-Kordic et al. 2017; Knight et al. 2021; Lambe et al. 2020; Laine, Anttila, and Välimäki 2016; Bucci et al. 2019), four for bipolar disorder (Matthews et al. 2015; Matthews et al. 2016; Marcu, Bardram, and Gabrielli 2011; Bardram, Frost, and Szanto 2012), four for people with SMI in general (Kip et al. 2019; Grim et al. 2017; Klein et al. 2019; Vilardaga et al. 2018; one specifically targeted forensic psychiatry (Kip et al. 2019)), three for schizophrenia (Terp et al. 2016; Ben-Zeev et al. 2013; Valimaki et al. 2008) and one for borderline personality disorder (Derks et al. 2017). The most common innovations were smartphone apps and internet-based interventions (n = 15). The three remaining innovations were information websites (n = 3), VR interventions (n = 3) and decision aids (n = 1). The main difference between internet-based interventions and information websites was the level of interaction: information websites offered static resources, whereas internet-based interventions facilitated interaction with the end-user.

People with SMI were mostly involved in the evaluative stage (n = 14), followed by the generative stage (n = 13) and pre-design stage (n = 12). Most studies involved participants in two or three stages (n = 16) and none involved participants in all stages. Participants were least involved in the post-design stage (n = 2). The methods that were most often carried out with SMI participants were interviews (n = 12) and usability testing (n = 11), followed by workshops and focus groups (n = 9 each). Other conducted methods with SMI participants were surveys (n = 6), observation (n = 1) and beta-testing (n = 3). Most studies (n = 19) used tell tools (Sanders and Brandt 2010) like group discussions and storytelling. Make tools (Sanders and Brandt 2010) such as sketches, wireframes and prototypes were used in 12 studies. Enact tools (Sanders and Brandt 2010), for example, scripting and role-playing were used in six studies.

The setting in which the studies took place was not specified in 11 out of 21 studies. Of the ones that did, most were conducted in a clinical setting such as a hospital or clinic (n = 9). Of these nine studies, five combined the clinical environment with additional settings: community centre (n = 2), online (a) synchronous communication (n = 2) and university plus participants’ home environment (n = 1). One study was conducted in a university setting combined with online synchronous communication. Individual sessions were often combined with group sessions (n = 13). Four studies only conducted group sessions and four other studies only conducted one-on-one sessions.

Survey respondent characteristics

25 out of 29 respondents completed the survey (86.2% completion rate). Most respondents came from the Netherlands (n = 16), followed by New Zealand (n = 5), United Arab Emirates (n = 2), Belgium (n = 1) and Bulgaria (n = 1). The majority of respondents carried out their research in knowledge institutions (n = 15). Most respondents had little to moderate experience in conducting co-design with vulnerable groups (n = 11 each). The roles in which they carried out their research were mainly that of the researcher (n = 10), lecturer (n = 5) or student (n = 5). Four respondents were involved in projects that targeted SMI or forensic psychiatry. Other target groups were people with a low-socio-economic status (n = 6) and people with dementia (n = 3). See Supplementary Appendix V for a complete list of target groups.

Workshop interviews

All six co-design workshop participants participated in the interviews. The participants included four people with SMI undergoing treatment and two people with lived experience. Three of the participants were male and forensic, while the remaining three were female and not forensic. The age of participants ranged from 29 to 57.

Best-practices

The following best-practices resulted from merging the lessons learned from each individual method. Four distinct categories with four to seven items each were found, namely (1) plan and structure study, (2) create and maintain a collaborative team, (3) accommodate vulnerable participants and (4) strive for equal partnership (see Table 1). The best-practices and recommendations mainly relate to overarching concepts of co-design projects, rather than specific methodological activities within such projects. Below Table 1, concrete examples are provided to illustrate how each item could be applied to practice. Supplementary Appendix VI provides an overview of Table 1 in more detail where the specific sources for each recommendation are listed. Throughout the following text, the sources for each concrete example are indicated in brackets with an author name for literature, ‘R#’ for a particular survey respondent or ‘I#’ for a specific interviewee.

Table 1. Best-practices for co-design processes involving people with SMI.

Best-practices		Sources
Plan and structure study		Review	Survey	Interview
1	Combine multiple methods (e.g. focus groups, interviews and design workshops) to gain different types of data and completeness of information.	x	x
2	Set up a flexible study design to allow for an iterative and adaptive development approach.	x	x
3	Determine the recruitment strategy in collaboration with stakeholders (e.g. vulnerable members, care organizations, supervisors, people with lived experience).	x	x	x
4	Secure the availability of sufficient resources (i.e. time, budget, materials and participants) in the early preparation phase of the study.	x	x	x
5	Provide and coordinate a clear structure between and within study activities (e.g. setting regular meetings).	x	x
6	Reflect with an open and critical mindset on the chosen methods, tools and materials.	x	x
Create and maintain a participatory team		Review	Survey	Interview
7	Collaborate with multiple stakeholders, including patients, to incorporate all relevant perspectives in the design-, development- and delivery process.	x	x	x
8	Ensure voluntary and informed participation by briefing participants on study goals and set-up, through clear but brief session introductions and informative flyers.	x	x
9	Contact team members and participants in between sessions to send reminders and updates, secure participation and provide them with support for their role as contributor.	x	x
10	Stimulate a collaborative work relationship between team members and participants through ice-breaker activities.	x	x	x
11	Provide transparency in design decisions by showing (concept) designs and explaining decision rationale.	x	x	x
Accommodate vulnerable participants		Review	Survey	Interview
12	Use concrete tools (e.g. scenarios, personas and prototypes) that account for the cognitive abilities (e.g. abstract reasoning and decreased attention span) of the SMI participants.	x	x	x
13	Adapt (the order of) research activities in a bespoke manner during data collection, according to the state and preference of the SMI participants.	x	x
14	Prior to the study, determine with participants how to solve practical barriers that might affect participation to improve (repeated) attendance.	x	x
15	Offer intrinsically valuable incentives (e.g. money, coupons, personal goals) to participants.	x		x
16	Employ researchers skilled in conducting qualitative research (e.g. asking effective probing questions) to guarantee quality of data.	x	x
17	Put measures in place that minimize the risk of harm or distress for vulnerable participants, such as having psychologists in attendance and offering ample opportunities for breaks.	x	x
Strive for power balance		Review	Survey	Interview
18	Approach participants with lived experience as experts and equal partners to minimize sense of power imbalance.	x	x	x
19	Stimulate equal dialogue and interaction by encouraging reserved members to provide input.	x	x
20	Conduct formal evaluations (e.g. brief interviews, exit questionnaires) with vulnerable participants to reflect on both the designs and the co-design process.	x	x
21	Create an informal and relaxed physical environment by altering artefacts (e.g. decorations and refreshments) during data collection sessions.	x
22	Enhance accessibility for participation by offering remote research methods to vulnerable participants.	x	x
23	Provide skills training on digital literacy and research methods to promote equal participation.		x

‘x’ marks the general source type that supports the recommendation.

Note that within items ‘team members’ refer to the stakeholders involved in the research team, which may include individuals with SMI, whereas ‘participants’ refer to stakeholders involved in separate data collection methods, which always include, but are not limited to, SMI participants. Furthermore, due to the clinical context in which this study took place, people with SMI may be referred to as ‘patients’ throughout the text in the recommendations.

Plan and structure study

The following items relate to activities to carry out before the start of the data collection for the co-design process.

(1) Combine multiple methods.

Data from the literature and survey respondents suggested combining multiple methods to bring forth different types of information. For example, focus groups – in which group discussions can be used for initial exploration of broad topics and reaching consensus – can be supplemented with one-on-one interviews to gather more in-depth information on the topics on which consensus was reached (Kip et al. 2019; Berry et al. 2020; Knight et al. 2021; Marcu, Bardram, and Gabrielli 2011).

(2) Set up a flexible study design.

Literature and the survey both showed the importance of a flexible and iterative study design with continuous cycles of data collection and evaluation to facilitate a constant generation of ideas (Matthews et al. 2015; Sin et al. 2019; Ben-Zeev et al. 2013), spontaneous bottom-up opportunities [R19] and rapid implementation of feedback (Matthews et al. 2015). It also allowed to take a step back in the design process when the project experienced drawbacks or required concepts to be revised [R22].

(3) Determine recruitment strategy.

It was viewed as important to pay close attention to the recruitment strategy of vulnerable participants. Partnerships with stakeholders in close connection with the target group – such as care organizations, supervisors, and people with lived experience – provided increased access to vulnerable participants [Berry et al. 2020, R1, R5, R11, R13, R20]. Involving vulnerable members in the creation of the recruitment materials allowed vulnerable participants to feel more included and less intimidated by avoiding jargon and using understandable wording [Berry et al. 2020, Terp et al. 2016, R4, R12, R13, R20]. An example towards the term of informal caregivers was provided: terms such as ‘caregivers’ could be replaced with ‘people who provide unpaid care for a spouse/family member’ [R4].

(4) Secure sufficient resources.

The review, survey respondents and interviewees emphasized the importance of securing resources such as time, budget, and necessary materials to facilitate the involvement of end-users and non-researchers in the development process. An example is ensuring funding for the contributions of a study ambassador (i.e. co-investigator, care staff) to arrange study logistics such as organizing a room, promoting the study and contacting participants (Berry et al. 2020).

(5) Coordinate study structure.

Through the literature, it was found beneficial to appoint a dedicated project manager to coordinate the different stakeholders and monitor the project structure before and during the data collection (Kip et al. 2019). The structured approach may be seen within the sessions (i.e. setting an explicit agenda, and choosing appropriate generative tools and techniques) (Terp et al. 2016) and between the sessions (i.e. setting regular meetings) (Kip et al. 2019).

(6) Adopt an open and critical mindset.

To let future users shape the innovation, the literature and survey data showed the importance to adapt an open mindset of (co-)researchers towards unconventional research methodologies, such as generative design methods. An open mindset allows researchers to step outside of their comfort zone and divert control to enable the mindset shift from ‘participation’ (i.e. contribution according to the needs of the researchers) to ‘co-design’ (i.e. serving the vision of the participant) [R15, R16]. Simultaneously, a critical mindset was needed to select and develop research methods, tools and materials that are suitable for the target group [R6].

Create and maintain a collaborative team

The following items regard strategies to ensure a fruitful collaboration between the co-design team members and participants.

(7) Collaborate with multiple stakeholders

Involving a range of stakeholders in the data collection was seen as beneficial to account for all relevant perspectives in the design of the technology, as was reported by all three data sources. In addition to patients [I6] and designers (Derks et al. 2017), it was advised to include clinicians (Marcu, Bardram, and Gabrielli 2011), management [R5] and caregivers [R7], as they play an important role in, for example, the implementation process. Further, by including patients in the project team, increased empathy and understanding of the research set-up and delivery of research methods could be achieved (Berry et al. 2020). For example, patients could resonate more deeply with the vulnerable participants during interviews due to their mutual experiences.

(8) Ensure voluntary and informed participation

Transparent communication before participation through participant briefing (Terp et al. 2016) could aid the alignment of participants during research activities [R5], and improve team functioning (Knight et al. 2021). Two main areas for informing were found. First, it was important to emphasize the relevance of the research and chosen approaches [R2, R7, R9, R13, R15, R17-R20]. Second, the practicalities of the data collection process must be shared, such as the agenda (Terp et al. 2016), aim (Knight et al. 2021; Terp et al. 2016; Kip et al. 2019), timeframe (Terp et al. 2016), design artefacts (Terp et al. 2016), roles and responsibilities (Knight et al. 2021; Kip et al. 2019), data-handling [R10] and the voluntary nature of participation (Terp et al. 2016). Additional strategies for informed participation were to have meeting minutes readily available with relevant information and decision rationale (Kip et al. 2019), to provide participants with the ability to ask questions throughout [R10]. Also, send information booklets before the meeting that includes the aforementioned information [R12, R21]. Such booklets may be personalized to the type of stakeholder and include various levels of detail or reading level [R12].

(9) Frequently communicate with participants

Throughout the co-design process, frequent communication in between sessions may be used to inform team members and participants on updates and activities or provide them with support for their role as contributors (Berry et al. 2020). It may also serve as a reminder of the importance of the study [R19] and to secure continued participation and a feeling of involvement [R10]. With regards to participants, Terp et al. (Terp et al. 2016), suggested for researchers to use an appreciative tone as a way to establish a relationship and sense of trust.

(10) Stimulate a collaborative work relationship.

Ensure that participants acknowledge the differing skillsets within the group during data collection as it could create empathy towards one another [Berry et al. 2020, R8]. A way to highlight the value of everyone’s contribution, was through ice breaker activities (e.g. ‘co-designing a pizza’) (Nakarada-Kordic et al. 2017). It was important to manage varying viewpoints [R1, R9, R18]. If stakeholder opinions conflicted, mediation could be achieved with the help of a framework helped guide the process. One specifically mentioned was the Patient-Clinician-Designer Framework (Marcu, Bardram, and Gabrielli 2011)

(11) Provide transparency in design decisions.

Show participants the (concept) designs to help them witness the impact of their input [I2, I5] and increase participant engagement [Nakarada-Kordic et al. 2017, R8]. If participants’ input could not be processed in the design, two actions were reported. First, to see if it was possible to find compromises within the time and budget constraints of the project (Berry et al. 2020; Marcu, Bardram, and Gabrielli 2011). Second, to provide reasoning on why certain changes were not possible (Berry et al. 2020).

Accommodate participants

The following items target strategies to offer the target group a specialized approach that accommodates their needs, limited capacity in abstract reasoning, reduced memory capacity and low concentration.

(12) Account for vulnerable participants’ skills and abilities

Limit the need for abstract reasoning by using concrete examples such as scenarios, personas, prototypes, familiar examples, storyboards, visualizations, and graphics [Nakarada-Kordic et al. 2017, Kip et al. 2019, Terp et al. 2016, R4, R13]. It also helped to test more mature prototypes [R4]. Further, to comply with low concentration levels, it was preferred to keep research methods such as interviews short [Kip et al. 2019, R12, I2, I3], to limit the amount of text and presented materials [R9, I2], to introduce the materials in increments [I2].

(13) Adjust activities and materials in a tailored manner

Offer a flexible and tailored approach to the research activities during data collection to help accommodate the varying needs of vulnerable participants. It was seen as helpful to set up pre-specified activities, but to tweak them during the execution depending on the needs of the participants (Knight et al. 2021; Terp et al. 2016; Nakarada-Kordic et al. 2017). Also ensure that the materials are adjusted to fit the comprehension level of participants [R1, R13].

(14) Solve practical barriers.

Before data collection, find ways to minimize the negative effects of practical barriers, such as time constraints, on the co-design process. Examples of strategies are to (1) consult participants in the scheduling process rather than offering pre-specified times (Berry et al. 2020), (2) visit participants in their living environment to learn to tailor communication and the co-design process to their needs and preferences [R21], and (3) set up the data collection at a location convenient for participants [I6].

(15) Offer personally relevant rewards.

Before the data collection, identify which rewards are relevant for participants (Kip et al. 2019). Then, provide such rewards as a token of appreciation to participants after each session or to stimulate ongoing engagement [I6]. The rewards may be extrinsic in the form of materials, grocery vouchers and refreshments. But, they can also be intrinsic where being endorsed as experts, sharing personal stories (Nakarada-Kordic et al. 2017), working in a team environment [I3, I6], gaining pleasurable experiences [I1–I6], creating something meaningful [I2], and knowledge attainment [I1] were mentioned as examples.

(16) Employ skilled researchers.

A way to uphold the integrity of data and maintain the open and explorative nature of co-design was to employ researchers skilled in qualitative methods to carry out the data collection. Examples of such skills were good interviewing practices (e.g. asking effective probing questions) (Kip et al. 2019), to understand what the participants were trying to say [R13], and remaining unbiased [R10]. Further, to achieve a balance between openly listening to participants’ stories and obtaining relevant data from researchers, a replicable data collection procedure was beneficial [R9].

(17) Minimize risk of distress

Put measures in place to decrease distress as participant distress may arise due to participants being exposed to, thinking of, and discussing experiences with SMI (Knight et al. 2021). Examples of such protective measures were to have psychologists, family and support workers in attendance (Knight et al. 2021), to offer ample opportunities for breaks [R11] and to exclude participants at any time if they experienced negative effects caused by the research activities, in consultation with care givers or supervisors [R3, R7].

Strive for equal partnership

Co-design creates an environment where stakeholders with varying knowledge and skill levels collaborate. Measures to mitigate challenges surrounding the creation and maintenance of power balance are needed when conducting co-design with vulnerable groups. Otherwise, there is an increased risk of vulnerable participants not being taken seriously. Thus, the following items offer ways in which the challenges surrounding power balance can be addressed before, during and after data collection.

(18) Promote equal partnership

Treat participants as equals and approach them as a ‘designer of need’ rather than a ‘patient in need’ (Terp et al. 2016). Prerequisites for a collaborative environment were trust [Nakarada-Kordic et al. 2017, Kip et al. 2019, R6, R13] and mutual respect (Kip et al. 2019), and a flat democratic structure where each participants’ input is seen as worthy [R5]. Specific tactics were to sit amongst the participants (Nakarada-Kordic et al. 2017), dress casually (Nakarada-Kordic et al. 2017), avoid jargon (Berry et al. 2020), discuss manners (Berry et al. 2020) and use first name introductions (Nakarada-Kordic et al. 2017),

(19) Facilitate involvement of vulnerable participants

Conduct co-design with multiple team members in attendance. Vulnerable participants might feel hesitant or uncomfortable to express their opinions due to the experience of power imbalance or personal reasons such as social anxiety (Grim et al. 2017). To combat this, a facilitator (i.e. co-researcher) could speak up for reserved participants and help them provide input by asking questions (Knight et al. 2021). To help facilitators observe group interactions and intervene accordingly, it was beneficial for co-researchers to sit amongst group members (Berry et al. 2020).

(20) Conduct evaluations

Conduct evaluations to determine participants’ satisfaction on the experience, pace, frequency and intensity of the sessions [Berry et al. 2020, R9]. The feedback could be implemented in future sessions. In expansion, one-on-one evaluations with vulnerable participants after the sessions could be documented and ‘and preserved as an aspect of the next decision or meeting’ (Grim et al. 2017). This would enable participants to express their honest opinion on decisions that were made during research activities if they felt uncomfortable to express them during the session.

(21) Alter physical environment.

To establish an environment that stimulates creativity and interaction, an informal physical surrounding was viewed as desirable (Terp et al. 2016). Strategies to support a relaxed ambience were to alter the environment in which co-design takes place with the use of physical artefacts such as music, flowers, drawings, decorations and providing refreshments (Terp et al. 2016).

(22) Enhance accessibility.

The survey question with regards to remote research methods was asked due to the circumstances during the COVID-19 pandemic. However, remote research methods can also be implemented in non-COVID-19 times [R8]. Therefore, offer remote research methods for participants that are unable to attend in person (Berry et al. 2020). This allows participants to attend in a comfortable environment with reduced social pressure [R25]. Ways to make the virtual engagements a success were to create smaller groups [R20], to test (mature) virtual prototypes to enable distanced usability testing [R23], to be extra cautious to use straightforward language when not able to see each other through video [R5] and to use programs appropriate for people with low (digital) literacy [R4]. However, when collecting data online, the preconditions necessary for conducting virtual methods, such as access to devices, work surface, good quality internet, should be available [R4, R12].

(23) Close knowledge and skills gap.

Invest in participants’ skill development to strive for equal participation by participants’ that lack digital-, research- or technical skills and experience [R8, R25]. Three tactics to close the knowledge gap were to (1) send the materials and information prior to the session to let participants prepare the session beforehand, (2) dedicate time within sessions to teach participants the necessary skills to effectively use the chosen tools, and (3) provide a phone number for additional support outside sessions [R20].

Discussion

This study aimed to uncover the best-practices for conducting co-design with people with SMI according to literature, co-design experts and people with SMI using a qualitative multi-method approach. In total, 23 best-practices were identified divided into four distinct overarching concepts of co-design projects. The four categories were: (1) ‘plan and structure study’ which include the necessary activities to carry out prior to the start of the data collection, (2) ‘create and maintain a collaborative team’ to ensure the fruitful collaboration of the co-design team, (3) ‘accommodate vulnerable participants’ which covers the bespoke approach within co-design to support the participants’ skills and abilities, and (4) ‘strive for equal partnership’ that helps mitigate the challenges surrounding power balance. The best-practices can be used by stakeholders, such as healthcare professionals or healthcare living labs, who want to pursue transdisciplinary collaborative design projects with designers and SMI end-users.

Based on the findings, central themes to what makes co-design with people with SMI successful were: diversity, flexibility, equal collaboration and communication. Diversity in people and methods were important to gain as much insight into the different stakeholder perspectives as possible. Flexibility in adapting the tools and activities was necessary to support the participants’ cognitive abilities. Finally, equal collaboration and communication was necessary to allow each participant with SMI to contribute throughout the process while being valued as experts and equal partners.

Implications for co-design

Previous studies have identified considerations for co-design in general, however, this study focussed on co-design involving people with SMI. Therefore, this study adds novel insight to the current literature on best-practices to conduct co-design with the SMI population. When comparing results with the current literature, several considerations seemed consistent to co-design, such as collaborate with multiple stakeholders and ensure voluntary and informed participation. It seems that such considerations require even more attention and adaptation to suit co-design with people with SMI. Further, special considerations such as addressing stigma and distress were found in this study. Such aspects were not included in guidelines that were geared towards the general public (Cruickshank, Coupe, and Hennessy 2013) or a broader range of vulnerable groups (e.g. asthma patients, caregivers) (Ozkaynak et al. 2021). These are important points to consider, as the consequences of not including them may provide negative experiences for vulnerable participants (Friesen et al. 2021). Further, some best-practices targeted the cognitive challenges faced by people with SMI, such as decreased abstract reasoning and lower concentration levels. Since these challenges are SMI-specific, it is important to take them into account to not overburden participants with an abundance of materials, long data collection methods or overly complicated prototypes.

Tailor co-design to the context of the research

The best-practices urge co-design researchers to adapt the materials and order of activities to the needs of the vulnerable participants. One specifically mentions the need for a critical mindset towards which methods are suitable for the relevant target group. The right choice depends on the context, the project aims and the abilities of specific participants. Each development environment comes with its own nuances that require an adaptive mindset of researchers (Orlowski et al. 2016).

Gather client values, while giving clients value

The best-practices prompt researchers to determine intrinsically valuable incentives for participants to ensure repeated attendance. The incentives mainly focused on their involvement in the project. However, the interviewees in this study expressed a positive attitude towards working as a team, creating something meaningful, knowledge attainment and an overall sense of enjoyment of the process in general. This might suggest that co-design can provide personal benefits rather than solely ones related to study outcomes. It could impact the overall mood of participants (Sharma, Conduit, and Rao Hill 2017), their sense of empowerment, and skill-building (Cargo and Mercer 2008). One best-practice specifically targets skill development for the vulnerable participants. Such skill development could aid participants in their personal lives and empower them to partake in activities that influence their care. Thus, ideally, long term benefits for SMI participants could be identified from a clinical point of view (Ramirez and Coskun 2020). That way, participants may become beneficiaries of co-design projects rather than strictly contributors. This poses a new perspective on co-design that needs further research to uncover which benefits may be obtained for participants and how these can be optimized.

Broadly skilled co-design practitioners are a necessity

Because of the diverse nature of the best-practices, it seems that co-design with people with SMI requires practitioners to be all-rounders. Examples of added responsibilities were the creation of a collaborative and informal environment, mitigating power imbalance and fulfilling roles such as researcher, confidant, project manager, mediator and participant. Researchers should be aware of the multitude of responsibilities that come along with conducting co-design in this complex environment, and gain the necessary knowledge and skills to execute the approach properly.

Personalization and communication as a key to power sharing

Throughout the results, power sharing and equal partnership were recurring themes. However, establishing a true flat power hierarchy might be hard to achieve due to the underlying power structures imposed onto the target group (Farr 2018). Thus, it is harder said than done. Yet based on the results of this study, power equity may be achieved through transparent communication, clearly defined roles and responsibilities, and a genuine sense of community and collaboration. This may also be achieved through the ability to adapt the tools and activities according to the needs of the participants, and provide them with skills training to empower them to contribute as equally as possible (Guo and Goh 2014).

However, personalizing co-design is labour intensive, requires ample resources and is a costly endeavour (Biagianti, Hidalgo-Mazzei, and Meyer 2017). Therefore, efficient ways to achieve tailored research will be important. Providing best-practices, such as the one from this study, allows for a starting point for a specialized approach as it prevents researchers from needing to re-invent the wheel. They can utilize the best-practices to compose their study and gain awareness of the challenges they might face. Through this awareness, researchers are able to anticipate or prepare suitable solutions that will avoid unnecessary hiccups.

Strengths and limitations

A strength of this study is the qualitative multi-method approach designed to fit the three pillars of evidence-based medicine. Similar studies (Ramirez and Coskun 2020; Ozkaynak et al. 2021), were solely supported by literature. Therefore, the multi-method approach adds an original methodology to the current best-practice literature for co-design. By combining the different methods, more comprehensive and exhaustive best-practices could be formed. For example, within ‘account for vulnerable participants’ skills and abilities’, the literature was able to capture concrete examples on what kind of materials and tools to use to limit the need for abstract reasoning. But the interviewees added practical information on how to present such materials. Further, some methods offered new best-practices, such as the ‘close knowledge and skills gap’ that was generated from survey data.

Another strength was the setting of the interviews. The workshops preceding the interviews ensured that the opinions of participants were based on real life experiences. This decreased the need for abstract reasoning and putting a claim on long-term memory. Therefore, the available data from literature could be substantiated and fine-tuned by real-world scenarios.

This study also faced limitations. First, if at least two reviewers carried out the coding process, increased transparency and trustworthiness of the analysis could be offered through an inter-coder reliability score (O’Connor and Joffe 2020). However, the negative impact on the trustworthiness of the results was believed to be minimal as three independent researchers iteratively generated or provided feedback on the codes.

Second, due to time constraints, the interviews with SMI participants were short. Although this is seen as a strength for conducting research with people with SMI, it obstructs the amount of information gathered from participants. More information could be obtained with separate interviews. In future studies, perhaps other ways of eliciting client values may be used. Additionally, the interview participants were involved in one single development process. This could result in bias as their opinions only reflect the experience of one co-design project.

Third, there is a potential for self-selection bias within the survey and interviews, which can lead to a misrepresentation of the target group (Lavrakas 2008). A majority of survey respondents had low to moderate experience with co-design with vulnerable groups (22/25). It is unclear whether respondents with extensive experienced were reached. Purposive sampling may have been used to ensure the involvement of researchers with more experience. With regard to the interviews, the preferences of people with SMI with more severe disabilities or fewer interest in eMH may remain unheard as often the most willing and able people are most likely to participate (Hardy, Wojdecka, Jonathan, et al. 2018).

Future research

Future research may determine if the best-practices made the overall co-design process more effective. Several other studies advocate for the execution of more research on the effectiveness of co-design approaches (Moll et al. 2020; Moore et al. 2019). To determine the opinions and lessons learned of using the best-practices by designers, researchers, and participants, methods such as focus groups or interviews seem preferred over surveys due to the ability to ask probing questions.

Another recommendation for future research is to create reporting criteria to increase the coherence and transparency of co-design studies. Overall, the studies showed inconsistent reporting with regard to lessons learned. The best-practices were formulated based on 14/21 (66.7%) of the included studies. Thus, a third of studies did not report any lessons learned. In the ones that did, it was regularly suggested ‘what’ should be done, but not ‘how’; concrete examples were often missing. The lack of practical examples, combined with the nature of this review, have contributed to the review targeting broader concepts of design projects. It would be valuable for future researchers to zoom in on effective methodological design activities for the SMI target group.

Additionally, certain information, such as setting, type of participation and sampling procedure, were frequently left out in the description of the studies (Biagianti, Hidalgo-Mazzei, and Meyer 2017). Consequently, useful insight was missing towards what worked and what did not work within specific contexts. This makes co-design somewhat of a black box as it becomes difficult for researchers to make informed decisions on which methodologies to apply in which contexts (Moore et al. 2019). Therefore, reporting criteria could increase the coherence and transparency of co-design studies. The GRIPP2 check-list can offer a blueprint for the creation of an eHealth design specific check-list (Moll et al. 2020; Staniszewska et al. 2017). That way it can become more clear which tools were used, which setting the study was conducted in and how the participants were involved in the process. Further research is needed to investigate the adaptation and suitability of such a checklist.

A last recommendation for future research is to disseminate the best-practices to both scholars and non-scholars, such as designers and care staff, who carry out co-design projects. An infographic is shown in Figure 2 as an example of making scientific findings more digestible to non-scholars. Such graphical representations can help to convey practical findings to a diverse range of disciplines, as scientific journal entries will not suffice for non-scholars (Ross-Hellauer et al. 2020).

Figure 2. Infographic best-practices for co-design involving people with SMI.

Conclusion

This qualitative multi-method study provides concrete best-practices for conducting co-design projects with the SMI population. Themes such as diversity in stakeholders and methods, flexibility, and equal collaboration and communication can aid in the success of such projects. The best-practices may help researchers, healthcare professionals and designers offer the SMI population a more specialized approach for co-design, which in turn will likely make the innovative output of eMH projects more effective and better adopted by the target group. Before embarking on a co-design project, researchers ought to equip themselves with the right skills through proper training or education to fulfil the many roles and responsibilities that are required. In addition, more research should be performed on the personal and clinical benefits of participation for people with SMI, rather than solely focussing on the benefits for research and the to-be-developed eHealth intervention.

Acknowledgements

A special thanks to the workshop participants and survey respondents who took the time and effort out of their day to provide input for this study.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are available from the corresponding author, SES, upon reasonable request.

Additional information

Funding

This work was supported by the ZonMw under [Grant 555003023].

Notes on contributors

Stephanie E. Schouten

Stephanie E. Schouten focuses on the design and development of evidence-based eHealth technology through a collaborative approach.

Hanneke Kip

Hanneke Kip conducts research on technology to improve mental healthcare for vulnerable and hard-to-reach populations such as forensic psychiatric patients.

Tessa Dekkers

Tessa Dekkers is interested in the role of personalization and customization in intervention design, patient experience, and eHealth implementation.

Jeroen Deenik

Jeroen Deenik mainly focuses on lifestyle behaviour, in relation to sedentary behaviour and physical activity, and the implementation of lifestyle interventions for the psychiatric population.

Nienke Beerlage-de Jong

Nienke Beerlage-de Jong performs multidisciplinary research into changing patients’ and healthcare professionals’ behaviours through eHealth technology, for wicked healthcare problems (e.g. antimicrobial resistance One Health).

Geke D. S. Ludden

Geke D. S. Ludden focuses on how the design of products and services influences people’s behaviour, motivation and ability to support healthy behaviour and therapy at home.

Saskia M. Kelders

Saskia Kelders is interested in positive psychology, persuasive technology and digital interventions. Her current VENI project focuses on individual engagement to personalize eHealth interventions.