Using co-creation methods for research integrity guideline development – how, what, why and when?

ABSTRACT

Existing research integrity (RI) guideline development methods are limited in including various perspectives. While co-creation methods could help to address this, there is little information available to researchers and practitioners on how, why and when to use co-creation for developing RI guidelines, nor what the outcomes of co-creation methods are. In this paper, we aim to address this gap. First, we discuss how co-creation methods can be used for RI guideline development, based on our experience of developing RI guidelines. We elaborate on steps including preparation of the aims and design; participant sensitization; organizing and facilitating workshops; and analyzing data and translating them into guidelines. Secondly, we present the resulting RI guidelines, to show what the outcome of co-creation methods are. Thirdly, we reflect on why and when researchers might want to use co-creation methods for developing RI guidelines. We discuss that stakeholder engagement and inclusion of diverse perspectives are key strengths of co-creation methods. We also reflect that co-creation methods have the potential to make guidelines implementable if followed by additional steps such as revision working groups. We conclude that co-creation methods are a valuable approach to creating new RI guidelines when used together with additional methods.

KEYWORDS:

• Co-creation
• guideline development
• research policy
• research integrity
• online research methods

Introduction

Research integrity (RI) is about conducting research according to the highest ethical and professional standards (Boehme et al. 2016). RI is important to ensure the trustworthiness and quality of research. RI is thought to be the responsibility of all research stakeholders, including researchers, research institutions, funders, and journals (Bouter 2018). To help research stakeholders address their RI responsibilities, in the past years various guidance documents on RI (e.g., All European Academies 2017) have been produced. Guidelines show research stakeholders how to conduct, organize, support or regulate research practices. Guidelines can take various forms, such as providing a checklist, a list of recommendations, or best practices.

Guidelines are developed using a combination of evidence-based methods and expert discussion (Brouwers et al. 2010; McAlister et al. 2007; Qaseem et al. 2010; Trepanier et al. 2021). There are various ways to organize expert discussion, with the most common being informal (Fretheim, Schünemann, and Oxman 2006; Murphy et al. 1998), and formal consensus approaches (James and Warren-Forward 2015). Informal consensus approaches entail the use of working groups or panel discussions (Fretheim, Schünemann, and Oxman 2006; Murphy et al. 1998). In contrast, formal consensus approaches, such as Delphi studies and the nominal group technique, use structured means of reaching consensus (James and Warren-Forward 2015).

As Fretheim and colleagues explain (2006), informal consensus methods are not ideal because they lack transparency, and are prone to undesirable group dynamics such as certain voices dominating and biasing the discussion. Despite these limitations, many research integrity guidance-providing documents are currently based on such methods (ALLEA 2017; ENERI 2019; NESH 2019). James and Warren-Forward (2015) explain that some formal consensus approaches, such as Delphi studies, are transparent and use various strategies to reduce the influence of group dynamics on decision making. However, they also have limitations. Nie and colleagues (2020) argue that the focus on reaching agreements among the group majority might lead to missing out on diverging views. Yet, such views could be particularly vital for developing guidelines that are sensitive to the specific needs of diverse users across countries and disciplinary fields.

In the past two decades, new methods for engaging participants have been developed in the field of industrial design, often referred to as “co-creation methods.” Sleeswijk Visser and colleagues (2005) describe a form of co-creation methods used to engage a wide range of expert and non-experts to express and reflect on earlier experiences. This is a generative design research approach to co-creation, which engages stakeholders not as research subjects, but as partners who are “experts of their experience”; promotes out-of-the-box thinking; and makes stakeholders’ tacit values explicit (Sanders and Stappers 2012). Additionally, this form of co-creation steps beyond “stakeholder consultation” as criticized by Arnstein (2020), in that stakeholders’ views are not merely considered as research “data” to take into account. Van Woezik and colleagues (2016) explain that co-creation methods are especially valuable for dealing with complex problems. These are problems where multiple intertwining factors and stakeholders are involved and there are no easy solutions, hence requiring flexible approaches which take into account various perspectives. Considering that guidelines often deal with complex problems involving multiple stakeholders and relevant factors, it could be that co-creation methods are helpful for guideline development. As co-creation methods are increasingly being expanded from the fields of industrial design and marketing to fields in the social sciences (Brandsen, Steen, and Verschuere 2018; Langley, Wolstenholme, and Cooke 2018), there is a rise in the availability of public co-creation tools and resources (Foster Open Science n.d.; GoNano n.d.; SISCODE n.d.). This is valuable for researchers interested in developing guidelines.

However, there is a gap in the literature regarding how researchers can use these tools and resources to specifically design RI guidelines using co-creation methods. Furthermore, experience-based information on the value of using co-creation methods for RI guideline development is also lacking. In this paper we aim to expand on the knowledge base regarding using co-creation methods for guideline development, by sharing insight with other researchers and practitioners about using online co-creation methods to develop RI guidelines, based on our own experiences with with this. First, we provide some reflections on how co-creation methods can be used to develop guidelines, using insights gained from our experience of using co-creation methods to develop guidelines targeted at research institutions and funders on how to foster RI. Secondly, we present the resulting guidelines to show what the outcome of co-creation methods are. Thirdly, we elaborate on why and when co-creation methods can be used for RI guideline development, based on our own experiences and insights as researchers, as well as by sharing the perspectives and insights from participants included in our guideline development process.

How to use co-creation methods for RI guideline development?

In our guideline development process, we aimed to develop guidelines for research institutions and funders across Europe on RI topics not currently addressed by high quality publicly available existing documents, together with lead users using co-creation methods (Labib et al. 2020; Lechner et al. 2020). Our intention was to create guidelines which addressed the responsibilities of institutions and funders at the organizational level, namely the level of rectors, deans, directors, RI officers, policy staff, and advisors. The guidelines that we intended to develop for research institutions addressed the topics: 1) RI education and training, 2) building a responsible research environment, and 3) fostering responsible supervision; while the guidelines targeted at funders focused on the topics: 4) safeguarding the independence of funded research, 5) selecting and evaluating proposals responsibly, and 6) monitoring funded projects. Prior to developing the guidelines, we had conducted several studies to explore the gaps and lacunas of current practices as institutions and funders (Gaskell et al. 2019; Labib et al. 2021a, 2021b; Mejlgaard et al. 2020; Sørensen et al. 2021). We did not pre-specify the format of the guidelines before the workshops, as we intended to address this issue during the co-creation process, and have participants decide on the most appropriate format.

There are various approaches to using co-creation methods – methods which engage users in interactive exercises involving role-playing, story-telling, card games, drawing, and other techniques promoting creativity (De Couvreur and Goossens 2011; Lee et al. 2018; Sanders and Stappers 2008). These include approaches focused on the development of user-centered products and services in the commercial sector (e.g., the development of shaving products as in Sleeswijk Visser et al. 2005); in addressing public sector questions which require novel ideas (for instance on how to become a better elementary school teacher, or how to create better healthcare services, e.g., Sanders and Stappers 2012, 88–89, 106–1112012; and to engage members of the general public in matters of research and innovation (GoNano n.d.; Robinson, Simone, and Mazzonetto 2020; e.g., SISCODE n.d.). However, none of these approaches were fully adequate for the purpose of RI guideline development, because the tools created in other contexts – for instance, exercise toolkits, sensitization materials, and card games – were not aimed at creating concrete guidance documents. To meet our needs, we needed to develop our own approach to co-creation methods, which allowed for developing and discussing RI guidelines usable by research institutions and funders across Europe, and ensured the appropriateness of all tools for our specific target group of research stakeholders.

Due to the COVID-19 pandemic, it was not physically possible to organize workshops where research stakeholders from different countries in Europe could come together in real life to work on the guidelines. Therefore, we found it most convenient to organize the workshops in an online environment, as this allowed for the inclusion of participants from countries across Europe. At the time, there was less published literature about using online methods for co-creation than is available now more than two years into the pandemic (e.g., Dexter, Atkinson, and Dearden 2013). We worked together with two co-creation experts – one a professor in Design (PJS), and the other a researcher and professional facilitator (KB) – to combine their methodology expertise with the rest of the team’s topic expertise to design the RI guideline co-creation methods .

Steps to co-creating RI guidelines

Step 1: Preparation

In our experience, using co-creation methods involves extensive preparatory work (Sanders and Stappers 2012), which should not be underestimated. Preparation involves the following steps which are elaborated further below: a) setting clear aims, b) designing the method, as well as c) selecting a suitable recruitment strategy for finding participants.

a) Aims

Researchers have the option to choose between a more exploratory aim such as reflecting on how supervision can be improved, and a more concrete outcome oriented aim such as a guideline on supervision for research institutions (Bhalla 2016; Ida 2017; Liu et al. 2018; Nambisan & Nambisan 2013). The former could be helpful in allowing participants to openly explore the general problem at hand, and jointly agree on an outcome based on this initial exploration (Nambisan and Nambisan 2013). While this approach is more participatory, it requires sufficient workshop time for exploring the problem at hand and jointly constructing a project aim. As such, it may be most suitable in cases where there is a lack of available literature on the problem at hand. Alternatively, an outcome oriented aim (particularly if not based on prior research with stakeholders), is at risk of not sufficiently taking into account stakeholders’ actual needs and preferences, but can be much more efficient and easier to work with (Nambisan and Nambisan 2013).

We decided to go with the latter option since we had decided on creating our RI guidelines – i.e., the intended outcomes of the co-creation process – based on an earlier extensive deliberation process supported by multiple empirical steps, in which we already consulted with various stakeholders (Labib et al. 2020, 2021b; Lechner et al. 2020). In addition to setting a concrete product-oriented aim (i.e., to create RI guidelines on the pre-specified topics), we also set two additional exploratory aims: investigate which guideline formats participants prefer, as well as delve into potential implementation issues of the guidelines. This allowed us to not only make steps toward producing the guidelines, but also helped us to look forward to how the guidelines might be implemented in practice.

b) Methods

One of the dominant approaches to using co-creation methods is the “Double Diamond” (Figure 1a), referring to a 4 step process to co-creation including: 1) discovering new ideas and opportunities, 2) defining a creation strategy by filtering, selecting and discarding ideas, 3) developing the ideas chosen in step 2 into a product, and 4) delivering a product, including testing and launching (Design Council 2015, 2021). As such, the “Double Diamond” approach consists of alternating divergent and convergent stages, where participants first go through a process of opening up and creating many ideas, and then closing in by filtering and selecting ideas (Design Council 2015, 2021; Stelzle, Jannack, and Noennig 2017). This ensures that a broad range of options is considered in the (product or service) development process, promoting the inclusion of novel and innovative ideas, as well as safeguarding that of all potential options, only the highest quality ideas (as defined by participants) are included in the final product or, in our case, guidelines (Stelzle, Jannack, and Noennig 2017).

Figure 1. Co-creation process of diverging and converging. Figure 1a visualizes the Double Diamond design approach shown by the Design Council. Figure 1b shows our adaptation of the Double Diamond design approach. Each of our workshops in the content generation set (i.e., workshops 1 and 2)) consisted of a divergent phase where participants developed a wide range of ideas for the guidelines, and a convergent phase where a selection of ideas was made and prioritized. Following the content generation step, the researchers analyzed the data and compiled the first version of the guidelines. Each of the workshops in the subsequent content refinement set (i.e., workshops 3 and 4) consisted of a divergent phase where participants evaluated the guidelines, and a convergent phase where participants came to agreements about refinements needed in the guidelines. Following the content refinement workshops, the researchers analyzed the data and refined and finalized the guidelines.

In our work, we adapted the Double Diamond model to design our guideline development process (Figure 1b). We decided to organize four workshops for each of the six topics we aimed to create guidelines for; therefore we conducted twenty-four workshops in total. The first two of the workshops per topic were focused on creating content for the guidelines, whereas the last two workshops were focused on refining the guideline content. Within each workshop, there was a divergent and convergent step where participants first had to develop a wide range of ideas, and then select and prioritize some ideas over others. The first two workshops focused on the first diamond, while the second two workshops focused on the second diamond. In addition to preparing and facilitating the workshops, the researchers’ role was to draft the first version of the guidelines after the content creation workshops, and revise the guidelines after the content refinement workshops.

c) Recruitment strategy

Participant selection and recruitment is similar in co-creation methods compared to other qualitative research methods (Sanders and Stappers 2012). One notable difference is that it is typical for workshops using co-creation methods to include fewer participants – two to six people – than other methods, such as focus groups (Sanders and Stappers 2012). This is because close and intensive collaborations between a small group is necessary to allow sufficient room and time to discuss diverse ideas and to come to conclusions (Sanders and Stappers 2012). Due to challenges in keeping everyone engaged and active in the online environment, our experience is that it is even more important in the online setting to include only a small number of participants per workshop.

Taking these considerations into account, we aimed to recruit four participants per workshop, so as to keep the group small but still allow for input from diverse perspectives. We identified and subsequently invited participants who would be future lead users of the guidelines (i.e., RI officers, educators, researchers, funders, policy makers, administrators, etc. from various parts of Europe) using our networks, as well as through snowballing. We aimed to include participants with diversity in country, gender, and position. To allow for some continuity across workshops, we included one to two participants in both the “content creation” and “content refinement” workshops for that RI guideline topic. This led to the inclusion of seventy-five participants in total across our twenty four workshops (i.e., six RI guideline topics, with four workshops for each topic), with twenty one participants taking part in both a “content creation” and “content refinement” workshop; for more details, please see Pizzolato et al. (2021). We had two to seven participants per workshop.

To familiarize participants with the online tools used for the workshop, we organized a fifteen-minute one-on-one call with each participant prior to the workshop to test the online tools and practice using them. This was to minimize potential problems that might arise during the workshops due to technical issues, and to therefore safeguard the quality of the workshops. Participants’ familiarity with online tools and tech-savviness can influence the quality of workshop collaborations (Fuglerud, Halbach, and Snaprud 2021; Wallgren, Babapour, and Eriksson 2021).

Step 2: Sensitization

As Sanders and Stappers (2012) explain, “creativity,” a key element in the divergent phase of workshop using co-creation methods, does not happen instantaneously. Instead, creativity is a process requiring sufficient preparation in terms of priming and activation to ensure that individuals can generate a wide range of ideas, link initially separate ideas into new combinations, and make associations between interconnected information (Sanders and Stappers 2012). Because of this, it is standard practice before a workshop using co-creation methods to “sensitize” participants, i.e., give participants some tasks to complete in preparation of the workshop (Sleeswijk Visser et al. 2005). The general advice regarding sensitization is to provide participants with a task which engages them to think about concepts related to the workshop aim, without necessarily specifying the exact aims of the workshop (Sanders and Stappers 2012; Sleeswijk Visser et al. 2005). This is considered important to not restrict participants’ thinking process, allowing for “out-of-the box” ideas to be formed before the workshop (Sleeswijk Visser et al. 2005). Researchers can expect that many participants will not complete all sensitization exercises before the workshop. Nonetheless, mere exposure to the sensitization exercise instructions can be helpful in the – conscious or unconscious – activation of ideas and priming among participants.

Especially presuming that the workshop participants, i.e., research stakeholders, might be more accustomed to more “analytic” types of workshops (which are focused more on critical thinking or convergent processes) than “creative” workshops (which are focused on opening up to different ideas), we found it important to carefully design simultaneously stimulating and serious sensitization exercises and materials that would foster creativity in our workshops. For the “content creation workshops,” we designed “inspirations” – small pieces of text or visual depictions of ideas related to the workshop topic –, which we circulated to participants one week prior to the workshop (https://osf.io/8cs42/). We asked participants to browse through the inspirations, select three which they found most striking, and explain why they found them striking (please see https://osf.io/6sqau/ for more details). This was in order to “sensitize” them before a workshop. The “inspirations” were intentionally designed to be “ambiguous,” or allow room for different interpretations, as ambiguity is considered a valuable tool in co-creation methods for nurturing richer discussions (Gaver, Beaver, and Benford 2003). For instance, for our workshops focusing on guidelines for RI education, one of our “inspirations” was a picture of a devil and angel heart (Figure 2). This could be interpreted in various – potentially conflicting – ways such as: 1) that RI education helps researchers become good in research, and prevents them from being bad, 2) that researchers have both good and bad tendencies, and 3) that RI trainings treat research in absolute terms of good and bad, rather than seeing the nuances involved in doing research. We piloted the exercise with colleagues before sending them to our participants, as suggested by Sleeswijk Visser et al. (2005) to check whether they work as expected.

Figure 2. Example inspiration sent to participants in the RI education workshops.

Step 3: Workshop exercises

Because workshops using co-creation methods are focused on creating something with a group (Galvagno et al. 2014), the organization and facilitation of such workshops requires some specific considerations to optimize development of outcomes and foster creativity. Virtual collaborative software programs such as MIRO (MIRO 2021) or MURAL (MURAL n.d.) allow for real-time interaction between participants and have many useful built in tools, such as sticky notes (Busse and Kleiber 2020; Kaur, Kaur, and Blomkamp 2021). In our workshops, we used Zoom (Zoom Video Communications, Inc 2021) to connect with our participants, and MIRO (MIRO 2021) to interact and create our guidelines collaboratively.

To allow sufficient time to meet our workshop objectives, and yet account for the limitations involved in doing online work (e.g., becoming fatigued more easily and finding it more difficult to concentrate), the duration of each of our co-creation workshops was 3–3.5 hours long. To ensure that the workshop addressed the challenge at hand, and led to a concrete outcome, our workshops were broken down into a number of smaller exercises, with each exercise building on the previous one (Sanders and Stappers 2012; Sleeswijk Visser et al. 2005). More specifically, each of our workshops contained four to five exercises, ranging between 15–45 minutes in duration. Although we found various toolboxes providing standard exercises that can be used in a co-creation workshop (Hyper Island n.d.; Skalska 2017), we found it difficult to use existing exercises as most were not suitable for our aims. Instead, we opted to look at available toolboxes as inspiration for designing our own exercises. An example of an exercise used in our workshops can be found in Figure 3.

Figure 3. Example exercise used in one of our co-creation workshops. This exercise focused on RI education for bachelor, master and PhD students, and was part of the content creation workshops for RI education and training.

Each of our workshop exercises’ objective contributed to the overall workshop, and thereby also the overall co-creation aims. For instance, the aim of the workshop from Figure 3 was to create content for guidelines on RI education, while the specific goal of the exercise shown was to discuss the content for RI education specifically targeted at the level of bachelor, master and PhD students. To stimulate creativity, but also safeguard the final outcome being produced (Stelzle, Jannack, and Noennig 2017), each exercise was composed of divergent and convergent elements. Individual elements – such as steps 1, 3 and 5 shown in Figure 3 – were helpful for generating initial ideas, while group exercises – such as steps 2, 4, and 6 – were particularly valuable for creating more and better ideas, through building on individuals’ ideas through recombination, transformation and merging (Chung 2018). While exercises were outcome oriented, we also asked participants to explain why they selected certain ideas or made certain choices, which allowed for a deeper understanding of stakeholders’ tacit needs and values (e.g., steps 2 and 4 in Figure 3).

Using MIRO allowed us to visualize all our exercises before the start of the workshops and place all our materials on one virtual board. We asked participants to use pictures of the inspirations used in the sensitization exercise to think of ideas, sticky notes to write down ideas, and dots to vote on ideas selected in convergent exercise steps. In the content creation workshops, the exercises focused on generating and selecting various ideas for the guidelines at hand, as well as is in discussing the preferred guideline format (see https://osf.io/8x3b2/ for examples). Alternatively, the content refinement workshops provided participants with the opportunity to comment on any gaps, inconsistencies, discrepancies, disagreements, or other issues in the drafted guidelines available, as well as to reflect on potential implementation concerns (see https://osf.io/kx8dj/ for examples). Prior to each set of workshops, we piloted different workshop exercises with colleagues to check that they would work as expected.

In each workshop, there was at least one facilitator and one co-facilitator present in the video call. The facilitator was responsible for moderating the session, whereas the co-facilitator helped with technical issues and any other problems during the session. In workshops in which more participants joined than initially expected (i.e more than five), there was also a second co-facilitator who assisted the co-facilitator. Having at least one co-facilitator in the session was crucial for the success of the online workshops, since co-facilitators could help participants struggling with the online tools. Because of the structured nature of the workshops, we wrote detailed facilitator instructions to ensure that the facilitators were well prepared (Appendix 1). However, since it is not possible to predict exactly how workshops will proceed (Pointon 2018), facilitators were also instructed to be flexible and adapt the workshop program when necessary – in consultation with our co-creation expert (KB) – without compromising on the workshop objectives.

Step 4: Analysis

As is common with qualitative research, co-creation methods generate a substantial amount of data (Sanders and Stappers 2012; Sleeswijk Visser et al. 2005). This consists of not only the workshop transcripts, but also the actual products of the workshop (e.g., the ideas on the MIRO board). As explained by Sanders and Stappers (2012), researchers have three options regarding how in-depth they conduct their analysis: i) “inspiration only” (i.e., immersion in data without rigorous analysis), ii) traditional “database” (i.e., line by line coding using software), and iii) “analysis on the wall” (i.e., clustering data on a wall). Although it might seem that a traditional database approach is ideal as it is most rigorous, as explained by Sleeswijk Visser (2005) this approach “does not offer an inspiring and flexible workspace for analyzing fragmentary information about context of product use … .and [does] not encourage the team to view data with empathy.” Therefore, we used an “analysis on the wall” approach, where data is clustered into groups on a real-life or virtual “wall,” rather than on a database, (Sanders and Stappers 2012). The rationale for this decision was twofold: this approach prevents (particularly the visual) data from becoming hidden in a database, and is time-efficient.

We used inductive and deductive thematic analysis to analyze the results from the “content creation” and “content refinement” workshops, respectively. The deductive themes used for the later analyses were based on the main guideline items produced earlier in the guideline development process. After analyzing the data in a small group (of 2–5 researchers per workshop topic), we visualized the results in analysis posters (example shown in Figure 4). This was to keep the analysis results close to the situations discussed by participants during the workshops. When developing the guidelines, we looked at the results of the analysis posters to write and revise each guideline items. We formulated the guidelines as a list of recommendations, each based on the results of the analysis (Pizzolato et al. 2021).

Figure 4. Example overview of the analysis process used in the SOPs4RI co-creation methods. Step 1 shows an overview of how the data were clustered by the group in an “analysis on the wall” approach. Step 2 shows the resulting poster, based on which a first draft of a guideline on RI education for bachelor, master and PhD students was developed.

What do co-creation methods result in when applied to RI guideline development?

Through the use of co-creation methods, we developed RI guidelines for research institutions on 1) RI education and training, 2) responsible supervision and leadership, 3) research environment; as well as RI guidelines for research funders on 4) selecting and evaluating proposals, 5) preventing unjustifiable interferences, and 6) monitoring funded projects. The guidelines take into account the diversity of stakeholders involved in the research process (such as the needs of researchers across ranks, as well as differences among institutions and funders). They provide recommendations that are practical, and yet can be tailored to different research contexts. Furthermore, they are built on the consensus of the co-creators involved in developing them. Table 1 provides a breakdown of the guidelines created per topic. In this section, we provide an overview of the content of the guidelines for each topic to help readers understand what the co-creation methods we applied in RI guideline development result in. An example of the second version of the guidelines – which was the direct output of the workshops – can be found in Appendix 3, while the most updated versions are available on the Open Science Framework (e.g., https://osf.io/z7m3v/). We delve deeper into the specific guideline content for each topic in separate manuscripts (e.g., Pizzolato et al. 2022).

Table 1. Breakdown of RI guidelines developed per topic using co-creation methods.

Topic	# of guidelines	Guideline content	Target of guidelines
RI education and training	4	RI training of bachelor, master and PhD students RI training of post-doctorate and senior researchers RI training of other RI stakeholders Continuous RI education	Research institutions
Responsible supervision and leadership	3	Supervision PhD guidelines Leadership	Research institutions
Research environment	4	Culture building Adequate education and skill straining Managing competition and publication pressure Diversity and inclusion	Research institutions
Selecting and evaluating proposals	3^a	RI plan Methodological requirements Diversity and inclusion	Research funders
Preventing unjustifiable interferences	4^a	What counts as an unjustifiable interference? Interference by funders Interference by commercial influences Interference by political/other external influences	Research funders
Monitoring of funded projects	3a	Execution of the research grant RI requirements Financial monitoring	Research funders

aThis is the number of guidelines created as a result of the co-creation workshops for this topic. However, after the co-creation workshops, at a guideline revision step, it was decided that these guidelines would all be merged into one overall guideline to cover the entire topic.

The guidelines for research institutions

RI education and training

In line with recommendations in the literature (e.g., Fanelli 2019), the co-creation workshop participants considered it important that RI education and training is provided to all research stakeholders, including students, junior and senior researchers, as well as others involved in the research endeavor (e.g., ombduspersons, research managers, RI officers, policy staff). During the co-creation workshops, the participants discussed specific recommendations that would be appropriate for the RI education of different stakeholders, based on their own experiences with RI education. For instance, they suggested full RI courses for PhD students, small workshops for more senior researchers, and peer-to-peer learning events for other RI stakeholders. Furthermore, they emphasized the importance of approaches to learning about RI that fall outside the scope of “formal training” in the classroom. This allowed us to create guidelines on RI education and training that capture various approaches to learning about RI, and are specific to the needs of various stakeholders. An infographic of the finalized guidelines can be found on OSF: https://osf.io/6zbqc.

Responsible supervision and leadership

The co-creation workshop participants highlighted research institutions’ responsibilities regarding communicating the responsibilities of and requirements for good supervisors and leaders, but also emphasized the need for research institutions to provide adequate support and training to supervisors and leaders to achieve these. In this way, the resulting guidelines went beyond outlining what responsible supervision and leadership mean, but actually focused on how institutions can empower supervisors, PhD students and research leaders (e.g., principle investigators) to ensure responsible supervision practices. This included recommendations on providing structures for peer-to-peer support, paying sufficient attention to researchers’ well-being, and providing bodies to consult in cases of conflict. The finalized guidelines have been visualized on this infographic which can be found on OSF: https://osf.io/8n5ud.

Research environment

The guidelines on building a responsible research environment provide practical steps that institutions can take to addressing this key, but less tangible, issue for RI (as seen by participants in another study we conducted, Labib et al. 2021b). One of the guidelines focused on the general question of how to create community building for a responsible research environment, and this guideline addressed various areas that institutions need to address such as conducting responsible research assessments; creating an open, safe, diverse and inclusive research culture; providing researchers with appropriate support structures and training; and dealing with competition and publication pressure. While the guideline was broad, participants provided concrete in-practice examples that could serve as inspiration for institutions on implementing these points. One such example was to publish institutional staff survey results, including negative comments, so as to create more transparency in the institution. The other three guidelines under the topic of responsible research environment, provided more detailed recommendations on specific aspects that were already highlighted under the general guideline on community building. The guideline focused on diversity and inclusion, for instance, emphasized the importance of taking an intersectional approach to diversity that accounts for different types of diversity (e.g., race, gender, class) and how they intersect, and provided concrete suggestions on how to do this (for instance by including diverse researchers in a bottom up way when developing diversity policies in the institution). A more detailed overview of the guidelines can be found on the OSF: https://osf.io/jcpgq.

Guidelines for research funders

The guidelines for research funders have been visualized on this infographic, available on OSF: https://osf.io/q2wra.

Selecting and evaluating proposals

When creating the guidelines for this topic, participants emphasized the difficulty to standardize detailed recommendations given the large diversity in funders in terms of size, funding streams (i.e., governmental or private), and culture. However, they agreed on some basics that apply across funders, such as requiring proposals to include a plan on how to safeguard RI; paying sufficient attention to the methodology section of submitted proposals; and removing biases from the selection and evaluation process (e.g., by ensuring that the language used to communicate to grant applicants is inclusive).

Preventing unjustifiable interferences

Co-creation workshop participants highlighted that not all interferences in the research process are unjustified and came to agreement about which are and are not justified. For instance, they concluded that the funder can influence the research agenda, but that interference in the publication of results is unjustified. In addition to recommending having a clear definition of “unjustifiable interferences” for each research funder, the guidelines on this topic also provide recommendations on how to prevent such interference and deal with it. These recommendations address various phases of the research and funding process, from the moment of selecting and evaluating proposals (in which preventing conflicts of interests is considered crucial), to providing guidelines about projects co-funded by commercial parties, and addressing how to keep researchers independent in the publication step of the research process.

Monitoring of funded projects

Co-creation workshop participants emphasized that while having funders monitor funded projects is crucial as a way to increase the trustworthiness of research, this monitoring process should be collaborative and cooperative (rather than employ a policing approach). While participants provided various recommendations on how to monitor projects in a manner that is fruitful and minimally bureaucratic (e.g., providing a checklist of points that researchers can report on to the funder), they also emphasized the need to create a quality assurance system for the monitoring process. Therefore the monitoring guidelines provide both “what to monitor” recommendations for funders, as well as recommendations on “how to” ensure that this is done in a cooperative and productive manner.

Why and when to use co-creation methods for RI guideline development?

Since the use of co-creation methods for RI guideline development is a novel approach, we frequently interrogated the advantages and disadvantages of co-creation methods and their suitability for RI guideline creation. In this section, we reflect on why and when it is suitable to use co-creation methods for developing guidelines. For this, we are combining our own experiences with those of our participants. To learn about our participants’ reflections, we conducted a set of informal interviews with one participant from each of our workshops, with the aim to explore how participants evaluated co-creation methods for developing RI guidelines. More details about the interviews, including the interview guide, characteristics of interviewees (including their interviewee numbers, demarcated as “IN”), and interview procedures can be found in Appendix 2.

Why use co-creation methods for developing RI guidelines?

Stakeholder engagement

Our interviewees identified close stakeholder engagement as a key benefit to co-creation methods. They indicated that the stimulating exercises of our workshops kept them closely engaged and willing to contribute to the RI guidelines. Interviewees expressed that our workshops were: “interactive and colorful, and not boring … ” (IN 3), “quite remarkable” (IN 6), “insightful and interesting” and “innovative” (IN 7), “fun” and “rewarding” (IN 8). One of our interviewees mentioned that when “you have [people] on board [and engaged], they will implement [the guidelines] because they like it and because they contributed to it” (interviewee number, IN 18). Another participant remarked that engaging stakeholders actively is especially necessary to get “buy-in,” i.e., to ensure that all stakeholders support the guidelines (IN, 14). Others mirrored these views by stating that stakeholder engagement is especially valuable to increase the likelihood that the guidelines will be actively used by institutions after the development process is complete. There were also some participants who appeared less comfortable with the workshops, more specifically to the online setting of the workshops, with a few remarking that MIRO was difficult to use. Others expressed that while their initial reaction to the workshop invitation was hesitant due to the online setting, they were then surprised to experience the online workshops as interesting and productive.

These results are promising, especially because we were initially concerned – also due to challenges outlined in Deserti, Rizzo, and Smallman (2020) – that it might be difficult to stimulate research stakeholders such as policy makers to embrace the creative aspects of co-creation methods (e.g., work with images to create a broad range of new ideas and alternative perspectives). Features of co-creation methods that make it particularly suitable for engaging stakeholders involve a) giving stakeholders the opportunity to create outputs based on their own needs, which they can then use themselves, and b) using interactive and playful exercises, by design (Sanders and Stappers 2012). This level of stakeholder engagement in our co-creation process is not fully surprising given that co-creation methods have also been previously used successfully with other groups of analytical, or “critical,” participants, including CEOs, healthcare workers, and policy makers (e.g., Agrawal, Kaushik, and Rahman, 2015; Morell and Senabre Hidalgo, 2020; Kimbell & Bailey, 2017; Sanders and Stappers 2012; Waseem, Biggemann, and Garry, 2018). Even stakeholders who might initially be hesitant about participating in creative workshops can be stimulated to be creative and engage in “serious play,” using and offering the appropriate tools to evoke creativity (e.g., “inspirations” and stimulating workshop exercises) and the reassurance that the “play” will lead to productive and valuable outputs for the participants (Hinthorne & Schneider, 2012; Sanders and Stappers 2012).

We encountered a few difficulties with ensuring that all participants were comfortable online. However, we experienced the use of a whiteboard interactive software like MIRO to facilitate engaging workshop sessions virtually positively, since it allowed participants from different parts of the world – time differences allowing – to collaborate together on one platform in real time, using various creative tools (Busse and Kleiber 2020; Kaur, Kaur, and Blomkamp 2021). Since MIRO requires some familiarization before it can be used optimally, we found it helpful to organize one-on-one calls with each participant ahead of time to explain the tool and help them practice with it. It might be, however, that participation in more than one workshop is necessary for all participants to feel comfortable with tools like MIRO, and that simpler tools are needed when engaging one-time participants who are not very tech-savvy (e.g., Google n.d.) . However, this has to be weighed against the inconveniences that simpler tools might present, such as limitations for facilitators and not providing all technical options needed for a workshop.

Inclusion of diverse perspectives

Co-creation methods were also considered valuable by our interviewees in stimulating the inclusion of a broad range of perspectives in the RI guidelines. As put by one of our interviewees, discussing diverse views is important “to prevent skipping some steps and starting with a one-sided perspective” (IN 1). Another interviewee also appreciated that “co-creation is not only meant to see what is mainstream but what are possibly dissenting views,” since when it comes to RI guidelines, “minority views are as important as majority views,” as they are likely “more problematic, more novel, and innovative” (IN 6). Our workshop participants appreciated that co-creation methods actively encourage participants to share their diverse and unique perspectives in various ways, including starting workshops by first encouraging participants to be open to various ideas – and even be stimulated to “dream” about an ideal guideline (IN 8) – and only afterward select ideas which are more practical and feasible; giving all participants “time to reflect on an issue” individually and then stimulating them to share these reflections in discussion with the group (IN 22); the framing of questions in ways that lead to different interpretations; combining visual and textual elements during the workshops (as people “respond differently when [they] have cartoons or words” (IN 22)); and “focusing on real-life experiences” of the participants to “come up with bigger perspectives” (IN 9). However, our interviewees also highlighted that the diversity of perspectives that can be included in a workshop is limited to the characteristics of the workshop participants. We received some criticism that despite there being sufficient diversity among our participants in terms of gender, stakeholder type, and country in Europe, we did not include enough participants from junior ranks, countries outside Europe, and diverse cultural backgrounds in our co-creation workshops.

Indeed, compared to other guideline development methods we have previously used, such as Delphi studies, we also found co-creation methods to more actively focus on evoking a broad range of ideas. These results confirm that co-creation methods are suitable for addressing complex problems, since they actively promote the inclusion of various perspectives (van Woezik et al. 2016). However, the results also suggest that to include a broad range of perspectives, both diversity in the demographic characteristics of participants, as well as the use of techniques that evoke various perspectives among a specific group (e.g., using phrasings that lead to different interpretations), are needed. However, including diversity in all dimensions of the demographic characteristics of participants will require a larger number of participants and, hence, workshops. The decision as to whether to hold workshops online or in-person will also have an influence on the diversity of the participants included in the workshops.

When it comes to guideline development, including diverse perspectives is helpful to ensure that the guidelines are sensitive to the needs of all relevant stakeholders. For RI guidelines, this includes research stakeholders across disciplines, countries, and institutions. Of course, at the end of the guideline development process, many ideas will need to be abandoned so that only the highest quality ideas are used. Yet, starting out with an open approach and allowing diverse users to define priorities allows guidelines to address the most important needs of all stakeholders.

When use co-creation methods for developing RI guidelines?

Our advice to other researchers and practitioners is to use co-creation methods early in the guideline development process. Early use of co-creation methods allows for – as described by one participant (IN 22) – guideline “details [and nuances] that would be missed in a different setting.” Additionally, early use of co-creation methods in the RI guideline development process – as we did – allows for a timely understanding of the level of agreement about the RI guideline content among stakeholders, as well as for consensus building. Many of our participants, for instance, said that they were satisfied to see that after discussions in the workshops, many points of agreement emerged and remaining differences in opinion – although well represented in the final outputs – were small and mainly related to the specific context in which the participants worked in (e.g., country, institution type).

Co-creation methods can be followed up by additional steps in the guideline development process to ensure that guidelines are well refined. This was highlighted by our interviewees, one of whom mentioned that the guidelines are still “very general,” whereas they “should be specific and … offer a way of action” (IN 17), and another who suggested that further steps are needed to write “things [i.e., the guideline items] in a clear and academic language” (IN 6).

Although the interviews were held before the interviewees had the chance to see the guidelines resulting from the workshops, these results already indicate that the workshops led to an abundance of ideas for the guidelines addressing important aspects of the workshop topic, which needed fine tuning later. This is in line with what we observed, since at the end of our guideline development process, we had a comprehensive set of guidelines for each of our 6 RI topics, which included agreements formed by participants across workshops. Furthermore, the workshops helped to elucidate differences in how institutions and funders from various countries approach RI (e.g., regarding laws, definitions, existing infrastructures and policies). In line with the interviewees’ concerns, while the resulting guidelines were comprehensive, they were less “actionable” (i.e., ready to be used). This is because our workshops produced a lot of ideas, but 1) the organization of these ideas (including merging, regrouping, and simplifying ideas) was not finished after the workshops, 2) the formulation of the ideas into concrete recommendations needed further fine-tuning (e.g., some items needed to be made general enough to be implementable across different institutions and funders in Europe). This is not surprising; given that workshop discussions using co-creation methods are focused on broader ideas even during convergent steps, there is less room for fine-tuning the details of the guideline formulations during workshops.

This suggests that guideline developers should use co-creation methods as the first major phase of the RI guideline development process – akin to the “fuzzy front end” of design described by Sanders and Stappers (2012), – to bring diverse and out-of-the-box perspectives to the floor, and then follow up with other methods to finalize the RI guidelines (such as expert working groups, surveys, and consensus methods). In line with this view, we used additional steps after the co-creation methods to finalize our guidelines, including a small expert working group to refine the guidelines, input from additional experience and content experts, as well as piloting of the guidelines. We expect that using such a multi-stage guideline development process helped to engage diverse stakeholders closely throughout the guideline development process to incorporate diverse perspectives and safeguard the quality of our guidelines and promote their implementation. Furthermore, such an approach allows the joint development of not only the guideline content, but also the format, as well as an early exploration into potential implementation challenges and opportunities. However, we also acknowledge that this process costs substantial time and human resources, which might not always be available, particularly in smaller RI guideline development projects.

Conclusions

In this paper, we reflect on what we learned about co-creation methods when developing RI guidelines to share insights with other researchers and practitioners on how, why and when they can use co-creation methods for developing RI guidelines, as well as sharing what the outputs of co-creation methods can be. Regarding the “how” question, we discuss that careful and extensive planning is required to prepare co-creation methods. This includes setting a clear and suitable research aim, designing the guideline development process using alternating diverging and converging steps, and recruiting diverse participants into a number of small and intense workshops. Additionally, we discuss the importance of sensitizing participants prior to the workshop to prepare them for creativity, as well as organizing and facilitating engaging and structured workshop exercises to stimulate productivity. Finally, we discuss that researchers have a variety of options regarding how to analyze their data in order to develop the guidelines, depending on the time available and purpose of the analysis.

Regarding the question of “what” co-creation methods can result in, we present an overview of the resulting guidelines for the six topics we addressed using co-creation methods: 1) RI education and training, 2) responsible supervision and leadership, 3) research environment, 4) selecting and evaluating funding proposals, 5) preventing unjustifiable interferences in the research process, and 6) monitoring of funded projects. The results show that co-creation methods help to develop RI guidelines that are sensitive to the needs of diverse RI stakeholders.

As to “why” use co-creation for guideline development, in our view, co-creation methods are unique and valuable to the guideline development process. They are particularly helpful in terms of engaging stakeholders closely throughout the guideline development process, as well as evoking a broad range of ideas and including diverse perspectives in the guidelines. This allows for the development of guidelines that meet diverse stakeholders’ actual needs.

To address the “when” question, our experiences indicate that co-creation methods are most helpful at the early phase of the guideline development process. We would recommend guideline developers to use a multi-stage approach to co-creating guidelines; co-creation methods likely need to be followed up by additional guideline development methods (e.g., expert working groups and consensus methods) to further organize the ideas generated by co-creation, and make guidelines precise, actionable and implementable.

Authors’ contributions

KL drafted and revised the co-creation study protocol and design, conducted the follow-up evaluation interviews, analyzed the data, and drafted and revised the manuscript. DP, IL, and NE revised the co-creation study protocol and design, analyzed the data and revised the manuscript. PJS, KB, GW, LB, and KD revised the co-creations study protocol and design, and revised the manuscript. JT revised the co-creation study protocol and design, conducted the interviews, analyzed the data, revised the manuscript, and supervised the work.

The workshops were designed by KL, in collaboration with all other authors. The workshops were overseen by KB. Facilitators included NE, JT, and KL, and co-facilitators included DP, IL, and NS and BT (please see acknowledgements).

Acknowledgments

We would like to acknowledge all the interviewees who shared their time and valuable insights with us. We would also like to thank all members of the SOPs4RI consortium who contributed to the co-creation study including Nik Claesen, Borana Taraj, and Nikolaos Skoulikaris.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This study was funded by the Horizon 2020 Framework Programme of the European Union for Research and Innovation under Grant Agreement Νo 824481. The funder was not involved in the design, conduct, analysis or interpretation of the stud; H2020 Science with and for Society.