Playing the new devil’s advocate role in facilitated modelling processes to address group homogeneity

Abstract

To address complex issues, facilitated modelling aims to represent and accommodate plural worldviews from many stakeholders and experts. In these contexts, group homogeneity can become problematic when participants’ plurality of perspectives and information is missing and people attending facilitated sessions have similar problem perceptions and interests. This is a challenge because it can lead to narrow discussion, groupthink and undermine output quality. Despite not being uncommon, effective approaches to deal with homogeneity are hardly reported. This paper presents a new role—the New Devil’s Advocate—in which some facilitators leave their neutrality-oriented stance and act as the missing stakeholders. The paper illustrates a first application to a group model building process aimed at supporting the development of energy efficiency policies in the UK. To evaluate the results, workshop transcripts were coded, participants’ and facilitators’ feedback collected, and the modelling output assessed with respect to the New Devil’s Advocate interventions during the workshop. Although the role performance appears to increase facilitators’ workload and be influenced by role performers’ personality and background, the analysis shows positive results as a promising practice to address homogeneity. Additionally, it offers a practical experience of how facilitation teams may temporarily abandon neutrality and intervene on content.

Keywords:

• Community OR
• group decision making
• problem structuring
• soft OR
• system dynamics
• decision support systems

1. Introduction

Interdisciplinary and participatory approaches have a long history (Sims et al., 1981) and are increasingly used (Voinov et al., 2016; Voinov & Bousquet, 2010) to deal with complex ‘messy problems’ where plural values are advocated by different stakeholder groups, affecting and affected by the system under analysis (Vennix, 1996; Videira et al., 2009). Among those participatory group approaches¹, facilitated modelling methods “combine group facilitation with participative modelling” involving “an interactive process by which models are jointly developed with a group, face-to-face, in a workshop environment” (Andersen et al., 2007; Franco & Montibeller, 2010; Tavella & Franco, 2015, p. 452). As these group characteristics introduce and emphasise a behavioural component during the modelling process, facilitated modelling is an important stream of study in behavioural operational research (Brocklesby, 2016; Franco et al., 2021; Kunc et al., 2016). The literature reports and elaborates on several facilitated modelling methods (Franco & Montibeller, 2010), for example: facilitated problem structuring, for understanding complex problems and identifying strategies (Eden & Ackermann, 2006; Mingers & Rosenhead, 2004); facilitated decision analysis, for selecting preferences in contexts with multiple objectives and uncertainty (Belton & Stewart, 2002); facilitated discrete event simulation, for developing discrete event models (Kotiadis & Tako, 2018; Robinson et al., 2014; Tako & Kotiadis, 2015); and facilitated system dynamics, also known as Group Model Building (GMB — Vennix, 1996) or Participatory System Dynamics Modelling (PSDM - Stave, 2010), a set of methods for engaging participants in the construction of a system dynamics (SD) model (Andersen et al., 2007; Lane, 1992; Peck & Vannix, 1998; Vennix et al., 1990). Overall, facilitated modelling applications aim at producing useful models with various degrees of stakeholder engagement in the sharing and co-production of knowledge, supporting the achievement of broad commitments to action and policy implementation (de Gooyert et al., 2017; Franco & Montibeller, 2010; Franco & Rouwette, 2011; Rouwette, 2011; Scott et al., 2016b). These methods are typically used to build rigorous models richly grounded in diverse experts’ and stakeholders’ experiences. This allows for incorporating various sources of information into the model, challenging assumptions, sharing knowledge and allowing participants to have a voice in the process and further develop their mental models (Eden, 1992; Franco & Montibeller, 2010; Vennix, 1996). Ultimately, plural viewpoints are expected to improve the model’s quality, foster stakeholders’ commitment regarding project results, and consequently to increase chances of success during the policy implementation phase (Giordano et al., 2017; Papamichail et al., 2007; Rouwette, 2011; Scott et al., 2016b). Such facilitated modelling projects usually require facilitators to accommodate participants’ plural worldviews over relatively long time frames of several months or even a few years (Vennix, 1996; Videira et al., 2012).

Methodological challenges related to time requirements and stakeholders engagement may arise because it might be impossible to access relevant stakeholders, participants may be time-constrained, or results might be needed quickly (Butler & Adamowski, 2015; Pyrko et al., 2019; Voinov & Bousquet, 2010). Thus, facilitated modelling might occur with participants from the same stakeholder group, who share similar mental models about the situation or systems of interest (homogeneity), and/or in very short periods. The combination of the two is called pressure cooker conditions (Gerrits & Vaandrager, 2018). They may negatively affect the intervention, and modelling efforts then need to be adapted to the context in which they occur (Bots & van Daalen, 2008; Gerrits & Vaandrager, 2018). Examples of successful GMB projects conducted in time-constrained settings already exist. For instance, Videira et al. (2012) and de Gooyert et al. (2016) describe situations in which they constructed qualitative models in a half-day workshop by adapting the workshop scripts, i.e., the recipe-like ‘chunks’ that facilitated modelling workshop sessions tend to be divided into (Ackermann et al., 2011; Andersen & Richardson, 1997). These scripts make sessions flexible and adaptable to various contexts and time frames.

However, the homogeneity issue poses an even greater methodological challenge. In fact, homogeneity, in “the sense that [stakeholders] have similar interests and problem perceptions” (Barreteau et al., 2017; Bots & van Daalen, 2008, p. 398), can undermine output quality when plural perspectives are required to broaden the scope of discussions (Beers et al., 2006; Gerrits & Vaandrager, 2018; Sedlacko et al., 2014; Vennix, 1996). Homogeneity may also generate groupthink effects and premature consensus, further narrowing down the group work (Eden, 1992; Esser & Lindoerfer, 1989; Flippen, 1999; Goodwin & Wright, 2004; Janis, 1972). Facilitated modelling methods have shown to address such risks by supporting the understanding of unstructured problems and countering decision-making pitfalls (DeSanctis & Gallupe, 1987; Eden, 1992; Miranda, 1994). Nonetheless, facilitated modelling methods do not guarantee to avoid the abovementioned problems in homogeneity contexts. In fact, divergent and creativity exercises (e.g., nominal group techniques, brainstorming) do not assure that the perspectives of missing stakeholders will be raised, acknowledged and considered by the participants. Even if participants are encouraged to think differently and about the positions of missing stakeholders, they may be unwilling to raise them or simply unaware of their stances.

Very few specific strategies have been tried to overcome drawbacks arising from group mental model homogeneity². The most widespread example is the Devil’s Advocate role play in management (Herbert & Estes, 1977; Janis, 1972; Schwenk, 1984), in which a group member is appointed to raise concerns and counter-arguments against premature group consensus. Vennix (1996, p.276) further elaborates on this technique combining it with dialectic inquiry (Mason, 1969)³ to develop the Devil’s Advocate script, adapted for GMB and, more generally, facilitated modelling sessions. It suggests dividing participants into two groups, one questioning the assumptions and plans made up to a certain point of the model development process, while the other is in charge of defending them, iterating it until an agreement between the two groups is reached.⁴ Despite its advantages, this strategy still retains a major limitation similar to the techniques mentioned above to address groupthink: it does not introduce new sources of information into the workshop room or avoid the risk of repeating shared knowledge; it uses the knowledge that is already there. Therefore, finding an improved solution to the homogeneity issue in facilitated modelling is a substantial methodological and practical challenge.

Given that, and in the spirit of improving practice (Ackermann et al., 2011; Andersen et al., 2007; Hovmand et al., 2011), this article introduces and tests a new facilitation role and associated guidelines, the New Devil’s Advocate (NDA), consisting of a developed form of the Devil’s Advocate script proposed by Vennix (1996). In the suggested new role, a sub-set of facilitators leave their neutral stance and try to act as the missing stakeholders in an attempt to bring their mental models into the room. This work aims to explore the NDA role’s nature, effectiveness and impact by employing it in a GMB case study. The article proceeds as follows: first, we outline the study’s relevance in the context of previous examples of addressing group homogeneity described in the literature; second, we describe the NDA role; third, we analyse and discuss a case study in which we applied the approach and, finally, present conclusions and suggestions for further research.

2. Strategies for managing group homogeneity in workshops

In facilitated modellers’ toolbox, there is a set of divergent techniques (and related scripts to perform them) to promote creative thinking and counter groupthink (Andersen & Richardson, 1997; Franco & Montibeller, 2010) which can be considered potential tools to address homogeneity. Examples are brainstorming, nominal group techniques, inquiry techniques and role-plays. Brainstorming aims to generate as many ideas as possible over a specific issue and has the effect of increasing the range of options considered by the group (Osborn, 1957). The nominal group technique ensures every member voices their ideas, which can then also be ranked (Delbecq & Van de Ven, 1971). However, these techniques increase creativity and ideas (McCrae, 1987) without addressing group homogeneity per se. In inquiry techniques, to enhance critical and heterogeneous thinking in groups, facilitators challenge participants by repeatedly asking questions and clarifications about their statements and assumptions to bring out additional elements to the debate (e.g., Socratic questioning) (Snyder & Snyder, 2008; Walker, 2004). The traditional Devil’s Advocate and its variations (such as dialectical inquiry (Mason, 1969) or strategic assumptions analysis (Mason & Mitroff, 1981; Mitroff & Emshoff, 1979) in which the devil’s advocates also have to propose counterplans) are inquiry techniques (Schwenk, 1982). However, there is no guarantee that a missing stakeholder’s point of view would be raised by participants (e.g., they may not even be aware of the missing stakeholders’ stances). Role plays address this shortcoming by stimulating participants to adopt another person’s point of view to experiment with perspectives that might have been overlooked (Ertmer et al., 2010). For instance, Bryson (2004) invites session participants to impersonate different stakeholders on the spot to explore how they would behave. It is similar to what is done in the area of cybersecurity (Nagarajan et al., 2012; Namin et al., 2016), where groups responsible for organisations’ cybersecurity are divided into two teams that try to take each other down. This is a way to test defence mechanisms’ effectiveness and explore hackers’ thoughts. Although these approaches have benefits, participants need reliable knowledge of their role; otherwise, the technique’s usefulness can decline drastically. Consequently, all these techniques are not evaluated as the best option to deal with homogeneity in facilitated modelling sessions because they rely only on the group’s internal resources (i.e., participants’ knowledge) and do not guarantee reliable representation of the missing stakeholders. Workshop participants with very similar mental models may thus have limited success if the goal is to understand how the system under study actually works.

Gerrits and Vaandrager (2018) tried to go beyond this methodological gap and reported on a pressure-cooker situation in which they specifically adapted a GMB process to increase heterogeneity among a participant group, yet without success. Their idea was to send participants into the field, where they would be confronted with many diverse points of view. The intention was to stimulate participants to collect information on the missing points of view to have at disposal some reliable knowledge of the missing stakeholders’ mental models during the modelling workshops. Facilitators asked the group to interview people outside the organisation, build SD models from the interviews using some training they had received and send the results to the research team. The modellers’ plan was to summarise individual data and construct a preliminary model to be discussed and further developed in the second session. Unfortunately, the majority of the causal maps sent by participants to facilitators were of low quality. The authors reported: “few officials had even taken the effort […] those who had made an attempt had usually done it incorrectly […] most of the civil servants had only interviewed direct colleagues at approximately the same hierarchical level. […] they only recorded and emphasised those views that confirmed their own […]” (Gerrits & Vaandrager, 2018, p. 7). Gerrits and Vaandrager evaluated their process as ineffective because they could not achieve the client’s goal, no change in the organisation was observed, and the organisation rejected the method deciding not to continue the collaboration after the two sessions. The authors suggest that the causes of failure were attributable to the high workload posed to the participants in terms of time demanded and cognitive effort, which hindered them to complete satisfactorily the necessary tasks to increase heterogeneity.

3. New Devil’s Advocate: Rationale, role development, and guidelines

3.1. Rationale

Therefore, a strategy to overcome possible mental-model homogeneity during facilitated modelling projects is still needed. The techniques reported in Section 2 have the limitation that they rely on the knowledge available to the participants and do not guarantee reliable representation of missing stakeholders’ perspectives, whereas the Gerrits and Vaandrager (2018) attempt to expand participants’ knowledge about missing stakeholders by assigning participants information collection tasks in preparation for the workshop was unsuccessful. In this respect, a still unexplored direction consists of giving the facilitator the responsibility for collecting reliable information and representing the missing stakeholders. This procedure is similar to Gerrits and Vaandrager (2018) but shifts the burden and workload of gathering information on the missing stakeholders from the participants to the facilitation team. In fact, it is argued that facilitators may more easily overcome time and cognitive obstacles because this process can be part of the workshop preparation and they may be more experienced with collecting and organising stakeholder information. Such an approach assumes that informed and prepared facilitators can better understand the perspectives of the missing stakeholders than (unprepared) workshop participants with only their knowledge at their disposal. However, it also somehow implies that facilitators leave their neutral stance during the modelling sessions. This represents a novelty, entailing some risks, since facilitators are usually supposed to be neutral and non-judgmental (Schuman, 1996; 2005), a notion that is commonly adopted in facilitated modelling (Ackermann, 1996; Elsawah et al., 2023; Franco & Montibeller, 2010; Papamichail et al., 2007; Phillips & Phillips, 1993) and GMB methodologies (Scott et al., 2016a; Vennix, 1996). Specifically, if facilitators leave their neutral stance, they could alter group processes that are vital for the facilitated modelling intervention to be successful (Phillips & Phillips, 1993) since groups are rather sensitive to different facilitation styles (Bell & Morse, 2013; Papamichail et al., 2007). In particular, issues about perceptions of fairness of the process (Korsgaard et al., 1995) may arise as participants feel the facilitator is not being impartial. This could undermine the facilitator-group relationship (Harper et al., 2021; Phillips & Phillips, 1993), leading participants to distrust the method or the facilitator as a conflict mediator, demotivating participants to share knowledge and contribute to new content (Beers et al., 2006), and weakening ownership and commitment towards facilitating modelling outputs (Ackermann, 1996; Harper et al., 2021; Vennix, 1996). The tension resulting from the trade-offs of assigning such a role to facilitators has been under scrutiny for a while (Colin & Radford, 1990; Huxham & Cropper, 1994; Sims et al., 1981). Although orthodox facilitated modelling wisdom calls for facilitators’ neutrality, some researchers advocate (Huxham & Cropper, 1994) or demonstrate (Papamichail et al., 2007; Tavella & Papadopoulos, 2015a, 2015b) that the neutrality ‘boundary’ could be less rigid. Notably, the facilitator’s deliberate intervention and sharing of information on the problem could be advantageous when the group discussion on the content is inaccurate and narrow (Huxham & Cropper, 1994), as it could be in cases of homogeneity, outweighing the risks posed by the non-neutral intervention. Some researchers state that it may be challenging to maintain neutrality when some participants go against established knowledge and compromise the outcomes of the facilitated modelling effort (Voinov & Bousquet, 2010) or when the group discussion and content developed work against the facilitator’s understanding and point of view (Voinov et al., 2016; Voinov & Bousquet, 2010). As Brocklesby (2016) reflects, the modellers may have, to a certain extent, the ethical responsibility of the model’s outcomes. Thus, when it becomes impossible for a facilitator to maintain neutrality, a possible solution is to be substituted by another external facilitator (Voinov & Bousquet, 2010), which guarantees process neutrality, while the original facilitator gets involved in the debate. Alternatively, some authors report that it is unrealistic to expect facilitators not to have beliefs on the content under discussion, especially if this is a pressing and relevant issue (e.g., environmental protection - Voinov et al., 2016). In these cases, it might be more effective for the facilitator to inform the group of their ideas (Glynn, 2017). On the one hand, this could make the facilitator and participants aware that there may be conscious or unconscious biases in the facilitation process and help take counter-measures (Glynn, 2017). On the other hand, it could help to build trust, as people could be suspicious of those who declare to have no stances or ideas on a sensitive issue (e.g., Kahan, 2012).

3.2. Role development

To fill the presented methodological gap, we developed the NDA role, an adaptation of the traditional Devil’s Advocate script (Vennix, 1996), integrated with ideas on how other methods increase heterogeneous thinking among groups. The approach proposed here stands out from previous versions of the Devil’s Advocate in that the NDA player is not a participant but a facilitator from the modelling team, a possibility mentioned (but not tested) by Herbert and Estes (1977). Ultimately, the NDA aims to provide reliable additional information that workshop participants may not think about, bring in new perspectives, and directly influence the model-building process. This might lead the group to ‘aha moments’, also known as critical learning incidents (Thompson et al., 2016), and subsequently create a new shared understanding that goes beyond participants’ initial homogenous mental models. In this way, the NDA performer becomes an asynchronous knowledge broker between missing stakeholders and workshop participants (Sedlacko et al., 2014).

In summary, the NDA role takes the basic principles of role games, where workshop participants impersonate other relevant people, and the intuition of Gerrits and Vaandrager (2018) of collecting information about missing perspectives in advance, to be then used in the workshop, and it combines them to overcome the limitations of the two techniques so to address homogeneity.

3.3. Guidelines and role-plot

During a workshop, one or more facilitators leave their role, impersonate the unrepresented or absent stakeholder groups and bring their mental models into the room. Notably, the NDA is a role that can run in parallel with other scripts for part of the workshop or the whole meeting (Richardson & Andersen, 1995). While scripts report a precise series of steps that need to be followed to perform a task during the facilitated modelling sessions (Ackermann et al., 2011; Andersen & Richardson, 1997), the NDA is instead instructed by guidelines. Those differ from the script in that they do not encompass a list of sequential steps to follow but a set of behavioural postures to comply with. In fact, the role performance in a facilitated modelling workshop depends on many factors (e.g., discussion dynamics, topics, priorities, etc.) that may not be anticipated and scripted in advance, and room is inevitably left to the judgment of the facilitator playing the NDA. The other main differences with Vennix’s approach are summarized in Table 1.

Table 1. Comparison between Vennix (1996) adapted and the New Devil’s Advocate.

	Vennix’ Devil’s Advocate	New Devil’s Advocate
Type	Script	Role
Performers	Subgroup of participants	Subgroup of facilitators
Task	Subgroup challenges the assumption and ideas adopted so far	The NDA impersonates the missing stakeholders
Preparation	None	Analysis of the missing stakeholders and information collection
Facilitation requirements	At least one facilitator	At least two facilitators (one acting as NDA)

We propose the following behavioural guidelines to conduct the NDA role in a workshop. Those indications are developed following two premises. First, one goal is to minimise the risks posed to the group work by the loss of neutrality by someone who previously was a facilitator. Particularly, attention has been devoted to identifying practices that try to maintain participants’ ownership of the model while clarifying that the NDA exclusively intends to represent the missing stances. Second, the guidelines are designed to make the NDA performer more comfortable with getting involved in the group discussion and eliminate the possible barriers preventing the group from seeing the NDA as an actual participant.

• The issues that the NDA should pose are studied, analysed and assessed for relevance and robustness before the meeting with the research team. All issues raised should be: i) related to a given stakeholder group, ii) supported by a source backing that claim (e.g., literature, interviews with stakeholders or experts), iii) summarised into a sentence to guide the NDA on how to present the issue to other participants and iv) associated to a list of possible relevant variables.

• The NDA sits and stands among the participants and not among the facilitators.

• The NDA suggests variables and links like any other participant. They can also participate in nominal group techniques for model conceptualisation and formalisation.

• The NDA does not interfere in group activities that require reaching a consensus among participants. This includes when the facilitator asks participants if they agree on a suggested model item. During these moments, the NDA should not interact because reaching a consensus is an internal group process through which the commitment to the model is built. If something appears to be wrong, the NDA can ask for clarifications but should not veto or start a conflict.

• The NDA can suggest variables and model modifications and share arguments and data sources to support them in a low-key style (Janis, 1972) and an inquiry mode (McCardle-Keurentjes & Rouwette, 2018; Schein, 1990). Bad practice: “the model is wrong…the literature says …” Good practice: “I see. But, have you ever considered this option described in literature source X?” or “This variable might be connected to the other one. Is it useful? Why don’t you think so?” With the latter formulations, the NDA challenges participants’ ideas but does not impose their own. The burden of thinking about the stimulus provided by the NDA is shifted to the participants. Moreover, questions are considered less harmful than statements in such a discussion (Schein, 1990).

• Ultimately the role of the NDA is to pose ideas (backed by literature and data sources from missing stakeholders) and explain the reasoning why these ideas could be relevant. Participants are the ones who make a decision.

• The NDA should show flexibility, considering this is a novel approach and there is the possibility of unexpected reactions from participants.

The preparation for the role requires considerable effort. First, the missing stakeholders need to be identified before the workshop in which the NDA is used. This task is performed by the facilitation team and it may be done based on participants’ suggestions and comments (e.g., inputs and discussions occurred during workshop sessions, ad-hoc consultation), literature, researchers’ expertise and other formal techniques (e.g., snowballing technique, Prell et al., 2007). The team utilising the NDA may screen scientific and grey literature, conduct interviews and analyse technical reports to profile their points of view. All relevant information obtained may subsequently be organised in a table format, including the possible issues to raise concerning unrepresented stakeholders, how to frame them during the facilitated workshop, suggestions on how to represent these concepts in the form of variables, causal relationships in the case of GMB and, finally, the corresponding information sources. The resulting table containing the information collected and provided to facilitators for playing the NDA role is called role-plot and should instruct the NDA player on the content of their interventions during the role performance. In the supplementary material, we illustrate the prototypical headings structure of the role-plot table (Annex A). Once NDAs have familiarised themselves with this information, they may envision a possible behavioural strategy to adopt during the workshop (e.g., prioritise some issues that will guide them during the session). Overall, the behavioural guidelines are expected to stay the same for the role performance in any context, as they deal with the general nature of the NDA, while the role-plot needs to be adapted to the specific content of the case.

To perform the proposed role in a workshop, at least two facilitators are required: one facilitates the workshop while the other plays the NDA. The facilitator should introduce the NDA to workshop participants at the beginning of the session. If this setting is not adequately explained to the group, it may generate misunderstandings, tension and even conflicts and thus undermine the whole project. It is also a good practice to ask participants whether they are comfortable with this approach and allow them to refuse it.

4. Case study

4.1. Case study setting

A joint project between the UK Government Department for Business, Energy and Industrial Strategy (BEIS) and the University College London (UCL) led to the development of the NDA role (Cunico, 2017; Cappuccio et al., 2017). BEIS started this collaboration to explore holistic approaches to address the complexities of the housing retrofit system and improve on past policy-making processes, which had not sufficiently understood the systemic effects of different agents’ actions and their interrelations (Rosenow & Eyre, 2016). Specifically, the failure of the previous policy (‘Green Deal’) in promoting the uptake of economically convenient energy efficiency measures was sought to errors in capturing homeowners’ motivations, constructors’ necessities and lending mechanics for retrofit measures (Guertler et al., 2013; Rosenow & Eyre, 2016), due to policymakers’ reduced “capacity to perceive and tackle problems with the Green Deal’s development outside the ‘corridors of power’”, i.e., due to barriers to stakeholders’ engagement (Guertler et al., 2013, p. 162). The project’s scope was to use GMB and further modelling to explore the dynamics behind homeowners’ retrofit uptake and how the policies under investigation in BEIS would trigger systemic change. The project occurred between the end of February and April 2017, a short period for a GMB intervention. However, due to high time pressure and some confidentiality concerns, only members from different areas of the BEIS policymaking and analysis team attended the sessions. Therefore, the project presented conditions of homogeneity and a short time span and was thus ideal for testing the NDA role, especially to decrease the likelihood that past policies’ misconceptions due to limited system understanding could be repeated.

4.2. Overall GMB process

The whole process consisted of two preliminary interviews in February 2017 with a gatekeeper (i.e., a contact person at BEIS acting as liaison with the modelling team), two workshops to build the model’s structure interspersed by an interview with the gatekeeper, five interviews with BEIS experts to refine and quantify specific model variables, and a final follow-up meeting with participants, which was open to all policymakers interested in the outcomes (see Figure 1).

Figure 1. Project timeline in which the case study was conducted (adapted from Cappuccio et al., 2017).

Preliminary interviews served to familiarise the client with the method, define the scope, focus and outcomes clearly and get to know the project constraints, e.g., concerning stakeholder availability. Following GMB best practices, we structured workshops in scripts and used them to build the SD model structure with participants (Andersen & Richardson, 1997; Hovmand et al., 2011). The outcomes of the workshops were maps in the form of causal loop diagrams (CLDs). All workshops were audio-recorded. At the end of each workshop, we collected participants’ verbal feedback. In-between workshops, we delivered a workbook with the outcomes of the first session to the group, as recommended to increase transparency and sustain participants’ interest (Ackermann, 1996; Vennix, 1996). The workbook also served as a validation step since participants were asked to report any disagreement with the map. The interview with the gatekeeper in-between workshops collected intermediate feedback on perceived satisfaction with the process, discussed possible changes and set up the next workshop. We also used it to ask BEIS about their willingness to try a new approach to overcome the homogeneity issue. We used and tested the proposed NDA during the second workshop because we did not know what the missing perspectives were before the first workshop. We also preferred to build a relationship with the participants through the first workshop before attempting the new role.

Behind the scenes, we constructed a simulation model based on the outcomes of the workshops. After the two workshops, we conducted interviews with experts to collect further data for the model and to clarify specific details on the model’s structure, thus providing a form of validation (Barlas, 1996; Forrester & Senge, 1980). Finally, less than two months after the first workshop, we presented modelling results to the client organisation.

4.3. NDA activities during the second workshop

The second workshop was devoted to revising, expanding and improving previous work. The research team prepared a cleaned-up and slightly modified map based on the first workshop outcomes to be used as a starting point for the group work and discussion. The workshop lasted four hours and included five participants plus two regular and two acting like NDA facilitators. One of the participants had not attended the first session.

The more experienced system dynamicist and a PhD student performed the NDA role since they had broader expertise on the issues under discussion. Specifically, the two had substantive content expertise, while only the first had GMB process expertise (Huxham & Cropper, 1994).

Reflecting on the discussion results after the first workshop, the facilitation team realised that three main points of view were potentially lacking: lenders, supply chain stakeholders and homeowners. However, one participant had worked in financial institutions for many years and therefore had the expertise to represent the mental model of lenders. To collect information on the interests and issues of the remaining two stakeholder groups, we conducted three interviews with experts (two on the supply chain and one on homeowner decision processes) between the two workshops. We complemented these by reviewing the literature and collecting information from the facilitators’ previous research experience. Although both NDA performers had content expertise (Huxham & Cropper, 1994), performing interviews and exploring the literature was important to ensure more completeness and to reduce the potential biases or erroneous beliefs that NDA players in the role-plot could introduce. Subsequently, all these inputs were collected in the role plot (see Table 2 for a sample) provided to the two NDA performers a few days before the session. The full role plot used in the second workshop consisted of 45 inputs, 10 of which the facilitation team considered to be of high priority.

Table 2. Sample of the role-plot table with the information provided to the facilitators playing the New Devil’s Advocate.

Issue to raise	Stakeholder bearer of that issue	How to frame it	Possible variables related to	Source
Existing dwellings are likely to represent 70–80% of the 2050 stock. 30% are going to be new houses. In 2016, 140,660 houses were completed. Legislation “Building Regulations and associated technical guidance”, regulation called “Zero carbon emissions” that have been stopped	Constructors / other policymakers	“At the moment, new houses are not relevant in the model. It is a stock outside the core structure. But I think that if we adopt a long-term view, they can be important and therefore valuable to be taken into account.” Are there any legislative requirements for energy efficiency and GHG emissions from new buildings?	“New building rate”, “demolition rate”	(Shrubsole et al., 2014), (Government of United Kingdom, 2017)
Top renovation payments methods: 1) Savings/personal finances 85% 2) Cash from home mortgage refinance 14% 3) Credit card—to be paid off over time 14% 4) Gift/inheritance 11% 5) Personal loan from friends/family 4%	Renovation expert	From what I know, only 14% of the people that are renovating choose to pay with money from home mortgage refinance. Is this data realistic according to you? Do you agree that knowing the magnitude of people paying in this way is important?	“Percentage of people using mortgage refinance for paying for renovations”	(Houzz & Home, 2016)
People in the UK are reluctant to take loans and mortgages	Industry stakeholders/ Households	Households in the UK are not very willing to take a loan or refinance their mortgage to pay for their renovations. This can have an impact on the policy.	“Reluctance towards mortgages and loans”→ “Percentage of people using mortgage refinance to pay for renovations”	Interview with a lecturer from the UCL Institute of Environmental Design and Engineering with multi-year work experience in the construction sector

At the beginning of the workshop, the facilitation team introduced the NDA role, asked participants for permission to implement it, and informed them that it was their right to interrupt the NDA role performance (Annex B reports how this introduction was scripted). The workshop was composed of two major parts. First, the map was revised and inconsistencies with the perceived ‘real system’ corrected. Second, participants wrote down the names of important missing variables on post-its and stuck them on the map. Then, the group discussed the added variables one by one and, once agreement was achieved, new variables were connected to the preliminary structure. The two NDA players were actually performing as participants, took part in all the group tasks (except the ones in which participants were reaching a consensus) and facilitators treated them like participants throughout the session.

5. Data analysis methods

The execution of the new NDA approach potentially impacted several aspects of the facilitated modelling project. The NDA primarily alters the dynamics of the workshop in which it is performed and the resulting model. Yet, the case of Gerrits and Vaandrager (2018) showed that it could also affect the whole project. Therefore, we considered it necessary to collect information on different levels of analysis and phases of the project potentially affected by the NDA performance, specifically, on the general outcomes of the project and reactions from the organisation, the impacts of the proposed role on the model output and the interaction dynamics among NDAs, participants and facilitators during the workshop. Such an approach constitutes the evaluation framework (Fig. 2) that we used to assess the overall impact of the NDA on the facilitated modelling effort. The remaining section describes the specifics of the data collected and how it was analysed.

Figure 2. Evaluation framework adopted.

To understand the performance and impact of the new NDA approach within the modelling workshop, we followed the methodological considerations and guidelines proposed by Franco and Rouwette (2011) and relied on different data sources and types to minimise biases through cross-comparison (e.g., using self-evaluation exclusively may affect results - de Gooyert et al., 2022). First, we used workshop recordings (transcribed by a contracted professional; duration: approx. 3h45min; length: approx. 26,000 words). Second, to capture participants’ perceptions related to the setting, we orally collected evaluation feedback from regular participants at the end of the workshop (recorded and transcribed) given their limited availability, and the first author (who was one of the two regular facilitators) interviewed the two NDA performers and the other traditional facilitator a few days after the workshop (Annex C reports the list of questions used to structure the interviews). Third, the first author wrote a memo with his reactions, perceptions and ideas about the session to complement other feedback (Miles et al., 2013). It is assumed that integrating hard data, such as the workshop transcript, with more soft ones, like memos and feedback, complements and enriches the overall dataset (Tavella & Franco, 2015).

Building on the workshop transcript, we then conducted a micro-level analysis of the participants’ interactions to understand better the NDAs’ behaviour and impacts on the output. The study of the micro-processes in workshops, such as statements, tone, body movements, etc., (e.g., Franco & Greiffenhagen, 2018; Franco & Nielsen, 2018; Tavella & Franco, 2015) are believed to help unpack the rich communicative interactions a single workshop can provide (Franco & Rouwette, 2011; Tavella & Franco, 2015). In particular, focusing “on data from a single workshop is appropriate where the interest is on examining communicative interactions in depth, particularly those mediated by technological artefacts such as models” (Tavella & Franco, 2015, p. 458) as it is the case for the NDA role. Therefore, we decided to code all NDAs interventions with the intent to assess how they contributed to the workshop.

To code and interpret the data collected, we combined deductive and inductive approaches, a common practice in coding workshops (Franco & Rouwette, 2011), which meant iteratively going back and forth between data and theory (Franco & Nielsen, 2018; Orton, 1997; Tavella & Franco, 2015). Thus, although the analytical steps outlined below are reported linearly, the actual inquiry comprised some iterations between these steps. First, we sensitised ourselves to the data by reading and taking notes on the key and recurring attributes of the NDAs’ interventions and confronting our insights. Second, given our interest in NDAs’ contributions to the workshop’s dynamics, we looked for existing theories on workshop communication in the literature. Thomas et al. (2011, p.26) provide a coding framework to categorise communicative practices in workshops that suited our purposes, covering the behaviours previously identified, which was already successfully adapted and utilised in micro-analysis of facilitated modelling workshops (e.g., Tavella et al., 2021; Tavella & Franco, 2015). Therefore, we based our codes on Thomas et al. (2011) scheme and adapted it to the specifics of our context (Table 3 below reports the codes and related descriptions). Specifically, building on the simplified version of Tavella et al. (2021), three additional codes were added as those emerged during the process and were not precisely captured by Thomas et al. (2011) framework: (i) modelling to capture explicitly interventions demanding work on the model (e.g., how this item should be modelled?); (ii) facilitating to catch facilitation interventions (e.g., how would you prioritise these elements? Let’s do some brainstorming) not precisely covered by existing codes (e.g., inviting, reiterating); (iii) repetition to account for cases in which the NDAs were interrupted and then continued their previous statement and so do not to overcount twice the same interventions. To assess the NDAs’ intervention, we used the turns as the level of analysis, meaning any NDAs’ contribution from the moment they started speaking until they finished, which could consist of a few words or long and multiple sentences. This possibility was mentioned by Franco and Rouwette (2011) and fruitfully used by Herrera et al. (2016) and Tavella et al. (2021) to capture workshop participants’ contributions through a session. Although not coded, all the other workshop participants’ interventions were also analysed since their understanding was instrumental in coding the NDAs’ interventions. The first author coded the workshop transcript and, to extend reliability, by an external researcher not involved in the research, following best practices provided in the literature (O’Connor & Joffe, 2020). The second coder was instructed on the context of the study and the workshop and provided with the transcript (suitably anonymised) and the coding scheme along with descriptions and generic examples for the codes. The agreement rate between the coders (McHugh, 2012) was 80.3%, and Cohen’s Kappa 0.728 (Kim, 2017; Neuendorf, 2017), indicating a substantial level of agreement (Neuendorf, 2017; O’Connor & Joffe, 2020). Then, following the example of Tavella and Franco (2015), we identified three relevant, general and functional recurrent patterns of behaviour emerging from the coded communicative practices: production (i.e., interventions proposing and contributing to the production and sharing of knowledge), decisional (i.e., interventions deciding on the model content and its legitimacy, i.e., whether including specific elements in the model or not), and procedural (i.e., facilitation interventions to address the participants’ discussion and needs and to address the work towards the model). How these patterns reflect the coding scheme is reported in Table 3. Overall, production and decisional patterns refer to the discussion related to content development (Huxham & Cropper, 1994; Papamichail et al., 2007), which workshop participants typically do. The role guidelines encouraged production interventions by NDA performers, while they discouraged the decisional ones because they interfered with the group ownership of the model. On the other hand, the procedural one relates to the workshop process and activities management (Huxham & Cropper, 1994; Papamichail et al., 2007), which the neutral facilitator usually does. Therefore, the expectation was to see a predominance of production patterns emerging from the coding exercise if the role were conducted respecting NDA role guidelines.

Table 3. New Devil’s advocate coding scheme.

Pattern	Code	Description
Production	Proposing	Turns that introduce, suggest and/or propose new concepts/ideas/items
	Building	Turns that engage with, elaborate, develop and provide information about concepts under discussion
	Clarifying	Turns that demand and/or open up a reflection on clarification of concepts and content
Procedural	Modelling	Turns that direct group discussion explicitly toward the model structure and construction
	Facilitating	Turns that aim to facilitate group discussion and work, like prioritising, prompting brainstorming, etc.
	Inviting	Turns that encourage participation by other group members
	Reiterating	Turns that repeat and summarise previous content
Decisional	Affirming	Turns that agree with the content proposed, under discussion, and/or to be included in the model
	Dismissing	Turns that serve to rebuff or ignore content proposed under discussion and/or to be included in the model
	Challenging	Turns that reject or critique alternative content proposed by other members
	Undermining	Turns that criticize other members for discrediting their proposed content
	Deploying Authority	Turns that contain directives that eliminate alternative content proposed by other participants referring to superior knowledge or expertise to justify the legitimacy of a proposed concept
Other	Interruption	Code to account for interruptions to avoid double counting the same communicative behaviour

Scaling up from the coding scheme to more general patterns allowed us to streamline the analysis of the behavioural postures of the NDAs, while still rooting it in a granular and robust coding method. However, this higher-level grouping is far from perfect, as workshop members’ communicative behaviour is much more nuanced and cannot be rigidly categorised. For example, clarifying interventions are characteristic of regular participants. Still, enquiring about clarifications can also be a communicative method used by facilitators to understand the content under discussion better and prompt further reflections within the group (Schuman, 2005; Vennix, 1996). Also, inviting and modelling interventions, typical of facilitators encouraging silent participants to join the discussion and directing discussion towards the model, are not exclusive to them. Workshop participants can also invite others to enter the debate or be concerned with the model construction. That is to say, the generalisation has functionally been done based on the archetypical behaviour of workshop members to analyse NDAs’ behaviour, but it has limitations that readers must be aware of.

Finally, to assess the impact of the NDA performance on the model output, the transcript was cross-compared with the resulting model to identify variables and connections in which the NDAs had a substantial role in their development. Kim and Andersen (2012) provide a robust approach to translating purposive texts into SD structures. Their lessons on discovering themes, identifying variables in texts, and relating texts to diagrams were our guidelines in this context. First, the additions and variations made in the second workshop were identified. We then compared them with the inputs given in the role-plot table provided to the NDA performers. This allowed assessing to what extent the stances of the missing stakeholders collected have been incorporated into the model. Secondly, we spotted the segment of the workshop (i.e., a portion of the discussion focused on a specific argument) where those model modifications were discussed and agreed upon by the group. The combination of this information with the NDAs interventions and production patterns allowed us to preliminarily identify the variables and connections in which the NDAs had a rather clear impact (i.e., substantially shaped and contributed to the discussion on the inclusion of that model element) and thus show the potential effect of the NDAs on the model development graphically (see Fig. 3). Although it is impossible to have the counterfactual evidence of what the model would have been like had the NDA not enacted, this allows to develop a general and indicative sense of the relevance and impact on the content of the new role played.

Figure 3. New Devil’s Advocates’ impact on the system map (Stock and Flow Diagram). The initial model’s structure at the beginning of the workshop is depicted in black, additions made during the session are presented in red, and the structure directly suggested by the New Devil’s Advocate performers is shown in blue.

6. Results

6.1. General outcomes of the facilitated modelling project

Overall, the project’s outcomes can be considered positive. The research team built a causal map and subsequently a simulation model of the housing retrofit system. Policies under consideration by BEIS were tested and their impact assessed, which was considered useful by the client organisation. Positive feedback was received by the client organisation, in particular the gatekeeper, on the project and method. After that, the organisation continued collaborating with the research team on this topic.

6.2. Second workshop model outcomes

Participants provided positive comments on the workshop and expressed appreciation for the NDA setting. The research team was satisfied with the session and its outcomes. The resulting map was used to build the final version of the quantitative SD model. Figure shows the map, distinguishing the structure developed at the first workshop and used as a starting point (black) from all additions done during the second workshop (coloured). Among those, implemented inputs in which the NDAs played a key contribution are reported in blue, while the remaining additions and modifications in which the NDAs did not contribute substantially but were proposed and discussed by the other regular participants are in red. As explained in section 5, those were identified by tracking back the new elements and variations into the workshop transcripts and then cross-comparing those textual segments with the patterns identified through coding. Links supported only by weak evidence are dotted. Specifically, the NDAs were found to play a substantial role in 28% and 36% of the variables and connections added to the model (Table 4 below). Of the 45 potential inputs provided to the NDAs in advance, 20 could be identified in the map (see Annex D). Overall, these results show how the NDAs had an impact on the resulting model and added part of the identified instances of the missing stakeholders.

Table 4. Second workshop New Devil’s Advocates’ and participants’ cumulative interventions, variables, and connections.

Items accountable to	Interventions	Variables	Connections
NDAs	112 (20%)	15 (28%)	27 (36%)
Other participants	452 (80%)	39 (72%)	48 (64%)
TOTAL	564	54	75

6.3. Coding results

Table 4 shows the overall results of the coding process, namely the sum of interventions in the workshop and the number of variables and connections in which the NDAs played a key role or were developed mainly by the other participants.

The two coders agreed that both NDAs kept a gentle and collaborative tone throughout the entire session. NDAs actively participated in the discussion and contributed to defining the model’s structure as they made 20% of the interventions and contributed to 28% and 36% of new variables and causal links, respectively. Table 5 presents how the NDA interventions have been categorised, showing how the NDAs mostly engaged in content development and directly contributed to the group co-production of knowledge (production pattern − 84%). However, most of the contributions (96%) came from only one NDA, the more experienced one. Moreover, this same NDA, on a few occasions, abandoned her NDA role to intervene in the workshop functioning (procedural pattern − 12%) and the group decisions (decisional pattern − 2%), in particular, towards the end of the workshop. Regarding the procedural interventions, the modelling coded turns are not particularly concerning as this behaviour could also emerge in participants particularly committed to and interested in the model. However, the interventions coded as ‘facilitating’ and ‘reiterating’ are exclusive to facilitators and highlight a trespass of role. An analysis of those turns showed how the NDA intervened by playing back participants’ statements when the facilitators struggled for a moment, she summarised previous discussions to engage the other participants, suggested new areas to investigate when the groups’ contribution was slowing down or expressed concerns on how the modelling team could translate a specific structure into a set of equations in the subsequent project phases. While ‘dismissing’, ‘challenging’, ‘undermining’, and ‘deploying authority’ have not been found, the two decisional interventions (both coded as ‘affirming’) are particularly interesting. In one case, she verbally nodded to another participant while he was talking as a sign of support, which did not seem to influence the group’s decisions. In the other case, she intervened with the other participants when the facilitator asked the group whether to include a variable in the model. The transcript also showed that right after, the facilitator reiterated his request and directed it to the rest of the participants seeking their opinion.

Table 5. Coding results for the New Devil’s Advocates’ interventions.

Pattern	Code	Total NDAs interventions
Production	Proposing	18 (16%)	94 (84%)
	Building	41 (36%)
	Clarifying	35 (31%)
Procedural	Modelling	7 (6%)	13 (12%)
	Facilitating	5 (4%)
	Inviting	0
	Reiterating	1 (1%)
Decisional	Affirming	2 (2%)	2 (2%)
	Dismissing	0
	Challenging	0
	Undermining	0
	Deploying Authority	0
Other	Interruption	3 (3%)	3 (3%)

6.4. Participants’ feedback

After explaining the new setting, all participants welcomed the idea of having someone trying to act as some of the missing stakeholders, which would challenge their usual way of thinking and help avoid groupthink. Regarding the assessment of their experience with the NDA approach, participants replied in the end when questioned for feedback. Similarly to Kotiadis and Tako (2018) and Zeijlemaker et al. (2022), we report the extracts from the workshop to provide the reader with a more nuanced understanding of the participants’ reactions:

It’s good to have challenge and more input. I think it works well.

In the same line, another participant added:

For avoiding groupthink I think it’s really important… Because we will all ultimately work together… it’s good to have a challenge.

Then the participants were asked how they felt during the process:

I thought it was quite useful [referring to the NDA], just a different thought process.

Yeah, working in a different way.

Lastly, when researchers explained to participants their initial concerns regarding a possible lack of familiarity and effectiveness of the NDA role, a participant said:

[By being transparent] You got trust.

In general, there were no negative comments about the new role and workshop attendants all had a positive reaction.

6.5. New devil’s Advocate performers’ feedback

Both NDA performers reported they felt a friendly atmosphere during the session. They thought it went well and responded affirmatively when asked whether they would consider using this new role again if in a similar homogeneity situation. Both NDAs highlighted the experience required to perform the role. They specify that experience, in this case, comprehends a multiplicity of concepts. First, it means more experience in the NDA role could make a difference. Namely, it would be much easier if they had to play it again. Second, they reported that having experience with GMB procedures helps (process expertise). The different ability to be engaged in the group discussion shown in the coding of the two NDAs’ contributions supports this view. Lastly, they believe that having a solid knowledge of the topic under discussion (content expertise) certainly affects the ability of the NDA player to perform the role. In other words, according to them, without a general comprehension of the topic of which the facilitated session is about, the role-plot with missing stakeholders’ information alone might not be enough to perform the NDA role satisfactorily. They also reported some issues they faced. The less experienced facilitator, a PhD student with robust content expertise but little in group facilitation, found it challenging to step into the conversation several times because she was not feeling entirely comfortable, and she was trying to formulate her intervention as consistently as possible with the SD language, and this slowed her down. On the contrary, the other facilitator did not have this problem. This seems to be coherent with their claim of the importance of facilitated modelling expertise. As coding captured, the experienced GMB practitioner mixed, a few times, her NDA role with the one of a regular facilitator. When asked, she said she was not always aware of it and realised it only a few times after it happened.

6.6. Facilitators’ feedback

Overall, the two facilitators judged the new workshop setting as positive. It did not create any conflict among participants and the group dynamics were fluid; namely, no friction due to the NDAs interventions was noted. According to the facilitator, the NDAs did not appear to steer in any way the process with their SD knowledge, and other participants seemed to accept rather smoothly the idea of having two new ‘experts’ in the room. Moreover, the whole process seemed to be much more structured than the first session, in which participants tended to discuss among themselves rather than formulate items for the map. Both facilitators also noted the partial role-switching of the more experienced NDA player towards the end of the session, which is consistent with the coding outcomes. In this respect, they mentioned that this workshop was more exhausting for them than the previous one and that, towards the end, they had a few issues in handling the session when the NDA intervened as a facilitator. However, from the facilitators’ perspective, it seems that the participants did not notice the role change or did not rate it as important as they did not show any change in their attitude, behaviour or body language. Both facilitators appreciated and believed teamwork among the whole research team was essential for the results. For example, when the NDA intervened on the group ownership of the model, the facilitators said they noticed it and covered the mistake by promptly asking the rest of the participants whether they agreed or not with including the structure under discussion, which is in line with the transcript. Lastly, both facilitators were satisfied with the quality of the model resulting from the workshop.

7. Analysis and discussion

7.1. Insights on role performance and group dynamics

Data analysis and comparison provide an interesting snapshot of the NDA role performance and its effect on the group and model. Overall, in this facilitated setting, the importance of the human component in the facilitation team seemed to be even higher than usual. Feedback and coding concur in showing how the two NDA players performed differently. While both had content expertise, only the more active one also had experience in the GMB process. In addition, the two had different confidence levels and personalities; the more engaged NDA performer showed higher confidence. Thus, we believe that the background and personality of the NDA performers influenced the way the role was carried out. A few times, the more experienced facilitator stepped out of the role and intervened in the group’s decisions or behaved like a facilitator. While the decisional intervention only occurred once and could be attributed to a momentary slip-up, the facilitation ones instead happened multiple times. Although those interventions occurred during the whole workshop, they mainly occurred towards the end. One potential explanation may be that the role performance can be energy demanding, and a person who usually is a facilitator while playing the NDA may unintentionally behave as they are used to. Another possible explanation is that the NDA did it deliberately to support the facilitation process, which seems to be the case when the NDA helped while the facilitators were momentarily struggling. Stepping out of the role could have posed several risks since it could have altered the group ownership of the model and made the NDA not perceived as a regular workshop participant anymore. However, given the feedback, the transcription, and the group commitment to the project, it seems that other participants did not notice the role change or did not rate it as important. This could be because of the limited number of times the role change happened or the marginality of these occurrences considering the whole NDA performance.

Concerning group dynamics, the NDA role deals with a very sensitive matter in facilitated modelling: facilitator neutrality. As presented in section 3, this aspect is recognised as essential to the role (Phillips & Phillips, 1993; Schuman, 1996; Scott et al., 2016a, Elsawah et al., 2023), although other researchers advanced the idea that facilitators might directly intervene in the content from their privileged position (Huxham & Cropper, 1994; Papamichail et al., 2007; Tavella & Papadopoulos, 2015a). This debate is vivid and has substantial practical implications (Voinov et al., 2016; Voinov & Bousquet, 2010). This work shows a case in which, after taking adequate precautions, a previously content-neutral facilitator intervened substantively with the declared aim of contributing to increasing output quality. However, at the same time, the NDA corroborates the importance of having a neutral figure during a facilitated modelling session. In fact, a previous facilitator could play the NDA only because someone else was maintaining the facilitator’s content-neutral role. If the facilitation team is composed of at least two members, one of them can play the NDA role and intervene in the content if the context allows - group culture and necessities (Phillips & Phillips, 1993), etc.) and with precautions taken (transparency, group previously informed, all participants accept, facilitators’ adequate preparation, etc.). In particular, as feedback suggested, trust and participants’ willingness to be challenged seemed to be critical elements for the success of the new role, as it was identified for the success of a traditional Devil’s Advocate technique (De Rivera, 1968; Jervis, 1968). More generally, this is consistent with reflections on the importance of trust among the three components of a facilitated modelling effort (stakeholders-models-facilitators) for the good outcome of those projects (Harper et al., 2021). The feedback collected and the project outcome show that the role did not negatively affect model ownership and participants’ commitment (Rouwette, 2011). Also, the overall perception of the facilitation team (Ackermann, 1996) did not appear to be distorted. This suggests that it is feasible for part of the facilitation team to leave their neutral stances for at least the duration of a workshop under specific circumstances without undermining the good outcome of the facilitated modelling project. This process allowed injecting into the discussion the missing stances and expertise that would have probably been overlooked by the group and have undermined the model’s usefulness. Besides countering homogeneity, this new proposed role could also be a safe way for facilitating modellers feeling the necessity to intervene in the content without risking damaging the relationship between the facilitation team and the group or the need to leave the facilitation role for the whole project (Voinov et al., 2016; Voinov & Bousquet, 2010). In addition, the feedback collected suggests that the new role improved the structuration of the workshop discussion, which could be an expected positive side effect. The NDAs may have shown other participants how to express their ideas more coherently with the SD language (e.g., how to suggest variables and links), being instrumental in the structuration of the group actions and group appropriation of the GMB method (Scott Poole & DeSanctis, 1992; Tavella et al., 2021). Unfortunately, the current level of evidence is not sufficient to substantiate this claim, as the difference in behaviours shown by participants could be due to their growing familiarity with causal mapping or the two facilitators were more experienced with the group dynamics and more capable of conducting the second session. If further studies support this finding, it could be one alternative application of the approach as a method to guide inexperienced facilitated groups (i.e., groups who have not yet appropriated the facilitated modelling method) to adopt more adequate ways to discuss and perform tasks.

Finally, concerning the relationship between facilitators and NDA players, teamwork (Richardson & Andersen, 1995; Whiteley & Garcia, 1996) was signalled as essential to overcome the challenges posed by the performance of a new role. Teamwork, intended as the spirit of the facilitation team to cover and support each other, was necessary during the preparation phase, in the selection of the roles, during the workshop and in the unusual interactions during the session (the NDAs had a new relationship with their facilitator colleagues). For example, how the facilitators promptly handled the NDAs’ decisional intervention could be seen as a teamwork spirit; similarly, as another example, the procedural interventions made by the NDA when the actual facilitators had difficulties in managing the discussion. Adaptability also seemed to be an important skill (Andersen & Richardson, 2010) when facilitating this specific role and, more in general, original and untested new approaches. In this case, adaptability could be seen in the ability of the facilitation team to perform a new role with different duties and interaction dynamics and adapt relatively quickly to the new context.

7.2. Role assessment

Overall, the evidence collected through this application (feedback, impact on the model and project outcome) supports an initial positive view of the NDA role as being what Hovmand et al. (2011) define as a promising tool to address homogeneity in need of more applications and tests to become an established practice. An analysis of strengths, weaknesses, opportunities and threats (SWOT) (Osita et al., 2014) was undertaken to summarize the current knowledge, resulting from the assessment conducted on the state of development of the NDA role and facilitate future practical applications (see Figure 4). Specifically, strengths and weaknesses capture aspects encompassed in the role, while opportunities and threats depict situations that may potentially occur. The SWOT analysis is expected to help to identify external and internal factors that are favourable and unfavourable to achieve the heterogeneity objectives of the setting. However, it does not provide a prioritisation, so, for instance, weak opportunities may seem to counterbalance substantial threats, which is a limitation of such analysis.

Figure 4. New Devil’s Advocate SWOT and mitigations analysis.

The SWOT analysis identifies several strengths. In general, the NDA role respects confidentiality and time constraints posed by participants: no external persons and modifications to the schedule are needed. Compared to the approach used by Gerrits and Vaandrager (2018), benefits include the much smaller demands the proposed setting poses on participants. The effort to prepare the setting is on the facilitation team. The NDA performers prepare to act as the missing stakeholders before the workshop; thus, it is also anticipated to work under time constraints among participants. It also requires a lower level of initial trust and own initiative by group members. Moreover, the NDA guarantees a minimum standard over the missing stakeholders’ representation, which does not happen when other approaches to tackle groupthink are used to address homogeneity where regular participants might not be able to bring such input promptly during a workshop. In this vein, the benefits appear to be multiple. First, since it is not just a script for part of a workshop but a role present throughout the whole session, it is expected to have an in-depth impact on the model quality because it may affect several group tasks. Second, participants locked-in in their mental models might not even realise such limitations when using inquiry techniques such as the traditional Devil’s Advocate and its evolutions (including the adaptation for GMB proposed by Vennix, 1996), while an external input from an NDA could help overcome them.

A potential opportunity needing further research is that the NDA role may help better organise the deliberative process. Having an expert in modelling/facilitation as a participant can contribute to structuring the group’s work. For example, the NDA may demonstrate to other participants how to suggest variables and express concepts in the form of causal relations and thus lead by example on how to interact fruitfully during a facilitated modelling workshop.

The NDA role also encompasses four weaknesses. First, it requires the facilitator playing the NDA to have enough knowledge to represent the missing stakeholders; otherwise, they may suggest unrealistic issues, which lowers the model’s quality and undermines the relationship with other participants. This risk is not present in other approaches potentially suitable for homogeneity since facilitators tend not to get involved in the group content discussions. Second, compared to other methods addressing potentially addressing homogeneity, where additional preparation time for the facilitation team is almost null, another weakness is the significant time required to prepare the scripts for the setting. Third, the facilitation team may lose some capacity since at least one plays the NDA during the entire workshop. This increases the workload for the remaining facilitators. Lastly, this new role involves uncertainty. Although careful analysis can be conducted in advance to assess potential workshop situations, it is impossible to anticipate all the factors that will interplay during a facilitated session where the NDA is used. Thus a high degree of flexibility is required for the facilitation team (Andersen & Richardson, 2010).

Related to these weaknesses, a few potential threats (risks) must be considered in future applications of the NDA. First, the participants’ acceptance of this original setting is unknown and unpredictable. The shift in the role of the facilitator and the consequential loss of a neutral stance may harm group dynamics and trigger an adverse reaction. Although the risk of undermining the fundamental relationship with the group of participants is, to some extent, always present (Ackermann, 1996; Eden & Sims, 1979; Scott et al., 2016a), the proposed NDA role increases the likelihood of this happening. Moreover, the credibility of the NDA depends on the perceived quality of the issues and comments raised. In addition, the NDA performer could struggle to step into the discussion due to previously established group dynamics. Alternatively, the NDA player could steer the process. Due to more profound experience in facilitated modelling as compared to other participants, an NDA could more or less voluntarily impose the point of view they represent. The risk that the NDA player behaves like a facilitator during the session can also arise. Overall, the preliminary evidence collected showed that NDA role performance may be susceptible to personality and process expertise. On top of that, the facilitation team is reduced by at least one member. This could make the job harder for the remaining facilitators, and fatigue can lower the quality of their performance, especially in long workshops. In addition, if it is performed only in a one-off session, a former NDA player who acts as a normal facilitator in a later workshop may find it challenging to treat neutrally the structures they brought forward versus the ones that other participants suggested and might have a preference for their own suggested variables.

To counteract these threats, we developed corresponding mitigation strategies expected to decrease the likelihood of these potential risks to occur. Some are already integrated into the NDA guidelines, while others have explicitly been elaborated to diminish the related risks. Their development and refinement was done based on collected feedback, workshop transcriptions and analysis of the map developed with BEIS. Mitigation strategies include (a) transparency about the NDA role, (b) the building of trust with participants, (c) proper analysis and documentation of the stakeholders the NDA represents, (d) gentle interference of the facilitators if the NDA player is too silent or too active, (e) breaks and other adaptations to the workshop schedule to help with the reduced number of facilitators, and (f) selection and training of an appropriate NDA performer. We expect these mitigation strategies to highlight further improvements for future experimentation with the proposed NDA role.

7.3. Limitations

Several limitations also emerged regarding the NDA role and the method used to analyse it in the described case study application. Concerning the role, although guidelines on how to behave (role-plot) were provided, ultimately, how to intervene and what to say (words and tone selected) was left to the NDA player. Even if they attempt to follow the role-plot blindly, to some degree and even unwillingly, they may also express personal opinions and biases. This component appears unavoidable at the moment since it depends on the intrinsic human nature of group interactions. Therefore, it would be more precise to affirm that the NDA firstly works as an asynchronous knowledge broker between the participants and the missing stakeholders but also, to a limited extent, as a way in which facilitators express their opinion on the workshop’s content. As said, this is not necessarily entirely negative since it could be helpful for facilitators who may need/want a way to safely raise their opinions to the group without damaging group dynamics and undermining the neutrality of the entire facilitation team (Voinov et al., 2016; Voinov & Bousquet, 2010). In line with that, personal factors, such as background and personality, can affect how the role is played. Thus, the choice of who plays the part is a very sensitive issue: an NDA player interpreting the role in the wrong way could have zero or negative impact on the model output and ‘group life’ (mood, necessities, feelings, etc.) (Phillips & Phillips, 1993). Moreover, the selection of the inputs provided to the NDA performers in the role-plot can be improved. Even though data about stakeholders’ mental models were rigorously collected, it is challenging to determine to what extent the important issues to a given stakeholder group have been included and categorised adequately. How this information is organised in the script is left to the judgment of the facilitators preparing the role, and further standardisation might benefit the setting. Finally, the NDA does not affect one of the main deficiencies of the traditional and Vennix (1996) Devil’s Advocate approaches and other methods that can be attempted to counter homogeneity (e.g., techniques to address groupthink). In line with a behavioural operational research perspective (Franco et al., 2021; Kunc et al., 2016; Lane & Rouwette, 2023), facilitated modelling projects not only serve to co-create knowledge (Franco & Montibeller, 2010) and improve the quality of the resulting model, but also work as spaces in which different stakeholders gather to share the information and re-evaluate it in the light of the modelling and group discussion with the result of developing further their mental models and, eventually, agree on shared interventions (Franco & Montibeller, 2010; Rouwette et al., 2011; Scott et al., 2016b). This stakeholder interaction is an inner issue related to homogeneity that also the new proposed setting role cannot overcome. The NDA role can bring the view of the missing stakeholders into the session, but it cannot be a substitute for their actual presence. To partially overcome this limitation, and if feasible, facilitators could attempt to organise separate workshops with the missing stakeholders using as inputs the outcomes of the sessions in which the NDA was performed. However, for these limitations, the NDA role should never be seen as a substitute for the engagement of the stakeholders but only as a possible backup plan to use whenever this is not possible and homogeneity issues threaten the facilitated modelling effort.

Concerning the method adopted to analyse the proposed role, transcription coding and its integration with feedback and model cross-comparisons provides a rich dataset allowing micro-analysis of specific facilitation aspects (Tavella & Franco, 2015). Still, several aspects remain unexplored. No knowledge is available on the body language and physical interaction between facilitators, participants and NDAs. Such information and its integration with existing data could unveil additional nuances on the NDA role and its execution. Furthermore, since all facilitated practices are heavily context-dependent (Faure, 1999; Gerrits & Vaandrager, 2018; Sadia, 2014), separating the results from the context in which the approach was tested is complicated. Tavella and Franco (2015) noted that this is a drawback encompassed by using a single case study. While “data from a single workshop is appropriate where the interest is on examining communicative interactions in depth, particularly those mediated by technological artefacts such as models” (Tavella & Franco, 2015, p. 458), at the same time, it does not allow to generalise the role to established practice (Hovmand et al., 2011). Moreover, the NDA role was played only in the final session of a GMB project, which did not permit to see if there were any long-term consequences on the group dynamics.

8. Conclusions and future research

This research aimed to explore the effectiveness of the proposed NDA role in a facilitated modelling exercise; in line with the efforts toward increasing the adaptiveness of facilitation modelling scripts, settings and roles (Ackermann et al., 2011; Andersen et al., 2007; Andersen & Richardson, 1997; Hovmand et al., 2011). It responds to the calls for increasing stakeholder plurality (heterogeneity) in facilitated modelling workshops when physical or remote inclusion of participants proves to be a challenge (homogeneity) for reasons of time pressure, availability, and confidentiality, among others. Existing techniques to address groupthink have important limitations when used to counter homogeneity issues. They do not guarantee a minimum standard on the quality of the missing stakeholders’ representations because they rely on the participants’ knowledge. Thus, we developed an adaptation of the traditional Devil’s Advocate for GMB (Vennix, 1996). This approach tries to artificially recreate the mental models of missing stakeholders in the workshop room by having a facilitator acting in their substitution (NDA), constituting a ‘knowledge brokerage’ platform between workshop participants and stakeholders that could not attend the session. The NDA, who is a facilitator and not a participant, collects knowledge about missing stakeholders that other workshop participants might lack and thus can (at least partially) represent these stakeholders indirectly. The NDA was utilised in a GMB project, and our analysis relied on a rich dataset, including workshop transcripts, feedback collected from participants, NDA performers, and facilitators and a cross-comparison of the input provided to the NDAs with the workshop transcription and the resulting model. The analysis shows that the new setting brought new elements that could have been otherwise overlooked. Overall, the study revealed positive results for the proposed design, and the NDA served the purpose. Participants appreciated the setting and did not have major problems with the loss of neutrality of part of the facilitation team, which can be seen as a contribution to the academic debate on facilitators’ neutrality. This case highlights that facilitator neutrality can be temporarily abandoned by part of the facilitation team if the context allows, specific precautions are taken, and someone else is still maintaining the facilitation’s neutral stance. Finally, the NDA role also satisfied the client’s time constraints showing that it can also be applied to deal with generic pressure cooker conditions (Gerrits & Vaandrager, 2018).

Although the proposed role shows benefits compared to the alternatives available to tackle homogeneity, it also comes at some costs. It may require a significant preparation time. The choice of who plays the role is critical since experience and personality may substantially affect the role’s effectiveness. Moreover, playing the role may be challenging even if a suitable professional performs the NDA role. It might be difficult to step into the group discussion or avoid behaving as a facilitator and just act as a stakeholder. Finally, despite the participants accepting the NDA player without any noted effect on the facilitator-group/client relationship, a single case study cannot fully show that the recommended precautions completely avoid any threats due to the loss of neutrality in all possible cases. For this reason, the proposed role can reasonably be defined as a promising practice (Hovmand et al., 2011) and, only after comparative studies based on the collection of multiple cases, generalised and considered an established one (Hovmand et al., 2011). Whether and when the NDA reaches established practice status, it could be added to the traditional roles of facilitation modelling (Richardson & Andersen, 1995).

From a broader perspective, the precautions adopted in this experiment might be valuable lessons for practitioners who want to test new facilitation strategies. In particular, we identified the facilitators’ ability to understand group life (Phillips & Phillips, 1993) as crucial to deciding how to tailor novelties to participants’ needs and avoid counterproductive applications. We believe this type of understanding is important for every facilitated modelling exercise, but a requirement in new settings, scripts, or variations where prior guidance, particularly on the risks, is not available and the possibility of failure is higher. For example, the attempt to address homogeneity suggested by Gerrits and Vaandrager (2018) may not have worked not because it was not technically robust but because it did not consider the specific group necessities in which it was tested.

Overall, although the case study was performed in a GMB setting, we argue that, with the proper adjustments, the NDA role could also be translated and used in other facilitated modelling methods (Franco & Montibeller, 2010), like facilitated problem structuring (Mingers & Rosenhead, 2004), facilitated decision analysis (Belton & Stewart, 2002), or facilitated discrete event simulation (Kotiadis & Tako, 2018). More broadly, this work should be seen as a further contribution to the increasing body of literature in behavioural operational research (Franco et al., 2021; Gooyert et al., 2022; Lane & Rouwette, 2023), unpacking the processes of utilising facilitated modelling (behavioural aspects of modelling - Brocklesby, 2016). Specifically, our effort provides a new potential method to use in facilitated modelling contexts as well as, among others, reflections on the problems of homogeneity and the neutrality of facilitation.

Further research on the role is necessary. Future efforts could be directed in two main directions. First, in real case studies and controlled lab experiments, additional NDA applications need to be performed and evaluated to understand the NDA role more in-depth and improve the materials developed to perform the role. This will help to comprehend to what extent contextual factors influence the outcomes obtained in this case study. Currently, the way information about the missing stakeholders is categorised in the role-plot is left to the judgment of facilitators preparing the role, and further standardisation might benefit the setting. Additionally, the role was tested only in one workshop of a GMB project; future efforts could also try to apply it in a series of sessions in the same project. More applications will also help to unveil particular nuances of the technique. For example, variations of the role could be explored, like leaving the task of information collection and compiling about the missing stakeholders to the facilitation team but then making some of the participants enact as NDAs (while it can bypass the loss of neutrality by the facilitators, it is unclear whether this could be feasible and without substantial negative impacts from the participants’ stances as it could be hard to play the NDA and also raise their personal opinions). In particular, a potential hypothesised positive side effect of the NDA was that the group discussion became more structured than it was in the previous session. Future research can assess explicitly if the NDA role contributes to improving the group’s actions structuration and modelling method appropriation (Scott Poole & DeSanctis, 1992; Tavella et al., 2021). Similarly, on paper, the NDA role might also be effective in addressing groupthink directly; thus, a specific analysis should be conducted to substantiate this view (e.g., study of the observable antecedents and consequences in a workshop with NDA players to observe variations in creativity within the discussion - see Esser & Lindoerfer, 1989). Moreover, in the aftermath of Covid-19, online workshops have recently increased, posing new challenges (Wilkerson et al., 2020; Zimmermann et al., 2021), which create opportunity to test the NDA role also in these contexts. As the second direction of research, the NDA assessment is still open to revisions in order to achieve comprehensive coverage of all aspects influencing the role performance and impact. For example, in future experiments, the collection and analysis of video recordings could supply additional critical information on the workshop dynamics in which the NDA role is performed (for instance, see the analyses of Franco and Greiffenhagen (2018) and Franco and Nielsen (2018)). Furthermore, the laboratory comparison of workshops with and without the use of the NDA role can provide additional meaningful data. Finally, to the best of our knowledge, an established method to analyse new roles in facilitated modelling workshops is not yet available. The experience described in this paper could be seen as a starting point in this direction.

Acknowledgements

We are very grateful to the BEIS team, who accepted to be challenged by this new role in their workshop. We are deeply indebted to Nicole Bernardi, Yekatherina Bobrova, Valerio Cappuccio, Luca Cunico and Juan Pablo Ríos Ocampo for all their precious help along the way. The authors also acknowledge Radboud University Nijmegen (The Netherlands), NOVA University Lisbon (Portugal), University of Bergen (Norway) and University of Palermo (Italy) for supporting this research within the European Master Programme in System Dynamics (EMSD).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the UK Government Department for Business, Energy and Industrial Strategy (BEIS) [no grant number], by a UCL Public Policy Small Grant [no grant number] and by the EPSRC Platform Grant on ‘Built Environment Systems Thinking’ [EP/P022405/1]. CENSE is financed by Fundação para a Ciência e Tecnologia, I.P., Portugal [UID/AMB/04085/2020].

Notes

1 Note that those approaches also tend to be labelled and referred to as group decision support approaches, problem structuring methods or soft OR (for an overview of the terminology, similarities and differences see, for instance, Andersen et al., 2007; Friend, 2006; Jackson, 2006).

2 In this article, for improving readability, we use the terms “homogeneity/heterogeneity”, “group homogeneity/heterogeneity” and “group mental model homogeneity/heterogeneity” interchangeably, although we strictly refer to the group mental model homogeneity/heterogeneity. Please note that the terms homogeneity/heterogeneity have been used in the literature referring to the notion of diversity within groups (background, culture, ethnicity, etc.), see Mayo et al. (2017) for a recent review. Although, group diversity and mental model homogeneity/heterogeneity can be related (i.e. a group with low diversity is expected to be more likely to have mental model homogeneity) the two do not perfectly overlap.

3 In dialectic inquiry, meeting participants are divided in two subgroups. A group develops written recommendations based on certain assumptions and the other takes the key assumptions of the first group, negates them and based on these counter assumptions develops alternative recommendations. Then, the two groups will present and debate their recommendations and assumptions.

4 Although not explicitly mentioned by Vennix (1996), his Devil’s Advocate script also has many similarities with the Strategic assumptions surfacing and testing method proposed by Mitroff and Emshoff (1979).

Annex A

Sample of the headings of the role-plot table with the information provided to the facilitators playing the New Devil’s Advocate

Table

ISSUE TO RAISE	STAKEHOLDER BEARER OF THAT ISSUE	HOW TO FRAME IT	POSSIBLE VARIABLES RELATED TO	SOURCE

Annex B

NEW DEVIL’s ADVOCATE ROLE—Introduction to the participants

The facilitators introduce themselves and their role (mainly their loss of neutrality) in the beginning of the workshop.

[Facilitator] “As you saw last time, facilitators don’t interfere with the content under discussion among the participants. What they can do is to ask question that challenge the participants. Today N and Y will leave their neutral facilitation role and act as one of the participants for most of the session. The reason why we made this choice is that we want to bring some issue into ‘the workshop room’ that might be raised by the stakeholders (that unfortunately can’t attend the workshop). The idea is to stress the model/map as much as possible in order to increase its truthfully with respect of reality. We expect this process to increase the quality of final output since it can lead to integrate into the model ‘things’ that were not considered before.”

“However, it is your right to stop it. It means that if you think that this approach is annoying or useless just let us know and say that you do not want that facilitator to be there.”

What does the New Devil’s Advocate (NDA) do? He/she acts as one of the participant, but a “special one.”

Annex C

New Devil’s Advocate: Facilitation Team Debrief

1) How do you think it went?

2) Why do you think went in that way?

3) What went well?

4) How did you feel during the session? Did you have any concern? If so, what were your concerns?

5) What was difficult and why?

6) What was your challenge/struggle?

7) What was easier than expected? Why do you think it was?

8) What was unexpected? Is there anything that surprised you? Before, during and after the workshop?

9) What would you have changed? What can be done differently? What could be improved?

10) Will you consider doing it again? Would you suggest it to someone in a similar position?

11) How difficult was to separate the role from the facilitation one? How difficult was to carry out the new role?

12) How important do you think the group attitude was for the good results of the DA?

13) What are the differences if you compare the session with your past experiences?

14) How did you find the guidelines and role-plot? Was it useful? What would you improve?

15) How would you assess the impact of the NDA?

16) If I am going to document this script, what are your suggestions?

Annex D

Correspondence between New Devil’s Advocate inputs and structure

Table

Structural input in the role-plot^a	Corresponding structure in the map^a
“Rebound effect”	“Rebound effect”
“Probability of unintended consequences”	“Unintended consequences”
“Probability of unintended consequences”→ “Negative word of mouth”; “Speed of negative word of mouth”	“Unintended consequences” → “Measure performance” → “Word of mouth effect”
“Quality of retrofit”→ “Probability of unintended consequences”	“Quality” → “Measure performance”
“Energy use” depends on the “EE level of the house” and on the “Compliance of householders behaviour with the new retrofit measure”	“Behavioural compliance” → “Measure performance”
“Fragmentation→ “Quality of retrofit”; “Skills of installer”→ “Quality of retrofit”	“Market fragmentation” → “Standards” → “Quality”; “SC skills” → “Quality”
“Quality of the retrofit”→ “Performance of the retrofit measure”; “Maintenance capacity”→ “Performance of the retrofit measure”; “Customer behaviour”→ ”Performance of the retrofit measure”	“Quality” → “Measure performance”; “Maintenance frequency” → “Measure performance”; “Behavioural compliance” → “Measure performance”;
“Industrial trust for making investments”	“Industry trust”
“Industry capacity”; “time to adjust capacity” (estimated in 1-2 years)	“Capability of response to demand”
“Installer skills”, “Time to gain skills, Skilled workers”	“SC skills”
“Unskilled workers” → “Skilled workers gap” → “Installers skills” → “Quality of retrofit”	“SC skills” → “Quality” → “Measure Performance”
“Quality”→”Credibility of the industry”→ “Demand”	“Quality” → “Measure performance” → “Uncertainty Costs” → “Perceived costs”
“Performance of the measures” → “Actual savings”	“Measure performance” → “Perceived monthly savings”
“Economic cycles”→ “Industry staff dismissed” → “Skills”	“Business Cycles”
“Uncertainty costs”	“Effectiveness uncertainty”
“Monitoring” → “Quality”	“Monitoring” → “Quality”
“Maintenance costs” → “Actual savings”	“Maintenance costs” → “Financial attractiveness”
“Disposable income” → “Priority given to retrofit”	“Households preference for energy efficiency”
“Marginal improvements” → “Technological level” → “Performance of the measure”	“Technological development of deep retrofit” → “Measure performance”
“Trusted information (credibility)”	“Tailored feedback to consumer on impact of their work”

^a The symbol ‘→’ stands for a causal connection.