Are “wrong” models useful? A qualitative study of discrete event simulation modeller stories

ABSTRACT

Little is known about models deemed ”wrong” by their modellers or clients in Operational Research (OR). This paper aims to improve our understanding of “wrong” Discrete Event Simulation (DES) models based on empirical evidence. We interview 22 modellers who describe projects where modelling did not go as expected and explain how they dealt with those situations. This resulted in 54 stories reporting that a model was identified ”wrong” either by the modeller, the client or both. We perform a qualitative text analysis of the stories to identify the factors that define a ”wrong” model as well as potential uses of ”wrong” models. The results show that some models even though considered ”wrong” may still be useful in practice and provide valuable insights to users and modellers. This study offers practical suggestions for users and modellers to consider when dealing with a model that is considered ”wrong”.

KEYWORDS:

• Discrete event simulation
• wrong models
• model use
• model characteristics
• stories

1. Introduction

Simulation is a well-established tool used in Operational Research (Tako et al., 2010; Kotiadis & Mingers, 2014; Robinson, 2014; Tako, 2015). Yet, the development of a model may be subject to constraints such as time or resource limitations. As a result, this could lead to a model considered “wrong“ by the users (clients/third parties), the modellers or both based on their perspective and understanding. This suggests that the concept of wrongness may be subjective. From our exploration of the literature, we note that there is limited exploration of “wrong“ models, even though there is a general belief that they can be useful (Hodges, 1991). Two main questions arise: what constitutes a “wrong” model and whether a ”wrong” model can still be useful. In this study we explore both questions. We note that in this study we consider ”wrong” models as any models that could subjectively be perceived wrong by anyone involved during their development and/or use.¹

To date there is no clear definition of what constitutes a “wrong“ model in the OR field. Some authors refer to the reasons (called here factors) that may lead to “wrong” models, while others refer to specific model characteristics related to wrongness. For example, Robinson (1999) describes experimentation as a source of model inaccuracy in simulation studies. Vennix (1999) and Salt (2008) refer to clients’ biases and beliefs affecting model creation. Gass (1983) and Kleijnen (1995) on the other hand, explain how to build valid models. We thus find a range of different opinions and viewpoints as to what could be a ”wrong” model.

Following, modellers or users could be faced with the following question: if a model is “wrong“, should it be discarded or can it still be of use? There is little evidence in the literature on possible uses of such models as papers usually showcase successful models (2006) with few references to failed attempts. The limited mentions found are mostly theoretical in nature. For example, Hodges (1991) and Hodges and Dewar (1992) explain uses for promotional or training purposes. An experimental study by A. Tako et al. (2020) approaches the topic of models that may be considered “too simple“ for use and provides evidence that simpler versions of a model could still be useful for users. This is a relevant study to this paper as sometimes models may be simplified to the extent that they are considered “wrong”. However, this is the only study that offers some empirical evidence on this topic, where a simple model could be considered as one type of a ”wrong” model. As such, we still need to understand the potential usefulness of ”wrong” models when encountered in practice.

The research presented here aims to shed further light to our understanding of “wrong“ simulation models in practice by categorising reasons of wrongness and potential uses of “wrong“ models based on empirical evidence. This topic has received little attention in the simulation field. For practical reasons we have chosen to focus our empirical work on Discrete Event Simulation (DES) which is the authors’ main field of research. This also ensures that all examined practical cases use the same type of Simulation. We note, however, that the same type of work could be applied for any other simulation approach, such as System Dynamics or Agent-Based Simulation. We interview DES modellers who discuss real-life examples of developing “wrong“ models and how these are treated by the modellers and users. Two objectives are set: to investigate factors of wrongness related to ”wrong” model characteristics, and to explore possible uses. We perform qualitative text analysis which shows evidence that some ”wrong” models may indeed be used in practice and even provide valuable insights. Modellers and practitioners could utilise the findings from our empirical analysis presented in this paper to identify cases where the use of ”wrong” models could be justified and hence ensure that modelling efforts are not completely lost.

The rest of the paper is structured as follows. Section 2 explores OR and Simulation literature referring to definitions and uses of “wrong” models in relation to the two study objectives. Next, the methodology is briefly explained and results of our analysis are presented. Section 5 discusses our findings against existing literature and puts forward potential reasons that such models could be useful. We conclude by considering limitations and future work.

2. Background on “wrong” models

2.1. Definition and possible uses of “wrong” models

A famous quote by George Box states that “all models are wrong, but some are useful“ (Box & Draper, 1987). This suggests that even bad models may offer some value to users. We provide an analysis of the concept of ”wrong” models.

“Wrong“ models are rarely presented (Rouwette et al., 2002), not allowing learning outcomes to derive from them (2006). Even when mentioned, different authors prefer different definitions. In OR, Hodges (1991) refers to “bad“ and “inadequate“ models. He distinguishes between models that can be validated but haven’t yet (“unvalidated“) and models that simply cannot be validated (“invalidated“). A similar categorisation is found between validatable models used for predictions, and “unvalidatable“ models for other non-predictive uses (Hodges & Dewar, 1992). A different clustering is made between “consolidative“ and “exploratory“ models where “consolidative“ is a correctly constructed and validated model while “exploratory“ is a model used for solutions with unavailable data (1993; Hooker, 2007). Authors in Simulation papers use terms like ”false” or ”incorrect” (Bankes, 1998; 2007) or simply ”wrong” (Hinkkanen et al., 1995) to denote a problematic model. Bankes (1998) offers further descriptions like “demonstrably incorrect”, “strongly predictive”, and “plausible” or “weakly predictive”. In another work (Phillips, 1984), it is hinted that a model is either “at fault” (“not sufficient to solve the problem”) or “requisite” (has shown new intuitions).

The use of multiple terms to describe “wrong” models suggests that there is no consolidated definition of “wrong” models nor a consistent approach to how they are dealt with in OR. The existing literature often uses subjective definitions of “wrong” models. In this study, we consider a model “wrong” if any problems can be identified with it. We note that our concept of “wrong” models encapsulates any term found in literature that refers to problems found with the models mentioned.

Nevertheless, it is suggested that even a “wrong” model may be useful. There is limited discussion on this with two main papers presenting possible uses (Hodges, 1991; Hodges & Dewar, 1992). Their suggested uses can be summarised as follows:

• promotion or selling,

• training users when accuracy is not required,

• storing information within a management system,

• creating new knowledge on results that don’t have to be exact, and

• implementing “wrong” models’ knowledge to test hypotheses when precision is not required.²

Other authors refer to model usefulness even if they do not specify whether/how “wrong“ models may offer benefits. For instance, Bankes (1998) states that “exploratory” and ”weakly predictive” models may assist in decision-making, Mens and Van Gorp (2006) refer to usefulness as a quality that helps understand systems and make decisions. This is also supported by Jessop (2002). 1999 comments that usefulness is found in how a model addresses expected problems.

Usefulness of “wrong” models is rarely considered. A study that provides relevant evidence on this topic shows that simpler models may be as useful as their more complex equivalents for decision-making (A. Tako et al., 2020), where simpler models can be considered a type of a “wrong” model. Yet, to the best of our knowledge, there has not been any empirical explorations about the use of “wrong” models in OR.

2.2. Characteristics of “wrong” models

Models in OR need to be rational to help with decision-making and learning (Ormerod, 2018). Since users learn by connecting model results with their understanding, this creates subjectivity for modellers’ and users’ approach to models (Pidd, 2009) and to outcomes’ interpretation (Williams, 1990).

This means that validity of a model – “the process of ensuring that the model is sufficiently accurate for the purpose at hand” (Robinson, 2014) – may be different to its credibility which concerns a model’s credential value from clients (Gass, 1983). This is further supported in recent literature through the exploration of the idea of trust (Harper et al., 2021; Yilmaz & Liu, 2020). Yilmaz and Liu (2020) posit that model credibility is not only based on numerical accuracy but also on the extent that model users place trust in models and find them believable. Hence, model credibility depends on the cognitive aims and interests of its intended users (Gelfert, 2019). Harper et al. (2021) explore trust as an integral part of all model development stages between the stakeholders, modeller and model, highlighting a fundamental need for the clients to acquire confidence from the insights gained from this process. Robinson (2002a) makes a similar observation about the importance of process, as well as content, in forming opinions about the credibility of simulation study.

Following the above, the interpretation of a model being considered “wrong“ can be subjective. We use the framework developed by Robinson (2014) to help us identify the characteristics of a “wrong“ model based on two main aspects: the process followed in developing the model and the content of a model. The characteristics are defined based on the authors’ interpretation of the references found in the existing literature. Hence, we identify two main categories of factors that lead to “wrong” models: process-related factors which refer to the underlying reasons a model may be ”wrongly” developed during a modelling study (model development process), and content-related factors which are related to elements of the model that could be considered ”wrong”. In this research, factors are split under distinctive common model traits called model characteristics. These characteristics were created and assigned to either category based on our analysis and understanding of the existing literature which relates to modelling process and implementation. We provide at the footnote of Table 1 a brief operational definition for each of these characteristics.

Table 1. Categorisation of factors of wrongness from the literature analysis.

Model characteristic	Category	Factors and references from OR and Simulation
Validation	Process-related	1. Infeasible or very difficult (Hodges, 1991; Hodges & Dewar, 1992) 2. Improper validation (Gass, 1983; Sargent, 1991), experimentations (Robinson, 1999) 3. Lack of validation (Kleijnen, 1995; Salt, 2008)
Development aspects	Process-related	1. Limitations/restrictions of time/funding (Balci et al., 2002) 2. Short-term focus (Chussil, 2005) 3. Technological constraints (Bankes, 1998; 2015; Kolkman et al., 2016)
Decision-making related to model stakeholders	Process-related	1. Clients Empirical criteria, bad assumptions, personal analysis, limited experience/training with models (Chussil, 2005; 2007; Kolkman et al., 2016) Personal reasons (distorted memory, wishful thinking, bias, professional background/position, methodolatry, lack of willingness to change, belief reality can be simulated perfectly; Vennix, 1999; Chussil, 2005; Salt, 2008) Unquestionable belief in model (Salt, 2008) Wrong comparison with system (Brooks & Tobias, 1996) Selection of wrong alternatives (2007) 2. Modellers Trying to model what is easy to model (Bankes, 1998), lack of skills (Robinson & Pidd, 1998) Lack of objectivity (1993) 3. Group dynamics Different interests (Hodges, 1991) Absence of competition, lack of customer inclusion (Chussil, 2005) Messy problems (Vennix, 1999)
Technical aspects	Process-related	1. Data issues Contradiction of existing data (Hodges, 1991) Sample data and data fit (2007) 2. Component issues Bad/missing model components (Hodges, 1991; Pidd, 2004) Component performance failure (Brooks & Tobias, 1996)
Level of model detail	Content-related	1. Too simple models/excessive use of simplification (Prieto et al., 2012; 2015) 2. Too complex models/lack of adequate simplification (1993; Goldberg et al., 1990; Brooks & Tobias, 1996; Salt, 2008)
Model results	Content-related	Values or results different from expectations/reality (Goldberg et al., 1990; Hodges & Dewar, 1992; Brooks & Tobias, 1996)
Structural or application-related model aspects	Content-related	1. Structure Model structure (Chussil, 2005; Bankes, 1998) Conceptual model not representative (Robinson, 1999) 2. Application Translation of problem into wrong model (Lombaers, 1979) Application to wrong problem (Law & McComas, 1991; Kotiadis et al., 2014)

The characteristics are defined as follows: Validation refers to any steps/process followed for a model to abide by its users’ requirements. Development aspects refer to topics that restrict or limit the development of a model. Decision-making related to model stakeholders regards any decisions connected to involved parties (clients, modellers, third parties) during model creation. Technical aspects concern considerations of data and components during development. Level of model detail refers to simplification issues of a model. Model results refer to problems of a model’s outcomes. The last category is putting together any aspects of a model’s structure and whether the model is applied.

2.2.1. Process-related characteristics

The first category regards process-related factors under four model characteristics: validation, model development, decision-making related to model stakeholders, and technical aspects. Each one is next explained in more detail.

Validation is a very important part of modelling to ensure that a model corresponds to its purpose (Robinson, 2014). If validation is not proper, the model may be considered “wrong“. Within OR, Hodges (1991) and Hodges and Dewar (1992) separate models based on validation as mentioned above – if validation is infeasible or difficult to occur, the model may be considered “wrong“. In Simulation, improper validation is another concern: if validation is done “wrongly“ it could lead to a ”wrong” model (Sargent, 1991). This is related to experimentation as a source for model inaccuracy as it may produce deductions based on insufficient model replications and result analysis (Robinson, 1999). Kleijnen (1995) proposes that lack of validation may offer wrong decisions. This can be supported by the ”black-box mistake” idea where a model’s outcomes are not questioned because they are considered inaccessible (Salt, 2008) leading to a ”wrong” model being developed. Still, proper validation is not the only adequate premise to avoid model wrongness (Kleijnen, 1995) as we suggested earlier when mentioning credibility.

Model development is a set of reasons related to factors of limitation or restriction like time or funding (Balci et al., 2002). Indeed, if resources are limited or run out during a project, a model may be developed hastily or not even fully. Short-term focus is also included (Chussil, 2005) as a model could become obsolete after some time if during development we do not consider its use for the future. Certain more theoretical factors like unmatched expectations of technology that OR and Simulation didn’t meet (Bankes, 1998; Kolkman et al., 2016; 2015) are also encountered.

Decision-making related to model stakeholders is commonly connected to model wrongness. It regards decisions of those involved in model development and can be attributed to clients, modellers or interactions (dynamics) between them. In view of clients, a person may call a model “wrong“ due to empirical criteria (wishful thinking, bias, overconfidence) or bad assumptions (Chussil, 2005). In Simulation, we find analysis by clients (2007), unquestionable belief in a model (Salt, 2008), wrong model selection instead of alternative models that may have been better (2007) or wrong comparison with the system (Brooks & Tobias, 1996). Additionally, training and experience of users (Kolkman et al., 2016), methodolatry, lack of willingness to change a model, and belief that reality can be simulated perfectly (Salt, 2008) are cited. Specifically in System Dynamics we find distorted memory, professional background and position (Vennix, 1999). Equally, a model may also be developed ”wrongly” due to modellers’ actions. Factors include lack of objectivity from modellers (1993), modelling only the easy parts of a system (Bankes, 1998), and even possible lack of skill (Robinson & Pidd, 1998). Lastly, group dynamics can also be connected to wrongness: different interests of clients and modellers (Hodges, 1991), lack of customer inclusion or absence of competition (Chussil, 2005), and team member communication (Vennix, 1999).

Technical aspects is the last characteristic here. Having data problems may lead to a “wrong” model as its development basis would be inaccurate. This is referenced both in OR as contradiction of existing data leading to bad outcomes (Hodges, 1991) and in Simulation as problems arising from sampling data (2007). Also, components’ performance and technology appear in literature. In OR, a model may have good partial components but considered bad in its totality (Hodges, 1991) and in Simulation a model may be lacking a component or a variable (Pidd, 2004) or its components’ performance may fail (Brooks & Tobias, 1996).

2.2.2. Content-related characteristics

A similar analysis is performed for factors related to the content of the model.

Level of model detail regards factors in view of a model’s simplification. Simple models may be considered “wrong“ as they may not represent the initial problem adequately (Brooks & Tobias, 1996) due to “far abstraction“ (2015). This is the first factor here, addressing that a model may be considered “wrong“ because it is viewed as too simple. In OR, we find that abstraction is needed for a model but it may create inaccuracies (Prieto et al., 2012) while in Simulation even a simple model could offer solutions but it could also imply problems (Robinson 2015). On the other hand, a model may be considered ”wrong” because it is viewed as too complex as it may include unnecessary details (2017). In OR, 1993 suggests that large-size models could be a cause for concern and Goldberg et al. (1990) point that a high level of detail may not actually be required. In Simulation, creation of a more detailed model may be limited by resources (Brooks & Tobias, 1996) with relevant factors of wrongness here referencing beliefs that more content leads to better outcomes and connecting good models together also returns a good model (”trifle-worshipping” and ”connectionism”; Salt, 2008).

Model results is another model characteristic that is found in cases where wrongness is discussed. If a model is providing wrong outputs, it may be regarded as “wrong“. For instance in OR we find that comparisons with specific values can denote a “wrong” model (Goldberg et al., 1990; Hodges & Dewar, 1992), while in Simulation inaccurate results and values may also be used to describe a model as ”wrong” (Brooks & Tobias, 1996).

Structural or application-related model aspects categorises factors related to issues with the structure and application of a “wrong“ model together. This is because issues with a model’s structure can affect whether the model is applied. In view of structure, Chussil (2005) proposes that it can indeed result in an OR model being considered “wrong“ due to the need for model calculations to be based on well-defined infrastructures. In simulation, Robinson (1999) refers to issues that may be invoked because of bad conceptualisation, while Bankes (1998) supports that if an internal model structure is different to that of a known system, it could lead to the model being considered “wrong”. For model application, reasons identified are less explicit. We find reports that the OR problem may have been translated into the ”wrong” model (Lombaers, 1979) and in DES that a wrong problem may have been solved (Kotiadis et al., 2014; Law & McComas, 1991). This suggests the need for an appropriate interpretation of a problem to avoid subsequently building a ”wrong” model.

Table 1 summarises the factors of wrongness as categorised under model characteristics.

2.3. The need for exploring “wrong” models

Our analysis of existing literature shows that there is no single definition on the concept of wrongness and that many reasons can contribute to a model being considered “wrong”.

It is our belief that wrongness is the result of subjective decision-making based on how modellers and users perceive the modelling process and model content. This creates the question of what is considered a “wrong“ model. We also note that authors rarely mention a “wrong“ model and that there is a limited discussion on the reasons that lead to a “wrong” model. The use of ”wrong” models is discussed at a theoretical level without practical guidance. There is hence a need for practical guidance in assessing models and whether a ”wrong” model can be useful and how.

This research is a first attempt to address this gap, by categorising factors of wrongness and taking an empirical view in identifying such factors in practice. Furthermore, we aim to provide evidence about the extent to which “wrong” DES models are used in practice and their potential uses.

3. Methodology

3.1. Objectives

The main aim of this paper is to define in practice what is considered a “wrong” model and whether such a model can still be useful. Based on the identified gap (Section 2.3), we set out two main objectives as follows:

• Objective 1 (O1): to provide a definition of “wrong“ models. O1 addresses the question ”what is considered a ‘wrong’ model”. The research is expected to denote specific factors of wrongness categorised under model characteristics. These factors will be mapped against those found in the literature to establish whether they match modellers’ experience in practice. Factors are expected to be corroborated and additions to be made.

• Objective 2 (O2): to explore uses of “wrong“ models. O2 addresses whether a ‘wrong’ model can still be useful and identify potential uses“. The first part is establishing whether a “wrong” model may actually be used in practice. If such a model can be considered useful, its uses are further explored. It is expected to find evidence on possible use as well as actual uses of ”wrong” models to accept or refute usefulness for ”wrong” models.

3.2. Choice of research method and data analysis strategy

To select the best research method and way to analyse the collected data, certain requirements are considered.

The two objectives are defined by the need for empirical data to show what consists a “wrong“ model and whether it can be of use. To gather enough information, a method of a qualitative nature would be required, as it would provide detailed information but also allow us to compare outcomes with previous literature suggestions. Interviews are deemed as the most suitable method to gather sufficient data to address the objectives of the study, considering also time restrictions for this research. We conduct semi-structured interviews and ask participants to describe stories reporting examples where DES models deemed ”wrong” were developed, based on their modelling experience. Other researchers have used interviews to research DES modelling practices. Tako (2015) interviewed expert modellers by asking them to speak aloud when they develop DES models. Gogi (2016) interviewed DES modellers to convey stories to explore insight generation from using simulation models in Operational Research. Here we explore factors of wrongness in DES models as well as whether and how ”wrong” models may have been used. We solely focus on DES models, as already explained in Section 1, primarily for practical reasons and due to the lack of previous studies on this topic.

All collected data was transcribed and anonymised. Thematic analysis was undertaken looking for repetitive themes with relevant information to the objectives of this research. (Bryman, 2012) content analysis. Interpretation of meaning from text data (Bazeley, 2013) was undertaken through rigorous and repetitive exploration of the interview transcripts. The content was grouped into relevant themes and then compared with topics found in the literature. Further details of the qualitative research methodology and data analysis strategy are provided in Appendix A.

3.3. Study participants and interview process

We carry out interviews with experienced DES modellers to explore real-life cases of unsuccessful DES modelling projects. We contacted a number of UK-based simulation companies who helped us to identify potential interviewees. The topic was met with interest, resulting to a total of 22 participants from 6 different companies from the private sector. A total of 54 stories/models were discussed that addressed models of various content, all of which were found relevant to the objectives around “wrong“ models as they contained either issues around modelling or the view that a model was considered ”wrong”. Table 2 provides a summary of the interviewees in each company, their experience and number of stories shared.

Table 2. Information of the interviewed participants.

Company	Type/Sector	Interviewees	Experience	Stories shared
A	Consulting	6	5 J/I, 1 S	21
B	Manufacturing	7	5 J/I, 2 S	16
C	Consulting	6	1 J/I, 5 S	11
D	Consulting	1	1 S	2
E	Services & consulting	1	1 S	2
F	Consulting	1	1 S	2

All companies focused on consulting but for one whose focus was manufacturing.

Experience is deduced by the researchers based on participants’ background. Senior (S) is considered a participant with at least 10 years of experience, regardless of position as opposed to Junior (J) or Intermediate (I).

Further details about the profile of the participants’ experience, clients’ sectors and problem types they were facing are provided in Appendix B.

Regarding the process, an ethic clearance was performed (Brinkmann & Kvale, 2015) to inform every interviewee around confidentiality. Each interview was then held with one of this paper’s researchers and the DES modeller, lasting between 40 and 60 minutes. After exploring each interviewee’s background, they were asked to provide details of projects that did not go well. Questions were used to obtain information for each project on model size, industry of application and project aims. Questions on wrongness followed on the problems with each model and who identified them. These would offer information for O1 on factors related to wrongness in practice. Further questions were asked to clarify model acceptance and use by clients, how that happened, etc. This set of questions aimed to understand usefulness of “wrong“ models, addressing O2 on using “wrong” models. It is noted that due to the nature of semi-structured interviews, specification of questions and additional questions were asked if required to conform by the peculiarities of each situation. The generated data here is primary data collected for this research on the use of ”wrong models” while the timeline for the interviews was approximately 3 months, taking place between November 2018 and January 2019.

4. Results from the interviews

4.1. Defining a “wrong” model

The analysis uses the two categories of factors defining “wrong” models as presented in Section 2. It is noted that although all stories came from the modellers’ descriptions, some of them referred to clients perceptions of the modelling problems (as narrated by the modellers). This distinction in perception is displayed in Table 3. All factors are identified based on the researchers’ interpretation of the interviewees’ narrations.

Table 3. Results on factors of wrongness from the interviews per perspective.

Characteristics		Unique references (stories) per category
		Modellers	Clients	Total
Process-related	Decision-making related to participants	23	8	30
	Technical aspects	18	5	20
	Development aspects	9	0	9
	Validation	8	0	8
Content-related	Model results	18	21	27
	Structural or application-related aspects	9	8	17
	Level of model detail	12	8	13
	Usability	0	2	2

Characteristics are referenced only when connected to a specific story as mentioned throughout Sections 4.1.1 and 4.1.2. The column “Total” does not summarise the two other columns to avoid counting a story twice. Instead, a characteristic is only counted once even if it is referenced both by modellers and clients.

4.1.1. Factors of wrongness under process-related characteristics

The same four model characteristics that contain factors of wrongness for process were encountered at various stories.

The most common set of problems reported in 30 of those stories regards Decision-making related to participants. More specifically, factors attributed to the clients that led to a model being considered “wrong“ concern guesses the clients may have made during development or even lack of planning for creating a model, contradicting decisions, and lack of training or understanding of DES. Other problems include evasion of liability or lack of participation when developing a model, dismissal of certain results due to bias and persistence on unhelpful details. Additionally, disbelief towards a model or mismatch of questions with the agreed scope are considered as possible issues towards building a model. Bad communication or details not communicated, lack of strict specifications, and loss of project control are also referenced. There are also certain factors that are attributed to how the modellers have approached a model that was considered ”wrong” and include lack of experience or expertise, bad presentation of model or outcomes, bad communication or misinterpretation of data, guesses or assumptions alongside any uncertainty on clients’ operations, decision to use too many or unnecessary details, and unrealistic or unsuitable promises made to the client. Group dynamics are also discussed, with the main factors being conflicts of interest, internal or external politics causing discrepancies, insistence for simulation use, lack of collaboration or communication, inclusion or interest for a project, departure of important project members, and different processes followed.

The second most recurring set of factors regards data and technical topics (Technical aspects) connected with 20 stories. In view of data concerns, models are developed “wrong“ due to lack of data or because of the involvement of contradicting, messy, ambiguous or bad data. Other stories reference extreme availability of data causing problems, lack of trust in data, and different data specifications within the same company resulting to discrepancies in analysis. Accordingly, modellers mention issues with components or coding that could have been responsible for ”wrong” models. These regard use of wrong distributions, calculations, components or variables, incomplete models with flaws on mapping/layout and missing or wrong logic that were still put to use, wrong implementation of variables or numbers, model speed concerns, and bad rules or priorities causing performance failure.

The next most referenced set of factors is found under Development aspects. There are nine unique references of such factors that could be connected to developing a model “wrong”. Concerns could be summarised under two main factors of wrongness: time limitations to complete a model and the possible frustration this causes, and, short-term focus of model design including incorrect or continuously changing scope as the reasons for developing a bad model.

Lastly, Validation problems are mentioned in eight stories. The factors regard infeasible, impossible or very difficult validation, improper/bad or failed validation, and lack of validation. Interestingly enough, validation factors are always mentioned alongside other factors of wrongness as the cause for developing a “wrong” model and all of them come from the modellers’ side – meaning that their clients did not complain about this topic.

4.1.2. Factors of wrongness under content-related characteristics

All three model characteristics for content were mentioned at various stories, also encountering factors that could be put under a fourth characteristic.

Starting again with the most commonly referenced factors, we find that Model results are considered crucial for finding a model good or bad. The actual factors mentioned here in 27 stories concern values, results or visuals different from the clients’ reality or expectations, results that are doubted wrongly by the clients, and lack of outputs from the model.

The second set of factors with the most references regards the Structural or application-related model aspects characteristic, mentioned in 17 stories. In view of structure, factors include lack of representativeness for model structure, incomplete models, and models wrongly altered by the client. Application issues concern when the wrong question is addressed, not delivering what is promised due to scope change, and when clients’ demands or internal disagreements are involved.

Level of model detail considers the complexity represented in the model. It is noted that due to confidentiality purposes, we were not able to access the models accounted for in the stories. Instead, the analysis is based on the modeller’s interpretation of a too simple or too complex model, if their level of model detail was reported to be a concern. A total of 13 unique references are encountered: models could be “wrong“ due to being either too simple or too complex, while another factor of wrongness suggests that simplification may not be correct. The finding that both too simple and too complex models may be considered ”wrong” is very interesting especially when considering that opinions could be different even for the same model. Indeed in two cases of our study, a relatively complex model for the modeller is considered as too simple by the clients for their needs.

The analysis of the interviews also brought to light another set of factors that literature did not highlight under the characteristic of Usability. There are only two references in stories, but these are very elaborative to consider it separately. Factors here refer to difficulty to use a model, lack of interest to use a model, different format of results than required, and DES not preferred as a method by clients.

4.1.3. Summary for O1

We found a total of eight different model characteristics populated with factors of wrongness. In various cases, more than one factor was referenced, but correlations exceed our current analysis. The distinction of whether factors are coming from modellers and clients is presented in Table 3.

Some interesting points arise: Validation does not seem of high concern for clients, as opposed to Usability that modellers do not reference, while Results are equally split – meaning it is a very important characteristic to account for both sides.

4.2. Exploring uses of “wrong” models

The analysis of the stories revealed that “wrong“ models could actually be used, offering benefits to their users. Additional results on why a ”wrong” model was decided to be used and how it was implemented are presented in Appendix C.

4.2.1. Using a “wrong” model in practice

We first explore the extent to which “wrong” models are used in practice. The following diagram (Figure 1) shows the process followed.

Figure 1. Analysis followed for the 54 stories to consider uses.

All 54 stories contained a “wrong“ model. As such, we analysed iteratively what modellers mentioned that happened in each case. We use the following operational definitions for our categorisation: If a model was clearly put to some use or meant to be put to some use by the clients regardless of who considered it ”wrong”, it was noted as used. If there was a clear reference that clients would not use the model, it was considered as not used. Stories without information on the topic were categorised as unclear as their use could not be communicated either because the modeller did not follow up to find out, or, because the client did not offer specifications due to confidentiality. Table 4 summarises the outcomes.

Table 4. Results on interviews regarding whether models were used.

Category	Number of models
Used	25 (46.3%)
Not used	24 (44.4%)
Unclear	5 (9.3%)
Total	54

We notice that almost half of the stories regarded a “wrong” model that was considered used.

4.2.2. Uses of “wrong” models in practice

We now focus on the actual uses of those models to provide an understanding of their role in practice. Six types of uses are identified by the analysis, presented in Table 5.

Table 5. Results of interviews regarding uses of models.

Types of uses	Total	Cases found
Decisions or actions	17 (68%)	-Resource or production changes like introduction of new lines, addition of equipment to assist production, factory redesign, storage allocation, and time-based vehicle assignments -Cases where a “wrong” model is used for exploring options in deciding production for a new facility, finding bottlenecks, suggesting how to set up new lines, determining why queues are jammed, and testing new facility throughputs -In 2 cases, “wrong” models are used as tools for scheduling to support ordering equipment, predict storage and time needs, and to support investments
Hypothesis/scenario testing	8 (32%)	-Exploration of system times, testing of new equipment impact, prediction of newly invested Emergency Departments, and measuring of efficiency/costs due to regulations and work load changes -In 2 cases “wrong” models are considered by clients as the experimental basis to get a better understanding of their systems’ complexities
Surface issues	7 (28%)	-“Wrong” models that reveal operational dysfunctions with managing personnel, how decisions are really made by personnel, and ambiguous records by clients -Model use to find issues between involved parties like proving a third party wrong or showing bad communication between departments and discrepancies between team
Better understanding or thinking development	3 (12%)	-A simple model showing an overview of expense flows that offered insights to the clients -A model that showed the actual problem to be different than the initial scope -A model that made clients realise the actual constraints they should address
Promote/communicate selling	2 (8%)	-Use of a “wrong“ model to assist ”commercially” for bidding as a sales tool (which was different than its original design for scheduling) -Use of a “wrong” model as evidence to prove a client’s interest for a bidding opportunity.
Store or monitoring	1 (4%)	- A “wrong” model used by an analyst who was to leave the project thus needed to produce certain results for peers who couldn’t use the model.

A model may abide by more than one categories. Each percentage is out of 25 stories.

The cases of the last column are the actual examples found in the explored stories.

The majority of stories (17/25) shows that “wrong“ models were used for decision-making (case descriptions in Table 5). The second type of use found in eight stories regards use of “wrong“ models for hypothesis or scenario testing. Some overlapping with the first category is noted. The third encountered type in seven stories regards use of “wrong“ models to help surface issues. The fourth encountered type in three stories regards use of ”wrong” models for better understanding of the real world or to develop clients’ thinking of a problem. This use is always accompanied by other uses. The fifth encountered type in two stories refers to use of ”wrong” models for promoting or communicating a selling idea. Lastly, the sixth encountered type in one story addresses use of ”wrong” models for storing information or monitoring/controlling processes. Table 5 summarises types and cases.

4.2.3. Summary for O2

We found that in almost half of the interviews (25/54) a “wrong“ model was utilised or was expected to have been utilised. Their actual uses were categorised over 6 overlapping types based on the literature analysis. These are all actual uses of ”wrong” models in practice. Appendix C holds some additional findings on why those models were used and how they were implemented.

5. Discussion of findings

This novel study investigated the unexplored topic of “wrong” models. We next discuss research contributions and the findings of our interviews against existing literature, highlighting additional insights gained and considerations on how different domains of application can benefit from our analysis.

5.1. Contribution of research

This research has for the first time considered why and how “wrong“ models are used, exploring existing literature and DES practice to find factors that lead to the development of “wrong“ models and identify their potential uses. The main contributions regard categorisation and analysis on wrongness and usefulness both for theory and practice. From a theoretical perspective, this study offers a list of factors leading to “wrong” models and their uses. From a practical perspective the described cases offer empirical evidence to support clients and modellers when faced with a ”wrong” model. The very process of developing a model can offer insights even if the outcome may not be as expected. Furthermore, the list of factors can be considered in practice as a topic of discussion when exploring what may go or have gone wrong during development while the uses can be considered as alternative options for finding benefits even from ”wrong” models.

5.2. Factors of wrongness

To address factors of wrongness in practice, the first objective explored them through real-life cases.

We first explored factors leading to the development of a “wrong” model. We identified characteristics for Validation, Model Development, Decision-making related to model stakeholders, and Technical aspects, matching interview findings to literature. More specifically, when discussing Validation, infeasible, impossible or difficult to be carried out validation align with Hodges (1991) and Hodges and Dewar (1992). Most Model Development factors are encountered in practice, meaning that planning and proper scope are elementary in practice, while Decision-making is again split into issues related to clients, to modellers, and to group dynamics. On the contrary, some factors were not repeated from Table 1. For example, Model Development funding restrictions and technological constraints (Bankes, 1998; 2015) are not explicitly mentioned in stories. For Decision-making factors attributed to modellers, we notice a higher variety in interviews than literature. Also, there were some significant additions like Technical aspects factors on speed of models and problems with coding. We noted that Validation concerns are reported only from modellers which could suggest that clients may not be fully acquainted with the concept to reference it as an issue.

We then explored characteristics related to model content. These are the Level of model detail, Model results, Structural or application-related model aspects, and Usability. It is again noted that most findings were matched to literature. For example, for the Level of model detail, the interviews supported that both too simple and too complex models could be considered “wrong”, while for Model results, numerical citations within literature (e.g Goldberg et al., 1990) were corroborated. Still, there were factors from literature not fully matched as for instance conceptual concerns under Structural or application-related aspects probably due to firm scoping techniques that companies adapt. On the contrary, additions did not only regard factors but also a new category, Usability, which was suggested as a characteristic related to how easy a model and its components are considered to use.

To summarise, the study discusses a variety of characteristics of “wrong“ models. Their categorisation was possible under the same two categories of Section 2, confirming and expanding most factors, and adding another characteristic. The most commonly cited factors for modellers and clients were found under the same categories (Decision-making and Model results). Still, the findings show that there can be a difference of opinions between modellers and clients on whether and why a model is ”wrong” with complex dynamics on decision-making. Especially in view of level of model detail, simple and complex models may even reflect opposite opinions.

5.3. Usefulness and potential uses of “wrong” models

Following, our study considers uses of “wrong” models from a practical perspective.

In total, 25 of the 54 examined models were found to have been used in at least some way. This derived from different interviewees and companies – which shows that a variety of models may be considered as inadequate but still useful. This also supports the assumption that using a “wrong” model does not imply that a modeller would willingly offer a bad model to their clients – instead clients may even insist on using a model because they find it credible for their needs or because a modeller showed possible benefits attributed to this model’s use (Appendix C).

Six types of uses were found in practice. The most common use (17/25 stories) regarded Decisions or actions that according to modellers offered benefits to clients. This use resembles the exploitation of “wrong“ models to create knowledge on results that don’t have to be exact (Hodges, 1991; Hodges & Dewar, 1992). In other words, ”wrong” models seem possible to be considered for decisions if precision is not the main concern.

Next, use for Hypothesis or scenario testing (8/25 stories) was found which is also identified in literature on how “wrong“ models can implement acquired knowledge for hypothesis generation (Hodges, 1991; Hodges & Dewar, 1992). This shows that ”wrong” models may be useful in testing system interactions even if their numbers are inaccurate.

The next most encountered use (7/25 stories) is deemed to Surface problems within an organisation. This use describes that models may offer insights on operational or decision discrepancies for a company which could be connected to using Soft OR for initiating discussions (Mingers, 2003) and using simulation to create conversations (Robinson et al., 2014).

Another use (three stories) addresses “wrong” models utilisation for Better understanding the clients’ systems and developing their thinking process. This is a complementary use to other types to create knowledge on non-precise results (Hodges, 1991; Hodges & Dewar, 1992).

A fifth use (two stories) concerns Promotion or communication on a commercial level, encountering models for sales tool and as evidence of interest when bidding. This use is a direct reference to literature (Hodges, 1991; Hodges & Dewar, 1992), showing a business-engaging use of “wrong” models in practice.

Lastly, a sixth type is connected to Store and monitor results. This matches storage of information as part of a management system (Hodges, 1991; Hodges & Dewar, 1992) and suggests that “wrong” models do not need to directly be implemented into actions to offer benefits but instead be saved for later use.

Although the absence of practical studies on uses does not allow comparison with past findings, certain deductions can still be reached. 2007 mention how “false“ models may occasionally offer a better fit than “true” models which seems compatible to the current study’s outcomes, while 1993, 1998) mentions ”exploratory” models as those used for investigating system assumptions and hypotheses regardless of validity. These also lead towards a more complex discussion for future research on model qualities and whether validity should be the main attribute for a model or if usefulness could be adequate. A first consideration may still be reached: the modelling process needs to focus on pragmatic topics and include contents in a model that would deem usefulness to its users – ideally corroborated through model validity. If validity is not achievable, a model could then be considered for uses as the ones presented here.

5.4. “Wrong” models in different domains of application

A last consideration here addresses the potential applicability of “wrong” model uses across different domains and whether these differ.

This study is the first to tackle the idea that certain models may be “wrong“ but still useful. As such it has not focused on separating the results by domain of application. Still, we offer a first indication of whether differences are encountered based on our understanding of the collected data. We notice from Tables 2 and 5 that the modellers worked in a range of sectors. We can see from our data that “wrong” models have been used for decisions in manufacturing, and for capacity planning and resource scheduling in services. Similarly, we find ”wrong” models applied to scenario testing in Emergency Departments of clinics.

As such, we have no specific evidence to refute the possibility that a “wrong“ model may be useful in any application domain. This aligns with previous work that explored how simulation could be used in different domains (Robinson, 2002b). It appears that the ”mode of practice” employed is more important than the domain of application. For instance, a model used in facilitated workshops (Kotiadis & Tako, 2018; Robinson et al., 2014; Tako & Kotiadis, 2015) requires a lower level of fidelity than a model focused on providing an accurate representation of the real world.

The above also conforms with our previous work which found that simpler simulation models offer similar levels of learning as more complex ones (A. Tako et al., 2020). We consider that there is a connection between simple and “wrong“ models in that a simple model could be simplified to such an extent that it could be considered “wrong”. This approach to the definition of wrongness fits with the subjectivity of identifying wrongness in the model. Indeed, our literature analysis has identified the level of detail (Table 1) as one of the reasons for ”wrong” models (Section 4.1.2). Yet future research could identify potential thresholds of simplification to define model boundaries.

6. Conclusion

This paper is the first empirical study to explore the concept of “wrong” models and their uses.

We conducted interviews with DES modellers offering real-life stories where modelling efforts did not go as planned. This does not mean that “wrong“ models are intentionally developed but rather highlights that the concept of ”wrong” models is subjective and such models may still have some use.

We analysed literature as well as a total of 54 stories on “wrong“ models and defined the main factors leading to the development of such models. We also investigated uses of ”wrong” models in practice, and found that they can be fruitfully used. We recommend that discussions are held between model developers and their users on how best to take advantage of such models.

We also identified concepts that related to “wrong“ models. The authors acknowledge that there is no objective definition of a ”wrong” model as this depends on the interpretation of the model’s validity from the modellers’ or client’s point of view. Our analysis shows that these perceptions may not match and hence highlights the need for reaching a common understanding between clients and modellers. The study also identified the concept of having a useful model, in that a model that offers value to the clients is not the same as that of a valid model.

A number of limitations apply to the study. The subjective categorisation of factors of wrongness and model characteristics is based on the researchers’ viewpoint. Certain characteristics could be considered as more intersectional and under different provided operational definitions. Other researchers could have given different interpretations to the topics and content accounted in the modelling stories and interviews. Also, all stories come from the description of the modellers, meaning that clients’ reactions and inputs were implied based on the modeller’s recollections. The choice of interviewees was subjective and based on availability. A different selection could have led to different outcomes. It is noted that the perceived complexity of each model was subjective and based on each modeller’s description and the researchers’ understanding, without the option for a more objective or quantified interpretation. Due to confidentiality reasons, it was not possible for the researchers to access or analyse these models and their content.

Lastly, the study has identified areas for further analysis. These include considering more specific factors that may have an adverse effect on the perception of “wrong” models and the possibility that different “modes of practice” and domains of application have implications on model “wrongness”. The authors aim to revisit the interviews to explore learning and insights gained from “wrong” models as well as to expand the research to address simplification and usefulness in combination with model wrongness.

Ethical approval

This study has been logged with the School of Business and Economics, Loughborough University, and received ethical clearance on September 20 2018.

Acknowledgments

The authors would like to acknowledge the financial support received from Simul8 Corporation to administer the interviews for this research. Also, we would like to thank the simulation modellers that participated in the interviews and for their input towards this research.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/17477778.2022.2108736

Notes

1. This research is an extension to the study presented in the Simulation Workshop 2021 (Tsioptsias et al., 2021).

2. This last use is known as the “a fortiori” argument (Hodges, 1991) and within OR it could refer to a model being used under a new setting to get more reliable insights than those of its initially expected use.