Are “wrong” models useful? A qualitative study of discrete event simulation modeller stories

Figures & data

Table 1. Categorisation of factors of wrongness from the literature analysis.

Model characteristic	Category	Factors and references from OR and Simulation
Validation	Process-related	1. Infeasible or very difficult (Hodges, 1991; Hodges & Dewar, 1992) 2. Improper validation (Gass, 1983; Sargent, 1991), experimentations (Robinson, 1999) 3. Lack of validation (Kleijnen, 1995; Salt, 2008)
Development aspects	Process-related	1. Limitations/restrictions of time/funding (Balci et al., 2002) 2. Short-term focus (Chussil, 2005) 3. Technological constraints (Bankes, 1998; 2015; Kolkman et al., 2016)
Decision-making related to model stakeholders	Process-related	1. Clients Empirical criteria, bad assumptions, personal analysis, limited experience/training with models (Chussil, 2005; 2007; Kolkman et al., 2016) Personal reasons (distorted memory, wishful thinking, bias, professional background/position, methodolatry, lack of willingness to change, belief reality can be simulated perfectly; Vennix, 1999; Chussil, 2005; Salt, 2008) Unquestionable belief in model (Salt, 2008) Wrong comparison with system (Brooks & Tobias, 1996) Selection of wrong alternatives (2007) 2. Modellers Trying to model what is easy to model (Bankes, 1998), lack of skills (Robinson & Pidd, 1998) Lack of objectivity (1993) 3. Group dynamics Different interests (Hodges, 1991) Absence of competition, lack of customer inclusion (Chussil, 2005) Messy problems (Vennix, 1999)
Technical aspects	Process-related	1. Data issues Contradiction of existing data (Hodges, 1991) Sample data and data fit (2007) 2. Component issues Bad/missing model components (Hodges, 1991; Pidd, 2004) Component performance failure (Brooks & Tobias, 1996)
Level of model detail	Content-related	1. Too simple models/excessive use of simplification (Prieto et al., 2012; 2015) 2. Too complex models/lack of adequate simplification (1993; Goldberg et al., 1990; Brooks & Tobias, 1996; Salt, 2008)
Model results	Content-related	Values or results different from expectations/reality (Goldberg et al., 1990; Hodges & Dewar, 1992; Brooks & Tobias, 1996)
Structural or application-related model aspects	Content-related	1. Structure Model structure (Chussil, 2005; Bankes, 1998) Conceptual model not representative (Robinson, 1999) 2. Application Translation of problem into wrong model (Lombaers, 1979) Application to wrong problem (Law & McComas, 1991; Kotiadis et al., 2014)

The characteristics are defined as follows: Validation refers to any steps/process followed for a model to abide by its users’ requirements. Development aspects refer to topics that restrict or limit the development of a model. Decision-making related to model stakeholders regards any decisions connected to involved parties (clients, modellers, third parties) during model creation. Technical aspects concern considerations of data and components during development. Level of model detail refers to simplification issues of a model. Model results refer to problems of a model’s outcomes. The last category is putting together any aspects of a model’s structure and whether the model is applied.

Table 2. Information of the interviewed participants.

Company	Type/Sector	Interviewees	Experience	Stories shared
A	Consulting	6	5 J/I, 1 S	21
B	Manufacturing	7	5 J/I, 2 S	16
C	Consulting	6	1 J/I, 5 S	11
D	Consulting	1	1 S	2
E	Services & consulting	1	1 S	2
F	Consulting	1	1 S	2

All companies focused on consulting but for one whose focus was manufacturing.

Experience is deduced by the researchers based on participants’ background. Senior (S) is considered a participant with at least 10 years of experience, regardless of position as opposed to Junior (J) or Intermediate (I).

Table 3. Results on factors of wrongness from the interviews per perspective.

Characteristics		Unique references (stories) per category
		Modellers	Clients	Total
Process-related	Decision-making related to participants	23	8	30
	Technical aspects	18	5	20
	Development aspects	9	0	9
	Validation	8	0	8
Content-related	Model results	18	21	27
	Structural or application-related aspects	9	8	17
	Level of model detail	12	8	13
	Usability	0	2	2

Characteristics are referenced only when connected to a specific story as mentioned throughout Sections 4.1.1 and 4.1.2. The column “Total” does not summarise the two other columns to avoid counting a story twice. Instead, a characteristic is only counted once even if it is referenced both by modellers and clients.

Figure 1. Analysis followed for the 54 stories to consider uses.

Table 4. Results on interviews regarding whether models were used.

Category	Number of models
Used	25 (46.3%)
Not used	24 (44.4%)
Unclear	5 (9.3%)
Total	54

Table 5. Results of interviews regarding uses of models.

Types of uses	Total	Cases found
Decisions or actions	17 (68%)	-Resource or production changes like introduction of new lines, addition of equipment to assist production, factory redesign, storage allocation, and time-based vehicle assignments -Cases where a “wrong” model is used for exploring options in deciding production for a new facility, finding bottlenecks, suggesting how to set up new lines, determining why queues are jammed, and testing new facility throughputs -In 2 cases, “wrong” models are used as tools for scheduling to support ordering equipment, predict storage and time needs, and to support investments
Hypothesis/scenario testing	8 (32%)	-Exploration of system times, testing of new equipment impact, prediction of newly invested Emergency Departments, and measuring of efficiency/costs due to regulations and work load changes -In 2 cases “wrong” models are considered by clients as the experimental basis to get a better understanding of their systems’ complexities
Surface issues	7 (28%)	-“Wrong” models that reveal operational dysfunctions with managing personnel, how decisions are really made by personnel, and ambiguous records by clients -Model use to find issues between involved parties like proving a third party wrong or showing bad communication between departments and discrepancies between team
Better understanding or thinking development	3 (12%)	-A simple model showing an overview of expense flows that offered insights to the clients -A model that showed the actual problem to be different than the initial scope -A model that made clients realise the actual constraints they should address
Promote/communicate selling	2 (8%)	-Use of a “wrong“ model to assist ”commercially” for bidding as a sales tool (which was different than its original design for scheduling) -Use of a “wrong” model as evidence to prove a client’s interest for a bidding opportunity.
Store or monitoring	1 (4%)	- A “wrong” model used by an analyst who was to leave the project thus needed to produce certain results for peers who couldn’t use the model.

A model may abide by more than one categories. Each percentage is out of 25 stories.

The cases of the last column are the actual examples found in the explored stories.

Hodges, J. S. (1991). Six (Or So) things you can do with a bad model. Operations Research, 39(3), 355–365. https://doi.org/10.1287/opre.39.3.355

 Web of Science ®Google Scholar

Hodges, J., & Dewar, J. (1992). Is it you or your model talking? A framework for model validation. RAND Publication Series.

 Google Scholar

Gass, S. I. (1983). Decision-aiding models: Validation, assessment, and related issues for policy analysis. Operations Research, 31(4), 603–631. https://doi.org/10.1287/opre.31.4.603

 Web of Science ®Google Scholar

Sargent, R. G. (1991). Simulation model verification and validation. In B. L. Nelson, W. D. Kelton, & G. M. Clark, eds., Proceedings of the 23rd conference on Winter simulation, (pp.37–47).

 Google Scholar

Robinson, S. (1999). Three sources of simulation inaccuracy (and how to overcome them). In P. A. Farrington, H. B. Nembhard, D. T. Sturrock, and G. W. Evans (Eds), Proceedings of the 1999 Winter Simulation Conference. (pp.1701–1708).

 Google Scholar

Kleijnen, J. P. C. (1995). Statistical validation of simulation models. European Journal of Operational Research, 87(1), 21–34. https://doi.org/10.1016/0377-2217(95)00132-A

 Web of Science ®Google Scholar

Salt, J. D. (2008). The seven habits of highly defective simulation projects. Journal of Simulation, 2(3), 155–161. https://doi.org/10.1057/jos.2008.7

 Web of Science ®Google Scholar

Balci, O., Nance, R. E., & Arthur, J. D. (2002). Expanding our horizons in verification, validation, and accreditation research and practice. In E. Yücesan, C.-H. Chen, J. L. Snowdon, and J. M. Charnes, eds. Proceedings of the 2002 Winter Simulation Conference, (pp.653–663).

 Google Scholar

Chussil, M. (2005). With all this intelligence, why don’t we have better strategies? Journal of Business Strategy, 26(1), 26–33. https://doi.org/10.1108/02756660510575023

 Google Scholar

Bankes, S. (1998). Policy analysis for complex and uncertain systems through computational experiments. In IEEE Aerospace Conference, vol.1, (pp.9–21).

 Google Scholar

Tolk, A. (2015). Learning something right from models that are wrong: Epistemology of simulation. In L. InYilmaz (Ed.), Concepts and Methodologies for Modeling and Simulation. Simulation Foundations, Methods andApplications. Springer. (pp. 87–106). https://doi.org/10.1007/978-3-319-15096-3_5

 Google Scholar

Kolkman, D. A., Campo, P., Balke-Visser, T., & Gilbert, N. (2016). How to build models for government: Criteria driving model acceptance in policymaking. Policy Sciences, 49(4), 489–504. https://doi.org/10.1007/s11077-016-9250-4

 Web of Science ®Google Scholar

Lance, C. B., Woehr, D. J., & Meade, A. W. (2007). Case study : A Monte Carlo investigation of assessment center construct validity models. Organizational Research Methods, 10(3), 430–448. https://doi.org/10.1177/1.094.428.106.289.395

 Web of Science ®Google Scholar

Vennix, J. A. (1999). Group model-building: Tackling messy problems. System Dynamics Review, 15(4), 379–401. https://doi.org/10.1002/(SICI)1099-1727(199.924)15:4<379::AID-SDR179>3.0.CO;2-E

 Web of Science ®Google Scholar

Brooks, R. J., & Tobias, A. M. (1996). Choosing the best model: Level of detail, complexity, and model performance. Mathematical and Computer Modelling, 24(4), 1–14. https://doi.org/10.1016/0895-7177(96)00103-3

 Web of Science ®Google Scholar

Robinson, S., & Pidd, M. (1998). Provider and customer expectations of successful simulation projects. Journal of the Operational Research Society, 49(3), 200–209. https://doi.org/10.1057/palgrave.jors.2600516

 Web of Science ®Google Scholar

Bankes, S. (1993). Exploratory Modeling for policy analysis. Operations Research, 41(3), 435–449. https://doi.org/10.1287/opre.41.3.435

 Web of Science ®Google Scholar

Pidd, M. (2004). Systems modelling theory and practice. John Wiley & Sons.

 Google Scholar

Prieto, D. M., Das, T. K., Savachkin, A. A., Uribe, A., Izurieta, R., & Malavade, S. (2012). A systematic review to identify areas of enhancements of pandemic simulation models for operational use at provincial and local levels. BMC Public Health, 12(1), 1–13. https://doi.org/10.1186/1471-2458-12-251

 PubMedGoogle Scholar

Robinson, S. (2015). A tutorial on conceptual modeling for simulation. In L. Yilmaz, W. K. V. Chan, I. Moon, T. M. K. Roeder, C. Macal, and M. D. Rossetti (Eds), Proceedings of the 2015 Winter Simulation Conference, (pp.1820–1834).

 Google Scholar

Goldberg, J., Districh, R., Chen, J. M., Mitwasi, M., Valenzuela, T., & Criss, E. (1990). A simulation model for evaluating a set of emergency vehicle base locations: Development, validation, and usage. Socio-Economic Planning Sciences, 24(2), 125–141. https://doi.org/10.1016/0038-0121(90)90.017-2

 PubMed Web of Science ®Google Scholar

Lombaers, H. J. M. (1979). Operational research in practice. Engineering and Process Economics, 4(2-3), 245–248. doi:10.1016/0377-841X(79)90036-6.

 Google Scholar

Law, A. M., & McComas, M. G. (1991).Secrets of successful simulation studies. Proceedings of the 1991 Winter Simulation Conference, (pp.21–27).

 Google Scholar

Kotiadis, K., Tako, A. A., & Vasilakis, C. (2014). A participative and facilitative conceptual modelling framework for discrete event simulation studies in healthcare. Journal of the Operational Research Society, 65(2), 197–213. https://doi.org/10.1057/jors.2012.176

 Web of Science ®Google Scholar