2,035
Views
6
CrossRef citations to date
0
Altmetric
Research Article

Increasing situation awareness in healthcare through real-time simulation

ORCID Icon, ORCID Icon &
Pages 2339-2349 | Received 30 Jun 2021, Accepted 01 Nov 2022, Published online: 16 Nov 2022

Abstract

Research into real-time simulation applications outside of manufacturing environments has extended to sociotechnical systems such as healthcare over the past decade, where a number of published studies have demonstrated proof-of-concept models for near-future resource planning. Using real-time decision-support systems, people take decisions supported by the output of simulations. However real-time simulation frameworks abstract human intervention to an “external decision-maker,” with little regard to the complexities of underlying decision-making constructs, and how design and development decisions can impact the quality of decision-support. One such construct is situation awareness (SA), which is a precursor to decision-making. It is a dynamic state of knowledge about how a situation is unfolding; one approach to enhancing situation awareness is the provision of appropriate real-time information. We argue that design, development and implementation decisions should be focused at the interface between decision-making and decision-support. This integrative literature review proposes a SA framework integrating models of SA with a technical perspective for real-time simulation, to support an understanding of the cognitive needs of users alongside technical details during the development process. The implications for the usefulness and usability of real-time decision-support tools are discussed with application to Emergency Departments.

1. Introduction

With greater availability of data and computer power, the last decade has seen an increase in real-time simulation research and its challenges for short-term decision-support. A 2010 review of real-time simulation identified applications in power generation, automotives, transport, aerospace, and education (Bélanger et al., Citation2010). Around the same time, the approach began to be proposed in healthcare (Tavakoli et al., Citation2008; Marmor et al., Citation2009). The purpose of the real-time simulation in this context is to serve as a means of projecting the development of a situation in an existing system over a short time period to support safe short-term operational decisions. To date, few healthcare applications of real-time simulation have been published, with research lagging behind that of other industries.

The application of simulation modelling as a decision-support tool for complex systems has a proven track record, supporting an understanding of the interdependencies between human and system variables. The potential value of simulation for healthcare operational improvement is undisputed (Jahangirian et al., Citation2017; Zhang et al., Citation2020), despite persistent low evidence of results’ implementation (Katsaliaki & Mustafee, Citation2011; Roy et al., Citation2021). This lack of successful application of simulation studies in the healthcare domain has been attributed to the complexity of healthcare as a sociotechnical system, which characterises a system as an interconnected network of people and technology (Klein & Young, Citation2015; Long et al., Citation2020). Tasks have high diversity and are safety-critical, with a large number and variety of dynamically interacting elements, often operating under time and capacity constraints (Tako & Robinson, Citation2015).

Where care is operating close to the threshold of capacity, as has been the case in the UK for some years now (Amalberti & Vincent, Citation2020), the risk of a critical event occurring is high. A timely response to a critical situation requires effective short-term decision-making and adaptive behaviour to maintain system functioning. The quality of decision-making can be affected by workload and fatigue (Endsley, Citation1995; Endsley & Garland, Citation2000), for example, unpredictable workload can interfere with effective decision-making as work demands can exceed the capacity of available cognitive resources (Levin et al., Citation2012). This reduces system resilience by reducing the ability of decision-makers to anticipate, react and recover from critical situations.

With a focus on the needs of the users of the system, one approach to improving decision-making and supporting effective performance is by focussing attention on the type of information needed when and by whom to support system goals. Real-time simulation can provide much-needed information. However, current conceptualisations of real-time simulation abstract human decision-processes to an “external decision-maker,” with little regard to the complexities of underlying decision-making constructs, or how design and development decisions can impact the quality of decision-support (e.g., Aydt et al., Citation2008; Onggo et al., Citation2021). Real-time information can have an important role in contributing to awareness of the current state of a situation by updating users’ immediate knowledge and experience to make fast decisions to inform adaptive action. This is achieved by enhancing situation awareness (SA), a knowledge state that is considered to be essential for decision-making and performance in dynamic environments (Endsley, Citation2016; Chiappe et al., Citation2015). Loosely defined as a worker’s understanding of “what is going on” while interacting with a complex, dynamic system, SA is an important constituent in decision-making processes, and can be enhanced or detrimentally reduced by the introduction of environmental stimuli, including new information (Endsley, Citation1995; Citation2016).

For this reason, we argue in this paper that design, development and implementation decisions for real-time simulations should be focused at the interface between decision-making and decision-support. Towards this, we propose a sociotechnical view of real-time simulation. Our contribution is a proposed high-level framework which embodies decision-making and decision-support, through an examination of the construct of SA in the human factors literature, in particular the highly influential work of Endsley (Citation1995; Citation2016), and of real-time approaches to decision-support in the OR literature. This is done using an integrative literature review, which reviews, critiques, and synthesises representative literature on a topic in an integrated way such that new perspectives on the topic are generated (Torraco, Citation2005). The intention is to initiate new conversations around the purpose, use, and design of real-time simulation for short-term decision-support tools in sociotechnical systems such as healthcare, by re-framing existing understanding. Our framework can be used to inform methods choices, conceptual modelling activities and design in real-time simulation studies in healthcare.

The paper is structured as follows: Section 2 reviews the current state of real-time simulation in healthcare, and real-time simulation frameworks. Section 3 reviews theoretical models of SA and proposes a sociotechnical view of real-time simulation which accounts for human decision-making processes. Section 4 examines the SA framework with reference to emergency department short-term decision-support. Section 5 concludes the paper and provides pointers for future research.

2. Literature review

Healthcare 4.0, a collective term for data-driven digital health technologies, is expanding rapidly toward smart automation, protection of the critical functionality of healthcare infrastructure, and the privacy of personal data (Thuemmler & Bai, Citation2017; Jayaraman et al., Citation2020). Within this, real-time simulation has application for dynamic, goal-directed decisions in systems that continuously make decisions in real-time. A simulation is initialised and driven by real-time (or near real-time) data, adding flexibility to the monitoring of operational systems. Such data applications can be categorised in a widely-used functional classification (e.g., Shao et al., Citation2014):

  • Descriptive analytics involve observing real-time data to understand what is happening;

  • Diagnostic analytics involve exploratory analysis to determine why something is happening;

  • Predictive analytics involve prediction of future observations to determine what is likely to happen;

  • Prescriptive analytics enable the best course of action to be determined under certain circumstances, supporting the ability to influence the system towards its goal performance.

Adra (Citation2016) outlined how real-time simulation can be used for descriptive (real-time visibility), predictive, and prescriptive purposes, while Hoot et al. (Citation2008) developed a real-time simulation for diagnostic purposes, by indicating where bottlenecks would result in system congestion. Alternatively, real-time data may be used with simulation to support different stages of a modelling and simulation study (Mustafee et al., Citation2020). For example, predictive analytics using real-time, time-series data can inform system KPIs for a future system state, which then serves as the basis for comparing the results of scenarios in the experimentation stage of a simulation study.

In healthcare, several examples of real-time simulation have been published. Tavakoli et al. (Citation2008) and Mousavi et al. (Citation2011) adapted an approach from manufacturing to healthcare, while Espinoza et al. (Citation2014) and Marmor et al. (Citation2009) investigated the feasibility of real-time simulation in emergency departments (ED) for short-term resource allocation. Similarly, Tan et al. (Citation2013) and Bahrani et al. (Citation2013) developed prototype real-time DES models for staff planning. Due to the fast-paced, system-driven nature of the work, ED is a particular focus of application for real-time simulations in healthcare. For example, Hoot et al. (Citation2008) developed and validated a DES model to predict a range of ED operational indicators. Harper and Mustafee (Citation2019a; Citation2019b) described a model which combined time-series forecasting and real-time DES for predicting ED crowding, while Augusto et al. (Citation2018) proposed a prescriptive framework for real-time simulation in ED planning. A self-adaptive framework was proposed by Kotiadis (Citation2016) incorporating model reuse and sensor automation, illustrated with application to ED. Outside ED, Oakley et al. (Citation2020) used a proof-of-concept DES model for hospital bed management focussing on validation, a technical challenge as real-time simulation inputs and outputs are time-dependent. Technical challenges continue to exist, for example, data acquisition and integration. However, as cyber-physical systems and enabling technologies continue to evolve, the interaction between users and technology presents potentially more significant challenges.

2.1. Real-time simulation frameworks

The execution of real-time simulation has been in use in manufacturing systems for decades, using terms such as “online simulation,” “data-driven simulation,” “digital twin,” and “symbiotic simulation” (Onggo, Citation2019; Onggo et al., Citation2021). The conceptualisation by Fujimoto et al. (Citation2002), adapted by Aydt et al. (Citation2008), emphasised a mutual benefit between the simulation and the physical system through a continuous execution of the simulation and its real-time interaction with the real-world system ().

Figure 1. Symbiotic simulation decision support system, reproduced from Aydt et al. (Citation2008, p. 112).

Figure 1. Symbiotic simulation decision support system, reproduced from Aydt et al. (Citation2008, p. 112).

This is done via a control feedback from the simulation to the real system, either through an actuator or a human decision-maker (Onggo et al., Citation2021), and represents a closed-loop system. In a closed-loop system, there is feedback between the simulation and the real system, and this feedback affects the real system. In sociotechnical systems, the feedback closure is performed via human decision-making processes. The decision-maker retains control over the decision and subsequent action, and any action which changes the physical system will be subsequently reflected in the real-time data used to initialise the simulation model. Several authors have presented similar high-level architectures (Mousavi et al., Citation2011; Bahrani et al., Citation2013; Augusto et al., Citation2018; Onggo et al., Citation2021). In each case, human intervention is abstracted to an “external decision-maker” or “decision-process.”

Although much can be done to automate systems, in sociotechnical systems humans typically still need to take the information provided to determine a course of action, which means human judgement is integral to the decision, and there is always the possibility of human error. None of the above studies have addressed this issue, in particular with regard to how users might interface cognitively with the information provided by the real-time simulation, alongside their day-to-day work and multiple competing information sources. Cognitive processes such as SA are an integral part of decision-making using information in the environment.

For a decision-support system to constantly interface with the real-world requires some understanding of the “external decision-maker,” namely how characteristics of human decision-making may be influenced by design, development and implementation to maximise efficacy, efficiency and safety. The next section reviews the literature on SA, a measurable construct in cognitive psychology and human factors which describes the degree to which a decision-maker is aware of events and elements in their environment, both spatially and temporally, and the effect of actions on goals and objectives now and in the future.

3. Situational awareness framework

SA provides the primary basis for subsequent decision-making and is a state of knowledge, not the processes used to achieve that knowledge. Knowledge is the understanding gained from the analysis of information (Kuiler, Citation2014); or information combined with experience, context, interpretation, and reflection (Albert & Bradley, Citation1997). Viewing knowledge as a systemic property of an organisational system rather than within an individual supports a sociotechnical perspective, with information held by people, artefacts, and their interactions (Stanton et al., Citation2017). Boisot and Canals (Citation2004) saw data, information and knowledge as possessing specific types of utility: data utility in that it can carry information about the physical world; information utility in that it can modify an expectation or state of knowledge; and knowledge utility in that it allows an agent to act in an adaptive way upon and within the physical world. Once sufficient awareness of the situation has been gained, a match between past experience and knowledge about the current situation can be sought, which determines the appropriate course of action (Salas et al., Citation2010). The utility of the results of simulation experiments are their contribution to such awareness.

According to Endsley (Citation1995; 2000), SA occurs at three levels:

  • Level 1: The perception of elements in the environment;

  • Level 2: Comprehension of their meaning;

  • Level 3: The projection of their status into the near future.

Endsley’s (Citation1995) theoretical model of SA illustrates a closed-loop system with an undefined feedback loop from the real system that reflects the outcomes of an action (). The feedback may not be immediate, as the results of actions need to be perceived and comprehended in the environment. Real-time information can support this feedback loop by updating users’ immediate knowledge, and all real-time simulations share enhancing SA as part of their common purpose, for example, by predicting near-future patient volumes or wait-times. Nonetheless, how this information influences SA is rarely made explicit.

Figure 2. Three-level model of situation awareness in dynamic systems, adapted from Endsley (Citation1995, p. 35).

Figure 2. Three-level model of situation awareness in dynamic systems, adapted from Endsley (Citation1995, p. 35).

Based on its role in dynamic decision-making, considerable research has investigated the relationship between SA and a variety of individual and environmental factors (Endsley, Citation2020). Environmental limiting factors to SA include workload, stress, and system complexity, and their effects on the ability to process information and make effective and timely decisions. Stress and anxiety reduce the capacity of available memory, such that individuals may be more likely to rely on external sources of information than internal memory storage. Endsley (Citation2020) noted that performance would be impeded where SA is incomplete or inaccurate, yet competing demands of tasks for attention can exceed a staff member’s limited cognitive resources (e.g., Riveiro et al., Citation2008; Weigl et al., Citation2020). While there are many parallels between workload and performance, as task load increases, workload will increase but performance can remain stable as a result of a range of adaptive strategies (Parasuraman & Hancock, Citation2001). However, at some point, a sustained high workload may prevent staff from responding effectively to an increase in task load demand (Naderpour et al., Citation2016). For system design, these distinctions are important, as designs which support or improve task performance are different to those which support SA. Poor information designs can add to task load, for example, by being difficult to interpret, and can have a detrimental effect on SA and subsequent performance.

People are generally aware of information that is not the current focus of their attention. This peripheral awareness of background information enables people to rapidly switch attention to new matters if it becomes salient to them. A common example is the “cocktail party” effect, whereby in a noisy room, the sound of a person’s name can focus ones’ attention. During routine operations, SA is partial and selective, and varies according to job role and level of expertise. Consideration of the salience of simulation outputs applies to where and how the information is presented and accessed, for example, whether the user is able to find the information, whether the information is trigger-activated or continuous, and to what degree accessing the information interrupts workflow and stands out amongst the “noise” of other information sources in the work environment.

Being able to perceive and comprehend a system state, and make mental projections about the expected future development is crucial for safety, particularly where the context is time-pressured and high-risk (Gillespie et al., Citation2013; Tscholl et al., Citation2020). A major component of the job of a healthcare provider involves developing SA and keeping it up-to-date in a rapidly changing environment, requiring team members to have an understanding of the type of information needed by others, the devices used to distribute SA (e.g., visual displays or dashboards), shared team processes to facilitate information sharing (e.g., communication, coordination, cooperation), and shared mechanisms such as a common mental model (Salmon et al., Citation2008). Designing outputs that support these processes is therefore critical. The next section describes our SA framework for real-time simulation and its value in informing the design of real-time simulation studies in healthcare and other sociotechnical systems.

3.1 A situational awareness (SA) framework for real-time simulation in sociotechnical systems

For real-time systems, technical aspects combine with usability features, as real-time simulations are usually developed as recurrent-use tools, adding complexity to conceptual modelling design. We refer to our contribution as a SA framework as it has both conceptual and technical elements, and its intended purpose is to inform the design of real-time simulations. Additionally, core to our framework is collaborative engagement with stakeholders to understand the system and its requirements (Robinson et al., Citation2014; Tako & Kotiadis, Citation2015), with development likely to require collaboration across all stages of the study lifecycle (Kotiadis et al., Citation2014). Jones et al. (2022) developed an overarching conceptual frame for hybrid simulation, emphasising the frame’s importance in capturing the why in hybridisation. Their overarching frame can be used to inform conceptual model development for hybrid simulation studies, and to communicate the value of the chosen approach to modellers and stakeholders. Similarly, the purpose of our proposed SA framework is to provide a high-level representation of the system components to consider how to maximise the system value of the real-time simulation. The framework operates at a higher level of abstraction than a conceptual model, which focuses on specific development decisions such as precise objectives, inputs, outputs, content etc. (Robinson, Citation2020). It can be used to support methods choices; specify the need for and approaches to collaborative activities; inform conceptual modelling processes; and support design, implementation and evaluation decisions through its broad representation.

Of the many SA models published in the literature (see Tremblay (Citation2017) for a comprehensive overview), Endsley’s (Citation1995) 3-stage model of SA in dynamic systems has been the most influential. Its closed-loop design can be readily mapped to closed-loop real-time simulation conceptualisations, such that the decision-maker creates a control feedback upon the physical system, while consequent changes to the data, simulations, and outputs create a control feedback updating the SA of the decision-maker. Real-time simulation can output descriptive, diagnostic, predictive or prescriptive information (Adra, Citation2016), alone or in combination with other methods. Salient outputs support a perception of the current system state for Level 1 SA, clarity and presentation of descriptive and diagnostic outputs support comprehension for Level 2 SA, and predictive and prescriptive outputs support projection of future states for Level 3 SA.

presents our proposed SA framework for real-time simulation in sociotechnical systems as a 2 × 2 matrix, which represents the system across the two dimensions. The social (the decision-maker, and their decisions and actions), and technical components (the physical system and the simulation model) form the horizontal axes. On the vertical axes, the physical system and any actions performed upon it characterise the real system. Representations of the system are composed of the simulation model as an external representation of the system (the model and its outputs can be visualised and are therefore standardised across all users), while the SA of human decision-makers forms a mental model or internal representation (knowledge of the system state and the effect of actions upon it are held conceptually by decision-makers, and may vary across decision-makers) (Löhner et al., Citation2003).

Figure 3. A SA framework for real-time simulation of sociotechnical systems.

Figure 3. A SA framework for real-time simulation of sociotechnical systems.

SA may be impeded by distractions, stressful situations, high workload, vigilance failures, poorly presented or ambiguous information, forgetting key information, and poor mental models, reducing human decision quality and speed (Endsley, Citation2016). The outputs of the real-time simulation should aim to guide constrained and enabled safe action, and design, development and evaluation choices can ensure these are as intended. Outputs which are confusing, difficult to understand, incomplete, do not follow standard procedures, or do not align with mental models can adversely affect SA by increasing workload in order to make sense of the information (Pennathur et al., Citation2011; Peute et al., Citation2013; McGeorge et al., Citation2015; Dixit et al., Citation2020).

A degraded state of SA increases reliance on external sources of information, so poorly perceived information can result in a negative SA spiral, of particular importance under high workload conditions (Blandford & Wong, Citation2004; Brennan et al., Citation2020). Our high-level SA framework positions the decision-maker as a central component of the sociotechnical system, and emphasises the core purpose of the simulation outputs: to update the users’ knowledge of the current and projected system state and to ensure the influence of the information on SA is as intended.

4. The relevance of the SA framework for emergency departments (ED)

Many features of ED position it as a complex sociotechnical system, and illustrate how it may benefit from short-term operational decision-support, including the event-driven nature of the work, and variable demand and requirements of patients (Carayon, Citation2016). ED workflow accounts for both clinical care and time-limited targets, hence workflow is both clinical and organisational, and staff manage pressures by making in situ adaptations and goal trade-offs toward safe, quality care (Woods & Branlat, Citation2011). Levin et al. (Citation2012) reported growing evidence of a relationship between ED crowding, reduced SA, and patient safety, finding the number of patients managed (i.e., high task load) contributes most to a reduction in SA and its potential effects on patient safety (i.e., performance). These features position ED as a relevant domain for examining the SA framework.

As described in Section 2, ED has been a particular focus of work in real-time simulation, characterised as the technical represented system in the SA framework. Here, decisions about appropriate methods can focus specifically on their contribution to SA. For example, Aydt et al. (Citation2008) proposed forecasting a critical indicator, with simulation to support system reconfiguration before the critical condition occurs, offering constant “projected” SA support. Ardito et al. (Citation2020) proposed the use of a real-time tool for emergency dispatch integrating process mining to understand patient flows and highlight bottlenecks, with simulation to support system recovery. SA is addressed through the use of visualisations (perception), process mining (comprehension) and simulation (projection). Their focus was on methods support for each stage of SA to align the needs of stakeholders with their SA requirements. Early consideration of interoperability, focusing on rapidly meeting user needs, and prioritising evaluation criteria with stakeholders suggest the need for collaborative design (Dixit et al., Citation2020). This is likely to be required across the study lifecycle, and identification of methods for addressing this may form part of the conceptual modelling process.

Design features can be used to support the “external representation” of simulation outputs. Research on health information system design and evaluation has provided insights into factors contributing to successful system design, safety-critical aspects, system user-friendliness and usability. For example, Dixit et al. (Citation2020) recommended providing “in-progress” visualisations, and designing for stakeholders in term of metrics and user-literacy, especially for inexperienced users. Blandford and Wong (Citation2004) found that the integration and presentation of information should support immediate quick-glance interpretation, with minimal reliance on “drilling down” for details, or comparing information sources. They also found that the level of certainty in the information should be indicated; this is considered important for predictive analytics used for decision-support (Petropoulos et al., Citation2022).

SA is structured and supported by an underlying mental model. The “external representation” of information informs the “internal representation,” or mental model, of users, hence the importance of design and understanding workflow when developing and implementing new technology for decision-support. Weigl et al. (Citation2020) reported that high rates of interruptions were significantly associated with low levels of ED providers’ SA. Whilst ED clinical staff continuously cope with disruptions and interruptions, technical malfunctions and other interruptive workflow environments impede SA, hence technology-related disruptions should be avoided. When implementing technologies in ED, factors such as proximity to staff task-space, amount of view detail at a time (quick-glance view vs. interactive scrolling), and amount and type of interaction need to be considered (Pennathur et al., Citation2011). In addition, researchers need to be aware of the impact of real-time simulation implementation on secondary task performance in a multitasking domain, where additional information can potentially hinder performance in tasks using other technologies. Investigation of such features may form part of an evaluation plan as part of the overall study design.

A collaborative design process should drive development, putting the needs of stakeholders at the forefront of the design and development process. In ED, where task load is high, the importance of this cannot be understated if the outputs are to be used by frontline staff for informing adaptive behaviours. Where medium-term planning is proposed, a different range of considerations may apply, however usability, clarity, accessibility, accuracy, and reliability remain essential concerns which require stakeholder engagement and testing. Addressing these is likely to require a context-dependent design, implementation and evaluation plan.

5. Discussion

In this age of Industry 4.0, interest in the use of real-time simulation outside of manufacturing environments has extended to sociotechnical systems such as healthcare, where a number of published studies have demonstrated proof-of-concept models for near-future planning. Developers of real-time simulation models should view themselves as system designers, investigating the needs of users with design and development decisions alongside technical development. At this interface lies the cognitive construct of SA, the dynamic state of knowledge which perceives and interprets environmental information, and projects the state of the environment into the near future to inform decisions and action. A real-time simulation, alone or in combination with other methods, provides information that can support perception, comprehension and projection of the system state through descriptive, diagnostic, predictive and prescriptive information.

Our proposed SA framework can inform methods selection, including collaborative activities. It can support the conceptual modelling process by providing an overarching conceptualisation. Finally, it can enable a structured approach to design and development. Without this focus, at best, the real-time simulation may deliver additional noise in an already noisy environment; at its worst, it may impact on the ability of users to make safe decisions, reducing, rather than supporting system resilience. At a time of rapid evolution of real-time simulation tools in multiple domains, focusing efforts on technical challenges are essential, but without simultaneously attending to the needs of the decision-maker, these tools will continue to remain “proof-of-concept” in sociotechnical systems. In emergency care, where the majority of real-time simulation applications have been proposed and tested, there is significant opportunity to advance real-time simulation prototypes toward implementation.

This paper opens up substantial opportunity for further research. Studies which take a SA approach to information design recognise the methodological challenges in studying this area. Wickens (Citation2000) summarised Endsley’s conceptualisation of SA, with particular regard to measurement, while Endsley (Citation2020) reviewed a wide range of subjective and objective SA measurement tools to draw conclusions around their divergence. For example, Endsley and Smolensky (Citation1998) discuss the use of controlled laboratory settings; while Blandford and Wong (Citation2004), Dixit et al. (Citation2020) and Weigl et al. (Citation2020) relied on observations and qualitative data in naturalistic settings. A weakness of the qualitative approach is getting reliable data at times when the requirements of SA are highest. For example, where safety-critical tasks rely on verbal communication, a think-aloud protocol can interfere with task performance, distracting staff during periods of high workload. Nonetheless, real-time simulation tools should be tested under realistic conditions with experienced staff, and for measuring SA, a range of validated methods are available (e.g., Pennathur et al., Citation2011).

People have low self-awareness of their own SA, so shortcomings can be difficult to detect qualitatively (Wickens, Citation2000). Endsley (Citation2020) argued that subjective measures of SA appear to better reflect a person’s confidence in their SA, which independently affects performance. A person who has poor SA but is overconfident is likely to act confidently and incorrectly, and may even influence the actions of others. This has design implications, as overconfidence in incorrect model outputs that reinforce faulty mental models will lead to poor decisions (Sulistyawati et al., Citation2011). Yilmaz and Liu (Citation2022) suggest that simulation design should be context-sensitive to mitigate against over-trust as well as distrust. One approach to design support is the use of participatory modelling and simulation approaches (e.g., Kotiadis & Tako, Citation2021; Tako & Kotiadis, Citation2015; Robinson et al., Citation2014), which can facilitate complex sociotechnical applications through communication and collaboration. Problem structuring methods and other qualitative techniques can enable exploration of SA during the model development lifecycle, alongside addressing technical challenges such as accessing and protecting sensitive data, interoperability, and model validation.

While our framework has been discussed in relation to healthcare, the underlying principles are more generally applicable where real-time simulations are used for decision-support, such as education, transport control and crowd management. In each case, there is a need to develop the simulation model with an awareness of the needs of users and stakeholders, and a design, implementation, and evaluation plan that considers the relevant features of usability, safety, efficiency, and efficacy. Ignoring the needs of human decision-makers can result in failed implementation, or worse, implemented models that negatively impact decision-making and reduce safety.

Acknowledgements

This report is independent research supported by the National Institute for Health Research Applied Research Collaboration South West Peninsula. The views expressed in this publication are those of the author(s) and not necessarily those of the National Institute for Health Research or the Department of Health and Social Care. This work was supported in part by the ESRC ES/P000630/1 and ESRC ES/W005875/1.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

Economic and Social Research Council.

References

  • Adra, H. (2016). Real-time predictive and prescriptive analytics with real-time data and simulation. In 2016 Winter Simulation Conference (WSC). (pp. 3357–3364). IEEE.
  • Albert, S., & Bradley, K. (1997). Managing knowledge: Experts. Agencies and Organisations. Cambridge University Press.
  • Amalberti, R., & Vincent, C. (2020). Managing risk in hazardous conditions: Improvisation is not enough. BMJ Quality & Safety, 29(1), 60–63. https://doi.org/10.1136/bmjqs-2019-009443
  • Ardito, C., Di Noia, T., Lofu, D., Mallardi, G. (2020). An adaptive architecture for healthcare situation awareness. Proceedings of i-CiTies.
  • Augusto, V., Viallon, A., & Murgier, M. (2018). A modelling and simulation framework for intelligent control of emergency units in the case of major crisis. In Proceedings of the 2018 Winter Simulation Conference M. Rabe., A.A. Juan, N. Mustafee, A. Skoogh, S. Jain, and B. Johansson, eds. (pp. 2495–2506). IEEE.
  • Aydt, H., Turner, S. J., Cai, W., & Low, M. Y. H. (2008). Symbiotic simulation systems: An extended definition motivated by symbiosis in biology. Proceedings - Workshop on Principles of Advanced and Distributed Simulation, PADS, b, 109–116. Rome, Italy: IEEE.
  • Bahrani, S., Tchemeube, R. B., Mouttham, A., & Amyot, D. (2013). Real-time simulations to support operational decision making in healthcare [Paper presentation]. In Proceedings of the 2013 Summer Computer Simulation Conference, (p. 53). Society for Modeling & Simulation International.
  • Bélanger, J., Venne, P., & Paquin, J.-N. (2010). The what, where and why of real-time simulation. Planet RT, 1(0), 37–49.
  • Blandford, A., & Wong, B. W. (2004). Situation awareness in emergency medical dispatch. International Journal of Human-Computer Studies, 61(4), 421–452. https://doi.org/10.1016/j.ijhcs.2003.12.012
  • Boisot, M., & Canals, A. (2004). Data, information and knowledge: Have we got it right? Journal of Evolutionary Economics, 14(1), 43–67. https://doi.org/10.1007/s00191-003-0181-9
  • Brennan, P. A., Holden, C., Shaw, G., Morris, S., & Oeppen, R. S. (2020). Leading article: What can we do to improve individual and team situational awareness to benefit patient safety? The British Journal of Oral & Maxillofacial Surgery, 58(4), 404–408.
  • Carayon, P. (Ed.). (2016). Handbook of human factors and ergonomics in health care and patient safety. CRC Press.
  • Chiappe, D., Strybel, T. Z., & Vu, K. P. L. (2015). A situated approach to the understanding of dynamic situations. Journal of Cognitive Engineering and Decision Making, 9(1), 33–43. https://doi.org/10.1177/1555343414559053
  • Dixit, R. A., Hurst, S., Adams, K. T., Boxley, C., Lysen-Hendershot, K., Bennett, S. S., Booker, E., & Ratwani, R. M. (2020). Rapid development of visualisation dashboards to enhance situation awareness of COVID-19 telehealth initiatives at a multihospital healthcare system. Journal of the American Medical Informatics Association, 27(9), 1456–1461. https://doi.org/10.1093/jamia/ocaa161
  • Endsley, M. R. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors: The Journal of the Human Factors and Ergonomics Society, 37(1), 32–64. https://doi.org/10.1518/001872095779049543
  • Endsley, M. R. (2016). Designing for situation awareness: An approach to user-centered design. CRC press.
  • Endsley, M. R. (2020). The divergence of objective and subjective situation awareness: A meta-analysis. Journal of Cognitive Engineering and Decision Making, 14(1), 34–53. https://doi.org/10.1177/1555343419874248
  • Endsley, M. R., & Garland, D. J. (2000). Theoretical underpinnings of situation awareness: A critical review. Situation Awareness Analysis and Measurement, 1(1), 3–21.
  • Endsley, M. R., & Smolensky, M. W. (1998). Situation awareness in air traffic control: The picture. InSmolensky, M.W., Stein, E.S. (Eds.), Human Factors in Air Traffic Control. Academic Press. pp. 115–150
  • Espinoza, C., Pascual, J., Ramis, F., & Broquez, D. (2014). Real-time simulation as a way to improve daily operations in an emergency room. In Proceedings 2014 Winter Simulation Conference., (pp.2600–2608). IEEE.
  • Fujimoto, R., Lunceford, D., Page, E., & Uhrmacher, A. M. (2002). Grand Challenges for Modeling and Simulation. In Schloss Dagstuhl, 350. www.dagstuhl.de/02351
  • Gillespie, B. M., Gwinner, K., Fairweather, N., & Chaboyer, W. (2013). Building shared situational awareness in surgery through distributed dialog. Journal of Multidisciplinary Healthcare, 6, 109.
  • Harper, A., & Mustafee, N. (2019a). A hybrid modelling approach using forecasting and real-time simulation to prevent emergency department overcrowding [Paper presentation]. 2019 Winter Simulation Conference (WSC), In (pp. 1208–1219). IEEE. https://doi.org/10.1109/WSC40007.2019.9004862
  • Harper, A., & Mustafee, N. (2019b). Proactive Service Recovery in Emergency Departments: A Hybrid Modelling Approach using Forecasting and Real-time Simulation [Paper presentation]. In Proceedings of the 2019 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation (pp. 201–204).
  • Hoot, N. R., LeBlanc, L. J., Jones, I., Levin, S. R., Zhou, C., Gadd, C. S., & Aronsky, D. (2008). Forecasting emergency department crowding: A discrete event simulation. Annals of Emergency Medicine, 52(2), 116–125.
  • Jahangirian, M., Taylor, S. J., Young, T., & Robinson, S. (2017). Key performance indicators for successful simulation projects. Journal of the Operational Research Society, 68(7), 747–765.
  • Jayaraman, P. P., Forkan, A. R. M., Morshed, A., Haghighi, P. D., & Kang, Y. B. (2020). Healthcare 4.0: A review of frontiers in digital health. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(2), e1350.
  • Jones, W., Kotiadis, K., O’Hanley, J. R., & Robinson, S. (2021). Aiding the development of the conceptual model for hybrid simulation: Representing the modelling frame. Journal of the Operational Research Society, 1–19. https://doi.org/10.1080/01605682.2021.2018368
  • Katsaliaki, K., & Mustafee, N. (2011). Applications of simulation within the healthcare context. The Journal of the Operational Research Society, 62(8), 1431–1451.
  • Klein, J. H., & Young, T. (2015). Health care: A case of hypercomplexity? Health Systems, 4(2), 104–110. https://doi.org/10.1057/hs.2014.21
  • Kotiadis, K. (2016). Towards self-adaptive discrete event simulation (SADES). In Proceedings of the Operational Research Society Simulation Workshop 2016 (SW16) (pp. 1–11). UK or Society.
  • Kotiadis, K., & Tako, A. A. (2021). A tutorial on involving stakeholders in facilitated simulation studies [Paper presentation]. In Proceedings of the Operational Research Society Simulation Workshop 2021 (SW21), (pp. 42–56). The Operational Research Society.
  • Kotiadis, K., Tako, A. A., & Vasilakis, C. (2014). A participative and facilitative conceptual modelling framework for discrete event simulation studies in healthcare. Journal of the Operational Research Society, 65(2), 197–213. https://doi.org/10.1057/jors.2012.176
  • Kuiler, E. W. (2014). From big data to knowledge: An ontological approach to big data analytics. Review of Policy Research, 31(4), 311–318. https://doi.org/10.1111/ropr.12077
  • Levin, S., Sauer, L., Kelen, G., Kirsch, T., Pham, J., Desai, S., & France, D. (2012). Situation awareness in emergency medicine. IIE Transactions on Healthcare Systems Engineering, 2(2), 172–180. https://doi.org/10.1080/19488300.2012.684739
  • Löhner, S., van Joolingen, W. R., & Savelsbergh, E. R. (2003). The effect of external representation on constructing computer models of complex phenomena. Instructional Science, 31(6), 395–418. https://doi.org/10.1023/A:1025746813683
  • Long, K. M., McDermott, F., & Meadows, G. N. (2020). Factors affecting the implementation of simulation modelling in healthcare: A longitudinal case study evaluation. Journal of the Operational Research Society, 71(12), 1927–1939.
  • Marmor, Y. N., Wasserkrug, S., Zeltyn, S., Mesika, Y., Greenshpan, O., Carmeli, B., Shtub, A., & Mandelbaum, A. (2009). Toward simulation-based real-time decision-support systems for emergency departments. In Proceedings of the 2009 Winter Simulation Conference (WSC) (pp. 2042–2053). IEEE.
  • McGeorge, N., Hegde, S., Berg, R. L., Guarrera-Schick, T. K., LaVergne, D. T., Casucci, S. N., Hettinger, A. Z., Clark, L. N., Lin, L., Fairbanks, R. J., Benda, N. C., Sun, L., Wears, R. L., Perry, S., & Bisantz, A. (2015). Assessment of innovative emergency department information displays in a clinical simulation center. Journal of Cognitive Engineering and Decision Making, 9(4), 329–346. https://doi.org/10.1177/1555343415613723
  • Mousavi, A., Komashie, A., & Tavakoli, S. (2011). SIMulation-based real-time performance MONitoring (SIMMON): A platform for manufacturing and healthcare systems. In Proceedings of the 2011 Winter Simulation Conference (WSC) (pp. 600–611). IEEE.
  • Mustafee, N., Harper, A., & Onggo, B. S. (2020). Hybrid modelling and simulation (M&S): Driving innovation in the theory and practice of M&S [Paper presentation]. 2020 Winter Simulation Conference (Wsc), In (pp. 3140–3151). IEEE. https://doi.org/10.1109/WSC48552.2020.9383892
  • Naderpour, M., Lu, J., & Zhang, G. (2016). A safety-critical decision support system evaluation using situation awareness and workload measures. Reliability Engineering & System Safety, 150, 147–159. https://doi.org/10.1016/j.ress.2016.01.024
  • Oakley, D., Onggo, B. S., & Worthington, D. (2020). Symbiotic simulation for the operational management of inpatient beds: Model development and validation using Δ-method. Health Care Management Science, 23(1), 153–169.
  • Onggo, B. S. (2019). Symbiotic simulation system (S3) for. industry 4.0. In Simulation for Industry 4.0. (pp. 153–165) Springer.
  • Onggo, B. S., Corlu, C. G., Juan, A. A., Monks, T., & de la Torre, R. (2021). Combining symbiotic simulation systems with enterprise data storage systems for real-time decision-making. Enterprise Information Systems, 15(2), 230–247. https://doi.org/10.1080/17517575.2020.1777587
  • Parasuraman, R., & Hancock, P. A. (2001). Adaptive control of workload. In P. A. Hancock & P. E. Desmond (Eds.), Stress, workload, and fatigue. (pp. 305–320). Erlbaum.
  • Pennathur, P. R., Cao, D., Bisantz, A. M., Lin, L., Fairbanks, R. J., Wears, R. L., Perry, S. J., Guarrera, T. K., Brown, J. L., & Sui, Z. (2011). Emergency department patient-tracking system evaluation. International Journal of Industrial Ergonomics, 41(4), 360–369. https://doi.org/10.1016/j.ergon.2011.02.003
  • Petropoulos, F., Apiletti, D., Assimakopoulos, V., Babai, M. Z., Barrow, D. K., Ben Taieb, S., Bergmeir, C., Bessa, R. J., Bijak, J., Boylan, J. E., Browell, J., Carnevale, C., Castle, J. L., Cirillo, P., Clements, M. P., Cordeiro, C., Cyrino Oliveira, F. L., De Baets, S., Dokumentov, A., … Ziel, F. (2022). Forecasting: Theory and practice. International Journal of Forecasting, 38(3), 705–871. https://doi.org/10.1016/j.ijforecast.2021.11.001
  • Peute, L. W., Driest, K. F., Marcilly, R., Da Costa, S. B., Beuscart-Zephir, M. C., & Jaspers, M. W. (2013). A framework for reporting on human factor/usability studies of health information technologies. In CSHI, 54–60.
  • Riveiro, M., Falkman, G., Ziemke, T. (2008). Improving Maritime Anomaly Detection and Situation Awareness through Interactive Visualization. 2008 11th International Conference on Information Fusion, 47–54.
  • Robinson, S. (2020). Conceptual modelling for simulation: Progress and grand challenges. Journal of Simulation, 14(1), 1–20. https://doi.org/10.1080/17477778.2019.1604466
  • Robinson, S., Worthington, C., Burgess, N., & Radnor, Z. J. (2014). Facilitated modelling with discrete-event simulation: Reality or myth? European Journal of Operational Research, 234(1), 231–240. https://doi.org/10.1016/j.ejor.2012.12.024
  • Roy, S., Prasanna Venkatesan, S., & Goh, M. (2021). Healthcare services: A systematic review of patient-centric logistics issues using simulation. Journal of the Operational Research Society, 72(10), 2342–2364. https://doi.org/10.1080/01605682.2020.1790306
  • Salas, E., Rosen, M. a., & DiazGranados, D. (2010). Expertise-Based Intuition and Decision Making in Organisations. Journal of Management, 36(4), 941–973. https://doi.org/10.1177/0149206309350084
  • Salmon, P. M., Stanton, N. A., Walker, G. H., Baber, C., Jenkins, D. P., McMaster, R., & Young, M. S. (2008). What really is going on? Review of situation awareness models for individuals and teams. Theoretical Issues in Ergonomics Science, 9(4), 297–323. https://doi.org/10.1080/14639220701561775
  • Shao, G., Shin, S. J., Jain, S. (2014). Data analytics using simulation for smart manufacturing. In Proceedings of the Winter Simulation Conference 2014 (pp. 2192–2203). IEEE
  • Stanton, N. A., Salmon, P. M., Walker, G. H., Salas, E., & Hancock, P. A. (2017). State-of-science: Situation awareness in individuals, teams and systems. Ergonomics, 60(4), 449–466.
  • Sulistyawati, K., Wickens, C. D., & Chui, Y. P. (2011). Prediction in situation awareness: Confidence bias and underlying cognitive abilities. The International Journal of Aviation Psychology, 21(2), 153–174. https://doi.org/10.1080/10508414.2011.556492
  • Tako, A. A., & Kotiadis, K. (2015). PartiSim: A multimethodology framework to support facilitated simulation modelling in healthcare. European Journal of Operational Research, 244(2), 555–564. https://doi.org/10.1016/j.ejor.2015.01.046
  • Tako, A. A., & Robinson, S. (2015). Is simulation in health different? Journal of the Operational Research Society, 66(4), 602–614. https://doi.org/10.1057/jors.2014.25
  • Tan, K. W., Tan, W. H., Lau, H. C. (2013). Improving patient length-of-stay in emergency department through dynamic resource allocation policies. IEEE International Conference on Automation Science and Engineering (CASE) 984–989.
  • Tavakoli, S., Mousavi, A., Komashie, A. (2008). A generic framework for real-time discrete event simulation (DES) modelling. Proceedings of the 2008 Winter Simulation Conference, 1931–1938.
  • Thuemmler, C., & Bai, C. (2017). Health 4.0: How virtualization and big data are revolutionizing healthcare. Springer.
  • Torraco, R. J. (2005). Writing integrative literature reviews: Guidelines and examples. Human Resource Development Review, 4(3), 356–367. https://doi.org/10.1177/1534484305278283
  • Tremblay, S. (2017). A cognitive approach to situation awareness: Theory and application. Routledge.
  • Tscholl, D. W., Rössler, J., Said, S., Kaserer, A., Spahn, D. R., & Nöthiger, C. B. (2020). Situation awareness-oriented patient monitoring with visual patient technology: A qualitative review of the primary research. Sensors, 20(7), 2112. https://doi.org/10.3390/s20072112
  • Weigl, M., Catchpole, K., Wehler, M., & Schneider, A. (2020). Workflow disruptions and provider situation awareness in acute care: An observational study with emergency department physicians and nurses. Applied Ergonomics, 88, 103155. https://doi.org/10.1016/j.apergo.2020.103155
  • Wickens, C. D. (2000). The trade-off of design for routine and unexpected performance: Implications of situation awareness. In Endsley, M.R., Garland, D.J. (Eds.), Situation Awareness Analysis and Measurement. Lawrence Erlbaum Associates, Inc. Publishers.
  • Woods, D. D., & Branlat, M. (2011). Basic patterns in how adaptive systems fail. Resilience Engineering in Practice, 2, 1–21.
  • Yilmaz, L., & Liu, B. (2022). Model credibility revisited: Concepts and considerations for appropriate trust. Journal of Simulation, 16(3), 312–325. https://doi.org/10.1080/17477778.2020.1821587
  • Zhang, X., Lhachimi, S. K., & Rogowski, W. H. (2020). Reporting Quality of Discrete Event Simulations in Healthcare—Results from a Generic Reporting Checklist. Value in Health, 23(4), 506–514.