Modeling driver behavior in critical traffic scenarios for the safety assessment of automated driving

Abstract

Objective

Before market introduction, the safety of highly automated driving systems needs to be assessed prospectively. BMW has developed a holistic approach for the assessment of the traffic safety impact by these systems in which stochastic traffic simulations play a significant role. A driver behavior model which represents realistic driver behavior ranging from performance in non-critical everyday driving toward performance in critical situations is key for this approach. To ensure trustworthy results, validation of the driver model is needed. The paper aims at demonstrating that the presented driver model acts realistically in different critical real-world traffic scenarios.

Methods

BMW has been developing the Stochastic Cognitive Model (SCM) which models cognitive processes in traffic situations. These processes range from information acquisition by gaze behavior, mental representation of the environment, recognition of situations from the visual information and reaction to the situation. The driver model combines these cognitive processes with stochastic driver parameters to obtain a variation in driver behavior in simulations. Especially visual attention modeling is key to realistic traffic interactions in simulations as this is the input for the sequential cognitive processes, i.e., the recognition of situations and the reaction to the situation. Modeling of driver’s gaze behavior with SCM is thus shown in this paper.

Results

SCM is applied in three critical real-world traffic scenarios in which gaze behavior, brake reaction times and time-to-collisions are evaluated and compared to the real-world data. Due to the stochastic approach not only a single SCM agent but a collective of virtual SCM test drivers is assessed. Results show that SCM is capable to simulate the influence of visual inattention on collision risk.

Conclusion

Realistic driver behavior in simulations can be achieved by using SCM. Especially in the presented critical scenarios SCM is able to represent real-world driving behavior which is determined particularly by its gaze behavior and subsequent reaction. Driving performance varies over different SCM agents which mean that different driving behavior can be simulated with SCM as well. However, the investigation in this paper included only three real-world cases. Therefore, further critical, and additionally non-critical scenarios need to be investigated in the future.

Keywords:

• Driver behavior model
• gaze behavior
• stochastic traffic simulation
• safety assessment
• automated driving

Introduction

Automated driving systems (ADS) are about to enter the automotive market. Highly related to their market introduction is the question about their safety. The European and German Ethics commission requests the demonstration of ADS’s positive effects on traffic safety respectively a “positive risk balance” (Di Fabio et al. 2017; Bonnefon et al. 2020). BMW has subsequently developed the “Positive Risk Balance” framework that serves as a holistic approach for developing safe ADS (Kauffmann et al. 2022).

The prospective safety assessment before market introduction is an essential part of the framework. Different approaches for the prospective safety assessment exist, ranging from driving simulator studies, real-world tests in a controlled environment (test-tracks) or field operational tests (Breuer et al. 2015). These approaches have different disadvantages—they are either time and resource consuming or they investigate only surrogate measures instead of the reduction of collisions. The simulation-based safety assessment approach (ISO Technical Report 2021) allows to overcome these issues by investigating a high number of scenarios with reasonable effort. It also measures the reduction of collisions. As the approach is based on simulations, validation and verification measures are crucial for proving the simulation’s correctness.

Approaches for simulation-based safety assessment

The ISO Technical Report, “Road vehicles—Prospective safety performance assessment of pre-crash technology by virtual simulation” describes four approaches for the simulation-based safety assessment (ISO Technical Report 2021). The first two approaches are accident-based approaches in which real-world crashes are either resimulated with or without modification. For these two approaches trajectories of the crash-involved traffic participants are available. But the trajectories are only available for a couple of seconds before the collision. Hence, these approaches are limited in their usefulness to evaluate the safety impact for ADS, which cover a larger scenario space and are in operation already before a scenario gets critical. The safety assessment of ADS needs to cover the effects during normal driving (e.g., comply with the speed limit) as well, i.e., the assessment needs to cover larger time frames.

According to ISO 21934 (ISO 202) the other two approaches are traffic-based approaches which derive scenarios by sampling the scenario parameters from real-world data distributions. While the third approach considers only a limited number of scenarios for the assessment (scenarios should represent the most relevant ones), the fourth approach covers a high number of scenarios to investigate the complete scenario space. Therefore, BMW sees a clear advantage in using the fourth approach for ADS. However, for this approach only the starting parameters are sampled, which means that no pre-defined trajectories exist. Therefore, a sophisticated driver behavior model is required to get realistic trajectories for the vehicles involved in the scenario. The model must represent the entire variety of realistic driver behavior ranging from non-critical driving to critical situations.

Driver models in traffic simulations

Car following models for traffic simulations have been used in traffic flow simulations for a long time. Widely known examples of these models are the Wiedemann model (Wiedemann 1974), the Gipps model (Gipps 1981) or the Intelligent Driver Model (Treiber et al. 2000). However, as these models have been designed for generating traffic flow, the human behavioral aspect in certain traffic scenarios is not sufficiently represented for assessing traffic safety. In particular, failures in the cognitive processes of road users, e.g., perception, are commonly reported as a major factor for criticalities and traffic collisions (van Elslande et al. 2008).

Other driver models focus on driver’s perception. An important approach in this context is the tau theory by Lee (1976) or the steering control model by Salvucci and Gray (2004) which basically link visual perception and vehicle control. The second model was further extended to a driver model using a cognitive architecture (Salvucci 2006). These cognitive models are especially useful for representing cognitive processes within humans. However, overall traffic flow, particularly traffic interactions between several road users, has so far not been part of these models.

Until now, there is no holistic driver model available that fulfills the requirements of being able to simulate realistic driving behavior for the fourth approach in the simulation-based safety assessment. A model for this approach shall

• represent the overall traffic flow and traffic interactions realistically,

• shall include behavioral aspect of perception, decision-making and reactions in all kinds of traffic scenarios, and

• shall cover these aspects also in critical situations including behavior that might lead to collisions, e.g. incomplete perception of the situation or incorrect decisions.

Method

Overview of the Stochastic Cognitive Model

For these reasons BMW has been developing the Stochastic Cognitive driver Model (SCM) together with partners since 2014 within the open-source simulation environment openPASS (Witt, Ring et al. 2018, Appendix A1; Witt, Wang et al. 2018; Fries et al. 2022). Composed of six major modules SCM combines the implementation of cognitive processes from information acquisition via the mental representation of the environment to the assessment of the situation and the derivation of actions. These cognition processes take place in the modules Information Acquisition, Mental Model, Situation Manager, Action Manager, Action Implementation. The model is parameterized by stochastic means, which is achieved by the module Driver Characteristics, influencing all other modules. An overview is given in Figure 1.

Figure 1. Overview of the modules in SCM.

Modeling of driver’s gaze behavior with SCM

As most of all relevant information during driving is perceived visually (Sivak 1996), the modeling of human visual perception constitutes an essential part of SCM. The realistic simulation of human gaze behavior is highly relevant, because all subsequent cognitive processes, including situation assessment as well as reaction to the given situation, are highly dependent on the information that comes from the gaze behavior. Thus, only visual data, which is perceived by the Information Acquisition module in SCM, is available for subsequent cognitive processes in SCM. The basic gaze allocation mechanism in SCM is a stochastic process i.e., allocation in a specific area of interest and gaze fixation time in a specific area of interest are sampled from distributions. These distributions were parametrized with real-world driving data. Gaze allocation of SCM is further influenced by top-down and bottom-up mechanisms.

A cognitive processing of visual information is possible by allocating the fovea centralis onto a visual stimulus. The fovea is the (circular) spot of sharpest sight in the center of the retina (line of sight) and is about 2° to 4° wide. Stimuli being presented eccentrically to the line of sight are perceivable in the peripheral field of view (elliptical shape; about 180° wide horizontally and 110° vertically). The quality of extracted information increasingly worsens with further distance between the stimulus and the fovea. However, the periphery is very sensitive toward changes in stimuli (e.g., movement, luminance, color). Therefore, perceived stimuli in the periphery cause an unconscious allocation of the fovea toward them (see e.g., Stapel et al. 2022). These mechanisms are part of the bottom-up (external) control of attention. The counterpart to this is the top-down (internal) control of attention. It is intentionally triggered, when vital information for a task execution is missing or uncertain (e.g., control mirror gazes before a lane change). As especially bottom-up mechanisms are highly relevant for the application in the first scenario presented in this paper (see following section) these mechanisms are further explained subsequently.

Changing fixation spots of the fovea upon gaze targets requires eye movements, which are commonly known as saccades. During a saccade, the visual cortex is not able to perceive any information (not consciously noticeable due to a grey stimulus interpolation), the saccadic suppression. The duration d_sac of a saccade depends on the amplitude Δφ of angular movement of the eyes between gaze targets (Schweigert 2003): (1) $$d_{\mathit{sac}}=21\mathrm{ }\mathrm{ms}+2.2\mathrm{ }\frac{\mathrm{ms}}{\mathrm{degree}}·\Delta \phi$$(1) The saccadic suppression starts already about 75 ms before the start of a saccade and lasts about 50 ms after the end of a saccade (Pannasch 2003). The start of a saccade after the appearance of a stimulus is delayed, which is called saccade latency l_sac. The duration of the saccade latency depends on several factors, generally when the perception threshold of a specific stimulus on the spot of presentation φ on the retina is exceeded. There are several models available for different perception functions, e.g., perception of time-to-collisions (TTC) according to the tau theory (Lee 1976), luminance contrast perception (Adrian 1989), flicker fusion frequency (Ferry 1892; Porter 1898) and their change over φ (Lachenmayr 1995), change of other perception functions over φ according to cortical magnification (Rovamo and Virsu 1979). However, as a simplified approach, Lachenmayr (1995) introduced a regression function for the saccade latency l_sac, which depends only on the amplitude Δφ: (2) $$l_{\mathit{sac}}=242\mathrm{ }\mathrm{ms}+3.59\mathrm{ }\frac{\mathrm{ms}}{\mathrm{degree}}·\Delta \phi$$(2) The gaze allocation time t_gat defines the first part of the driver’s reaction time. It starts with the onset of a visual stimulus in the periphery and lasts until the start of cognitive processing of the visual information after gaze fixation. According to the explanations above, t_gat can be calculated as: (3) $$t_{\mathit{gat}}=l_{\mathit{sac}}+d_{\mathit{sac}}+50\mathrm{ }\mathrm{ms}$$(3) Possible gaze angle amplitudes Δφ of a saccade are within ±90° to the left and right of the fovea because the maximum extend of the periphery is 180° horizontally. The resulting gaze allocation times of this model range from 313 ms up to 834 ms, which is within typical boundaries for human behavior in a traffic context. For instance, Ringhand (2013) executed a human subject study on a proving ground, where subjects were surprised by pop-up obstacles while driving. The gaze allocation times observed for the baseline condition, which meant no assistance by forward collision warnings, ranged from 182 ms up to 819 ms (see Mai 2017).

Application of SCM in critical traffic scenarios

For demonstrating SCM’s capabilities, the model is applied in three critical real-world scenarios which have been identified from BMW’s field operation test data (see Appendix A2 for more information on the selection process of the scenarios). These scenarios are a passive cut-in maneuver, an emergency braking and a distraction scenario. Each scenario is investigated with SCM in 2,000 single simulation runs to account for the variation in the stochastic driver parameters of SCM. Each time SCM is being instantiated in a new simulation run, a new set of driver parameters are sampled from pre-defined distributions in the Driver Characteristics module (see Witt, Wang et al. 2018). These driver parameters (e.g., desired velocity and reaction time) are based on driving data (e.g., FOT data) and literature (regarding reaction times considering for instance from Gold et al. 2013). The objective of this study is to determine SCM’s reactions to the given situation and compare it to the real-world reaction of the human driver. For the application of SCM in simulations BMW uses the open-source simulation platform openPASS (see Appendix A1).

Scenario 1: passive cut-in scenario

Scenario 1 is a passive cut-in maneuver. The ego vehicle (EGO) drives in the left lane of a two-lane German motorway. Another vehicle (OV) passes illegally EGO on the right adjacent lane and starts a lane change to EGO’s lane. Due to a much slower vehicle (SV) on the left lane OV and EGO must perform an emergency braking maneuver during OV’s lane change. The scenario’s criticality is characterized by EGO’s braking acceleration of 8.8 m/s². The initial setup of the vehicles is given in Appendix A3.

There are four relevant points in time within the scenario that are used for the comparison of the FOT data with the SCM simulation namely: the start of the lane change (t_solc), the activation of the indicator (t_ind), the end of lane change (t_eolc) and the start of deceleration by OV (t_decel). In this scenario the end of lane change is defined as the first point of time where OV is completely in the left lane.

In 1,943 simulation runs of total 2,000 runs (97.2%) the SCM agent avoids a collision. In the remaining 57 runs (2.8%) a collision occurred. The velocity over time for both the FOT driver and SCM agents are given in Figure 2. Like the FOT driver the SCM agents start to brake between the points in time t_ind (7.4 s) and t_eolc (10.3 s). Some SCM agents even start to brake before indicator activation. This shows the anticipation capabilities of SCM. These are triggered by the agents’ individual cooperative behavior, the overall situation, and the interpretation of the movement of other traffic participants.

Figure 2. Velocity of FOT driver and SCM agents over time in scenario 1. Start of lane change (t_solc), indicator activation (t_ind), end of lane change (t_eolc) and start of deceleration (t_decel) by OV are marked as reference points.

There is no difference in braking acceleration between collided and non-collided agents, see Figure A4. However, all the agents in simulation independently of the collision status had lower braking accelerations than the FOT driver. Reaction times after OV’s indicator activation are higher for agents with a collision (median of 1.6 s) compared to non-colliding agents (median of 0.6 s). A Kolmogorov-Smirnov test indicates a significant difference at the 5% level. Still, 75% of collided agents have a lower reaction time than the FOT driver, see Figure A5.

For the collision runs a larger overall amount of agents’ gaze fixation to the instrument cluster (IC) is found. Specifically, after the indicator activation (7.4 s) there is a peak of agents looking at the IC in the collision cases (45.6%) compared to the cases without any collision (14.5%), see Figure 3. The described bottom-up mechanism within the gaze behavior of SCM leads to a subsequent peak of gaze allocations to the right due to the stimulus of the overtaking OV.

Figure 3. Distribution of gaze allocation of SCM agents over time for collision cases (n = 57) and no collision cases (n = 1,943) in scenario 1. Start of lane change (t_solc), indicator activation (t_ind), end of lane change (t_eolc) and start of deceleration (t_decel) by OV are marked as reference points.

For those agents looking in the IC at t_ind, a difference in brake reaction times is observable. The colliding agents have higher brake reaction times (median 1.25 s) compared to the non-colliding agents (median 0.8 s), see Figure A6. The Kolmogorov-Smirnov test indicates a significant difference at the 5% level.

For agents looking in the IC at t_decel there is no significant difference in the distribution of brake reaction times. The majority of agents is already braking at that point in time, see Figure A7. However, there is a difference in minimal TTCs for the entire time span in which the agents look at the IC. During this time the minimal TTCs are significantly lower for collided agents (median 1.3 s) compared to agents in the non-collision simulation runs (median 6.9 s), see Figure A8.

Scenario 2: emergency braking scenario

Scenario 2 is initiated by a hard braking maneuver of the vehicle in front (FV) with 5 m/s² in a traffic jam scenario which leads to an emergency braking by the FOT driver (EGO) of 8.7 m/s² and a minimal TTC of 1.5 s. Relevant points in time for this scenario are the first braking by FV at 2.3 s (t_B1) and FV reaching braking accelerations higher than 4 m/s² at 4.2 s (t_B2). The initial setup of the vehicles is given in Appendix A4.

Out of the 2,000 simulation runs, 1,991 SCM agents prevent a collision and 9 SCM agents collide with FV. The velocity over time for both the FOT driver and SCM agents are given in Figure 4. Like the FOT driver 96% SCM agents start to brake between t_B1 and t_B2. Some SCM agents start to brake earlier to already increase the following distance.

Figure 4. Velocity of FOT driver and SCM agents over time in scenario 2. First braking of FV (t_B1) and FV reaching braking accelerations higher than 4 m/s² (t_B2) are marked as reference points.

Regarding braking accelerations, colliding SCM agents have significantly higher braking accelerations than SCM agents that prevent a collision, see Figure A11. The reason might be that colliding SCM agents have a longer reaction time. Their braking start later (median 3.1 s after start of braking by FV) than the non-colliding agents (median 1.7 s), see Figure A12. A Kolmogorov-Smirnov test indicates a significant difference between the distributions at the 5% level. The colliding SCM agents encounter a more critical situation and thus SCM reacts with stronger braking.

Regarding gaze behavior, all nine colliding agents are visually distracted after FV starts braking, see Figure 5. The difference to those non-colliding agents is the duration of distraction while FV is braking. Durations for EGO’s distraction are lower for non-colliding agents (median 0.8 s) than for colliding agents (median 1.5 s), see Figure A13. A Kolmogorov-Smirnov test indicates a significant difference at the 5% level.

Figure 5. Distribution of gaze allocation of SCM agents over time for no collision cases (n = 1,991) and collision cases (n = 9) in scenario 2. First braking of FV (t_B1) and FV reaching braking accelerations higher than 4 m/s² (t_B2) are marked as reference points.

Scenario 3: distraction scenario

Scenario 3 is a scenario with a distracted FOT driver (EGO) while approaching a much slower front vehicle (FV). This leads to a hard braking maneuver of EGO when recognizing FV after looking back in the front area. EGO’s braking acceleration is 6.0 m/s². Thus, in terms of EGO’s reaction it is the least critical scenario. The initial setup of the vehicles is given in Appendix A5. For the simulation of the FOT driver’s distraction, SCM’s gaze behavior and velocity is externally controlled until the point of time when the FOT driver looks back to the front area at 4,4 s (t_FA) to ensure comparability.

In the 2,000 simulation runs, all SCM agents prevented a collision, which underlines the lower criticality of the scenario. The velocity over time for both the FOT driver and SCM agents are given in Figure 6. Braking accelerations of all SCM agents are between 6.7 and 7.4 m/s² compared to 6.0 m/s² for the FOT driver. Brake reaction times of the SCM agents range between 0.4 and 1.2 s. The FOT driver’s brake reaction time is 0.7 s. Minimal TTCs for the scenario range from 2.1 s to 3.0 s for the SCM agents compared to 2.7 s for the FOT driver, see Figures A16–A18 in the Appendix.

Figure 6. Velocity of FOT driver and SCM agents over time in scenario 3. The FOT driver looking back to the front area (t_FA) is marked as a reference point.

Discussion and conclusion

The vast majority of the SCM agents can cope with the selected critical FOT scenarios as the real drivers and avoid a collision (collision rate in simulation: 1^st scenario 2.8%, 2^nd scenario 0.5% and 3^rd scenario 0%). This demonstrates that SCM can deal with critical scenarios. However, some agents collide in scenario 1 and 2. Results indicate that collisions are caused by visual distraction or having longer reaction times compared to non-collided agents or both. Not surprisingly, numerous authors found that visual inattention, caused by internal (e.g., in-vehicle devices) or external elements (e.g., billboards) is linked to a decrease in the perception of situational changes (Metz et al. 2011) and to a deterioration of driving performance (Kaber et al. 2012; Nabatilan et al. 2012). The results by Metz et al. (2011) indicate that an inappropriate distribution of the visual attention, i.e., fewer and shorter road glances, might lead to a higher risk in collisions. Victor et al. (2015) found that gaze behavior for crashes and near-crashes involved higher percentages of eyes-off-path glances a few seconds before the crash or critical event occurred compared to the examined baseline. These findings are highly consistent to the behavior that is shown by the colliding SCM agents. However, the percentage of colliding SCM agents cannot be validated. Firstly, different FOT drivers do not encounter the same scenario and secondly collisions are rare events which cannot be observed in the underlying FOT data. For further validation of this aspect separate studies, e.g., in a driving simulator, are necessary to account for these limitations.

Results indicate that some agents can anticipate critical situations ahead as they are reacting before the cut-in maneuver of the other vehicle and decelerating (scenario 1) or trying to increase the following distance before the braking of the front vehicle (scenario 2). However, as braking accelerations for all SCM agents in scenario 1 remain lower compared to the FOT driver and in scenario 3 are higher compared to the FOT driver, the question arises if the calculation for the intensity of braking is sufficient in the SCM model or whether further enhancements need to be made.

Overall, it can be concluded that the SCM driver behavior model is able to model and replicate real world driving behavior mechanisms in critical scenarios which is shown by three exemplary FOT scenarios. The focus of this paper is on gaze behavior. Effects of gaze behavior differences over a virtual driver population can be simulated, making virtual assessments and the impact of their cognitive processes, particularly perception, in traffic scenarios possible. Still, SCM visual distraction is momentarily simulated only by gaze fixations on the instrument cluster. Here, future developments regarding modeling other visual inattention causes might be beneficial to improve perception and reaction in critical situations.

It is important to note that a comprehensive validation of SCM requires further analyses—also in other driving scenarios measured during FOT or in drone data and in crash scenarios from German in-depth crash database GIDAS PCM. An analysis of SCM’s behavior in cut-in maneuvers compared to real world accidents is for instance given in Fries et al. (2022). SCM is a quite complex driver behavior model, which cannot be described completely at detailed level in one paper. Thus, this paper focuses only on one aspect. Nevertheless, to make the model transparent for future research and development as well as to foster the scientific discussion about driver behavior models for simulation, the SCM model will be published as an open-source model in 2023.

Disclosure statement

The authors declare we have no conflict of interest to disclose.

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.