Improvement of driver active interventions during automated driving by displaying trajectory pointers—A driving simulator study

Abstract

Objective: The handover of vehicle control from automated to manual operation is a critical aspect of interaction between drivers and automated driving systems (ADS). In some cases, it is possible that the ADS may fail to detect an object. In this event, the driver must be aware of the situation and resume control of the vehicle without assistance from the system. Consequently, the driver must fulfill the following 2 main roles while driving: (1) monitor the vehicle trajectory and surrounding traffic environment and (2) actively take over vehicle control if the driver identifies a potential issue along the trajectory. An effective human–machine interface (HMI) is required that enables the driver to fulfill these roles. This article proposes an HMI that constantly indicates the future position of the vehicle.

Methods: This research used the Toyota Dynamic Driving Simulator to evaluate the effect of the proposed HMI and compares the proposed HMI with an HMI that notifies the driver when the vehicle trajectory changes. A total of 48 test subjects were divided into 2 groups of 24: One group used the HMI that constantly indicated the future position of the vehicle and the other group used the HMI that provided information when the vehicle trajectory changed.

The following instructions were given to the test subjects: (1) to not hold the steering wheel and to allow the vehicle to drive itself, (2) to constantly monitor the surrounding traffic environment because the functions of the ADS are limited, and (3) to take over driving if necessary.

The driving simulator experiments were composed of an initial 10-min acclimatization period and a 10-min evaluation period. Approximately 10 min after the start of the evaluation period, a scenario occurred in which the ADS failed to detect an object on the vehicle trajectory, potentially resulting in a collision if the driver did not actively take over control and manually avoid the object.

Results: The collision avoidance rate of the HMI that constantly indicated the future position of the vehicle was higher than that of the HMI that notified the driver of trajectory changes, χ² = 6.38, P < .05. The steering wheel hands-on and steering override timings were also faster with the proposed HMI (t test; P < .05).

Conclusions: This research confirmed that constantly indicating the position of the vehicle several seconds in the future facilitates active driver intervention when an ADS is in operation.

Keywords:

• Automated driving
• active interventions
• human machine interface
• sense of agency
• driving simulator

Introduction

Automated driving systems (ADS) are being actively developed as a promising means of helping to reduce traffic accidents, alleviating congestion, and facilitating the freedom of movement of all members of society (Cabinet Office, Government of Japan 2018).

Under the definitions established by the Society of Automotive Engineers (SAE) International, a level 2 ADS requires the driver to intervene and take over operation when the system cannot continue operating the vehicle safely (SAE International 2014). For example, if an object is detected on the road and the ADS determines that automated operation cannot be safely continued, the system will prompt the driver to resume control of the vehicle. Faced with this type of takeover request, the driver must immediately grasp the situation and perform the appropriate driving task, such as slowing down or maneuvering the vehicle. For this reason, the handover of vehicle control from automated to manual operation is a critical aspect of research into the interaction between drivers and ADS.

Eriksson and Stanton (2017) summarized research related to takeover requests and reported that drivers require 2.96 (±1.96) s to respond to a request. Takeover requests can be sent to the driver visually, audibly, or haptically (Eriksson and Stanton 2017; Ho et al. 2005; Politis et al. 2014). It was also found that the method of notification affected the driver’s evaluation of the request (Dzindolet et al. 2003; Tokushige et al. 2017; Walch et al. 2015).

The methods described above require the driver to grasp the situation immediately after the request and take over driving. However, the complexity of actual driving environments makes it difficult for drivers to decide the appropriate action immediately after receiving a takeover request from the system. Therefore, notification methods that present an instruction at the same time as the takeover request have been proposed (Borojeni et al. 2016; Ho et al. 2005; van Veen et al. 2017).

However, because the detection of objects on the road by ADS and localization using a Global Positioning System are not infallible processes, in some cases ADS may fail to detect an object on the road. In this event, ADS will be unable to either provide the driver with information about the object or make a takeover request. Therefore, the driver is required to identify the existence of potential collision objects independent of the system and, if an undetected object is present, actively judge whether manual control must be resumed to avoid the object. Consequently, the driver must fulfill the following 2 main roles when using an ADS:

1. Monitor the vehicle trajectory and surrounding traffic environment.

2. Actively take over control of the vehicle if a potential issue is identified along the trajectory.

This article describes the results of tests designed to validate the effectiveness of constantly indicating the future location of the vehicle as support to facilitate roles 1 and 2.

Constant indication of the future location of the vehicle refers to the continuous display of the vehicle location at a set point in time in the future of the vehicle trajectory planned by the ADS. By indicating the vehicle’s location at multiple points, the driver can be informed of the vehicle speed, acceleration, and planned maneuvers a short time in the future.

Wegner (2002) stated that if an actual causal relationship can be interpreted as the cause of an action—that is, if a thought is consistent with a succeeding action—people perceive a sense of responsibility for that action and obtain what is known as a “sense of agency.” This research found that people perceived themselves to be responsible for an action or result if it was preceded by a thought consistent with that action or result, even if the person did not actually perform a concrete action (Wegner et al. 2004; Wegner and Wheatley 1999). Based on this idea, constantly indicating the future position of the vehicle might create a sense of agency in the driver with respect to the operations of the ADS.

Roles 1 and 2 are fulfilled naturally by the driver when driving manually. If the driver experiences a sense of agency with respect to the operations of the ADS, it may be possible to facilitate the performance of roles 1 and 2 even when the driver is not manually driving the vehicle.

Kondo et al. (2018) used a simulator consisting of a computer game steering wheel and a personal computer to superimpose the future location of the vehicle onto the road ahead. This research concluded that this system made the driving plan of the ADS easier to understand and also reduced the number of unnecessary driver interventions.

Allowing drivers to compare the position of the vehicle a short time in the future with the status of the surrounding traffic environment more easily may facilitate awareness of issues in the traffic environment and judgments about the necessity of actively taking over control. This hypothesis was validated based on the following paradigm.

As a result of properly fulfilling role 1 above, drivers who have access to a method of predicting the status of the vehicle a short time in the future have a significantly higher success rate of avoiding objects by actively taking over control of the vehicle when, for example, the ADS fails to detect an object on the road than drivers without such access. Consequently, a system that constantly indicates the vehicle location a short time in the future should also help the driver to fulfill role 2.

Method

Toyota Dynamic Driving Simulator

The experiments that were carried out to validate this hypothesis used the Toyota Dynamic Driving Simulator (TDDS; Murano et al. 2009). The TDDS, which is equipped with the cabin of a Lexus HS and can be operated by the driver, is installed within a dome (diameter: 7 m) that surrounds the vehicle and covers the full 360° field of view via 6 projectors (BARCO F70-4K6). The dome creates a highly realistic driving experience by moving and tilting in accordance with driver operations to generate physical sensations of acceleration.

Human–machine interfaces

Continuous trajectory pointers

The continuous trajectory pointer (CTP)-based human–machine interface (HMI) indicated the future position of the vehicle using a continuous series of markings called CTPs while the ADS was in use. In a standard state, 3 CTPs were indicated one above the other in vertical alignment. In these experiments, the CTPs indicated the lateral position of the driver’s vehicle 1, 2, and 3 s in the future, respectively.

The CTPs were indicated above an icon representing the front of the driver’s vehicle, between white lines representing the left and right edges of the vehicle’s lane. Figure 1 shows an example of the CTP-based HMI display when the vehicle avoided an object on the road. When an object is detected, the HMI shows it as a yellow triangle when the time-to-collision (TTC) reaches 8.2 s. Icons b to g show the sequence of displays that occur when the vehicle avoids an object. If the ADS does not detect an object, the HMI continues to show icon a.

Figure 1. Example of display transition of CTP-based HMI when avoiding an object on the road.

Notification of trajectory changes

In contrast, the notification of trajectory changes (NTC)-based HMI used the start and end points of arrows to indicate the current position of the driver’s vehicle and its lateral position after steering control when the planned trajectory of the ADS changed. Figure 2 shows an example of the NTC-based HMI display when a lane offset occurred to avoid the object.

Figure 2. Example of display transition of NTC-based HMI when avoiding an object on the road.

Similar to the CTP-based HMI, the white lines representing the left and right edges of the vehicle’s lane were indicated constantly while the ADS was in use. The NTC system also shows a yellow triangle and the arrow in the same timing as the CTP-based HMI when an object is detected. Icons b to g show the same sequences as Figure 1, and if the ADS does not detect an object, it shows icon a.

Apparatus

The 2 HMIs were indicated at an appropriate location on the head-up display. This head-up display indication was projected onto the screen in front of the driver via a projector (CANON WUX600) installed on the roof of the Lexus HS. In each case, the size of the HMI was approximately 200 × 140 mm.

Participants

This investigation was approved and conducted according to the Ethical Guidelines for Research defined by Toyota Motor Corporation (2014). Informed consent was obtained from all participants, including details of the experimental procedures and privacy protection policies.

A total of 48 people were recruited from within the company to participate in the investigation. The average age of the participants was 36.3 ± 13.3 years. The youngest participant was 19 years old and the oldest was 64 years old. Of the 48 participants, 41 were male, and all drove at least 3 days a week.

The 48 participants were divided into 2 equal groups of 24 with similar representations of age and gender. One group tested the CTP-based HMI and the other tested the NTC-based HMI. Data for 4 out of the 48 participants were discarded from the analysis due to abnormalities in the obtained values. Consequently, the CTP-based HMI was analyzed using data from 21 participants and the NTC-based HMI was analyzed using data from 23 participants.

Design

The following instructions were given to all participants before driving the simulator: (1) to not hold the steering wheel and to allow the vehicle to drive itself, (2) to constantly monitor the surrounding traffic environment because the functions of the ADS are limited, and (3) to grasp the steering wheel and take over driving if necessary.

The driving simulator experiments using the ADS lasted approximately 20 min and simulated a 25-km stretch of an urban expressway with two lanes in each direction (Figure 3). The first 10 min of the experiment was defined as an acclimatization period in which the ADS followed preceding vehicles, overtook slower vehicles, and passed safely through branching and merging points. The purpose of this period was to demonstrate via actual operations that control can be returned to the driver by turning the steering wheel or depressing the pedals even during automated driving.

Figure 3. Experiment route (all participants followed the same route).

After acclimatization, the route transitioned seamlessly into a 10-min evaluation period. During the evaluation period, the driver encountered various scenarios in which the ADS successfully overtook slower vehicles and avoided an object on the road. Approximately 10 min after the start of the evaluation period, a scenario occurred in which the ADS failed to detect an object in the vehicle’s trajectory, potentially resulting in a collision if the driver did not actively take over control and manually avoid the object. The collision avoidance rate, steering wheel hands-on timing, and steering override timing were analyzed in this scenario.

Procedure

Before the experiment started, participants completed a questionnaire (5 min) and were given instructions about the experiment by an operator (10 min). Participants then performed the acclimatization and evaluation sections of the experiment (20 min) and completed another questionnaire after the experiment ended (10 min).

Instructions from the operator included explanations about how to operate the vehicle, necessary displays such as the vehicle speed, and the operation of the ADS. The participants were also given cautions about the ADS (see Design subsection).

Participants entered the TDDS after receiving these instructions. During the acclimatization period, each participant followed the operator’s instructions to experience automated driving. This allowed the participants to become sufficiently accustomed to the operation of the ADS and vehicle within a short space of time.

Analysis

At the end of the evaluation route, a scenario was arranged in which the ADS failed to detect an object in the vehicle’s trajectory, potentially resulting in a collision if the participant did not actively take over control and manually avoid the object. In this scenario, the following avoidance performance indices were compared between the group using the CTP-based HMI and the group using the NTC-based HMI.

Object avoidance rate

The collision avoidance rate was compared based on whether or not driver intervention enabled the vehicle to avoid a collision with the object on the road. A significant rate of successful avoidance was defined as a risk rate of 0.05 or less calculated using a chi-square test.

Steering wheel hands-on timing and steering override timing

Participants were determined to be holding the steering wheel when the flag of the steering wheel capacitance sensor in the TDDS recorded a value of 1. Participants were determined to have taken over steering when the input torque to the steering wheel exceeded a predetermined threshold value (1.3 Nm).

In the hands-on and steering override timing comparisons, a significant difference between HMIs (in this case, the difference in the TTC between the vehicle and the object when the driver judged it necessary to hold the steering wheel and override the steering, respectively) was defined as a risk rate of 0.05 or less calculated using a t test.

Results

All participants were driving without holding the steering wheel when the potential object collision scenario occurred.

Figures 4 and 5 shows the cumulative results for the collision avoidance rate, hands on steering wheel, and override timing.

Figure 4. Cumulative total of collisions.

Figure 5. Steering wheel hands-on timing and steering override timing (average and SD).

In the case of the NTC-based HMI, 13 of the 23 applicable participants avoided a collision. In contrast, 19 of the 21 applicable participants in the CTP-based HMI group avoided the object. When the number of participants who successfully avoided the collision was compared using the chi-square test, the collision avoidance rate of the CTP-based HMI was significantly higher, χ² = 6.38, P < .05.

The average TTC when the NTC-based HMI participant group grasped the steering wheel was 2.20 s. In contrast, the average TTC of the CTP-based HMI group was 4.02 s. Results of the t test showed that the steering wheel hands-on timing was significantly faster with the CTP-based HMI, t(35) = 2.55, P < .05. The analysis discarded data from 2 participants who were holding the steering wheel when the object was seen.

In addition, the average TTC after steering override was 2.02 s with the NTC-based HMI and 3.11 s with the CTP-based HMI. Similar to the results for the steering wheel hands-on timing, the t test found that steering override was performed significantly faster with the CTP-based HMI, t(42) = 2.24, P < .05.

In conclusion, this research confirmed that constantly indicating the position of the vehicle several seconds in the future facilitates active driver intervention when an ADS is in operation.

Discussion

The CTP- and NTC-based HMIs displayed warnings about evasive maneuvers at the same timing. Therefore, as the vehicle approached the final undetected object scenario, the absence of an evasive maneuver warning before that timing should have made the drivers indirectly aware of the possibility that the ADS might not avoid the object. However, although all drivers should have recognized the possibility of collision, the average hands-on and steering timings differed by more than 1 s depending on the HMI. Consequently, constant indication may be regarded as a more effective means of facilitating driver judgment of a potential system failure to avoid an object than periodic indication.

Possible reasons for this are as follows. Drivers using the CTP-based HMI were able to determine a potential collision because the CTPs did not show the vehicle going around the object. In contrast, drivers using the NTC-based HMI were not notified of the vehicle’s trajectory and only realized that the ADS could not avoid the object when the vehicle was facing an impending collision. This difference facilitates driver judgment that the ADS has failed to detect an object on the road. However, it will be necessary to investigate whether differences in the sense of agency generated by the HMIs was responsible for this result.

Topics for future research include investigating whether the results described above are sustainable after the driver becomes fully accustomed to the ADS and CTP-based HMI, identifying a suitable validation method, and evaluating the extent of the sense of agency generated by the HMIs.