Concept of an enhanced V2X pedestrian collision avoidance system with a cost function–based pedestrian model

ABSTRACT

Objective: State-of-the-art collision avoidance and collision mitigation systems predict the behavior of pedestrians based on trivial models that assume a constant acceleration or velocity. New sources of sensor information—for example, smart devices such as smartphones, tablets, smartwatches, etc.—can support enhanced pedestrian behavior models. The objective of this article is the development and implementation of a V2Xpedestrian collision avoidance system that uses new information sources.

Methods: A literature review of existing state-of-the-art pedestrian collision avoidance systems, pedestrian behavior models in advanced driver assistance systems (ADAS), and traffic simulations is conducted together with an analysis of existing studies on typical pedestrian patterns in traffic. Based on this analysis, possible parameters for predicting pedestrian behavior were investigated. The results led to new requirements from which a concept was developed and implemented.

Results: The analysis of typical pedestrian behavior patterns in traffic situations showed the complexity of predicting pedestrian behavior. Requirements for an improved behavior prediction were derived. A concept for a V2X collision avoidance system, based on a cost function that predicts pedestrian near future presence, and its implementation is presented. The concept presented considers several challenges such as information privacy, inaccuracies of the localization, and inaccuracies of the prediction.

Conclusion: A concept for an enhanced V2X pedestrian collision avoidance system was developed and introduced. The concept uses new information sources such as smart devices to improve the prediction of the pedestrian's presence in the near future and considers challenges that come along with the usage of these information sources.

KEYWORDS:

• Active safety
• vehicle-to-vehicle communication
• crash avoidance
• driver assistance systems
• pedestrian protection

Introduction

Since 2007, approximately 1.25 million people have died each year in traffic accidents worldwide (World Health Organization [WHO] 2015). Half of these deaths are pedestrians, cyclists, or motorcyclists (WHO 2015). This group of traffic participants is called vulnerable road users (VRUs). Prediction estimates indicate that traffic accidents will rise to become the seventh leading cause of death by 2030 if no suitable actions are taken (WHO 2015). To counteract this trend, the United Nations (2015) adopted the 2030 Agenda for Sustainable Development and set a road safety target of halving the global number of deaths and injuries due to traffic accidents by 2020. Collision avoidance systems are necessary to meet this target. Advanced systems based on braking (automated emergency braking) have already shown a positive impact on collision avoidance and mitigation, contributing to the reduction in collisions and mitigation of their results (Shibata et al. 2014). However, in order to meet the set target described above, additional intelligent systems that evade or mitigate accidents by steering are also necessary.

In Germany, 9.2% of all road traffic accidents in 2014 involved pedestrians (Statistisches Bundesamt 2015). In 2015, pedestrian deaths accounted for 14% of all traffic fatalities in the United States (NHTSA 2015). In Germany, more than 40% of all pedestrian–vehicle accidents resulted from a line-of-sight obstruction (Rasshofer et al. 2009). Most state-of-the-art collision avoidance systems are not able to detect pedestrians without a line-of-sight to the pedestrian. New systems that use vehicle-to-pedestrian (V2P) or vehicle-to-infrastructure (V2I) communication, such as the ones introduced in Westhofen et al. (2012), Pudenz (2012), Bawaba (2013), and Anaya et al. (2014), are able to react to pedestrians without direct sight; for example, if the pedestrian is waiting behind a vehicle.

In addition to improved detection of pedestrians, behavior and movement prediction is essential to further improve collision safety. Themann et al. (2015) showed that the prediction accuracy of the future VRU position for vehicle velocities of 50 km/h should provide a standard deviation of less than 55 cm. Today's systems, as well as the previously mentioned V2P or V2I systems, predict the future position of pedestrians based on the assumptions that pedestrians will maintain a constant velocity or acceleration and heading (Westhofen et al. 2012). Many studies on the behavior of pedestrians showed that several influence pedestrians' decisions and movements. In their patent, David and Flach (2011) suggested the use of smart device information to improve the prediction of pedestrian behavior. By 2020, 6.1 billion smartphone users are expected globally. This equals 70% of the world's population, showing the potential of this new information source (Ericsson 2016). Smart devices, with their included sensors and user interface, can provide information about the current movement state (velocity, acceleration, and heading) and the position and characteristics of their users, such as age, gender, habits, and so on.

State of the art

The short-term prediction of pedestrian positions and trajectories is difficult because human behavior is affected by different observable and unobservable influences. Today's driver assistance systems are only equipped with on-board sensors to predict this behavior. Thus, only the current position, speed, heading, and acceleration of pedestrians in line-of-sight (in certain cases also in situations without line-of-sight) and within the detection range of on-board sensors can be used for the prediction. Advanced systems—for example, those introduced in Köhler et al. (2013)—additionally use the potential of head and foot positions for the prediction. New information sources allow new prediction models to use additional information about pedestrians that is not directly measurable by on-board sensors of the vehicle. For this purpose, it is necessary to understand what information and of what quality supports the prediction of pedestrian behavior.

The pedestrian decision-making process can be distinguished into strategic, tactical, and operational levels (Daamen 2004; Hoogendoorn and Bovy 2004). Long-term decisions, mid-term decisions, and immediate decisions are made at the strategic, tactical, and operative levels, respectively. Collision avoidance systems focus on the prediction of other traffic participants within the next seconds and therefore need prediction models operating on the tactical and operative levels. Information at the strategic level can support the prediction on the other two levels as valuable input.

Studies show that diverging factors, on different levels, influence pedestrians' decisions. The results of these studies are stored in a database of relevant pedestrian characteristics within the framework of the presented work. In the following, the main findings and a resulting concept for an improved collision avoidance system are presented.

The age of the pedestrian is an important factor to be considered in prediction modeling. Several studies investigated the impact of pedestrian age on pedestrian speed, acceleration, and the decision whether to cross a street or not. Figure 1 shows the effect of gender and age on pedestrian speeds. The figure is an extension of the work from Bartels and Erbsmehl (2014).

Figure 1. Pedestrian speeds based on gender and age (m = male/f = female). Sources: EBE (Eberhardt and Himbert 1977); KRA (Kramer and Raddatz 2010); BUR (Burg and Moser 2009); ZEB (Zebala et al. 2012); VAU (Vaughan and Bain 2000); BAR (Bartels and Liers 2014).

In addition, physical performance, response capacity (Fugger et al. 2000), and risk appetite decrease with age (Hamed 2001). Risk appetite influences, for example, the minimum gap between pedestrian and vehicle at which pedestrians will cross (Hamed 2001). Figure 2 shows the average speeds based on pedestrian pace, age, and gender. The figure is an extension of the work from Bartels and Erbsmehl (2014).

Figure 2. Average speeds depending on pace, age, and gender (m = male/f = female). Sources: Eberhardt and Himbert (1977), Kramer and Raddatz (2010), Burg and Moser (2009), Zebala et al. (2012), Vaughan and Bain (2000), Bartels and Liers (2014).

It can be seen that the influence of pace on speed is strongly dependent on age. The effect of pace peaks around age 25 and then decreases with age. Gender has a small influence on pedestrian speed (e.g., 0.1–1.13 m/s at a sprinting pace), but male pedestrians cross faster after traffic lights switch from red to green than females. Male pedestrians in aged between 14 and 64 cross, on average, 1.83 s after the light switches, whereas female pedestrians in this age range cross after 2.01 s. In comparison, over the age of 65, males cross on average after 2.39 s and females after 2.57 s (Knoblauch et al. 2007). Similar influences are also observable for acceleration and rotation behavior. Furthermore, crossing speed also depends on the interacting vehicle size, street width, weather conditions, number of pedestrians that cross a street together (Bartels and Erbsmehl 2014; Gates et al. 2006; Knoblauch et al. 2007), traffic light cycle, parked cars, and infrastructure (Weidmann 1992; Zhang et al. 2009). Additional factors, such as physical restrictions (Eberhardt and Himbert 1977), luggage (Bartels and Liers 2014; Finnis and Walton 2007; Rahman et al. 2012), moving direction (Eberhardt and Himbert 1977), travel purpose (Bartels and Liers 2014; Weidmann 1992), use of consumer electronics, pedestrian density (Fitzpatrick et al. 2006; Hagen et al. 2012; Weidmann 1992), lighting (Fitzpatrick et al. 2006), and country of origin (Goh et al. 2012; Rahman et al. 2012), influence pedestrians' movements.

The decision whether to cross a road or not also depends on many factors. Referring to Schnadt and Schübbe (1975), pedestrians act according to a Reiz-Person-Verhaltensmodell or “stimuli-person behavior model,” in that situational and environmental stimuli trigger reactions. Situational stimuli include traffic density (Lenhoff 2004; Schnadt and Schübbe 1975), interaction partner configuration (e.g., speed, distance, vehicle type; Kotte 2016), length of red phase (Lehnhoff 2004), group size (Schnadt and Schübbe 1975), weather, and infrastructure (Hagen et al. 2012). Personal factors that influence the decision are experience, knowledge, individual characteristics (e.g., age; Hagen et al. 2012; Oxley et al. 2005; Petzoldt 2014), risk appetite (Hamed 2001), near-term motivations, and attention (Hatfield and Murphy 2007; Pešić et al. 2015; Schnadt and Schübbe 1975).

In addition to prediction of pedestrian behavior based on a complex model, it is possible to detect pedestrians' intention and use this to better predict whether the pedestrian crosses or not. In particular, the evaluation of head position (Kloeden et al. 2014) and foot movement (Köhler et al. 2013) shows high potential for this task.

Methods

New sensor information sources can be used to improve the prediction of pedestrian presence in the near future due to available information such as age, past behavior (e.g., pedestrian speed), etc. However, the prediction of human behavior is difficult due to large variabilities (Burgett et al. 2008). The influence of different impact factors must be studied to develop suitable behavior prediction models (Carsten and Nilsson 2001). Due to the large number of factors and their dependencies among each other, the methodology shown in Figure 3 was developed and applied. This article focuses on the main findings of the literature review, requirement definition, concept development, and implementation.

Figure 3. Overview of methodology.

A literature review on studies focusing on state-of-the-art collision avoidance systems, new sensor information sources, and studies on pedestrian behaviors was the first step. The results of the analyzed studies were stored in a database of relevant pedestrian characteristics and further influence factors, not yet adequately investigated, were identified.

One important factor that has not been investigated in detail yet is pedestrian–vehicle interaction. This factor was analyzed through traffic observations and a controlled environment study (Kotte 2016). The results are included in the database of relevant pedestrian characteristics. Based on this database, requirements for an enhanced vehicle-to-everything (V2X) pedestrian collision avoidance system and an improved pedestrian prediction model are derived. A concept based on these requirements is developed and implemented. The developed pedestrian model is parameterized based on the relevant pedestrian characteristics in the database. Subsequent work will be needed to analyze the potential of the proposed enhanced system.

Results

Analysis of all factors shows how complex the prediction of pedestrian behavior and movement is and why it is important to analyze the relationship among each of these factors. Many factors mentioned above can be determined and aggregated with today's and future technologies so that the information can be used to improve short-term prediction of a pedestrian's presence (up to 3 s) in an enhanced V2X pedestrian collision avoidance system. Therefore, different information sources must be combined. Because personal information such as age, gender, and risk appetite is also important, the enhanced V2X pedestrian safety system concept must ensure privacy. Furthermore, the concept must be able to use different information sources and be capable of handling the varying availability and quality of this information. Due to the complexity of the prediction, the output must be combined with a probability. This must be considered in the trajectory planning approach to calculate the best avoidance or mitigation maneuver (braking, steering, or a combination of both). Due to high inaccuracies in the position determination with GNSS sensors, which are included in smart devices today, the concept should have the ability to match information from these devices with information from environmental sensors to obtain reliable position information for vehicle interventions.

A concept that meets these requirements is introduced and shown in Figure 4. It consists^{Figure 5}of the three main elements: pedestrian's smart device, cloud and roadside unit, as well as vehicle. To ensure privacy, prediction of the pedestrian's position is calculated based on the pedestrian's smart device. This has the advantage that private information can be considered without the necessity to stream it to the environment. The output of the smart device is an array of possible positions together with a probability, called the probability of the pedestrian's presence, combined with the current heading and global position of the pedestrian. This approach has two additional advantages. On the one hand, the vehicle only has to consider the information about pedestrians who will enter its path with a defined probability. Thus, the computational power can be decreased in situations with many pedestrians around, because this distributed approach uses the computational power of each smart device to predict the position of its own user. On the other hand, the update rate of smart devices is much faster than that of vehicle systems so that their performance can be adapted much faster to updates in hardware. Therefore, it is important that the output of the smart device can be used easily in standard in-vehicle environmental models. To be able to consider the vehicle–pedestrian interaction in the prediction, information sent by the vehicle is also considered.

Figure 4. Overview of overall concept.

Inside the vehicle, a wrapper is necessary to transform the probability of the pedestrian's presence into the local environmental model of the vehicle. For this purpose, the global position information of the pedestrian, provided by the smart device, must be converted into the local coordinate system of the vehicle. An intelligent matching algorithm is necessary to match the information from the smart device with the correct object of the vehicle's environmental sensors. Due to high inaccuracies in the GNSS-based pedestrian position information and the possibility that pedestrians walk close to each other, the matching algorithm must evaluate the position information along with the pedestrian's recent trajectory history. The same approach can be used in the future to enable faster object classification by vehicle sensors by fusion on the raw data level. Based on the combined information in the environmental model, suitable vehicle trajectories can be determined.

A connection between the cloud or roadside unit and the smart device can provide information about critical locations and paths as well as behaviors of other pedestrians at this location. Because roadside units will not be available at all locations, the concept must be able to also predict the behavior without this information. However, particularly at locations with many pedestrian accidents, roadside units are conceivable. They can also be used to determine the position of pedestrians with a high degree of accuracy even if they are not in line of sight with the vehicle sensors and can therefore improve safety at accident hotspots. Thus, location-specific patterns can be considered in the pedestrian prediction.

The prediction of the pedestrian's future position is based on a cost function that determines the cost to reach a possible position inside the pedestrian's movement area. Costs are determined and parameterized based on the information from the relevant pedestrian characteristics database and interaction parameters, such as the current vehicle's deceleration to safety time (Hupfer 1997), as well as the type of interacting vehicle. The pedestrian's movement area is defined by the physiological ability to move and is calculated by considering maximum accelerations, speeds, and turning rates for the current pedestrian speed and heading (Zecha and Rasshofer 2009). If the information of one cost portion is not available—for example, if no roadside unit is installed at a location—these costs are not considered. Therefore, it is an extension of the weighted motion model introduced in Westhofen et al. (2012).

Westhofen et al. (2012) determined the probability field of the pedestrian's presence by evaluating only the current heading and speed. Two examples for the probability of the pedestrian's presence calculated using this article's approach are shown in Figure 5. On the left-hand side, the probability of the pedestrian's presence at 3 s in the future for a desired speed of 0 m/s is shown. On the right, the probability for a desired speed of 1.5 m/s is shown. The intention detection evaluates smart device sensors to adapt the weightings of the costs in case of a detected intention to cross (leg movement) or a detected interaction with the driver (head position).

Figure 5. Example for the probability of the pedestrian's presence 3 s in the future.

The figure shows the^{Figure 6}movement areas, which are the same for both situations and the probabilities of the pedestrian's presence. Dark areas represent areas with a high probability of presence after 3 s, whereas bright areas represent areas with low probabilities. The influence of the desired speed is clearly seen in the figure.

The usage of probability values for pedestrians' presence is not common in current active safety systems. One approach that considers probability values is introduced in Westhofen et al. (2012), where areas that can be reached with the current speed and heading are weighted with higher probability values. The situation risk in this approach is calculated based on an evaluation of the intersection area between the movement area of the vehicle and the movement area of the pedestrian. This approach is efficient but has disadvantages if additional impact parameters such as impact speed and position are considered in the trajectory planning and if evasion maneuvers are possible.

Within the FP7 European Community–funded project AsPeCSS, injury risk curves were developed that describe the impact of collision speed on the probability of injuries in pedestrian accidents based on an analysis of German In-depth Accident Study accident data (Edwards et al. 2014). To be able to include this information and to determine evasion and combined maneuvers (braking and steering), a trajectory planning approach was implemented in our test vehicle that calculates the time to react for different predefined maneuvers (braking, evade left, evade right, or combined maneuvers). Several pedestrians' presence areas with a specific probability (e.g., 90%, 80%, 70%, …,) and the collision speed, if a collision occurs, are evaluated to determine the best suitable trajectory for this situation. In the end, the trajectory with the best available time to react (latest intervention time), if the accident can be avoided or, in case the accident can only be mitigated, the trajectory with the lowest injury risk (considering injury risk curves and probabilities of presence) is chosen. The available room for the trajectory planning is provided by the environmental model. An example of the trajectory planning is shown in Figure 6.

Figure 6. Determination of suitable vehicle maneuver.

Misuse and false positives will be avoided by redundancy of the critical information and safety mechanism. Due to high inaccuracies of today's GNSS systems, a critical vehicle maneuver (e.g., harsh braking) will only be triggered if the pedestrian's position is also provided by a second information source (e.g., vehicle or infrastructure sensors) and if the values of a calculated information quality vector are high enough. This vector includes necessary information and provides it to the decision level of the vehicle (e.g., age of information, number of information sources, etc.). Noncritical maneuvers such as smoothly increasing the gap to a vehicle on the right side, if a pedestrian is present, can be also triggered with lower quality values of the information. Parametrization of the overall system has to ensure that the false-positive rate is as low as possible.

Discussion

The introduced concept connects different information sources and considers privacy. It can be used to avoid or mitigate pedestrian accidents via an enhanced pedestrian presence prediction with a higher accuracy. Thus far, the higher accuracy of the developed prediction approach can only be derived from the literature review of studies on pedestrian behavior and still needs to be validated with real-world recordings and clinical studies in the future. Because today's accident analysis data do not include the real trajectory of the pedestrian before the accident, a solid impact assessment is difficult. However, based on many studies on pedestrian behavior, the impact and improvement can be estimated for specific situations. One simple example is given in Figure 7. The top plot shows an approximated starting speed profile of children/elderly (4 and 75+ years, respectively, average speed 1 m/s) and young adults (10–15 years, average speed 1.75 m/s) over movement time. Speed and acceleration values are chosen based on the literature review above. The bottom plot compares the prediction results of the model introduced in this article, which uses age as an additional parameter, with the prediction results of state-of-the-art models, which predict the pedestrian's presence based on a constant speed model for prediction times of 1, 2, and 3 s. A prediction error of 1 m at a movement time of 1 s for a prediction time of 3 s means that a prediction of the pedestrian's presence after 1 s of walking for 3 s in the future deviates from the real position by 1 m. As shown in the figure, the prediction error of the constant speed model shortly after the pedestrian starts walking is in the range of 0.5 m (prediction time 1 s, children/elderly and young adults) and 3.7 m (prediction time 3 s, young adults) and decreases until the pedestrian reaches its crossing speed. Considering the results from Themann et al. (2014), which showed that the prediction accuracy of the future VRU position for vehicle velocities of 50 km/h should provide a standard deviation of less than 55 cm, the state-of-the-art model is not adequate. Including an estimated target speed, which can be derived from personal information on the pedestrian (e.g., age) or the short-term behavior history, can significantly improve such pedestrian collision avoidance systems. The introduced enhanced pedestrian prediction model considers this kind of information together with many other factors to improve prediction accuracy.

Figure 7. The top plot shows a typical speed profile of children/elderly (4 years and 75+, average speed 1 m/s) and young adults (10–15 years, average speed 1.75 m/s) over movement time derived from the literature. The bottom plot shows the related prediction errors of the constant speed model compared to the prediction model introduced in this article for prediction times of 1, 2, and 3 s. At a movement time of 0 s, the pedestrian starts walking following the profiles of the top plot.

A prerequisite of the introduced system is a precise localization and speed determination. Today's smart devices do not provide this information with the required accuracy. However, several papers introduced promising approaches for an improved localization of pedestrians. The GNSS position can, for example, be improved by patches (DGNSS) up to an accuracy of 0.1 m (Rizos et al. 2012) or up to an accuracy of 3.33 m in narrow urban canyons by 3D maps (Wang et al. 2013). Yang et al. (2016) introduced an approach that determines the heading of the pedestrian based on smart device sensors with an accuracy of up to 12.5°. First, analyses with algorithms that match the smart device information (e.g., age of pedestrian) with information of a vehicle LiDAR showed promising results for many situations. Thus, smart device information seems to be usable for improvement in pedestrian prediction after detection of the pedestrian by vehicle sensors. In this case, the pedestrian location information is redundant and the advantage of the introduced system is that through the matching of the information, personal information (e.g., age) on the pedestrian can be used to improve the prediction. Given improved smart device sensor accuracies, the system can already be used before the vehicle sensors detect the pedestrian.

A V2X pedestrian collision avoidance system based on a cost function to predict the probability of the pedestrian's presence is developed and introduced. The model uses new information sources such as smart devices to improve the prediction of pedestrian presences in the near future and considers challenges that arise with the use of these information sources. Relevant pedestrian characteristics were derived from a literature review, analyzed, and stored in a database. An overall concept for an enhanced vehicle-to-everything (V2X) pedestrian collision avoidance system is presented. A method for the calculation of the probability of the pedestrian's presence is introduced together with an appropriate trajectory planning approach. The planning approach can handle probabilities of the pedestrian presence and can consider injury risks for calculation of trajectories in real time. The calculation of the pedestrian presence is based on the relevant pedestrian characteristics database. Further real-world observations and a clinical study with the system in real traffic are planned as next steps to analyze the impact of the developed concept.

Funding

This article is part of the work in the project UR:BAN funded by the German Federal Ministry for Economic Affairs and Energy based on a decision of the Deutsche Bundestag.