Abstract
Objective
Advanced Driving Assistance Systems (ADAS) have the potential to reduce occurrences and severity of collisions with Vulnerable Road Users (VRUs). However, the nearly infinite number of possible VRU visual appearances (e.g., size, clothing, accessories) represent a technical challenge as systems need to correctly detect and identify VRUs to take adequate mitigation measures. The aim of this study was to determine, through track testing, which parameters affect systems’ capabilities in detecting pedestrians.
Methods
The standardized articulated adult male pedestrian (EPTa) and seven-year-old articulated child pedestrian (EPTc) targets were used as the control group. Evaluations on the track followed the Euro NCAP AEB-VRU test protocols, and derivatives thereof. An iterative test approach was used to benchmark the detection capabilities of systems with variations in target configuration and environmental conditions against the control group (baseline condition). Over 1,000 track tests using 24 configurations and 13 vehicles (model years 2019-2021) were conducted. The environmental conditions included nighttime and snow-covered roads. Pedestrians were dressed in winter clothing and/or equipped with accessories, including a hat, jackets of different colors, backpack, umbrella, and a scooter. Other scenarios involved parked vehicles as obstructions or using multiple pedestrian targets (to simulate a parent crossing the road with child or a crowd waiting at an intersection) to challenge the vehicles with realistic urban-like scenarios.
Results
This study illustrates how the variation of parameters outside the baseline condition can affect a vehicles’ safety performance. Weather conditions and urban-like scenarios on the test track affected some systems more than others. The validity of these findings is however limited by the small vehicle sample size and number of tests performed per scenario.
Conclusions
In Canada, vehicles are exposed to less-than-ideal road conditions and a wide range of pedestrian profiles. The vehicles tested demonstrated various levels of performance and capabilities when mitigating collisions with pedestrians. The research illustrates the safety risks associated with weather and types of VRUs.
Introduction
In 2020, Vulnerable Road Users (VRUs) were victims in 18.1% of all Canadian road fatalities and 32.4% of serious injuries according to Transport Canada (TC)’s National Collision Database (Transport Canada Citation2022). Recent advances in crash avoidance technology may provide an avenue to reduce the risks to VRUs. These technologies use camera, radar, or lidar (individually or in combination) to detect static and moving objects in the vehicle’s environment and enable the vehicle’s systems to warn and react.
Advanced Driver-Assistance Systems (ADAS) are typically evaluated using test scenarios representing specific configurations. The current methodologies require tests to be performed under ideal conditions, without precipitation (rain or snow), within a range of ambient light parameters, and without any road infrastructure such as, curbs, sidewalks, signalization posts and permanent adjacent buildings, that could interfere with the vehicle’s sensors (EuroNCAP Citation2021). However, the common pedestrian does not always cross the road wearing blue pants, a black shirt and in day light, as prescribed in typical performance test protocols. In Canada, since the temperature often drops below freezing, pedestrians can be seen wearing large clothes and accessories (jackets, hats) during the winter months. Also, with a yearly average of 6 months of snow-covered ground (Canada, Citation2022), a white background is a common scene in Canada.
TC hypothesized that inclement weather and change in dummy appearances might affect the detection capabilities of commercially available ADAS to Canadians. Quantifying the detection capabilities will assist in understanding the real-world benefits of ADAS. Since 2015, PMG Technologies Inc. (PMG), the official engineering contractor and operator of TC’s Motor Vehicle Test Center (MVTC), and TC have been conducting research on ADAS with a focus on Pedestrian Automatic Emergency Braking systems (P-AEB). The intent of this specific work was to use standard test procedures as a baseline to evaluate ADAS technologies and to quantify how the detection capability varied with changes in environmental factors and target appearance. The testing approach was progressive with the aim of quantifying how night conditions, winter testing, and variations in pedestrian appearance affected the performance of driver assistance systems. This approach provided a way to assess the robustness of vehicle systems and their sensitivity to differences from optimal test conditions.
Methods
In order to obtain baseline scenarios for the evaluation of more complex scenarios, the Euro NCAP AEB VRU Test Protocol V3.0.4 (EuroNCAP Citation2021) methodology was used. The following scenarios were selected: Car-to-Pedestrian Nearside Adult at 75% impact point (CPNA-75), Car-to-Pedestrian Nearside Child at 50% impact point with two obstructing vehicles (CPNC-50) and Car-to-Pedestrian Longitudinal Adult at 25% impact point (CPLA-25). The baseline tests were performed using the standardized adult and child Euro NCAP Pedestrian Targets (EPTa and EPTc), representing a 50th percentile male and seven-year-old child, respectively. Subsequent variations included night testing (with and without illumination from lampposts), pedestrian targets equipped with additional clothing or accessories (jacket, hat, backpack, umbrella and scooter), or different backgrounds (snow or asphalt). Other variations involved combining the EPTa and EPTc on the same propulsion system (CPNAC-75) to simulate multiple pedestrians crossing simultaneously, obstructing the pedestrian target by positioning static dummies at the start of its path to re-create a busy intersection (CPNAP-75), or conducting the false positive test of the UN Regulation No. 152 (UNECE Citation2020) with the child pedestrian target in front (CPNCF-50) or behind (CPNCR-50) the target’s path to blend the child’s motion to the vehicle. Tables A1–A3 illustrate the different scenario variations and Table A4 illustrates the scenario configurations.
The test vehicles were selected based on availability, ADAS offerings and their representativeness of common vehicle types purchased by Canadians. No alterations were made to the test vehicles. A total of 13 vehicles were tested with each one only performing a subset of the total possible variations, as specified in Tables A5–A7. The test vehicles were equipped with Inertial and GNSS Navigation Data Acquisition Systems (RT) to collect accurate vehicle position and dynamics data (speed and acceleration), driving robots to perform precise and repeatable vehicle paths, and alert detectors to record the timing of audio warnings produced by the vehicles.
Two methods were used to determine the test speed for assessing the performance of the vehicles’ P-AEB systems depending on the type of variation: maximum performance speed and specified speed. The maximum performance speed method was used for baseline and nighttime conditions speed determination using the following procedure: the test vehicle was tested at an initial speed of 20 km/h. If there was complete avoidance, the test speed for the next test was increased by 10 km/h until an impact occurred. Once three impacts occurred at a given speed, the test speed was reduced by 5 km/h. This process was repeated until five avoidances were achieved over a maximum of seven repetitions at a given speed, defined as being the maximum performance speed. When determining the maximum performance speed for the CPLA-25 scenario, which only looks at the timing of the warnings, instead of finding the speed with full avoidance, the performance criterion was the onset of the Forward Collision Warning (FCW) corresponding to a Time-to-Collision (TTC) of at least 1.7 s (Euro NCAP, 2021). For clarity and conciseness, the term maximum performance speed is used to reference both the maximum avoidance speed (for scenarios CPNA-75 and CPNC-50) and the maximum speed which produced a TTC at FCW of at least 1.7 s (for scenario CPLA-25).
The specified speed method was used for clothing, snow, multiple actor/obstructions and the scooter variations. It consisted in performing five repetitions of the test scenario at a single speed: corresponding to the maximum avoidance speed determined during the baseline condition for each specific variation. If an impact at a speed greater than 20 km/h occurred, the speed was reduced by 5 km/h and tests were instead performed at this new speed.
The average TTC at the onset of audio FCW and the number of tests with full avoidance were determined and compared between each scenario variation and its corresponding baseline to identify which configuration had the most influence on each system’s capability to detect and react to VRUs. Vehicles were tested as black boxes since no information on the proprietary detection and control algorithms was available. Therefore, the TTC at onset of FCW was used as a metric of detection performance in order to compare data from tests performed in different environmental conditions. This approach allowed the exclusion of the friction coefficient effect on the braking outcome of AEB tests. This parameter also provided a uniform approach to evaluate vehicles’ detection capabilities across test variations. It is assumed that an earlier detection resulted in a larger TTC while a later detection resulted in a smaller TTC value.
Results
The results are presented in the following sections by variation type: clothing, snow background, multiple targets/obstructions, scooter, and nighttime. In , which are used to present the results for the clothing, snow background, multiple targets/obstructions, and scooter variations respectively, the average TTC at FCW is represented by the bar lengths while the test speed and the ratio of avoidances over the number of runs are presented next to the variation name. presents the maximum performance speed for each of the different scenarios in both daytime and nighttime conditions. The data for each vehicle are presented in Table B1–Table B13.
Figure 1. CPNA-75 clothing variations test results.
Figure 2. CPNC-50 clothing variations test results.
Figure 3. Snow background variation test results.
Figure 4. Multiple targets/obstructions variation test results.
Figure 5. Scooter variation test results.
Figure 6. Night variation test results.
Clothing
CPNA-75
Both the “hat, light jacket” and the “hat, dark jacket” variation gave similar or better results than the baseline condition for Vehicles C and D in terms of both number of collision avoidances and TTCs at FCW. The other two vehicles tested (Vehicles A and B) had decreased TTCs at FCW and an increased number of impacts compared to the baseline.
All three vehicles (Vehicles A, C and D) that were tested with the “hat, dark jacket and bag” variation had increased TTCs at FCW compared to the baseline. However, only Vehicles C and D avoided all impacts as they did during the baseline tests, while Vehicle A did not avoid a single impact in any of the five repetitions.
All five vehicles (Vehicles A, C, D, E and F) that were tested with the “hat, dark jacket, bag and umbrella” variation had worse performance compared to the baseline in terms of crash avoidances. For Vehicles C, D and F, the test speed was reduced by 5 km/h due to severe impacts that occurred at the planned test speed. Vehicles D and E had increased TTCs at FCW compared to their baselines but still resulted in more impacts.
CPNC-50
The three vehicles (Vehicles B, C and D) tested to the “hat, pink jacket” variation had similar TTCs at FCW compared to the baseline (within 0.03 s). Vehicles C and D were challenged by the “hat, blue jacket” variation compared to their baselines (decreased TTC at FCW and more impacts) while Vehicle B performed better.
Four out of the five vehicles (Vehicles B, C, D and E) tested with the “hat, blue jacket and bag” variation had decreased TTCs at FCW and an increased number of avoidances compared to their baselines. The fifth vehicle (Vehicle F) had an increase in TTC at FCW compared to the baseline but only avoided the target once over six repetitions.
Snow background
Tests performed with the adult pedestrian (CPNA-75) with a snow background resulted in more impacts for four out of five vehicles (Vehicles A, D, E and F). The fifth vehicle (Vehicle C) resulted in the same outcome as its baseline condition (five avoidances out of five tests). The TTCs at FCW were decreased for all vehicles except for two: one being increased (Vehicle D at a test speed of 45 km/h) and one being the same (Vehicle E) versus the baseline condition.
For the CPNC-50 scenario, three vehicles out of four (Vehicles C, D and E) had more impacts when the background was covered in snow compared to the asphalt background (baseline condition). Vehicles D and E had decreased TTCs at FCW while Vehicle C’s were increased. Vehicle F detected the dummy earlier against the snow background, therefore increasing the TTC at FCW and reducing the frequency of impacts.
Multiple targets/obstructions
Vehicles G and H were tested with multiple targets simultaneously (CPNAC-75 and CPNAP-75). The variations had a negative effect on the TTC at FCW and on the avoidance ratio for Vehicle H. In comparison, Vehicle G’s TTC at FCW was increased in the CPNAC-75 variation although these tests still resulted in more impacts than the baseline condition. On the other hand, its TTC at FCW was decreased for the CPNAP-75 but the avoidance ratio remained similar.
For the same two vehicles (Vehicles G and H), positioning obstructions in front of the pedestrian’s path (CPNCF-50) resulted in increased TTCs at FCW compared to the baseline (CPNC-50). However, testing Vehicle H under this configuration resulted in a severe impact compared to the full avoidance experienced under the baseline condition. Placing obstructions behind the pedestrian’s path (CPNCR-50) resulted in decreased TTCs at FCW except at a speed of 35 km/h for Vehicle H. Overall, both Vehicles G and H were challenged by this configuration as compared to their baselines since more impacts occurred.
Scooter
shows that the two vehicles (Vehicles A and I) tested using the scooter target detected the target in a similar time, or faster, when compared to the baseline tests. The warnings were produced at increased TTCs and the number of avoidances was the same or greater. Vehicle I detected the scooter faster for the longitudinal direction (CPLA-25) while Vehicle A detected faster with the scooter in the transversal direction (CPNA-75).
Nighttime
As shown in , all vehicles had at least one nighttime scenario in which the maximum performance speed was lower than during daytime. The CPNA-75 scenario generated degraded results for all vehicles. Vehicles A, H and J were not affected by the nighttime condition for the CPLA-25 scenario as compared to the baseline. A reduced maximum performance speed was observed in all scenarios performed for 8 of the 11 vehicles tested under the nighttime condition (all except for Vehicles A, H and J).
Discussion
The purpose of this study was to quantify the effect of target variations on the capabilities of P-AEB technology to detect and emit a warning. It is important to note that both the speed and trajectory of the pedestrian were not modified from the standard procedure. This enabled the physical and environmental parameters chosen for this study to be isolated from any other changes. Understanding the influence of these parameters helps with assessing the robustness and limitations of standard procedures. The results highlight the key elements that can influence the detection of pedestrians: the shape/silhouette and the contrast with the background.
For the purposes of this analysis, the FCW is used as an indication of the detection. Although the authors recognize that a vehicle’s system is likely to have detected the targets much earlier than the time the warning is emitted, the assumption is that, for a same scenario configuration (vehicle and target speed), if the detection and classification is done later, the warning will inevitably be emitted later than if the detection had been done earlier.
Changing the shape of the pedestrian reduced the detection capabilities of most tested systems. In fact, adding an umbrella to the adult target and adding a backpack to the child target had the most influence, as reflected by the increased number of impacts and later detections. Visually, these accessories more greatly altered the silhouette from the standard dummies in comparison to the other clothing variations (coloured jackets and hat). Clothing color was determined to have no significant effect on detection capabilities. Additionally, the systems had mixed results when multiple pedestrians were grouped together before or during a crossing. It is hypothesized that the proximity of the pedestrians to each other may have caused the perceived shape of the crossing pedestrian to differ from the standard system baseline shape, leading to hesitation from systems in identifying risks and performing mitigation responses consistently. On the other hand, placing the pedestrian on a scooter improved the detection capabilities of the systems. In this variation, the profile silhouette was not altered as much as the others (carrying an umbrella for example).
As seen from the results, adding obstructions in a standard scenario can affect system response times. When the background behind the pedestrian’s path was modified by the addition of snow or static vehicles, the detection capability of most systems was reduced compared to their baseline results. Also, the majority of the vehicles were challenged by low light nighttime conditions, resulting in lower maximum performance speeds when compared to daytime tests.
Canada’s many ecozones are such that both urban and rural roads have a large variety of shapes and colors that differ from season to season. This means that there are an essentially countless number of detection scenarios possible for Canada alone. As such, more work should be conducted on these systems to understand how their technologies cope with these highly variable conditions.
The findings presented in this paper show that most of the selected variations challenged current P-AEB systems. The variations were representative of urban vehicle-VRU interactions across multiple seasons and times of day. Vehicles had different baseline performances based on their design, technology, and operational characteristics. The key to an effective standard procedure is to select a critical threshold where safety benefits can be measured. Here, it was decided to assume that a non-detection would result in an impact and a delayed TTC at FCW would result in less time and distance to engage the P-AEB system properly. As systems become more complex and become capable of responding to more real-world challenges, understanding the range of operation of P-AEB will be critical, as demonstrated by these variations.
Representing the real world on closed test tracks is also a challenge and is part of the limitations of this study. The limited number of vehicles, scenarios, and repetitions per variation represent only a small sample of a normal driving day. Also, for the purpose of this study, the surface coefficient of friction was not factored in for winter conditions. The test surface was optimized for best braking performance to avoid potential impacts between targets and vehicles, and to mitigate unnecessary damages. Another limitation of this study is understanding the effects of using the TTC at FCW as a metric. The small variance measured using precise instrumentation is a quantification of the technology and not the time gain for a human response. The fractions of a second collected were used to illustrate nuances between test conditions. Finally, the research team recognizes that the test vehicles may not have been designed to respond to the selected scenarios and may have been evaluated in conditions that fall outside of their respective operational design domain.
ADAS-equipped vehicles are routinely exposed to less-than-ideal road conditions and a wide range of pedestrian profiles. Understanding system limitations in a controlled environment enables the evaluation of their potential benefits in the real world, their performance, and which parameters affect their responses. The results confirm the hypothesis that varying the overall visibility, the shape of the target, and the background has an impact on the performance of vehicles across multiple brands and models.
Vehicles tested showed various levels of performance and capabilities when mitigating risks to VRUs. This paper presents opportunities to quantify potential safety benefits when vehicles are operated in less-than-ideal environments. The results also serve as an indicator of robustness of this type of system when only a subset of parameters is varied. TC and PMG will continue to conduct research on P-AEB technologies and explore more realistic scenarios as new systems are made available. The results will be compared to published procedures currently used to evaluate the differences, inform the development of potential requirements and help to quantify the effect on the systems’ reliability.
Supplemental Material
Download Zip (10.5 MB)Acknowledgments
The authors would like to thank the technical members of the PMG team for their support during testing and data analysis.
Disclosure statement
The authors report that there is no competing interest to declare, the test facility and equipment are property of Transport Canada and vehicles were purchased by Transport Canada for the sole purpose of research activities. The brands and models selected were part of a research fleet available to the department and were purchased through a competitive process without the manufacturer’s knowledge. The observations made in this report are the sole expression of the authors and does not necessarily reflect Transport Canada’s policies and regulations.
Data availability statement
The data that support the findings of this study are available from the corresponding author, [DC], upon request.
Additional information
Funding
References
- Environment and Climate Change Canada. 2022. SNOW COVER: Canadian Environmental Sustainability Indicators. Gatineau (QC): Government of Canada [accessed 2022 Nov 20]. https://www.canada.ca/content/dam/eccc/documents/pdf/cesindicators/snow-cover/2022/snow-cover-en.pdf.
- EuroNCAP. 2021. TEST PROTOCOL – AEB VRU Systems Version 3.0.4. https://cdn.euroncap.com/media/62795/euro-ncap-aeb-vru-test-protocol-v304.pdf.
- Transport Canada. 2022. Canadian Motor Vehicle Traffic Collision Statistics: 2020. Ottawa (ON): Government of Canada [accessed 2022 Aug 10]. https://tc.canada.ca/en/road-transportation/statistics-data/canadian-motor-vehicle-traffic-collision-statistics-2020.
- UNECE. 2020. UN Regulation No. 152, Amend.1 – Advanced Emergency Braking System (AEBS). https://unece.org/fileadmin/DAM/trans/main/wp29/wp29regs/2020/R152am1e.pdf.