How certain are we that our automated driving system is safe?

References

• Antona-Makoshi J, Uchida N, Yamazaki K, Ozawa K, Kitahara E, Taniguchi S. 2019. Development of a safety assurance process for autonomous vehicles in Japan. Paper presented at: 26th International Technical Conference on the Enhanced Safety of Vehicles (ESV). p. 1–18, https://www-esv.nhtsa.dot.gov/Proceedings/26/26ESV-000286.pdf.
   Google Scholar
• Bunge J, Fitzpatrick M. 1993. Estimating the number of species: a review. J Am Stat Assoc. 88(421):364–373. doi: 10.1080/01621459.1993.10594330.
   Web of Science ®Google Scholar
• Burns P. 2002. Robustness of the Ljung-Box test and its rank equivalent. Technical report. doi: 10.2139/ssrn.443560.
   Google Scholar
• Cadre B. 2006. Kernel estimation of density level sets. J Multivariate Anal. 97(4):999–1023. doi: 10.1016/j.jmva.2005.05.004.
   Web of Science ®Google Scholar
• Chan C-Y. 2017. Advancements, prospects, and impacts of automated driving systems. Int J Transp Sci Technol. 6(3):208–216. doi: 10.1016/j.ijtst.2017.07.008.
   Google Scholar
• Chen Y-C. 2017. A tutorial on kernel density estimation and recent advances. Biostat Epidemiol. 1(1):161–187. doi: 10.1080/24709360.2017.1396742.
   Google Scholar
• de Gelder E, Adjenughwure K, Manders J, Snijders R, Paardekooper J-P, Op den Camp O, Tejada Ruiz A, De Schutter B. 2023. PRISMA: a novel approach for deriving probabilistic surrogate safety measures for risk evaluation. Under review. https://arxiv.org/abs/2303.07891.
   Google Scholar
• de Gelder E, Elrofai H, Khabbaz Saberi A, Op den Camp O, Paardekooper J-P, De Schutter B. 2021. Risk quantification for automated driving systems in real-world driving scenarios. IEEE Access. 9:168953–168970. doi: 10.1109/ACCESS.2021.3136585.
   Web of Science ®Google Scholar
• de Gelder E, Manders J, Grappiolo C, Paardekooper J-P, Op den Camp O, Schutter D. 2020. B. Real-world scenario mining for the assessment of automated vehicles. Paper presented at: IEEE International Transportation Systems Conference (ITSC). p. 1073–1080. doi: 10.1109/ITSC45102.2020.9294652.
   Google Scholar
• de Gelder E, Paardekooper J-P, Khabbaz Saberi A, Elrofai H, Op den Camp O, Kraines S, Ploeg J, De Schutter B. 2022. Towards an ontology for scenario definition for the assessment of automated vehicles: An object-oriented framework. IEEE Trans Intell Veh. 7(2):300–314. doi: 10.1109/TIV.2022.3144803.
   Google Scholar
• de Gelder E, Paardekooper J-P, Op den Camp O, Schutter D. 2019. B. Safety assessment of automated vehicles: How to determine whether we have collected enough field data? Traffic Inj Prev. 20(S1):162–170. doi: 10.1080/15389588.2019.1602727.
   Google Scholar
• Duong T. 2007. ks: Kernel density estimation and kernel discriminant analysis for data in R. Journal of Statistical Software. 21(7):1–16. doi: 10.18637/jss.v021.i07.
   Web of Science ®Google Scholar
• Efron B. 1979. Bootstrap methods: another look at the jackknife. Ann Stat. 7(1):1–26. doi: 10.1214/aos/1176344552.
   Web of Science ®Google Scholar
• Elfring J, Appeldoorn R, van den Dries S, Kwakkernaat M. 2016. Effective world modeling: Multisensor data fusion methodology for automated driving. Sensors. 16(10):1–27. doi: 10.3390/s16101668.
   Web of Science ®Google Scholar
• Elrofai H, Paardekooper J-P, de Gelder E, Kalisvaart S, Op den Camp O. 2018. Scenario-based safety validation of connected and automated driving. Technical report. Helmond (the Netherlands): Netherlands Organization for Applied Scientific Research, TNO. http://publications.tno.nl/publication/34626550/AyT8Zc/TNO-2018-streetwise.pdf.
   Google Scholar
• European Commission. 2022. Commision implementing act AD v4.1. Draft implementing regulation. https://eur-lex.europa.eu/legal-content/EN/ALL/?uri=PI_COM:Ares(2022)2667391.
   Google Scholar
• Goodman LA. 1960. On the exact variance of products. J Am Stat Assoc. 55(292):708–713. doi: 10.1080/01621459.1960.10483369.
   Web of Science ®Google Scholar
• Gyllenhammar M, Johansson R, Warg F, Chen D, Heyn H-M, Sanfridson M, Söderberg J, Thorsén A, Ursing S. 2020. Towards an operational design domain that supports the safety argumentation of an automated driving system. Paper presented at: 10th European Congress on Embedded Real Time Systems (ERTS). https://www.diva-portal.org/smash/get/diva2:1390550/FULLTEXT01.pdf.
   Google Scholar
• Hauer F, Schmidt T, Holzmüller B, Pretschner A. 2019. Did we test all scenarios for automated and autonomous driving systems? Paper presented at: IEEE Intelligent Transportation Systems Conference (ITSC). p. 2950–2955. doi: 10.1109/itsc.2019.8917326.
   Google Scholar
• Hayward JC. 1972. Near miss determination through use of a scale of danger. Technical Report TTSC-7115. State College: Pennsylvania State University. https://onlinepubs.trb.org/Onlinepubs/hrr/1972/384/384-004.pdf.
   Google Scholar
• ISO 21448. 2022. Road vehicles – safety of the intended functionality. Geneva (Switzerland): Standard, International Organization for Standardization. https://www.iso.org/standard/77490.html.
   Google Scholar
• ISO 26262. 2018. Road vehicles – functional safety. Geneva (Switzerland): Standard, International Organization for Standardization. https://www.iso.org/standard/68383.html.
   Google Scholar
• Kaur P, Taghavi S, Tian Z, Shi W. 2021. A survey on simulators for testing self-driving cars. Paper presented at: Fourth International Conference on Connected and Autonomous Driving (MetroCAD). p. 62–70. doi: 10.1109/MetroCAD51599.2021.00018.
   Google Scholar
• Ljung GM, Box GE. 1978. On a measure of lack of fit in time series models. Biometrika. 65(2):297–303. doi: 10.1093/biomet/65.2.297.
   Web of Science ®Google Scholar
• Luetge C. 2017. The German ethics code for automated and connected driving. Philos Technol. 30(4):547–558. doi: 10.1007/s13347-017-0284-0.
   Google Scholar
• Menzel T, Bagschik G, Maurer M. 2018. Scenarios for development, test and validation of automated vehicles. Paper presented at: IEEE Intelligent Vehicles Symposium (IV), p. 1821–1827. doi: 10.1109/ivs.2018.8500406
   Google Scholar
• Milanés V, Shladover SE. 2014. Modeling cooperative and autonomous adaptive cruise control dynamic responses using experimental data. Transp Res Part C: Emerging Technol. 48:285–300. doi: 10.1016/j.trc.2014.09.001.
   Web of Science ®Google Scholar
• Mullakkal-Babu FA, Wang M, Farah H, van Arem B, Happee R. 2017. Comparative assessment of safety indicators for vehicle trajectories on highways. Transp Res Rec. 2659(1):127–136. doi: 10.3141/2659-14.
   Google Scholar
• Najm WG, Smith JD, Yanagisawa M. 2007. Pre-crash scenario typology for crash avoidance research. Technical Report DOT HS 810 767, U.S. Cambridge (MA): Department of Transportation Research and Innovative Technology Administration. https://rosap.ntl.bts.gov/view/dot/6281/dot_6281_DS1.pdf.
   Google Scholar
• Owen AB. 2013. Monte Carlo theory, methods and examples. https://statweb.stanford.edu/∼owen/mc/.
   Google Scholar
• Paardekooper J-P, Montfort S, Manders J, Goos J, de Gelder E, Op den Camp O, Bracquemond A, Thiolon G. 2019. Automatic identification of critical scenarios in a public dataset of 6000 km of public-road driving. Paper presented at: 26th International Technical Conference on the Enhanced Safety of Vehicles (ESV). https://www-esv.nhtsa.dot.gov/Proceedings/26/26ESV-000255.pdf.
   Google Scholar
• Parzen E. 1962. On estimation of a probability density function and mode. Ann Math Stat. 33(3):1065–1076. doi: 10.1214/aoms/1177704472.
   Google Scholar
• Ploeg J, de Gelder E, Slavík M, Querner E, Webster T, de Boer N. 2018. Scenario-based safety assessment framework for automated vehicles. Paper presented at: 16th ITS Asia-Pacific Forum. p. 713–726. https://arxiv.org/abs/2112.09366.
   Google Scholar
• Pütz A, Zlocki A, Bock J, Eckstein L. 2017. System validation of highly automated vehicles with a database of relevant traffic scenarios. Paper presented at: 12th ITS European Congress. p. 1–8, https://www.pegasusprojekt.de/files/tmpl/pdf/12th%20ITS%20European%20Congress_Folien.pdf.
   Google Scholar
• Riedmaier S, Ponn T, Ludwig D, Schick B, Diermeyer F. 2020. Survey on scenario-based safety assessment of automated vehicles. IEEE Access. 8:87456–87477. doi: 10.1109/ACCESS.2020.2993730.
   Web of Science ®Google Scholar
• Roesener C, Sauerbier J, Zlocki A, Fahrenkrog F, Wang L, Várhelyi A, de Gelder E, Dufils J, Breunig S, Mejuto P, et al. 2017. A comprehensive evaluation approach for highly automated driving. Paper presented at: 25th International Technical Conference on the Enhanced Safety of Vehicles (ESV). https://www-esv.nhtsa.dot.gov/Proceedings/25/25ESV-000259.pdf.
   Google Scholar
• Rosenblatt M. 1956. Remarks on some nonparametric estimates of a density function. Ann Math Stat. 27(3):832–837. doi: 10.1214/aoms/1177728190.
   Google Scholar
• Rosique F, Navarro PJ, Fernández C, Padilla A. 2019. A systematic review of perception system and simulators for autonomous vehicles research. Sensors. 19(3):648. doi: 10.3390/s19030648.
   PubMed Web of Science ®Google Scholar
• SAE J3016. 2021. Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. Technical report. Warrendale (PA): SAE International.
   Google Scholar
• Scholtes M, Westhofen L, Turner LR, Lotto K, Schuldes M, Weber H, Wagener N, Neurohr C, Bollmann MH, Körtke F, et al. 2021. 6-layer model for a structured description and categorization of urban traffic and environment. IEEE Access. 9:59131–59147. doi: 10.1109/ACCESS.2021.3072739.
   Web of Science ®Google Scholar
• Thorn E, Kimmel S, Chaka M. 2018. A framework for automated driving system testable cases and scenarios. Technical Report DOT HS 812623. Washington (DC): National Highway Traffic Safety Administration. https://rosap.ntl.bts.gov/view/dot/38824.
   Google Scholar
• Turlach BA. 1993. Bandwidth selection in kernel density estimation: a review. Technical report, Berlin: Institut für Statistik und Ökonometrie, Humboldt-Universität zu. https://www.researchgate.net/publication/2316108_Bandwidth_Selection_in_Kernel_Density_Estimation_A_Review.
   Google Scholar
• Wang C, Xie Y, Huang H, Liu P. 2021. A review of surrogate safety measures and their applications in connected and automated vehicles safety modeling. Accid Anal Prev. 157:106157. doi: 10.1016/j.aap.2021.106157.
   PubMed Web of Science ®Google Scholar
• Wang W, Liu C, Zhao D. 2017. How much data are enough? A statistical approach with case study on longitudinal driving behavior. IEEE Trans Intell Veh. 2(2):85–98. doi: 10.1109/tiv.2017.2720459.
   Google Scholar
• Wasserman L. 2006. All of nonparamatric statistics. New York (NY): Springer.
   Google Scholar
• Westhofen L, Neurohr C, Koopmann T, Butz M, Schütt B, Utesch F, Neurohr B, Gutenkunst C, Böde E. 2022. Criticality metrics for automated driving: A review and suitability analysis of the state of the art. Arch Comput Methods Eng. 30:1–35. doi: 10.1007/s11831-022-09788-7.
   PubMed Web of Science ®Google Scholar
• World Forum for Harmonization of Vehicle Regulations. 2021. E/ECE/TRANS/505/Rev.3/Add.156 Uniform provisions concerning the approval of vehicles with regard to Automated Lane Keeping Systems. Geneva (Switzerland): United Nations. https://unece.org/sites/default/files/2021-03/R157e.pdf
   Google Scholar
• Xiao L, Wang M, van Arem B. 2017. Realistic car-following models for microscopic simulation of adaptive and cooperative adaptive cruise control vehicles. Transp Res Rec. 2623(1):1–9. doi: 10.3141/2623-01.
   Google Scholar
• Zhang P. 1996. Nonparametric importance sampling. J Am Stat Assoc. 91(435):1245–1253. doi: 10.1080/01621459.1996.10476994.
   Web of Science ®Google Scholar