Safety assessment of automated vehicles: how to determine whether we have collected enough field data?

References

• Alvarez S, Page Y, Sander U, et al. Prospective effectiveness assessment of ADAS and active safety systems via virtual simulation: a review of the current practices. Paper presented at: 25th International Technical Conference on the Enhanced Safety of Vehicles (ESV); 2017.
   Google Scholar
• Bashtannyk DM, Hyndman RJ. Bandwidth selection for kernel conditional density estimation. Comput Stat Data Anal. 2001;36:279–298.
   Web of Science ®Google Scholar
• Blair SN, LaMonte MJ, Nichaman MZ. The evolution of physical activity recommendations: how much is enough? Am J Clin Nutr. 2004;79:913S–920S.
   PubMed Web of Science ®Google Scholar
• Broggi A, Buzzoni M, Debattisti S, et al. Extensive tests of autonomous driving technologies. IEEE Trans Intell Transp Syst. 2013;14:1403–1415.
   Web of Science ®Google Scholar
• Bunge J, Fitzpatrick M. Estimating the number of species: a review. J Am Stat Assoc. 1993;88:364–373.
   Web of Science ®Google Scholar
• Calonico S, Cattaneo MD, Farrell MH. On the effect of bias estimation on coverage accuracy in nonparametric inference. J Am Stat Assoc. 2018;113:767–779.
   Web of Science ®Google Scholar
• Chen Y-C. A tutorial on kernel density estimation and recent advances. Biostat Epidemiol. 2017;1:161–187.
   Google Scholar
• Chiu S-T. A comparative review of bandwidth selection for kernel density estimation. Stat Sin. 1996;6:129–145.
   Web of Science ®Google Scholar
• de Gelder E, Paardekooper J-P. Assessment of automated driving systems using real-life scenarios. Paper presented at: IEEE Intelligent Vehicles Symposium (IV); 2017.
   Google Scholar
• Dingus TA, Guo F, Lee S, et al. Driver crash risk factors and prevalence evaluation using naturalistic driving data. Proc Natl Acad Sci USA. 2016;113:2636–2641.
   PubMed Web of Science ®Google Scholar
• Duin. On the choice of smoothing parameters for Parzen estimators of probability density functions. IEEE Trans Comput. 1976;25:1175–1179.
   Web of Science ®Google Scholar
• Elrofai H, Paardekooper J-P, de Gelder E, Kalisvaart S, Op den Camp O. Scenario-Based Safety Validation of Connected and Automated Driving. Helmond, The Netherlands: Netherlands Organization for Applied Scientific Research, TNO; 2018.
   Google Scholar
• Elrofai H, Worm D, Op den Camp O. Scenario identification for validation of automated driving functions. In: Meyer G, ed. Advanced Microsystems for Automotive Applications 2016. Berlin, Germany: Springer; 2016:153–163.
   Google Scholar
• Gandolfi A, Sastri CC. Nonparametric estimations about species not observed in a random sample. Milan J Math. 2004;72:81–105.
   Google Scholar
• Geyer S, Baltzer M, Franz B, et al. Concept and development of a unified ontology for generating test and use-case catalogues for assisted and automated vehicle guidance. IET Intelligent Transport Systems. 2014;8:183–189.
   Web of Science ®Google Scholar
• Goodman LA. On the exact variance of products. J Am Stat Assoc. 1960;55:708–713.
   Web of Science ®Google Scholar
• Guest G, Bunce A, Johnson L. How many interviews are enough? An experiment with data saturation and variability. Field Methods. 2006;18:59–82.
   Web of Science ®Google Scholar
• Hall P. On global properties of variable bandwidth density estimators. Ann Stat. 1992;20:762–778.
   Web of Science ®Google Scholar
• Jones MC, Marron JS, Sheather SJ. A brief survey of bandwidth selection for density estimation. J Am Stat Assoc. 1996;91:401–407.
   Web of Science ®Google Scholar
• Kasper D, Weidl G, Dang T, et al. Object-oriented Bayesian networks for detection of lane change maneuvers. IEEE Intelligent Transportation Systems Magazine. 2012;4:19–31.
   Web of Science ®Google Scholar
• Klauer SG, Dingus TA, Neale VL, Sudweeks JD, Ramsey DJ. The Impact of Driver Inattention on Near-Crash/Crash Risk: An Analysis Using the 100-Car Naturalistic Driving Study Data. Blacksburg, VA: NHTSA; 2006.
   Google Scholar
• Krajewski R, Bock J, Kloeker L, Eckstein L. The HighD dataset: a drone dataset of naturalistic vehicle trajectories on German highways for validation of highly automated driving systems. Paper presented at: IEEE 21st International Conference on Intelligent Transportations Systems (ITSC); 2018.
   Google Scholar
• Marks IH, Fong ZV, Stapleton SM, Hung Y-C, Bababekov YJ, Chang DC. How much data are good enough? Using simulation to determine the reliability of estimating POMR for resource-constrained settings. World J Surg. 2018;42:2344–2347.
   PubMed Web of Science ®Google Scholar
• Marron JS, Wand MP. Exact mean integrated squared error. Ann Stat. 1992;20:712–736.
   Web of Science ®Google Scholar
• Paardekooper J-P, Montfort S, Manders J, et al. Automatic identification of critical scenarios in a public dataset of 6000 km of public-road driving. Paper presented at: 26th International Technical Conference on the Enhanced Safety of Vehicles (ESV); 2019.
   Google Scholar
• Parzen E. On estimation of a probability density function and mode. Ann Math Stat. 1962;33:1065–1076.
   Google Scholar
• Ploeg J, de Gelder E, Slavík M, Querner E, Webster T, de Boer N. Scenario-based safety assessment framework for automated vehicles. Paper presented at: 16th ITS Asia-Pacific Forum; 2018.
   Google Scholar
• Pütz A, Zlocki A, Bock J, Eckstein L. System validation of highly automated vehicles with a database of relevant traffic scenarios. Paper presented at: 12th ITS European Congress; 2017.
   Google Scholar
• Rosenblatt M. Remarks on some nonparametric estimates of a density function. Ann Math Stat. 1956;27:832–837.
   Google Scholar
• Sadigh D, Driggs-Campbell K, Puggelli A, et al. Data-driven probabilistic modeling and verification of human driver behavior. Paper presented at: 2014 AAAI Spring Symposium Series; 2014.
   Google Scholar
• Schölkopf B, Platt JC, Shawe-Taylor J, Smola AJ, Williamson RC. Estimating the support of a high-dimensional distribution. Neural Comput. 2001;13:1443–1471.
   PubMed Web of Science ®Google Scholar
• Scott DW. Multivariate Density Estimation: Theory, Practice, and Visualization. Houston, TX: John Wiley & Sons; 2015.
   Google Scholar
• Scott DW, Sain SR. Multi-dimensional density estimation. Handbook of Statistics. 2005;24:229–261.
   Google Scholar
• Silverman BW. Density Estimation for Statistics and Data Analysis. London: CRC Press; 1986.
   Google Scholar
• Stellet JE, Zofka MR, Schumacher J, Schamm T, Niewels F, Zöllner JM. Testing of advanced driver assistance towards automated driving: a survey and taxonomy on existing approaches and open questions. Paper presented at: IEEE 18th International Conference on Intelligent Transportation Systems; 2015.
   Google Scholar
• Turlach BA. Bandwidth Selection in Kernel Density Estimation: A Review. Berlin, Germany: Institut für Statistik und Ökonometrie, Humboldt-Universität zu Berlin; 1993.
   Google Scholar
• Ulbrich S, Menzel T, Reschka A, Schuldt F, Maurer M. Defining and substantiating the terms scene, situation, and scenario for automated driving. Paper presented at: IEEE 18th International Conference on Intelligent Transportation Systems; 2015.
   Google Scholar
• Wang W, Liu C, Zhao D. How much data are enough? A statistical approach with case study on longitudinal driving behavior. IEEE Transactions on Intelligent Vehicles. 2017;2:85–98.
   Google Scholar
• Williamson A, Lombardi DA, Folkard S, Stutts J, Courtney TK, Connor JL. The link between fatigue and safety. Accid Anal Prev. 2011;43:498–515.
   PubMed Web of Science ®Google Scholar
• Xie J, Hilal AR, Kulić D. Driving maneuver classification: a comparison of feature extraction methods. IEEE Sens J. 2018;18:4777–4784.
   Web of Science ®Google Scholar
• Yang H, Van Dongen BF, Ter Hofstede AH, Wynn MT, Wang J. Estimating Completeness of Event Logs. 2012. BPM Center Report 12. Eindhoven, The Netherlands: Eindhoven University of Technology.
   Google Scholar
• Zambom AZ, Dias R. A review of kernel density estimation with applications to econometrics. Int Econ Rev. 2013;5:20–42.
   Google Scholar
• Zofka MR, Kuhnt F, Kohlhaas R, Rist C, Schamm T, Zöllner JM. Data-driven simulation and parametrization of traffic scenarios for the development of advanced driver assistance systems. Paper presented at: 18th International Conference on Information Fusion; 2015.
   Google Scholar