Finding and understanding pedal misapplication crashes using a deep learning natural language model

Abstract

Objective

The objective of this study was to develop a system which used the BERT natural language understanding model to identify pedal misapplication (PM) crashes from their crash narratives and validate the accuracy of the system.

Methods

The training dataset used for this study was 11 cases from the NMVCCS study and 952 cases from the North Carolina state crash database. Cases for this study were selected from their respective full datasets using a keyword search algorithm containing terms indicative of a pedal-related mistake. A BERT language model was used to classify each case narrative as either no pedal misapplication, PM by vehicle 1, PM by vehicle 2, or PM by vehicle 3. After training, the language model was used to determine the incidence of pedal misapplication in a test dataset of 8,668 North Carolina and NMVCCS cases and these results were compared to a manual review of the dataset. After manual review, 2,969 cases were pedal misapplications.

Results

The model’s AUC ROC performance at detecting PM was quantified on the entire testing dataset to evaluate the power of the system to generalize to case narratives unseen at training time. The AUC ROC value was 0.9835, indicating strong generalization to all crash narratives. By choosing the optimal threshold using the ROC curve, the system correctly identified PM in 95.7% of crash narratives. When pedal misapplication was correctly identified, the correct vehicle was identified in 95.9% of cases. A total of 3,062 pedal misapplications were identified. The model labeled cases 353 times faster than a researcher.

Conclusions

The strong performance of the model suggests that the automated interpretation of case narratives can be used for future research studies without any manual review. This would save time and enable the use of datasets where manual review would be infeasible. The automated extraction of information from crash narratives using deep learning natural language models has not been demonstrated previously in the literature, to the best of the authors’ knowledge. This technique can be applied to large, infrequently used datasets of crash narratives and extended to extract useful vehicle, occupant, or environment information to make these datasets amenable to traditional statistical analyses.

Keywords:

• Pedal misapplication
• NMVCCS
• deep learning
• BERT
• NLP

Acknowledgments

The authors would like to thank Alexandra Haynes and Jordan Moon at Virginia Tech for their help labeling the training and validation datasets.

Additional information

Funding

This work was supported by Collaborative Safety Research Center (CSRC) and Toyota Motor Corporation.