Testing the reliability of accident analysis methods: a comparison of AcciMap, STAMP-CAST and AcciNet

References

• Ahmadi, O., S.B. Mortazavi, A. Khavanin, and H. Mokarami. 2019. “Validity and Consistency Assessment of Accident Analysis Methods in the Petroleum Industry.” International Journal of Occupational Safety and Ergonomics: JOSE 25 (3): 355–361. doi:10.1080/10803548.2017.1387400.
   PubMed Web of Science ®Google Scholar
• Annett, J. 2002. “A Note on the Validity and Reliability of Ergonomics Methods.” Theoretical Issues in Ergonomics Science 3 (2): 228–232. doi:10.1080/14639220210124067.
   Google Scholar
• Chicco, D. 2017. “Ten Quick Tips for Machine Learning in Computational Biology.” BioData Mining 10 (1): 35. doi:10.1186/s13040-017-0155-3.
   PubMed Web of Science ®Google Scholar
• Chicco, D., and G. Jurman. 2020. “The Advantages of the Matthews Correlation Coefficient (MCC) over F1 Score and Accuracy in Binary Classification Evaluation.” BMC Genomics 21 (1): 6. doi:10.1186/s12864-019-6413-7.
   PubMed Web of Science ®Google Scholar
• de Ruijter, A., and F. Guldenmund. 2016. “The Bowtie Method: A Review.” Safety Science 88: 211–218. doi:10.1016/j.ssci.2016.03.001.
   Web of Science ®Google Scholar
• Goncalves Filho, A.P., G.T. Jun, and P. Waterson. 2019. “Four Studies, Two Methods, One Accident – an Examination of the Reliability and Validity of Accimap and STAMP for Accident Analysis.” Safety Science 113: 310–317. doi:10.1016/j.ssci.2018.12.002.
   Web of Science ®Google Scholar
• Goode, N., P.M. Salmon, N.Z. Taylor, M.G. Lenné, and C.F. Finch. 2017. “Developing a Contributing Factor Classification Scheme for Rasmussen’s AcciMap: Reliability and Validity Evaluation.” Applied Ergonomics 64: 14–26. doi:10.1016/j.apergo.2017.04.014.
   PubMed Web of Science ®Google Scholar
• Grant, E., P.M. Salmon, N.J. Stevens, N. Goode, and G.J.M. Read. 2018. “Back to the Future: What Do Accident Causation Models Tell us about Accident Prediction?” Safety Science 104: 99–109. doi:10.1016/j.ssci.2017.12.018.
   Web of Science ®Google Scholar
• Green, D.M, and J.A. Swets. 1966. Signal Detection Theory and Psychophysics. Ney York, USA, Wiley New York.
   Google Scholar
• Hallgren, K.A. 2012. “Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial.” Tutorials in Quantitative Methods for Psychology 8 (1): 23–34. doi:10.20982/tqmp.08.1.p023.
   PubMedGoogle Scholar
• Heinrich, H.W. 1931. Industrial Accident Prevention. A Scientific Approach. New York, USA, McGraw-Hill.
   Google Scholar
• Hollnagel, E. 2004. Barriers and Accident Prevention. Aldershot: Ashgate.
   Google Scholar
• Hollnagel, E. 2012. FRAM, the Functional Resonance Analysis Method: Modelling Complex Socio-Technical Systems. Farnham, England, Ashgate Publishing, Ltd.
   Google Scholar
• Hollnagel, E., D.D. Woods, and N. Leveson. 2006. Resilience Engineering: Concepts and Precepts. Farnham: Ashgate.
   Google Scholar
• Hulme, A., N.A. Stanton, G.H. Walker, P. Waterson, and P.M. Salmon. 2021a. “Are Accident Analysis Methods Fit for Purpose? Testing the Criterion-Referenced Validity of AcciMap, STAMP-CAST and AcciNet.” Safety Science 144: 105454. doi:10.1016/j.ssci.2021.105454.
   Web of Science ®Google Scholar
• Hulme, A., N.A. Stanton, G.H. Walker, P. Waterson, and P.M. Salmon. 2021. “Complexity Theory in Accident Causation: Using AcciMap to Identify the Systems Thinking Tenets in 11 Catastrophes.” Ergonomics 64 (7): 821–838. doi:10.1080/00140139.2020.1869321.
   PubMed Web of Science ®Google Scholar
• Hulme, A., N.A. Stanton, G.H. Walker, P. Waterson, and P.M. Salmon. 2021b. “Testing the Reliability and Validity of Risk Assessment Methods in Human Factors and Ergonomics. Ergonomics under Review.” Ergonomics 65 (3): 407–428. doi:10.1080/00140139.2021.1962969.
   PubMed Web of Science ®Google Scholar
• Hulme, A., N.A. Stanton, G.H. Walker, P. Waterson, and P.M. Salmon. 2019. “What Do Applications of Systems Thinking Accident Analysis Methods Tell us about Accident Causation? A Systematic Review of Applications between 1990 and 2018.” Safety Science 117: 164–183. doi:10.1016/j.ssci.2019.04.016.
   Web of Science ®Google Scholar
• Igene, O.O., C.W. Johnson, and J. Long. 2022. “An Evaluation of the Formalised AcciMap Approach for Accident Analysis in Healthcare.” Cognition, Technology & Work 24 (1): 161–181. doi:10.1007/s10111-021-00669-w.
   Web of Science ®Google Scholar
• Johnson, C.W, and I.M. de Almeida. 2008. “An Investigation into the Loss of the Brazilian Space Programme’s Launch Vehicle VLS-1 V03.” Safety Science 46 (1): 38–53. doi:10.1016/j.ssci.2006.05.007.
   Web of Science ®Google Scholar
• Kanis, H. 2000. “Questioning Validity in the Area of Ergonomics/Human Factors.” Ergonomics 43 (12): 1947–1965. doi:10.1080/00140130050201418.
   PubMed Web of Science ®Google Scholar
• Kanis, H. 2014. “Reliability and Validity of Findings in Ergonomics Research.” Theoretical Issues in Ergonomics Science 15 (1): 1–46. doi:10.1080/1463922X.2013.802058.
   Google Scholar
• Katsakiori, P., G. Sakellaropoulos, and E. Manatakis. 2009. “Towards an Evaluation of Accident Investigation Methods in Terms of Their Alignment with Accident Causation Models.” Safety Science 47 (7): 1007–1015. doi:10.1016/j.ssci.2008.11.002.
   Web of Science ®Google Scholar
• Khanzode, V.V., J. Maiti, and P.K. Ray. 2012. “Occupational Injury and Accident Research: A Comprehensive Review.” Safety Science 50 (5): 1355–1367. doi:10.1016/j.ssci.2011.12.015.
   Web of Science ®Google Scholar
• Lee, S., Y.B. Moh, M. Tabibzadeh, and N. Meshkati. 2017. “Applying the AcciMap Methodology to Investigate the Tragic Sewol Ferry Accident in South Korea.” Applied Ergonomics 59 (Pt B): 517–525. doi:10.1016/j.apergo.2016.07.013.
   PubMed Web of Science ®Google Scholar
• Leveson, N. 2011a. “Applying Systems Thinking to Analyze and Learn from Events.” Safety Science 49 (1): 55–64. doi:10.1016/j.ssci.2009.12.021.
   Web of Science ®Google Scholar
• Leveson, N. 2004. “A New Accident Model for Engineering Safer Systems.” Safety Science 42 (4): 237–270. doi:10.1016/S0925-7535(03)00047-X.
   Web of Science ®Google Scholar
• Leveson, N., N. Dulac, K. Marais, and J. Carroll. 2009. “Moving beyond Normal Accidents and High Reliability Organizations: A Systems Approach to Safety in Complex Systems.” Organization Studies 30 (2-3): 227–249. doi:10.1177/0170840608101478.
   Web of Science ®Google Scholar
• Leveson, N.G. 2011b. Engineering a Safer World: Systems Thinking Applied to Safety. Cambridge, USA, The MIT Press.
   Google Scholar
• Lindberg, A.K., S.O. Hansson, and C. Rollenhagen. 2010. “Learning from Accidents – What More Do we Need to Know?” Safety Science 48 (6): 714–721. doi:10.1016/j.ssci.2010.02.004.
   Web of Science ®Google Scholar
• MAIB 2015. Flooding and capsize of ro-ro passenger ferry Herald of Free Enterprise with loss of 193 lives. Retrieved 7 May, from https://www.gov.uk/maib-reports/flooding-and-subsequent-capsize-of-ro-ro-passenger-ferry-herald-of-free-enterprise-off-the-port-of-zeebrugge-belgium-with-loss-of-193-lives.
   Google Scholar
• Marcum, J.I. 1948. 19. A statistical theory of target detection by pulsed radar. Retrieved August, from https://www.rand.org/pubs/research_memoranda/RM754.html#related.
   Google Scholar
• Matthews, B.W. 1975. “Comparison of the Predicted and Observed Secondary Structure of T4 Phage Lysozyme.” Biochimica et Biophysica Acta 405 (2): 442–451. doi:10.1016/0005-2795(75)90109-9.
   PubMed Web of Science ®Google Scholar
• NTSB 2018. National Transportation Safety Board Preliminary Report Highway: HWY18MH010. https://www.ntsb.gov/investigations/AccidentReports/Pages/HWY18MH010-prelim.aspx
   Google Scholar
• Olsen, N.S. 2013. “Reliability Studies of Incident Coding Systems in High Hazard Industries: A Narrative Review of Study Methodology.” Applied Ergonomics 44 (2): 175–184. doi:10.1016/j.apergo.2012.06.009.
   PubMed Web of Science ®Google Scholar
• Perrow, C. 1984. Normal Accidents: Living with High-Risk Technologies. New York: Basic Books.
   Google Scholar
• Rasmussen, J. 1997. “Risk Management in a Dynamic Society: A Modelling Problem.” Safety Science 27 (2-3): 183–213. doi:10.1016/S0925-7535(97)00052-0.
   Web of Science ®Google Scholar
• Rasmussen, J., Svedung, I. (2000). Proactive risk management in a dynamic society, Swedish Rescue Services Agency.
   Google Scholar
• Reason, J. 1990. Human Error. New York: Cambridge University Press.
   Google Scholar
• Salmon, P.M. 2016. “Bridging the Gap between Ergonomics Methods Research and Practice: Methodological Issues in Ergonomics Science Part II.” Theoretical Issues in Ergonomics Science 17 (5-6): 459–467. doi:10.1080/1463922X.2016.1200693.
   Web of Science ®Google Scholar
• Salmon, P.M., M. Cornelissen, and M.J. Trotter. 2012. “Systems-Based Accident Analysis Methods: A Comparison of Accimap, HFACS, and STAMP.” Safety Science 50 (4): 1158–1170. doi:10.1016/j.ssci.2011.11.009.
   Web of Science ®Google Scholar
• Salmon, P.M., N. Goode, N. Taylor, M.G. Lenné, C.E. Dallat, and C.F. Finch. 2017a. “Rasmussen’s Legacy in the Great Outdoors: A New Incident Reporting and Learning System for Led Outdoor Activities.” Applied Ergonomics 59 (Pt B): 637–648. doi:10.1016/j.apergo.2015.07.017.
   PubMed Web of Science ®Google Scholar
• Salmon, P.M., A. Hulme, G.H. Walker, P. Waterson, E. Berber, and N.A. Stanton. 2020a. “The Big Picture on Accident Causation: A Review, Synthesis and Meta-Analysis of AcciMap Studies.” Safety Science 126: 104650. doi:10.1016/j.ssci.2020.104650.
   Web of Science ®Google Scholar
• Salmon, P.M., A. Hulme, G.H. Walker, P. Waterson, and N.A. Stanton. 2020b. “The Accident Network (AcciNet): a New Accident Analysis Method for Describing the Interaction between Normal Performance and Failure.” Proceedings of the Human Factors and Ergonomics Society Annual Meeting 64 (1): 1676–1680. doi:10.1177/1071181320641407.
   Google Scholar
• Salmon, P.M., Read, G.J.M., Walker, G.H., Stevens, N.J., Hulme, A., McLean, S., Stanton, N.A. (2020c). Methodological issues in systems Human Factors and Ergonomics: Perspectives on the research–practice gap, reliability and validity, and prediction. Human Factors and Ergonomics in the Manufacturing and Service Industries: 1–14.
   Google Scholar
• Salmon, P.M., N.A. Stanton, M. Lenne, D.P. Jenkins, L. Rafferty, and G.H. Walker. 2011. Human Factors Methods and Accident Analysis: Practical Guidance and Case Study Applications. Surrey: Ashgate Publishing, Ltd.
   Google Scholar
• Salmon, P.M., G.H. Walker, G.J.M. Read, N. Goode, and N.A. Stanton. 2017b. “Fitting Methods to Paradigms: Are Ergonomics Methods Fit for Systems Thinking?” Ergonomics 60 (2): 194–205. doi:10.1080/00140139.2015.1103385.
   PubMed Web of Science ®Google Scholar
• Sauer, J., A. Sonderegger, and S. Schmutz. 2020. “Usability, User Experience and Accessibility: Towards an Integrative Model.” Ergonomics 63 (10): 1207–1220. doi:10.1080/00140139.2020.1774080.
   PubMed Web of Science ®Google Scholar
• Shappell, S.A, and D.A. Wiegmann. 2001. “Applying Reason: The Human Factors Analysis and Classification System (HFACS).” Human Factors and Aerospace Safety 1: 59–86.
   Google Scholar
• Shorrock, S.T, and C.A. Williams. 2016. “Human Factors and Ergonomics Methods in Practice: Three Fundamental Constraints.” Theoretical Issues in Ergonomics Science 17 (5-6): 468–482. doi:10.1080/1463922X.2016.1155240.
   Web of Science ®Google Scholar
• Stanton, N.A. 2014. “Commentary on the Paper by Heimrich Kanis Entitled ‘Reliability and Validity of Findings in Ergonomics Research’: where is the Methodology in Ergonomics Methods?” Theoretical Issues in Ergonomics Science 15 (1): 55–61. doi:10.1080/1463922X.2013.778355.
   Google Scholar
• Stanton, N.A. 2016. “On the Reliability and Validity of, and Training in, Ergonomics Methods: A Challenge Revisited.” Theoretical Issues in Ergonomics Science 17 (4): 345–353. doi:10.1080/1463922X.2015.1117688.
   Web of Science ®Google Scholar
• Stanton, N.A., J.W. Brown, K.M.A. Revell, J. Kim, J. Richardson, P. Langdon, M. Bradley, N. Caber, L. Skrypchuk, and S. Thompson. 2022. “OESDs in an on-Road Study of Semi-Automated Vehicle to Human Driver Handovers.” Cognition, Technology & Work 24 (2): 317–332. doi:10.1007/s10111-021-00682-z.
   Web of Science ®Google Scholar
• Stanton, N.A., P. Salmon, D. Harris, A. Marshall, J. Demagalski, M.S. Young, T. Waldmann, and S.W.A. Dekker. 2009. “Predicting Pilot Error: Testing a New Methodology and a Multi-Methods and Analysts Approach.” Applied Ergonomics 40 (3): 464–471. doi:10.1016/j.apergo.2008.10.005.
   PubMed Web of Science ®Google Scholar
• Stanton, N.A., P.M. Salmon, and G.H. Walker. 2018. Systems Thinking in Practice: Applications of the Event Analysis of Systemic Teamwork Method. Boca Raton, FL: CRC Press.
   Google Scholar
• Stanton, N.A., P.M. Salmon, G.H. Walker, and M. Stanton. 2019. “Models and Methods for Collision Analysis: A Comparison Study Based on the Uber Collision with a Pedestrian.” Safety Science 120: 117–128. doi:10.1016/j.ssci.2019.06.008.
   Web of Science ®Google Scholar
• Stanton, N.A, and M.S. Young. 2003. “Giving Ergonomics Away? The Application of Ergonomics Methods by Novices.” Applied Ergonomics 34 (5): 479–490. doi:10.1016/S0003-6870(03)00067-X.
   PubMed Web of Science ®Google Scholar
• Stanton, N.A, and M.S. Young. 1999. “What Price Ergonomics?” Nature 399 (6733): 197–198. doi:10.1038/20298.
   Web of Science ®Google Scholar
• Svedung, I., and J. Rasmussen. 2002. “Graphic Representation of Accident Scenarios: Mapping System Structure and the Causation of Accidents.” Safety Science 40 (5): 397–417. doi:10.1016/S0925-7535(00)00036-9.
   Web of Science ®Google Scholar
• Tanner, W.P, and J.A. Swets. 1954. “A Decision-Making Theory of Visual Detection.” Psychological Review 61 (6): 401–409. doi:10.1037/h0058700.
   PubMed Web of Science ®Google Scholar
• Taylor, R. 1990. “Interpretation of the Correlation Coefficient: A Basic Review.” Journal of Diagnostic Medical Sonography 6 (1): 35–39. doi:10.1177/875647939000600106.
   Web of Science ®Google Scholar
• Thoroman, B., P.M. Salmon, and N. Goode. 2020. “Evaluation of Construct and Criterion-Referenced Validity of a Systems-Thinking Based near Miss Reporting Form.” Ergonomics 63 (2): 210–224. doi:10.1080/00140139.2019.1694707.
   PubMed Web of Science ®Google Scholar
• Underwood, P., and P. Waterson. 2013. “Systemic Accident Analysis: Examining the Gap between Research and Practice.” Accident; Analysis and Prevention 55: 154–164. doi:10.1016/j.aap.2013.02.041.
   PubMed Web of Science ®Google Scholar
• Underwood, P., and P. Waterson. 2014. “Systems Thinking, the Swiss Cheese Model and Accident Analysis: A Comparative Systemic Analysis of the Grayrigg Train Derailment Using the ATSB, AcciMap and STAMP Models.” Accident; Analysis and Prevention 68: 75–94. doi:10.1016/j.aap.2013.07.027.
   PubMed Web of Science ®Google Scholar
• Waterson, P., D.P. Jenkins, P.M. Salmon, and P. Underwood. 2017. “Remixing Rasmussen’: The Evolution of Accimaps within Systemic Accident Analysis.” Applied Ergonomics 59 (Pt B): 483–503. doi:10.1016/j.apergo.2016.09.004.
   PubMed Web of Science ®Google Scholar
• Waterson, P., M.M. Robertson, N.J. Cooke, L. Militello, E. Roth, and N.A. Stanton. 2015. “Defining the Methodological Challenges and Opportunities for an Effective Science of Sociotechnical Systems and Safety.” Ergonomics 58 (4): 565–599. doi:10.1080/00140139.2015.1015622.
   PubMed Web of Science ®Google Scholar