Testing the reliability and validity of risk assessment methods in Human Factors and Ergonomics

Abstract

There is growing interest in the use of systems-based risk assessment methods in Human Factors and Ergonomics (HFE). The purpose of this study was to test the intra-rater reliability and criterion-referenced concurrent validity of three systems-based risk assessment approaches: (i) the Systems-Theoretic Process Analysis (STPA) method; (ii) the Event Analysis of Systemic Teamwork Broken Links (EAST-BL) method; and, (iii) the Network Hazard Analysis and Risk Management System (Net-HARMS) method. Reliability and validity measures were obtained using the Signal Detection Theory (SDT) paradigm. Whilst STPA identified the highest number of risks, the findings indicate a weak to moderate level of reliability and validity for STPA, EAST-BL and Net-HARMS. There were no statistically significant differences between the methods across analyses. The results suggest that there is merit to the continued use of systems-based risk assessment methods following a series of methodological extensions that aim to enhance the reliability and validity of future applications.

Practitioner summary The three risk assessment methods produced weak to moderate levels of stability and accuracy regarding their capability to predict risks. There is a pressing need to further test the reliability and validity of safety methods in Human Factors and Ergonomics.

Keywords:

• Risk assessment
• STPA
• EAST-BL
• Net-HARMS
• reliability validity

Acknowledgements

We would like to thank Mrs Kerri Salmon for assisting with the logistical aspects of the study including planning, organising and booking out multiple testing locations. We would also like to thank the three participant groups for their valuable time in taking interest in the use of new methods and applying them on two separate occasions. This involved a considerable amount of travelling, in some cases crossing state lines and borders, just to contribute the necessary expertise.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by an Australian Research Council (ARC) Discovery Project grant [grant number: DP180100806].