Evaluation of construct and criterion-referenced validity of a systems-thinking based near miss reporting form

Abstract

The validity of methods is an ongoing issue in ergonomics. Inconsistent definitions and approaches to evaluation exacerbate this challenge. In this study, the construct and criterion-referenced validity of a new near miss reporting form was evaluated to determine the extent to which it comprehensively captures near miss incidents and is aligned with the systems thinking approach to accident causation. Interview data were used as the reference standard in the evaluation. Using signal detection theory (SDT), a high average hit rate (HR), predictive value (PV) and sensitivity index (SI) were found, with an almost perfect ranking for the index of concordance. The findings show that the reporting form has strong construct and criterion-referenced validity. It is proposed that the approach used in this study could be used by researchers and practitioners when testing the validity of incident data collection tools.

Practitioner summary: The validity of methods is a key issue in ergonomics. In this study, we test the validity of a near miss reporting form using interview data as a standard. This approach could be used by practitioners when testing the validity of other ergonomics methods.

Keywords:

• Near miss
• systems thinking
• construct validity
• criterion-referenced validity
• Incident reporting

Acknowledgements

The authors thank Dr. Colin Solomon for his input to the paper through discussions around validity, reliability, and accuracy as well as his comments on drafts of the paper. The authors thank Ms. Lauren Coventon B.S. Hons (Psych) for her assistance with the coding of the data. The authors thank the reviewer’s for taking the time to provide comment on the improvement of the article.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This project was supported by funding from the Australian Research Council (ARC; LP150100287) in partnership with Australian Camps Association, Outdoor Council of Australia, The Outdoor Education Group, Sport and Recreation Victoria, Victorian YMCA Accommodation Services Pty Ltd, Outdoors Victoria, Outdoor Recreation Industry Council (Outdoors NSW), Outdoors WA, Outdoors South Australia, Queensland Outdoor Recreation Federation, Wilderness Escape Outdoor Adventures, Venture Corporate Recharge, and Christian Venues Association. Paul Salmon’s contribution was funded through his current Australian Research Council Future Fellowship (FT140100681).