Tackling reliability and construct validity: the systematic development of a qualitative protocol for skill and incident analysis

ABSTRACT

It is important to understand factors contributing to and directly causing sports injuries to improve the effectiveness and safety of sports skills. The characteristics of injury events must be evaluated and described meaningfully and reliably. However, many complex skills cannot be effectively investigated quantitatively because of ethical, technological and validity considerations. Increasingly, qualitative methods are being used to investigate human movement for research purposes, but there are concerns about reliability and measurement bias of such methods. Using the tackle in Rugby union as an example, we outline a systematic approach for developing a skill analysis protocol with a focus on improving objectivity, validity and reliability. Characteristics for analysis were selected using qualitative analysis and biomechanical theoretical models and epidemiological and coaching literature. An expert panel comprising subject matter experts provided feedback and the inter-rater reliability of the protocol was assessed using ten trained raters. The inter-rater reliability results were reviewed by the expert panel and the protocol was revised and assessed in a second inter-rater reliability study. Mean agreement in the second study improved and was comparable (52–90% agreement and ICC between 0.6 and 0.9) with other studies that have reported inter-rater reliability of qualitative analysis of human movement.

KEYWORDS:

• Qualitative methods
• biomechanics
• sport injury

Acknowledgements

The authors would like to acknowledge the significant contributions made by Simon Kemp and Colin Fuller, who led the research group at the RFU, towards the development of the tackle analysis protocol; Beccy Cancea, who assisted with IRR data collection at the RFU; and the participants of the IRR studies. The participation of the High performance and Community rugby departments at the ARU and the Sports medicine department at RFU in developing the protocol was greatly appreciated. The research was undertaken by Trevor Savage at the University of New South Wales former School of Risk and Safety Sciences where Andrew Stuart McIntosh worked and Trevor Savage worked and studied at the time. Bertrand Fréchède made a significant contribution to the development of the protocol and provided helpful feedback during drafting of the manuscript. Andrew Stuart McIntosh is a lead researcher at the Australian Centre for Research into Injury in Sport and its Prevention (ACRISP). ACRISP is one of the designated International Olympic Committee centres for research into the prevention of injury and protection of athlete health. The authors are grateful for the ongoing support and interest of rugby players and administrators who make this and other research possible.

Contributors

Trevor Nicholas Savage and Andrew Stuart McIntosh developed the research protocol. Trevor Nicholas Savage managed the IRR studies and analysed the IRR results with input from Andrew Stuart McIntosh. Trevor Nicholas Savage drafted the manuscript. Both authors edited and approved the final manuscript.

Ethics approval

The study protocol was approved by University of New South Wales Human Research Ethics Committee via the Science Human Research Ethics Advisory Panel: reference 08/2010/30.

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplemental data

Supplemental data for this article can be accessed at http://dx.doi.org/10.1080/02640414.2016.1172722

Additional information

Funding

The parent projects in which this protocol was used were funded by the International Rugby Board (IRB). The reliability assessments and the drafting of this paper were prepared without the financial assistance of the IRB.