Evaluating the Reliability of the Needs in Recovery Assessment (NiRA) with Simulated Patients

Abstract

Introduction

The Needs in Recovery Assessment (NiRA) is a tool designed to support recovery-oriented and person-centred approaches in mental health services through facilitating the identification and prioritisation of needs. The aim of this study was to evaluate the interrater reliability of the NiRA. Method: Ten mental health clinicians from various professional backgrounds used the NiRA to facilitate assessment interviews with Simulated Patients. Completed and semi-completed NiRA forms, questionnaires, and audio-visual recordings of assessment interviews were collected for analysis. The interrater reliability of the NiRA was calculated using percent agreement and Gwet’s Agreement Coefficient (AC)¹. Results: Percent agreement across all items of the finalised tool was 0.84 (item range: 0.55 to 1.0). Overall interrater reliability (Gwet’s AC¹) was 0.70 (95% CI 0.64-0.76) with items ranging from −0.08 to 1.0. Conclusion: The NiRA is a reliable tool and is ready to be trialled in a feasibility study in clinical settings. It is anticipated that the NiRA will facilitate a deeper understanding of service users’ needs and a more targeted approach to meeting unmet needs.

Acknowledgments

The authors acknowledge and thank a number of people who made this study possible. Firstly, to participants who dedicated time for training and participation in this study. Also, Assoc Prof Adam Montagu and the actors who adopted the roles of SPs from Adelaide Health Simulation, University of Adelaide. We also thank Dr Scott Clark for providing his clinical expertise and Prof Malcolm Battersby for facilitating recruitment through the Southern Adelaide Local Health Network.

Declarations of interest

There are no conflicts to report.

List of supplemental digital content

• Supplemental Digital Content 1: Final paper-based version of NiRA

• Supplemental Digital Content 2: Version of NiRA used in reliability study

• Supplemental Digital Content 3: Study details

• Supplemental Digital Content 4: NiRA Reliability Study Questionnaire

• Supplemental Digital Content 5: SP case – Sophie Langford

• Supplemental Digital Content 6: SP case – Lucy Parker

• Supplemental Digital Content 7: SP case – Kai Stevenson

• Supplemental Digital Content 8: SP case – Paul Nolan

• Supplemental Digital Content 9: Mental health clinician post-encounter questionnaire

• Supplemental Digital Content 10: SP questionnaire

• Supplemental Digital Content 11: SP accuracy checklist

• Supplemental Digital Content 12: Priority needs identified on NiRA

Additional information

Funding

The corresponding author was supported to undertake this research through an Australian Government Research Training Program Scholarship.