Abstract
Aim: Clinical practice improvement carried out in a quality assurance framework relies on routinely collected data using clinical indicators. Herein we describe the development, minimum training requirements, and inter-rater agreement of indicators that were used in an Australian multi-site evaluation of the effectiveness of early psychosis (EP) teams.
Methods: Surveys of clinician opinion and face-to-face consensus-building meetings were used to select and conceptually define indicators. Operationalization of definitions was achieved by iterative refinement until clinicians could be quickly trained to code indicators reliably. Calculation of percentage agreement with expert consensus coding was based on ratings of paper-based clinical vignettes embedded in a 2-h clinician training package.
Results: Consensually agreed upon conceptual definitions for seven clinical indicators judged most relevant to evaluating EP teams were operationalized for ease-of-training. Brief training enabled typical clinicians to code indicators with acceptable percentage agreement (60% to 86%). For indicators of suicide risk, psychosocial function, and family functioning this level of agreement was only possible with less precise ‘broad range’ expert consensus scores. Estimated kappa values indicated fair to good inter-rater reliability (kappa > 0.65). Inspection of contingency tables (coding category by health service) and modal scores across services suggested consistent, unbiased coding across services.
Conclusions: Clinicians are able to agree upon what information is essential to routinely evaluate clinical practice. Simple indicators of this information can be designed and coding rules can be reliably applied to written vignettes after brief training. The real world feasibility of the indicators remains to be tested in field trials.
Acknowledgements
The work of the New South Wales Early Psychosis Research and Evaluation Committee (Jane Beckman, John Brennan, Joanne Gorrell, Anthony Harris, Deborah Howe, Louise Nash, Sian O'Brien, Maryanne O'Donnell, Marc Reynolds, Alan Rosen, Elizabeth Spencer, Maree Teesson, Mark Welch), the New South Wales State Coordinators for Early Psychosis (Bernadette Dagg, Kathy Paterson, and Vickie Ryan) and the support of the former Director of the New South Wales Centre for Mental Health (Beverley Raphael) and her adviser (Kym Scanlon) are gratefully acknowledged. We thank the New South Wales Project Officer Elizabeth Temple.
Declaration of interest: We gratefully acknowledge National Health and Medical Research Council of Australia funding for the CPIN-EP project. The authors alone are responsible for the content and writing of the paper.