Development and reliability of performance indicators for measuring adherence to a guideline for depression by insurance physicians

Abstract

Introduction. We wanted to measure adherence to the guideline for depression in disability assessments. The research questions we addressed were: How can we develop performance indicators (PIs) for adherence to the Dutch guideline for disability assessment of patients with depression and how can we measure the quality of the scores? What is the inter-rater reliability of these PIs? What is the quality of the PI scores?

Methods. PIs, developed by the researchers, were reviewed on various aspects, by a panel of seven experts in several consulting rounds. After adjustments, senior insurance physicians (IPs) attended two training sessions and scored the PIs on 10 different simulated case reports. Two researchers developed proxy ‘gold standard’ scores for these 10 case reports. To assess the inter-rater reliability and the quality of the scores, we calculated the intra-class correlations (ICC) and 95%% confidence intervals (CI) of the PI scores and of the PI scores compared to the proxy ‘gold standard’, respectively.

Results. Six specific and relevant PIs resulted from the consultation of the panel of experts. The PI scores for the 10 case reports, rated by seven (of the eight) senior IPs who completed both training sessions, showed that the PIs were not reliable at individual level (ICC  ==  0.543; 95%% CI 0.426–0.642). However, the ICC became more reliable as an average of two raters was calculated (ICC  ==  0.704). The ICC of the PI scores with the proxy ‘gold standard’ was 0.538 (95%% CI 0.419–0.640), but the quality was higher when calculated as an average of two raters (ICC  ==  0.700).

Conclusion. The PIs for adherence to the guideline were sufficiently reliable, and the quality of their scores was adequate if at least two well-trained raters were involved. The senior IPs evaluated the feasibility of the PIs as good, with a prerequisite of sufficient training. This method may be interesting for measuring guideline adherence and quality of disability assessments in general.

Keywords:

• Performance indicators
• reliability
• insurance medicine
• guidelines

Acknowledgements

The authors thank the senior insurance physicians who participated in this research. The Research Center for Insurance Medicine AMC-UMCG-UWV-VUmc, in Amsterdam, is a joint initiative of the Academic Medical Center (AMC), University Medical Center Groningen (UMCG), National Institute for Employment Benefit Schemes (UWV), and the VU University Medical Centre (VUmc).

Declaration of interest:

The authors declare that they participated in the study and made the following contributions to the study, and that they have seen and approved the final version. They declare that they have no competing interests. Funding sources: FZ, JRA and AJMS are (partially) funded by UWV. The study sponsor had no (decisive) role in the study design, in the collection, analysis, or interpretation of the data, in the writing of the case reports, or in the decision to submit the paper for publication. The design of this study was laboratorial and for data collection fictitious but realistic case reports were used. Consequently, the Medical Ethics Committee was not involved.