205
Views
4
CrossRef citations to date
0
Altmetric
Research Papers

Development and reliability of performance indicators for measuring adherence to a guideline for depression by insurance physicians

, , , &
Pages 2535-2543 | Accepted 01 Apr 2011, Published online: 17 May 2011
 

Abstract

Introduction. We wanted to measure adherence to the guideline for depression in disability assessments. The research questions we addressed were: How can we develop performance indicators (PIs) for adherence to the Dutch guideline for disability assessment of patients with depression and how can we measure the quality of the scores? What is the inter-rater reliability of these PIs? What is the quality of the PI scores?

Methods. PIs, developed by the researchers, were reviewed on various aspects, by a panel of seven experts in several consulting rounds. After adjustments, senior insurance physicians (IPs) attended two training sessions and scored the PIs on 10 different simulated case reports. Two researchers developed proxy ‘gold standard’ scores for these 10 case reports. To assess the inter-rater reliability and the quality of the scores, we calculated the intra-class correlations (ICC) and 95%% confidence intervals (CI) of the PI scores and of the PI scores compared to the proxy ‘gold standard’, respectively.

Results. Six specific and relevant PIs resulted from the consultation of the panel of experts. The PI scores for the 10 case reports, rated by seven (of the eight) senior IPs who completed both training sessions, showed that the PIs were not reliable at individual level (ICC  ==  0.543; 95%% CI 0.426–0.642). However, the ICC became more reliable as an average of two raters was calculated (ICC  ==  0.704). The ICC of the PI scores with the proxy ‘gold standard’ was 0.538 (95%% CI 0.419–0.640), but the quality was higher when calculated as an average of two raters (ICC  ==  0.700).

Conclusion. The PIs for adherence to the guideline were sufficiently reliable, and the quality of their scores was adequate if at least two well-trained raters were involved. The senior IPs evaluated the feasibility of the PIs as good, with a prerequisite of sufficient training. This method may be interesting for measuring guideline adherence and quality of disability assessments in general.

Acknowledgements

The authors thank the senior insurance physicians who participated in this research. The Research Center for Insurance Medicine AMC-UMCG-UWV-VUmc, in Amsterdam, is a joint initiative of the Academic Medical Center (AMC), University Medical Center Groningen (UMCG), National Institute for Employment Benefit Schemes (UWV), and the VU University Medical Centre (VUmc).

Declaration of interest:

The authors declare that they participated in the study and made the following contributions to the study, and that they have seen and approved the final version. They declare that they have no competing interests. Funding sources: FZ, JRA and AJMS are (partially) funded by UWV. The study sponsor had no (decisive) role in the study design, in the collection, analysis, or interpretation of the data, in the writing of the case reports, or in the decision to submit the paper for publication. The design of this study was laboratorial and for data collection fictitious but realistic case reports were used. Consequently, the Medical Ethics Committee was not involved.

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.