Quality indicators for diagnosis and treatment of respiratory tract infections in general practice: A modified Delphi study

Abstract

Objective.To develop a set of quality indicators focusing on the diagnosis and treatment of respiratory tract infections in general practice. Design. A modified 2-round Delphi study. Setting.General practice. Subjects. A panel of 27 experts (13 countries) comprising mainly general practitioners, clinical microbiologists, and clinical pharmacologists were asked to rate the relevance of 59 quality indicators for diagnosis and treatment of respiratory tract infections with regard to reducing antimicrobial resistance and improving patient health. A thorough literature review was carried out to ensure that all potential quality indicators were considered. Outcome. Consensus for a quality indicator was reached if ≥75% of experts scored the item ≥5 on a 7-point Likert scale, ranging from 1 (=completely disagree) through 4 (=uncertain) to 7 (=completely agree). Results. A 96% response rate was achieved in both Delphi rounds. A total of 41 of the proposed 59 quality indicators attained consensus. None of the quality indicators focusing on the diagnostic process achieved consensus. Consensus was attained for 14 quality indicators focusing on the decision regarding antibiotic treatment and for 27 quality indicators focusing on the choice of antibiotics. Conclusion. This study resulted in a final set of 41 quality indicators concerning respiratory tract infections in general practice. These quality indicators may be used to strengthen general practitioners’ focus on their management of patients with respiratory tract infections and to identify where it is possible to make improvements.

Key Words::

• Antibiotics
• Delphi technique
• family practice
• general practice
• quality indicator
• respiratory tract infection

The majority of antibiotics prescribed in general practice are attributable to treatment of patients with respiratory tract infections, despite the fact that many of these infections are harmless, self-limiting conditions. The varying problems in the quality of diagnosis and treatment of respiratory tract infections induce a need for development of quality indicators.

• For assessing the quality of diagnosis and treatment of respiratory tract infections in general practice a panel of experts agreed that a total of 41 quality indicators were relevant.

• These disease-specific quality indicators may be used to strengthen general practitioners’ focus on their management of patients with respiratory tract infections and to identify where it is possible to make improvements.

Antibiotic resistance is a growing problem worldwide and excessive and inappropriate use of antibiotics is considered to be the most important cause [1–3]. The majority of antibiotics prescribed in general practice are attributable to treatment of patients with respiratory tract infections (RTIs) despite the fact that many of these infections are harmless, self-limiting conditions or caused by virus [4,5].

General practitioners’ (GPs) prescribing patterns for RTIs differ considerably between countries and, moreover, there are large seasonal variations in antibiotic prescribing in many countries [6,7].

The varying problems in the quality of diagnosis and treatment of RTIs in different countries induce a need for development of quality indicators that can be applied in countries with high as well as low antibiotic use. Quality indicators are measurable elements of practice for which there is evidence or consensus that they reflect quality [8,9]. Indicators allow comparison to be made between practices over time or against quality standards and such comparisons may stimulate and motivate change [10]. Most indicators have been developed for use in hospitals, but they are also necessary in general practice, as the majority of antibiotics are prescribed here. Previously, so-called drug-specific indicators have been developed to assess the quality of antibiotic use in primary care in Europe [11]. However, the value of these indicators is limited because it is difficult to evaluate the quality of an antibiotic treatment without knowledge about the background for treatment or the diagnostic process performed. There is a need for the development of quality indicators that encompass the diagnostic process, the decision concerning antibiotic treatment and the choice of antibiotics in relation to the presumed diagnosis. The aim of this study was to develop a set of quality indicators focusing on the diagnostic process and treatment of RTIs in general practice by means of the Delphi method.

Material and methods

This study is part of the EU-funded project HAPPY AUDIT which aims to improve the quality of diagnosis and treatment of RTIs in general practice [12]. The HAPPY AUDIT project involves GPs from Lithuania, Russia, Argentina, Spain, Sweden and Denmark.

A two-round modified Delphi study was conducted from April to July 2008. The Delphi method is in essence a series of sequential questionnaires or “rounds” interspersed by controlled feedback, seeking to gain the most reliable consensus of a group of experts [13]. Quasi-anonymity was sustained in this study, meaning that the respondents may be known to one another, but their judgements and opinions remain strictly anonymous [14].

The panel of experts

Studies employing the Delphi method make use of experts, who are individuals experienced in the topic being investigated [15]. A panel of 27 experts were invited: 19 GPs, four clinical microbiologists, two clinical pharmacologists, one full-time senior researcher (MD) and one pharmacist. All invited experts accepted to participate. The 27 experts originated from 13 countries and the panel comprised members of European projects concerning RTIs and different European organisations (appendix).

Study design

The flowchart (Figure 1) illustrates how the list of proposals for quality indicators for the Delphi study was generated. At first, members of the HAPPY AUDIT steering committee were invited to a workshop focusing on development of quality indicators. All members of the steering committee were clinicians or scientists with profound experience in RTIs in general practice. The workshop consisted of plenary sessions as well as smaller working groups and resulted in a list of 20 proposals. Subsequently an e-mail correspondence was initiated. The members of the steering committee were asked to add additional proposals according to national guidelines. A thorough literature review was carried out to ensure that all potential quality indicators were considered. A draft list of 87 proposals was attained. The draft list was edited by the research group (the authors) by removing duplicates and grouping equal proposals. In the next step the edited list of 58 quality indicators was sent to each of the 27 experts in the Delphi panel for additional suggestions and comments. This resulted in a new draft list of 82 proposals. Again this draft list was shortened by the research group by removing duplicates and grouping equal proposals. A final list of 59 proposals for quality indicators for diagnosis and treatment of RTIs was established.

Figure 1. Process of the development of proposals for quality indicators. n = number of proposals for quality indicators.

The 59 quality indicators were then classified according to the International Classification of Primary Care (ICPC) into groups concerning: acute sinusitis, acute otitis media, acute tonsillitis/pharyngitis, acute bronchitis, pneumonia, and exacerbation of chronic obstructive pulmonary disease (COPD) [16,17]. Some quality indicators were aggregated according to the NICE guidelines in lower respiratory tract infection (LRTI) comprising acute bronchitis, bronchiolitis, pneumonia, and tracheitis and in respiratory tract infections (RTI) comprising any infectious disease of the upper or lower respiratory tract [18].

The indicators focused on the quality of (1) the diagnostic process, (2) the decision concerning antibiotic treatment, and (3) the choice of antibiotics (narrow-spectrum penicillin, broad-spectrum penicillin +/-clavulanic acid, macrolides, cephalosporins, or quinolones) [19].

The experts were asked to rate the relevance of the 59 proposed quality indicators on a 7-point Likert scale, ranging from 1 (=completely disagree) through 4 (=uncertain) to 7 (=completely agree). Each indicator had to be assessed for two dimensions [11]:

• relevance in measuring quality focusing on microbiological issues, i.e. reduction in antimicrobial resistance;

• relevance in measuring quality focusing on patient health benefit, i.e. reduction in symptoms and/or duration of the disease.

The agreement rate was defined as the percentage of experts rating the quality indicator >5 on the 7-point Likert scale in the second Delphi round. Consensus for an indicator was achieved if the agreement rate was >75% for one of the dimensions mentioned above. The definition of consensus was established before data analysis [20,21].

Between the two Delphi rounds experts were given two types of feedback for each of the 59 indicators for the two dimensions:

• A bar chart showing the distribution of ratings in the first Delphi round with the experts’ own rating marked in the figure (Figure 2).

Figure 2. Example of feedback on the experts’ rating between the two Delphi rounds. The experts rating is marked as “your score”.

• Comments from the experts collected during the first Delphi round.

All 59 quality indicators were rated in both Delphi rounds. Questionnaires in English were distributed electronically. Data were analysed using Stata, version 10.0 [22].

Results

A total of 41 of the proposed 59 quality indicators attained consensus for at least one dimension after the second Delphi round (Tables I–III). Of the 41 quality indicators 40 were found relevant for reducing antimicrobial resistance. Only two quality indicators were found relevant for patient health benefit: Patients with discharging ear treated with antibiotics and patients with acute tonsillitis/ pharyngitis treated with narrow-spectrum penicillin. One quality indicator achieved consensus on both dimensions: Patients with acute tonsillitis/pharyngitis treated with narrow-spectrum penicillin (data not shown).

Table I. Quality indicators focusing on the diagnostic process.

Quality indicators	Relevance for antimicrobial resistance	Relevance for patient health benefit
Patients with acute sinusitis:
Number of patients with symptoms for less than 1 week	42 (2)	35 (1)
Number of patients examined with a CRP test	42 (4)	35 (4)
Patients with acute otitis media:
Number of patients >2 years with symptoms for less than 3 days	23 (2)	23 (3.5)
Patients with acute tonsillitis/pharyngitis:
Number of patients examined with a StrepA test	46 (4)	50 (4.5)
Number of patients fulfilling only 1 Centor criterion ^2 examined with a StrepA test	27 (3)	15 (4)
Patients with acute lower respiratory tract infections:
Number of patients examined with a CRP test	38 (4)	27 (4)
Number of patients examined with an X-ray of thorax	23 (2.5)	31 (4)
Number of patients not examined with either a CRP test or X-ray of thorax	23 (4)	35 (4)

Notes: The values represent agreement rates¹ in% (median on a Likert scale, range 1-7). CRP test =C-reactive protein rapid test. Strep A test =rapid Streptococcus A antigen detection test. 1 Percentage of experts who scored the dimension ≥5 in the second Delphi round (n =26) on a Likert scale, range 1-7. ² Fever >38.5, tonsillar exudate, no coughing, enlarged angular glands.

Table II. Quality indicators focusing on the decision concerning treatment with antibiotics.

Quality indicators	Relevance for antimicrobial resistance	Relevance for patient health benefit
Patients with acute sinusitis:
Number of patients treated with antibiotics	92* (7)	35 (4)
Number of patients treated with antibiotics without a diagnostic test	38 (4)	15 (4)
Number of patients treated with antibiotics with a CRP test < 10 mg/l	73 (6)	50 (4.5)
Patients with acute otitis media (AOM):
Number of patients treated with antibiotics	92* (7)	50 (4.5)
Number of patients < 2 years treated with antibiotics	85* (7)	69 (5.5)
Number of patients >2 years with less than 3 days of symptoms of AOM treated with antibiotics	96* (7)	46 (6)
Number of patients with discharging ear treated with antibiotics	73 (6)	85* (6)
Patients with acute tonsillitis/pharyngitis:
Number of patients treated with antibiotics	88* (7)	65 (5)
Number of patients treated with antibiotics without a StrepA test	62 (6)	31 (4)
Number of patients treated with antibiotics with a positive StrepA test	77* (6.5)	50 (4.5)
Number of patients treated with antibiotics with a negative StrepA test	69 (6.5)	27 (4)
Patients with acute bronchitis:
Number of patients treated with antibiotics	96* (7)	35 (4)
Patients with pneumonia:
Number of patients treated with antibiotics	62 (5)	58 (6)
Patients with acute exacerbation of chronic obstructive
pulmonary disease:
Number of patients treated with antibiotics	88* (6)	50 (4.5)
Number of patients not fulfilling all the Anthonisen criteria ^2 treated with antibiotics	88* (7)	62 (5)
Patients with acute lower respiratory tract infections:
Number of patients treated with antibiotics	85* (7)	50 (4.5)
Number of patients treated with antibiotics without a preceding CRP test or X-ray of thorax	31 (4)	15 (4)
Number of patients treated with antibiotics with a CRP test < 20 mg/l	81* (6.5)	42 (4)
Patients with acute respiratory tract infections:
Number of patients treated with antibiotics	85* (7)	50 (4.5)
Number of patients with no history of penicillin allergy treated with macrolides	92* (7)	42 (4)

Notes: The values represent agreement rates ¹ in% (median on a Likert scale, range 1-7). CRP test =C-reactive protein rapid test. Strep A test =rapid Streptococcus A antigen detection test. *Consensus (agreement rate ≥75%). ¹ Percentage of experts who scored the dimension ≥5 in the second Delphi round (n =26) on a Likert scale, range 1-7. ² Increased dyspnoea, increasing expectorate, and increasing purulence of expectorate.

Table III. Quality indicators focusing on choice of antibiotics (relevance for antimicrobial resistance).

Quality indicators	Patients with acute sinusitis	Patients with acute otitis media	Patients with acute tonsillitis/pharyngitis	Patients with pneumonia	Patients with acute exacerbation of COPD	Patients with acute LRTI
Number of patients treated with narrow-spectrum penicillin	85* (7)	92* (7)	96* (7)	92* (7)	62 (5)	88* (7)
Number of patients treated with broad-spectrum penicillin +/-clavulanic acid	92* (7)	92* (7)	92* (7)	100* (7)	92* (6)	92* (7)
Number of patients treated with macrolides	88* (7)	85* (7)	85* (7)	88* (6)	77* (6)	88* (6.5)
Number of patients treated with cephalosporins	81* (7)	81* (7)	88* (7)	81* (6)	73 (6)	81* (7)
Number of patients treated with quinolones	81* (7)	81* (7)	65 (6)	81* (6.5)	85* (6)	81* (7)

Notes: The values represent agreement rates ¹ in% (median on a Likert scale, range 0-7). COPD =chronic obstructive pulmonary disease. LRTI =lower respiratory tract infection. *Consensus (agreement rate ≥75%). ¹ Percentage of experts who scored the dimension ≥5 in the second Delphi round (n =26) on a Likert scale, range 1-7.

None of the quality indicators focusing on the diagnostic process achieved the predefined consensus, i.e. an agreement rate >75% (Table I). Highest agreement rate (50%) was obtained for the quality indicator: Patients with tonsillitis/pharyngitis examined with a StrepA test. For CRP rapid test the highest agreement rates were 42% (acute sinusitis) and 38% (LRTI), respectively.

Consensus was attained for 14 of the 20 quality indicators focusing on the decision about antibiotic treatment (Table II). The highest agreement rates were related to the relevance for antimicrobial resistance, and the majority of experts agreed on the indicators concerning the number of patients treated with antibiotics. For acute sinusitis, 73% of experts agreed on the indicator concerning the patients treated with antibiotics with a CRP < 10 mg/l and for acute tonsillitis/pharyngitis, 77% of experts agreed on the indicator concerning patients treated with antibiotics with a positive Strep A.

Consensus was attained for 27 of the 30 quality indicators focusing on the choice of antibiotics (Table III).

Discussion

Main findings

For assessing the quality of diagnosis and treatment of RTIs in general practice the panel of experts agreed that a total of 41 quality indicators were relevant. Almost all of these indicators were found to be relevant for reducing antimicrobial resistance while only two were found to be relevant for patient health benefit. None of the quality indicators focusing on the diagnostic process achieved consensus. The experts, however, agreed on indicators based on both StrepA (acute tonsillitis/pharyngitis) and CRP (LRTI) in relation to the decision concerning antibiotic treatment. About two-thirds of the quality indicators focusing on the decision regarding antibiotic treatment and almost all quality indicators concerning choice of antibiotics achieved consensus.

Strengths and limitations

In Delphi studies it is common to start the study using a qualitative approach by generating ideas that are used to form the questionnaire items for the subsequent quantitative rounds [23]. In our study proposals were collected both from members of the HAPPY AUDIT steering committee and from all 27 experts in the Delphi panel. The development of the quality indicators was initiated at a workshop. One of the merits of this procedure is the opportunity to discuss potential proposals and thereby inspire one another for further proposals. One of the drawbacks of a workshop is the risk of missing some quality indicators and a thorough review was carried out to ensure that all potentials were considered.

The experts were told that their assessments should be based on what they found to be best practice, irrespective of national or local conditions or potential access to laboratory testing. However, the tradition of use of laboratory tests in primary healthcare differs considerably between countries and the heterogenous availability of, for example, Strep A and CRP rapid tests might have influenced the uneven assessment of the diagnostic quality indicators.

The classical Delphi method has four Delphi rounds, and one may argue that more than two Delphi rounds were needed to reach a stable consensus. We decided to predefine the number of rounds, so the experts knew from the very beginning how many rounds the study consisted of. Too many rounds may lead to fatigue among participants and the number of Delphi rounds was kept at a minimum to ensure a high response rate. We obtained a response rate of 96% in both Delphi rounds.

Conducting a study including different countries may create language barriers, but all the experts included in our study were proficient in English. To diminish potential misunderstandings concerning the interpretation of the quality indicators we provided the group of experts with manuals with the definition of a quality indicator and including cases explaining the interpretation of potential quality indicators. This contributed to a common understanding of the concept of quality indicators in patients with respiratory tract infections.

Reliability

The Delphi method has been criticized for having no evidence of reliability, and the results of a Delphi study reflect the opinion specifically of the invited panel [15]. In our panel, all 27 experts included were experienced in general practice. Most of them were specialists in general practice, and the representativeness of the panel was ensured by including specialists from different specialties related to the diagnosis and treatment of RTIs, among these microbiology and pharmacology. All experts had been involved in a number of research studies or quality improvement activities focusing on patients with RTIs in primary care. The international representativeness of the panel was ensured by inviting experts from 13 different countries. According to the face validity we find it important that the main part of our panel consisted of GPs. It is essential that GPs are involved in the development of quality indicators for use in general practice, and it is important that they find them applicable for use in daily practice.

Comparison with other studies

We found that the agreement rate for the quality indicators varied considerably when focusing on the relevance for patients’ health benefit and only two indicators reached consensus if this dimension was taken into account. Obviously, it was harder for the experts to agree on which indicators were relevant for patient health benefit than on which were relevant to reducing antimicrobial resistance. The study by Coenen et al. attained a result similar to ours with the indicators scoring higher on the dimension resistance than on patient health benefit [11]. The Coenen study was, however, designed to develop so-called drug-specific quality indicators, but it did not include indicators related to diagnoses or the diagnostic process. The value of drug-specific indicators is limited by the absence of knowledge concerning

the background for treatment. In our study we developed disease-specific quality indicators focusing on different RTIs in general practice.

Perspectives

This final set of 41 disease-specific quality indicators or parts of it is the first step in improving the quality of diagnosis and treatment of RTIs. They can be used to strengthen GPs’ focus on their management of patients with RTIs and to identify where it is possible to make improvements. Policy-makers might also apply the set of quality indicators as a tool to assess quality and for implementation of new strategies in general practice.

Future studies employing the developed set of quality indicators should focus on defining benchmarks for diagnosis and treatment of RTIs related to clinical practice and local conditions.

Appendix 1

Greece (n =1), Portugal (n =1), Croatia (n =1), United Kingdom (n =1), Belgium (n =1), The Netherlands (n =2), Norway (n =2), Argentina (n =2), Russia (n =2), Spain (n =3), Lithuania (n =3), Sweden (n =4) and Denmark (n =4)

Genomics to combat Resistance against Antibiotics in Community-acquired LRTI in Europe (GRACE) http://www.grace-lrti.org/

European Surveillance of Antimicrobial Consumption (ESAC) http://www.esac.ua.ac.be/ Changing behavior of Health care professionals And the general public towards a More Prudent use of antimicrobial agents (CHAMP)

Health Alliance for Prudent Prescribing, Yield And Use of antimicrobial Drugs In the Treatment of Respiratory Tract Infections (HAPPY AUDIT) http://www.happyaudit.org/

World Organization of Family Doctors (WONCA) http://www.woncaeurope.org/

European Drug Utilization Research Group (Euro– DURG) http://www.eurodurg.com/

World Health Organization, Collaborating Centre for Drug Statistics Methodology (WHO-CC) http:// www.whocc.no/

General Practice Respiratory Infections Network (GRIN) http://www.almen.dk/grin2008/

Acknowledgements

The authors would like to thank all 27 participants in the Delphi study for their valuable contribution to this project.

Funding

This study was supported by grants from the EU-funded HAPPY AUDIT project, the Research Foundation for General Practice in Denmark and the University of Southern Denmark.

Conflict of interests

All authors declare that they have no conflict of interests.