Quality assessment of clinical practice guidelines

Overview of evidence-based medicine

Fundamentally, evidence-based medicine (EBM) represents an effort to improve the quality of information about benefits of healthcare services and their related effectiveness [1]. One of the most important contributions of EBM is to draw attention to gaps in the scientific evidence upon which healthcare decisions are made, thereby helping to shape a research agenda that focuses on answering the questions important to medicine. Clinical practice guidelines (CPGs) take the next step, helping to translate science into policy for practitioners and administrative decision-makers. The EBM movement has given rise to evidence-based practice guidelines, which offer clinical guidance as well as documentation of the quality of evidence or opinion supporting practice recommendations. CPGs are used increasingly by government agencies and professional organizations around the world to improve patient care [2,3]. There is a pressing need for recognized criteria to assess guidelines that are valid, clear, cost-efficient, usable and reliable [4,5,101].

EBM has been recognized as an effective mechanism for not only improving healthcare quality, but also for reducing medical errors precipitated in part by clinical practice variation [1]. It provides for an integration and translation of evidence, experience and values into clinical and policy decision-making. EBM provides an organized structure to explicitly link clinical and public health policy to a systematic examination of the quality of supporting scientific evidence [6]. EBM can be a tool to assess the effectiveness of a new technology and its benefits, thereby informing decisions related to health policy and access to new technologies [1,5,6]. An important application of EBM is in the development of CPGs.

Clinical practice guidelines

The Institute of Medicine (IOM) defined CPGs as systematically developed statements to assist practitioners’ and patients’ decisions about appropriate healthcare for specific clinical conditions and/or circumstances [5]. These statements should be based on the best available scientific evidence and practical knowledge. However, an increasing number of CPGs and variations in their recommendations have stimulated discussion about their value.

Why is a quality assessment of clinical practice guidelines important?

Over the past two decades, CPGs have become an increasingly popular tool for the implementation of scientifically based clinical information to improve the quality of healthcare [6,7,8]. They are intended to present a synthesis of current evidence and recommendations performed by expert clinicians and may affect the practice of large numbers of physicians. Therefore, there must be agreement as to how the guidance is generated and how the evidence is interpreted [10]. Guideline developers attempt to identify, appraise and collate the best evidence to ensure that the highest quality information is available for clinicians and patients. It would also be beneficial if guideline developers could follow a common standard of reporting similar to that used by consolidated standards of reporting trials (CONSORT) for randomized controlled trials [9,11]. The resulting quality assessments should be made available to practitioners, policy-makers and the public to facilitate informed decision-making about the quality and usefulness of particular guidelines. Coleman indicated that published sources of evidence-based guidance were used in clinical practice, but there were clear differences in knowledge, use and perceived influence of different sources of guidance among the professionals based in different healthcare settings [12]. Those responsible for health policy were much more likely to believe that evidence-based guidance had influenced their practice than doctors who provided clinical care in hospitals or primary care. Their study showed that awareness, use and perceived impact of evidence-based guidance was much greater among those involved in health policy and administration, responsible for determining patient access issues or purchasing healthcare products and services, than among consultants in hospitals or primary care physicians.

Is there doubt about the quality of published CPGs?

CPGs are now a common feature of clinical practice [3,13]. They are expected to facilitate more consistent, effective and efficient medical practice. In principle, a good guideline is one that eventually leads to improved patient outcomes. However, applying guidelines to individual care is likely to require the exercise of professional judgment, even when recommendations are properly linked to evidence. Cranney assessed the quality of osteoporosis guidelines produced in the 1998–2001 period [14]. The Appraisal Instrument for Clinical Guidelines (version 1, 1999) was used to assess the quality of guidelines. It consists of 37 items in three dimensions with yes, no and not sure format. This instrument was used widely in the UK and Canada during that period. It had shown acceptable reliability and criterion validity. They found the methodological quality of current osteoporosis guidelines was low, although their scores for clinical content were high and no guidelines covered the dissemination issue. Choudhry concluded that any influence that CPG authors experience from their interactions with pharmaceutical companies may be transmitted to the CPG readers [15]. If CPG authors have relationships that pose a potential conflict of interest, readers may wish to know about them to evaluate the merit to those guidelines. Thus, a systematic assessment of the process used in CPG development can provide users of these recommendations a means to mitigate potential biases.

Potential explanation for why recommendations are different among CPGs

Over the past 20 years, guidelines have been developed to bridge the gap between research and practice. There has been a concerted effort to base clinical decisions on research evidence and to make this evidence available globally [9,101]. Since bibliographic databases (e.g. Medline) are readily available, one might expect that this would lead to international consensus on the evidence chosen to support recommendations for clinical care and a consequent convergence of recommendations made in guidelines. Nevertheless, clinical recommendations often differ in guidelines on the same topic [16,17]. Investigators suggest that differences are due to cultural factors, such as differing expectations of apparent risks and benefits, socio-economic factors, varied characteristics of healthcare systems, insufficient evidence, differing interpretations of evidence, the influence of professional bodies and unsystematic guideline development methods.

What factors might influence the quality of CPGs?

Some guidelines published in the peer-reviewed medical literature during the past decade do not adhere well to methodological standards. While all areas of guideline development may benefit, the greatest improvement is needed in the areas of identification, evaluation and synthesis of the scientific evidence. There is evidence that guidelines can improve clinical practice, but their successful utilization is dependent on many factors, including the clinical context, methods of development, dissemination and implementation [101,102]. Successfully addressing all of these issues in routine practice can prove difficult, but is necessary if CPGs are to improve the quality of healthcare.

There are numerous potential barriers to the development of good clinical guidelines [8,101]. Some CPGs may not have been developed by a fully multidisciplinary group that is representative of those clinicians who will use them or patients affected by them, with the result that there is a lack of ownership. Value judgements made by a guideline group may not accurately reflect those of the patients. The CPGs may include recommendations that do not comprehensively take into account the scientific evidence, potentially leading to suboptimal or ineffective practice. There can be insufficient, misleading or misinterpretations of the scientific evidence about what to recommend. Guideline development groups may lack the time, resources and skill to gather and scrutinize evidence in detail. Recommendations may be biased by the opinions, clinical experience and composition of the guideline group. Conflicts or variations in guidelines from different professional bodies can lead to confusion and frustration among practitioners wanting to utilize CPGs.

Literature on quality assessment of CPGs

Despite improvement over time, the quality of practice guidelines developed by specialty societies may be unsatisfactory in some cases [18]. Increasing concern is related to the number of guidelines of low quality and guidelines that contain conflicting recommendations [5,15,16]. A properly performed evaluation of the scientific evidence is critical in ensuring the scientific validity of a guideline. An important goal for guideline use is to increase the efficiency in the utilization of healthcare resources. According to Shaneyfelt, almost 60% of guidelines reviewed did not mention costs and only 14% provided any quantitative cost estimates [5]. Clearly, if guidelines are to improve the cost-efficiency of healthcare, greater attention must be given to economic analysis. Graham concluded that the quality of all CPGs in Canada should be assessed in a systematic fashion by an independent body using a standardized appraisal instrument [21].

The AGREE Collaboration published the international appraisal instrument for CPGs, the ‘AGREE’ instrument [18]. It was tested on 100 guidelines selected from 11 participating countries by 194 independent appraisers. The median time for appraising a guideline was 1.5 h, including reading of the guideline and completing the appraisal instrument [8]. This adoption of a common standard has the potential for improving the consistency and quality of the reporting of guideline development worldwide and provides a framework to encourage international comparison of CPGs based on the quality of their development.

The AGREE instrument identifies criteria for the development of high-quality CPGs [8]. CPGs should contain a specific statement about the overall objective(s), clinical questions and descriptions of the target population. They should provide information about the composition, discipline and relevant expertise of the guideline development group, and they should involve patients in their development. They should also clearly define the target users, with pilot testing among them prior to CPG publication. CPGs should provide detailed information on the search strategy, the inclusion and exclusion criteria for selecting the evidence and the methods used to formulate the recommendations. Recommendations should be linked explicitly to the supporting evidence, with a discussion of the health benefits, side effects and risks. There should be an external review before publication, with detailed information about the procedure for updating the guideline. CPGs should contain specific recommendations on appropriate patient care and consider different possible options. Key recommendations should be found easily, as well as a summary document and patient information. We would expect to see a discussion regarding the organizational changes and cost implications of applying the recommendations, and a presentation of review criteria for monitoring the use of the guidelines. There should be an explicit statement regarding the nature of involvement of the funding body in the CPG development process, to ensure that they have not unduly influenced the final recommendations. Also, members of the guideline development group should declare possible conflicts of interest.

Benefits of quality assessment of CPGs

A properly performed evaluation of the scientific evidence is critical in ensuring the scientific validity of a guideline. Quality assessment tools can be helpful for clinical practice in two ways: to help clinicians appraise, then adopt recommendations from different sources; as an information support to be used more confidently in the continued quality improvement of medical practice [8,102].

Conclusions

Concern about the quality of healthcare, uncertainty and variability in decision making, as well as rising costs, have stimulated a marked growth over the past 5 years in the development and use of CPGs [20,21]. Clinicians, policymakers and payers may see CPGs as a tool to facilitate more consistent and efficient decision-making, useful in closing the gap between what clinicians do and what the scientific evidence supports. Implementing the findings of evidence-based guidelines is a quality-improvement process that provides accountability through the monitoring of the reliability of practices to models that have been demonstrated by research to be effective. Using this framework, policy makers can approach payers with greater confidence. They can argue for resources to implement evidence-based guidelines with a greater assurance of accountability and value for money. The achievement of consistently positive outcomes is at the heart of an evidence-based practice. Providers, both clinicians and administrators, must understand new clinical practice options and their utility before they can be expected to adopt them.

An increasing concern is the number of disease-specific guidelines that offer inconsistent recommendations. Many reasons have been put forward to explain this variability, ranging from lack (or differing interpretation) of underlying research findings, different values given to anticipated outcome (for example clinical versus economic), questioning the achievement of consensus and possible bias introduced through conflicts of interest [15]. Several approaches could be used to improve the quality of CPGs. Guideline developers should become more familiar with CPG development standards and strive to incorporate them into their efforts [8,101].