Guidelines for the use of biomarkers: Principles, processes and practical considerations

Abstract

With the growing availability of new health care technologies and rapidly emerging biomarker discoveries, clinicians need advice on the clinical validity and utility of new tests and whether they improve clinical, patient-centred, organizational or economic outcomes. High quality clinical practice guidelines (CPGs), based on well-designed and conducted test evaluation studies, are tools for translating research into practice and in promoting a value- and evidence-based approach for clinical utilization and reimbursement of new biomarkers. Such study protocols should be appropriate for the questions addressed at each stage of biomarker development: 1/ Basic research into the association of disease with the new biomarker; 2/ Modelling the potential use of the new biomarker in clinical practice; Studies on the 3/ analytic validity; 4/ clinical validity (efficacy); 5/ clinical utility (effectiveness); and 6/ clinical impact (efficiency) of testing. Irrespective of the facts that CPGs potentially influence important clinical decisions and thus patient outcomes, current approaches to CPG development often do not follow the rigorous processes of scientific publications. Guidelines should be outcome oriented; reliable and free from any forms of bias; based on high quality research or on formal consensus when evidence is conflicting or lacking; multidisciplinary; flexible and applicable to various clinical circumstances and patient preferences; clear; cost-effective; appropriately disseminated and implemented; amenable to measurement of their impact in practice; and regularly reviewed and updated. Therefore until guideline-making and reporting standards are improved, all CPGs should be carefully scrutinized for methodological and content validity before being adopted, adapted and used in clinical practice.

Key Words::

• biomarker
• evidence-based laboratory medicine
• guideline
• translational medicine

Abbreviations:
ACCE	=	Analytic validity Clinical validity Clinical utility and Ethical legal and social implications
AGREE	=	Appraisal of Guidelines for Research and Evaluation
CPG	=	Clinical Practice Guideline
RCT	=	Randomized Controlled Trial

Introduction

Translational research, which aims to bridge the gap between the identification of new biomarkers and proving that these are clinically effective and improve patient outcomes, is still in its infancy and faces two major obstacles. The first is the translation of basic science discoveries into clinical studies; the second is to translate clinical proof-of-concept studies to evidence-based personalized treatment guidelines or health policies [1]. With the growing availability of new biomarkers, clinicians need advice on their validity and utility. New laboratory biomarkers have clinical value only if they provide additional benefit to patients at acceptable costs. Laboratory tests by themselves rarely influence patient-centred outcomes directly, and often they are disease-centred predictors or surrogates to patient-relevant endpoints. However, tests initiate a cascade of decisions which subsequently determine the course and costs of patient management and thus indirectly contribute to patient-centred and economic outcomes. Recognizing the importance of testing in medical decisions, on the background of limited health care resources, it is now widely promoted that clinical utilization and reimbursement of diagnostic tests should move from a cost-based, towards a value- and evidence-based approach. It is also commonly believed that high quality, evidence-based clinical practice guidelines (CPGs) might be most suited in transmitting this message to practicing clinicians and their patients.

How to evaluate biomarkers before recommending them for practice?

Recommendations of using a new test or incorporating it into the patient management pathway should be preceded by and based upon specific test evaluation studies and technology appraisals. Test evaluation should be carried out in a step-wise fashion, with carefully planned study designs appropriate for the questions addressed at each stage of research (Figure 1). The rigour and depth of evaluation should be proportionate to the questions and expected outcomes being addressed.

Figure 1. Phases of biomarker evaluation.

Phase I: Basic research into the association of disease with the new biomarker

In the initial phases of biomarker discovery the key question is: Do patients with the target disorder have different test results from normal individuals? Case-control studies are often sufficient for answering these early stage questions.

Phase II: Modelling the potential use of the new biomarker in clinical practice

In the next phase careful consideration should be given to the purpose, the clinical context and pathway of testing, the population and healthcare setting in which the test is intended to be used, and its potential downstream consequences in clinical practice. No new test should be subjected to tedious evaluation if the test is unlikely to result in improved clinical actions or additional measurable outcomes. Decision analytic modelling could be a cost-saving approach for assessing the potential clinical utility of the new biomarker in various practical scenarios.

Phase III: Analytic validity of new biomarker

Only if modelling confirms potential clinical utility it is worth moving to the next phases of establishing clinical and analytic performance goals. Analytic validity studies for e.g. technical sensitivity, specificity, imprecision and trueness should be performed, quality control procedures worked out and analytical characteristics improved, if clinical performance goals justify such needs.

Phase IV: Clinical research into the validity of tests (efficacy study)

Clinical validity of a biomarker defines its ability to detect or predict a disorder or a response to interventions. Common questions can be: In patients suspected of having the target condition, will the test distinguish those with and without the disorder? This question is usually investigated in diagnostic accuracy studies in a representative spectrum of patients, in order to obtain the clinical sensitivity and specificity of the test. It is important to emphasize that analytic and clinical validity studies are not sufficient to justify recommendations for the clinical use of new biomarkers. Therefore evaluation should move to the next stage.

Phase V: Clinical application/utility of tests (effectiveness study)

Clinical utility of a test relates to the balance between benefits and harms associated with the use of the biomarker: Do patients who undergo the test fare better, in terms of health outcomes, than those who do not? The most suitable study design for this type of question is a randomized controlled trial (RCT) or a systematic review or meta-analysis of multiple RCTs.

Phase VI: Impact of testing in practice (efficiency study)

In this phase the ethical, legal, financial or social implications of testing are investigated. This is best explored by health technology assessment and when guideline teams formulate recommendations.

Guideline developers should be particularly concerned and familiar with processes from Phase IV onwards when making judgments about recommending the use of new biomarkers. The subsequent chapters describe key principles and processes of guideline development and will point to special areas of concern related to the development and application of guideline recommendations.

Principles and problems of guideline development

Clinical practice guidelines (CPG) are systematically developed statements providing recommendations about the care of specific diseases [2]. Good CPGs should be [3,4]:

–outcome oriented

–internally valid – i.e. based on high quality research evidence or on formal consensus when evidence is conflicting or lacking

–reliable – i.e. developed in an explicit, transparent and reproducible manner free from commercial influence or bias

–multidisciplinary

–externally valid – i.e. clinically applicable

–flexible – i.e. adaptable to various clinical circumstances and patient preferences

–clear – i.e. specific and readily understood by users

–regularly reviewed and updated

–appropriately disseminated and implemented

–cost-effective and

–amenable to measurement of their impact in clinical practice.

Before describing the ideal processes and methods of guideline development, the below fundamental questions need to be answered by also reflecting on the current state-of-the-art:

–Do current guideline development methods work?

–Does guideline quality matter?

–Do we need guidelines at all?

–Do we need so many CPGs for the same condition?

–Do we need separate CPGs for covering different aspects of care (e.g. screening, diagnosis, treatment, monitoring) of a clinical condition?

Do current guideline development methods work?

Critical appraisal of the methodology of CPGs by the AGREE Instrument [5] has shown that CPGs, often issued by prestigious authorities, lack the above listed desirable attributes in many medical fields [5,6]. Our review of 712 CPGs [6] along with a more recent systematic review of 626 CPGs published between 2003 and 2008 [7, personal communication] showed (Table I) that in most guidelines the scope and purpose of recommendations (Domain 1 see Table I) are clearly defined and guidance is given in a clear format (Domain 4). There are significant shortcomings, however, in the multidisciplinary composition of guideline teams and involvement of patients in formulating recommendations (Domain 2). It has been shown that the composition of a guideline panel could grossly influence the focus of guidelines and could enhance the interests of certain specialties, or governmental agencies or industry [8] as opposed to the interests and preferences of patients in decisions about their care.

Table I. Overview of publications investigating the quality of guidelines with the AGREE Instrument [7]. Figures in brackets represent 95% confidence intervals.

				AGREE SCORES (%)
Reference	Publication date of guidelines included in the review	Origin of guidelines	Number of guide-lines	Domain 1 Scope and purpose	Domain 2 Stakeholder involvement	Domain 3 Rigour of development	Domain 4 Clarity and presentation	Domain 5 Applicability	Domain 6 Editorial independence
Horvath et al, 2007 [6]	2003-2007	Mainly from Europe and North America	712 ^*	65	34	34	63	26	30
Alonso-Coello et al, 2009 [7]**	2003-2008	42% Europe 41% North America 17% Other	626	64 (61.9-66.4)	35 (33.9-37.5)	43 (41.0-45.2)	60 (57.9-61.9)	22 (20.4-23.9)	30 (27.9-32.3)

*Figure includes unpublished assessment of 189 national guidelines and care pathway protocols retrieved from the “ grey literature ”.

**Personal communication by authors.

The scores in these critical assessments were also low for the rigour (or reporting) of an evidence-based CPG methodology (Domain 3). There is particular concern about the quality and reliability of diagnostic recommendations in CPGs [9], as most recommendations are based on poor quality research and therefore there is over-reliance on expert opinion [10]. Making recommendations requires subjective judgments which are often dominated by some individual experts and their experience, rather than objective information gathered and evaluated systematically from the medical literature. Guidelines often fail the criteria of editorial independence, i.e. reporting on funding and potential conflicts of interest (Domain 6). Furthermore, both reviews in Table II found that most recommendations lack external validity, i.e. applicability in practice (Domain 5), have a one-size-fits-all mentality and rarely build flexibility or contextualization into the recommendations or allow for individualization of care.

There are a number of other practical problems with CPGs [8]. More extensive CPGs tend to be more meticulous in their methodology and more likely to be evidence-based [9], however, they are not necessarily the most specific and practice-based as well [11]. So, does guideline quality matter in practice? Is there any relationship between guideline quality and content, or compliance with its use in clinical practice? Does guideline methodological quality impact on clinical outcomes? We provide further insight into these relationships below [6].

Does guideline methodological quality matter in practice?

Critical appraisal by the AGREE Instrument is based on the theoretical assumption that poor methods potentially reflect biased or invalid results. Thus AGREE provides only an assessment of the predicted validity of a guideline, and the likelihood that it will achieve its intended outcome, but it does not assess the impact of a guideline on any outcomes. Whilst a number of studies confirm this assumption in the literature [12,13], other studies found no straightforward correlation between CPG quality and validity of content when comparing specific recommendations to available systematic reviews on the actual guideline topic [14,15]. Few studies investigated whether there is a relationship between guideline quality and their impact on practice patterns, health outcomes and healthcare costs. From our point of view it is also noteworthy that CPGs of poor methodological quality on benign prostate hyperplasia and lower urinary tract symptoms recommended more diagnostic tests than those of better quality [16].

Few studies investigated the correlation between guideline attributes and the use of CPGs in practice. Non-specific recommendations resulted in a higher frequency of inappropriate testing behaviour in low back pain syndrome than specific and clearly presented ones [11]. Controversial versus noncontroversial recommendations were followed in 35% and 68%, respectively. Vague recommendations that demanded a change in existing practice were followed in 44% and those that did not in 67%. Evidence-based recommendations were used more than non-evidence-based ones in 71% versus 57%, respectively [3]. These studies highlight the importance of external validity, i.e. applicability of recommendations in adhering to guidelines. Many guidelines are too often “lost in translation” [17]. Therefore guideline development teams should pay more attention to the specific attributes of CPGs that determine their use in practice.

In order to get research into practice more effectively, we need to move from “science-driven” guideline programs towards scientifically based but “customer-driven” approaches [17]. Guideline developers should provide unambiguous and clear statements, decision support tools, patient education materials and practical measurement tools with their guidelines if they wish to achieve that their recommendations have any measurable impact in clinical practice.

Do we need guidelines at all?

It is commonly accepted that guidance is universally needed to aid physicians in harmonizing the approaches and standards of care and in synthesizing the body of often contradicting research findings. It is also a growing expectation that CPGs are based on the best available research evidence preferably coming from systematic reviews and meta-analyses addressing well focused questions related to critical aspects of patients care. So one may argue whether we need guidelines at all, or would it be better to have high quality trials or systematic reviews or evidence summaries in form of well structured, quality rated evidence-tables that would provide a universal answer to clinically important questions? It is a particularly important notion as it has been demonstrated that the quality of systematic reviews and meta-analyses prepared for formulating recommendations, is often poor, and guideline developers do not assess the quality of the underlying evidence in a systematic process [18]. It has also been shown that systematic reviews, conducted solely for the purposes of guidelines or economic analyses, especially in the field of diagnostics, are of poorer quality than single overviews performed by experts trained in review techniques and evidence-based medicine [19]. The current state-of-the-art therefore indicates that probably we should concentrate our efforts and resources on producing more high quality research evidence, and less low quality guidelines.

Do we need so many guidelines for the same condition?

There is a plethora of CPGs for the management of many conditions, freely available on the World Wide Web [20,21,22]. There are 2442 CPGs currently available in the National Guideline Clearinghouse [20] with an additional 366 CPGs being under development. On diabetes mellitus, there are 135 CPGs on [20] alone. Thus for the practicing physician it is often difficult to know which guideline to choose and use in everyday practice.

A number of CPGs sometimes provide contradicting recommendations for the same condition. This is partly understandable as guidelines are developed in many countries where the local organizational, societal and cultural circumstances and the availability of resources may justify variations in the interpretation and application of evidence. Other valid reasons include differences in clinical questions, patient subgroups, time-span of retrieved literature, and judgments about the local relevance of research, and costs [23]. However, these differing or contradicting interpretations are often due to the lack of a systematic evidence retrieval and critical appraisal approach, failure to consider outcomes that are important to patients and to the dominance of certain personalities and their biased opinions or beliefs when recommendations are formulated. Therefore a more transparent process for CPG development would be needed where reasons for deviations from the research evidence are clearly explained and justified.

Do we need separate guidelines for covering different aspects of care of a clinical condition?

The reason for the high number of CPGs is also that in many situations there are parallel developments of recommendations for various aspects of care by relevant professional bodies. This might be seen as a pragmatic approach, as this way, for example diagnostic guidelines, can better focus on specific diagnostic problems. CPGs are often huge documents of several hundred pages which hardly anyone reads, let alone uses in practice. Others argue that specific recommendations developed in isolation by subspecialty experts rarely reach their target audience as they are not part of a larger leading clinical guideline. Whilst both arguments are valid, our view is that subspecialty organizations often develop more focused and specific recommendations. To achieve their aims, however, these also need to be conceived in a multidisciplinary process or at least consulted with and endorsed by relevant clinical or patient organizations who are the primary target groups of the recommendations. Such subspecialty recommendations can either be adopted and cross-referenced by clinical organizations or incorporated into the relevant larger CPG without unnecessarily duplicating the efforts of guideline teams.

Processes of guideline development

How can we change the current situation and how CPGs should be developed, particularly when making recommendations about the use of new biomarkers? Guidelines can be adopted or adapted or developed de novo. Adoption of guidelines means that recommendations are used in the same format as issued by the authority responsible for releasing the CPG. Guideline adaptation refers to the modification of a CPG produced for use in one cultural and organizational context to be applied in a different setting [24]. Adaptation can be used as an alternative to de novo guideline development or for customizing an existing guideline to suit the local context. CPG adaptation is carried out in three key phases as detailed in Figure 2. Guideline adaptation is particularly useful in low resource countries as they provide a cost- conscious solution to standardizing practice.

Figure 2. The process of guideline adaptation [24].

The flowchart of developing new CPGs is shown in Figure 3. Once the remit and clinical questions of the CPG are defined the critical steps in the process are how systematically the underlying research evidence is collected, selected, appraised and synthesized to give unbiased information which the CPG team can interpret further. This is probably the most time-consuming element of CPG development which needs special skills and training. Often busy clinicians or laboratory professionals neither have the time, nor the necessary training to carry out a thorough investigation. It is also unrealistic that all recommendations in a CPG would be based on systematic reviews of the relevant literature. Therefore CPG teams should prioritize their key questions and cover those with more meticulous reviews that are likely to influence patient relevant outcomes.

Figure 3. The process of de novo guideline development.

In good guidelines the overall quality or strength of the evidence and the strength of recommendations are graded separately. The quality of evidence indicates the degree of confidence that the evidence is adequate to support recommendations. This can be judged by considering several factors: 1/ the level of evidence of individual studies, which refers to the detailed study methods and the quality of their execution; 2/ the precision (confidence interval) of effect estimates; 3/ the consistency of results across various studies; 4/ the directness of the evidence, i.e. the extent to which the study's patients, interventions, and outcomes are similar to those in practice (NB: diagnostic studies using surrogate outcomes usually provide indirect evidence). These pieces of information can be gathered from phase IV and V biomarker evaluation studies (Figure 1).

One of the difficulties of this process is that most grading systems were developed for therapeutic interventions or test-treatment combinations that cannot be uniformly applied to all types of diagnostic questions in CPGs. It is often not recognized that different types of clinical questions can be addressed by differing study designs, and not all questions are answerable by a randomized controlled trial that currently represents the highest level of evidence of most grading schemes. In laboratory medicine, most recommendations are related to the clinical performance of tests for diagnosing, monitoring or prognosing conditions, or to some practical issues regarding the pre-analytical (e.g. requesting, taking and transporting or storing of specimens to the laboratory) and post-analytical phases (e.g. interpreting or calculating test results) of the diagnostic process. Some guideline manuals acknowledge these difficulties and provide special appraisal checklists for rating the quality of diagnostic studies [25].

Beyond the body of evidence, guideline developers need to give due consideration to other practical aspects as well: the balance between benefits and harms; the transferability of the evidence to the given population, condition, or outcomes (i.e. Phase V); the preferences of the patient; impact on health care organization and costs (i.e. Phase VI) [26]. These value judgments help in grading the strength of recommendations, which indicates the extent to which one can be confident that adherence to the recommendation will do more good than harm [26]. The process of considered judgment introduces subjectivity into the CPG development process. To avoid the dominance of opinion leaders, this phase should be carried out by an unbiased expert team in a transparent and well-documented process.

What can a guideline development team do if, in spite of all efforts, the literature search has found no evidence that addresses a key review question or if the quality of the clinical evidence found is poor or conflicting, or not directly applicable to the population covered by the guideline? In these situations, which are not uncommon when diagnostic recommendations are made, the CPG team should explicitly state the root of the problem and should consider using a robust formal consensus method to identify current best practices. Other, lower priority questions could be also addressed in a less formal way and turned into “good practice point” recommendations mostly driven by expert consensus and professional experience and agreement, or widely accepted standards of best practice [25]. This category mostly applies to technical (e.g. pre-analytical, analytical, post-analytical), organizational, economic or quality management aspects of laboratory practice where the question does not directly address health-related outcomes of care. In these cases recommendations are often based on observational studies or empirical data, audit reports, case series or case studies, non-systematic reviews, guidance or technical documents, personal opinions, expert consensus or position statements, usual practice, quality requirements and standards set by professional or legislative authorities or accreditation bodies.

However, care should be exercised with over- interpreting the importance of clinical experience, as it was shown with cholesterol testing that the more clinical experts were involved in the CPG development process, the less recommendations reflected the best available research evidence [27]. It is common knowledge that when doctors’ beliefs are confronted with the underlying evidence, the latter is usually over-ruled by expert opinion and experience, and guideline development is not immune to this attitude either. Therefore, any deviation from the evidence, due to the differing views of expert consensus, should be documented and the process and rationale made transparent for the users of recommendations.

Guidelines, issued to give advice on patient management, are supposed to be the most highly ranked and influential publications. Irrespective of their high status, CPGs are not always subjected to independent external peer review. Whilst the rigorous processes of scientific publications can be bypassed, their consequences cannot [28], therefore we advise that all CPGs are carefully scrutinized both for methodological and content validity before they are adopted, adapted and used in clinical practice.

Conclusions

The ultimate aim of biomarker development is that new diagnostic technologies improve disease management and clinical effectiveness of care and patient-centred outcomes. To support faster diffusion and rational use of new biomarkers of proven effectiveness and efficiency, a multidisciplinary, more responsive and proportionate risk assessment during pre-market approval of new tests is needed. Clinical practice guidelines are developed to close the gap between research and practice, but the appearance of guidelines created a new gap between their development and use in practice.

Often current diagnostic guidelines fail their purpose due to deficiency in formulation, methodological rigour and transparency in the development process. It is vital that such shortcomings are overcome to satisfactorily translate biomarker research into practical, fit-for-purpose recommendations e.g. by formulating methodological and reporting standards and agreeing on a unified grading system.

These tools, together with higher quality Phase I-VI research studies might also help in improving the validity, reliability, flexibility, clarity, transparency and other important attributes of good diagnostic guidelines related to the use of newly emerging biomarkers.

Although common sense dictates that rigorously developed CPGs, based on high quality translational research evidence, are more likely to improve clinically important outcomes, it remains to be seen whether this is the case. Until such research data and guidelines become available, CPGs should be critically evaluated not only for methodology but also for validity of their content before changing clinical practice. Translational research needs to be a two-way and reciprocal process. Feed-back on the clinical utility and impact of new biomarkers needs to be communicated to scientists, industry and guideline teams to enable them to ask new research questions, design new biomarker evaluation studies and formulate better evidence-based recommendations that are more responsive to real clinical and patients’ needs.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.