Does assessing the value for money of therapeutic medical devices require a flexible approach?

Abstract

Regulation criteria for licensing pharmaceuticals and medical devices (MDs) are asymmetric. This has affected the type, quantity and quality of the evidence produced in support of MDs. This paper has three objectives: to examine the reasons behind the current licensing criteria for MDs; to identify key methodological challenges associated with pre- and post-market evaluation of MDs and to assess the extent to which existing methods for the economic evaluation of pharmaceuticals can be applied to the evaluation of MDs. The belief that MDs cannot be properly evaluated stems from a combination of historical events and complexities in implementing rigorous RCTs in this field. Existing challenges to conduct sound economic evaluation of MDs have begun to be addressed in medical research using mixed research methods. While more challenging to implement, robust evaluations of therapeutic MDs can and need to be carried out to safeguard individual’s wellbeing.

Keywords:

• Bayesian methods
• economic evaluation
• licensing regulation
• medical devices

Healthcare systems around the world are under increasing pressure to use their available resources as efficiently as possible. This requires ensuring a provision of health technologies (e.g., pharmaceuticals, medical devices [MDs], surgical procedures, etc.) that maximize population health given existing (budget, legal, ethical and structural) constraints. To guide efficient resource allocation and research prioritization decisions, formal post-market health technology assessment (HTA) processes have been set up in many countries.

HTA processes involve identification and critical appraisal of the existing evidence base, to inform the clinical and economic evaluation (EE) of competing health technologies [1]. Implementation of these processes has not been without challenge; a lack of clarity regarding the definition of HTA and other related terms has recently been highlighted. This paper adopts the definition proposed by Luce et al.: ‘HTA is a method of evidence synthesis that considers evidence regarding clinical effectiveness, safety, cost–effectiveness...[A] major use of HTAs is in informing reimbursement and coverage decisions, in which case HTAs should include benefit-harm assessment and economic evaluation’ [2].

By providing an ‘organizing framework’ of the typology, nomenclature and interrelationships of the evidence process in healthcare, Luce et al. propose the following: there are different stages of evaluation for healthcare technologies in the pre-market (i.e., licensing) and post-market (e.g., comparative effectiveness research, HTA); licensing, comparative effectiveness research and HTA contribute to the evidence generation process and each is interlinked with preceding stages; HTA is positioned toward the end of the spectrum of evaluation and has an iterative nature; EE studies are a tool for HTA; and HTA contributes to healthcare decision-making and research prioritization. The interrelationships between the different stages of the evaluation process in healthcare point out the extent to which the robustness of HTA processes crucially rely on the quantity and quality of the available evidence base as well as the appropriateness of the methods used to analyze it.

The existing pre-market regulatory regime that controls the introduction (licensing) of healthcare technologies does not provide the same incentive mechanisms to promote the generation of high-quality evidence across different types of health technologies (i.e., pharmaceuticals, MDs, procedures, etc.). In the case of pharmaceuticals, pre-market (licensing) regulation - across all jurisdictions, notably UK/Europe, the USA, Canada, Asia, Australia and others - requires demonstration of their quality of manufacturing, safety and efficacy (the famous three hurdles). For MDs, however, licensing criteria require evidence of efficacy only for those MDs associated with the highest level of risk [3].

Evaluating all types of health technologies according to a common set of criteria is desirable and may have a number of potential advantages. These include: simplification of the regulatory environment, convergence of evaluation criteria and promoting consensus on definitions of quality, safety, efficacy, effectiveness and cost–effectiveness. Yet many authors have expressed concerns about what they perceive to be barriers to a sound clinical evaluation of therapeutic MDs [4]. Ethical concerns regarding the appropriateness of sham procedures and random allocation have supported a perspective that MDs cannot be evaluated with the same level of rigor as pharmaceuticals [5]. In view of the impact of the current licensing criteria for MDs on the ability to evaluate their value for money in EE studies, it is important to understand the rationale of the current licensing criteria and how much of the status quo can or cannot be realistically changed.

In 2011, the medical technology industry generated more than US$200 billion in annual revenue worldwide (excluding sales of diagnostics) with the US market accounting for nearly half of the global market [6]. In the same year, EUCOMED reported €95 billion sales for the sector in the EU, which accounted for approximately 30% of the market [7]. It is apparent that there is potential value in implementing rigorous HTA processes to inform reimbursement and/or coverage decisions as well as guide post-market research prioritization of MDs. In spite of this, explicit HTA guidance for the EE of MDs has been lacking with researchers and policy makers ‘being left to their own devices’ when applying to MDs the requirements of a post-market regulatory framework designed primarily for pharmaceuticals [1,8].

A key issue for consideration is whether EE methods currently used for pharmaceuticals can also be used to evaluate MDs. If the primary objective of EE studies is to inform resource allocation decisions aimed at maximizing population health given limited healthcare resources, then the dearth of evidence regarding MDs efficacy/effectiveness (as compared with pharmaceuticals) leads to technology assessment recommendations associated with high levels of uncertainty. This decision uncertainty manifests itself in terms of population health benefits potentially forgone by the healthcare system.

For ease of presentation, this paper limits its focus to MDs whose primary aim is to treat existing medical conditions, that is, ‘therapeutic MDs’. Section 1 ‘pre-market evaluation of MDs’ describes the results of a targeted literature review to explore the rationale for the current MDs’ pre-market regulatory system. Section 2 ‘Post-market evaluation of therapeutic MDs (HTA)’ examines methodological challenges often encountered when implementing EE of MDs and proposes alternative methods to begin addressing them. Section 3 ‘Does the evaluation of medical devices require an alternative approach?’ discusses whether identified challenges support the view that current methods for EE are not suitable for MDs. Similarly, it explores whether a flexible approach to evaluate the value for money of MDs needs to be taken. The discussion section summarizes the findings, reflects on their implications and provides final conclusions.

Pre-market evaluation of MDs

Medical device is an umbrella term that encompasses a large number of healthcare technologies ranging from surgical gloves to heart valves – see definition at end of paper – (GHTF Study Group 1, 2005). Despite risk being a key common criterion for the classification of MDs across countries and of efforts from the International Medical Device Regulators Forum (formerly the Global Harmonization Task Force) [9], a universally agreed classification of MDs is not yet available. While the EU groups MDs into four risk categories (I, IIa, IIb and III), with I being the lowest risk, the USA classify them in categories low (I), moderate (II) or high risk (III) [10–12]. Several factors determine the level of risk, including the intended duration of use, the degree of invasiveness, whether the device is implantable and whether it contains an active substance (e.g., drug-eluting stents). Current licensing regulation of MDs only require evidence of efficacy for technologies carrying the highest levels of risk to the patient (e.g., class IIb devices and above in the EU and class III in the USA).

Differences in licensing criteria between MDs & pharmaceuticals

Our review of relevant policy, regulation and methods literature as well as websites of regulatory institutions and HTA agencies did not yield a definite justification for the different licensing criteria faced by MDs and pharmaceuticals. We found that regulatory regimens have evolved over time in response to public health concerns, international legislation and political agendas [13]. Other factors may have also contributed to create an asymmetric pre-market regulatory system between the pharmaceutical and the MD industries. First, historical events and the response they triggered from regulatory bodies may have contributed. The experience with thalidomide, a sedative used to reduce nausea in pregnancy, was a turning point in the evaluation of pharmaceuticals. In view of the side effects of this agent on newborn babies, the US FDA changed its criteria for pre-market regulation [3] and, ever since, licensing of pharmaceuticals has required evidence of their ‘efficacy’ in addition to that on safety and quality of manufacturing [14].

Second, significant differences in the way MDs and pharmaceuticals come into existence and their mode of interaction with the human body may also have a role [12]. The intended use of MDs is not typically achieved by a chemical and system-wide biological reaction in the human body, as is the case with pharmaceuticals [15]. This intrinsic characteristic of MDs as well as their heterogeneous nature and the type of device scientist have in mind when thinking about MDs may explain a widely held view that MDs pose lower health risks than pharmaceuticals. Hence, less stringent regulatory criteria for the approval of MDs into healthcare markets may be sufficient.

Third, is the implicit assumption of equivalent (if not superior) clinical efficacy associated with incremental device innovations, and the concept of ‘substantial equivalence’. If ‘a new MD has the same (similar) mechanism of action and intended use and thus is at least as safe and effective as an earlier model or a model produced by another manufacturer’, it is considered to be substantially equivalent to the earlier model [16]. Unlike the European Council Directives concerning MDs [11], the FDA regulation [10] explicitly addresses the need to demonstrate claims of ‘similarity in mechanism of action and intended use’ of MDs. The submission route for MDs, Pre-Market Notification (PMN) or 510(k), allows manufacturers to notify the FDA their intention to market a device that is substantially equivalent to another product already in the market [16].

Fourth, a long-held view that implementing sound RCTs on MDs is impossible. There is increased complexity associated with the design and implementation of RCTs of MDs [17]. A recent review paper [18] reported on 96,346 studies from the US clinical trials registry: clinicalTrials.gov, in three clinical areas. For cardiovascular, mental health and oncology, when compared with trials on drugs and biologics, procedure and device trials were less likely to use blinding and randomization. Similarly, Dhruva et al. reviewed 123 safety and effectiveness summaries from 78 pre-market approvals – the most stringent FDA process – submitted between January 2000 and December 2007 for high-risk cardiovascular devices. Fewer than one-third of these studies was RCTs and only 14% used blinding. Dhruva et al. concluded that FDA’s pre-market approval of cardiovascular devices was often based on studies that lacked adequate strength and could be prone to bias [19].

In contrast, Black sets out a thorough discussion of how, in the context of MD evaluation, RCTs have been deemed unnecessary, inappropriate, impossible and inadequate. For each one of these apparent limitations, he also provides a potential alternative solution [20]. Table 1 gives an overview of the challenges associated with the implementation of RCTs of MDs. A detailed discussion of the above challenges is beyond the scope of this paper. Yet, as sound evidence on clinical efficacy/effectiveness is one of the pillars for HTA and EE, a brief discussion of relevant issues for designing pre- and post-market RCTs of MDs is provided in the section ‘Solutions to improve evidence generation’.

Table 1. RCTs in medical device evaluation.

Limitations	Potential solutions
Unnecessary  Dramatic treatment effect – small ignorable confounding factors, e.g., thyroxin, immobilization
Inappropriate  Small sample size   Rare condition   Infrequent adverse events  Outcomes of interest far in the future  Health professionals and patients preferences	Large trials Continue follow-up for over many years Introduce parallel preference arms
Impossible  Reluctance or refusal to participate  Ethical objections  Political and legal obstacles  Contamination/quality control monitoring  Scale of task	More acceptable methods of randomization Simplify methods to obtain informed consent Persuasion Video evidence/ pre-trial norms of success rates Expand funding budgets
Inadequate  Low external validity (atypical healthcare professionals, patients or treatment)	Undertaking pragmatic trials instead of explanatory trials

Unnecessary: large clinical effect and small likelihood of confounding factors being important.

Inappropriate: not large enough, rare events, outcomes of interest are long term.

Impossible: ethical objections, key stakeholders refusal to participate.

Inadequate: low external validity, that is, atypical treatment, patients and healthcare providers.

This table contents and foot notes are a summary from [20].

The added level of complexity associated with the design of RCTs of MDs is often communicated in support of the view that RCTs should not be considered as the gold standard methodology to investigate the clinical efficacy/effectiveness of MDs. This is in contrast with the main source of evidence for pharmaceuticals’ efficacy/effectiveness (i.e., Phase III and post-market RCTs). Califf et al. and Dhruva et al. found most MD clinical evidence comes from observational studies and registries [18,19]. Methodological challenges associated with conducting RCTs on MDs may in part explain this. The incentive structure provided by MDs’ pre-market regulatory requirements, however, may also have a role. MD regulatory agencies accept both experimental and observational clinical studies as valid scientific sources of efficacy. If MDs manufacturers do not have an incentive to produce data in a way that is meaningful for HTA agencies, they might not voluntarily invest in evidence generation. Indeed, this represents an issue of proportionality: what is the benefit of an appraisal as compared with its cost?

How much of the status quo can/cannot be realistically changed

A recent recall of a popular silicone breast implant that was approved in the EU, raised European concerns about the clinical evaluation of high-risk implantable MDs [21,22]. Similarly, detrimental long-term effects associated with metal-on-metal hip prosthesis triggered reactions among MDs regulators worldwide. These events [23–25] stressed a need to revise existing criteria for their pre-market approval and post-market surveillance [26], due to the ‘...unsatisfactory and unscientific way that medical devices are approved for use particularly in Europe; the failures in regulatory oversight during clinical use; and the lack of transparency in publishing research findings, device related complications, and competing interests’ [27]. The EU has now proposed a first set of modifications to their current regulation [28,29]. These, however, do not include a universal (i.e., for all MD) requirement to assess the efficacy of MDs using RCTs as gold standard in pre-market evaluations.

Reed et al. in an economic analysis [30] of the MD sector found that applying the same evidence standards expected of pharmaceuticals to MDs would offer improved financial incentives (i.e., earlier revenue streams) for MD companies. Reed et al. argued that this would require implementing a regulatory change consistent with the Coverage with Evidence Development [31] – the ‘Coverage with Study Participation (CSP) policy proposed by the Centers for Medicare and Medicaid Services, whereby Medicare will pay for beneficiaries to receive new devices that are not currently determined to be ‘reasonable and necessary’ if the patients participate in clinical studies or registries [30].

While appealing, substantial modification of the existing pre-market evaluative process might not be implemented sufficiently rapidly or be entirely desirable. Reed et al. noted requests for more rigorous pre-market regulation of MDs may inevitably result in fewer small and medium MD companies with a potential reduction in innovation [30]. Consequently, the following discussion assumes no alignment of the licensing criteria for MDs and pharmaceuticals.

Post-market evaluation of therapeutic MDs (HTA)

Healthcare EE analyses are a valuable tool to assist decision-making in HTA [1]. While current guidelines for conduct of EE have been primarily developed with pharmaceuticals in mind, a clear distinction between different types of healthcare technologies has not been made [8]. As a result, the same methods for EE and criteria of assessment were, until recently, used in many countries to ascertain value for money of pharmaceuticals and MDs. The Medical Services Advisory Committee and the Pharmaceutical Benefits Advisory Committee in Australia, for instance, require cost–effectiveness evidence before providing public funding and support access to particular MDs [32].

In 2009, NICE created the Medical Technologies Evaluation Programme (MTEP) whose methods guidance recommends use of cost–consequences analysis – please see definition at end of paper – to evaluate the value for money of a very specific subset of MDs (i.e., those likely to be cost saving, and evaluated both as a single technology and on a short time scale). This is in contrast with HTA Program of NICE, which requires full cost–effectiveness evidence to inform value for money decisions of innovative expensive health technologies, primarily pharmaceuticals but also MDs [33]. Once again, the asymmetry between different economic appraisal methods within NICE for different types of MDs may be perceived as supporting the view that post-market evaluation of MDs cannot or should not be as comprehensive as the one used for pharmaceuticals [34,35]. This section explores a number of key challenges analysts come across when ascertaining the value for money of MDs. For each identified challenge, the author proposes alternative quantitative, qualitative or mixed research methods to address it.

Challenges associated with EEs of MDs

A number of methodological challenges have been associated with EE of MDs [36,37]. While many of these are common to the evaluation of all types of health technologies, some are typical, if not exclusive, to the devices sector. For ease of presentation, the discussion is structured around three main sections representing the cornerstones of any HTA: clinical efficacy and effectiveness, individual preference-based health-related quality of life (HRQoL) and resource use.

Clinical efficacy & effectiveness

Class effect

Clinical efficacy relates to the extent to which an intervention can work under ideal conditions. By contrast, clinical effectiveness relates to the extent to which an intervention does work in normal clinical practice [38]. While licensing focuses on the former [10,11], it is the latter that HTA processes are concerned with. For the reasons discussed in section ‘Differences in licensing criteria between MDs and pharmaceuticals’, the evidence base on clinical effectiveness of most MDs has traditionally been weaker than that available for pharmaceuticals [39]. The lack of comparative effectiveness data for post-launch evaluations has been primarily circumvented by assuming a class effect between MDs that have the same primary indication (even when these are produced by different manufacturers). While advantages and limits of using ‘class effect’ in the context of pharmaceuticals’ evaluation are acknowledged in the literature [40], the validity of its application to the MDs context seems to go unchallenged from the clinical, scientific or regulatory perspective.

Class effect is related to a key concept in pre-market evaluation discussed in section ‘Differences in licensing criteria between MDs and pharmaceuticals’: ‘substantial equivalence’. This concept was introduced by the FDA in the context of its PMN/510(k) process that was created as a way to simplify and facilitate market approval of low- to moderate-risk MDs in the US healthcare market. Currently, PMN is one of the main licensing channels for MD, estimated to enable the introduction of more than 3000 devices per annum [16]. Nonetheless, FDA’s 510(k) process could be interpreted as promoting ‘class effect’ between similar MDs that are licensed through this route.

According to Stiegman and Eggleton [41], for many devices, there are very straightforward pathways to earn a substantial equivalence determination, that is, most of the time, sufficient data are available in the literature or the company’s own documentation that can be used for comparison of results. This suggests that in the pre-market, ‘substantially equivalent’ devices are treated like generic drugs rather than ‘me too’ products (i.e., for the former, once chemical similarity is established further proof of efficacy is not needed, for the latter, efficacy trials are still needed, often using intermediate end points). As a result, comparative efficacy evidence between similar devices is often not available to inform post-market HTA.

The ability to group healthcare technologies according to a common criterion may bring benefits to manufacturers: more rapid introduction of health technologies into markets; reduced R&D cost; marketing tool to promote the use of ‘me too’ products; and simplified medical education and practice [42]. There is, however, danger in interpreting the term class effect to mean all members of a given class can be used interchangeably (i.e., statistically exchangeable or therapeutically similar).

The fact that lower risk MDs can be licensed on the basis of safety and quality of manufacturing evidence alone, could be misinterpreted as supporting the view that perfect exchangeability exists between MDs with a common therapeutic indication within class. In reality, there can be vast differences in sophistication and price across products identified as comparators in an EE. One example is found in the area of wound care, with competing therapies such as negative pressure wound care therapy, spun hydrocolloids, alginate and foam dressings for the treatment of severe grade 3 and/or grade 4 pressure ulcers [43]. In this context, it is impossible to make rational treatment and funding decisions without considering differential effectiveness and costs for the various competing MDs. Furthermore, estimates of clinical effectiveness may vary when using alternative outcome measures. For example, a class effect estimated in clinical terms may not translate into a class effect in terms of generic preference-based measures of HRQoL, a key measure of health benefit in EE.

Device innovation

Related to the concept of class effect is the notion of rapid change/innovation, a special feature of the MDs market. In this sense, it is important to distinguish between processes of innovation resulting in development of brand new products (i.e., something substantially different from existing ones) and those processes concerned with incremental developments of existing products. This distinction is relevant since the methods required to consider the effect of innovation in estimation of treatment effect may well differ. For example, the collection of primary data on clinical effects and resource use will be paramount to assessing economic value of a new technology. In contrast, estimating the value (to society) of incrementally developing an existing product, in the first instance, may be based on the evidence base available on clinical and cost–effectiveness of preceding products (notwithstanding what was said with regard to the assumption of a class effect in MDs).

RCT & observational evidence & combination thereof

European Directives on MDs as well as FDA’s regulation recognize data from randomized and non-randomized experimental and observational clinical studies as valid scientific sources of efficacy [10,11,44]. This pre-market evidence can be used to inform post-market assessments on value for money of MDs. NICE in its reference case, for instance, recognizes the value of considering both experimental and observational evidence in the Institute’s HTA process [33]. The need to consider and quantitatively synthesize all relevant (randomized and non-randomized) evidence raises a number of methodological issues both for decision-makers and analysts. Above all, the question is how to make use of a fragmented, heterogeneous and potentially biased evidence base when assessing the clinical and cost–effectiveness of MDs, in the face of a serious dearth of available data.

Learning curve effect

Another methodological issue related to evaluation of the cost–effectiveness of MDs is the potential effect on observed outcomes induced by the level of experience (i.e., learning curve effect) of the user (e.g., surgeon, surgical team, hospital) [45]. Methods to deal with this are readily available if one is interested in exploring this issue on clinical measures of health benefit [46,47]. Some of these include graphical CUSUM techniques, generalized linear models and multilevel models, among others. The availability of primary evidence to characterize the learning process is a key challenge to investigate the impact of learning on measures of efficacy and effectiveness. The intrinsic association between learning and performance may be related to the lack of routinely collected data on the process of learning of healthcare providers at individual, team or institutional level. Ramsay et al. recommend ‘systematic collection of data of factors known to influence the learning curve’ [46]. These include: number and order of all procedures performed by unit of interest (e.g., surgeon, surgical team, hospital) and total number of procedures performed up to the point at which systematic data collection began.

It is worth noticing that to estimate the value for money of healthcare technologies, the issue is somehow different. In this context, the main objective is to ascertain the impact of learning on clinical outcomes, and how this translates into an effect in terms of HRQoL outcomes. The sensitivity of existing generic preference-based HRQoL instruments to capture learning effects has not been investigated.

Preference-based measurement of HRQoL

Instruments to measure HRQoL have been distinguished between condition-specific and generic and between preference-based or non-preference based [48].

Unlike pharmaceuticals, MDs are often associated with specific characteristics such as portability, ease of use, appearance and size that have potential to influence an individual’s preferences and domains of their HRQoL. In an attempt to capture these likely effects, MD manufacturers have frequently estimated HRQoL using condition-specific instruments. Cost–effectiveness assessment aimed to inform funding decisions in HTA, however, require the use of generic preference-based measures of patients’ HRQoL to capture morbidity and mortality impact of a given technology; and equally importantly allow comparison of incremental benefits (estimated using societal preference values) across several different clinical areas [49].

Drug manufacturers are increasingly aware of the value of collecting HRQoL data using generic preference-based instruments such as the EQ-5D [50]. Implementing EEs of MDs, however, frequently requires describing (mapping) the relationship between different outcome measures (e.g., clinical effectiveness, condition-specific HRQoL and generic- or condition-specific preference-based ones).

Resource use & cost

Indivisibility by indication

As mentioned before, the range of technologies included under the ‘medical device’ umbrella term is quite extensive. It ranges from simple bandages to extremely sophisticated apparatus such as MRI equipment. The ability to associate a cost per individual or single use of MDs is central for the conduct of EE studies. In the case of reusable MDs with multiple indications, additional complexity is added to the costing process. Often it is not possible to accurately estimate the cost associated with the use of a device for each indication. In instances where the indications and indivisibility are common to all the technologies under evaluation, this issue may be simply overcome by using the same denominator to estimate unit cost for each technology. In cases where there are differences in the divisibility by indication between technologies, a potential for bias arises in the estimation of mean costs.

Price variation

Large variations in MDs manufacturers’ list unit prices are often observed between same-class products. To an extent, this is a direct consequence of the current simplistic characterization of MDs according to risk. These variations add complexity to EE studies of MDs when a number of similar products manufactured by the same or different industries are compared. Using a weighted average of the list prices of all products included in the evaluation, in combination with the assumption of class effect, is a popular strategy in evaluation of pharmaceuticals. When large differences in unit costs between products are observed, however, using a weighted average cost may significantly dilute potentially relevant differences in value for money between products. While re-running the economic analysis by individual products may seem computationally demanding, this may be the best alternative to ascertain the impact of large variations in unit prices between allegedly equivalent products. Alternatively, one-way sensitivity (threshold) analyses can be conducted to ascertain the value for money of products at both ends of the spectrum.

Implications & potential solutions

The characteristics and methodological challenges associated with MDs evidence base as discussed, imply HTA processes and resulting policy recommendations on MDs are often associated with several sources of uncertainty [51,52]. These uncertainties will include; parameter uncertainty: we do not know the true value of relevant parameters (e.g., mean time to hip fracture), and is determined by the quantity of evidence available; risk of bias in the estimated parameter: caused by uncontrolled systematic distortions and more likely to be present in non-randomized studies; methodological uncertainty: discrepancies in the choice of analytic methods used (e.g., perspective of an EE will influence the categories of costs relevant for the analysis) and structural uncertainty: to develop a model, analysts have to make a number of assumptions, simplifications and scientific judgments regarding different aspects in the model (e.g., mechanism of action between cardiac arrhythmia and myocardial infarction). Structural assumptions and judgments may also relate among others to: the likely duration of the treatment effect; whether the treatment effect is the same across devices considered to be substantially equivalent (class effect); the device medium to long-term failure rate; the type of failure and whether this impacts on both health outcomes and costs; short-term and long-term device-related adverse events not due to failure.

High levels of uncertainty call for a cautious approach to using health technologies in routine care, as pointed out by Claxton et al. ‘…without sufficient and good quality evidence subsequent decisions about the use of technologies will be uncertain, i.e. there will be a chance that the resources committed by the approval of a new technology may be wasted if the expected net health benefits are not realised. Equally rejecting a new technology will risk failing to provide access to a valuable intervention if the net health benefits prove to be greater than expected…’ [53]. In these circumstances, it is desirable to reduce this uncertainty through the generation of additional evidence. Typical examples include NICE recommendation for use ‘only in research’ in its guidance on use of stent-graft placement in abdominal aortic aneurysm; the evidence base for the effectiveness of the device comprised 77 studies: only 4 were RCTs, 17 non-RCTs, 22 comparative observational studies, 28 case series and 6 were registry publications. The guidance on use of percutaneous mitral valve annuloplasty is illustrative as NICE stated ‘...evidence on the safety and efficacy of percutaneous mitral valve annuloplasty is inadequate in quality and quantity, the procedure should only be used in the context of research, which should clearly describe patient selection, concomitant medical therapies and safety outcomes’.

Solutions to improve evidence generation

The design of surgical and non-surgical RCTs of MDs has been associated with a number of challenges, among others, RCT’s rigidity: the low external validity usually associated with RCT evidence; blinding: difficulties ensuring any concealment in RCTs of MDs and in those cases when this is achieved through a sham, an effect – over and above placebo – may not be discarded; rapid incremental development: MDs are constantly being modified before and after launch; variations in technical proficiency: learning curve treatment effect may be significantly influenced by the experience of healthcare providers; practitioner and patients preferences: evidence suggests these can compromise unbiased estimation of treatment effect; outcome measurement time span: consideration of intermediate outcomes as primary ones in RCTS of MDs because recording final clinical outcomes would require unaffordable long follow-ups; suitable comparator identification: there are ethical concerns associated with the delivery of sham procedures.

Alternative strategies to deal with the above challenges, in the design and/or analysis stages of prospective clinical studies, have been discussed in the literature. Pragmatic RCTs offer a solution to the rigidity of explanatory RCT designs and provide a valuable tool for decision-making [54]. Non-standard creative strategies – use of sham, preventing disclosure of study hypothesis, among others – can be used to minimize the effect of blinding as a potential source of bias [55,56]. Tracker trials are the most flexible design for an RCT and allow use of incremental developments of health technologies as they become available [57]. Attempts to neutralize (pre-specifying level of experience or ensuring all interventions are provided by the same clinician) and explain (e.g., using hierarchical models) learning curve effects are emerging in the literature [58]. The complementary use of RCTs and prospective observational studies, in comprehensive cohort studies, has been proposed as a potential solution to address issues related to individual’s treatment preferences and collection of long-term final clinical outcomes [59]. Discrete choice experiments can be used to identify and value characteristics/attributes that influence patients and practitioners’ treatment preferences (e.g., portability and usability of product, delivery setting, etc.) [60]. Active comparators such as ‘standard care’ are a suitable alternative to overcome ethical concerns associated with the use of sham procedures [61].

Potential solutions given current characteristics of MDs evidence base

While promoting the generation of primary research may be described as the ‘gold standard’ approach to resolve uncertainty, it would be inefficient to discard the existing evidence base on MDs’ clinical and cost–effectiveness through poor quality and its usual heterogeneous and fragmented nature. If nothing else, such an evidence base could be used to guide primary research prioritization and generation. This process will require a flexible analytical framework enabling coherent synthesis of all the existing evidence. In this context, the Bayesian approach to statistics emerges as a promising framework for evaluation.

Significant progress with computers’ power and sampling methods to resolve previously intractable statistical issues have motivated development of a wide range of statistical methods from a Bayesian perspective. Strong and continuous development of these methods since 1990 has brought about many potential solutions to the challenges posed by the evidence base in the MD field [62]. Diffusion of these methods among health economists and modelers has been somewhat slow, partly due to a combination of technical challenges, time pressure and that the validity of more sophisticated statistical methods rests on the appropriateness of their assumptions (e.g., about the data generating process).

Bayesian methods are particularly well suited to address issues associated with a complex and fragmented evidence base as that usually available for MDs. These methods naturally enable the synthesis of multiple sources of evidence with heterogeneous designs (e.g., evidence from RCT and observational data). Early examples focused on down-weighting observational evidence based on quality score of the studies forming the evidence base [63]. Subsequent approaches combined randomized and non-randomized evidence using weighting as a sensitivity analysis tool (with weights attached to observational evidence ranging from zero [no value at all] to one [same validity as randomized evidence]) to ascertain the impact of including observational evidence on final results [64]. Others have attempted to incorporate multiple sources of bias in estimation of specific parameters in the model, using multiple-bias models and weighting to account for study quality [65,66]. More recently, Bayesian elicitation mixed methods, to simultaneously consider internal and external sources of bias on the estimation of parameter(s), have been proposed and applied to the evaluation of a UK NICE appraisal on antenatal care [67].

Hierarchical models provide a powerful platform to estimate the clinical effect associated with incremental innovation [68], for example, Bayesian hierarchical models for clustered data to deal with learning curve effect [46,58] or to conduct network meta-analysis [69]. Hierarchical models can take into account the correlation between the clinical effect associated with a newly developed product and those associated with former versions of it. In other words, they do recognize the hierarchical structure embedded in device development processes where sometimes changes to products and use of protocols often occur during the conduct of clinical studies to demonstrate safety and manufacturing performance. In ascertaining whether an assumption of class effect is consistent with available evidence, other two methodological areas stand out as particularly relevant: Bayesian statistical methods to account for internal (e.g., selection, attrition) and external (e.g., lack of external validity) biases [69–71] and Bayesian mixed methods for the elicitation and synthesis of expert judgment [72–75]. A recent review of the use of Bayesian statistical methods in assessment of implantable MDs found that while these methods are still significantly underused, their application to date has primarily focused on investigation of treatment equivalence and surrogate outcomes predictors [76].

Does the evaluation of MDs require an alternative approach?

Sections ‘Pre-market evaluation of MDs’ and ‘Post-market evaluation of therapeutic MDs (HTA)’ argued that: some characteristics of the evidence base on MDs are a consequence of the incentive mechanisms derived from current pre-market regulation, that is, licensing; it is unlikely that the licensing criteria for pharmaceuticals and MDs will be aligned any time soon; in the current regulatory framework, the burden of proof is transferred to post-market regulatory agencies called to assess the relative effectiveness and cost–effectiveness of competing MDs with the same indication; the fragmented and poor quality evidence base usually available on MDs artificially introduces a higher level of complexity to estimate their effectiveness and value for money; to conduct sound EE of MDs seven typical characteristics of MDs that translate into key methodological issues need to be addressed; flexible and creative RCT design methods, mixed methods for Bayesian expert elicitation, Bayesian statistical methods and discrete choice experiments represent a powerful tool kit to generate robust clinical evidence and to characterize and address methodological issues associated with the evidence base for MDs. In spite of these, ability to implement well-established full methods of EE in healthcare, that is, cost–effectiveness analysis, cost–utility analysis and cost–benefit analysis to ascertain the value for money of MDs, is not compromised by any of these issues.

While NICE’s MTEP decision to recommend cost–consequences analysis and clinical outcomes, in its methods guidance [35], as appropriate measures of health benefit, it could be interpreted as promoting the view that main stream full EE methods are not suitable for MDs. This, however, would ignore the different pathways of evaluation for MDs currently available at NICE. MTEP was designed to help the NHS in England and Wales adopt MDs and diagnostics – likely to be effective and cost-saving or cost neutral – more rapidly and consistently [77]. The Medical Technologies Assessment Committee reroutes all other MDs to be evaluated by the most appropriate NICE evaluation program, for example, the Technology Appraisal or the Diagnostics Assessment Programmes both recommending the use of full EE methods to ascertain value for money of MDs [78,79].

Aside from the choice of EE method used for estimating value for money of MDs, it is paramount that mean costs and health benefits are rigorously estimated. This would entail acknowledging and addressing key methodological issues associated with the available evidence base using appropriate methodology. The implementation of the methods described in section ‘Potential solutions given current characteristics of MDs evidence base’ does require additional time and investment, particularly regarding the generation of human capacity. However, initial attempts to characterize the uncertainty associated with MDs’ evidence base through the implementation of Bayesian methods are beginning to emerge in the literature [80].

Similarly, Black’s recommendation to consider RCTs and observational studies as complementary rather than alternative provides a sound and comprehensive framework for the clinical evaluation of MDs. Promoting Black’s view would be a first step toward overcoming long established perceptions that the efficacy/effectiveness of MD cannot be thoroughly evaluated [20]. To be effective, this recommendation would need to be added to the methods guidance of pre- and post-market regulatory institutions to incentivize the generation of a sound clinical evidence base for MDs.

The introduction of MDs into markets without sufficient evaluation of clinical effectiveness has been associated with high long-term adverse event rates. In addition, they have been delivered as part of irreversible procedures [81]. It is unclear whether pre-launch studies would avoid safety issues in areas such as joint implantation. Nonetheless, provided that patients are informed and willing to face uncertain long-term effects potentially associated with innovative implantable MDs, additional efforts could focus on improving post-implant surveillance. This could be done by incentivizing timely identification of medium- and long-term adverse effects without delaying market access.

In fact, the European Commission’s review of its Directives regulating licensing of MDs is considering introducing modifications to its surveillance mechanisms [29]. Similarly, detrimental long-term effects associated with metal-on-metal hip prostheses motivated worldwide initiatives. In 2012, the Agency of Healthcare Research and Quality in the USA started a more efficient use of patient registries by creating a Registry of Patient Registries. They hope this initiative will expedite worldwide identification and dissemination of medium- and long-term adverse effects [82]. Many other countries have set up similar registries with post-marketing surveillance purposes [83].

Expert commentary

The estimation of clinical and economic effects associated with MDs in pre- and post-market evaluations imposes a number of challenges. These, however, do not support a departure in the methodology used to conduct EE between MDs and pharmaceuticals. Nonetheless, it is important to recognize that HTA process of MDs may need to consider issues over and above those for pharmaceuticals. Additional issues may include the continuous effort from MD manufacturers to improve the portability and usability of their products, organizational issues associated with adoption of new MDs (i.e., implementation and service delivery) and procurement negotiations. To an extent, consideration of some of these issues can be accommodated within existing methods for EE (i.e., considering additional costs associated with changes in implementation and service delivery in costing exercises as part of EE studies). Other issues may require use of alternative methods (e.g., discrete choice experiments to estimate user preferences associated with different configurations of MDs) that may complement and further inform the HTA process.

As previously discussed, sophisticated research methods are now available to recognize and robustly address specific characteristics of MDs’ evidence base. Their wider implementation will require dissemination and capacity building, but recent experiences with unexpected detrimental long-term effects, associated with the implantable MDs mentioned in section ‘How much of the status quo can/cannot be realistically changed’ suggest that additional efforts to safeguard the wellbeing of individuals may be warranted.

Five-year view

Following the experiences with the metal-on-metal hip replacements and the breast implants in 2011–2012, four large research initiatives were funded by the European Commission Funding Programme 7 (FP7) to look at the evaluation of MDs. The ongoing EUnetHTA, MedtecHTA, AdvanceHTA and ECRIN EU-funded projects are looking at different aspects regarding the clinical evaluation EE of MDs in the context of HTA [84–87]. EUnetHTA will produce guidelines for the conduct of MDs’ HTA [84]. Similarly, MedtecHTA will ‘…focus on improving the existing methodological framework within the paradigm of HTA for the assessment of MDs, and to develop this framework into a tool that provides structured, evidence-based input into health policies’ [85].

Research initiatives and MD regulators’ willingness to improve the evaluation of MDs worldwide will significantly increase stakeholders’ awareness of key challenges and opportunities. Manufacturers, regulators, healthcare providers, service commissioners, patients and policy makers will be encouraged to implement robust evaluations of the clinical effectiveness and value for money of MDs. In fact, before 2020 it is expected that alternative theoretical frameworks to evaluate MDs as well as practical ways to implement them will begin to emerge in the HTA literature. To make a difference, these efforts will require commitment from the different stakeholders to promote, support and engage with robust evaluations of MDs. For example, professional societies and hospitals involved in the MD evidence generation process may promote the conduct of clinical studies that recognize the complementarity of RCT and observational evidence. Similarly, offering ‘new and advanced MDs’ could be complemented by the provision of critical appraisals of the evidence that supports claims of benefit and cost–effectiveness. As we have discussed here, MDs can and need to be rigorously evaluated.

Endnotes

According to the Global Harmonization Task Force (now the International Medical Device Regulators Forum) [9]:

• ‘Medical device’ means any instrument, apparatus, implement, machine, appliance, implant, reagent for in vitro use, software, material or other similar or related article, intended by the manufacturer to be used, alone or in combination, for human beings, for one or more of the specific medical purpose(s) of: diagnosis, prevention, monitoring, treatment or alleviation of disease; diagnosis, monitoring, treatment, alleviation of or compensation for an injury, investigation, replacement, modification, or support of the anatomy or of a physiological process; supporting or sustaining life; control of conception; disinfection of MDs; providing information by means of in vitro examination of specimens derived from the human body; and does not achieve its primary intended action by pharmacological, immunological or metabolic means, in or on the human body, but which may be assisted in its intended function by such means.

• Note: Products which may be considered to be MDs in some jurisdictions but not in others include: disinfection substances; aids for persons with disabilities; devices incorporating animal and/or human tissues; devices for in vitro fertilization or assisted reproduction technologies [88].

The NICE defines cost–consequences analysis as [35]:

• ‘Cost-consequence analysis considers the costs and resource consequences resulting from, or associated with, the use of the technology under evaluation and comparator technologies, as well as considering relevant clinical benefits (e.g., effectiveness outcomes) alongside the cost analysis.

• The range of costs and resource consequences to be included in the analysis depends on the clinical characteristics of individual medical technologies and their comparators. Generally, the following apply:

  • – Typically, cost–consequence analysis frameworks include calculating and presenting estimates of resource use and of clinical benefits as separate domains of the evaluation.

  • – Estimates of resource use should include comparative costs of technology (and infrastructure) acquisition, use and maintenance. Focusing on these costs may be particularly applicable when the clinical effects of the technology can be assumed to be almost the same as those of comparator technologies.

  • – Estimates of resource use may also include the comparative value of healthcare service use outcomes (such as length of hospital stay, or number of hospitalizations, outpatient or primary care consultations) associated with the use of the technology or its comparators.’

Key issues

• Some characteristics of the evidence base on medical devices (MDs) are a consequence of the incentive mechanisms derived from current pre-market regulation that is, licensing.

• The fragmented and poor quality evidence base usually available for MDs artificially introduces a higher level of complexity to estimate their effectiveness and value for money.

• Flexible and creative RCT design methods, discrete choice experiments, mixed methods for Bayesian expert elicitation and Bayesian statistical methods represent a powerful tool kit to generate robust clinical evidence and begin characterizing and addressing methodological issues associated with the evidence base on MDs.

• Well-established full methods of economic evaluation in healthcare that is, cost–effectiveness, cost–utility analysis, cost–benefit analysis are suitable to ascertain the value for money of MDs.

Acknowledgements

The author would like to thank the three anonymous reviewers for their valuable comments on an earlier draft of this manuscript. Equally she is grateful to professors TA Sheldon, RT Edwards, MF Drummond, K Claxton, MJ Sculpher, J Hutton, K Bloor and A Maynard for comments and discussions on earlier drafts of this manuscript.

Disclaimer

The views expressed in this manuscript are those of the author and do not necessarily reflect those of the UK Medical Research Council or of the National Institute for Health and Care Excellence.

Financial & competing interests disclosure

C Iglesias is currently recipient of a MRC post-doctoral fellowship in Health Services Research and Health of the Population. She is also member of the Medical Technologies Assessment Committee at the National Institute for Health and Care Excellence. The author has no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.