Validating pharmaceutical product claims: questions a formulary committee should ask

Abstract

Claims, justifying the acceptance and placement of new products on health system formularies, are all too often presented in terms that are either unverifiable or only verifiable in a timeframe that is of no practical benefit to formulary committees. One solution is for formulary committees to request that (i) all predictive claims made should be capable of empirical testing and (ii) manufacturers in making submissions should be asked to submit a protocol that details how their claims are to be assessed. Evaluation of claims can provide not only a significant input to ongoing disease area and therapeutic reviews, but can also provide a needed link to comparative effectiveness research and value-based healthcare. This paper presents a set of protocol standards (PROST) together will questions that should be addressed in a protocol review.

Keywords::

• Pharmaceutical
• Formulary
• Protocol
• Standards
• Comparative effectiveness
• Collaboration

Introduction

Pressure from the US government, and from employer groups who pay health insurance premiums has shifted the focus of the US healthcare system from fee-for-service healthcare offerings to one that is value-based1^,2. Value-based care is focused on improving outcomes, lowering costs, and increasing overall access to care. Healthcare organizations have been making the shift toward increased collaboration, outcome-based payment, and new benefit design. This is changing the way entities pay for healthcare and how healthcare is delivered3.

Value-based contracting models represent an evolution in clinical and payment methodologies aimed at creating quality and cost outcomes, fostering greater accountability, and taking advantage of innovations in medical technology2^,3. These contract models align incentives across providers, members, employers, and payers to improve clinical outcomes and the patient experience along with improving cost efficiency2–4. For example, many markets currently have physician and hospital performance-based contracts, or capitation arrangements. In response to healthcare reforms at national and state levels, there also has been growth in accountable care organizations (ACOs), with payment and care delivery models that tie provider reimbursements to quality metrics and reductions in the total cost of care for an assigned population of patients.

The same pressure being applied to health-systems, providers, and patient care programs is likely for value-based assessment of pharmaceuticals. Questions about (1) improving the patient experience of care, (2) improving the health of populations and (3) reducing the per capita cost of healthcare (the Triple Aim) are relevant in the pharmaceutical domain5. The purpose of this article is to describe ideas for how pharmaceutical product claims can be validated and propose questions a formulary committee should ask during this process.

Formulary submission guidelines: a foundation for value-based care

Formulary submission guidelines developed by various health plans and organizations such as WellPoint (in 2014 renamed Anthem), the Academy of Managed Care Pharmacy (AMCP), and the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), can provide useful starting points for discussion. For example, in 2005, WellPoint released a number of guideline documents to support the process of formulary submission for both new products and for the review of existing products as part of ongoing disease area and therapeutic reviews6. These were reviewed and re-issued in November 2007. The two key documents were (i) guidelines to support new products, indications, and formulations, and (ii) guidelines to support product re-evaluations as part of disease area and therapeutic reviews7^,8. In addition, a supporting document was issued summarizing the evidentiary and analytical standards expected by WellPoint Next Rx in health technology assessments9. It should be noted that the standards described were recommended and not mandatory.

Underpinning the guidelines and WellPoint’s assessment methodology was the concept of an outcomes-based formulary. That is, interventions should be assessed in terms of the outcomes achieved10. This requirement for the substantiation or validation of claims was seen as a critical input to the process of formulary review; a review process that was intended to accompany products over their life cycle. The emphasis in the instructions given was that claims for clinical performance, cost-effectiveness, and budget impact were to be appropriate to the short and medium term—a time horizon of 2–5 years. All claims submitted should be empirically verifiable. Longer-term claims were not of interest as they were unlikely to be verifiable in a reasonable timeframe. Claims that could not be monitored or validated would be rejected; accepted claims should be re-visited on a regular basis as part of a program to support disease area and therapeutic class reviews. Manufacturers were expected to be prepared to devote resources to claims validation—although not necessarily within the WellPoint treating environment, as WellPoint would not release data specific to its treating environment. As part of the initial submission for a new product, indication or formulation, manufacturers were asked to provide a protocol detailing how they proposed to validate claims made, noting that the results of such a validation would be expected to be reported in any re-submission to support ongoing disease area or therapeutic class reviews.

If we accept the concept of an outcomes-based formulary, then the question of the monitoring and validation of product claims (value-based) is central to any commitment to a program of disease-area and therapeutic-class reviews. Our purpose here, therefore, is to build upon the earlier efforts around outcome-based formularies such as the WellPoint formulary submission model, and consider key questions that a formulary committee should consider in formulating evidentiary standards.

Overall guidelines for making evaluations practical

As mentioned earlier, health reform has resulted in payment and care delivery models that tie provider reimbursements to quality metrics and reductions in the total cost of care for an assigned population of patients. The use of products and services is being evaluated in terms of (1) improving the patient experience of care, (2) improving the health of populations, and (3) reducing the per capita cost of healthcare (the Triple Aim). Increasingly, the use of pharmaceuticals is seen as a program of care. From a program evaluation standpoint, decision-makers seek evaluations that are practical. The Centers for Disease Control and Prevention recommend four guidelines for making evaluations practical to decision-makers. They are (1) utility, (2) feasibility, (3) propriety, and (4) accuracy. The four standards are described next.

• Utility standards ensure that information needs of evaluation users are satisfied. Seven specific utility standards are11^,12: (1) stakeholder identification, (2) evaluator credibility, (3) information scope and selection, (4) values identification, (5) report clarity, (6) report timeliness and dissemination, and (7) evaluation impact (that is, findings should be reported in ways that foster follow-through by stakeholders).

• Feasibility standards ensure that the evaluation is viable and pragmatic. Three specific feasibility standards are11^,12: (1) practical procedures, (2) political viability, and (3) efficiency.

• Propriety standards ensure that the evaluation is conducted with regard for the rights and interests of those involved and affected. Eight propriety standards are11^,12: (1) service orientation, (2) formal agreements, (3) rights of human subjects, (4) respectful human interactions, (5) completeness, (6) full disclosure of findings, (7) conflict of interest procedures, and (8) fiscal responsibility.

• Accuracy standards ensure that the evaluation produces findings that are considered correct. Twelve accuracy standards are11^,12: (1) program documentation, (2) context analysis, (3) purposes and procedures, (4) defensible information sources, (5) valid information, (6) reliable information, (7) systematic information, (8) analysis of quantitative information, (9) analysis of qualitative information, (10) justified conclusions, (11) impartial reporting, and (12) meta-evaluation.

Interested readers may find more information about these standards on-line11^,12. Other program evaluation standards may be applied to the validation of pharmaceutical product claims. We propose that the standards of (1) utility, (2) feasibility, (3) propriety, and (4) accuracy should be applied. A priori agreement between the formulary committee (or other decision-making groups) and the product manufacturer (or their representative) should be made in writing regarding how to make the evaluations practical.

Technical standards

Practical evaluations must still be judged as scientifically and methodologically valid. Thus, there also is a need for analytical and technical standards in health technology assessment. For example, the CHEERS checklist13, the AMCP guidelines14, and a number of the ISPOR good practice reports15–17 for model building and presentation of results would be entirely consistent with a requirement for modeled claim validation. In the case of the ISPOR good practice reports, the latest budget impact good practices report15 provides a comprehensive framework for modeling budget impacts and predicting the financial implications in the short- and medium-term for ongoing product impact assessments. Similarly, the ISPOR good standards for undertaking observational studies16 are entirely relevant. Where there is a disconnect, however, is in the ISPOR standards for model validation17 as detailed by Langley in his critique of modeled claims18. The only standard for validating a claim consistent with the value-based philosophy would be for predictive claims; claims that can be validated within a 3–5 year timeframe. Claims which are either not intended to be validated, such as cost-per-QALY reference cases, or those that have no practical chance of being validated are not relevant.

With the caveat that any claim—whether it is for clinical (quality of care and safety outcomes), resource utilization, and costs or, more broadly, budget impact claims—must be testable within a meaningful timeframe, the AMCP guidelines provide a useful starting point. The important point to note is that the AMCP guidelines (as with other formulary or health agency guidelines) are static, while healthcare technology development is dynamic, highly competitive, and marked by rapid product enhancement and new product development. The guidelines are non-progressive in the sense that, while they formalize the standards and process for formulary listing and pricing, there is no follow-through. Once the product is on formulary, there has been little or no interest in evaluating claims and reporting results as part of ongoing disease area and therapeutic reviews. There are no standards for subsequent re-assessments, nor any link between the claims made and consequent effectiveness evaluations.

On the assumption that a formulary committee or health system is concerned to follow-up claims made in initial submissions, the question then arises as to how such a link can be implemented. One answer—and it is an obvious one—is that, as part of the initial submission, manufacturers should be asked to present a claims validation protocol that they will underwrite in collaboration with Pharmacy and Therapeutics committees and Health Technology Assessment panels, to support ongoing reviews.

Validation protocol

The critical link between claims made and their subsequent assessment is the validation protocol. This protocol should set out the detail of the claims, together with an agenda for validating and presenting the results to a formulary committee. The protocol should be seen as both integral to the initial submission and, when the protocol has been implemented, the results reported as part of a requested re-submission to the formulary committee or health system for continued placement and support of the product. Claims could be continually modified and reviewed over the life of the product, with future assessment agendas capturing new entrants to the treatment space.

It should be emphasized that, if a protocol is requested and/or proposed by a manufacturer, then it should be negotiated between the parties. The formulary committee or health technology assessment panel must be satisfied that the protocol meets minimum standards, that it is practical, and that the manufacturer is committed to implementing and reporting outcomes.

Application of a protocol is entirely consistent with the objectives in comparative effectiveness research (CER). A major objective of CER is to provide new evidence that links to ongoing systematic reviews of benefits and harms in different interventions. Considerable resources, for example, have been invested by the Agency for Healthcare Research and Quality (AHRQ) in their Effective Health Care Program (EHC). This program provides guidance on the conduct of systematic reviews of technologies and interventions, together with new research and the translation of findings for different audiences. Following the AHRQ contribution to comparative effectiveness reviews and their emphasis on moving away from evaluations of single interventions, a validation protocol should be seen as considering the benefits and harms of a range of options, focusing on the clinical context so that the validation focuses on the appropriate population and interventions. This pre-supposes that modeled claims for a product are presented for the appropriate population in the clinical context of options in the standard of care.

The protocol should not only be comprehensive in its coverage of treatment standards and treatment options, linking modeled comparator claims in the submission to the treatment environment within disease and therapy areas, but it must also recognize standards for comparative effectiveness research and evidentiary hierarchies in methods proposed. Outcomes end-points should be consistent with initial cost-outcomes claims, capturing resource utilization data and patient reported outcomes. As a manufacturer will, presumably, have submitted similar modeled claims to both commercial and non-commercial providers, the benefits of underwriting and reporting on such a research program would apply to a range of payers. Study designs may link retrospective administrative claims data to clinical or laboratory markers and apply principles of epidemiology and big-data techniques.

Given the accumulating evidence under the broad umbrella of comparative effectiveness research, it is important for the formulary committee to be able to put the proposed validation protocol in context. The primary purpose is, presumably, to generate new evidence in supporting predictive modeled claims. Manufacturers should be required to put validated claims in context, describing how this may change perceptions of treating options and intervention guidelines.

The validation protocol, once agreed upon with the manufacturer, may address concerns that the formulary committee has with the scope of the modeled clams and techniques used to extrapolate from clinical trials in claiming treatment effects over the next 3–5 years. Examples could include the treatment cohort characteristics, unreasonably narrow phase 3 protocols given the indication for the product, the need to consider the presence of co-morbidities on treatment effect claims and sub-populations in the protocol.

Irrespective of the proposed and ultimately agreed study design, time is of the essence. If a formulary committee undertakes disease area and therapeutic class reviews of products and associated interventions on a 3-year cycle, then manufacturers would be expected to have completed a study and presented the results of their claims validation well in advance of that meeting. At the same time, of course, other manufacturers would also be engaged in presenting reports on validating prior claims.

Protocol standards (PROST) proposal format

If a validation protocol is to be assessed and agreed on by a formulary committee, it should meet minimum standards for comparative effectiveness research (CER) and the dissemination of evidence-based assessments to clinicians, patients, and other decision-makers.

It is recommended that a representative national organization prepare a standard format for proposed validation protocols. These can then be tailored to the needs of individual formulary committees to include both study designs for validating cost-outcomes claims as well as formats for validating budget impact claims. If the formulary committee is not interested in budget impact claims being assessed by the manufacturer, that requirement can be dropped. It is important, however, that regular progress reports are made to the committee to ensure a timely delivery of results. A possible format for protocol standards (PROST) is given in Table 1.

Table 1. Protocol standards (PROST) proposal format.

Section	Section no.	Requirements
Title	1.0	Title: Proposed Validation Protocol for Modeled Product Claims: [name of product]
Abstract	2.0	Structured summary of protocol objectives, claims to be validated, target population, study design, data source, timelines, and reporting date
Context	3.0	A statement of the context of the proposed study to include a systematic review of effectiveness studies and a statement of the contribution this study is expected to make in evaluating competing interventions in this disease or therapy area
Claims	4.0	A statement of the claims to be quantitatively evaluated: 4.1 Clinical effectiveness (including patient reported outcomes) 4.2 Safety 4.3 Resource utilization 4.4 Direct medical costs 4.5 Cost-effectiveness 4.6 Budget impact
Target Population	5.0	Characteristics of study target population (age, gender, ethnicity, race) to include sub-groups and clinical markers (presence/exclusion of co-morbidities; stage of disease)
Data	6.0	Description of proposed data source(s) and any permissions required to access data/target population (e.g., IRB approvals, EMR approvals, access to administrative claims data) and confidentiality requirements
Study Design: Cost-Effectiveness Claims	7.0	Rationale and description of study design: 7.1 Rationale for choice of study design 7.2 Concurrence with good practice guidelines 7.3 Estimation and reporting of treatment effects 7.4 Timeframe 7.5 Treatment comparators 7.6 Data assembly 7.7 Confidentiality
Study Design: Budget Impact	8.0	Rationale and description of budget impact study design
Progress reporting	9.0	Proposed study timeline and interim progress reporting
Reporting	10.0	A brief description of proposed final report format, to include 9.1 Title 9.2 Abstract: study objectives and assessment results 9.3 Study design: brief description, including risks of bias and other limitations. 9.4 Claims validation: target population and sub-groups 9.5 Claims review: degree of confirmation of predictive claims and implications for place of product in therapy

Protocol standards (PROST) formulary committee review

In framing the questions a formulary committee should ask, it is not the intention to be unduly prescriptive. There are monographs and papers detailing ‘questions to be asked’ together with pro forma reporting standards and guides to assist in study design and reporting14^,18^,19. Rather, the focus here is on the committee addressing the core issue of how the modeled claims are to be validated. Manufacturers would be given considerable scope in study design, data requirements, study location, and assessment analytics. Manufacturers could be advised that protocol proposals may be sent for external assessment and that publication of results are encouraged. Table 2 lists some of the key questions that a formulary might ask in reviewing protocol proposals.

Table 2. Protocol standards (PROST) review questions.

Section	Questions
Objectives	Has the validation protocol provided a summary of the study objectives? Does the protocol abstract provide a summary of (i) population targeted; (ii) comparators; (iii) timeframe; (iv) data source; (v) endpoints; and (vi) reporting format?
Context	Has the validation protocol provided a systematic review to establish the context for the claims to be validated and the contribution this validation may have to an overall assessment of comparative treatment benefits and harms in the population targeted and indication for the product?
Claims	Has the validation protocol provided a summary of the modeled claims for cost-effectiveness to be validated? Has the protocol identified claims for (i) clinical benefits; (ii) safety; (iii) resource utilization; (iv) direct medical costs; (v) cost-effectiveness; and (if required) (vi) budget impact?
Targeted population	Has the validation protocol provided a rationale for the population to be targeted, to include (i) demographic characteristics; (ii) clinical status; (iii) disease stage; (iv) co-morbidities?
Study design	Has the validation protocol provided a rationale for the study design (e.g., prospective effectiveness trial, cohort/observational study)? Were alternative study designs proposed? Were risks of bias and other limitations described?
Reporting	Has the validation protocol provided a reporting format for study results and a timeline for delivery of those results?

Access to data through collaboration

An important challenge for validating pharmaceutical product claims is access to data, such as big data, that are needed for such reviews21. It is unreasonable to demand ongoing evaluation studies without access to data that would satisfy the concerns of a formulary committee or health technology assessment panel regarding target populations and local practice patterns. Failure to take such variation into account, from the perspective of payers, could result in claims of biased selection of data by the manufacturer, serious criticism of study findings, and dismissal of evidence. The burden cannot rest entirely on pharmaceutical manufacturers; co-operation by the formulary committee—minimally access to site-specific data if outside databases are not satisfactory—is essential. The restrictive access policies that have been claimed by health plans in the name of data security or proprietary business practices are counterproductive, and, with collaboration, resolvable.

The approach proposed in this paper involves joint decision-making among autonomous, key stakeholders of an inter-organizational domain20–24. Working together enables the participating organizations to create and capture mutual advantages that translate into positive return on investment and more efficient management. Necessary next steps may appear daunting. Thus, we propose that Collaboration Theory20–24 may serve as a guide for developing co-operative behaviors and formal partnerships that are necessary next steps. Collaboration Theory is comprised of five parts:

1. Collaborative performance system,

2. Information sharing,

3. Decision synchronization,

4. Incentive alignment, and

5. Integrated processes.

Collaborative performance system

A collaborative performance system devises and implements performance metrics that guide the collaborating organizations to improve overall performance20^,22. This process resolves the related issues of (1) who should be involved in determining the mutual objective, and (2) what performance objectives should be specified with respect to the mutual objective.

Information sharing

Information sharing denotes access to private data in all partners’ systems, enabling the monitoring of the progress of service provision as customers pass through each process in the overall system20^,22. This includes data acquisition, processing, representation, storage, dissemination, status metrics, cost data, and performance data. Such access to information enables the participating organizations to elicit the bigger picture of the situation that takes into account important factors for making effective decisions.

Decision synchronization

Decision synchronization can be defined as the extent to which the participating organizations are able to orchestrate critical decisions for optimizing overall performance20^,22. This includes re-allocating decision rights in order to synchronize planning and execution that seeks to match capacity for service provision with demand for services. The way to judge effective decision synchronization is based on its effects on (1) response towards fulfilling customer demands (logistical benefits) and (2) profitability and efficiency for participating organizations (commercial benefits).

Incentive alignment

Incentive alignment refers to the process of sharing costs, risks, and benefits among the participating organizations. This motivates entities to act in a manner consistent with their mutual strategic objectives, including making decisions that are optimal for the overall domain and revealing truthful private information20^,22. It covers estimating costs, risks, and benefits, as well as formulating incentive schemes such as pay-for-performance and pay-for-effort20^,22. The assumption is that the actions of individual organizations are based on the expectation the act will result in mutual benefit and will result in benefit to the individual organizations.

Integrated processes

Integrated processes refer to the extent to which participating organizations design efficient processes that deliver services to customers in a timely manner at lower costs20^,22. Explicit description of these processes allows organizations to synchronize the entire sequence of integrated work activities required to deliver services that fulfill customer needs. Flexibility is needed in order to respond to the variety of customer requirements at minimum costs with respect to supply capacity.

Conclusions

This paper presented ideas for taking next steps in validating pharmaceutical product claims. The principles were rooted in (1) addressing value-based care, (2) making evaluations practical, (3) using technical standards, (4) developing validation protocols, (5) using explicit formats, (5) conducting formal review and (6) collaborating on access to data. We propose that validating claims for pharmaceutical products and devices should be seen as a necessary part of any program of continuous quality improvement in healthcare delivery. The absence of feedback to formulary committees, healthcare providers and other decision-makers regarding product performance is a major gap in the ability to deliver effective, affordable and high quality healthcare.

The proposals set forward in this paper for claims validation protocols to accompany product submissions for formulary listing provide a link that, at present, is absent; a disconnect between prior claims and subsequent product performance. Implementation of a protocol requirement as an integral part of formulary decision-making and ongoing disease area and therapeutic class reviews would not only be consistent with established and recommended quality control standards, but go a long way to meeting a critical evidentiary need. Future work is needed for converting the ideas presented in this article into everyday practice.

Transparency

Declaration of funding

This manuscript has no funding to declare.

Declaration of financial/other relationships

AMC is a consultant to Prime Therapeutics, Applied Analytics. JME peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Acknowledgments

The authors thank Dr Paul Langley for his valuable guidance and insights during the preparation of this manuscript.