Perspectives on benefit-risk decision-making in vaccinology: Conference report

Abstract

Benefit/risk (B/R) assessment methods are increasingly being used by regulators and companies as an important decision-making tool and their outputs as the basis of communication. B/R appraisal of vaccines, as compared with drugs, is different due to their attributes and their use. For example, vaccines are typically given to healthy people, and, for some vaccines, benefits exist both at the population and individual level. For vaccines in particular, factors such as the benefit afforded through herd effects as a function of vaccine coverage and consequently impact the B/R ratio, should also be taken into consideration and parameterized in B/R assessment models. Currently, there is no single agreed methodology for vaccine B/R assessment that can fully capture all these aspects. The conference “Perspectives on Benefit-Risk Decision-making in Vaccinology,” held in Annecy (France), addressed these issues and provided recommendations on how to advance the science and practice of B/R assessment of vaccines and vaccination programs.

Keywords:

• benefit/risk
• conference report
• vaccine
• methodology

Introduction

Despite their major positive impact on public health, vaccines, like medicines, can also have undesirable effects. For more than 50 years, benefit/risk (B/R) assessment of vaccines has been one of the cornerstones for regulatory approval, though until recently the focus has been predominantly one of safety (risk). Nevertheless, systematic methods for B/R evaluation are rather new and few well-accepted models are in common use. The B/R assessment of vaccines, as compared with drugs, is further complicated by some specific attributes of vaccines and their use. For example, vaccines are usually given to healthy individuals, and, for some vaccines, the benefits may extend from the individual to the population level due to their herd effects.

To take a broad look at the B/R appraisal of vaccines, the Fondation Mérieux organized a conference from June 23–25 2014 entitled: “Perspectives on Benefit-Risk Decision-making in Vaccinology” in Annecy, France (“Les Pensières” Conference Center). Fondation Mérieux is an independent family foundation established in 1967 by Doctor Charles Mérieux with the aim of strengthening local capacities of developing countries to reduce the impact of infectious diseases. A multi-disciplinary group of experts drawn from academia, industry, international organizations and national public health institutes gathered to:

• Explore, through case studies, the specificities of B/R evaluation of vaccines

• Discuss data needs and methodologies for analysis of B/R in vaccine development and in post-marketing surveillance of vaccination programs

• Evaluate societal aspects of vaccination that should be taken into consideration in the analysis of B/R balance. This included perception of vaccines in the general population, reasons for vaccine refusal, motivation and incentive measures that could increase vaccine coverage and consequently impact the B/R ratio

• Put forth recommendations that would advance the science and practice of B/R assessment of vaccines and vaccination programs.

This report provides a summary of selected issues discussed by participants, key findings and recommendations for future approaches to addressing benefit-risk evaluation of vaccines.

Case Studies of Benefit-Risk Evaluation of Vaccines

For many established and well-characterized vaccines, benefit-risk assessment is straightforward and overwhelmingly positive. For others, analysis of B/R may be less straightforward as some important variables such as optimal endpoints, the benefits or the etiology of risks are not simple to define. Moreover, the B/R of a given vaccine might be different in the context of specific populations (e.g., pregnant women, children, etc.), geographical area (e.g., developing versus developed countries, high vs. low disease risk), the measured outcomes, or over time (e.g., pathogen evolution). B/R evaluation in some of these contexts is discussed below.

Influenza

Due to their high rates of influenza-related hospitalization and death, elderly (≥65 y old in most countries) remain the primary target population for reducing the burden of influenza through vaccination in most high-income countries. The rationale for vaccinating this population rather than the transmitters (children and young adults) comes mainly from observational studies ¹ and meta-analyses that reported 47 to 68% reduction of all winter deaths in vaccinated elderly.^2-4 Evidence accumulating since 2005 suggests that the benefit of classical influenza vaccines (i.e., inactivated, standard dose, non-adjuvanted) may be lower than previously believed, especially in the elderly where there is age-related decline in immune function, or immunosenescence.^5-8 Controversies over influenza vaccine B/R may be due to difficulties in estimating their benefits, thus leading to uncertainties and need for better data.

Pregnancy

The rationale for vaccinating pregnant women against certain diseases may be based on increased risk from complications of infection to the mother (e.g., influenza) or in order to promote passive transfer of maternal antibodies to the fetus in order to protect the subsequent newborn until the benefits from active vaccination can be attained (e.g. pertussis). Vaccination of pregnant women is not new: maternal vaccination against smallpox (1879)⁹ and pertussis (1938)¹⁰ were shown to confer protection to mothers and their newborns. However, after the thalidomide tragedy of the 1950s, regulatory authorities in Europe and the USA emphasized safety (risk) and excluded pregnant women from clinical trials, thus ensuring that any benefits from medicines or vaccines would not be assessed systematically in this group [Carol J. BAKER, USA]. The influenza pandemic of 2009 and the increased number of pertussis related deaths in young infants in the USA¹¹, UK and Wales in 2011¹² reopened the debate. During the 2009 H1N1 influenza pandemic, vaccine coverage in pregnant women, considered as a priority group by the World Health Organization, rose in the USA from 12% to 49%.¹³ Since that time, some data on benefits and safety of immunization in pregnancy with influenza ^14-16 and pertussis vaccines¹⁷ have been published. These contributed to the argument that pregnant women should no longer be systematically excluded from at least some vaccination programs because maternally-derived immunization can prevent mortality of their offspring and/or offer direct benefit to the mother.

HPV

Human papillomavirus (HPV) vaccine is an example of a product for which the B/R analysis is challenging because the intended benefit, prevention of cervical cancer, can take decades to observe. Accordingly, the 2 currently marketed HPV vaccines were developed and licensed based on a surrogate marker of protection, namely CIN 2+ (Cervical intraepithelial neoplasia) lesions, since most of these untreated lesions will progress to cervical cancer. Prevention of these lesions is therefore considered to be a reliable surrogate of efficacy. Due to the high clinical efficacy of these vaccines, CIN lesions are rarely, if ever, seen in vaccinees.^18,19 The B/R assessment is complicated by uncertainties in the knowledge about the beneficial effects (persistence of protection, extrapolation of data from one age group to younger and older ones) and unfavorable effects (rare events, potential effects in pregnancy). However, with high efficacy in the prevention of infection and CIN lesions and in the absence of clear adverse events from randomized controlled trials and data accumulated post-licensure, the B/R balance continues to be acceptable.^20,21 Accumulation of new data from extended follow-up of clinical trials continues to reduce uncertainty regarding duration of protection¹⁹ and will be needed in order to monitor the B/R of 2-dose strategies which may replace the initial 3-dose strategy.

Rotavirus

The example of rotavirus vaccines and the risk of intussusception (IS) highlights the issue of completeness of information for decision-making. The first licensed rotavirus vaccine, RotaShield^®, was introduced in the USA in 1998 and withdrawn one year later because of the observed increased risk of IS. As a result, the future of a potentially lifesaving vaccine for developing countries, which represent the highest burden of rotavirus-related diarrhea, was put in jeopardy.²² Clinical trials and post-licensure data of the 2 subsequent rotavirus vaccines licensed in 2006 and currently part of the national immunization programs of over 50 countries still demonstrate a small increased risk of IS,^23-26 but it is considered low compared with the large health benefits resulting from vaccination. Another specificity of rotavirus vaccine and its B/R evaluation concerns its dosing schedule. To limit the risk of IS, the WHO initially recommended that rotavirus immunization should be initiated by the age of 15 weeks and completed before age 32 weeks. However, in case of delay in childhood vaccination, this age restriction could impact vaccine coverage, thereby reducing its benefits.²⁷ Removing the age-restriction in developing world settings, where the vast majority of deaths from rotavirus occur, would likely outweigh the potential risk of IS-related deaths. [Umesh D PARASHAR, USA]. Indeed, based on a B/R model comparing excess rotavirus deaths averted with excess IS deaths due to vaccination, the WHO revised its recommendations in 2013.²⁸ While still encouraging early vaccination, the new position is that rotavirus vaccines should be given to infants along with recommended DTP vaccines regardless of age.

Polio

The example of polio vaccination emphasizes the importance of B/R assessment as a function of time and disease evolution. Oral polio vaccine (OPV) was first licensed in the 1950s as the exclusive critical tool for polio control but is now, ironically, turning out to be an obstacle to achieving eradication [Joel CALMET, France]. The choice for widespread OPV use was based on several key properties of the vaccine: low cost, conferring gut immunity and achievement of high routine coverage due to ease of administration. However, the emergence of circulating vaccine-derived poliovirus (VDPV) has been responsible for more than 500 cases of paralysis in immunocompromised individuals from 2000 to 2012 in several countries in an era when wild-type polio infection was becoming increasingly rare. Vaccination with OPV alone is therefore no longer considered sufficient for disease eradication by the WHO as some vaccinated individuals could excrete the VDPV for extended periods, thus sustaining virus circulation. The introduction of a single IPV dose in the immunization schedule of current OPV-using countries is expected to mitigate the risk of emergence of type 2 poliovirus as the world shifts from trivalent to bivalent OPV nearing the polio eradication end-game.²⁹

Methodologies For Assessing Benefits/Risks of Vaccines

Benefit/risk evaluation is inherently difficult because desirable and undesirable effects have to be balanced despite often having different measures and importance. B/R methods are defined as any tools, templates or models which attempt to summarize and integrate benefits and risks. This is ideally not a static but rather a dynamic process as it should change as data are accumulated during different stages of development and also after products are marketed [Christoph CONRAD, Germany]. Newly discovered unfavorable effects with uncertainties about their importance can also require updating the B/R assessment. The finding of porcine circovirus (PCV) sequences or infectious circovirus in rotavirus vaccines in 2010 ³⁰ is an example of the detection of a previously unrecognized adventitious agent that required a new B/R assessment in the post-marketing era. The World Health Organization (WHO) developed a guideline for the scientific assessment of risk with any new finding of a potential adventitious agent in an already registered biological product. The model used in the context of evaluating the impact of PCV sequences concluded that potential unforeseen risks linked to PCV are low and that the B/R balance for rotavirus vaccines remains positive.

Favorable effects of vaccines are frequently identified or better characterized after licensure. In the case of vaccines, herd immunity, i.e., benefits of vaccination beyond any individual but rather at the community level, cannot be estimated before widespread vaccine introduction and therefore is not often taken into consideration in B/R analysis. An increase in vaccine coverage leads to a decrease in pathogen circulation, thus providing additional protection to vaccinated individuals as well as to those not vaccinated. One recent example of this is the observed reduction in the incidence of disease caused by vaccine-containing serotypes of streptococcus pneumoniae in non-vaccinated age groups following introduction of pneumococcal conjugate vaccines.^31,32

Main methods of B/R assessment

Qualitative B/R assessment has been the primary means of assessment by regulatory and public health agencies. Such tools are often templates or guidelines and rely on expert judgment.

Quantitative methods involving modeling, based on principles from epidemiology and pharmacoepidemiology, are increasingly finding their way into use by various stakeholders. They complement qualitative approaches but are usually complex, may lack transparency and may or may not have well developed scales for many of the criteria. Challenges of quantitative approaches include assignment of numerical weights to outcomes, interpretation of complex models (e.g., stochastic models) and use of sensitivity analysis.

Currently, there is no single agreed upon methodology for vaccine B/R assessment. In fact, no single approach can fully capture all aspects of a B/R assessment. Choice of methodology should match the complexity of the problem [Shahrul MT-ISA, UK].

IMI-PROTECT project

To assess available B/R methodologies and develop tools for the vizualization of benefits and risks, the IMI-PROTECT (Innovative Medicine Initiative-Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium) project conducted a review of literature that identified 49 methodologies which they classified into 4 categories: frameworks, metrics, estimation techniques and utility survey techniques.³³ Application of the results in case studies has led to the development of a recommendation roadmap, with a very useful website,³⁴ to aid the selection of methodologies. It is organized around 5 stages: 1) planning, 2) evidence gathering and data preparation, 3) analysis, 4) exploration, and 5) conclusion and dissemination.³³ The process is not linear as there is a need to iteratively analyze the data.

Benefit-risk action team (BRAT) framework

Having more formal B/R decision tools should afford consistency, facilitate discussion between stakeholders and make the current B/R process more explicit and transparent. To respond to this, several B/R tools and frameworks have been developed in recent years. One such tool is the BRAT (Benefit–Risk Action Team) that was developed to standardize and communicate B/R assessment between pharmaceutical companies and regulators.³⁵ The US Food and Drug Administration (FDA) considers that presentation of benefit-risk considerations involves focusing on the individual (i.e., not population-level) benefits and risks, their frequency, and weighing them appropriately [Norman W BAYLOR USA].

Multi-criteria decision analysis framework

The multi-criteria decision analysis (MCDA) framework is another tool that assesses and integrates multiple benefits and risk criteria and compares multiple options. Through a 7 step approach, MCDA is used to construct a model for B/R assessment.³⁶ Among those 7 steps, 4 represent the most critical elements of the MCDA framework. The value tree (i.e., potential desirable and undesirable outcomes) is created by starting with all possible effects and then “pruning” based on pre-specified criteria in order to have a more workable and simple final product. Second, the effects table (i.e., source of data and effect estimates) is created according to a data selection strategy to determine which studies are eligible for inclusion. Third, scales are set in order to allow comparability of some otherwise disparate measures. The final and perhaps the most complex step involves assigning weights to each criterion according to their relative importance to the decision. The MCDA model was used to evaluate the B/R balance of the use of the Gardasil® in preventing anal cancer in males.

Accelerated development of vaccine benefit-risk collaboration in Europe

Further to the requirement for more formal methods to assess B/R, initiatives that involve regulators, industry and academia are also needed to harmonize such evaluations at a global level. Marketing authorization holders (companies) are responsible for monitoring the B/R of their vaccines. However, vaccine manufacturers often cannot directly access the necessary data required for B/R assessment as they may be collected and managed by governmental public health agencies. Through the European Commission Innovative Medicines Initiative (IMI), the ADVANCE (Accelerated Development of Vaccine benefit-risk Collaboration in Europe) project was established to build an integrated and sustainable framework for continuous vaccine effect monitoring [Alena KHROMAVA, Canada]. The project was initiated with the mission “to establish a prototype of a sustainable and compelling system”³⁷ that rapidly provides best available scientific evidence on vaccination benefits and risks post-licensure for well-informed decision-making [Miriam STURKENBOOM, Netherlands]. The vision of the project is to look from multiple perspectives at the same time, to monitor B/R during the life cycle of a vaccine in a real-time manner and to use existing data more efficiently (integrate information from different data sources, pooled across countries). The work-plan includes 5 steps that will be achieved by developing and testing a code of conduct, rules of governance, technical infrastructure, data sources, methods, and workflows in a European network of stakeholders.³⁸

Social Sciences and Benefits/Risks of Vaccines

The “real world” impact of vaccines on the burden of a disease, i.e., vaccination program effectiveness, is related both to the efficacy of a vaccine as well as its coverage in the population. Herd protection offered by high coverage enhances the overall vaccine effectiveness above the inherent clinical efficacy of the vaccine at the individual level. As a consequence, vaccine coverage may affect benefit-risk balance. Under-vaccination limits vaccine program effectiveness and its causes are heterogeneous; they include impaired access to vaccines, psychosocial stressors in families and vaccine refusal.³⁹ In many parts of the world, some parents of school-age children are increasingly claiming nonmedical exemptions to refuse vaccinations. The result can lead to pockets of under-vaccinated individuals which allows for propagation of more transmissible diseases and which have been linked with outbreaks diseases including mumps, measles, and pertussis.⁴⁰ Two main determinants of vaccine uptake, i.e., awareness and acceptance, are in the domain of social science. Understanding perception of risk is an important part of risk assessment and communication. Individuals make decisions based on value perception; therefore, providing accurate information and reliable sources for more information is crucial. By identifying psychological and emotional characteristics of vaccine acceptance that could be further addressed by public health deciders, social science can help policy makers improve vaccine coverage rates thereby contributing to the B/R equation.

Conclusions and Recommendations

Benefit/risk assessment methods are increasingly used by regulators and companies as important decision tools and their outputs as the basis of communication. However their use in vaccinology is still in its infancy and should consider certain specificities of vaccines, their use in vaccination programs, and their preventive nature. B/R assessment requires incorporating all relevant information on the safety and efficacy, including value judgment (clinical relevance of a given treatment, acceptable incidence of an adverse effect), uncertainties (statistical uncertainty, gaps in efficacy and safety data) and temporal effects. In the particular case of vaccines, other factors such as the additional population benefit afforded through herd effects should also be taken into consideration and parameterized in these models. B/R can be viewed from multiple perspectives, including that of the individual or of society. While national regulatory authorities evaluate B/R for the individual and public health agencies for the population, health care providers and individuals tend to evaluate the balance in term of patient and personal value respectively. These different perspectives should be weighted and integrated in the B/R tools. Effective communication about B/R of vaccines to the general public is one of the key factors influencing vaccine coverage, consequently impacting the B/R balance.

In addition to the above-mentioned challenges, evaluation of B/R in vaccinology suffers from the lack of harmonized and gold-standard methodology. Initiatives involving industry, regulators and public health authorities are ongoing to move toward consensus. A summary of challenges related to B/R assessment, recommendations and means to reach them, as provided by the expert group, is presented in Table 1.

Table 1. Recommendations to overcome challenges in B/R assessment of vaccines

Challenge	Recommendation	Approach to overcome the challenge (examples)
Assess B/R over time	B/R assessment should be considered as a continuum. It should start at the early stages of vaccine development; continue through the marketed life of the product and be updated as new data become available and/or by considering additional relevant endpoints.	Periodic Benefit-Risk Evaluation Report (PBRER) of European Medicines Agency
Optimize B/R assessment	Tools for assessing B/R of vaccines exist but need to be optimized by having more accurate data on vaccine exposure as well as outcomes (e.g. laboratory confirmation of cases). While these methodologies will support decision-making, they are not intended to replace medical judgment for individual patients.	Initiatives for better data and data sharing (ADVANCE, principles of data sharing, databases)
Establish harmonized frameworks for the evaluation of B/R in vaccinology	Networking, more interaction between regulatory authorities, vaccine manufacturers and academics	Set-up of more collaborative studies such as ADVANCE (EU), Mini-Sentinel (US)
Assess from a population perspective	Including not only qualitative and quantitative data, but also subjective values from a population perspective for B/R assessment. This can lead to more clinically relevant decisions.	Involvement of stakeholders such as patients and the public
Specific populations	It is crucial to be able to assess the B/R balance in specific medical conditions or risk groups not studied in clinical trials.	Consider adapting Pediatric Investigation Plan (EU) for monitoring, registers for pregnant women
Communication	Reporting of B/R assessments should be transparent as to how they were conducted	integrate Social Sciences activities in B/R evaluation and communication for more user-friendly reporting

While this conference arguably provided one of the most comprehensive review of the status and challenges of vaccine benefit-risk assessment today, it also highlights the remaining gaps to be addressed moving forward.

Disclosure of Potential Conflicts of Interest

MG is an employee of Sanofi Pasteur. FS worked for Sanofi Pasteur MSD at the time of the conference. Other authors declare that they have no conflicts of interest to report.

Acknowledgements

The authors express their gratitude to members of the steering committee: V Picot, C Jouquelet-Royer, J Louis, and L Marcelon for their contribution to the preparation of the manuscript and the scientific program. We are indebted to all speakers who shared their findings. Thanks are also due to Cindy Grasso (meeting coordinator) and staff of the Mérieux Foundation conference center for outstanding local organization.