Defining the benefits and challenges of stakeholder engagement in systematic reviews

Abstract

Background:

Although there is a growing literature on the process of engaging stakeholders in medical research, there are a lack of clearly-defined measures for reporting and evaluation, which limits the ability to learn from past experience, understand the effectiveness of engagement, or identify which approaches work best. Clearly defining the benefits and challenges of engaging stakeholders in the systematic review process is an integral first step toward developing a set of criteria that can be used to evaluate the impact and effectiveness on the conduct, quality, and dissemination of systematic reviews.

Methods:

We utilized two complementary approaches to examine the benefits and challenges of engaging stakeholders in the systematic review process: 1) a literature scan to understand the overall state of the field; and 2) a series of key informant interviews with systematic reviewers, program/policy officials, and stakeholders.

Results:

We identified six main expected benefits and five primary challenges of involving stakeholders in systematic reviews. Benefits included: establishing credibility; anticipating controversy; ensuring transparency and accountability; improving relevance; enhancing quality; and increasing dissemination and uptake of findings. Challenges included: time; training and resources; finding the right people; balancing multiple inputs; and understanding how to match the right type of stakeholder to the right time in the systematic review process.

Discussion:

The results of this study are an important first step toward developing mechanisms for evaluating the effectiveness of stakeholder engagement in systematic reviews. Future work should seek both to verify the appropriateness of these benefits and challenges and identify concrete criteria for evaluating the effectiveness of different methods, timing, and intensity of stakeholder engagement.

Keywords::

• stakeholder
• medical research
• key informants
• AHRQ
• Effective Health Care Program
• evidence based medicine

Introduction

Background and rationale

Medical research has evolved over the years from the paradigm of the lone research scientist to a model that encompasses a broader range of clinical research activities, including multidisciplinary team science. One of the more recent developments in this evolution is the active engagement of stakeholders (ie, a person or group with a vested interest in a particular clinical decision and the evidence that supports that decision¹) in the selection, design, funding, and conduct of medical research, including systematic reviews. According to the Agency for Healthcare Research and Quality (AHRQ) Effective Health Care (EHC) Program, stakeholders include: patients, caregivers, and patient advocacy organizations; clinicians and their professional associations; institutional health care providers, such as hospital systems and medical clinics; government agencies; purchasers and payers, such as employers and public and private insurers; health care industry representatives; health care policymakers at the federal, state, and local levels; and health care researchers and research institutions.¹ The overarching goal of stakeholder engagement in systematic reviews and other medical research is to generate evidence that is more relevant and useful to those making real-world health care decisions, with the hope that this will increase the dissemination and uptake of research findings in clinical practice.^2,3 Activities to ensure engagement range from relatively simple approaches, such as asking stakeholders to participate in research meetings or comment on research proposals, to more resource-intensive methods, including the use of Delphi techniques to reach consensus about research priorities and involving stakeholders directly in making funding decisions or conducting research.^4,5

Despite the growing emphasis on stakeholder engagement, there are a lack of clearly-defined measures for reporting and evaluation, which limits the ability to learn from past experience, understand the effectiveness of engagement, or identify what approaches work best.⁶ Clearly defining the benefits and challenges involved in this process is an integral first step toward developing criteria for evaluating the impact of stakeholder engagement on the conduct, quality, and dissemination of systematic reviews. Indeed, knowing how to make an activity “more effective” presupposes an understanding of the desired outcome or benefit as well as the challenges, tradeoffs, and adverse consequences. Toward this end, this paper examines the following questions: 1) what are the expected benefits of involving stakeholders in systematic reviews, and 2) what are the perceived challenges of involving stakeholders in systematic reviews?

We define expected benefits (hereafter referred to as benefits) as the expected positive changes to the quality and impact of the systematic review as a result of engaging stakeholders in the process. Additional positive externalities from engagement (ie, positive consequences of the process that did not directly affect the quality and impact of the review such as building relationships, educating stakeholders about the research process, and personal benefits of involvement) were not included in the scope of this report. We define perceived challenges (hereafter referred to as challenges) as the costs and difficulties associated with engagement that need to be balanced against the benefits of involvement.

Methods

To begin to answer these questions, we utilized two complementary approaches: 1) a literature scan; and 2) a series of key informant (KI) interviews.

Literature scan

We searched a range of databases to identify published and unpublished studies that address the benefits, challenges, and/or measurement of stakeholder engagement in order to: 1) understand the overall state of the field and whether there were existing efforts to explicitly define or evaluate the impact of stakeholder engagement in primary research or systematic reviews; and 2) better inform and focus the KI interviews. The complete search strategy is included in the full report by Cottrell et al.⁷

Three investigators reviewed 299 abstracts and 80 full text articles. We included articles on the benefits, challenges, and/or measurement of stakeholder engagement in systematic reviews or medical research more generally. We excluded articles that were published prior to 2005, those that primarily focused on how to engage stakeholders (instead of why or when), and those not relevant to medical research or systematic reviews. Twenty-four articles met the inclusion/exclusion criteria;^4,6,8–26 only four articles were specific to systematic reviews.^14–16,18 Information regarding benefits, challenges, and/or measurement/evaluation of stakeholder engagement was extracted from these articles and synthesized in conjunction with the results of the KI interviews. See the full report by Cottrell et al for details on included studies.⁷

Key informant interviews

To supplement the information gleaned from the limited literature in this area, we conducted a series of semi-structured interviews with systematic reviewers who engage stakeholders in their work, program/policy officials who commission and/or use systematic reviews, and stakeholders who have been involved in the systematic review or research process.

A balance of representatives from each of the above groups was invited to take part in a 90-minute individual or group interview. Of the 60 potential KIs invited, 34 agreed to participate – 15 systematic reviewers, ten program/policy officials, and nine stakeholders. In all, we conducted 12 discussion sessions with between one and four participants each. Interviews were conducted by the lead author using a semi-structured interview guide designed to elicit a multi-faceted understanding of the expected benefits and perceived challenges of engaging stakeholders in systematic reviews. Although the primary purpose of the interviews was to understand the perceived benefits and challenges of stakeholder engagement as a necessary first step toward developing metrics and measures, we also asked interviewees for their suggestions on how such evaluation might be done. See the full report by Cottrell et al for the interview guide.⁷

Transcripts were entered into NVivo software for analysis. Two investigators and two research assistants read all of the transcripts to identify recurring themes, with an eye toward articulating the benefits and challenges of engaging specific types of stakeholders at each phase of the systematic review. Finally, we noted any particular suggestions by KIs of how to measure or evaluate the impact of engagement. Themes from the KI interviews and literature scan were reviewed with the larger working group during bi-weekly calls; disagreements were resolved through consensus.

Results

Overall, the themes from the KI interviews largely echoed the benefits and challenges outlined in the literature on stakeholder engagement in systematic reviews and medical research more generally. However, KIs provided a more nuanced understanding of the tradeoffs involved and augmented the nascent literature with specific insights into the benefits and challenges of engaging stakeholders in systematic reviews.

In short, expected benefits identified by KIs and in the literature clustered into six overarching domains: 1) establishing credibility;^6,8–15 2) anticipating controversy;¹⁰ 3) ensuring transparency and accountability;^{6,8–10,13–20} 4) improving relevance;^{6,8–10,12–22} 5) enhancing quality;^{6,8–10,12–24} and 6) increasing dissemination and uptake of systematic review findings.^{6,9,10,12–15,18–22} Perceived challenges included: 1) time;^{10,12,14,17,18,20,22,23,25,26} 2) training and resources (both for stakeholders as well as systematic reviewers);^{4,10,12,14,17,18,21–23,25} 3) finding the right people;^{9,10,14,17,22,23,25} 4) balancing multiple, often competing inputs;^10,11,14,23 and 5) understanding when to engage specific types of stakeholders.^10,14,23 We discuss each of the overarching benefits and challenges below using context from the KI interviews. Additional details on the results, including descriptions of the articles reviewed in the literature scan and data from the KI interviews, are available in the full report by Cottrell et al.⁷

What are the expected benefits of involving stakeholders in systematic reviews?

Establishing credibility

One of the immediate benefits is establishing credibility and gaining buy-in from the stakeholder community.^6,8–15 Systematic reviewers and program/policy officials in particular suggested that engaging stakeholders from the beginning increases the likelihood that they truly understand the process and feel that their insights are incorporated into the design and conduct of the review. KIs who had been engaged in a review as a stakeholder reported that it gave them a sense of investment in the process and findings, as opposed to feeling that the results were simply “announced” to them at the end.

Anticipating controversy

Several KIs explained that a key benefit of engaging stakeholders is the ability to anticipate controversies and opposing views that could potentially derail a report’s relevance. Systematic reviewers and program/policy officials in particular underscored the imperative of understanding the context and history of a given topic, including areas of scientific uncertainty or debate and politically charged or “hot button” issues from a consumer or advocacy perspective. Moreover, they described instances when stakeholders were able to uncover “blind spots” including issues of current debate or concern that impacted what questions were asked or how the report’s findings were communicated.¹⁰

Ensuring transparency and accountability

Engaging stakeholders in the review process was described as a key aspect of ensuring transparency and accountability, particularly in situations where systematic reviews directly determine policy.^{6,8–10,13–20} KIs from the stakeholder and program/policy perspective suggested that engaging a broader community in the process is a democratic right or moral imperative. Systematic reviewers and program officials both noted that mandates to engage stakeholders are important aspects of displaying accountability, transparency, and ongoing support for systematic reviews.

Improving relevance

Another primary motivation for engaging stakeholders is the belief that engaging the end users – patients, clinicians, policymakers, or guideline developers – is necessary to produce reports that are timely and relevant.^{6,8–10,12–22} There was general agreement that engaging those who would use the evidence was the best way to ensure that a systematic review addresses the right questions, includes the right outcomes, and that the review team does not miss a critical perspective. KIs repeatedly emphasized that stakeholders bring an understanding of the context that comes from having in-depth professional or personal experience with a topic; without this input, systematic reviews could miss the target and end up gathering dust on shelves rather than changing practice.

Enhancing quality

Perhaps one of the strongest overriding themes was the sense that engaging stakeholders would improve the overall quality of the review.^{6,8–10,12–24} Systematic reviewers and program officials identified a number of specific areas where stakeholder input improved the scientific quality of the final report, including: 1) framing the review and defining the key questions; 2) helping the team to refine the scope of the review; and 3) establishing appropriate parameters for the population, intervention, comparator, outcome, timing, and settings considered. Systematic reviewers credited stakeholder input with identifying nuances that were integral in helping to clearly define the population, intervention, comparator, outcome, timing, and settings considered in a given review. Moreover, clinical experts were identified as instrumental in providing valuable context that may not be readily apparent from the published literature, including identifying tests and procedures that are obsolete, or highlighting dosing and side effect profiles of different classes of medications that may influence study findings.

Increasing dissemination and uptake of systematic review findings

All of our discussions and several of the articles reviewed touched in some fashion on the idea of engaging stakeholders as a route to improving the dissemination and uptake of systematic review findings.^{6,9,10,12–15,18–22} KIs from all perspectives agreed with the principle that involving stakeholders meaningfully in the process should, at least in theory, increase the likelihood that systematic reviews address the right questions; are relevant to the real-life decisions faced by patients, clinicians, and policymakers; and that the findings are actually used to guide and inform real-life medical decisions. Moreover, KIs expressed the hope that by involving stakeholders in the conduct of systematic reviews, they were more likely to become advocates for the results, thereby facilitating more widespread dissemination and uptake of the final product.

What are the perceived challenges of involving stakeholders in systematic reviews?

Time

KIs from all perspectives agreed that engaging stakeholders requires a significant investment of time, especially if it is to be done well; this idea resonated in the literature as well.^{10,12,14,17,18,20,22,23,25,26} Systematic reviewers repeatedly emphasized that engaging stakeholders in the systematic review process lengthens the project timeline, especially if stakeholders are engaged at more than one time point. Given how rapidly new research becomes available, this can be a significant problem for systematic reviews. There was disagreement as to whether the extra time involved resulted in an improved product, with some arguing that the time spent engaging stakeholders was at least partly recouped by avoiding missteps, and others arguing that getting the results out to stakeholders earlier was more important than engaging them in the systematic review process. This investment of time was also cited as an issue for stakeholders, who usually have other professional or caregiving responsibilities.

Training and resources

In addition to the extra time involved, a lack of appropriate training and resources can also limit the benefits of stakeholder engagement.^{4,10,12,14,17,18,21–23,25} Many investigators are unfamiliar with how to best utilize and engage stakeholders and lack the skills required for successful management of such a process. Moreover, stakeholders who do not have a clinical or research background may require additional training and ongoing support in order to make a meaningful contribution to the process. KIs from all perspectives, but especially systematic reviewers and those who had served as stakeholders, suggested that ensuring that both sides have the necessary background and training, and sufficient resources to support their role, would go a long way toward increasing the overall benefit of stakeholder engagement in systematic reviews.^{4,10,12,14,17,18,21–23,25}

Finding the right people

Identifying and inviting stakeholders to participate in the review process is not an exact science and figuring out who to engage can be a challenge.^{9,10,14,17,22,23,25} Moreover, because US Office and Management and Budget Paperwork Reduction Act regulations (the Office of Management and Budget Regulations, USA) limit the number of stakeholders who can be engaged in a given review conducted for the federal government, the choice of who to engage carries a special weight in those cases. Getting it wrong can limit or negate the value of the engagement. During one discussion among systematic reviewers, there was agreement that in many instances the benefit of engagement is highly correlated with which types of stakeholders are engaged and/or the make-up of the stakeholder group.

KIs who had been involved as stakeholders and systematic reviewers alike noted that not all stakeholders have the same level of commitment and/or ability to contribute significantly to the systematic review process.^{9,10,14,17,22,23,25} Engaging those with high commitment runs the risk of engaging those with the most conflicts of interest. Moreover, as highlighted above, stakeholders who are not intimately familiar with the scientific review process may need substantial training in order to make valuable contributions.

Both stakeholders and systematic reviewers also emphasized that beyond their clinical expertise or personal experience with the topic, less tangible individual characteristics often play a large role in shaping the value and impact of stakeholder engagement, including personality traits, level of social influence among peers, level of personal motivation and willingness to engage, comfort with sharing opinions among a room full of “experts”, and comfort with group processes. Some KIs even suggested developing a database of “good stakeholder” contacts so that those who are a proven asset to this type of work can be accessed for future work.

Balancing multiple inputs

Another challenge is defining the relative value of different stakeholder input.^10,11,14,23 Only a limited number of stakeholders are engaged per review and it can be a challenge to determine whether a particular stakeholder is an outlier or has a personal or political agenda. As one systematic reviewer emphasized, not all input is equal and it is often difficult to assess which points to incorporate. It is a delicate balance to make stakeholders feel that their insights are valued, while at the same time being mindful that not all ideas can be incorporated into the review given both scientific and resource constraints. Some systematic reviewers expressed concern about expectations associated with being able to respond to all opinions expressed through stakeholder engagement. On the other hand, KIs who had served as stakeholders reported feeling pressure to come up with new or interesting insights and contributions or raise issues that the systematic review team had not addressed. Stakeholders also reported feeling intimidated by the process or under pressure to serve as the representative voice for a large population of people.

Understanding how to match the right type of stakeholder to the right time in the systematic review process

An overarching theme from both systematic reviewers and stakeholders who had been engaged in a systematic review was the concern that mandates for more extensive stakeholder engagement in systematic reviews seem to suggest that “more is better”, but there has been less attention to understanding when in the process it is most useful to engage particular types of stakeholders.^10,14,23 There is a danger of “lumping” all stakeholders together into a homogenous group, without recognizing the relative value of different contributions at various points in the review process.

Discussion

Although the findings of our KI interviews largely echo the benefits and challenges outlined in literature,^4,6,8–29 KI interviews provided a more nuanced understanding of tradeoffs, including the variation in relative benefits and challenges depending on who is engaged and when they are engaged. Systematic reviewers in particular highlighted the importance of engaging the right stakeholders at the right time and explained that the benefit of engagement often depends on the characteristics of the stakeholders involved. They also noted the challenge of deciding what input to incorporate into the report, and the inability to respond to all stakeholder comments and/or concerns.

Overall, one of the strongest messages from the KI interviews was the importance of linking the benefits of engaging specific types of stakeholders at different stages of the review process, rather than simply engaging a broad range of stakeholders in all phases of the review as “insurance” or in response to programmatic requirements. Of all phases of the systematic review, the “Topic refinement and research development” phase was repeatedly identified as the point where stakeholder engagement yielded the greatest benefit, with a preponderance of comments focused on the importance of stakeholder engagement at this phase in order to get buy-in and credibility for the process and product, to ensure the systematic review was scientifically valid, relevant, and useful, and to ensure uptake and use of the review products.

Despite a general agreement on the utility, neither the literature scan nor the KI interviews revealed any explicit efforts to evaluate the effectiveness or impact of stakeholder engagement. When we asked KIs to suggest possible methods, the most frequently cited was to evaluate at the impact of stakeholder engagement by assessing the quality, relevance, and dissemination and uptake of a review. For example, KIs suggested that tracking how a review is used to guide policy decisions or to inform guideline development may be an important indicator of the impact of engagement. A number of systematic reviewers mentioned the possibility of looking at this question retrospectively by comparing the quality and usefulness of past reports with varying degrees of engagement. Other KIs proposed pragmatic measures of the effectiveness, such as tracking the changes in key questions, inclusion and exclusion criteria, or overall scope of the review.

The results of this study are an important first step toward developing mechanisms for evaluating the effectiveness of stakeholder engagement in systematic reviews. Although the findings of our KI interviews are limited by a relatively small sample, the themes are similar to those found in the handful of papers on stakeholder engagement in systematic reviews^14–16,18 and the relatively more developed (yet still nascent) literature on stakeholder engagement in medical research more generally.^{4,6,8–13,17,19–26} The congruence between themes suggests that the benefits and challenges of stakeholder engagement in systematic reviews may be generalizable to medical research, and vice versa. Future work should seek both to verify the appropriateness of the set of benefits and challenges highlighted in this paper and to take the next step of identifying criteria for evaluating the effectiveness of different methods, timing, and intensity of stakeholder engagement.

Author contributions

EKC conducted all interviews, participated in conception and design of the project, analyzed and interpreted project material, and drafted the manuscript. EPW helped analyze and interpret project material and critically revised the manuscript. EK helped analyze and interpret project material and critically revised manuscript. SU helped analyze and interpret project material and critically revised the manuscript. SB helped analyze and interpret project material and critically revised the manuscript. CC helped analyze and interpret project material and critically revised the manuscript. TH helped analyze and interpret project material and critically revised the manuscript. DOM helped analyze and interpret project material and critically revised the manuscript. HN helped analyze and interpret project material and critically revised the manuscript. KAR helped analyze and interpret project material and critically revised the manuscript. MM helped analyze project material and helped to draft the manuscript. JA helped analyze project material and helped to draft the manuscript. RAP helped analyze and interpret project material and critically revised the manuscript. JMG participated in conception and design of the project and helped draft the manuscript. All authors read and approved the final manuscript.

Acknowledgments

This project was funded under Contract 290-2012-00004-C from the Agency for Healthcare Research and Quality (AHRQ), US Department of Health and Human Services.

Disclosure

The author reports no conflicts of interest in this work. This paper is based on a Research White Paper produced for the Agency for Healthcare Research and Quality (AHRQ), and was posted to the AHRQ EHC Program Web site on March 18th 2014. The authors of this document are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the US Department of Health and Human Services.