From patient uncertainty to WHO policy in two years: A GP's experience of preparing and maintaining a Cochrane review

Abstract

Clinical encounters in daily practice can provide a fertile ground for identifying uncertainties that require further investigation. Addressing such uncertainties by undertaking a Cochrane review can be a rewarding educational process and result in important contributions to health care policy. This paper describes the experiences of a UK GP undertaking a Cochrane review whilst working in clinical practice. It outlines some of the practical issues when starting a review, the importance of effective mentorship and collaboration, the power of the modern medical media (BMJ, Wikipedia) and engaging with policy makers (WHO).

Keywords::

• systematic reviews
• meta-analyses
• research methodology
• general practice/family medicine
• treatment/intervention research medical education

KEY MESSAGE:

• General Practice provides a fertile ground for identifying uncertainties that require further investigation.

• Addressing such uncertainties by undertaking a Cochrane review can be a rewarding educational process.

• The impacts of preparing and maintaining a review can go far beyond simply completing the review itself.

INTRODUCTION

Clinical encounters in daily practice can provide a fertile ground for identifying uncertainties that require further investigation. Addressing such uncertainties by undertaking a Cochrane review can be a rewarding educational process. The impact of preparing and maintaining a systematic review can go far beyond the completion of the review itself, as demonstrated in this paper. Yet, undertaking a systematic review can seem a daunting prospect when faced with the realities and pressures of daily clinical practice. The aims of this paper are to demonstrate the impact that following an uncertainty through from routine general practice can have on health care policy. In doing so, I will also demonstrate that many of the skills required to complete a Cochrane review are actually essential features of being a GP. Where enhanced skills or additional support are necessary, I shall make these clear.

IDENTIFYING UNCERTAINTIES IN DAILY PRACTICE

In March 2010, during a routine morning GP surgery, a patient I knew well told me in passing he was going to India for his summer holidays that year. I offered to arrange the appropriate travel vaccinations for him and emphasized the importance of hepatitis A coverage— the most common vaccine-preventable virus acquired during travel. In reply, he asked me two simple questions—‘Does it work?’ and ‘Is it safe?’ The WHO website indicated that millions of doses of the inactivated vaccine had been administered world-wide, but to my surprise failed to offer a clear statement as to clinical efficacy and safety (1). I promised the patient I would go away and search for the best available evidence to address this uncertainty.

SEARCHING THE LITERATURE

A GP should be able to transform a clinical uncertainty in to an answerable search question, systematically retrieve and critically appraise the best available evidence (2). I used the PICO (patient, intervention, control, outcome) framework to develop a question that was answerable for the uncertainty I was presented with:

• Patient: Adults previously unexposed to hepatitis A

• Intervention: Hepatitis A vaccines (inactivated or live attenuated)

• Control: Placebo or no intervention

• Outcome: All-cause mortality; hepatitis A related mortality; clinical hepatitis A

This structure enabled me to search easily those databases freely available online such as PubMed and the Cochrane library. To my surprise, I could find few high-quality randomized controlled trials and systematic reviews addressing the outcomes of interest to my patient. Instead, much of the existing literature related to proxy outcomes such as antibody production.

STARTING A COCHRANE REVIEW

At the time of starting the review, I had just started working one day a week as an Academic Clinical Fellow in the University of Liverpool with the remainder of my time spent in clinical practice as a GP trainee. It was at this point I spoke with John Holden, a Cochrane review author about the prospect of undertaking a review (3). He suggested we undertake a review of our own and offered to mentor me through the process. As David Sackett highlights, having a good mentor at this stage brought me three things as a Cochrane newcomer: First, resources without obligation—possibly a place to work and some local funding, but certainly access to advice from someone who had made the journey before; second, opportunities without demand—an introduction to his contacts within the Cochrane collaboration and some hands on practical experience of undertaking a review; third, a degree of protection—insulating me from needless academic buffeting (4).

John emphasized the importance of collaboration— working together to achieve a shared goal. He recommended we invite Daniel Pope, an experienced Public Health researcher from the University of Liverpool to join us. We registered the title with the Cochrane Hepato-Biliary Group (one of the many Cochrane review groups) following the guidance on their website (5,6). The Editorial Team, Christian Gluud and his colleagues at the editorial office in Copenhagen, then helped us to develop and publish a protocol and at the same time, refine our search strategy (6). Many parts of developing the protocol were reasonably intuitive to me as a GP, such as identifying clinically relevant outcomes rather than proxy outcomes such as serology, which I realized are very hard to interpret. Other parts, such as using, refining the approach to data analysis were less straightforward and went beyond the skills required of a GP. However, all the problems faced were invariably covered by the Cochrane handbook, Cochrane training resources or advice from the editorial team (7).

DATA RISING IN THE EAST

Interestingly, our initial literature search alluded to the widespread use of a live attenuated vaccine in China. The three of us suspected that there might be some randomized trials, of relevance to our review, published in the Chinese literature. We invited Rongrong Yang—then a University of Peking PhD student visiting the University of Liverpool and working with Dan Pope—to join our review group and search the Chinese literature. The four of us soon found a large number of trials that would otherwise not have been identified, had we restricted our search to the usual Western medical databases. Some of the trials we identified were huge and included over half a million participants (8,9). However, many of the studies we identified were of poor methodological quality. Assessing potential studies for their risk of bias is a development of critically appraising evidence, and in Cochrane reviews it is performed using explicit criteria (7).

ENGAGING THE MEDIA

We presented our preliminary findings at the 2011 Wonca conference in Warsaw, in a session chaired by Domhnall MacAuley, the BMJ editor (10). Here we reported that both the inactivated and live attenuated hepatitis A vaccines were clinically effective, that whilst there was evidence to support the safety of the inactivated hepatitis A vaccine, more high quality evidence is required to determine the safety of live attenuated vaccines. We highlighted that many of the large trials included in our review were published in Chinese and only available through Chinese databases. Domhnall later wrote in his blog about the potential for Chinese research powered by the sheer size of their population (11). Geoff Watts then invited me to contribute to a BMJ feature article titled ‘Data rising from the East’— focusing on the near industrial scale of research output now coming from China, emphasizing key findings from our review. For example, issues of quantity over quality (12). As a GP, I had little prior experience of dealing with the media, but again managed to find support from within Cochrane Collaboration. Engaging with such a powerful media raised the profile of our work and served as an important stepping-stone to our next task—changing policy.

CHANGING WHO POLICY

Owing to formal links established between the Cochrane Collaboration and the World Health Organization (WHO), I was contacted by Steven Wiersma and Jördis Ott and invited to present our findings in Geneva to the experts working on hepatitis A vaccine policy. Our contribution strengthened several existing recommendations when measured against the GRADE criteria, but also identified a number of gaps in existing knowledge, thus raising questions that required further investigation (13–15). These included the need for more evidence on the effects of single doses of both inactivated and live attenuated vaccines; evidence on the efficacy and safety of the live attenuated vaccine with fewer risks of bias; and a review of the evidence on long-term protection. I then went on to co-author a paper with a team from the WHO addressing this question (16). I took on this additional challenge at the request of the WHO as they felt my knowledge of the available evidence, having undertaken the review, was as comprehensive as any other ‘expert.’

PUBLICATION AND WIKIPEDIA

The completed Cochrane systematic review was published on 13 July 2012; two years after my patient had posed the questions we had addressed. In terms of the number of participants included, our review appears to be the second largest vaccine-related review in the Cochrane Library (17). Only two days after publication an anonymous member of the public had used our findings to update the Wikipedia entry for hepatitis A vaccine—a wonderful example of social media bringing high quality evidence into the public domain (18).

DETERMINANTS FOR PRODUCING A SUCCESSFUL REVIEW

There were undoubtedly factors that contributed to the successful completion of this review. First, making efficient use of time in practice was essential. The review took hundreds of hours to complete from start to finish. Quieter times were seen as opportunities to take the project forward. Learning my own effective patterns and places of work improved my productivity enormously. This included making lists to prioritize tasks. I also found working entirely online and using a laptop made life much easier for me, especially if the day involved a train journey or prolonged time out of the office. Second, as discussed earlier, the importance of effective mentorship cannot be overstated. When I was feeling stuck and frustrated, John would often quote the maxim of Pete Goss, a former Royal Marine who received the Legion d’honneur for sailing two day in hurricane force winds day to rescue a fellow sailor in the round the world yacht race: ‘be professional at all times; never give up; make intelligent use of everything at your disposal.’ An illustration of this point was making use of at least eight experts within the institutions associated with the Cochrane collaboration itself, University departments and the WHO when the way forward was difficult or unclear. Finally, the importance of effective collaboration cannot be overstated. As a group, we worked together to utilize our complementary skills and achieve our shared goal. Having the right chemistry helped enormously to operationalize this relationship, especially that there were no ‘passengers,’ all authors made essential contributions. This fostered a sense of trust and a degree of ‘give and take;’ essential ingredients for effective collaboration.

I urge other clinicians to consider preparing and maintaining Cochrane reviews when inadequately addressed questions arise during clinical encounters. I believe they will find the experience a fulfilling use of their time. To get involved with the Cochrane movement as a GP, I recommend visiting the Cochrane Primary Health Care Field web site (19).

ACKNOWLEDGEMENTS

John Holden (Garswood Surgery), Daniel Pope (University of Liverpool), Rongrong Yan (University of Peking), Christian Gluud (Cochrane Hepato-Biliary Group), Steven Wiersma (WHO), Jördis Ott (WHO) and Sir Iain Chalmers (James Lind Alliance).

Declaration of interest: The author reports no conflict of interest. The author alone is responsible for the content and writing of the paper.