Sustaining surveillance: Evaluating syndromic surveillance in the Pacific

Abstract

Prior to the 2009 H1N1 pandemic, the Pacific Island Countries and Territories (PICTs) had agreed to develop a standardised, simple syndromic surveillance system to ensure compliance with International Health Regulations requirements (rapid outbreak detection, information sharing and response to outbreaks). In October 2010, the new system was introduced and over the next 12 months implemented in 20 of 22 PICTs. An evaluation was conducted to identify strengths and weaknesses of the system, ease of use and possible points for improvement. An in-country quantitative and qualitative evaluation in five PICTs identified that the most important determinants of the system's success were: simplicity of the system; support from all levels of government; clearly defined roles and responsibilities; feedback to those who collect the data; harmonisation of case definitions; integration of data collection tools into existing health information systems; and availability of clinical and epidemiological advice from external agencies such as the World Health Organization and the Secretariat of the Pacific Community. Regional reporting of alerts, outbreaks and outbreak updates has dramatically increased since implementation of the system. This syndromic system will assist PICTs to detect future influenza pandemics and other emerging infectious diseases and to rapidly contain outbreaks in the Pacific.

Keywords:

• syndromic surveillance
• Pacific
• evaluation
• early warning system
• infectious diseases

Introduction

In 2010, in recognition of the challenges posed by recent infectious disease outbreaks (Duffy et al. 2009, Thein et al. 2010) and the potential impact of emerging infectious diseases (Morens et al. 2004), Pacific Island Countries and Territories (PICTs) commenced implementing a standardised, simple syndromic surveillance system (Kool et al. 2012). This system was built on the introduction of influenza-like-illness (ILI) surveillance by all PICTs during the 2009 influenza pandemic. Further details on the history, functionality and implementation of this system are reported elsewhere (Kool et al. 2012).

Critics have questioned the value and sustainability of syndromic surveillance when public health resources are scarce (Reingold 2003, Koo 2005). Both formative and summative evaluations of syndromic surveillance systems are essential to explore determinants of successful system implementation and to allow adjustment during system expansion. Evaluation also ensures that opportunity costs associated with system development are justified by improved outbreak detection and response (Sosin 2003).

Towards the end of the first year of system implementation, a formative study was conducted to evaluate the implementation of the Pacific syndromic surveillance system, with a focus on degree of adoption, data quality, reporting timeliness and compliance and to ensure that required changes were identified early in the life of the system. The study examined the system's ability to act as an early warning system for detecting, investigating and responding to outbreaks, and explored factors that appeared to contribute to successful syndromic surveillance system functioning in low-resource settings.

Methods

The syndromic surveillance system evaluation was undertaken by an independent, external epidemiologist (BP) engaged by the World Health Organization (WHO), between May 2011 and September 2011, using: semi-structured key informant interviews during in-country visits in five PICTs; observational techniques including field inspections of raw data and data collection methods; and analysis of syndromic data reported to WHO by all participating PICTs. In-country evaluations of participating PICTs were chosen based on their participation in the system for a reasonable length of time; implementation approach; and time and resources for the evaluator to travel to participating PICTs. The evaluation was adapted from the Centers for Disease Control and Prevention (CDC) framework for evaluating syndromic surveillance systems (Buehler et al. 2004), the WHO guide for monitoring and evaluating surveillance and response systems for communicable diseases (World Health Organization 2006), and formative evaluation techniques commonly used in public health evaluations (Patton 2002, Braun and Clarke 2006). Immediate feedback was provided during in-country visits on how system performance could be improved. To limit possible withholding of relevant information, informants were notified that responses would be anonymised. Country level reports of in-country visits were provided to individual PICTs by the evaluator for confirmation and follow-up.

Key informants included national syndromic surveillance coordinators, senior Health Ministry staff, Directors of Public Health, doctors, nurses and clerks in hospitals and health clinics, statisticians or health information officials and other public health officials, including an expatriate epidemiologist. Additional interviews were conducted with WHO and Secretariat of the Pacific Community (SPC) staff. Interview guides were pre-prepared for different categories of informants, and interviews were recorded for transcription and thematic analysis (Braun and Clarke 2006). Where possible, collection points were visually inspected and data entry demonstrations observed. The reliability of identified themes was tested with public health officials from six other PICTs, who were interviewed by phone or completed a semi-structured questionnaire by email.

Quantitative data from participating PICTs from November 2010 to September 2011 were analysed in Excel 2007 for timeliness (percentage of participating PICTs reporting prior to the WHO weekly deadline) and compliance (percentage of participating PICT reporting to WHO each week and percentage of sites reporting from each PICT). Data quality was assessed during in-country visits by observing data collection practices and comparing patient-level data with data captured in the syndromic surveillance system.

Markers of system performance included the perceptions of key informants, the country participation and compliance rates and the capacity of the system to detect and respond to outbreaks, as recollected by key informants and contrasted with reports to WHO. The reporting of alerts, outbreaks and outbreak updates to the PacNet listserver, which has been established for longer than a decade for the rapid communication of epidemic threats in the region, were also examined and counted for the years 2010 and 2011 (before and after the implementation of the system). This count does not include reports duplicated in French. Regional alerts for PICTs and New Zealand were included in the count. Where alerts for a syndrome across multiple PICTs were notified in the same report, this was counted as a single alert.

Results

Six PICTs were approached for in-country interviewing with one refusal. Forty-three key informants were interviewed in-country: Cook Islands (n=10), Fiji (n=12), Kiribati (n=4), Nauru (n=8) and Tuvalu (n=9). A small number (7) of more remote informants were interviewed by phone or email: American Samoa (n=1), Guam (n=1), Palau (n=1), Papua and New Guinea (n=1), Solomon Islands (n=2) and Tonga (n=1).

System components and attributes

The purpose of the system is ‘To develop a simple, sustainable system that allows local health authorities to detect unusual cases and clusters of disease early, in order to respond rapidly to limit the impact of outbreaks’ (World Health Organization and Secretariat of the Pacific Community 2010). The system is based on the weekly reporting of four core syndromes (ILI, diarrhoea, prolonged fever and acute fever with rash), using common case definitions, to improve rapid outbreak detection, information sharing, response to outbreaks, and hence ensure compliance with International Health Regulation (IHR) requirements (Fidler and Gostin 2006, World Health Organization 2008). System components and attributes are described in Table 1.

Table 1. Pacific syndromic surveillance system: system components and attributes.

System components
System structure	Supported by WHO and SPC (including training); support at all levels of government; implementation manual ‘Practical guide for implementing syndromic surveillance in Pacific Island Countries and Territories 2010’ (World Health Organization and Secretariat of the Pacific Community 2010); sentinel sites generally based at major hospitals or health clinics; national focal-point for syndromic surveillance (responsible for tallying, analysing data, identifying outbreaks, initiating outbreak investigation and reporting).
Case ascertainment	Standardised case definitions of four core syndromes (influenza-like-illness – ILI, diarrhoea, prolonged fever and acute fever with rash); a fifth ‘syndrome’ is for unusual events; optional addition of syndromes at the local level; cases generally identified by doctor but occasionally by nurses or health information clerks.
Data collection and analysis	Paper-based (encounter forms, patient registers and logbooks) or electronic reporting mechanisms; guidance provided on analysis; threshold identification.
Reporting	Weekly reporting to WHO (including zero reports); unusual events reported immediately; regular feedback at the local level to surveillance sites and in-country stakeholders; weekly consolidated Pacific syndromic surveillance report sent to stakeholders via PacNet listserv (also made available on SPC and WHO websites).
Outbreak investigation and response	Thresholds for investigation; outbreak responses generally based on ‘Pacific outbreak manual’ (World Health Organization and Secretariat of the Pacific Community n.d.) or customised local outbreak manual; standard outbreak investigation steps; further detail on cases collected in the event of an outbreak; rapid local responses but provision of accessible public health advice or assistance from WHO and/or SPC on request.
System attributes
Simplicity	Based on the tallying and reporting of cases that meet four syndromic case definitions; does not require laboratory confirmation; high training needs due to staff turnover; perceived as a simple system by users.
Flexibility	Range of approaches implemented by PICTs; adapted from early pandemic influenza surveillance system and other earlier systems; includes a ‘fifth syndrome’ which captures unusual events; PICTs are able to include additional syndromes based on local needs.
Acceptability	Participants increased from 6/22 to 20/22 during the period November 2010 and September 2011; informants agreed that the system was useful and an improvement on previous systems; 84% of sites reported during the review period; assists PICTs in meeting their International Health Regulations (IHR) obligations.
Data quality	Variable data quality with some discrepancies between clinical diagnostic data and captured syndromic data; no regular data quality checks; training in the use of thresholds to be implemented in the next training round; high visibility case definitions improved case ascertainment accuracy.
Sensitivity	In areas where sentinel sites had been implemented, the system is sensitive enough to detect outbreaks; in remote areas or areas without sentinel sites, outbreaks could be missed.
Timeliness	Ninety one percent (575/631) of reports were received on time (weekly); rapid identification of increases in cases as based on syndromes rather than laboratory confirmation.

Acceptability and timeliness

The number of PICTs participating in the system increased from 6 to 20 of a possible 22 between November 2010 and September 2011 (Figure 1). During this period, 631 country reports were submitted to WHO. Ninety-one percent (575/631) of reports were received on time and this has steadily improved (Figure 1). After a significant decrease in 1 week (Week 9, 2011), it was recognised that reminders and encouragement by the central information unit at WHO was an important factor in ensuring on time reporting. From that date, WHO adopted standard operating procedures which included contacting non-responders at least twice by email or phone to remind them of the weekly reporting deadline. Informants noted that because the system was based on syndromes (identified immediately) rather than on laboratory confirmation (which could take weeks), the identification of outbreaks was more rapid than in previous years. There was great variability in the number of in-country reporting sites, ranging from 1 to 25, usually depending on population size, with some variation over time. On average 84% of participating in-country sites reported during the review period.

Figure 1.  Number of Pacific Island Countries and Territories (PICTs) reporting and number reporting on time (i.e., by Wednesday each week) to the WHO syndromic surveillance hub.

Note: There are 22 PICTs (not counting New Zealand), of which 20 participated in the system as of 30 September 2011. The month shown on the horizontal axis is the month containing the last day of the epidemiological week.

In-country key informants universally agreed that the syndromic surveillance system was valuable and a marked improvement over previous systems in allowing early detection and response to outbreaks. One informant commented: ‘this system … really everyone's taken to it; everyone feels like they own it, that it's useful’.

Implementation

Country-specific system review found that specific implementation elements varied between PICTs depending on available personnel, medical informatics systems and health system structure. A national surveillance coordinator, based within the Health Ministry, was generally assigned responsibility for system management and weekly reporting to WHO. System features common to all PICTs included: case ascertainment, case counting, outbreak detection, outbreak investigation, outbreak response, reporting and feedback. Some PICTs chose to collect data on other syndromes of local public health importance (e.g., conjunctivitis, dengue-like illness), in addition to the four core syndromes.

There was a wide variation in the type of technology employed to implement the systems. Technology included: automated, web-based mobile phone messaging systems; automatic extraction of the syndrome counts from the electronic medical records system; and paper-based recording and tallying. The use of sophisticated systems was often problematic due to variable Internet connectivity and technical failures. One clinician, when commenting on a highly technical system, noted, ‘it's a great system when it works’.

When commenting on how implementation could be improved and what contributed to system success, key factors identified by respondents included simplicity of the system; support at all levels of government; clearly defined roles and responsibilities; feedback, particularly to those who collected the data; harmonisation of case definitions across different in-country health information systems; integration of data collection tools into existing health information systems; and the availability of clinical and epidemiological advice from an external agency such as WHO or SPC.

Data quality

Data quality varied across the implementing PICTs. In-country data review revealed occasional discrepancies between clinical diagnostic data and captured syndromic counts. No regular data quality checks or protocols were identified during the in-country visits. Case ascertainment had generally improved over the first year, with a common feature being periods of improved data collection following training visits by WHO or SPC. It was noted that data collection could deteriorate when new untrained doctors or nurses were appointed in the surveillance sites. Despite data quality variability, increases in syndrome counts routinely triggered outbreak investigations; however, inconsistent data quality hampered preparation and understanding of trends. None of the PICTs visited had defined outbreak thresholds, in part, due to the limited available historical data and recognised variable data quality. WHO officials noted that threshold development was scheduled to occur during the next system training round. Follow-up interviews identified that thresholds are used in some other PICTs. An epidemiologist commented that ‘the system has benefited us by establishing thresholds for these syndromes so it can tell us if we are seeing something unusual’.

Application of case definitions varied, with some obvious confusion in certain PICTs where duplicate health information systems existed to capture syndromes for polio, measles and tetanus elimination programmes, using similar but not identical case definitions (e.g., elimination surveillance uses an age cut-off in its definition of ‘acute fever and rash’, which is primarily used to identify measles; the syndromic system, in order to be more sensitive, uses no age cut-off for this syndrome). Case definition awareness was highest where syndromes were integrated into the manual patient register or electronic health records system. High visibility case definitions in consulting rooms appeared to assist in achieving case ascertainment accuracy.

Improved outbreak communication and political acceptance

A common response from key informants was that the syndromic surveillance system had improved communication during outbreaks; often expressed as a ‘positive spin-off of the system’. This improved communication and openness was reported to have extended beyond outbreak responses. Respondents described how data sharing had improved, and noted that routine reporting of syndromic counts and outbreaks had decreased concerns about the political implications of outbreak reporting. ‘People are more comfortable sharing information because they feel that there is no shame and none of the repercussions that they used to fear’ (WHO staff member). The reporting of alerts, outbreaks and outbreak updates for the region through the PacNet listserver substantially increased from 29 in 2010 to 224 in 2011 (Figure 2). Informants also discussed how it was easier to share information about outbreaks once there were a larger number of PICTs participating in the system, as the regular reporting by many PICTs demonstrated that outbreaks were not uncommon and their PICT would not be singled out. Some concern was expressed by the Director of Health Services from one PICT, who noted negative trade ramifications following a media report implicating his country as the source of an influenza outbreak.

Figure 2.  The reporting of alerts, outbreaks and outbreak updates for the region notified through the PacNet listserver, by month, 2010–2011.

Clinicians consistently mentioned that regular information about outbreaks in the region improved their motivation to participate in the system. A senior medical officer reported: ‘It's useful so that we can alert our doctors and we can know if someone is coming from these countries then we can know they might have this [disease]’.

Requirement for system support

Interviewees commented that the clinical and epidemiological support provided by WHO and SPC was integral to system success. The agencies had collaborated to provide in-country training at system introduction, and respondents reported regularly seeking advice on appropriate responses to outbreaks, particularly for uncommon diseases. Senior in-country key informants remarked positively on the unified approach that had established system credibility and the high-level support (provided by the Pacific Health Ministers at their biennial Meeting) that had confirmed the international importance of the regional early warning system. High-level support was also considered a key requirement for success: ‘… the doctors are very responsive to authority; we need high level written support from the Minister either directing them or pleading with them [to motivate them to report]’ (Surveillance Coordinator).

Strengthened outbreak detection

Outbreaks were readily identified by the system, although in some smaller PICTs they were contemporaneously identified by outpatient clinicians. The value of the system was apparent to key respondents who reported that the only areas where outbreaks were initially identified through the media or by word of mouth were those where no sentinel reporting site existed, indicating that the system was sensitive enough to identify outbreaks. Respondents were extremely positive about the system's ability to detect outbreaks: ‘Late last year we had an epidemic of diarrhoea and gastroenteritis and we really picked it up from the surveillance system’ (Director of Medical Services). One outbreak was even identified during the initial weeks of system development. Interviewees also commented that it was easier to detect and respond to outbreaks earlier: ‘… we are now able to detect and respond to outbreaks earlier as the reporting from sentinel sites is based on symptoms rather than diseases which may take time to establish and report’ (country Head of Communicable Disease).

Simplicity – an improvement on earlier systems

The system was reported as an improvement on previous surveillance systems, which often included many more conditions, some of which were considered irrelevant to respondents: ‘Previously there were as many as 65 conditions – including overly common diseases like scabies and diseases that are almost impossible to confirm locally, like plague or smallpox – things that weren't useful for detecting outbreaks’ (WHO staff member). Clinicians also noted that it was easier to make a clinical diagnosis using the clearly defined case definitions in preference to laboratory confirmation which, because of the remoteness of many islands and the time taken for confirmation, limited its value for initiating an outbreak response.

Success of introducing the system for pandemic preparedness and meeting IHR requirements

A number of key informants commented on the implementation of regional ILI surveillance during the pandemic, and how with that structure in place, it was fairly simple to extend it to include the additional syndromes: ‘We learned from that experience [the pandemic] and we're just hoping that we can keep that structure and improve that structure’ (Health Protection Manager). Respondents provided examples of how they were updating their pandemic preparedness or disaster management plans, and the integration of the syndromic surveillance system into these plans: ‘… this is part and parcel of our disaster management plan. In fact, the surveillance system prompted the review’ (Director of Public Health). A number of key informants also mentioned the relationship between the system and IHR obligations, though not all were clear on whether they had met their IHR obligations: ‘IHR is one of the issues. We would like to know in detail what our IHR obligations are, so that we can see what we've done, what we need to improve and what we haven't done’ (Director of Public Health).

System quality improvement

A structural element that appeared important to timely outbreak investigation was the location of a surveillance officer within health programmes rather than in a statistical or health information area. Respondents remarked that this facilitated analysis of data and timelier outbreak responses. Key informants noted the importance of having clearly defined roles and responsibilities for each player to ensure an efficient outbreak response: ‘We should strengthen it by better supervision, by making sure the roles and responsibilities are clear’ (WHO staff member). There was some concern noted about the sustainability of the system and acknowledgement that there was still scope for further improving implementation in some PICTs. The requirement for regular training, particularly in PICTs with a high turnover of clinical staff, was raised by a number of country respondents: ‘My real concern is the sustainability … this is an issue that I've grappled with from day one … all my doctors are expatriates, all my clinical support are expatriates … I can't build capacity on the expatriates alone’ (Secretary of Health).

Discussion

The syndromic surveillance system has expanded from 6 to 20 participating PICTs within a year, indicating that there is a high level of system acceptance. The participating PICTs include a number of low-income countries, suggesting that there may be a similar syndromic surveillance system applied in developing countries in other parts of the world that are struggling to meet their IHR requirements.

Encouragingly, there is overwhelming agreement that the system is effectively acting as an early warning system and, while data quality and analysis are still variable, this has not lessened the ability of the system to identify outbreaks. Effective outbreak response is the critical ethical and functional element of any early warning system (Carrel and Rennie 2008). Consistent collection and examination of data and standardised approaches to reporting, outbreak investigation and response are necessary for a surveillance system to effectively serve as an early warning system. Regional reporting of alerts, outbreaks and outbreak updates has dramatically increased since implementation of the system. Improvements in case ascertainment, data quality and training will further enhance the system.

Support for the system at all levels of government is a key determinant of successful system implementation. Interestingly, the availability of clinical and epidemiological advice from an external agency such as WHO or SPC was considered equally important by respondents. Dedicated support and training from WHO and SPC, and local political support, appeared critical to initial implementation and sustainable functioning. This could have future implications for the sustainability of the system if this support is withdrawn. Pressure to conform to social norms may have assisted in catalysing participation as the system gained wider acceptance; several respondents cited the need to be seen actively collaborating in this IHR-related system.

One of the beneficial, and perhaps unintended, consequences of the syndromic surveillance system was a general improvement in communication and data sharing within the clinical setting, the broader health system, between different government departments and, externally, with regional neighbours and agencies. While the regular reporting of syndromes by a large number of PICTs appears to be mitigating the fear of political reprisal for outbreak reporting, which was identified as a barrier to reporting in some countries during the 2009 pandemic (Briand et al. 2011), the potential for political reprisal remains a possibility.

The inter-country variation in specific system characteristics appeared to be a positive attribute in a region with variable socio-economic and health system development. Despite marked differences in technology, personnel, health system resources and medical informatics, PICTs productively participated and contributed to the regional early warning system. Simple, manual systems were often more robust than sophisticated, automated systems, indicating that sophistication is not a requirement for participation. Harmonisation of the syndromic surveillance system and pre-existing local systems are issues that need to be addressed in some PICTs. Expansion with additional case definitions that can help detect important outbreak-prone diseases important in the Pacific, such as ‘dengue-like illness’, should be considered. However, the added disease detection capability should be weighed carefully against the increase of the reporting burden. Improving the visibility of case definitions could help improve system performance. High turnover of clinical staff in the Pacific region is a particular challenge and regular training is of major importance.

Although only key informants from five PICTs were interviewed in-country, the consistency of the derived themes was confirmed during interviews with key informants from five additional PICTs. A clear limitation is that the experience of PICTs during early system implementation cannot confirm sustainability and future success of the system. The WHO engaged an independent evaluator, and observations and reported data were compared with interview data to address the possibility that health officials might exaggerate the success of their country's implementation. A range of local respondents were interviewed to ensure a valid understanding of the local system. These comparisons confirmed that the systems were, in fact, functioning as described. Further evaluation of the system as it matures is recommended to provide longitudinal information.

There have only been a limited number of published evaluations of syndromic surveillance systems in developing settings (La Ruche et al. 2000, Durrheim et al. 2001, Nelesone et al. 2006, Jefferson et al. 2008, Meynard et al. 2008), although a number of authors have suggested that syndromic surveillance is appropriate for developing regions (Durrheim et al. 2001, Chretien et al. 2008, May et al. 2009, 2011). While the use of ‘low technology’ for syndromic surveillance has been recognised as having value in some developing settings (Chretien et al. 2008, Happel Lewis and Chretien 2008, May et al. 2009, 2011), there continues to be an emphasis on the technical aspects of syndromic surveillance systems and less emphasis on the importance of establishing a standardised process or framework (customised to each individual setting) for outbreak detection. The present evaluation demonstrates that technical capacity is only one component of a successful syndromic surveillance system and reliance on technology can be detrimental to a system when the technology fails, a not uncommon occurrence in developing countries.

While PICTs had agreed to establish a syndromic surveillance system prior to the 2009 pandemic, the need for systematic reporting of ILI cases during the pandemic encouraged countries to rapidly implement a system for collecting and transmitting syndromic data. The limited availability of laboratories for promptly confirming influenza demonstrated the usefulness of good case definitions to support clinical and public health interpretation and action. In the absence of this powerful incentive there may not have been the political will to implement the system so rapidly. The use of the system to assist countries in meeting their IHR obligations is another major incentive for implementation, though it is of concern that some PICTs reported not understanding if they were meeting their IHR obligations. These foundations made implementation of an expanded syndromic surveillance system an achievable aspiration. Detection of future influenza pandemics or other emerging infectious disease outbreaks in the Pacific will be greatly assisted by this syndromic surveillance system.

Acknowledgements

The authors would like to acknowledge the help and assistance of all those who so generously gave of their time to be interviewed during this evaluation. In particular, the assistance provided by Akanisi Dawainavesi, Surveillance Coordinator, Fiji, is acknowledged.