Abstract
Background: Eighty per cent of primary care patients with infectious conjunctivitis are treated with antibiotics, although in only 30%, there is a bacterial cause. An accurate diagnostic index to distinguish bacterial from viral conjunctivitis may help reduce unnecessary antibiotics.
Objectives: To validate and, if necessary, improve an existing diagnostic index for bacterial conjunctivitis.
Methods: Non-experimental validation and updating study of an existing diagnostic index in Dutch General Practice. We collected 210 adult patients with incident symptoms suggestive for acute infectious conjunctivitis. GPs completed a standardized questionnaire and a physical examination of the eye(s) and took a conjunctival sample for culture. Cultures were analysed masked for the GPs’ findings. On bad performance of the existing index on the new patients, we developed a new index combining the dataset on which the original model had been developed (n = 176) and the new dataset (n = 210). Bootstrapped backward variable selection and shrinkage of regression coefficients was used to protect the new index against bad performance in future patients.
Results: The bacterial culture was positive in 36.3%. The items age and number of glued eyes at awakening were consistent predictors. This model classified 48% (107/386) of patients at a low (< 25%) chance of having a positive culture and 2% as at high (> 70%) chance.
Conclusion: Correction of a previously derived diagnostic index for bacterial conjunctivitis yielded a simple index, based on history only. The index is potentially useful to rule out bacterial conjunctivitis in patients below 50 years of age with no history of glued eyes at awakening. This study underscores the importance of external validation of diagnostic indices.
KEY MESSAGE:
Only one third of cases of infectious conjunctivitis in primary care has a bacterial origin.
Discrimination between viral or bacterial origin on clinical grounds is difficult; a simple diagnostic index may help.
Younger patients and those without glued eyelids are less at risk for a bacterial origin.
INTRODUCTION
In primary care, acute infectious conjunctivitis is presented by 15 per 1 000 patients annually. Around 30% of cases are of bacterial origin, but more than 80% of patients receive antibiotics (Citation1–3). Prescription rates in general practice have diminished, but over the counter selling of chloramphenicol eye drops in the UK since 2005 provoked a net increase in the use of topical antibiotics by almost 50% (Citation4). Moreover, acute infectious conjunctivitis of bacterial origin, benefits only marginally from antibiotic treatment on the short-term (Citation5). Although the contribution of topical antibiotics to the increase in antibiotic resistance is difficult to demonstrate, restriction of their use is recommendable (Citation6).
Discrimination between bacterial and other causes of conjunctivitis on clinical grounds is difficult, and a systematic literature search showed that diagnostic indicators claiming to be informative were not evidence based (Citation1–3). General practitioners (GPs) claim that not so much medical reasons, but rather social factors such as reducing absence from work or missed school days urge them to prescribe topical antibiotics (Citation7).
To help reduce the number of unnecessary prescriptions of antibiotics, we previously developed a diagnostic index to estimate the probability of bacterial origin of conjunctivitis. This index uses ‘number of glued eyes at awakening,’ ‘itch,’ and ‘history of previous conjunctivitis’ as diagnostic indicators. Using these items, the overall pre-test probability of 32% of a bacterial conjunctivitis transformed into post-test probabilities between 4% and 77% (Citation8). An individual patient data meta-analysis of three previously published primary care trials showed that patients with ‘purulent discharge’ and ‘mild severity of redness of the eye’ had significant benefit from antibiotics. This analysis, however, included many children less than five years of age, in whom a bacterial aetiology is more often present (Citation9).
Since a diagnostic model often performs worse in populations other than that in which it was derived, external validation in another but similar population is warranted before implementation in clinical practice (Citation10–15). This paper reports on an attempt to validate externally our previously developed diagnostic index for bacterial conjunctivitis and the subsequent re-derivation of an adapted index.
METHODS
General design
Originally, the current study was designed as a formal external validation study of a previously derived diagnostic index for bacterial conjunctivitis. Therefore, it was executed according to a research protocol very similar to the previous study (dataset 1, n = 177) (Citation8). This led to a dataset (dataset 2, n = 212) that had the same structure as the previous one. Data from medical history and physical examination were modelled against the results of microbiological cultures (reference standard). The microbiologists performing the cultures were not informed about the results of medical history and physical examination; GPs did not receive the culture results (double blinding).
The Ethics Committee of the Academic Medical Center of Amsterdam exempted this study from formal approval, as no interference with usual healthcare occurred.
Participants
General practitioners (GPs). Sixty-two GPs from 22 practices in the Northern and Central parts of the Netherlands included adult patients with symptoms suggestive for acute infectious conjunctivitis. Fifteen GPs had been involved in the previous diagnostic index derivation study (Citation8).
Patients. Inclusion criteria were: red eye and either discharge or glued eyes at awakening. Exclusion criteria were: age younger than 18 years; pre-existing symptoms for longer than seven days; use of systemic or local antibiotics within the previous two weeks; herpes keratitis ever; recent loss of vision, recent eye trauma, or active infection with chlamydia trachomatis or Neisseria gonorrhoea. Participants were informed about the study and gave written informed consent. The inclusion and exclusion criteria were identical to those of the previous derivation study.
Data collection
General practitioners (GPs). Between April 2006 and November 2008 the GPs collected information for each consecutive patient, presenting with signs of infectious conjunctivitis.
Medical history. The medical history (questionnaire) contained questions about symptoms (itching, burning, foreign body sensation, tears, number of glued eyes at awakening, and duration of symptoms (days)), a history of other eye problems (allergic conjunctivitis, hay fever, pre-existent eye problems, eye surgery and previous infectious conjunctivitis) and medication.
Physical examination. The eyes were investigated with regard to the degree of redness of the eye (none, peripheral, whole conjunctiva but not pericorneal, or whole conjunctiva and pericorneal); the presence of periorbital oedema (absent, present); the kind of discharge (none, watery, mucous, or purulent); and the number of affected eyes (one or two).
Microbiological sample. GPs took one conjunctival sample for bacterial culture of the affected eye. In case of two affected eyes, the most severely affected eye as judged by the GP was designated as the study eye. If two eyes were affected equally severely, the eye which had shown the first symptoms was taken. After taking a sample of the conjunctiva of the study eye by rolling a cotton swab (Laboratory Service Provider, Velsen-Noord, The Netherlands) across the conjunctiva of the lower fornix, GPs put the sample in a Stuart transport medium and sent it the same day by surface mail to the laboratory for medical microbiology of the Medical Centre Alkmaar.
Microbiological procedures
At the laboratory, the swabs were inoculated directly after arrival onto blood agar enriched with 5% sheep blood, MacConkey agar, and chocolate agar. All media were made at the laboratory with standard ingredients (Becton Dickinson, Cockeysville, MD, USA). After standard inoculation, the blood agar and MacConkey agar plates were incubated for 48 h at 35°C; the chocolate plates for 48 h at 35°C, but in a 7% CO2 atmosphere.
Cultures were analysed daily according to the guidelines of the American Society for Microbiology (Citation16). All pathogens were identified using routine standard biochemical procedures. Suspected colonies were selected and further investigated by Gram stain. In case of gram-positive cocci, Gram stain was followed by catalase test, coagulase test or an optochine test. In case of gram-negative rods or cocci, sugar tests followed.
Statistical methods
External validation attempt. As a first step, we applied the regression equation underlying the previously derived index for bacterial conjunctivitis in the new dataset 2. We assessed discrimination (area under the ROC curve) and calibration (the extent to which predicted risks are close the observed risks). We used a calibration plot and the Hosmer–Lemeshow Chi-square test to investigate calibration. However, the validation model performed badly on both discrimination and calibration (see Results).
Re-deriving a new diagnostic index. Since the current (‘validation’) dataset 2 had the same structure as the previous (‘derivation’) dataset 1, we decided to combine datasets 1 and 2 to re-derive a diagnostic index for bacterial conjunctivitis, now built on a more robust dataset. Had the external validation of the previously derived index been satisfactory in terms of discrimination and calibration, combining the datasets and re-deriving a new model would not have been necessary.
Data preparation. Available were two datasets of 177 (set 1) and 212 (set 2) patients, respectively. In dataset 1, one patient was excluded because she scored negative on ‘redness of the eye,’ which was an inclusion criterion. In dataset 2, one 15 year-old patient had been inadvertently admitted and was analysed since the 18 year cut-off had no biological rationale for this study. Finally, two patients from dataset 2 were excluded because a bacterial culture result was missing. This left 176 (dataset 1) and 210 (dataset 2) patients for analysis (n = 386; ).
Table 1. Patient characteristics and pathogen species.
Collapsing variable categories. The type of redness was documented as ‘mild’ (peripheral only); ‘moderate’ (complete conjunctiva, but not pericorneal); ‘severe’ (complete conjunctiva). However, the moderate and severe categories were associated with a positive culture equally strongly. Therefore, we modelled redness as ‘peripheral’ versus ‘moderate/severe’. Similarly, for secretion the original categories of ‘none’ (n = 3), ‘clear,’ ‘mucous,’ and ‘purulent’ were recoded as ‘none/clear’ versus ‘mucous/purulent.’
Variable selection. In three separate analyses, we used backward elimination for variable selection starting with a model including all variables and eliminating those whose associations were weaker than P = 0.05, 0.10 and 0.157, respectively to assess the stability of the selected model across these thresholds (Citation17). For each threshold, 1 000 bootstrap samples were drawn, and the frequency with which a variable was selected into the model (bootstrap inclusion fraction (BIF) was counted (Supplementary Appendix Table 1. available at http://www.informahealthcare.com/doi/abs/10.3109/13814788.2013.842970). To enhance parsimony of the final model, we predefined a threshold for inclusion of predictor variables into the definitive model of 2/3, that is, 670 selections out of 1 000. To enhance clinical applicability, we first let the selection procedure select indicators that a physician may collect without actually seeing the patient. In a second step, we assessed the added value of indicators that necessitate visual inspection of the eye(s) by a medically trained person, namely, redness, secretion, and oedema. We plotted the shrunk predicted probabilities and their 95% CIs to enhance valid application of the diagnostic model in practice.
Parameter-wise shrinkage. The main reasons to combine both available datasets were: (a) the larger size of the combined dataset is likely to make the derivation of the new model more reliable; and (b) the larger size also enabled us to check the functional form of age (the only indicator that may be modelled not using dummy variables) more robustly (Citation17). To counteract over-optimism of the model, the regression coefficients of the final logistic model were made smaller using ‘parameter-wise shrinkage.’ This technique shrinks strong predictors less than weak ones since the likelihood of selecting a weaker predictor is more prone to chance than selecting a strong one (Citation18). The regression coefficients were multiplied (and attenuated) by these factors and their standard errors re-calculated.
RESULTS
External validation attempt of the previously derived index (n = 210)
Application of the existing diagnostic index to the newly collected dataset 2 (external validation; n = 210) clearly showed over-optimism of the index (data not shown). The AUC-ROC, as the overall measure of discrimination, fell from 0.72 to 0.58, while calibration proved particularly bad with a Hosmer–Lemeshow P-value < 0.0001. Therefore, datasets 1 and 2 were combined to a more robust dataset to re-derive a diagnostic index for bacterial conjunctivitis.
Patient and pathogen characteristics of the combined dataset (n = 386)
Patient characteristics of the combined dataset are presented in . The overall prevalence of a positive bacterial culture was 36.2% (140/386). The most prevalent pathogens were Streptococcus pneumonia (44/140 positive cultures; 31%) and Staphylococcus aureus (41/140 positive cultures; 29%).
Re-deriving a new diagnostic index using the combined dataset (n = 386)
shows the distributions of the candidate indicators stratified by culture result and their univariable odds ratios. (Supplementary Appendix Table 1. available at http://www.informahealthcare.com/doi/abs/10.3109/13814788.2013.842970) shows how stable indicators were selected across 1 000 bootstrap samples at alpha levels of variable selection of 5%, 10%, and 15.7%.
Table 2. Index tests and their univariable odds ratios for the combined dataset (n = 386). Values are numbers (%), unless stated otherwise.
‘History of glued eyes at awakening’ and ‘age’ were consistently selected for inclusion. The bootstrap inclusion fraction for ‘type of redness’ exceeded our 670/ 1 000 threshold at alpha levels of 10% and 15.7%, but not of 5%. Calibration was reduced by adding ‘type of redness’ (Hosmer–Lemeshow P = 0.756 down to P = 0.235) and we, therefore, omitted this indicator (). The shrinkage factors for the two coefficients of the dummies for ‘glued eyes’ and the coefficient for ‘age’ were 0.814, 0.892 and 0.929, respectively.
Figure 1. (Hosmer–Lemeshow) Calibration plots showing how the diagnostic probabilities of a positive bacterial culture predicted by the models correspond with the observed probabilities in ten groups of about equal size. (a) Model including age and the number of glued eyes at awakening derived on combined data (n = 386). (b) Model including age, the number of glued eyes at awakening and type of redness. Note how discrepancies between predicted and observed probabilities increased, particularly in the highest four deciles, compared to the left graph. The P-value for the fit has decreased from 0.756 to 0.235, indicating overall worse calibration.
The final model with ‘number of glued eyes at awakening’ and ‘age’ classified 47% (183/386) of patients at low risk (< 25%) of having a positive culture and 2% (7/386) at high risk (> 70%). facilitates the clinical interpretation of the re-derived index for bacterial conjunctivitis using ‘age’ and ‘history of glued eyes at awakening’ as the only diagnostic indicators.
Figure 2. Diagnostic probabilities of a positive bacterial culture as a function of age and the number of glued eyes at awakening in patients presenting to a general practitioner with symptoms of conjunctivitis. Diagnostic probabilities (prevalence) and 95% confidence limits of a positive bacterial culture as depending on age for patients with symptoms suggestive of acute conjunctivitis and who report to have no (a), one (b), or two (c) glued eyes at awakening. Probabilities and their 95% confidence limits were calculated after (parameterwise) shrinkage of the regression coefficients for age and the number of glued eyes at awakening to enhance application of the model to external populations.
DISCUSSION
Main findings
A previously derived index for bacterial conjunctivitis proved not to validate well in an independent dataset with a comparable patient mix as in the derivation study. In the current study among adults with signs of acute infectious conjunctivitis, a positive bacterial culture was present in 36%. The number of glued eyes at awakening and age were consistently associated with a positive culture result. Adding type of redness of the eye to this model contributed little to the discriminative capacity, but reduced calibration.
Strengths and limitations
Lack of external validation is probably one of the reasons of the failure to translate research into health practice (Citation14,Citation16). We started the current study with the aim of performing an external validation study of the previously derived and published index for bacterial conjunctivitis (Citation8), but performance of this index in the new (validation) population was not satisfactory. In general, two main reasons for worse performance of a diagnostic index in a validation cohort exist: inadequate development of the index and major differences between the derivation and validation population.
Our previous study—in retrospect—was of limited size, had relatively few events and the model was developed without adequate internal validation methods (Citation8). Our validation cohort (dataset 2), however, comprised a similar case-mix (in an exclusion criteria), the same measurements and outcome-parameters, and (partly) the same participating GPs. Combination of both datasets, therefore, offered the opportunity to update the existing index in a cohort of sufficient size and with more patients having bacterial infection. We now used the bootstrap resampling technique as a robust method of internal validation. We, therefore, think that our new index is more robust than the previously published index. This new index is simple and practicable, but doctors and pharmacists should, as always, be aware of several limitations. We excluded patients with symptoms of other (more severe) eye diseases. In particular keratitis and iritis might be overlooked by lay personnel; these patients generally present with pain and photophobia. Signs and symptoms, which are strong predictors but (very) seldom occur, will not be selected in our model, as is the case in most diagnostic models.
Comparison with existing literature
This study confirms that only about 30% of patients with conjunctivitis seen by a GP might have a bacterial infection. We found glued eyes (as reported by patients and as a proxy for purulent discharge) and age as predictors for a bacterial origin of a present infectious conjunctivitis.
In clinically diagnosed acute bacterial conjunctivitis in adults, topical antibiotics might improve the five-day remission rate by 31%, in comparison with placebo (Citation18). A recent meta-analysis showed that patients with mild severity of redness of the eye and purulent discharge might profit from topical antibiotics (Citation9). Purulent discharge probably is the cause of the eyelid(s) being glued.
Implications for future research
This study confirms the necessity to validate diagnostic indices in comparable, but other patients as those, in whom the derivation of the index took place.
Strictly speaking, this new index therefore, needs external validation to guarantee absence of over or under fitting and a management study to determine its effect on decision making in practice. Infectious conjunctivitis, however, mostly is a self-limiting disease and in general antibiotic treatment has only small benefits (Citation9). Application of our model might serve as a starting point for future trials, investigating the benefits of such treatment, as it will enrich the included population and thus enhance contrasts between intervention and placebo groups.
Implications for practice
Topical antibiotics generate side effects. In the Netherlands, an increasing in-vitro resistance (61%) against the most used ocular antibiotic (fusidic acid) is seen (Citation18). Prescribing and delivering (unnecessary) antibiotic treatment also generates costs. In the UK, about 4 million preparations for topical antibiotic treatment of the eye are issued annually while 74% of cases with acute infectious conjunctivitis recover within seven days without treatment (Citation9).
Our new index has the advantage that it may be used by lay personnel, including pharmacists. Using this model, it is possible to classify around 48% of patients with signs of infectious conjunctivitis below 25% chance of a positive culture. As antibiotic eye preparations nowadays are available without prescription in many European countries, our model might help doctors and pharmacists to persuade patients to refrain from antibiotic treatment. Patients with a conjunctivitis, but without glued eyes are at low risk of a bacterial origin (, maximum risk < 40%). For those younger than 25 years, the risk is lower than 10%. By contrast, in patients with two glued eyes and older than 65 years of age the risk is over 70% ().
In conclusion, when adult patients present with infectious conjunctivitis a simple diagnostic index might help reduce the number of antibiotic treatments, as benefits are small and half of the patients can be explained that they have a small chance of any profit, as their complaints probably will not be caused by a bacterial infection. Formally, however, a validation study still is necessary and the impact of applying this index on health care needs to be assessed.
Supplementary Appendix Table 1
Download PDF (24.5 KB)ACKNOWLEDGEMENTS
The authors thank the participating general practitioners and their patients for making this study possible and their willingness to invest their time in a study from which they did not have any direct advantage.
FUNDING
This study was supported by a grant of ZonMW, The Netherlands Organization for Health Research and Development. No 42000024.
Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.
REFERENCES
- Everitt H, Little P. How do GPs diagnose and manage acute infective conjunctivitis? A GP survey. Fam Pract. 2002;19:658–60.
- Hovding G. Acute bacterial conjunctivitis. Acta Ophthalmol. 2008;86:5–17.
- Rietveld RP, van Weert HCPM, ter Riet G, Bindels PJE. Diagnostic impact of signs and symptoms in acute infectious conjunctivitis: Systematic literature search. Br Med J. 2003; 327:789.
- Davis H, Mant D, Scott C, Lasserson D, Rose PW. Relative impact of clinical evidence and over-the-counter prescribing on topical antibiotic use for acute infective conjunctivitis. Br J Gen Pract. 2009;59:897–900.
- Sheikh A, Hurwitz B. Antibiotics versus placebo for acute bacterial conjunctivitis. Cochrane Database Syst Rev. 2006; 2:CD001211.
- Scott G. Over the counter chloramphenicol eye drops. Br Med J. 2010:340.
- Rose PW, Ziebland S, Harnden A, Mayon-White R, Mant D, on behalf of the Oxford Childhood Infection Study (OXCIS). Why do general practitioners prescribe antibiotics for acute infective conjunctivitis in children? Qualitative interviews with GPs and a questionnaire survey of parents and teachers. Fam Pract. 2006;23:226–32.
- Rietveld RP, ter Riet G, Bindels PJE, Sloos JH, van Weert HCPM. Predicting bacterial cause in infectious conjunctivitis: Cohort study on informativeness of combinations of signs and symptoms. Br Med J. 2004;329:206–10.
- Jefferis J, Perera R, Everitt H, van Weert H, Rietveld R, Glasziou P, et al. Acute infective conjunctivitis in primary care: Who needs antibiotics? An individual patient data meta-analysis. Br J Gen Pract. 2011;61:e542–8.
- Altman DG, Royston P. What do we mean by validating a prognostic model? Statist Med. 2000;19:453–73.
- Bleeker SE, Moll HA, Steyerberg EW, Donders ART, Derksen- Lubsen G, Grobbee DE, et al. External validation is necessary in prediction research: A clinical example. J Clin Epidemiol. 2003; 56:826–32.
- Reilly BM, Evans AT. Translating clinical research into clinical practice: Impact of using prediction rules to make decisions. Ann Intern Med. 2006;144:201–9.
- Steckler A, McLeroy KR. The importance of external validity. Am J Public Health. 2008;98:9–10.
- Steyerberg EW, Bleeker SE, Moll HA, Grobbee DE, Moons KGM. Internal and external validation of predictive models: A simulation study of bias and precision in small samples. J Clin Epidemiol. 2003;56:441–7.
- Toll DB, Janssen KJM, Vergouwe Y, Moons KGM. Validation, updating and impact of clinical prediction rules: A review. J Clin Epidemiol. 2008;61:1085–94.
- Hall GS, Pezzlo M. Ocular cultures. In: Isenberg HD, editor. Clinical microbiology procedures handbook. Washington: American Society for Microbiology; 1995.
- Harrell FE Jr. Regression modeling strategies: With applications to linear models, logistic regression, and survival analysis. New York: Springer; 2001.
- Royston P, Sauerbrei W. Multivariable model-building. Chichester: John Wiley; 2008.
- Rietveld RP, ter Riet G, Bindels PJE, Bink D, Sloos JH, van Weert HCPM. The treatment of acute infectious conjunctivitis with fusic acid: A randomised controlled trial. Br J Gen Pract. 2005;55:924–30.