The effect of an automated integrated management of childhood illness guideline on the training of professional nurses in the Western Cape, South Africa

Abstract

Background: Reducing under-five mortality rates is a global priority. Although under-five mortality has decreased in South Africa, it is still unacceptably high. The implementation of the Integrated Management of Childhood Illness (IMCI) guideline is a key World Health Organization intervention aimed at reducing under-five mortality. Most primary care consultations are with professional nurses and they are usually trained in IMCI in a year-long primary healthcare diploma, which qualifies them to be a clinical nurse practitioner. This study aimed to evaluate the effect on training outcomes of introducing an automated IMCI guideline.

Methods: Thirty professional nurses enrolled for the diploma course were purposively allocated in a quasi-experimental design to train with either paper-based or automated versions of the guideline. Their knowledge of IMCI was evaluated before and after the initial 12 hours of classroom teaching. Data on assessment, classification and management of children was extracted from the medical records of their consultations during supervised clinical training.

Results: Both groups improved their knowledge of IMCI, but were not significantly different at the end of classroom teaching. Nurses in the automated group performed significantly better in use of the IMCI guideline (p < 0.05): checking immunisations (68% vs 93%), making a complete assessment (62% vs 100%), prescribing correct medication (50% vs 85%) and correct dose (42% vs 85%).

Conclusion: Use of automated IMCI guidelines showed potential for improved training outcomes. The potential for improved quality of care and clinical outcomes needs to be further studied along with a cost–benefit analysis.

Keywords:

• automated guidelines
• clinical nurse practitioners
• computer assisted learning
• integrated management of childhood illness
• primary care
• training

Introduction

The fourth World Health Organization (WHO) Millennium Development Goal (MDG) is for a global reduction by two-thirds in the under-five child mortality rate between 1990 and 2015. This gives a target of 20 deaths per 1000 live births by 2015.¹ Although there had been a 47% decline in the global under-five mortality rate by 2012; the current rate of 48 deaths per 1000 live births is still unacceptably high.¹

In 2009 the under-five mortality rate of 63 deaths per 1000 live births in South Africa was increasing and the burden of disease in children was 2–3 times higher than that of similar income countries.² By 2012 the rate had decreased to 45 deaths per 1000 live births, but South Africa is still likely to miss the 2015 MDG target.³ South Africa has therefore made reducing child mortality a national priority.⁴

Globally over 70% of under-five deaths occur within the first year of life, of which pneumonia, diarrhoeal disease, preterm complications and birth asphyxia are the major killers.⁵ In Africa HIV/AIDS and malaria are also important contributors to the burden of disease in young children. In South Africa under-five mortality is mostly due to acute respiratory infections, diarrhoeal diseases, malnutrition, and underlying HIV.⁶

In 1995 the WHO and the United Nations Children’s Fund introduced the Integrated Management of Childhood Illness (IMCI) guideline, an easy to follow algorithmic approach to common conditions, to help developing countries reduce the major causes of mortality in children under five years of age.⁷

IMCI is a standardised integrated approach to child health that focuses on the well-being of the whole child. IMCI aims to reduce death, illness and disability, and to promote improved growth and development among children under five years of age. The strategy includes three main components: improving case management skills of health-care staff, improving overall health systems, and improving family and community health practices.⁷

In South Africa the IMCI guideline is predominantly used to assist primary care nurses in making an assessment and initiating management without the assistance of a doctor. In terms of case management the IMCI guideline is a syndromic approach, where a limited number of carefully selected symptoms and signs are the entry point for management of the child. Implementation of the IMCI guideline has been shown to lead to improved quality of care, better case management and improved child health.^8–10

In SA, the Department of Health adapted the WHO’s generic version of the IMCI guideline to the South African context.¹¹ The South African version of the IMCI guideline, for children aged one week to five years, includes the management of diarrhoea, acute respiratory infections, asthma, ear infections, malnutrition, tuberculosis, HIV/AIDS, child abuse, meningitis and the critically ill child.

However, the introduction of IMCI in South Africa in 1997 has not been associated with a significant impact on child mortality.² Reduction in under-five mortality in South Africa has been credited to the success of the HIV programme in reducing HIV transmission and the expansion of the essential immunisations programme.¹² Reasons for the lack of impact of the IMCI programme have been ascribed to insufficient IMCI coverage, in terms of the percentage of primary care staff trained, and by a lack of adherence to the guideline recommendations after training.^10,13 Ineffective implementation of IMCI due to poor adherence to the guideline recommendations, with incorrect or incomplete assessments of the child, has been identified as a critical factor in the poor quality of care for the sick child.^8,10

Previous studies indicate that adherence to the IMCI guidelines decreases over time due to inadequate initial training, shortage of staff and insufficient supervision.⁸ Critical recommendations within the IMCI guideline may also be omitted due to lack of knowledge and skills.^13,14

WHO recommends an 11-day standard (paper-based) training course for teaching IMCI to nurses. The course describes the problem of childhood illness, the need for integrated case management, and trains nurses in use of the case management charts contained within the guideline. This usually includes eight days of classroom teaching and three days of supervised clinical practice.⁷

There is evidence that the outcomes of training from a shortened course, of 5–10 days duration, can be as good in terms of the performance of health workers post-training.¹⁵ There is also evidence that technology can assist with training in the context of limited time and a shortage of expert trainers. For example, in a Ugandan study, IMCI trainees who did a nine-day IMCI training course with a computer-based guideline, showed better adherence to the guideline over time compared to the trainees using the paper-based format.¹⁶ Training time can also be reduced with the use of technology, as the computerised version can model use of the guideline to the trainees and trainees can engage with simulated practice in their own time outside of formal classroom teaching.^13,16,17 Technology helps to shift the mode of education from theory to modelling and practice. The cost of technology must be evaluated in light of the potential for shortened training time, reduced need for trainers, and better achievement of outcomes.^13,18

In SA, primary care nurses are trained to assess and manage patients through a one-year long Diploma in Clinical Nursing Science; Health Assessment, Treatment and Care. Training in the IMCI guideline is just one of the components of the whole curriculum that must be covered. At Stellenbosch University the diploma programme allocates only 1.5 days of classroom-based training to IMCI. This is followed by a period of supervised clinical practice at training sites one-day a week for three months. During supervised practice, nurses would see a mix of adults and children and would be able to practise using the IMCI guidelines during consultations with children under five years. However, not all clinical trainers are trained in IMCI and therefore a lack of expert supervision is a critical issue.

This study explored the use of a new automated version of the IMCI guideline during the training of professional nurses who were registered for the diploma course at Stellenbosch University. The automated guideline, in the format of a computer tablet (see Figure 1), aimed to improve the outcomes of training in IMCI, given the limitations described above of teaching time, exposure to clinical practice and expert supervision. Such an automated IMCI guideline has not previously been evaluated in SA.

Figure 1: Nurse using the convertible classmate personalised computer tablet with stylus.

The specific objectives of this study were to evaluate nurses’ knowledge of the IMCI guidelines at the end of classroom-based teaching and nurses’ use of the IMCI guidelines during supervised clinical training practice at their training sites.

Methods

Study design and setting

This was a quasi-experimental study comparing groups of professional nurses trained with the paper-based version versus the automated version of the IMCI guideline.

The study population was drawn from 230 professional nurses who enrolled for a one-year long diploma programme in 2010. The students worked at primary healthcare facilities for the City of Cape Town and the Provincial Government of the Western Cape, as well as for hospitals in both the private and public sectors. All students attended part-time theoretical training sessions on a once a month basis at the university as well as training practices in public sector primary care facilities. The diploma is recognised by the South African Nursing Council and allows nurses to practise as clinical (family) nurse practitioners within their work settings.

Selection and sampling

A questionnaire was administered to the 230 nurses who enrolled for the diploma programme to determine who had not previously received training in IMCI. One hundred nurses without prior IMCI training were identified and were all invited to participate in the study. Thirty of these nurses consented to participate and were then allocated to teaching either with the paper-based or the automated guideline. Allocation to groups was based on the location of their intended training practice, so that all nurses at a given location used the same version of the guideline, and on the need to balance nurses from rural and urban sites in both groups.

Development of the automated guidelines

The study instrument consisted of the pre-programmed automated IMCI national treatment algorithms (initial visit) for children from two months up to five years. The electronic device was a CTL-2Go touch screen Convertible Classmate PC, where users had the choice of either using the touch screen or a keyboard. Information was encrypted for patient confidentiality. The software included all aspects of the IMCI initial assessment including the general assessments and treatment of danger signs, cough or difficult breathing, wheezing, diarrhoea, fever, ear problems, malnutrition, anaemia, HIV infection and TB, immunisation status, measles, treatment of all the IMCI classifications, feeding assessment, counselling, follow-up recommendations and a summary of the data. Software was developed by Expert-24, a UK company with expertise in automation of clinical guidelines. The automated computer-assisted guideline was designed to systematically guide the nurse through the IMCI, reducing errors and making it more effective. It provided a warning to take urgent action where necessary and at the end offered a summary of the patient’s classification, key features and recommended management. The software guided the practitioner to gather information, make an assessment and decide on management of the patient, and was developed to mirror the IMCI flowchart (patient case recording form) that is currently documented in the patient folder. Minimal text could be typed if the child for example displayed symptoms outside of the IMCI classification. In addition the software included links from the guideline to short explanatory text, graphics and definitions designed to assist the novice IMCI practitioner.

Training in IMCI guidelines

Training was given by the researcher, who is IMCI trained, at Stellenbosch University. Three four hour workshops were run over three non-consecutive days. The process of training was identical for both groups except for the use of either the automated or the paper-based version of the guideline during training.

The automated group also received training on how to use and look after the computer tablet itself, which was separate from the training on IMCI. Use of the keyboard was not a requirement, but an option, as training utilised the touch screen modality.

Following on from the 12 hours of classroom-based training, clinical training practice was completed at the allocated training sites over a 3-month period. Nurses attended the training sites for 1 day a week and saw patients under the supervision of a nurse trainer. The total number of hours spent on consulting children under the age of five years was estimated as 40 hours or the equivalent of five days.

Data collection

Knowledge of IMCI was assessed before and after training in both groups using a standardised test of 30 single answer multiple choice questions (one-point per question) based on a previously validated tool.¹⁹ The same multiple choice questions were used for both pre- and post-test by rearranging the order of the questions.

After the initial classroom training the medical records of all children, in the relevant age group, that were seen by the nurses during their clinical training practice were examined using a standardised data collection tool. Information was collected on the assessment, classification, and management. The paper-based nurses made their notes on a standardised medical record form provided by the health services. The nurses in the automated group saved data on the computer and could print out a summary to include in the medical record.

Data analysis

Quantitative data was analysed to compare the two groups in terms of their improvement in knowledge after the initial classroom-based teaching and in terms of their use of the IMCI guidelines during clinical training practice. Statistical significance was reported using a 95% confidence interval and a 5% significance level (p < 0.05). The M-L (maximum-likelihood) chi square and observed frequencies were used for analysis of categorical data.

Assessments were classified as correct if the nurse’s assessment was congruent with the clinical information recorded and evaluated by the researcher according to IMCI and incorrect if the clinical information implied a different classification. If there was insufficient information to determine whether the assessment was correct then it was classified as indeterminate. Assessments were classified as complete if all of the presenting symptoms were classified as well as the four compulsory assessments of nutritional status, anaemia, HIV status and TB that are required in all patients by the IMCI guideline.

Ethical approval

Ethical approval was granted by the Health Research Ethics Committee of Stellenbosch University (N09/12/349) and permissions from the relevant departments of health. All nurses gave written informed consent and their identity was protected in data collection and analysis by means of study codes.

Results

The 30 nurse participants ranged in age from 23 to 62 years with only one male nurse. The nurses came from different cultural backgrounds of which five were white, 18 coloured and seven black African, and geographic locations that ranged from urban Cape Town to rural Western Cape.

Prior computer skills

Only 16 (53.3%) nurses had used email, while 23 (76.6%) had typed a Word document in the previous year and 24 (80.0%) were used to using a mobile phone. However, of those who had typed a Word document, nine (30.0%) had done this less than 10 times in the previous year. Only two (6.6%) of the participants had no confidence to use a personal computer, 12 (40.0%) felt somewhat confident, seven (23.3%) felt confident and five (16.6%) felt very confident.

Knowledge test scores

Nurses trained with the paper-based guideline showed a mean percentage improvement of 8.5 (95%CI - 1.9–18.9) in their knowledge test score even though this is not statistically significant. However, six nurses in the paper-based group did not complete the test thus reducing the power to show a difference. Furthermore, when the results of the whole class using the paper-based guideline were analysed (N = 84) which showed a statistically a significant (p < 0.00) improvement of 10.7 (95%CI 8.4–13.0). In addition, the nurses using the automated guideline showed a significant (p = 0.02) percentage improvement of 8.0 (95%CI 1.2–14.8). Neither group differed from the other in the post-test percentage score (p = 0.729), with the paper-based group achieving 62.2 (53.5–70.9) and the automated group 64.5 (57.7–71.2).

Assessment of consultations

Eighty children were consulted by these nurses and included in the study; 30 in the automated group and 50 in the paper-based group. On average each nurse contributed three to four consultations to the study.

The main signs and symptoms identified during the clinical assessment in descending frequency were cough (46, 57.5%), fever (23, 28.8%), ear problems (16, 20.0%) and diarrhoea (10, 12.5%). During the initial assessment the nurse should identify the reason for the child’s visit, record the child’s age, measure the child’s weight and temperature and record these in the patient’s medical record. Only six children had a temperature above 37 °C. The child should then be assessed further for general danger signs (indications that the child needs urgent referral), malnutrition, anaemia, HIV risk, TB contacts and immunisation status. Assessment of feeding should be done if the child has anaemia, is not growing well or is below the age of two years.

Table 1 presents the comparative results for key steps in this assessment process.

Table 1: Gathering information: comparison of outcomes for paper-based and automated groups

Variable	Paper-based (n = 50)	Automated (n = 30)	p value
	n (%)	n (%)
Was the reason for encounter recorded?	47 (94.0)	28 (93.3)	0.905
Was the age recorded?	44 (88.0)	30 (100.0)	0.014
Was the weight recorded?	48 (96.0)	30 (100.0)	0.166
Growth monitoring recorded in terms of nutritional status and anaemia	45 (90.0)	30 (100.0)	0.026
Temperature taken and recorded	45 (90.0)	29 (96.7)	0.246
Previous HIV testing in child or mother recorded	25 (50.0)	28 (93.3)	<0.001
Any close TB contact recorded	22 (44.0)	28 (93.3)	<0.001
Immunisation status recorded according to the immunisation schedule and age of child	34 (68.0)	28 (93.3)	<0.001
General danger signs observed and recorded	47 (94.0)	30 (100.0)	0.089

The results suggest that there was no difference between the groups in terms of assessment related to the immediate clinical problem; however, the automated group performed significantly better in terms of more comprehensive information regarding growth, immunisations, HIV and TB risk.

Table 2 presents the results for the correctness and completeness of the assessments in the two groups. The automated group recorded sufficient information to make a correct assessment in almost all children. The paper-based group had insufficient information recorded in more than a third of children, which made it impossible to determine whether the actual assessment was correct. In addition, the automated group made significantly more complete assessments as the systematic algorithm does not allow one to miss steps, resulting in more adherent and comprehensive clinical management.

Table 2: Comparison of assessment outcomes for children seen, using paper-based and automated IMCI guidelines

Variable	Paper-based (n = 50)	Automated (n = 30)	p value
	n (%)	n (%)
Correctness of assessments
Correct assessments	26 (52.0)	29 (97.0)	<0.001
Incorrect assessments	7 (14.0)	1 (3.0)	0.178
Indeterminate assessments	19 (38.0)	0 (0.00)	<0.001
Completeness of assessments
Completeness	31 (62.0)	30 (100.0)	<0.001

Table 3 compares the two groups in terms of their management of the children. Results for each item are calculated from the number of consultations in which the item was relevant as per the IMCI guideline.

Table 3: Comparison of management plan for paper based and automated groups

	Paper based (n = 50)	Automated (n = 30)	p value
	n (%)	n (%)
Correct medication prescribed	13/26 (50.0)	11/13 (84.6)	0.029
Correct dose prescribed	11/26 (42.3)	11/13 (84.6)	0.005
Advice on feeding given and recorded	34/46 (73.9)	25/26 (96.2)	0.01
Safety netting advice given and recorded	39/50 (78.0)	23/28 (82.1)	0.037
Advice on when to return immediately recorded	33/50 (66.0)	20/28 (71.4)	0.018
Enquiry regarding mother’s health	42/50 (84.0)	23/28 (82.1)	0.594

In those where medication should have been prescribed, the automated group performed significantly better in terms of the correct medication and dose. In terms of appropriate feeding advice and safety netting, the automated group offered better management of the patient compared to the paper-based group. The automated group therefore appeared to offer a more comprehensive and safer consultation.

Discussion

Overall, the nurses who were trained with the automated guideline performed significantly better in their clinical practice in terms of their history-taking, assessment and management. Their consultations were more comprehensive and paid attention to important preventative issues such as growth monitoring, immunisations, and risk of HIV or TB.

This improvement in clinical practice did not appear to be due to differences in the success of classroom-based teaching, but to the support of clinical practice given by the automated IMCI guidelines at the clinical training sites. In addition to guiding the nurse through all components of the IMCI consultation, the automated version also included a database of resources and explanatory material for quick reference.

The study suggests that the use of the automated guideline may enable the shortened course as part of the diploma programme to achieve better outcomes, especially in the context of a lack of expertise to provide clinical supervision.

At the time this study was performed, the cost of the computer tablet was ZAR 3 700 (US $450) and this is likely to reduce with economies of scale. The software can also be incorporated into personal digital devices. Most studies choose PCs for durability, practicality, convenience and robustness at the point of care.^17,20 Paper-based training can be more expensive in terms of the duration of training required and the cost of more human resources to facilitate learning.²⁰ A cost–benefit analysis should be done in future studies to clarify these issues.

Another limitation of the automated version is that nurses are not fully computer literate.²¹ However, although a few nurses were technologically challenged, all managed to use the automated version adequately. The approximate time taken to complete an assessment was observed and nurses also reported that the automated version was quicker to use once the nurse had familiarised herself with the tablet.

This study has many limitations and should be regarded as evidence for the potential benefit of automated guidelines, which needs further testing and evaluation in our context. Only 30 out of the possible 100 nurses agreed to take part in the study. Of these nurses six dropped out of the initial paper-based training course and four dropped out of the automated group during the clinical practice phase. Twenty-three of the consultations performed by nurses in the automated group were also lost for technical reasons, which contributed to the uneven number of consultations between the two groups.

A reverse power analysis suggested that the study had 75% power to detect a significant difference between the groups. Under-powering may lead to differences being non-significant when they would have been significantly different in the target population. However, despite the small number of nurses included in the study, the results demonstrated a number of statistically and clinically important differences, which are consistent with qualitative feedback²² and other studies.^13–16,19

Care must also be taken in generalising the results more broadly due to the non-random sampling technique and potential for unknown confounding factors to influence the results, which is always a concern with quasi-experimental studies.

The potential benefits of automated guidelines should be further established in a larger study with a more rigorous randomised study design. The ability of the automated version to maintain fidelity to the IMCI guidelines while working in clinical practice, when compared to usual levels of support, should also be evaluated. The best technological interface for the software must also be considered as newer devices, such as smart phones or tablets, may be more cost-effective than the devices used in this study. Future studies should include cost-benefit analyses.

Conclusion

The knowledge of IMCI by nurses trained with the automated guidelines was as good as those trained with the paper-based version at the end of the classroom-based course. During clinical training, use of the IMCI guidelines was significantly better in the automated group in terms of gathering information, assessment and management. The potential for improved quality of care and clinical outcomes needs to be further studied along with a cost-benefit analysis.

Competing interests

The hardware and software were provided by Expert-24 Limited (Ltd) World Doctor Project, which is a UK decision support software company. I declare that I have no financial or personal relationship that may have inappropriately influenced me in writing this article.

Acknowledgements

The author acknowledges the Division of Nursing at Stellenbosch University which allowed the study to be conducted in the diploma programme. Justin Harvey from the Centre for Statistical Consultation assisted with the analysis. The Provincial Government of the Western Cape (Department of Health) and the Director of Health for the City of Cape Town gave permission for the participants to conduct the study in the community health centres and clinics. I also wish to thank the study participants who gave of their time for the interviews and willingness to be part of the study.