Quality of implementation of a school mental health initiative and changes over time in students’ social and emotional competencies

Abstract

This paper reports the theoretical conceptualisation, statistical development, and application of an Implementation Index to evaluate the quality of implementation of the KidsMatter Primary school mental health initiative in Australia. Questionnaires were received from the parents and teachers of almost 5000 students, and also from KidsMatter project officers. A conceptual framework of fidelity, dosage, and delivery guided the selection of questionnaire items to create the Implementation Index, which was refined using Latent Class Analysis. Schools’ scores on the Index were classified into high, average, and low implementation categories. Profiles of high- and low-implementing schools provided insights into the characteristics of successful and less successful implementation. Next, hierarchical linear modelling showed that children's social and emotional competencies significantly improved over time in average- and high-implementing schools, but not in low-implementing schools. The Implementation Index can inform areas for attention in health promotion initiatives and can provide a framework for future evaluations.

Keywords:

• mental health promotion
• evaluation
• implementation quality
• latent class analysis
• hierarchical linear modelling

Acknowledgements

1: The authors wish to thank Professor John P. Keeves AM for his guidance with the Latent Class Analysis and Hierarchical Linear Modelling reported in this paper.

2: KidsMatter is an Australian national primary school mental health promotion, prevention and early intervention initiative. KidsMatter was developed in collaboration with the Australian Government Department of Health and Ageing, beyondblue: the national depression initiative, the Australian Psychological Society, and Principals Australia, and was supported by the Australian Rotary Health Research Fund. The authors gratefully acknowledge permission from beyondblue: the national depression initiative to use the data in this paper.

Notes

1.  For ease of reference, the term “parents” is used in this paper to refer to parents and other primary caregivers of children.

2.  KidsMatter was undertaken in 101 schools, but one hospital-based school did not participate in the evaluation due to the challenges of collecting longitudinal questionnaire-based data from the school's transient students.

3.  More detail about the aims and processes of KidsMatter can be obtained from the KidsMatter website, www.kidsmatter.edu.au

4.  Full details of item selection, verification, and factor analysis can be found in Dix et al. (2010).

5.  A few schools contained less than 76 students.

6.  The display in Figure 6 suggests that students in high-implementing schools started with higher SEC than students in low-implementing schools. However, an extension to the HLM rejected INDEX as having a significant influence on the intercept (p > .05).