Positive education in daily teaching, the promotion of wellbeing, and engagement in a whole school approach: a clustered quasi-experimental trial

ABSTRACT

In the current study, a Dutch whole school Positive Education Programme (PEP) was investigated. PEP is a bottom-up programme, aimed at shifting teacher’s attention from solely focusing on learning outcomes towards a more comprehensive approach that takes the wellbeing and engagement of students during classes into account. PEP was investigated through a clustered quasi-experimental trial. Four primary schools, 639 pupils, were included, of which two were allocated to PEP. Multilevel analyses showed no significant differences on any of the outcomes. However, the data indicated a trend towards more engagement in the intervention schools (p = .09). Also, at post-measurement, the proportion of sufficiently engaged students was significantly higher in the intervention schools. Although it seems too premature to draw conclusions regarding the effectiveness of such a whole school approach, the findings on engagement are promising and have the potential to engender a wide range of other beneficial outcomes.

KEYWORDS:

• Wellbeing
• engagement
• whole school approach
• positive psychology
• primary school

Introduction

Wellbeing and engagement

There is a growing interest in applying positive psychology elements in education (World Government Summit & the International Positive Education Network [IPEN], 2017). In 2000, Seligman and Csikszentmihaly (2000) presented positive psychology as the science of wellbeing and optimal functioning. Positive psychology in education, or positive education, is concerned with the development of pupils’ strengths and wellbeing, thereby getting them to flourish and enabling them to achieve academic goals. Students are flourishing when they experience high levels of emotional, psychological, and social wellbeing (Keyes, 2002). Positive education is addressed through the implementation of positive psychology interventions (PPI’s) aimed at cultivating positive feelings, positive behaviour, or positive cognitions in school settings (Norrish & Vella-Brodrick, 2009; Seligman et al., 2009; Sin & Lyubomirsky, 2009). A systematic review of 12 international school-based PPI’s indicated that those interventions had significant positive effects on students’ wellbeing, social relationships, and academic performances (Waters, 2011).

In addition to wellbeing, one vital focus in positive education is engagement. Engagement means that there is intense mental activity, that a pupil is functioning at the limits of their capabilities and with an energy flow that comes from intrinsic sources (Laevers, 1999). When pupils experience a high level of wellbeing and engagement, their learning curve and learning outcomes will be optimal (Laevers, 1999). Previous research showed that pupils’ levels of wellbeing and engagement can be linked to academic achievement (e.g., Hughes & Kwok, 2007; Pietarinen et al., 2014; Reyes et al., 2012), social-emotional learning (Buhs et al., 2006; Hosan & Hoglund, 2017), and mental problems including anxiety and depressive symptoms (Fredrickson, 2001).

It has been discussed that wellbeing and engagement should not only be seen as outcome measures, but should be used as daily guidelines for teaching. Instead of adjusting the teachers’ approach based on the learning outcomes, the approach should be altered based on the indicators of the learning process: wellbeing and engagement (Laevers, 1999). This theory stems from the Centre for Experience Based Learning (CEGO) in Leuven. This requires teachers to adopt a change in perspective. They should not be merely focusing on learning outcomes (e.g., grades), but rather be actively looking for ways to engage students and to get students to flourish (based on their interests and strengths). A number of qualitative studies showed the potential of this process-oriented teaching approach by describing its positive influence on wellbeing and academic achievement (Buyse et al., 2009; Laevers & Heylen, 2013). In the current study, a Dutch positive education programme focusing on the perspective change in line with the theory by CEGO is investigated in a clustered controlled trial.

Whole school approach

To date, the implementation of positive psychology programmes in schools primarily takes place at the level of the classroom, with relatively short curriculum-based interventions (Waters, 2011). However, research on social and emotional learning interventions suggests that interventions are likely to be more successful when (a) integrated into the school culture and daily school practices, (b) engaging all staff, (c) reinforcing social skills outside the classroom, (d) supporting parental engagement, and (e) supporting community engagement such as sport associations or local companies (Adi et al., 2007; Barry et al., 2017; Jones & Bouffard, 2012; Ttofi & Farrington, 2011; Weare & Nind, 2011; Wilson et al., 2003). Altogether, these characteristics point towards the potential of adopting a whole school approach for enhancing youth’s wellbeing and engagement.

A whole school approach defines the entire school community as the unit of change and aims to integrate skill development into daily interactions and practices using collaborative efforts that include principals, teachers, other staff, parents, and pupils (Jones & Bouffard, 2012; Meyers et al., 2015). Based on the World Health Organization’s (WHO) Health Promoting Schools Framework (1998), a whole school approach involves coordinated action between three components: (a) curriculum, teaching, and learning; (b) school ethos and environment; and (c) family and community partnerships. Effective curriculum teaching and learning involves teaching skills through the implementation of evidence-based programmes as well as modelling social-emotional competencies and providing continuous and consistent opportunities to practise these skills during everyday classroom situations (Oberle et al., 2016). At the school level, skills are reinforced in non-curriculum-based ways through policies, whole staff training, organisational structure, and daily activities in the school that are designed to promote a positive school climate which, in turn, helps youth to develop across academic, social, emotional, and behavioural domains (Jones & Bouffard, 2012; Meyers et al., 2015). Through involving not only school staff but also parents and communities in a whole school approach, learning is not limited to the school setting, but can also occur in the home setting.

A recent meta-analysis (Goldberg et al., 2019) of 45 studies evaluating the impact of social and emotional learning interventions which adopted a whole school approach reported that these interventions have a significant positive impact on behavioural adjustment (d = 0.13), social-emotional adjustment (d = 0.22), and internalising symptoms (d = 0.11). However, not all whole school approaches report positive effects. The evaluation of social and emotional aspects of learning (SEAL) indicated no significant outcomes in terms of social and emotional skills, mental health difficulties, and behaviour (Humphrey et al., 2010). The International Positive Education Network provides three international case studies of positive education applied via a whole school approach (World Government Summit & IPEN, 2017): Geelong Grammar School in Australia (Hoare et al., 2017); St. Peter’s College in Australia (White & Waters, 2015); and Universidad Tecmilenio in Mexico (Escamilla, 2017). These schools consider wellbeing to be equally important as academic achievement, ensuring that all staff sees the improvement of wellbeing as their main responsibility (Hoare et al., 2017; White & Waters, 2015). By implementing positive education as a whole school approach, a positive climate change has occurred at all three schools (World Government Summit & IPEN, 2017). The findings from the case studies report positive results; however, these findings have yet to be confirmed by more robust evidence.

Dutch Positive Education Programme

At the core of this study is the Dutch Positive Education Programme (in Dutch: Positief Educatief Programma; PEP). The key aim of PEP is to promote a shift in perspective among teaching staff, from a teaching approach solely based on learning outcome indicators towards a teaching approach informed by the learning process indicators wellbeing and engagement. To enable and foster this shift in perspective, schools participating in PEP implement activities at the whole school level based on the shared values of their team members.

PEP draws upon the principles of positive education and is derived from the applied model for positive education of Norrish et al. (2013). The applied model for positive education aims to increase flourishing by living, teaching, and embedding the principles of positive psychology (Norrish et al., 2013). During PEP, this is done by making the shared values of the team members a central theme. School teams identify their shared values and create (approximately five) life rules to give expression to these shared values. A school culture that reflects shared values can positively affect learning outcomes (MacNeil et al., 2009). In the 13 schools that currently have adopted the PEP approach in the Netherlands, a wide range of values are addressed through life rules (e.g., respect, safeness, kindness, humour). These life rules are expressed in the curriculum, in the physical school environment, and in the interaction with parents and amongst team members. By implementing activities in the lessons and school culture that are closely associated with the values of the teachers, the process indicators “wellbeing” and “engagement” are strengthened. If the process indicators appear to be insufficient, teachers change their approach. With high levels of wellbeing and engagement, behavioural problems are expected to be minimalised and social-emotional development is expected to be optimised.

A process evaluation (Elfrink et al., 2017) evaluating the implementation of PEP in two pilot primary schools in the Netherlands showed promising effects. Favourable results were identified across pupil and teacher outcomes including pupils’ self-reported wellbeing and problem behaviour (e.g., bullying), teachers’ awareness of pupils’ strengths, and the overall school climate. In addition, teachers showed positive attitudes towards the philosophy of PEP and its key components. Teachers reported that they were more aware of the wellbeing and engagement of their pupils as a result of PEP. Also, they appreciated the fact that PEP’s bottom-up approach starts from existing values of the team members. Goldberg et al. (2018) demonstrated similar findings in a second process evaluation of an adapted version of PEP conducted in two schools in special needs education. In this evaluation, teachers reported feeling more confident to focus on the process of learning instead of focusing on learning outcomes. Although PEP as a Dutch whole school approach proved acceptable and feasible in these pilot studies, there is a need for more robust research to determine the impact of PEP.

Current study

Despite emerging evidence for the effectiveness of whole school approaches in the social-emotional learning domain (Goldberg et al., 2019), to date little is known about whole school approaches that focus on the daily enhancement of wellbeing and engagement of pupils in primary schools. It is expected, however, that such interventions lead to happier students, less problematic behaviour, and a better connection with the school and its teachers. This study therefore sought to contribute towards closing this knowledge gap through evaluating the effects of PEP in a quasi-experimental clustered trial in Dutch primary education. Wellbeing and engagement are considered to be the main working mechanisms in PEP and are measured as proximal outcomes in the current evaluation. As distal outcomes, the current study aimed to assess the impact of PEP on bullying behaviour among students, as well as students’ emotional and behavioural problems.

Methods

Ethical approval was obtained from the Ethical Review Board of the University of Twente (Document No. BCE17604). All participants were treated in accordance with ethical standards of the American Psychological Association and were ensured anonymity, with no identification of individual responses.

Study design

This study employed a clustered quasi-experimental design with one intervention group and one control group. Both conditions consisted of two primary schools. Randomisation was not possible, since two of the included schools were only willing to participate provided that they would be allocated to the control condition. The PEP schools implemented the intervention from September 2017 to June 2018. The control schools were instructed to complete their school year as they would normally do, without implementing any positive psychology programmes in this time frame. Assessments took place before the start of PEP (i.e., baseline, September 2017) and at the end of the school year (i.e., post-measurement, June 2018).

Sample size calculation

Prior to this study, a power analysis was conducted so as to determine the required sample size, aiming to detect small significant effect sizes for the primary outcome wellbeing (d = 0.20). The alpha was set to .05, the power to .80 (1 − beta .20), and the correlation between pre- and post-measurement to .50. This resulted in a required sample size of 591 participants (Borm et al., 2007).

Procedure and participants

The four participating schools were recruited in the spring of 2017. The schools were part of a district-wide organisation (Stichting Consent, Enschede, the Netherlands) consisting of 32 primary schools. Of these 32 schools, five schools were already working with PEP and were, therefore, not eligible to participate in the present study. An information letter about the present study was spread across the 27 remaining schools, of which six schools communicated their interest in the programme. For each of these six schools, two meetings were organised to provide information about PEP and the study. Two out of the six schools appeared not eligible to participate because they were already implementing a whole school approach aimed at positive behavioural support. The remaining four schools were included in the trial. As all included schools were part of the same organisation and were located in the same city, we perceived their demographic factors to be similar.

In total, 639 pupils from four different primary schools in Enschede were enrolled in the present study, of whom 401 pupils (two schools) were allocated to the control condition and 238 pupils (two schools) to the intervention condition. Informed consent was obtained from their parents or guardians prior to including the pupils in the study. Additionally, informed consent was sought from the school principals and teachers. Informed consent was given passively, meaning that parents/guardians, principals, and teachers gave automatic consent unless they stated otherwise. They received an information leaflet with instructions on how to withdraw consent. The information letter also provided information on the study, including information about its purpose, the type of data that would be collected, and how the data would be utilised. Five percent of the pupils (N = 32) were not included in the research as their parents or guardians did not provide consent. All teachers and principals provided consent.

Data were collected from all school classes (i.e., Grades 1–8, ages 4–12), with on average 21 pupils per class. The mean age of the participants was 7.71 (SD = 2.34), with 50% boys and 50% girls. All four schools were located in the same city. At baseline, the PEP schools and control schools did not differ significantly on any of the outcome measures, with p values ranging from .08 (pro-social behaviour) to .78 (victimisation).

Intervention: Positive Education Programme (PEP)

PEP is a whole school approach based on positive education. Key components of PEP are the daily enhancement of pupil’s wellbeing and engagement and the shared values of the team members. The goal of PEP is to promote change in perspective among teaching staff, from a teaching approach based on learning outcome indicators towards a teaching approach informed by the learning process indicators wellbeing and engagement.

Implementation activities

At the start of the school year, both intervention schools selected three to five teachers to form a project team to arrange all practicalities. The entire school staff received training in the form of two workshops. The first workshop took place in September 2017 and concerned the observation and promotion of children’s wellbeing. In November 2017, the second workshop took place, which focused on values and life rules. Additionally, both intervention schools were visited on a monthly basis by the researcher (first author) and a PEP trainer, to participate in a PEP talk. During these PEP talks, the observation and enhancement of wellbeing and engagement as well as the activities around the values and life rules were discussed. The PEP trainer provided guidance to both intervention schools during the entire school year, emphasising the whole school approach of PEP. Schools were encouraged to implement activities to engage parents, to make PEP visible in the entire school, and to update their policy based on their shared values. Except for the two workshops and the PEP talks, the study was implemented using a bottom-up approach, without a strict manual. The implementation activities are listed in Table 1.

Table 1. Implementation checklist intervention schools.

Implementation activities
Establish a project team concerned with the practicalities of PEP
Implement a workshop on the topic of wellbeing and engagement
Implement a workshop on the topic of values
Implement monthly PEP talks with the entire team
All teachers conduct five engagement scans during the school year
Implement strategies to enhance wellbeing and engagement at the classroom level
Develop at least one life rule during the school year
Organise activities in the context of a life rule
Inform parents about PEP via a parent letter
Make PEP part of the yearly parent conversations
Make PEP visible in the school and on the playground
Change or update the policy of the school

Wellbeing and engagement as process indicators

Within the PEP programme, the focus was on the learning process rather than on learning outcomes. Wellbeing and engagement are considered two major indicators of the learning process, providing a daily direction for teaching. During the implementation of PEP, teachers were encouraged to be constantly aware of the degree of engagement and wellbeing amongst their pupils. On five pre-determined occasions during the school year, teachers gave observational scores based on the degree of engagement the pupils showed at that specific moment. These observations were not seen as outcome measures but were solely intended to provide the teacher with information about the current level of engagement amongst their pupils. These methods followed the procedures as described by the CEGO (Leavers & Laurijssen, 2001; Appendix 1). The engagement score was given on a 5-point scale, based on a 2-minute observation. Teachers observed all their pupils on these five occasions and discussed their observations with their colleagues. Teachers were provided with a student tracking system, which offered possible strategies and interventions at the individual level as well as at the classroom level, with the overall aim to increase the wellbeing and engagement of the pupils. An example of such an implemented intervention was the placement of a suggestion box in the classroom in which pupils can post ideas for activities or lessons. Through observing their pupils on a regular basis, teachers were aware of the wellbeing and engagement of their pupils.

Value clarification and implementation

During the second workshop, schools identified their shared values and cooperatively created life rules to give expression to their shared values. The created life rules were applied in four different domains, leading to a whole school approach in line with the definition of the WHO (1998): (1) in the curriculum; (2) in the physical environment of the schools (e.g., hallways); (3) in the communication with the parents; and (4) in the communication amongst the team members. These positively formulated life rules gave teachers and students a direction for actions but specifically did not provide strict behavioural rules. Every 4 to 6 months, the schools focused on one of their life rules. Both intervention schools implemented two life rules in the context of the current study (see Table 2 for an overview). The life rule “no strength remains unknown”, for example, was created based on the value “personal growth” and stimulates pupils, teachers, and parents to explore and make use of their strengths. Teachers tried out multiple activities that corresponded with the school’s life rules, including lessons in finding your strengths, organising a talent market for the entire school community (e.g., parents, guardians, neighbours, local companies), and displaying talents of pupils, teachers, and parents throughout the school (e.g., posters, drawings, banners).

Table 2. Shared values and formulated life rules of both intervention schools.

School	Value	Life rule
School A	Personal growth	No talent remains unknown
	Humour	Don’t worry be happy
	Passion
	Safety
School B	Respect	You, I and We, everybody is welcome here
	Fun	School is a party and you are invited
	Talent enhancement
	Safety
	Authenticity

Measures

Proximal outcomes

Wellbeing was assessed with the student self-reported KINDL-R questionnaire (Ravens-Sieberer & Bullinger, 1998). The KINDL-R was designed to measure six dimensions of youth’s health-related quality of life, including: physical wellbeing, emotional wellbeing, self-esteem, quality of relationships with family, quality of relationships with friends, and everyday functioning in schools. Items were scored on a 5-point Likert scale ranging from 1 (never) to 5 (always), with high scores reflecting greater levels of wellbeing. We administered two versions of the KINDL-R. Pupils aged 6 years or younger completed the 12-item Kiddy KINDL-R together with their teacher, whereas pupils aged 7 years and above individually completed the 24-item Kid KINDL-R. Previous research indicates that the KINDL-R is a reliable instrument that in the past yielded valid data (Ravens-Sieberer & Bullinger, 1998). In the current study, the reliability of the Kiddy KINDL-R total score as well as its subscales was insufficient (alphas ranging from .37 to .53), and therefore the Kiddy KINDL-R was excluded from the analyses. The internal consistency of the Kid KINDL-R was good with Cronbach’s alphas (α) of .82 and .81 at baseline and post-measurement, respectively. For the subscales (4 items each), only the subscale Self-esteem was found reliable with alphas of .74 and .70 for pre and post, respectively. The remaining subscales showed insufficient reliability (alphas ranging from .55 to .62). Hence, only total scores and self-esteem scores were analysed for the Kid version of the KINDL-R.

Engagement was measured through observations done by an independent researcher of the CEGO, using the Leuven Involvement Scale (LIS) developed by the respective organisation (Laevers et al., 1994). The CEGO researcher, who was blinded to the condition of each school (i.e., intervention or control), observed a random selection of approximately 10 pupils of Grades 1, 3, 5, and 7 per school (approximately 40 pupils per school). The selected pupils were observed twice per measurement (four times in total), whereby each student received a score between 1 and 5 points (Appendix 1), with high scores reflecting greater levels of engagement. Previous research demonstrated high interrater reliability of the LIS with a kappa of .86 (Laevers et al., 1994). Laevers and colleagues (1994) considered scores of 3.5 and higher as adequate. This cut-off point was used in this research as well. Pupils who scored ≥ 3.5 were therefore considered engaged, whereas pupils with a score < 3.5 were labelled as non-engaged.

Distal Outcomes

Pupils’ social-emotional and behavioural functioning was examined through the 25-item teacher-report Strengths and Difficulties Questionnaire (SDQ; Goodman, 2001). This questionnaire comprises five subscales with five items each, namely: emotional symptoms, conduct problems, hyperactivity, peer-relationship problems, and prosocial behaviour. Items deployed a 3-point Likert scale with answers ranging from 1 (not true) to 3 (absolutely true). Higher scores reflected greater levels of difficulties, except for the subscale prosocial behaviour. SDQ scores yielded valid data in previous research, attesting to the feasibility of the SDQ as a screening instrument (Goodman, 2001). In the current study, Cronbach’s α for the total scale was .42 at pre- and .38 at post-measurement, and, consequently, total scores were not used in the analyses. The subscales “emotional problems” (Pre α = .74, Post α = .74) and “prosocial behaviour” (Pre α = .81, Post α = .76), but not the remaining subscales, reached acceptable reliability and were included in the analyses.

The student–teacher relationship was examined with the 28-item teacher-report student–teacher relation questionnaire (in Dutch: Leerkracht-Leerling Relatie Vragenlijst; LLRV) of Koomen et al. (2007). The LLRV examines the student–teacher relationship in terms of conflict (11 items), closeness (11 items), and dependency (6 items). Items were rated on a 5-point Likert scale with answers ranging from 1 (certainly not applicable) to 5 (certainly applicable). High scores on closeness reflected a better student–teacher relationship, whilst for conflict and dependence it was the other way around. Criterion validity was assessed to be good in earlier research (Koomen et al., 2007). In the current study, both the total scale and the subscales showed acceptable to good reliability with alphas ranging from .73 to .88.

Among pupils in Grades 4 to 8, bullying and victimisation were assessed with one question each, filled in by the students. These questions were derived from the Olweus scale (Olweus, 1996). The first question asks how frequent the student has bullied others, and the second question asks how frequent the student has been bullied by others. Both items were answered on a 5-point Likert scale from 1 (never) to 5 (once per day or more), with higher scores reflecting greater levels of bullying or victimisation.

Implementation fidelity

To measure the implementation fidelity of PEP, an implementation checklist was filled in by the researchers for both intervention schools. This checklist consists of 12 actions determined by the researchers (all authors) prior to the start of the intervention (see Table 1).

Analyses

Attrition analyses were conducted to examine any baseline differences between pupils retained and pupils lost at post-measurement. Independent samples t tests were conducted to determine whether dropouts significantly differed from non-dropouts at baseline. Analyses were conducted according to the intention-to-treat (ITT) principle, using the expectation-maximization (EM) algorithm (Dempster et al., 1977; El-Masri & Fox-Wsylyshyn, 2005) to replace missing values on continuous variables at pre- (4.5%) and post- (5.5%) intervention.

Multilevel modelling procedures were performed in order to assess the impact of PEP on proximal and distal outcomes. The influence of clustering of the data was hereby considered, by assuming that the pupils were nested in grades and the grades were nested in schools. Clustered data violate the assumption of independent observations; hence, if the clustering is ignored, this may result in incorrect statistical inferences (Snijders & Bosker, 1999). Schools and grades were therefore included as random effects, condition as fixed effect, and condition as random slope (Field et al., 2012). To compare the proportion of pupils who conformed to the norm of engagement (i.e., LIS score ≥ 3.5; Leavers et al., 1994) across conditions at post-measurement, a chi-squared test of independence was performed and odds ratios (ORs) were calculated.

Additionally, to gain insight into differences on all outcome measures between the two intervention schools, independent samples t tests were conducted to compare pre-post change scores for both groups on each outcome. Multilevel procedures could not be performed for these additional analyses, because convergence was not reached. In interpreting the findings, we consider an alpha level of p < .05 as significant, yet effects with p values up to .10 are indicated as trends (Hoglund et al., 2012). The multilevel analyses were conducted in R-studio; all other analyses were conducted in IBM SPSS Statistics Version 25.

Results

Dropout

For each measurement, the proportion of dropouts was determined. Due to practicalities, it was not possible to observe the level of engagement in two of the classes at post-intervention. This concerned one school class in Grade 3 of the intervention condition and one school class in Grade 1 of the control condition. As a consequence, a high percentage of dropout was observed on that particular measurement (Table 3). The engagement scores of the dropouts were significantly higher at baseline than the scores of the non-dropouts. However, for none of the baseline measurements, a significant difference was found between dropouts in the PEP condition and dropouts in the control condition.

Table 3. Dropouts per measurement.

Outcome (Measure)	N consented to start	N filled in Pre	N filled in Pre and Post	N dropout (%)
Wellbeing & Self Esteem (Kid-KINDL-R)	399^a	319	305	14 (4%)
Engagement (observation)	173^b	173	132	41 (23%)
Emotional problems & Prosocial behaviour (SDQ)	639	605	576	29 (5%)
Student–teacher Relationship, Conflict, Closeness & Dependence (LLRV)	639	621	600	21 (3%)
Bullying & Victimisation (Olweus)	399^a	282	275	7 (2%)

^a Only completed by pupils in Grades 4–8; ^bSamples of 10 pupils from Grades 1, 3, 5, and 7 were scanned per school.

Intervention effects

In Table 4, for each outcome, means and SDs are presented per condition and per assessment. The multilevel analysis indicated no significant difference between the intervention schools and the control schools on any of the outcomes (Table 5). Regarding engagement scores, no significant difference was found between the groups, but the data indicate a trend in favour of the PEP schools (β = −0.53; p = .09). The proportion of engaged pupils in the PEP group slightly (non-significantly) increased between baseline and post-measurement from 48% to 52%, whilst the percentage of engaged pupils in the control group significantly decreased (X² = 8.64, p < .01) over the same time interval with 41% engaged pupils at baseline and 18% engaged pupils at post-measurement. This difference between PEP schools and control schools was found to be significant (X² = 8.48, p < .01). The odds ratio at post-measurement was 4.97, (95% confidence interval [CI] = 2.25, 10.98), meaning that pupils who followed PEP were nearly 5 times more likely to be engaged by the end of the school year compared to pupils who were in the control condition. The found odds ratio is comparable to a Cohen’s d of 0.88, indicating a large effect (Sánchez-Meca et al., 2003).

Table 4. Means and standard deviations for PEP schools and control schools.

	PEP schools			Control schools
Outcome	N	Pre M (SD)	Post M (SD)	N	Pre M (SD)	Post M (SD)
Wellbeing	143	3.93 (0.51)	4.05 (0.46)	162	3.86 (0.47)	3.90 (0.42)
Self-esteem	143	3.95 (0.84)	4.08 (0.70)	162	3.88 (0.74)	3.92 (0.70)
Engagement	65	3.21 (0.62)	3.32 (0.69)	68	3.10 (0.73)	2.68 (0.86)
Emotional problems	207	0.31 (0.41)	0.27 (0.39)	369	0.27 (0.34)	0.25 (0.35)
Prosocial behaviour	207	1.61 (0.46)	1.61 (0.45)	369	1.54 (0.42)	1.66 (0.38)
Student–teacher relationship	223	10.60 (1.40)	10.91 (1.65)	377	10.50 (1.45)	11.06 (1.34)
Conflict	223	1.42 (0.63)	1.49 (0.76)	377	1.53 (0.76)	1.40 (0.60)
Closeness	223	3.77 (0.57)	4.16 (0.63)	377	3.81 (0.52)	4.18 (0.58)
Dependence	223	1.76 (0.77)	1.77 (0.80)	377	1.78 (0.74)	1.72 (0.78)
Bullying	126	1.40 (0.91)	1.34 (0.81)	149	1.29 (0.80)	1.36 (0.81)
Victimisation	126	1.79 (1.26)	1.89 (1.35)	149	1.84 (1.30)	1.96 (1.33)

Note: N = number of respondents; M = mean score; SD = standard deviation.

Table 5. Intervention effects per outcome.

Outcome	N	AIC	BIC	LogLik	β	SE	p value
Wellbeing	305	2333.45	2366.99	−1157.72	−1.76	1.91	.454
Self-esteem	305	1644.24	1677.78	−813.12	−0.43	0.41	.410
Engagement	132	352.15	378.10	−167.08	−0.53	0.17	.090
Emotional problems	576	2253.10	2292.30	−1117.55	0.11	0.24	.688
Prosocial behaviour	576	2452.54	2491.74	−1217.27	0.55	0.29	.199
Student–teacher relationship	600	4673.42	4712.99	−2327.71	2.73	2.56	.398
Conflict	600	3955.89	3995.46	−1968.95	−2.42	1.30	.203
Closeness	600	3857.90	3887.48	−1914.95	−0.31	1.11	.807
Dependence	600	3433.17	3472.74	−1707.59	−0.62	0.77	.509
Bullying	275	785.10	817.65	−383.55	0.14	0.16	.492
Victimisation	275	989.70	1022.25	−485.85	−0.02	0.33	.955

Note: N = number of respondents; AIC = Akaike information criterion; BIC = Bayesian information criterion; LogLik = log likelihood; β = slope; SE = standard error.

When comparing both intervention schools with regard to teacher–student relationship scores, a significant difference was found in conflict scores (t(221) = 2.06, p = .04). School A remained stable in conflict (M = −0.01; SD = 0.05), whilst School B increased significantly in conflict (M = 0.18; SD = 0.07; t(95) = 5.83, p = .02). Additionally, both School A (M = 0.29; SD = 0.05; t(126) = 8.26, p < .01) and School B (M = 0.47; SD = 0.06; t(95) = 5.83, p < .01) improved significantly in change scores for closeness, though this improvement was significantly higher in School A compared to School B (t(221) = −2.27, p = .02). For the remaining outcomes, no significant differences between the PEP schools were found.

Implementation of PEP (fidelity checklist)

To assess the implementation fidelity of PEP, a list of 12 implementation activities was checked for the two intervention schools (Table 1). Both intervention schools did not update their policy based on the shared values identified during PEP. Also, both schools did not complete all five engagement scans over the course of the year. School A completed four scans, and School B completed three scans. Additionally, School A organised three more PEP talks than School B (respectively 10 and seven). Both schools reported significant changes in the school’s management during the year. In School A, the principal was replaced at the beginning of the school year, whilst in School B the principal was replaced halfway through the school year. Additionally, School B received a negative review of the national school inspection with regard to the quality of their education. The teachers explained in the PEP talks that this review led to a setback in their motivation for implementing PEP. These contextual difficulties led to School B not deploying any PEP-related activities in the two months before post-measurement.

Discussion

There is accumulating evidence regarding the impact of positive education on the wellbeing of youth (Waters, 2011). Fundamental in positive education is the focus on wellbeing and engagement. The key aim of the Dutch Positive Education Programme (PEP) is therefore to promote a shift in perspective among teaching staff, from a teaching approach solely based on learning outcome indicators towards a teaching approach informed by the learning process indicators wellbeing and engagement. To enable and foster this change in perspective, schools applying PEP implement activities at the whole school level based on the shared values of their team members. To date, there is a lack of empirical evidence regarding the effectiveness of interventions focusing on increasing engagement and wellbeing of children during regular classes. Increasing these concepts has the potential to lead to initiating a series of beneficial outcomes, for example, increased academic results, less behavioural problems, and less internalising problems.

Results from this cluster quasi-experimental trial indicated no significant effects on the proximal outcomes (wellbeing and engagement) or on the distal outcomes (social-emotional behaviour, student–teacher relationship, bullying and victimisation). Despite the lack of significant results, a positive trend was detected for pupil engagement. Also, the number of engaged pupils increased slightly in the intervention group and decreased largely in the control group. In the current study, the odds that pupils would be sufficiently engaged by the end of the school year was 5 times higher in the PEP schools compared to the control schools. The non-significant findings from this study are in contrast to most previous research which has shown the significant positive impact of whole school interventions on wellbeing, behavioural adjustment, and social-emotional adjustment (e.g., Barry et al., 2017; Goldberg et al., 2019). The findings are in line with a national evaluation of the whole school approach SEAL (Social and Emotional Aspects of Learning) that did not detect any significant impact outcomes in terms of social and emotional skills, mental health difficulties, and behaviour (Humphrey et al., 2010). Like PEP, teachers in the SEAL intervention expressed that they experienced beneficial effects of the programme, but the data failed to show the impact (Hallam, 2009).

A first possible explanation for the non-significant results in the multilevel analyses is the limited implementation fidelity. Findings in the current study revealed implementation difficulties for both intervention schools. According to Durlak et al. (2011), social-emotional learning programmes with a higher quality of implementation produce higher effect sizes on skills, attitudes, positive social behaviours, conduct problems, emotional distress, and academic performance. Other research indicated that whole school approaches tend to show difficulties in implementation fidelity because they require a substantial amount of planning, organisation, and support beyond the classroom (Wilson & Lipsey, 2007; Wilson et al., 2003). For both intervention schools in the current study, implementation difficulties arose because of changes at the principal level, leading to practical difficulties in the implementation of PEP. Results from the pilot study of PEP already demonstrated the importance of principal support for teachers to feel the confidence to focus on the process of learning (Goldberg et al., 2018). Additionally, the teachers identified changes in managerial staff as a hindering factor in the implementation of PEP (Elfrink et al., 2017). This aligns with research findings that adequate support from school principals is a crucial factor in intervention implementation in schools (Kam et al., 2003). Besides changes at the principal level, one of the intervention schools received a negative review of the national inspection. This negative review had a demotivating effect on the teachers, which is reflected in significantly higher conflict scores and significantly lower closeness scores on the student–teacher-relationship scale relative to the other intervention school. Lastly, there were nationwide teacher strikes during the implementation of PEP, leading to changes in the planning of the workshops and leading to agitation amongst team members and diminished focus on implementing PEP.

A second possible explanation for not finding any significant multilevel effects is the bottom-up approach deployed in the current study. As argued by Weare and Nind (2011), programmes that are more prescriptive tend to show larger effects in the short term, whilst bottom-up programmes lead to well-rooted and long-lasting changes of attitudes and policies. Institutions need to think carefully about a balance between adaptability and prescription (Lendrum et al., 2012), to avoid becoming vague and diluted. During PEP, schools use their own values as a starting point for the implementation. There is no strict manual or timeline that they have to follow. This requires school teams to take a lot of initiative to integrate the programme in their school culture. In an earlier pilot feasibility study, teachers reported that they needed some time to get used to this ownership (Elfrink et al., 2017; Goldberg et al., 2018). Teachers described that, at first, they were missing a manual or certain guidelines, but later on during the implementation process, PEP “got into their DNA”. Teachers described that by staying close to their own values, they got to own the programme, and they felt comfortable to continue with the programme for the upcoming years, also without the assistance of trainers. They also described that they noticed higher wellbeing in their pupils and less behavioural incidents (Elfrink et al., 2017; Goldberg et al., 2018). The bottom-up approach requires teachers to change not only their perspective but also their actions. Therefore, the duration of the current study may have been too short to identify the true effects of such a whole school approach. The evaluation of SEAL (Wigelsworth et al., 2013) found a similar result for their 2-year time frame, which was identified to be too short to fully analyse the impact of the programme.

Despite the potential disadvantages of the implementation challenges and the bottom-up approach, a positive trend was detected for pupil engagement. The odds that pupils would be classified as “sufficiently engaged” by the end of the school year was 5 times higher in the PEP schools compared to the control schools. On the one hand, this trend for engagement has the potential to lead to a wide range of positive outcomes in the future. On the other hand, it is equally likely that this result indicates that no changes in engagement truly exist between these groups. This result therefore needs to be interpreted with caution. Previous research has shown that engagement is a predictor of academic achievement (e.g., Pietarinen et al., 2014), wellbeing (e.g., Gase et al., 2017) and social-emotional learning (e.g., Buhs et al., 2006). Theoretically, this could mean that there is a hierarchy in the working mechanisms of PEP. To achieve improvements in pupils’ wellbeing may take a longer time investment than it does for engagement, and wellbeing may be influenced by the level of engagement. Practically, relative to most other countries, pupils in the Netherlands are found to perform well in many measures of wellbeing (Organisation for Economic Co-operation and Development, 2019), making it more difficult to detect significant improvements on wellbeing.

Limitations and recommendations

In the current study, several limitations and recommendations arose concerning the deployed measurements. First, multiple scales showed poor reliability and were therefore excluded from the analyses. In the case of measuring wellbeing, the exclusion of the Kiddy KINDL-R resulted in not being able to include pupils from Grades 1 to 3 in the analyses. Second, the level of wellbeing was determined using a health-related quality of life questionnaire. This questionnaire did provide insight into certain aspects of wellbeing, but did not fully cover the construct of wellbeing as defined by Keyes (2002). The choice of the instrument was based on the fact that it was used in the previous PEP studies and the fact that there are no other adequate wellbeing questionnaires available in Dutch. Therefore, the wellbeing results can be compared safely with the results of the previous PEP studies, but also should be interpreted with caution. It may be useful to get independent observation scores on wellbeing, similar to what is done for the outcome engagement. Third, the current study heavily relied on self-report questionnaires. Earlier research (Elfrink et al., 2017; Goldberg et al., 2018) already showed the added value of qualitatively investigating the impact of whole school approaches. This added value is supported by the current finding that most baseline scores were already quite high or low, making it difficult to determine improvements in the quantitative analyses. Combining qualitative and quantitative results in future research could provide more information on potential differences between pre and post on the outcomes. Fourth, the current study was merely focused on the pupils. For evaluation of the intervention, it would be important to measure how much the teacher buys into the PEP programme, as well as the parents or the community.

Another limitation of the current implementation is the sample. Although the sample size for the number of pupils included in the research is rather high (639 in total), only two schools were included per condition, and the schools were not randomly assigned to the experimental and control condition. The low number of intervention schools resulted in the research being too sensitive for implementation difficulties, as well as being too sensitive for dropout. Additional research should therefore aim to include a larger number of schools per condition and randomisation. Ideally, this could be done by finding regional or national support for the implementation of the programme, as research finds this to be an important factor for successful, effective, and sustainable whole school approaches (Barry et al., 2017; Mart et al., 2015).

To gain better insight into the effectiveness of a whole school approach to impact wellbeing and engagement, additional studies employing a longitudinal design are needed. It is currently unclear whether an increase of engagement in the short term leads to enhanced wellbeing, reduced behavioural problems, and improved academic achievement in the long term. To conduct such an extensive long-term research, it is advised to deploy a randomised controlled trial with a larger sample. Additionally, the desired outcomes could be measured more often during the implementation. This will provide more insight into the exact working mechanisms of the intervention and will provide more insight into potential short-term effects of the programme. Also, it is advised to make use of triangulation in the measures and to add qualitative outcomes as part of the evaluation. Finally, it is recommended to develop an implementation measure to keep track of the implementation quality, as an addition to implementation fidelity. Based on the research recommendations done by Greenberg and colleagues (2005), it is advised for future research to routinely assess implementation quality, broaden evaluation efforts, evaluate implementation quality together with stakeholders, and examine how variation in implementation affects the results.

Conclusion

This study did not yield evidence for the effectiveness of a whole school approach in primary schools aiming to increase wellbeing and engagement of pupils. Nonetheless, a positive trend was observed for engagement, which may be interpreted as a preliminary indication of the shift in perspective the whole school approach aimed to achieve, although significant proof remains absent. Future research with longer follow-up periods may reveal whether implementing a bottom-up whole school approach based on the shared values of all staff members leads to long-term and long-lasting effects on pupils’ wellbeing. It cannot be taken for granted that integrating positive psychology in every layer of the school leads to the desired effects in the short term, but that should not refrain schools from placing a consistent focus on making the pupils feel at their best and engaging them optimally in the given education. Importantly, staff of the participating schools were so convinced of the impact of the programme that they decided to continue its implementation.

Acknowledgements

We thank Fonds Kinderpostzegels for the collaboration in the PEP research.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by Fonds Kinderpostzegels.

Notes on contributors

Jochem M. Goldberg

Jochem M. Goldberg, MSc, is the lead author of this article. The development and evaluation of the Dutch Positive Education Programme, as described in the current article, is one of the key areas of his PhD research towards positive education.

Marion P. J. Sommers-Spijkerman

Marion P. J. Sommers-Spijkerman is a postdoctoral researcher at the University Medical Center Utrecht, Department of Rehabilitation, Physical Therapy Science and Sports. Her work focuses on improving the quality of life of patients and family caregivers through the use of education and psychological support.

Aleisha M. Clarke

Aleisha M. Clarke is Assistant Director of Evidence at the Early Intervention Foundation (EIF) in London. EIF is one of the UK government’s What Works Centres and supports the use of effective early intervention to improve the lives of children and young people at risk of experiencing poor outcomes. Aleisha leads EIF’s research on child and adolescent mental health. Her research interests lie in the understanding of what works, for whom, and under what circumstances and as part of this work bridging the gaps between research, practice, and policy.

Karlein M. G. Schreurs

Karlein M. G. Schreurs is a professor in psychology, a cognitive-behavioural therapist, and a supervisor.

Ernst T. Bohlmeijer

Ernst T. Bohlmeijer is a professor in mental health promotion and head of the Department of Technology, Human and Institutional Behaviour at the university of Twente. He specialises in developing and evaluating (online) mental health interventions based on frameworks such as acceptance and commitment therapy, positive psychology, and compassion-based therapy.

Appendix 1. Scoring engagement and wellbeing

Engagement – Score

1 = Rarely attains to actual activity; stares a lot, absent, apathetic; just brief moments of attention; hard to reach; when active, the actions are stereotypical, simple, and require minimal effort; mental activity is minimal; understands little

2 = Mostly interrupted activity

3 = Mostly attains to activity with progress in the actions; is there with his mind, but misses the engagement signals: often distracted, limited attention span, not really absorbed or touched by the activity

4 = The pattern mostly consists of engagement

5 = Concentrated and works continuously most of the time; hard to distract; alert; absorbed and fascinated; mentally active on a high level; appeals to his full potential; acts on the edge of his abilities; enjoys exploring

? = Not enough information, very unclear image or not yet determined

Wellbeing – Score

1 = Doesn’t feel good most of the time; lacks enjoyment; often tensed, misses inner peace; lots of signals of negative experiences; little confidence, low self-esteem; relationships with others are negatively loaded; mainly: not feeling happy

2 = The pattern mostly consists of discontentment

3 = A neutral or mixed pattern, signals of not feeling good; having fun is transitory and not intense; relationships with the environment are not optimal, but also not alarming; not happy nor unhappy

4 = The pattern mostly consists of wellbeing

5 = Feels optimal most of the time; enjoys at the highest level; appears to be full of vitality; is relaxed and experiences inner peace; is open for his environment and adjusts to it rapidly; is confident and acts in a resilient manner; is happy and content with himself

? = Not enough information, very unclear image or not yet determined