Effects of an early warning system on student absence and completion in Norwegian upper secondary schools: a cluster-randomised study

ABSTRACT

This cluster-randomised study investigated the effects of a Norwegian early warning system, the IKO model. IKO is a Norwegian acronym for identification, assessment, and follow-up, and the model aims to improve schools’ abilities to identify and support students who are at risk of dropping out during the school year. The study involved 7677 first-year students in 42 upper secondary schools, 20 schools randomised to the experimental group and 22 to the control group, and utilised administrative data. Two-level logistic and linear regression models with students nested in schools were applied. After two school years, there were no significant effects on absence from lectures, completion rates, or academic results. The analyses did not indicate stronger effects among students at risk of dropout before entering upper secondary education than among students in general.

KEYWORDS:

• School dropout
• cluster-randomised
• early warning system
• upper secondary
• Norway

Introduction

Achieving as high completion rates in upper secondary education as possible is a common policy goal (Nyen & Tønder, 2014, p. 12; OECD, 2020; St.meld. 30 [2003–2004]). Despite increased research, there is limited knowledge about effective measures to reduce the number of young people who leave school without gaining a diploma from upper secondary education (Freeman & Simonsen, 2015; Lehr et al., 2003; Lillejord et al., 2015; Wilson et al., 2011). Research reviews emphasise the complexity of addressing the school dropout issue. First, the term “dropout” refers to a heterogeneous group and includes both those who leave before finishing and those who stays but fail to pass all subjects. Second, causes and solutions may be found both at the individual level and in the institutional or social context (González-Rodríguez et al., 2019). Since there is no single cause or general solution to school dropout, policymakers, practitioners and researchers often recommend wider community- and school-level interventions that address organisational structures, social contexts and individual needs (Freeman & Simonsen, 2015; Rumberger et al., 2017). In the United States, there is growing experience with a specific type of school-level intervention that combines institutional and individual measures. Early warning systems allow schools to use administrative data both to monitor dropout at the school level and to identify and intervene with individual students at an early stage (for recent reviews see Balfanz & Byrnes, 2019; McMahon & Sembiante, 2019). However, there are only a few studies – and to our knowledge, no studies in Europe – that have evaluated the effectiveness of these early warning systems using a randomised controlled design (Corrin et al., 2016; Faria et al., 2017; Mac Iver et al., 2019).

This cluster-randomised study focusses on an early warning system – the IKO model – first developed by Akershus County in 2008 and later implemented in four other Norwegian counties. The two-year follow-up study (school years 2016/2017 and 2017/2018) involved 7677 first-year students in 42 upper secondary schools in the counties Nord-Trøndelag, Oppland, Hedmark and Aust-Agder. We analyse the effects of the IKO model on achievements, absence from lectures and completion, which all are factors expected to predict both early school leaving and completion of secondary education without gaining a diploma. As the model is designed to improve schools’ abilities to identify and intervene early among students at risk of dropping out, stronger intervention effects are anticipated for students who show signs of school disengagement and low achievements already before entering upper secondary, than for students in general. Hence, we measure the effects of the model for all students and for at-risk students.

The study was funded by the Norwegian Ministry of Education and Research through the “Programme for enhanced completion of upper secondary education and training”.

Early warning systems

The research literature includes a wide variety of theories on the causes that influence school dropout (González-Rodríguez et al., 2019). Comprehensive theories include both pull and push factors and differentiate between individual and institutional factors (Bradley & Renzulli, 2011; Rumberger & Lim, 2008). There is general agreement that individual factors (e.g., educational performance, behaviours, attitudes and social background) are effective in predicting graduation rates (e.g., Lamb & Markussen, 2011; Markussen et al., 2011; Ripamonti, 2017). However, several recent reviews have emphasised the role of institutional factors in dropout prevention (De Witte et al., 2013; Freeman & Simonsen, 2015; Ripamonti, 2017). In particular, push factors that schools can have some control over (e.g., use of resources, structural features, attitudes, policies and practices) are expected to be useful. When students are “pushed out” of schools, it is because schools fail to create appropriate relations and conditions for individual students. Hence, schools need effective systems to identify these at-risk situations and systems for continuously monitoring and adjusting how schools interact with individual students to offer more targeted interventions (Mac Iver & Mac Iver, 2009).

The basic idea behind early warning systems is to develop schools’ abilities to support students who are at risk of dropping out during the school year. The effectiveness of early warning systems is expected to derive from the following: 1) the ability of teachers and school managers to analyse large sources of data to identify at-risk students and intervene early; 2) a dedicated team of staff and mapping of roles, responsibilities and meeting structures; and 3) components for training and the exchange of best practices (see e.g., Balfanz et al., 2007; Faria et al., 2017; Mac Iver, 2011; Mac Iver & Mac Iver, 2009; Neild et al., 2007). The points are described in turn below.

First, the use of administrative data in a software-assisted early warning system implemented at the school level enables leaders and teachers to be continuously aware of students who are at risk. The procedure for identifying students in a software-assisted early warning system component builds on previous research, which has shown that indicators of school disengagement (absence rates, behaviour measured by suspensions) and level of pupil engagement through course performance (grades and credits) predict graduation rates (Balfanz et al., 2007; Macfadyen & Dawson, 2010; Márquez-Vera et al., 2016). Hence, tracking these indicators regularly enables schools and teachers to intervene early. From a theoretical point of view, the use of specific indicators in this identification process, such as absence and course performance, will keep the focus on schools’ responsibilities (push factors) to support pupils’ achievements rather than blaming individual pupils (pull factors) for failure (Lee & Burkam, 2003).

Second, the organisational structure around identifying students for interventions by working in teams of managers and teachers makes identifying and intervening among at-risk students a shared responsibility and less dependent on the activities of individual teachers. Moreover, manuals for activities and routines are implemented to ensure that teachers map and evaluate students’ progress often and with higher quality.

Third, communities of learning contribute to developing an organisational culture with shared beliefs, principles and purpose. This training of staff ensures that leaders and teachers are regularly inspired and informed about possible inventions and relevant methods.

Since causes and possible solutions to school dropout are complex (González-Rodríguez et al., 2019; Wilson et al., 2011), the model includes no standard “dose” or type of intervention to be given to all students identified as at risk (McMahon & Sembiante, 2019). The basic idea is that the measures and their intensity need to be matched to the individual students’ needs. The goal is to meet the student’s individual needs, and thus, increase engagement (e.g., attendance and motivation) and achievements (e.g., grades and completion rates) via tailored interventions. Expected short-term outcomes at the student level are increased engagement and improved achievements. Moreover, as school dropout is closely related to low engagement and achievements during upper secondary education, reduced dropout is expected as the long-term outcome. The anticipated effects (increased engagement and improved achievements) relate in particular to students identified as at risk of dropping out either before or during upper secondary education.

On-going lack of robust evidence

While the development of early warning systems and the selection of early warning indicators are based on a wide literature covering both theoretical and empirical knowledge, the empirical evidence supporting their effects is still rather weak. Nevertheless, there are some recent studies on this topic.

The experimental evaluation of the Diplomas Now model in US urban lower and upper secondary schools represents one example. The model combines a comprehensive school reform strategy with an early warning system followed by targeted interventions for students who are identified as at risk (Corrin et al., 2016), with the long-term goal of increasing students’ college readiness (indicated by attendance, behaviour and course performance). The researchers assigned 32 schools to the experimental group and 30 schools to the control group. After two years of implementation, middle schools using the Diplomas Now model were significantly more successful in helping at-risk students stay above the early warning thresholds related to all evaluated outcomes (attendance, behaviour and course performance) when assessed together, compared with schools randomised to the control condition (Corrin et al., 2016). Contrary to what was expected, the intervention had a greater effect on students who were not identified as at risk prior to middle school than it did on students identified as at risk. This study did not document effects at the upper secondary level.

Faria et al. (2017) used a cluster-randomised design to examine the impacts of early warning and monitoring interventions in 73 upper secondary schools in three US states (37 experimental and 36 control schools). Only two schools achieved moderate or high levels of implementation quality, and 8 out of 37 experimental schools stopped implementing the model during the first school year. Despite low levels of implementation, the study showed an early impact (after 14 months) and concluded that the implementation of the early warning and monitoring interventions reduced risk indicators related to chronic absence and course failure among students, but it did not document effects on students’ grade point average (GPA) or suspensions.

Mac Iver et al. (2019) published results from a cluster-randomised study that included 41 geographically and demographically diverse American high schools. The study investigated the effects of an early warning intervention comparable to the model investigated by Faria et al. (2017). The study estimated the effects after the second year of implementation. Analyses of attendance rates and course credits indicated no statistically significant impact of the model in treatment schools. The implementation findings suggested that the intervention did not make a significant enough difference in school practices to influence students’ outcomes. The study further underscores the importance of follow-through in delivering actual interventions to struggling students. Putting the early warning indicator system components in place at the institutional level is not enough; this action must be accompanied by interventions that address the underlying reasons for absenteeism and course failure.

The Norwegian context

The current study examines a version of an early warning system developed for Norwegian upper secondary schools. Almost all young Norwegians (96%) enter upper secondary education. In 2020, one out of five students did not complete upper secondary education within five to six years (the standard duration is three to four years) (Norwegian Directorate of Education and Training, 2020a, 2020b). As admission into different programmes and different secondary schools is based on academic performance during lower secondary education (Markussen et al., 2011), completion rates vary by schools and programmes. In upper secondary, students choose between five general programmes aimed at preparing students for higher education and eight vocational programmes leading to vocational degrees. Most schools offer general programmes, but the range of vocational programmes varies more between schools. Accordingly, upper secondary schools vary considerably in size.

Students in vocational programmes generally have weaker school performance, lower completion rates, and a higher risk of dropout than students in general programmes do (Lamb & Markussen, 2011). Moreover, completion rates tend to be lower among boys, immigrants, and students in families with low socio-economic status, and these groups tend to be over-represented in vocational tracks (Lamb & Markussen, 2011; Markussen et al., 2011). At the individual level, important school-related variables predicting early school leaving are earlier school performance, academic and social identification, and school engagement. Among these, the far most predictive variable is earlier school performance as measured by grades from last year of compulsory education (Markussen et al., 2011).

Study overview

The aim of this article is to analyse student-level effects of implementing the IKO model by comparing outcomes for students in schools randomised to the experimental condition with students in schools randomised to the control condition. Schools in the experimental group implemented the IKO model in autumn 2016, whereas schools in the control group continued their work on dropout prevention as before. We tested the following hypotheses:

H1: The IKO model promotes academic results, reduces absences from lectures, and increases completion rates.

H2: The IKO model is particularly efficient among students at risk of dropout.

As most Early warning systems the IKO model¹ aims to improve the way schools identify and follow up on students at risk of dropping out with temporary follow-up needs (i.e., not qualified for special education).² Figure 1 demonstrates the logic model, which includes the following components: (1) a software-assisted early warning system, (2) organisational structures and practices that enable schools to monitor progress and implement effective interventions and (3) communities of learning both within and outside of the school.

Figure 1. IKO Abridged theory of action.

In the organisation component, each school forms a small, dedicated IKO team, consisting of an IKO responsible (appointed among the school staff), school leader, one or two teachers and/or interprofessional staff. The IKO team is responsible for implementing an annual cycle showing the various IKO activities, including the roles and responsibilities of teachers, school leaders, and the IKO team. The IKO manual (distributed to all schools) describes these activities (the main activities listed in Table 1).

Table 1. Annual activities and responsibilities of the IKO model.

Activity	When	Who
Analyse data in the software: identify at-risk students through information transferred from lower secondary school	Before school start	IKO responsible
Use the identification process and additional information from lower secondary schools to construct heterogeneous classes	Before school start	School leaders
Information to class teachers about students who have been identified as at-risk students through analyses of student data + additional information	Planning day/before school start	IKO responsible
Send out information about forthcoming interviews to students and parents/guardians	At school start	School leaders
Assessment A: Interviews with students identified by the software at school start	Within three weeks	Class teacher
Assessment B: Interviews with all students	Within four weeks	Class teacher
Assessment C: Evaluate the need for follow-up measures for students identified before school start (A) and additional students identified during interviews with class teachers (B)	Within four weeks	Teachers
Class teacher meetings to agree on and plan follow-up needs at class level	Within five weeks	Class teacher/IKO responsible
Mid-term evaluations for all students. Preliminary marks are given and registered in the software	1 November, 1 April, half-year evaluation	Teachers
Class teacher meetings to agree on follow-up needs at class level and develop follow-up interventions and recommendations to school leaders	1 November, 1 April, half-year evaluation	Class teacher/IKO responsible
Register students who need a closer follow-up	1 November, 1 April, half-year evaluation	Class teacher
Reflect on the results from mid-term evaluations. Evaluate the need for measures at individual, class, and/or school level. Evaluate the need for increasing competence in the staff	1 November, 1 April, half-year evaluation	Leaders/teachers

Before schools start in August, schools identify at-risk students by analysing data from the lower secondary level in the software-assisted early warning system. The educational software company Conexus was responsible for developing the software used in the IKO model. The following five criteria are used to identify at-risk status: (1) accepted by first choice of upper secondary school³ and having a GPA of 2.5 or below,⁴ (2) accepted by second or lower-prioritised choice of school and having a GPA of 3 or below, (3) total absence higher than 6%, (4) a failing or unknown grade in one or more subjects and (5) prior history of special education. Identification of at-risk students continues during upper secondary. The IKO model includes two mid-term assessments (by 1 November and 1 April),⁵ where teachers register preliminary grades for all students. These assessments, together with the ordinary half-year assessment and records of absence, are included in the software-assisted early warning system.

Class teachers should pay special attention to identified students and interview them to assess the students’ total situation. Teachers use class teacher meetings (often held weekly) to share information from these interviews. Thereafter, decisions on interventions are made as a collective action in the class teacher meetings. To our knowledge, interventions typically range from measures to improve students’ academic skills, instruction on learning strategies and courses to improve test-taking skills to general counselling and efforts to increase motivation. The IKO team is responsible for ensuring regular evaluation and revision of the interventions if ineffective.

The communities of learning component includes yearly seminars where schools exchange “best practices” with other schools that use the IKO model in their region. In addition, school leaders and teachers were to participate in tailor-made web-based training to support organisational-level and student-level interventions to reduce dropout. This was a one-year training for schools randomised to experimental group, which included three modules, one module for school leaders on how to develop the organisation, one module for teachers and school staff on the theoretical approaches to learning (mindset etc.) and one module for all the staff in the school on how to develop the organisation.

Data and methods

All 42 upper secondary schools in four Norwegian counties took part in the study. Akershus County, which had developed the model, supported implementation. However, Akershus was not included in the cluster-randomised study. The four participating counties had completion rates at approximately the national average by the onset of the project (Norwegian Directorate of Education and Training, 2018). Schools were randomised to experimental and control groups within each county. Two of the four counties had an unequal number of schools, and both surplus schools were by chance randomised to the control group. Twenty were randomised to the experimental group, and 22 were randomised to the control group. None of the schools opted out prior to randomisation or during the study (no self-selection bias). The study follows an intention-to-treat principle, including all students independent of exposure, implementation level, and quality. We use data on all 7678 students enrolled in the first year of upper secondary school, of which 7364 (95.9%) were observed at all three time points (i.e., at the end of lower secondary, first and second year of upper secondary education). Students’ outcomes were measured at the end of the school years 2016/2017 and 2017/2018, giving us a follow-up period of two school years.

Research permissions were granted by the Norwegian Centre for Research Data (case no. 48657) and the Norwegian Data Protection Authority. In autumn 2016, we informed students and guardians about the study and the use of administrative data and explained that they could withdraw from the study at any time and for any reason. Twelve students chose to withdraw and were removed from the data file. The data was deidentified for the researchers. Before transferring administrative data to the researchers, the counties (as data owners) replaced personal information (e.g., names and social security numbers) with a unique personal identifier which enable linking information on students over time. The code to this personal identifier was kept by the counties and stored as sensitive data.

Administrative data on students

Our population comprised all first-year students enrolled in the 42 participating upper secondary schools in autumn 2016. The VIGO database⁶ was used as the primary source for all outcome measures in the study. This database is prepared for all Norwegian counties and includes administrative data on grades and absences from 10th grade in lower secondary school, application and acceptance status to upper secondary school and data on student status and absence for all years in upper secondary education.

The primary outcome measures are completion and absence (lectures and days) during the first and second years of upper secondary school and final grades by the end of these years. All these outcomes are highly predictive of students’ completion of upper secondary education (De Witte & Csillag, 2014; Ripamonti, 2017; Rumberger, 2011). Completion was assessed by the students’ registered status at the end of the two school years and recoded into a dummy variable. If a student is identified as having completed and passed, completion is coded 1; otherwise, it is coded as 0. Absence is assessed separately for missed lectures and days absent, as reported regularly in the school administrative data.⁷ Final grades are a mean of all grades on oral and written exams, final assessment grades, and report card grades (scale 1–6).

School size is measured as the number of students attending the school in the first year of upper secondary education. A number of students change schools during upper secondary education. Of the 7677 students included in this study, 5.1% (393 students) changed from an experimental school to a control school; 3.4% (260 students) changed from a control school to an experimental school; and 2.6% (202 students) changed to schools that are not included in the study. We controlled for these school changes in our final analyses.

Educational programme is measured as a dummy variable, holding the value of 1 if the student attends a general educational programme (students achieve a general university admissions certification and can take higher education) and 0 if the student attends a vocational educational programme (leading to vocational competence). All schools in the study included multiple educational programmes, both general and vocational.

Students can change school from first to second year of upper secondary school, which means that students can move in and out of treatment schools. In second year of upper secondary school, students where categorised into four groups depending on their school affiliation. Students who remained in the same allocation (experimental if experimental at first year, control if control at first year) were defined as “continuers”, students who changed from a control to an experimental school were defined as “moving in”, students who changed from an experimental to a control school were defined as “moving out”, and students who changed from either an experimental or a control school to a school beyond the study were defined as “moving beyond”. In the analyses, the variable is introduced as a categorical variable (using the i.-function in Stata), with “continuers” as reference value.

Analyses of missing values (Table 2) show no significant difference between the experimental and control group in any of the outcomes at first or second year of upper secondary school.

Table 2. Missing values in experimental and control conditions, % (n).

Variable	Experimental		Control		t-test
	%	(n)	%	(n)
Completion, 1st year	0.00	(0)	0.03	(1)	0.240
Completion, 2nd year	4.00	(178)	3.87	(125)	0.779
GPA, 1st year	4.00	(178)	3.69	(119)	0.484
GPA, 2nd year	10.11	(450)	9.24	(298)	0.201
Absence from lectures, 1st year	1.17	(52)	1.21	(39)	0.873
Absence from lectures, 2nd year	5.21	(232)	5.08	(164)	0.798
Days of absence, 1st year	1.15	(51)	1.21	(39)	0.801
Days of absence, 2nd year	5.19	(231)	5.15	(166)	0.928

The success of randomisation

In this study, we performed a blocked randomisation; that is, we randomised schools to experimental and control groups within each county. Table 3 shows a comparison of the school and student characteristics of the experimental and control conditions. By chance, the randomisation resulted in larger schools, and consequently, a higher number of students in the experimental condition than in the control condition. Further, there was a higher share of students in general programmes in the experimental condition compared with the control condition. Since only students who have completed lower secondary can start upper secondary education, there was full completion of lower secondary school in both experimental conditions. However, there were somewhat higher absence rates during lower secondary education and a higher share of students in general programmes in the experimental condition compared with the control condition.

Table 3. Characteristics by experimental and control condition.

Variable	Experimental	Control	t-test
Schools, N	20	22	n.a.
First-year students, N	4449	3229	n.a.
Number of students per school, M	222	147	0.025
Students in general study programmes, %	51.5	48.3	0.005
Completion of lower secondary, %	100	100	n.a.
Grade point average (GPA) lower secondary, M	4.06	4.02	0.072
Absence from lectures lower secondary, M	7.58	6.37	>0.001
Days of absence lower secondary, M	8.98	8.34	0.034

Note: All student characteristics represent the situation before entering upper secondary school (i.e., when leaving lower secondary).

Analysis plan

All analyses are conducted using two-level models with students clustered within schools. For the dichotomous outcome, we applied a two-level logit model. For continuous outcomes, we applied a two-level linear model. We report the effect of the IKO model on absence and GPA after the first and second years of upper secondary education as coefficients and the effect on completion as odds ratio (OR). To correct for the nested structure of the data, we applied two-level regression models. All models control for the outcome measurement at baseline (i.e., at the end of lower secondary education) in addition to educational programme, school size and absence from lectures and days in lower secondary school, as these variables significantly differ between experimental and control conditions. In the models analysing the effects of the IKO model after the second year of upper secondary school, we also control for whether students change schools between experimental and control conditions or move to schools that are not part of the study. The following mathematical equation represents the basic model: $y_{ij}={\beta }_1+{\beta }_2{\chi }_{2j}+{\beta }_3{\chi }_{3ij}+{\zeta }_{1j}+{\zeta }_{2j}{\chi }_{ij}+{\epsilon }_{ij}$where y is the outcome, χ = explanatory variables and covariates, i = individual, j = clusters (schools), and ϵ_ij are residuals that are independent across schools and individuals. The model has two parts, fixed and random effects. A fixed effect represents a single value, β, existing in the population and assumed to be shared by all clusters: β1 is the intercept (i.e., starting point) and β2j is the regression coefficient of cluster constant predictors (e.g., school size), while β3ij is the regression coefficient of variables that may vary across individuals within schools (e.g., days of absence). ζ1j and ζ2j represent the random intercept and random slope in the basic equation. For the purpose of study, the fixed effects (β) are presented and discussed.

All results are presented for the entire student population and for students identified as being at risk for school dropout (before entering upper secondary school). The at-risk group was identified by the same procedure as in the IKO software using administrative data from 10th grade as follows: 1) accepted by first choice of upper secondary school and having a GPA of 2.5 or below, 2) accepted by second or lower-prioritised choice of school and having a GPA of 3 or below, 3) total absence higher than 6% or 4) a failing or unknown grade in one or more subjects. The IKO model defines students as being at risk of dropout if they have at least one of these characteristics. There were 2745 students who met this definition. Most of these students (n = 2020) were identified by one criterion, whereas 725 students were identified by two or more criteria.

All analyses were conducted in Stata/MP 14.0. The syntax can be provided on request.

Results

Table 4 gives a descriptive comparison of students belonging to the experimental and control conditions after the first and second years of upper secondary education. The first two columns show results among all students, whereas the last two columns show results among students identified as at risk before entering upper secondary school.

Table 4. Student-level variables by experimental condition.

Variable	All students		At-risk group
	1st Year	2nd Year	1st Year	2nd Year
Grade point average (GPA), M
Control	3.90	3.94	3.41	3.53
Experimental	3.91	3.91	3.47	3.50
t-test	0.642	0.203	0.119	0.470
Design effect	5.687	5.211	4.172	3.664
Completion (%)
Control	86.6	85.3	72.5	71.7
Experimental	86.5	84.5	72.1	71.4
χ^2 test	0.945	0.381	0.795	0.868
Design effect	4.059	2.985	2.703	2.523
Absence from lectures, M
Control	10.7	14.5	17.3	19.3
Experimental	11.4	15.5	18.9	21.8
t-test	0.092	0.026	0.111	0.008
Design effect	4.743	3.796	3.319	2.306
Days of absence, M
Control	5.2	5.7	8.7	8.9
Experimental	5.3	5.8	9.0	9.0
t-test	0.446	0.524	0.474	0.758
Design effect	2.038	2.205	1.720	1.605

The comparison demonstrates no effects of the IKO model on GPA, completion or days of absence, but it shows an adverse effect on absence from lectures both among all students and in the at-risk group in the second year of upper secondary education. Mean levels of absence from lectures were significantly (0.05 level) higher among students belonging to the experimental conditions among all students (experimental = 15.5, control = 14.5, p = 0.026) and among students in the at-risk group (experimental = 21.8, control = 19.3, p = 0.008). However, the descriptive analyses are only indicative. To estimate the effects of the IKO model, we adjust for baseline differences in the final models (Tables 4 and 5).

The design effects vary from 1.605 to 5.687, depending on outcome, population, and time point. Simulations by Lai and Kwok (2015) suggest that single-level analyses are inappropriate if the design effect is larger than 1. Hence, we apply two level models to adjust for the nested structure of the data (Tables 5 and 6).

Table 5. Effect of the IKO model. All students.

	Completion (1)		GPA (2)		Absence from lectures (3)		Days of absence (4)
	1st year	2nd year	1st year	2nd year	1st year	2nd year	1st year	2nd year
	OR	OR	Coef.	Coef.	Coef.	Coef.	Coef.	Coef.
IKO model	1.144	1.066	0.005	0.021	−0.325	−0.631	0.094	−0.190
Standard errors	0.170	0.133	0.033	0.032	0.745	0.773	0.313	0.339
P-value	0.364	0.607	0.875	0.502	0.662	0.414	0.765	0.575
N of observations:	7,552	7,255	7,173	6,741	7,461	7,160	7,461	7,164
N of schools:	42	42	42	42	42	42	42	42
ICC	0.033	0.020	0.025	0.014	0.011	0.008	0.008	0.008

Note: All models control for educational programme, school size, baseline absence (days and lectures) and the outcome measurement at baseline. Second-year models also control for students’ change of school.

Table 6. Effect of the IKO model. At-risk students.

	Completion (1)		GPA (2)		Absence from lectures (3)		Days of absence (4)
	1st year	2nd year	1st year	2nd year	1st year	2nd year	1st year	2nd year
	OR	OR	Coef.	Coef.	Coef.	Coef.	Coef.	Coef.
IKO model	1.090	1.038	0.003	−0.014	−0.269	0.664	−0.012	−0.788
Standard errors	0.189	0.170	0.043	0.048	1.683	1.521	0.644	0.650
p-value	0.621	0.822	0.945	0.773	0.873	0.662	0.985	0.225
N of observations:	2,618	2,485	2,331	2,152	2,578	2,449	2,579	2,451
N of schools:	42	42	42	42	42	42	42	42
ICC	0.004	0.003	0.011	0.007	0.021	0.012	0.008	0.005

Note: All models control for educational programme, school size, baseline absence (days and lectures) and the outcome measurement at baseline. Second-year models also control for students’ change of school.

Multivariate analyses of intervention effects

Table 5 shows the adjusted results for all students. The OR for effects of the IKO model on completion was 1.144 in the first year of upper secondary education and 1.066 in the second year, and not statistically significant. The coefficients of the IKO model on GPA are close to zero in the first and second years of upper secondary education, indicating no effects of the IKO model. The coefficients of the IKO model on absence from lectures were not significant neither the first nor second year of upper secondary education.

Table 6 shows the adjusted results for students in the at-risk group. As for students in general, these results do not suggest any statistically significant effects of the IKO model. The OR for the IKO model on completion was 1.090 (p = 0.612) in the first year of upper secondary education and 1.038 (p = 0.822) in the second year. Regarding the coefficients for the IKO-model on absence, we found no significant effects on neither absence from lectures nor days of absence. Regarding GPA, the coefficients were close to zero and not significant the first and second years of upper secondary education, indicating no effect of the IKO model.

Discussion

Our first hypothesis was that students in schools randomised to the experimental condition would reduce their risk of dropout, measured by better academic results, lower levels of absence and higher rates of completion after the first and second years of upper secondary school. Our second hypothesis was that the IKO model would be particularly efficient among students identified as having an elevated risk of school dropout (measured by administrative data from 10th grade on GPA, absence and acceptance (or non-acceptance) by the first choice of upper secondary school). The study’s realised effects were not significant, and consequently, both hypotheses are rejected.

The results for students in general do not demonstrate significant effects of the IKO model on academic results, nor do they do so on completion rates and absences (days and lectures). The sensitivity analyses for the “at-risk” group shows similar results. In accordance with the study by Corrin et al. (2016), our analyses did not indicate stronger effects among at-risk students. Still, our analyses give some promising results on the reduction in absence from lectures (approximately by one lecture on average) and on days of absence in the second year for the at-risk group. Faria et al. (2017) found in their US study on early warning and monitoring interventions, favourable effects on reduced absence. Reducing absence from lectures at an early stage of upper secondary school could be important, as frequent non-attendance at an early stage may create a negative cycle that can be difficult to break (De Witte & Csillag, 2014; Havik et al., 2015; Markussen et al., 2011). Non-attendance reduces students’ involvement in learning activities that take place in the school and may lead to declining school performance over time. Moreover, a high absence rate may decrease school motivation by undermining relations to teachers and fellow students or by increasing involvement in unhealthy or deviant acts. However, in this study, we do not find a significant buffering effect of the IKO model on absence.

In sum, the results may indicate that the theoretical underpinnings are wrong and that the model is not performing as expected. The IKO model understands dropout as resulting from an interaction between individual student characteristics and schools and that students are “pushed out” because schools fail to create appropriate relations and conditions. Since dropout is also connected to factors that are difficult to address solely by adjusting institutional factors within schools (e.g., health issues, family problems, drug use), it may be that an effective intervention to a further extent should be able to address problems students face outside the school context. At the same time, before rejecting the model and its theoretical underpinnings, possible implementation issues, dissemination of treatment practices and study limitations that may explain the result should be addressed.

Implementation issues

Our analyses on programme fidelity have been reported in more detail elsewhere (Malmberg-Heimonen et al., 2019), showing variation in implementation quality between model components, counties and schools. These results indicate that schools (in the experimental group) worked more systematically with the identification of at-risk students, whereas assessments and follow-up of students improved less. Because the effectiveness of early warning systems is expected to derive from implementing a system that enables schools to continuously monitor and adjust how they interact with and intervene among individual students during the school year, this may explain the lack of significant effects after two years of implementation. These findings are in line with the implementation findings in Mac Iver et al. (2019) and Faria et al. (2017). In their studies, a school’s ability to follow through in delivering interventions was a better predictor of improvements in course passing and attendance than having the early warning indicator system in place.

In general, the failure to implement tailor-made interventions and follow up on them seems to be a particularly vulnerable element in early warning systems (Balfanz & Byrnes, 2019; Davis et al., 2013). Future studies should address this concern. It might be necessary to redesign elements of the IKO model and the intervention plan to strengthen the school’s follow-through in delivering interventions after identification of at-risk students. It is also possible that a more powerful model should have offered more detailed tools to the schools, also for intervening at the individual level. The current model – evaluated in this study – must, however, be understood as a school level intervention which is restricted to the introduction of a system (at the school level) for identifying at risk situations and for continuously monitoring and adjusting how schools interact with students to offer more targeted interventions. As the IKO model rests on the assumption that there is no standard “dose” or type of individual interventions, this study did not systematically acknowledge the interventions offered to students.

Dissemination of treatment practices

To increase graduation rates from upper secondary education, the Norwegian government launched in 2010 a large-scale three-year national effort called New possibilities⁸ (Ny GIV in Norwegian) (Holen et al., 2020; Huitfeldt et al., 2018). This initiative was extended by the “Programme for enhanced completion of upper secondary education and training”. During these years, most Norwegian upper secondary schools improved their systems and routines to identify, assess and follow up on at-risk students. In this period, the IKO model was used by Akershus County, and although it was not communicated systematically, theoretical underpinnings and elements of the IKO model were discussed in various inter-school and inter-county networks. Moreover, a new national absence limit was introduced in all upper secondary schools in August 2016. According to the new rules, students lose the right to a final grade with an absence rate greater than 10 per cent in a subject. Since keeping track of absence is a central element of the IKO model, this may also have reduced the contrast between the control and experimental settings.

In general, the possible disseminations of treatment components prior to the intervention may have influenced the work within the control schools. In this study, the control schools were aware of their status as controls and could compare their performance with that of the treatment group because there were both control and experimental schools within the same geographical and administrative contexts (i.e., county). It is possible that the dissemination of the model inspired control schools to implement organisational structures and routines that made them work more systematically with the following up of at-risk students. It could also be that the staff at the control schools perceived the dissemination as a kind of contest, motivating them to work harder to overcome the “disadvantage” of being in the control group This mechanism is sometimes labelled the “John Henry” effect and refers to the bias introduced to an experiment when members of the control group are aware that they are being compared to the experimental group and behave differently than they typically would to compensate for their perceived disadvantage (Frey, 2018).

The situation highlights the importance of documenting an experimental contrast (in administrative routines and the school staff’s ways of working) before making conclusions about the effectiveness of the different components in the IKO model. In contrast to the results of Mac Iver et al. (2019), our findings indicate some significant treatment contrasts (Malmberg-Heimonen et al., 2019). Teachers’ practices differed between treatment and control schools regarding the access to and use of information to identify at-risk students. However, the implementation findings did not show significant contrasts regarding schools’ routines for assessment or the follow-up of at-risk students after identification. Hence, the contrast between what happened in the experimental and control schools was smaller than planned.

Study limitations

In accordance with a possible weak treatment contrast, there are some further limitations related to the study design that should be considered when interpreting the results. First, we evaluated the short-term effects of the intervention. Given that the implementation of the model began in August 2016, these findings document an early impact of the IKO model: The effects were evaluated after only two school years. Second, regarding absence, we do not have the means to check the accuracy of national registers and are therefore unable to acknowledge that schools may have different routines for registration. Still, we assume that the randomisation of schools within counties would contribute to a similar distribution of this problem between schools in the experimental and control groups. Third, this is an intention-to-treat study comparing all schools that were randomised into experimental and control conditions, regardless of the level of implementation within experimental schools. Consequently, these are lower bound effects of programme impact. An intention-to-treat design does not account for different levels of implementation fidelity. This increases the risk of a type II error in that previous studies on dropout prevention programmes have found stronger effects in schools that had greater implementation fidelity (Freeman et al., 2016); alternatively, they have explained a lack of anticipated effects by pointing to a low implementation quality (Faria et al., 2017). In contrast, removing schools that did not implement the intervention well enough from the analyses would introduce a selection bias, conflicting with the fundamental idea of randomised controlled trials (RCTs).

Conclusion

This study investigated the effects of a Norwegian early warning system implemented as a school-level approach to dropout prevention – the IKO model. The research design is an example of a pragmatic cluster-RCT, with the intervention implemented in an ordinary school setting. After two school years, there were no significant effects on absence from lectures, completion rates or academic results. According to the basic idea of the IKO model, effects among the at-risk group of students could be expected to be stronger than among students in general because the model particularly focusses on improving the ways schools identify and follow up on this group of students. However, the study did not indicate stronger effects among at-risk students than among students in general. Still, it is too early to reject the IKO model and its theoretical underpinnings. The present study is the first to analyse the effects of early warning systems implemented in a Norwegian educational context. Additional research is needed, and in accordance with this short-term evaluation, it is also important to evaluate the longer-term effects.

Our discussion illustrates some of the challenges with randomised experimentation applied to an ordinary school setting. On the one hand, it is important for schools to identify interventions that are suitable and possible for schools to support and implement in their everyday routine. This normally implies that the intervention is close to ordinary practice and is implemented gradually with rather low intensity (e.g., Holen et al., 2020). On the other hand, since most effect studies follow interventions over a relatively short time span, to identify effects, there is a need to ensure enough contrast between treatment and control quickly. Despite these kinds of problems, experimental designs are vital to estimate programme effects within the educational sector (Raudenbush, 2018). In the future, studies should explicitly address this paradox – for example, by including longer periods for implementation prior to evaluation and a longer follow-up period. Moreover, theorising on the mechanisms that allow an intervention to succeed within varied timeframes and settings is crucial.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 IKO is a Norwegian acronym for identification, assessment and follow-up.

2 Special education refers to teaching given pursuant to the rights formulated in section 5 of the Norwegian Education Act. If a pupil does not or cannot receive proper academic benefit from the ordinary teaching, that pupil may receive an expert assessment, which may result in an individual decision triggering the right to special education. https://www.uv.uio.no/english/research/subjects/special-education/.

3 Final-year students within compulsory school apply for admission into different upper secondary educational programmes. Their grades at the end of lower secondary school determine whether they are admitted to their first choice of programme.

4 Norway uses a grade point assessment scale from 2 (minimum passing grade) to 6 (maximum).

5 Non-official, but obligatory for schools using the IKO model.

6 The VIGO administrative data system is owned jointly by all Norwegian counties. The main purpose of VIGO is the management of the enrolment of pupils in upper secondary education.

7 National statistics show an average absence rate of 15.3 lectures and 5.8 days among students in upper secondary education Norwegian Directorate of Education and Training (2018). Gjennomføring i videregående opplæring. https://www.udir.no/tall-og-forskning/finn-forskning/tema/gjennomforing2/gjennomforing-i-videregaende2/.

8 https://www.regjeringen.no/globalassets/upload/kd/kampanjer/nygiv/prosjektrapport_nygiv_2010_2013_8mb.pdf.