Three conjectures about school effectiveness: An exploratory study

Abstract

In this article, we address three broad conjectures about what really matters with respect to school effectiveness. Our review of previous evidence prompted us to look at three sets of factors connected with classroom teachers, school policies and processes, and matters of governance. All three have featured prominently in the public arena. In particular, we look for the relative contributions of teacher-, school-, and governance indicators for educational effectiveness (measured by Math achievement). About 100 Dutch primary schools form the database together with findings of international school effectiveness research (studies, reviews, and meta-analyses). School-level variables are the most substantial in explaining educational effectiveness. The sector effect (public/private) explains 16% of the between school variance, other school-level variables explain 51%, and the teacher- or classroom-level variables explain 32%. Some of the underlying variables are identified and we address three broad conjectures about what really matters with respect to school effectiveness.

Keywords:

• school effectiveness
• teacher effectiveness
• governance
• sector effect
• multi-level analysis

Public Interest Statement

We address three broad conjectures about what really matters with respect to school effectiveness. Our review of previous evidence prompted us to look at three sets of factors connected with classroom teachers, school policies and processes, and matter of governance. All three have featured prominently in the public arena. In particular, we look for the relative contributions of teacher-, school-, and governance indicators for educational effectiveness (measured by Math achievement). About 100 Dutch primary schools form the database together with findings of international school effectiveness research (studies, reviews, and meta-analyses). School-level variables are the most substantial in explaining educational effectiveness. The sector effect (public/private) explains 16% of the between school variance, other school-level variables explain approximately 50%, and the teacher- or classroom-level variables explain 32%. Some of the underlying variables are identified and we address three broad conjectures about what really matters with respect to school effectiveness.

1. Introduction and research problem

Research on school effectiveness has tended to go in phases and, on occasion, to follow fashions. We have found this problematic. In this article, we address three broad conjectures about what really matters with respect to school effectiveness. Our review of previous evidence prompted us to look at three sets of factors connected with classroom teachers, school policies and processes, and matter of governance. All three have featured prominently in the public arena. The vehicle through which we test these is a relatively small-scale study of Dutch primary schools in which considerable amounts of data were collected on all three areas.

Findings of school effectiveness research show that the concept of educational effectiveness is conditioned by the context of schooling. Predictors of effectiveness differ between countries (Hendricks, 2008; Hofman, Hofman, & Gray, 2010; Lockheed & Verspoor, 1991; Scheerens, 2001; Teddlie & Reynolds, 2000) between regions, school types and school populations (Patrinos & Psacharopoulos, 2011; Scheerens, 2008) and also between sector (public/private) of the schools (Coleman & Hoffer, 1987; Daly, 2005; Hofman, 1995). Moreover, conditions of schooling differ strongly between developed and (post) industrialized countries. Nevertheless, research on school and teacher effectiveness can be supportive for improving education in both contexts. There are several indications that some principles of schooling are applicable universally, while others are much more sensitive to local and cultural variation. However, research from all over the world seems to indicate that schooling is much more similar than different across countries and cultures (Brophy, 2000; Creemers & Reezigt, 2005; Hanushek & Woessman, 2007; Hofman, Hofman, & Gray, 2008; TALIS/OECD, 2009; Willms & Somer, 2001).

Research on effective schools and effective instruction generated a quantity of correlates of effectiveness. Different kinds of research and research traditions (outlier, survey, case study, correlational, and model steered research) sum up correlates that are supposed to explain differential effectiveness of classrooms and schools in their outcomes on schooling (Creemers, 2006; Daly, 2005; Lomos, Hofman, & Bosker, 2011a, 2011b). An important finding of these reviews is that it is not one isolated factor that contributes to the effectiveness of schools but several different factors at different levels in the school (multi-factor design).

Hattie (2009) has undertaken a synthesis of more than 800 meta-analyses including 50,000 studies regarding influences on student achievement and includes contributions regarding student, home, school, teacher, curricula, and finally teaching approaches. For example, important factors at teacher level are: teacher quality and teacher expectations, teacher–student relationships, and teacher professional development. However, Hattie also claims that the current mantra that teachers make a difference is misleading as not all teachers are effective and not all teachers have powerful effects on students (Hattie, 2009, p. 108). A similar argument can be made regarding school leadership; not all types of school leadership are related to student achievement; some do better than others. These are both examples of the universal relevance of concepts like school leadership and teacher effectiveness, while at the same time, they show the sensitivity of some of these concepts to local circumstances.

This study focuses on this context sensitivity of effectiveness factors from a multi-factorial and multi-level point of view while also taking into account the influence (relative weights) of different social contexts of schooling on educational performance of pupils. We formulate the following research questions:

1. Which teacher-, school-, and governance indicators contribute significantly to educational effectiveness?

2. What indications, if any, do we have about the context sensitivity of these effectiveness factors?

Our research is based on almost 100 primary schools in the Netherlands to answer the first research question. Subsequently, findings of international school effectiveness research (studies, reviews, and meta-analyses) are used to investigate the second one regarding the context sensitivity of the relevant factors and social contexts of schooling that have been found to contribute to educational effectiveness.

2. Indicators of educational effectiveness

To answer the first research question, we started to look for factors that should be included in our research on Dutch primary schools from the perspective of the contribution to educational effectiveness. We considered three broad areas in greater detail: teacher/classroom effectiveness, school effectiveness, and governance effectiveness.

2.1. Impact of teacher and classroom effectiveness factors

The organization and governance dimensions of schooling should map backwards from the concept of teaching and learning (Murphy, 1991). This implicates the instructional components of learning theories that can be seen as the basis for school effectiveness models. Within Carrol’s original instruction model (1963), the academic learning time (ALT) developed by the teacher in the class setting constituted the crucial factor. ALT is a derivative of the available time for learning divided by the quality of the instruction method and the capacity of the pupil. The distinction between available and effective learning time is an important one. Teacher management behavior will affect the effective learning time, as will his or her grouping system and instruction methods. Some instruction practices have a strong impact on school performance: reinforcement or rewards for correct responses or performance, remedial teaching on reading, cooperative learning, and person oriented counseling. Moderate associations with student performance were found in teacher expectations, individual instruction, and the use of “advance organizers.” Instruction time shows a strong correlation with student achievement and seems to be a prerequisite, though not sufficient, condition with respect to an optimal student performance (Scheerens, 2008).

The TALIS/OECD (2009) study provides an internationally comparative perspective on the conditions of teaching and learning in 23 participating countries (OECD and partner countries). The findings show that “At least half of teachers in most countries spend over 80% of their lesson time on teaching and learning. However, one in four teachers in most countries loses at least 30% of their lesson time. Country and school level differences are in this respect less important than differences among teachers within schools” (TALIS/OECD, 2009, p. 88). Creemers and Reezigt (2005) produce three overarching categories that contribute to the outcomes of instruction: the quality of the teachers’ instruction, time to learn and opportunity to learn. Furthermore, they expect that the school level will provide the conditions for effective instruction. They postulate that conditional factors for the quality of instruction are the rules and agreement or cohesion in the school concerning grouping procedures, curricular materials and teacher behavior.

In the latest model, the so-called dynamic model, Kyriakides and Creemers (2008) refer to eight effectiveness factors which describe teachers’ instructional role: orientation, structuring, questioning, teaching modeling, applications, management of time, teacher role in making classroom a learning environment, and classroom assessment. Regarding this key role of the teacher, Hattie’s (2009) meta-meta-analysis showed that the strongest predictors regarding teacher characteristics are: quality of teaching, reciprocal teaching, teacher–student relationships and providing feedback (Cohen’s d effect sizes > .70).

2.2. Impact of school effectiveness factors

According to Creemers and Reezigt (2005), there should be a policy at the school level to acquire consistency in the learning process and a policy of monitoring pupils’ achievement. Stability in the school team can be seen as a conditional factor for realizing this school policy (Hofman & Dijkstra, 2010). Conditional to time and opportunity to learn are different school effectiveness factors like the setting of homework by the teacher and good contact between home and school, an orderly school climate with clear rules, and high expectations. Moreover, Hofman and Hofman (2011) found strong differences in effective management styles between schools with different student populations.

Next to that, school effectiveness factors are found especially in the concept of school culture. School culture is connected closely with attitudes, values, and school climate. It represents factors such as expectations for student achievement, academic pressure, and emphasis on basic skills. Evaluation mechanisms of the education process emerge in many theories and models as an essential prerequisite to effectiveness (Lomos et al., 2011a). At the school level assessing pupils’ progress and the way teachers function is one of the pillars of educational leadership (Scheerens, 2008; TALIS/OECD, 2009).

A highly relevant factor with respect to school culture is community participation. This factor refers to the extent that participation of the local community, particularly the parents, is stimulated as an integral part of the school program. When educators involve minority parents as partners in their children’s education, these parents appear to develop a sense of efficacy which communicates itself to the children, with positive academic consequences. It seems possible to generate rather important effects on the educational attainment of minority pupils when school policy does not exclude parents from the school, but stimulates parental involvement in the curriculum (Hattie, 2009; Hofman, 1995; Teddlie & Reynolds, 2000). Research in primary education showed that the involvement of parents in school policy matters correlates positively to academic performances (Hofman et al., 2008).

2.3. Impact of the governance structure

Two different research lines seem to be relevant in explaining the possible impact of governance structure on school effectiveness. The first research line focuses on the (different) bureaucratic structures of public and private schools (Chubb & Moe, 1990; Hannaway, 1991; Hofman et al., 2008). Hofman (1995) indicates that the influence of the school community on decisions of the school boards seems to differ in line with the different governance structures and achievement levels of public and private schools. As an explanation of this finding, they point at the relationship between contextual characteristics of schools and the influence of the surrounding school community on school performances.

The second research line attempts to explain these differences by drawing attention to the social resources available in so-called functional communities that surround private, religious schools (Coleman & Hoffer 1987). A functional community includes a social and cultural network characterized by structural consistency in which an inter-generational network shapes social norms and social structures as well as their relationship. This perspective on schooling views the school as an extension of the family. The school is in loco parentis and bears the responsibility to carry out the parents’ will, transmitting the culture of the community from the older generation to the younger one (Coleman & Hoffer, 1987, pp. 3–4). Although some cross-national research (Dijkstra, Dronkers, & Hofman, 1997) seem to support this functional community theory, little is known yet about the mechanisms of transition between the characteristics of the community and the school’s influence on student outcomes (Sugarman & Kemerer, 1999).

To summarize, our review of earlier research alerted us to a number of possible conjectures for explaining variations in school performance.¹ At first glance we found each of the broad areas we considered equally plausible. Our concern in the rest of this article is to make sense of these various conjectures in the light of empirical evidence about their relative contributions to explaining variations.

3. Method

3.1. Sample and data collection

Different surveys were constructed to provide information for the different levels under study, that is, the relationship between parents and school staff, characteristics of school boards, school, teacher and classroom indicators of school effectiveness, and math test scores of pupils in the eighth grade of primary schools (age-group 11 or 12 years old). The original data-set consisted of 250 Dutch randomly selected primary schools. A subset of these schools was approached to obtain additional information regarding the governance structure on the one hand and teacher variables on the other. After combining these data-sets 90 schools remain from the population of Dutch primary education schools. This subset of schools shows no deviations from the original sample on crucial (aggregated) pupil-level variables like socioeconomic status, gender, and intelligence, nor with respect to school culture variables. Our data represent therefore a reliable national sample of Dutch schools for primary education. At micro-level about 700 pupils from grade eight (age-group 11/12 years old) were sampled; within every school one classroom was randomly chosen. Only pupils with all scores were included in the data-set. There are some very small schools in the sample with only a few pupils in a classroom resulting from the national policy of one school in each village. However, multi-level modeling takes into account these small sample sizes. Pupil achievement was measured with standardized achievement tests for math performances. Testing was conducted classroom-wise for both IQ and math achievement on the same day.

3.2. Hierarchical structure

The data-set used in this study has hierarchical structure. Accordingly, we assume a two-level structure (schools and pupils nested within schools). However, since research into school choice has found that self-selection may cause differences between the pupils’ populations of schools, possible individual correlates with math achievement will be taken into account. Pupil-level variables, such as gender, socioeconomic background, intelligence, and former achievements will therefore be handled as covariates in the model of analysis.

3.3. Variables

3.3.1. Individual variables

The next table describes the statistics for the pupil variables (covariates and dependent) at the individual level.

Table 1 shows some difference in the pupil covariates that relate to the sector effect. The student intake of public schools in the Netherlands shows the lowest IQ scores and lowest SES-scores. From the three categories of private schools, the pupils from neutral schools score highest on these covariates.

Table 1. Descriptive statistics of pupil-level variables and sector

	Mean	Std. Dev	Min	Max
Mathematics	103.2	13.6	58.9	122.4
Intelligence	103.7	14.3	54.5	147.8
Socioeconomic status	3.5	1.3	1.0	6.0
Gender	Male = 1
	Female = 2
	Mathematics	IQ	SES	N
Public	99.2	102.6	3.3	177
Catholic	105.2	104.3	3.5	349
Protestant	102.5	103.5	3.4	148
Neutral (secular)	105.6	105.6	3.8	24

At the output level, pupil attainment scores in arithmetic were measured with a standardized achievement test developed by CITO, the National Testing Institute. The arithmetic tests included mental and interpretative arithmetic, measurement, percentages, figures, and proportions. Pupils in the public schools show the lowest scores on the math test, while the catholic and neutral schools score highest.

3.3.2. Governance variables

The governance structure of school boards was measured using structural and cultural dimensions.

The structure of governance was measured with surveys administered to the chair and secretary of the school board. This structural side relates to indicators like the scale of governance (scale: how many schools does one board govern) and the board’s financial policy (finpolicy). The financial policy of the school board is measured using a scale of 13 items (α = .82) with the same question for each item: “how frequent is the school board involved with …” (for example) getting extra finances for the school they govern’ or with … “buying new learning materials for the school they govern.” Furthermore, we used a third indicator of the structural side of the school board. The variable regular represents how frequently the school board has regular meetings with various school parties like the principal, the teachers, the participation council, the parents’ council, or non-teaching personnel (five items with α = .80).

The cultural side of the functioning of school boards focuses more on the climate or external orientation. The indicator (influence) measures the influence that the school board ascribes to different members of the school like the principal, the teachers, the participation council, the parents’ council, and parents in general on the decisions of the school board. This influence was measured on a scale from 1 (no influence) to 4 (a lot of influence) on decisions of the school board (α = .73).

3.3.3. School variables

Several indicators have been used to measure school culture and school policy towards parents at the institutional level. School policy towards the relationship with home environment or the parents constitutes the second dimension at the institutional level. The participation of parents (oudpars) in the school focuses on concrete activities of parents (extracurricular activities, in lessons, and organizational aspects) and the influence of parents on the policy, organization, and educational goals of the school. Oudpart refers to the frequency of participation of parents in the school, whereas oudco refers to the cohesion in the school team on participation of parents.

The second dimension regards the school culture. This concerns variables that are conditional on the quality of instruction such as the degree of achievement oriented policy in the school (sexpect) and emphasis on basis skills (sbasics). There should be a degree of pupil evaluation oriented policy in the school policy to acquire consistency in the learning process and a policy of monitoring pupils’ achievement (evaluation). Stability in the school team (teamsta) can be seen as a conditional factor for realizing this school policy. The factor time and opportunity to learn consists of variables like the degree of orderly school climate with formally stated school rules (order and srules). In Table 2, the psychometric statistics for the variables at the school and governance level are presented.

Table 2. Psychometric statistics school-level variables

Variable	Mean	Std. Dev	Min	Max	Items	Reliability
Governance structure
Scale	5.93	8.22	1.00	44.00	1	–
Finpolicy	.04	1.05	-2.18	2.83	13	α = .83
Regular	6.25	2.31	2.00	10.00	5	α = .80
Influence	2.05	.45	1.38	3.38	8	α = .73
School–parent policy
Oudpars	2.84	.43	1.71	4.00	4	α = .80
Oudpart	2.59	.42	1.75	4.00	3	α = .71
Oudco	4.44	2.86	.00	18.00	1	–
School culture
Sexpext	2.52	.62	2.00	4.00	2	Sum
Sbasics	3.16	.69	2.00	5.00	6	H = .53
Order	4.49	.41	2.91	5.00	11	α = .83
Srules	1.97	.65	1.00	4.00	4	H = .67
Evaluation	4.18	1.27	1.00	6.00	3	Sum
Teamsta	73.81	22.06	.00	100.00	1	%

3.3.4. Teacher and classroom variables

At the classroom level, indicators have been used to represent the three overarching categories that, hypothetically, will contribute to the outcomes of instruction: the quality of instruction, time to learn, and opportunity to learn. The quality of the instruction process is represented by the clarity of classroom rules (crules), by the degree of regular feedback on achievement to pupils (feedback), by the influence of the teacher on the learning attitudes of the pupils and the freedom given to organize his/her learning process (independ), by participation of teachers in the decision-making process (participate), and by the achievement orientation (cexpect) and working together of teachers (together). Time and opportunity to learn is measured as the percentage of time spent on basics like arithmetic and language (cbasics) and also by the setting of homework by teachers (homework), the efficient planning of the instruction process (planning) and the planning by teachers of work by pupils with learning problems (planproblem).

Most of the indicators are measured by means of Likert scales which proved sufficiently reliable (reliabilities between .66 and .91) or, when an underlying hierarchy among items was assumed, with reliable Rasch scales (H between .43 and .67). Sometimes sum scores (in case of “hard” frequency measures e.g. number of contacts of teachers with parents) were used. In Table 3, the psychometric statistics for the classroom-level variables are presented.

Table 3. Psychometric statistics classroom-level variables

Variable	Mean	Std. Dev	Min	Max	Items	Reliability
Instructional quality
Crules	2.82	.25	1.50	3.00	8	α = .87
Feedback	2.76	.45	1.67	3.83	12	α = .72
Independ	2.22	.46	1.50	3.63	8	α = .73
Participate	31.08	6.53	12.00	40.00	8	α = .92
Cexpect	2.36	.58	1.22	3.89	9	α = .75
Together	1.06	.92	0	3	3	H = .38
Time and opportunity to learn
Cbasics	50.84	10.82	28	100	2	Ratio
Homework	1.97	.43	1.0	3.33	3	Sum
Planning	65.08	15.92	20	100	4	%
Planproblem	2.13	1.17	0	4	4	H = .43

4. Methodology: multi-level modeling of the research problem

The modeling strategy will be that, first of all, a so-called empty model will be estimated in which the pupil- and student-level variance will be determined. Next, the covariates model will be fitted, containing individual-level variables, such as intelligence, ethnic and socioeconomic background, and gender. We estimate the sector effect, which is represented by the factor “denomination” consisting of four sector categories: public, catholic, protestant, and neutral schools, in a separate model. The proof of the pudding will be the introduction of the governance structure, school, and classroom factors in three separate variable blocks.

Multi-level analyses consisting at the pupil level and school levels were conducted with the Varcl computer program. The following models are included in the analyses.

(A) Empty, covariate, and sector model:

• Model 1: Estimates of variance components for individual and school level

• Model 2: Pupil covariates

• Model 3: Sector effect: denomination

(B) Models including school level:

• Model 4: Governance structure of the school

• Model 5: School policy towards relationship with home environment/parents

• Model 6: School effectiveness factors; school culture: school policy towards norms, climate, and achievement

(C) Model including the teacher/classroom level:

• Model 7: Classroom effectiveness factors: quality of instruction, time, and opportunity to learn

Due to data collection procedures, we were not able to separate the classroom, school organization, and governance level in the multi-level analyses. Therefore, we used a stepwise input procedure of clusters of variables which gives an indication of the relative contribution of these factors.

5. Results

5.1. Basic variance components and individual-level effects

The so-called empty model (see the first column in Table 4) is the first model fitted. The outcomes of this model show significant between school variance (43.0) in math achievement scores, thus it makes sense to search for governance and school and classroom-level correlates that might account for this school-level variance. The intra-school correlation coefficient, that is, the indicator of the percentage of variance at group level is .23 (school variance 43.0/total variance 189.6). So, 23% of the total variance in math scores is due to school-level factors. This significant amount of between-school variance also indicates that the math scores at the aggregated school level, in terms of a 10% prediction interval, will vary between 89.7 and 115.5 points.

Table 4. Results of the multi-level regression

	Model 1		Model 2		Model 3		Model 4		Model 5		Model 6		Model 7
	Empty		Pupil		Sector		School I: governance		School II: parents		School III: culture		Classroom
(1) Grand mean	102.3		43.2		40.6		34.6		34.7		25.2		8.1
(2) IQ			.54	(.03)^**	n.s.d.		n.s.d.		n.s.d.		n.s.d.		n.s.d.
Male			.00	(.00)
Female			−1.4	(.8)^*
SES			1.3	(.3)^**
(3) Public					.00	(.00)	.00	(.0)	.00 (.00)		.00	(.00)	.00	(.00)
Catholic					4.6	(1.5)^**	−.80	(2.4)	−1.4 (2.4)		−1.9	(2.2)	−2.2	(1.8)
Protestant					2.0	(1.7)	−2.8	(2.5)	−2.9 (2.5)		−3.7	(2.3)	−2.8	(2.0)
Neutral					4.0	(3.2)	.51	(3.9)	.40 (3.5)		−.48	(3.4)	−1.1	(2.9)
(4) Regular							.50	(.40)					.72	(.29)^**
Influence							3.3	(1.5)^**			4.2	(1.3)^**	3.2	(1.0)^**
(5) oudpart									1.6	(1.4)	2.6	(1.3)^**	2.6	(1.0)^**
oudco									.01	(.21)			.53	(.16)^**
(6) Evaluation											2.6	(.48)^**	2.5	(.45)^**
(7) Rules													3.7	(2.0)*
Homework													−2.4	(1.3)*
Together													−1.8	(.61)**
Planning													.09	(.03)**
planprobl													1.4	(.44)**
School variance	43.0	(6.6)	25.1	(5.0)	21.2	(4.6)	16.7	(4.1)	16.4	(4.0)	8.3	(2.9)	.2	(.4)
Student variance	146.6		83.2		83.2		83.1		83.2		82.7		83.4
Gender/school*	–		12.3	(3.5)	10.9	(3.3)	10.8	(3.3)	10.1	(3.2)	10.6	(3.3)	9.7	(3.1)
Deviance	5562		5185		5175		5162		5160		5130		5086
Percentage school variance**	–		–		16%		33%		35%		67%		99%

Notes: Standard error in parentheses; only significant effects.

Gender/school: gender school variance; Percentage school variance: percentage explained school variance.

^* Coefficient significant at .05 level.

^** Coefficient significant at .01 level.

The pupil model shows three significant effects. Pupils with higher intelligence levels, from higher socioeconomic family backgrounds, and boys, perform relatively well. Introduction of the gender variable makes clear that there are also random effects, that is, schools differ in their success for boys and girls. The variance components concerned with this between schools’ gender effects are included in the model (see bottom lines of Table 4).

5.2. Governance effects

The governance model contains first school sector: public, catholic, protestant, and neutral schools. The public schools serve as baseline group in the analysis, they also score lowest of the four school sector categories on mathematics achievement. In upward order, we distinguish protestant and neutral schools. However, it is the catholic schools that show a significant higher achievement level than the public schools.

The new structural variable of school governance shows a significant impact on math achievements, that is, how frequently the school board has regular meetings with various school parties (regular). On the cultural side of the functioning of school boards, it is the influence of members of the school community on decisions of the school board (influence) that plays a significant role. Remarkable here is the mediating effect: the positive impact of attending a catholic school disappears after inclusion of this structural school governance factor. This finding expresses that in catholic schools, the influence of members of the school community such as the principal, the teachers, and certainly also the parents on decisions of the school’s governing body is stronger than in the other three school sector categories and especially strong compared to the public schools. The explained school variance increases by 17 points from 16 to 33%.

5.3. School effects

A block of variables regarding the school policy towards the relationship between school and home environment is incorporated in the analysis. None of the included variables shows a substantial impact on math achievements and neither do they have an impact on the sector effect. School effectiveness variables which are viewed as conditional to instructional effectiveness (the second variable block) exert a strong impact on math achievements. In particular, it is the emphasis in school policy on evaluation and monitoring of pupil performances (evaluation) which shows a significant effect. Interestingly, the incorporation of school culture in the model also leads to some changes in the effects of the school policy towards the relationship between school and home environment. The frequency of participation of parents in the school (Oudpart) becomes a significant factor at this stage of the analysis. This indicates some sort of interaction between parental involvement and school culture. It may be that this emphasis on evaluation and monitoring stimulates and implicates more school/parent contacts.

Inclusion of school culture leads to a) negative coefficients for the original positive effects on math for the private, especially the catholic, schools (again a mediating effect) and b) a quite substantial increase in explained between school variance, that is, with 32 points from 35 to 67%.

After inclusion of the school-level factors, we do not observe a significant decrease in the random variance component regarding gender. In other words: the school-level factors, such as, “the influence of members of the school community on the decisions of their school board” (influence) and “the degree of pupil evaluation oriented policy of the school” (evaluation), do not produce an egalitarian effect.

5.4. Teacher and classroom effects

Finally, we included the teacher and classroom effectiveness factors in the model. The expectation that the way teachers instruct their pupils and manage their classroom will affect math achievement significantly is affirmed by the results shown under column seven in Table 4. The extent to which teachers work together (Together), teachers use efficient planning procedures concerning the instruction process (Planning), and teachers’ make use of efficient planning procedures concerning pupils with learning problems (planproblem) exert strong significant effects on math achievement. We observe that teachers who work together in regular school activities (together) produce better results on math than teachers that work together only in extracurricular activities.

Furthermore, our model shows moderate effects on math achievement of classroom variables from the quality and quantity of instruction dimensions. An expression of the latter is the extent to which there exists clarity about the classroom rules (rules). Illustrations of the importance of time and opportunity to learn factors are the focus of teachers on curriculum basics like arithmetic and language (basics) and the extent to which teachers give homework systematically (homework). The joint impact of these factors on math achievement is large as is clearly expressed in the additional explained variance of 32 points (from 67 to 99%).

Moreover, after inclusion of the classroom factors, we observe a decrease in the random variance component on gender. The classroom factors, such as cooperation among teachers, teachers’ efficient planning, feedback on achievement, clarity of rules the teacher sets for his or her classroom, seem to produce some egalitarian effect concerning gender-specific achievement.

6. Summary and implications

The findings show that the school (level) variance component makes up 23% of the total variance in math achievement of pupils. When we examine the different portions of variance explained by the various blocks of variables in our analysis, it is apparent that the school variables are the most substantial. The sector effect (public/private) explains 16% of the between school variance, the school variables explain additionally 51% and the teacher or classroom variables 32%. Within the school-level variance, component school culture variables are dominant: governance structure takes 17% of the school variance (additional to the sector model), school–parent relationship 2% and school culture variables 32%. It is in particular the emphasis in school policy on evaluation and monitoring of pupil performances that seem important.

6.1. Implications for teachers and classrooms

The study shows that the extent to which teachers work together in regular school activities and use efficient planning procedures concerning the instruction process and concerning pupils with learning problems, exert significant effects on math achievement. Additionally, clarity about the classroom rules, the extent to which teachers give homework to all pupils, and the focus on basics show moderate effects on math achievement.

A supportive classroom climate as well as cooperative learning is well associated with the significance of collaboration of teachers that was found to be important in this study. Teachers will have to show cooperation themselves as good practices for pupils. These findings are in line with the outcomes of the synthesis of over 800 meta-analyses relating to the importance of classroom climate and cooperative learning of Hattie (2009) with moderate to strong effects for cooperative learning (Cohen’s d from .41 to .59). Furthermore, planning and working together improves the cohesion in the school which exerts positive effects on pupil performances. This aligns with outcomes from a recent review of Lomos et al. (2011a) into the possible impact of teacher collaboration, professional community and networks on student achievement. The review is based on an analysis of the quantitative studies that were performed in the last 30 years in secondary education Lomos et al. found small positive effects of professional community on student achievement (Cohen’s d between 10 and .40).

These findings imply that teacher training should dedicate a fair amount of time showing teachers effective ways to cooperate, plan, and work together with other teachers in their school. A specific reason for the importance of such teacher cooperation lies in the positive relationship of “clarity of rules” with math achievement of pupils. Teachers being part of a school team should be aware of the impact of (in) consistency of rules for classroom behavior. It should be obvious for pupils what their teachers stand for and which set of rules and norms pupils should comply with throughout the whole school. A safe and orderly classroom climate asks for a clear set of rules shared by all teachers and accordingly maintained in the same way in every classroom. Furthermore, the emphasis teachers put on curriculum basics (math, reading, and writing skills) can be seen in line with the curriculum alignment principle which puts forward the relevance of goal oriented curriculum development (Creemers, 2006).

6.2. Implications for schools: evaluation and monitoring

The evaluation factor consists of three elements which jointly indicate a school-level focus on evaluation of pupil achievement. The first element is the extent to which standardized school performance tests are administered. The second element to what extent the outcomes of these tests are being used to evaluate the adequacy of the subject matter that is being offered. The last aspect concerns the fact whether the achievement scores are part of a system of central registration. The evaluation factor expresses the awareness of the relevance of evaluating behavior of pupils and other school parties and using the generated information for adapting and improving the learning process. In this respect, Hattie (2009) found feedback to be the strongest predictor of student performance, but he also stresses that frequent testing is only effective if teachers receive feedback from the tests in order to fit their instruction to the strengths and weaknesses of the student performance (Cohen’s d = .72). Scheerens (2008) aligns with our findings as he concludes in his meta-analysis that next to effective use of learning time and an orderly and safe climate, an output oriented monitoring and evaluation policy is important for student achievement. The TALIS/OECD (2009) study also concludes that the evaluation of education (teachers’ appraisal and feedback) is important for changes in teachers’ practice to improve their teaching.

6.3. Implications for governance: school-based management

School board policies and their relationships with other parties, for example, teachers and parents are relevant in most education systems. Several studies on education management suggest that instructional leadership is associated with effective schools. However, in the setting of many developing countries principals apparently function as the lower link in an organizational chain that extends from the school through local educational boards, district supervisors to central staff (Hofman, Hofman, Gray, & Daly, 2004; Scheerens, 2001). Our study shows a positive impact of school governance on math achievement, if the influence of members of the school community on the boards’ decisions is relatively high. This finding indicates that the responsiveness of governance to the educational knowledge of staff and other parties involved in the school life, such as parents, is crucial.

In line with this finding, we observe the much discussed findings of the research by Chubb and Moe (1990) and Hofman et al. (2008) which indicates the significance of the different bureaucratic organization on variation in effectiveness of public and private (religious) schools. In this respect, it is interesting that Hattie (2009) found positive relationships between religious and charter type of schools and student performance. Our findings advocate an approach with less decision-making in the central office and board room and to give more authority to individual schools. The policy and decisions of the school board should rely more strongly on information and input from members of the school community, for example, teachers. Moreover, a “team approach” seems to be a preferred vehicle for involving others like school staff and members of the community in the decision-making process of the school or district. This finding is in line with a recent review of Lomos et al. (2011a) into the possible impact of teacher collaboration, professional community, and networks on student achievement. The review is based on an analysis of the quantitative studies that were performed in the last 30 years in secondary education. The findings show positive effects of professional community on student achievement.

It is important to note that we use math achievement as a single indicator for student achievement while in the studies conducted worldwide on educational effectiveness various (often cognitive but also non-cognitive) indicators are used. This can clearly be viewed as a limitation of the current study.

7. Conclusion

This analysis has sought to address some important but nonetheless perplexing questions that have clouded research on school effectiveness in recent years. It has become customary to argue vociferously that previous conceptions of the major sources of variance have been overturned by whatever perspective the most recent researchers have chosen to dwell on. Our research, by contrast, provides some support for a number of different viewpoints. Governance matters—the sector effect is significant in this study and there is certainly room for its continued exploration as a lever for policy development. But school and classroom variables are also highly significant and international comparative research supports the statement that these outcomes are valid for different contexts (Hattie, 2009; TALIS/OECD, 2009). In the analyses reported here, we have partitioned them into school-level and classroom-level components. The data suggests that school-level differences are important and these findings are consistent with other work on elementary school differences. At the same time, however, the evidence supports statements about differences between classrooms and teachers. They are, of course, highly interrelated. School principals establish the policy climate within which teachers can function but, by the same token, teachers’ behaviors determine the limits within which schools can develop. We started, adopting the conventional mind-set of the discipline, by asking the question which matters most? The conclusion of this exploratory study is that they all continue to have mileage.

Additional information

Funding

Funding. The authors received no direct funding for this research.

Notes on contributors

Roelande H. Hofman

Roelande H. Hofman is a senior researcher at the Institute for Educational Research of the University of Groningen in the Netherlands. She has a well-established track record as a researcher in the fields of school organization and school effectiveness, and has published various articles on these topics.

W.H. Adriaan Hofman

W.H. Adriaan Hofman is a professor in education Studies at University of Groningen, The Netherlands.

John M. Gray

John M. Gray is a professor of education at Cambridge University, United Kingdom.

Notes

1. Of course, we have to take into account that school performance is a broader concept than math achievement used in this study as outcome measure.