The informed choice: mathematics textbook assessment in light of educational freedom, effectiveness, and improvement in primary education

ABSTRACT

In many educational systems, textbooks are a leading factor for daily instructional practices, and consequently may affect educational outcomes. Whereas in many countries textbook assessment is a common practice in light of evaluating and improving educational effectiveness, in other countries these practices are absent, mainly under the realm of educational freedom. In this study, we attempt to highlight the potential contribution of textbook assessment to educational freedom, effectiveness, and improvement, by means of mathematics textbook assessment primary education (Grade 4) in Flanders. Multilevel (multivariate) modelling was used, studying the relation of textbooks with both mean achievement (overall and in different content and cognitive subdomains) and (social and ethnic) equity. Furthermore, multilevel quantile regression analyses were used to investigate the effect of textbooks in different quantiles of the achievement distribution. Implications of the results of textbook assessment are discussed in light of its contribution to educational freedom, effectiveness, and improvement.

KEYWORDS:

• Math instructional support materials
• math textbooks
• TIMSS 2015 math achievement
• social and ethnic equity
• multivariate multilevel analyses

Disclosure statement

No potential conflict of interest was reported by the authors.

Table 8. Results RQ1c for DAT: intercept and textbook regression weights.

Data availability statement

This manuscript makes use of data of TIMSS 2015, which are publicly available at https://timssandpirls.bc.edu/. It was up to the nation to decide on whether data on the textbook were collected. For the Flemish region of Belgium, these data are available upon request from the corresponding author.

Notes on contributors

Kim Bellens is Research Associate at the Centre for Educational Effectiveness and Evaluation at KU Leuven, Belgium. Her major research interests are international comparison of (trends in) quality and equity amongst educational systems and explaining existing differences across countries and time.

Wim Van den Noortgate is Full Professor at the Centre of Methodological Research for Educational Sciences at KU Leuven, Belgium. His major research interests are in multilevel modelling, meta-analysis, and learning analytics.

Jan Van Damme is Emeritus Professor at KU Leuven, Belgium. His special interests are in school trajectories and educational effectiveness from the class to the system level.

ORCID

Kim Bellens http://orcid.org/0000-0001-9695-5167

Notes

1 When we refer in this article to textbook, we refer to the textbook made available for students, together with the accompanying educational support material (e.g., workbook, worksheets, tests, teacher textbook, website, learning platform, etc.).

2 For further, detailed information on the organization of Flanders’ educational system, see Verstraelen et al. (2016).

3 Eurobasis was renewed to Kompas in 2006 (215 students of nine schools indicated using Eurobasis; 1,145 students of 39 schools indicated using Kompas). As it may be assumed that both textbooks are highly similar, they are taken into account as one textbook in our analyses.

4 In contrast to dummy coding, in which the effect of a category is compared to the effect of a(n) (arbitrary) reference category, in weighted effect coding the effect of a category/textbook is compared to the mean effect of all categories/textbooks on math achievement. The weighting of the categories results in the equal contribution of each category to the estimated mean, regardless of the (different) sizes of the categories (Te Grotenhuis et al., 2017).

5 In 10 classes, information on the textbook used was missing. For eight classes, we substituted the missing information with (non-missing) information on the textbook used in (the majority of) the other class(es) of that school. For two classes (46 students), substitution was not possible, as both schools consisted of only one Grade 4 class. These classes were removed from the data.

6 We left out data of the eight special education schools (174 students) that participated in TIMSS 2015 as textbooks play a much smaller role in special education compared to regular primary education. This is due to the high level of differentiation in educating special needs students. (Missingness on the textbook used and teachers indicating to primarily using their own material is 16.0% and 26.0% in special education schools.)

7 Due to the selective inclusion of schools in our final sample, weights calculated by TIMSS to come to terms with the two-stage stratified sample design are no longer correct, which made us decide to opt for unweighted analyses. As no systematic bias was detected in the schools that were removed from our analyses (e.g., all coming from the same educational network or belonging to the same strata in TIMSS), implications of the selective sample of schools may be rather limited. Nevertheless, this selection implies that caution is needed in generalizing results to the entire population.

8 Whereas TIMSS used a two-stage sample design (with sampling of schools in the first phase and sampling classes in the second phase), a three-level model with students nested in classes and classes nested in schools showed that the estimate of the variance of the achievement scores at the class level was small. The ICC (intraclass correlation coefficient) for MATH is .18 at the school level and .03 at the class level. The variance distribution for the content and cognitive subdomains is ICC_school= .16 and ICC_class = .03, and ICC_school= .15 and ICC_class= .02, respectively. However, we have to be prudent with the interpretation of these values, as in most countries the class and school level cannot be disentangled in TIMSS due to the fact that most schools only have one/two classes, as remarked by Rutkowski, Gonzalez, Joncas, and von Davier (2010).

9 Missing data in the independent variables included are very low, that is, 1.3% for SES, SCONF, and SEAS, 2% for gender, and 2.8% for LANG (other variables having no missingness).

10 In the analyses in R, we worked with the first plausible values, and used listwise deletion of missing cases (N = 658, 13.80%) as multiple imputation and FIML are not (yet) implemented in package lqmm.

11 In a model leaving out GROUP SES and GROUP LANG, CS showed significantly higher achievement levels compared to the mean. This may suggest that the relation between NETW and MATH is due to the existing differences in group composition of Flanders’ NETWs (or due to overcontrolling by taking into account group composition).

12 Upon request, results of Models 1, 2a, and 2b are available from the corresponding author.

13 Due to limitations of space, Table 8 includes only the coefficients of intercept and of textbook on DAT in Model 3. Upon request, coefficients of other variables included in Model 3 and other dependent variables, as well as results of Models 1, 2a, and 2b, are available from the corresponding author.

Additional information

Funding

This work was funded by the Flemish Department of Education and Training.