Relative strengths and weaknesses of countries in the TIMSS 2015 mathematics assessment framework categories

ABSTRACT

This study explored the mathematics achievement profiles of the countries participating in TIMSS 2015 based on the TIMSS assessment framework categories. More specifically, observed and expected performances of countries in the framework categories were compared to determine the relative strengths and weaknesses of countries in the respective item categories. Results from profile analysis and multidimensional scaling showed that, in general, countries with similar cultural and economic backgrounds had similar relative strengths and weaknesses. The Arab, Western and East Asian country clusters were identified considering these similarities. The difficulty and discrimination parameters of the TIMSS 2015 8th grade mathematics items were also more similar among countries in the same cluster than in countries in different clusters. The relative strengths and weaknesses of the countries in the assessment framework categories were related to countries’ human development levels. This relationship was dominant in the ‘algebra’ and ‘data and chance’ item categories. In general, ‘data and chance’ items favoured the developed countries and ‘algebra’ items favoured the less developed countries. Finally, the issue of item parameter equivalence in international surveys was discussed from a sociocognitive perspective.

KEYWORDS:

• Comparative education
• international large-scale assessments
• item response theory
• mathematics
• multidimensional scaling biplots
• profile analysis
• TIMSS

Disclosure statement

No potential conflict of interest was reported by the author.

Availability of data

Data sets used in this study come from TIMSS international database which is publicly available for use in research.

Notes

1 Technically, ${\hat{d}}_{ij}=a+b\left(\sqrt{1-r_{ij}}\right)$, where r_ij is the Pearson correlation coefficient in item parameter estimates between the countries i and j. The real numbers a and b are determined to minimize the Stress-1 measure in Equation (1).

2 The two maps on Figure 1 are Procrustean-fitted to eliminate meaningless differences between the configurations that may be attributable to the rotation, translation, and dilation of countries.

3 p values are obtained by a permutation test of 500 replications which generate MDS configurations from randomly generated data sets.