An Attempt to Reinterpret Student Learning Outcomes: A Cross-National Comparative Study

Abstract

Attracting significant attention from the media and policymakers, international student assessments are serving as pressures on and challenges for individual countries. Even so, paying attention only to publicly released rankings in international student assessments may lead to superficial interpretations of the assessment results. Simply equating the rankings with educational outcomes can be a questionable perspective because the assessment results can be interpreted differently. In this context, this article has explored student achievement in the Programme for International Student Assessment (PISA) tests in comparison with civic knowledge achievement in the IEA Civic Education Study across 17 countries. This article has also evaluated efficiency in educational processes by assessing outputs (student performance in PISA) against inputs (educational spending and learning time) directly used in the education system. These two approaches clarify that publicly released rankings in PISA change to some extent when the rankings are reevaluated by taking other factors into consideration. This implication should be allowed for in interpreting the results of international assessment and the relative rankings of participating countries.

Notes

The PISA evaluates not the extent to which students master a specific school curriculum but the ability to use the wider knowledge, concepts, and skills in core subjects to meet real-life situation (OECD, 2001).

Cognitive achievement is interchangeably used with student learning achievement on three core subjects (mathematics, reading, and science) in this article.

CIVED basically assesses the content of students’ civic knowledge and their civic attitudes and engagement. Specifically CIVED consists of five components: (a) civic knowledge; (b) skills in interpreting civic information; (c) concepts of democracy, citizenship, and government; (d) attitudes toward the nation, the government, immigrants, and women's political rights; and (e) willingness to participate in civic-related activities (Torney-Purta, Lehmann, Oswald, & Schulz, 2001).

CIVED 1999 was conducted in two phases. This article focuses on the second phase only in which nationally representative samples were tested on their civic knowledge and skills.

Sample students who participated in both assessments were not identical to each other. However, considering that CIVED was carried out 1 year before PISA was conducted, target populations appear to be the same. Target populations are based on the coverage of a specific age cohort: in PISA 15-year-olds and in CIVED 14-year-olds enrolled in educational institutions. Especially because this article compares national averages on international assessments, comparison of PISA 2000 and CIVED 1999 is reasonable.

PISA 2009 results in lieu of PISA 2000 are used mainly because of data availability.

The 17 participating countries are Australia, Belgium, Czech Republic, Denmark, Finland, Germany, Greece, Hungary, Italy, Latvia, Norway, Poland, Portugal, Russian Federation, Sweden, Switzerland, and the United States.

Among 17 sample countries, Latvia is at the bottom of rankings in both core subjects and civic knowledge.

TABLE 1 Student Achievement in PISA 2000 and CIVED 1999 and Rankings

Student test scores are converted to a scale of 0 to 10 by the author. Initial range of sum of test is 1400 to 1600 for core subjects and 90 to 110 for civic knowledge. Reference points in converting initial test scores are OECD average, 1500, for core subjects and international average, 100, for civic knowledge, which means that 5 on a scale of 0–10 is international average. Scores below 1400 in core subjects and 90 in civic knowledge are converted to 0, scores above 1600 and 110 to 10.

FIGURE 2 Dispersion of student achievement in PISA 2000 and CIVED 1999. Source: OECD (2001, 2010, 2011) and Schulz and Sibberns (2004). Note. AUS = Australia; BEL = Belgium; CZE = Crech Republic; DNK = Denmark; FIN = Finland; GER = Germany; GRE = Greece; HUN = Hungary; ITL = Italy; LAT = Latvia; NOR = Norway; POL = Poland; POR = Portugal; Russia = Russian Federation; SWE = Sweden; SWI = Switzerland; USA = United States (color figure available online).

Although not shown in this article, the scatter plots of mean scores in each of core subjects and civic knowledge are truly similar as the scatter plot shown in Figure 2.

The correlation between them is 0.48 with its p value of .05.

Outcomes in PISA 2003 are similar to those specified in Table 2, which is not shown in this article. The author just checked whether any big changes are observed. Poland is the most cost-effective country in PISA 2003 for which Russian Federation is unavailable.

TABLE 2 Student Performance in PISA 2009 and Cumulative Education Spending per Student

FIGURE 3 Relationship between student achievement in PISA 2009 and cumulative spending. Source: OECD (2010, 2011). Note. AUS = Australia; BEL = Belgium; CZE = Crech Republic; DNK = Denmark; FIN = Finland; GER = Germany; GRE = Greece; HUN = Hungary; ITL = Italy; LAT = Latvia; NOR = Norway; POL = Poland; POR = Portugal; Russia = Russian Federation; SWE = Sweden; SWI = Switzerland; USA = United States (color figure available online).

The average TE level is 2.21 and the average AE level is 0.46.

TABLE 3 Student Performance in PISA 2009 and Time Devoted to Learning

FIGURE 4 Relationship between student achievement in PISA 2009 and total learning hours. Source: OECD (2010, 2011). Note. AUS = Australia; BEL = Belgium; CZE = Crech Republic; DNK = Denmark; FIN = Finland; GER = Germany; GRE = Greece; HUN = Hungary; ITL = Italy; LAT = Latvia; NOR = Norway; POL = Poland; POR = Portugal; Russia = Russian Federation; SWE = Sweden; SWI = Switzerland; USA = United States (color figure available online).