Critical views of 8th grade students toward statistical data in newspaper articles: Analysis in light of statistical literacy

Abstract

Understanding and interpreting biased data, decision-making in accordance with the data, and critically evaluating situations involving data are among the fundamental skills necessary in the modern world. To develop these required skills, emphasis on statistical literacy in school mathematics has been gradually increased in recent years. The critical views of 8th graders with respect to statistical data presented in several newspaper articles were investigated from various aspects through this descriptive study. The study was conducted in a middle school with 9 students at the 8th grade level. The statistical data included in news articles selected from different national newspapers published in Turkey were used to collect the data. Through clinical interviews, the evaluations of the students were determined from the aspects of consistency, sample, data collection, data analysis, data presentation, and data support; the reasoning behind the students’ evaluations was also discussed. As a result, it was determined that the students did not perform well during their critical evaluations of newspaper articles that included research data.

Keywords:

• statistical literacy
• media
• newspaper
• critical questions
• critical thinking

Public Interest Statement

During the recent years, the importance of educating statistically literate people has become more crucial for societies. Being statistically literate is required to read, interpret and criticize statistical messages in various contexts such as radio, TV, magazine, newspaper, or www. On the other hand, information-laden earth in which we live forced policy-makers to develop a contemporary curriculum. Thus, the concept of “statistical literacy” is defined as a key concept of statistics education for K-8 curriculum. Current study aims to investigate the critical views of middle school students about news items included in the media. In order to reach this aim, we firstly selected articles in national newspapers. Secondly, the clinical interviews were conducted with 8 graders. In this step, each student was required to examine all the articles in terms of sampling, data collection, data presentation, and data support.

1. Introduction

Rapid development and transformation in the fields of science and technology have affected human life in a radical manner and lead to an increase in our knowledge. Therefore, instead of memorizing new information developed in different fields, one should develop fundamental literacy in specific fields to cope with the emerging data. To address this issue, the concept of mathematical literacy has arisen as a means for improving skills in interpreting the growing amount of data over the last century.

Although mathematics literacy has been described in different ways by various researchers and institutions, the implied meaning remains the same. In this respect, the Programme for International Student Assessment has identified mathematics literacy as the skills used in struggling with, understanding and describing mathematics, along with using those skills in order to find solutions for problems encountered in social life (OECD, 2006). According to Mkakure and Mokoena (2011), mathematical literacy, as derived from the practical applications of mathematics, is a concept that provides individuals with the confidence to solve problems through interpretation and critical evaluation of situations encountered in daily life. On the other hand, mathematical literacy has been interpreted by McCrone and Dossey (2007) as an understanding of the function of mathematics in the world that allows for forming sound decisions and constructing responses to individual needs. As can be inferred from all of these descriptions, mathematical literacy is a broad term including numerous skills, such as problem solving, analytical ability, judgment, and effective communication in the various types of situations that individuals may encounter.

Statistical literacy may be considered as a separate discipline due to fundamental aspects that are unique to statistics. According to Moore (1998), an understanding of statistics concepts such as data, change and chance is necessary for coping with situations that arise in everyday life; this importance has led to an increased emphasis on statistical literacy. According to Gal (2004a), statistical literacy refers to the skills that people use for interpreting data, evaluating them critically, explaining ideas stemming from statistical information, arguing the data, and being aware of related phenomena. Watson (1997) notes that these skills include understanding of probabilistic and statistical terminology, understanding of concepts in social discourse and statistical language, and inquiring into the attitudes formed for counter-situations. Likewise, according to Wallman (1993), statistical literacy entails understanding statistical results related to everyday life and the ability to make evaluations through a critical approach; proficiency in statistical thinking enables individuals to make professional and individual decisions, both in society and internally.

An examination of the related literature reveals frequent references to four dimensions of statistical literacy. These may be summarized as (1) understanding of statistics; (2) interpreting statistical data; (3) decision-making; and (4) developing critical positioning skills. In this respect, an understanding of statistics denotes having knowledge of statistical concepts in given situations, articulating these concepts, and analyzing and giving responses to problems related to these concepts (Chance, 1997). Furthermore, interpreting statistical data refers to the process undertaken by individuals both to understand the meaning of data that they encounter and to interpret data they have themselves collected when required for a particular situation (Rumsey, 2002). Since making inferences concerning the possibilities that may arise based on interpreted data entails making predictions for the future, people tend to interpret the statistical concepts that they understood through using statistical relationships, and then reach to decisions through making predictions in consideration of these relationships (Mosenthal & Kirsch, 1998).

However, purposefully or not, some data that may be encountered may include misleading, unilateral, and biased opinions. Therefore, in order to develop a more objective view, it has been emphasized that passively interpreting data is not satisfactory, and that it is important to take an interrogative and critical stance toward potentially misleading data (Frankenstein, 1989; Wallman, 1993). In this respect, the skill of critical positioning requires knowing which questions should be asked in a particular situation (Snell, 2002).

In the results of the International Adult Literacy Survey, it was reported that societal change has led to an increasing demand for developing statistical literacy skills. For instance, as Gal (1994) claims, workers may feel a need for statistical literacy in order to increase the quality of a process in order to respond to rapidly increasing demand. Likewise, this requirement applies to people who are data consumers, who may have a need to interpret discount rates in television and newspaper advertisements; newspaper circulars; statistical information related to political decisions; and so on. With this type of reporting, all of the information includes a message and aims to form a picture in the minds of consumers; these messages may be intended to make the consumer believe in an opinion or to convince them to accept a unilateral view by hiding pieces of the whole picture and presenting manipulated information (Clemen & Gregory, 2000). Therefore, in order to evaluate the validity and conclusiveness of the information, critical thinking skills are necessary (Beyer, 1987).

1.1. Critical thinking

Dewey (1909, as cited in Fisher, 2001) identified critical thinking as the processes of thinking about a situation, as well as the information and the beliefs that support it, in an active and careful manner. In this sense, Norris (1985) explained critical thinking as the evaluation of self-thinking skills and the behavioral changes that occur as a result of this evaluation. Furthermore, Ennis (1985, p. 45) described critical thinking as a logical and reflective process used in making decisions about “what to do” and “what to believe,” which Norris and Ennis (1989) outline a three-stage process wherein (1) critical thinking begins with a problem statement and an interaction with the environment to solve the problem; (2) an inference is made through reasoning and establishing a relationship using the previous information; and (3) a decision is made about what to believe as result of the process.

According to Aizikovitsh-Udi, Kuntze, and Clarke (2016), people who are able to think critically and statistically literate are prerequisites for democratic countries. Although there is a direct relation between two important areas (Royalty, 1995), researchers in the literature mainly focus on either critical thinking or competencies of statistical literacy. However, being critical in statistical context is not about an attitude, but also related to interpret and evaluate the given statistical data using some “abilities” (Aizikovitsh-Udi et al., 2016, p. 119). Considering the average Turkish watches more than four hours of television per day, as well as the 5 million newspapers sold per week and the 32 million Internet customers, (TurkStat, 2013), the average person is likely to be bombarded with information. This information, as Wallman (1993) contends, frequently contains statistical data that is biased, in terms of both special and public foundations. Among the reasons for these biases, use of unfamiliar technical terms and misleading presentation of the news have been cited (Gal, 1999; Laborde, 1990). Therefore, evaluation of the statistical data that people confront matters.

1.2. Literature review

In a study conducted by Gelman, Nolan, Men, Warmerdam, and Bautista (1998) in the context of an introductory statistics course, the researchers collected articles over a one-month period from two different newspapers (the New York Times & the San Francisco Examiner); these were published in the United States and included scientific studies and statistical data (e.g. graphics, table, etc.). The articles were given to students, who were asked to analyze them. The students’ critiques of the given articles were examined according to previously assembled and prepared questions. The students were then required to find a newspaper article and write a critique of the content. The students who participated in the study stated that the data placed in the newspaper articles included incomplete information used to sway their opinions.

In a similar study, Cerrito (1999) distributed various news articles and asked students to analyze them in the scope of a statistics lesson. It was observed that the students interpreted the data provided in the articles according to their personal beliefs. In this case, the researcher found that the students neglected the persuasiveness of the articles; furthermore, they failed to infer the purpose of the articles at the beginning of the lesson. As a result, Cerrito suggested that these types of articles may be useful in developing students’ critical thinking skills.

An investigation carried about by Watson and Moritz (2000) focused on student understanding and evaluation of newspaper articles around the concept of “sample” within the dimensions of the SOLO taxonomy: i.e. pre-structured, single-structured, multi-structured, and establishment of relationship. The consistency rate of the student responses for questions asked at the 3rd, 6th, and 9th grade levels were higher at the upper grade levels; it was concluded that the vast majority of the students had a lower level understanding of the concept of sample. Looking at the issue from another angle, Pfannkuch (2005) aimed to identify the types of problems that students encountered in the course of statistical evaluations and to determine the ability levels of the students with respect to the SOLO taxonomy. In this case, data found in newspaper articles were converted into tables and graphs and given to students, along with some critical questions. According to the results of the study, most of the students were ranked at the first and second level, or the pre-structured and single-structured level, respectively, and a limited number of students were able to reach the relationship and abstract structure level with respect to the SOLO taxonomy.

In another study, Schield (2006) presented results from an international project, conducted by W. M. Keck Statistical Literacy Project. The focus of the survey was reading statistics or rates of the percentages given in tables and graphs. The results showed that, participants of the study who were college teachers, college students, and data analysts had difficulties on reading statistical data. For instance, in reading an X-Y plot, about 4/5 of the college teachers misread a—times more than—comparison. Similarly, Özmen (2015) observed instructors and students teaching or attending statistics courses in 9 programs of 7 faculties during a year. In order to evaluate the outcomes of the course, she developed a statistical literacy test and conducted Rash analysis. Consequently, she found that many of the participants showed low performance in the test. In addition to determination of the current issue, some researchers focused on how to improve statistical literacy. Koparan (2012) investigated the effect of project-based learning approach on 8th grade students’ statistical literacy. He proved the effectiveness of project-based learning on students’ literacy levels toward probability concept in his experimental research. Merriman (2006) also designed a statistics course for 14 year old students using media reports and investigated the effectiveness of the intervention on student achievement. Results of that study also revealed a significant improvement in students’ statistical literacy at the end of the implementation process. In addition to summarized articles above, some studies introduced classroom activities or theoretical backgrounds (e.g. English & Watson, 2015; Watson, 2013).

A limited number of studies have been carried out in different countries in relation to students’ understanding, interpreting, and critical evaluation of data presented in the media (e.g. Gelman et al., 1998; Merriman, 2006; Pfannkuch, 2005). These studies reveal a need for further research to be carried out with different populations. Accordingly, the present study aimed to investigate the critical perspectives of 8th students toward newspaper articles including statistical data.

1.3. Theoretical framework

As a result of his research related to knowledge and disposition elements, Gal (2004a) classified statistical literacy according to two aspects, as shown below in Figure 1. The interests and attitudes of individuals toward given situations comprised the statistical data related to the dispositional element, while the knowledge element compromised literacy skills, statistical knowledge, mathematical knowledge, content knowledge, and critical perspective.

Figure 1. A model for statistical literacy (Gal, 2004a, p. 67).

According to Gal (2004a), knowing and interpreting statistical information is not enough in itself. It also requires accompanying reading-writing skills, as well as mathematical and content knowledge; and even these are not enough to perform evaluations. Rather, in addition to these, individuals need to ask “critical questions;” Gal identified the critical questions needed to evaluate statistical information in newspaper articles as follows (Table 1):

Table 1. Critical questions

1. Where did the data (on which this statement is based) come from? What kind of study was it? Is this kind of study reasonable in this context?

2. Was a sample used? How was it sampled? How many people actually participated? Is the sample large enough? Did the sample include individual units that are representative of the population? Is the sample biased in some way? Overall, could this sample reasonably lead to valid inferences about the target population?

3. How reliable or accurate were the instruments or measures (tests, questionnaires, interviews) used to generate the reported data?

4. What is the shape of the underlying distribution of raw data (on which this summary statistic is based)? Does it matter how it is shaped?

5. Are the reported statistics appropriate for this kind of data? For example, was an average used to summarize ordinal data? Is a mode a reasonable summary? Could outliers cause a summary statistic to misrepresent the true picture?

6. Is a given graph drawn appropriately, or does it distort trends in the data?

7. How was this probabilistic statement derived? Are there enough credible data to justify the estimate of likelihood given?

8. Overall, are the claims made here sensible and supported by the data? For example, is correlation confused with causation, or a small difference made to loom large?

9. Should additional information or procedures be made available to enable me to evaluate the sensibility of these arguments? Is something missing? For instance, did the writer “conveniently forget” to specify the base of a reported percent-of-change or the actual sample size?

10. Are there alternative interpretations for the meaning of the findings or different explanations for what caused them; e.g. an intervening or a moderating variable affecting the results? Are there additional or different implications that are not mentioned?

Following Gal’s (2004a) suggestion, some of the critical questions were simplified in this case in consideration of the purpose of the study, the existing knowledge, and the life context of the students who formed the sample according to the experts’ (2 university mathematics educators) opinions. In this study, the students’ interpretations of statistical data were investigated in consideration of how often they used the simplified critical questions classified in the knowledge element of Gal’s model.

2. Methods

This study, which aims to investigate the critical views of students about news items included in the media, is descriptive in nature; in this sense, the most important specification of the descriptive model is the ability of the researcher to identify an event or situation as it exists (McMillan & Schumacher, 2001). The method of case study was adopted in this study which is a form of qualitative descriptive research.

2.1. Research group

The research sample for the study included nine 8th graders, students of approximately 15 years of age, who were enrolled in a public school located in Trabzon, Turkey, in the 2011–2012 academic year. This school was chosen among other middle schools with respect to the results of the Level Determination Exam for high school. The views of the students’ mathematics teacher were considered during the selection of the research sample. In this respect, the teacher categorized the students in a class in consideration of their mathematics achievement according to 3 levels: low, intermediate and high. Three students were randomly chosen from each group by the researchers as a means to increase the variation of the sample. From this perspective, purposive sampling was performed. The learning objectives related to statistics in the Turkish middle school mathematics curriculum were investigated in order to determine the pre-knowledge and academic proficiency of the students. The previously achieved objectives for the middle school level students (Ministry of National Education [MoNE], 2005) comprising the research sample are presented in Table 2.

Table 2. Distribution of objectives for statistics according to grade level in the middle school teaching program

Grade	Objective
6th	(1) Create research questions for a problem, identify an appropriate sample, and collect data (2) Interpret and display the data in their statistical representation forms (3) Explain the situations resulting from incorrect interpretations of column charts (4) Interpret data through calculating arithmetic average and range (5) Make assumptions based on the data
7th	(1) Make and interpret column and line charts according to more than one criterion (2) Make and interpret a pie chart (3) Form opinions about real-life situations by interpreting and making statistical presentations (4) Make assumptions based on data (5) Explain the situations resulting from incorrect interpretations of line, picture, and figure charts (6) Calculate mean, median and quartiles (7) Interpret the measures of central tendency and quartiles
8th	(1) Form and interpret a histogram (2) Calculate standard deviation (3) Develop opinions about real-life situations by using appropriate statistical presentation forms, measures of central tendency, and standard deviation

2.2. Data collection tools and data collection

Articles published in Turkish national newspapers were used to collect the data. The newspaper articles were selected over a one-week period by experts; articles that included text, tables, and charts containing incorrect or missing information were chosen for the purposes of the study. The articles were given to the students, who were allowed one class period (approximately 45 min.) to read and think about them. Each student was required to examine all of the newspaper articles provided in terms of sampling, data collection, data presentation, and data support. So as not to interfere with their critical thinking, the students were guided through open-ended questions, such as: “Do you believe this information? Do you have any doubts about the accuracy of this information?” The data were collected through interviews, which were conducted individually and audiotaped. In regard to ethical considerations, permission to conduct the study was granted by school principal and participants were requested for consent to collect data.

2.2.1. Selected articles

Within the scope of the study, four articles were chosen that included statistical data with the potential to create misunderstandings. The article titles are listed below, in Table 3.

Table 3. Selected newspaper articles

Article 1	One in three smart ticket holders has an automobile
Article 2	The most comfortable shopping city of Europe is Istanbul
Article 3	The number of ATMs has increased twice in 6 years
Article 4	Twenty thousand new mobile customers per month

The first article, entitled “One in three smart ticket holders has an automobile” (see Appendix 1), contains the results of a questionnaire and reports that an increasing number of people prefer public transport, and that 32% of rail-users have an automobile. In the title of the article, the statement is made that one in three smart ticket holders is an automobile owner. However, a smart ticket can be used for all public transport systems, including buses as well as rail, especially in metropolitan areas. Furthermore, the article states that the rate of high school graduates who were smart ticket holders was 46.2%, yet the rate of literate smart ticket holders was 0.7% when the percentages of smart ticket holders are considered with respect to their level of education in the related table given in the article. Another variable mentioned referred to occupational categories. The sum of the percentages related to this variable is 107.6%. While it is possible that some of the participants surveyed could have more than one occupation, there was no explanation offered for the overage of 7.6% percentage. Furthermore, there was no information concerning the research sample to which the questionnaire was applied, nor was the population described.

In the second article, entitled “The most comfortable shopping city of Europe is Istanbul” (see Appendix 2), it was written that The Economist, a respected UK journal, evaluated 33 large cities with respect to 38 criteria in 5 different categories, selecting Istanbul as the winner in the category of “comfortable shopping.” However, while the categories used to evaluate the cities were mentioned in the article, there was little information about the related criteria. Furthermore, there was no information on the sample size, sample selection, adequacy of the sample to represent the population, or how the measurements were performed during data collection.

In the third article, entitled “The number of ATM has increased twofold in 6 years” (see Appendix 3), a table displays the distribution of ATMs according to given years. When the values provided in the table are considered, it can be seen that the number of ATMs in 2011 was almost double with respect to the number of ATMs in 2006. However, the twofold increase for a given value means that the value actually increased three times. Furthermore, there was no information about from whom and how these data were collected. The remainder of the article explains that there was a decrease in the use of common ATMs, but there was no information on how sharp the decrease was, either in terms of percentages or in general.

The fourth article, entitled “20 thousand new mobile customers per month” (see Appendix 4), relates that due to the increase in smart phone usage, banks have begun to address new smart phone applications for their customers; information about the increase in the number of customers is provided. However, there is no information about whether the stated data referred to an entire population; nor was there any description of the population. In the content of the article, it was stated that the number of active users increased by almost 100 thousand in six months; but it can be seen in the related table that this number was closer to 90 thousand. In addition, it was stated in the table that 20 thousand new costumers had used this service each month. On the other hand, although this was presented as the average value, it was found to be closer to 15 thousand when the average was calculated.

2.3. Data analysis

In the data analysis phase, Gal’s (2004a) 10 critical questions that should be asked in order to interpret and understand statistical information were categorized in 6 dimensions according to expert opinion (Table 4). Different researchers then listened to voice records of the interviews in terms of which and how much of the identified dimensions were used with respect to the critical questions used by the students in analyzing the statistical data in the articles. To increase the reliability of the research, the data was coded by two researchers and inter-coder reliability is found to be greater than .80 which shows the reliability of the analysis (Lombard, Snyder-Duch, & Bracken, 2002). Finally, a discussion was held between coders to provide a consensus between disagreements.

Table 4. Distribution of the critical questions with respect to dimensions

Categories	Questions
Consistency	Questions 1 and 8
Sampling	Question 2
Data collection	Question 3
Data analysis	Questions 4 and 5
Data presentation	Questions 6 and 7
Data support	Questions 9 and 10

Because the first and eighth question asked in the interviews were related to whether the study in the given newspaper article was logical, these items were evaluated in the “consistency” dimension. The second question was related to the participants of the studies presented in the newspaper articles as a whole; therefore, it was evaluated in the “sampling” dimension. The third question was evaluated in the “data collection” dimension, as it related to the use of data collection tools as a whole, while the fourth and fifth questions were evaluated as “data analysis,” since they inquired about the data distribution and statistical calculations presented in the articles. Questions six and seven were evaluated in the “data presentation” dimension, since they related to the presentation of the data obtained in the studies, and questions 9 and 10 were evaluated in the “data support” dimension, because they were related to whether the existing or supplementary explanations were sufficient for the presentation of the findings or data. The findings of the analysis are presented in frequency tables in the following section.

3. Findings

After completion of the interviews, the students’ recorded responses were analyzed to determine how often and/or whether the participants used the critical questions relating to the dimensions of consistency, sampling, data collection, data analysis, data presentation, and data support. Table 5 represents a matrix pertain to students’ critics about four different newspaper articles with respect to dimensions of Gal’s critical questions (as given in Table 4).

Table 5. Critical questions used by students pertain to newspaper articles

		S1				S2				S3				S4				S5				S6				S7				S8				S9
	Critical questions	1	2	3	4	1	2	3	4	1	2	3	4	1	2	3	4	1	2	3	4	1	2	3	4	1	2	3	4	1	2	3	4	1	2	3	4
Consistency	1a. What kind of study is it?	✓	✓			✓	✓	✓		✓	✓	✓			✓	✓			✓				✓	✓	✓		✓		✓	✓	✓	✓	✓		✓		✓
	1b. Is this kind of study reasonable in this context?					✓			✓				✓						✓			✓					✓		✓	✓	✓		✓	✓			✓
	8. Are the claims made here meaningful and supported by the data?							✓	✓			✓	✓						✓	✓			✓	✓	✓					✓	✓	✓		✓	✓		✓
Sampling	2a. How was it sampled?									✓		✓	✓			✓	✓	✓		✓	✓			✓	✓	✓	✓				✓	✓	✓	✓	✓	✓
	2b. How many people actually participated?	✓	✓							✓	✓✓							✓	✓			✓		✓	✓	✓	✓		✓	✓				✓		✓	✓
	2c. Is the sample large enough?							✓				✓		✓							✓					✓			✓	✓
	2d. Could this sample reasonably lead to valid inferences about the target population?																									✓										✓
Data collection	3. How reliable were the instruments or measures (tests, questionnaires, and interviews) used to generate the reported data?													✓	✓					✓	✓		✓		✓	✓		✓			✓	✓	✓	✓	✓
Data analysis	4. How was the data distribution accomplished?									✓	✓	✓										✓	✓										✓				✓
	5a. Are the reported statistics appropriate for this kind of data?									✓	✓	✓										✓		✓								✓	✓				✓
	5b. Could outliers cause a summary statistic to misrepresent the true picture?						✓								✓					✓	✓
Data presentation	6. Is a given graph (or table) drawn appropriately, or does it distort trends in the data?		✓		✓																✓														✓		✓
	7. How was this probabilistic statement derived? Are there enough credible data to justify the estimate of likelihood given?
Data support	9. Should additional information or procedures be made available to enable me to evaluate the sensibility of these arguments? Is something missing?	✓	✓			✓			✓		✓	✓		✓		✓		✓	✓					✓	✓	✓	✓	✓	✓		✓	✓	✓			✓	✓
	10. Are there alternative interpretations for the meaning of the findings or different explanations for what caused them?					✓	✓	✓	✓		✓		✓						✓				✓					✓	✓	✓			✓			✓	✓

Notes: 1: Article 1, 2: Article 2, 3: Article 3, 4: Article 4.

Table 5 shows a general picture about how students evaluated newspaper articles regarding to critical questions. Per tick in the table signifies that the questions were met for the related article by students. As it can be seen, students mostly focused on first item of consistency dimension and 9th question related to data support. Data presentation dimension is the dimension which is less considered. In an attempt to carry out in-depth investigation, each of the dimensions was handled as subtitles. The calculated frequencies are presented in the tables below.

3.1. Findings of the consistency dimension

The frequency table below illustrates how often the consistency dimension was used among the critical questions as the students interpreted the newspaper articles.

As illustrated in Table 6, approximately half of the students considered the critical question “what kind of study is it?” in relation to articles 1, 3, and 4; and all of the students considered the same question in relation to article 2. In addition to this, the question “is this kind of study reasonable in this context?” was considered by almost half of the students in relation to articles 1, 2, and 4, while none of the students considered this question in relation to article 3. Moreover, the question “are the claims made here meaningful and supported by the data?” was considered under the consistency dimension; nearly half of the students asked the question in relation to articles 2, 3, and 4, while 2 students asked the question in relation to article 2. Some examples from the student responses within the consistency dimension are presented below:

S1: Maximum number of people taking the bus, people who are at the high school level and not literate … (Question 1a—Article 1)

S4: … here, it was written that the shopping centers in Istanbul took 13th place in the general ranking among a lot of cities, but took first place in comfort … (Question 1a—Article 2)

S3: … how much did the number of ATMs increase from the year 2006 to the year 2011 … (Question 1a—Article 3)

S2: … they determined three things here, and all are true. Why are they true? First, the level of education was considered, then smart ticket usage was taken into account (owner of a car or not), and finally, occupation was regarded (whether they are bosses or workers). (Question 1b—Article 1)

S5: The study was performed in England, so ranking London was really normal for this. (Question 1b—Article 2)

S3: … for example, whether these were one-time customers or they were continuously using mobile banking … these data were not given … (Question 8—Article 4)

Table 6. Frequency of critical questions usage related to the consistency dimension

	Article 1		Article 2		Article 3		Article 4
	f	%	f	%	f	%	f	%
1a. What kind of study is it?	4	44	9	100	5	55	4	44
1b. Is this kind of study reasonable in this context?	4	44	3	33	0	0	5	55
8. Are the claims made here meaningful and supported by the data?	2	22	4	44	5	55	4	44

On examining the students’ responses in Table 6, it can be seen that the half of the students paid attention to the question related to the consistency dimension. In addition, it can be seen that for some of the articles, all of the students considered a critical question (Article 2-Q.1a), while for the other articles, none of the students applied a critical question (Article 3-Q.1b).

3.2. Findings of the sampling dimension

The frequencies and percentage values given below, in Table 7, illustrate how often the sampling dimension was used among the critical questions in students’ interpretations of the newspaper articles.

Table 7. Frequency of critical questions usage related to the sampling dimension

	Article 1		Article 2		Article 3		Article 4
	f	%	f	%	f	%	f	%
2a. How was it sampled?	4	44	3	33	6	66	5	55
2b. How many people actually participated?	7	77	4	44	2	22	3	33
2c. Is the sample large enough?	3	33	0	0	3	33	3	33
2d. Could this sample reasonably lead to valid inferences about the target population?	0	0	0	0	0	0	2	22

As the frequencies in Table 7 indicate, half of the students took note of how the sample was identified in articles 1, 2, and 4; and more than half of the students questioned the sampling technique used in article 3. On the other hand, only a limited number of students focused on the number of participants in the studies described in articles 2, 3, and 4, while 7 students considered this situation in article 1. The question under the sampling dimension entitled “is the sample large enough?” was considered by 3 of the students for articles 1 and 2 for article 3, and 4, while none of the students took this question into consideration in article 2. In addition, none of the students mentioned the question “could this sample reasonably lead to valid inferences about the target population?” in relation to articles 1, 2, and 3, while only 2 students remarked on this issue in article 4. Some examples from the student responses within the sampling dimension are presented below:

S5: I could not understand where the study took place, or who were the people included in the sample and how they were selected … (Question 2a—Article 1)

S9:33 large cities were selected from Europe, but, it is not clear how they identified the size of the cities … (Question 2a—Article 2)

S4: … was the number presented for smart phone users only in Istanbul, or in general, so in all of Turkey? (Question 2a—Article 4)

S3: … the number of people who participated in the study was not clear … (Question 2b—Article 2)

S7: … it is going to be closer to reality, as many more people were asked … (Question 2d—Article 1)

S9: … for example, we cannot find so many ATMs in Hakkari, but hundreds of them in Istanbul … (Question 2d—Article 3)

An examination of Table 7 and the related student responses show that the students paid attention to how the research samples were selected and how many participants were included in the samples. However, it can be seen that the students did not give attention to whether the selected sample was large enough and representative of the population.

3.3. Findings of the data collection dimension

Another dimension of the critical questions investigated in terms of the students’ reviews of the articles related to data collection. The frequencies and percentages belonging to the responses in this dimension are presented in Table 8.

Table 8. Frequency of critical questions usage related to the data collection dimension

	Article 1		Article 2		Article 3		Article 4
	f	%	f	%	f	%	f	%
3. How reliable were the instruments or measures (tests, questionnaires, and interviews) used to generate the reported data?	3	33	4	44	3	33	3	33

Table 8 indicates that 3 of the students considered the critical question of “how reliable were the instruments or measures (tests, questionnaires, and interviews) used to generate the reported data?” in articles 1, 3, and 4, while 4 students considered this question in article 2. In other words, it can be said that less than half of the students considered how the data was collected. Some examples from the student responses are presented below:

S4: … if I met with the person who wrote this article, I would ask, “Did you calculate these values by going and asking people one by one, or did you talk through your head to deceive the nation?” (Question 3—Article 1)

S8: … I wish he could say how he collected the data … (Question 3—Article 4)

According to Table 8 and the related responses, the students did not take into account how the data was collected and what kinds of data collection tools were used when they evaluated the data.

3.4. Findings of the data analysis dimension

The frequency and percentage values given below in Table 9 represent how often the data analysis dimension was used among the critical questions in students’ interpretations of the newspaper articles.

Table 9. Frequency of critical questions usage related to the data analysis dimension

	Article 1		Article 2		Article 3		Article 4
	f	%	f	%	f	%	f	%
4. How was the data distribution accomplished?	2	22	2	22	1	11	2	22
5a. Are the reported statistics appropriate for this kind of data?	2	22	1	11	3	33	2	22
5b. Could outliers cause a summary statistic to misrepresent the true picture?	0	0	2	22	1	11	1	11

The data presented in Table 9 demonstrates that 2 students considered the critical question “how was the data distribution accomplished?” for articles 1, 2, and 4, and only one student considered this issue for article 3. The question “are the reported statistics appropriate for this kind of data?” was considered by a limited number of students. Almost all of the students neglected the question “could outliers cause a summary statistic to misrepresent the true picture?” considered under the data analysis dimension. Some examples from the student responses are presented below:

S3: … I can’t be sure whether [the statistics] are true or not when I look at this, because only some data were given. Nothing like standard deviation was given; it would be better if the median, range were given … (Question 4—Article 1)

S8: … it was given only for 2011; the other years were not given. That is to say, there were no mobile applications; if there were, it was not stated. This is a deficiency. If I could, I would give the data from 2010 and compare them … (Question 5a—Article 4)

S2: … maybe there are world renowned department stores in London; maybe they prefer it because of this. For instance, 2 or 3 department stores in London… maybe there are more department stores in Istanbul, but not well-known. Presumably, there are more admirable brands there … (Question 5b—Article 2)

When Table 9 and the related student responses are considered, it can be seen that the students did not focus on or investigate the data analysis during their evaluations of the newspaper articles. In other words, the students considered the results of the analysis, but they did not evaluate how the analysis was carried out.

3.5. Findings of the data presentation dimension

The frequency and percentage values given below in Table 10 represent how often the data presentation dimension was used among the critical questions in students’ interpretations of the newspaper articles.

Table 10. Frequency of critical questions usage related to the data presentation dimension

	Article 1		Article 2		Article 3		Article 4
	f	%	f	%	f	%	f	%
6. Is a given graph drawn appropriately, or does it distort trends in the data?	0	0	2	22	0	0	3	33
7. How was this probabilistic statement derived? Are there enough credible data to justify the estimate of likelihood given?	0	0	0	0	0	0	0	0

Table 10 illustrates whether the students gave attention to whether the charts presented in the articles were drawn accurately or whether there was any distortion of the drawings in the newspaper articles. None of the students focused on this situation in articles 1 and 3; while on one hand, 2 students took note of this issue in article 2; and on the other hand, 3 students mentioned this situation in relation to article 3. Furthermore, the question “how was this probabilistic statement derived? Are there enough credible data to justify the estimate of likelihood given?” was neglected by all of the students who participated in the study. Some of the students’ responses are presented below as an example:

S1: I look at this, and it does not look like twofold the other. So, it is close to being double, but indeed, it is not … (Question 6—Article 4)

S9: The graphics look OK to me … (Question 2—Article 2)

When the data presented in Table 10 and the related student responses are considered, it can be seen that the students did not take note of the data presentation during their evaluations of the newspaper articles. In other words, the students interpreted the graphics as they saw them and investigated whether there was anything that could have an effect on study.

3.6. Findings of the data support dimension

Another dimension of the critical questions examined during the students’ evaluations of the newspaper articles related to support provided for the data. The frequency and percentage values related to this dimension are presented in Table 11.

Table 11. Frequency of critical questions usage related to the data support dimension

	Article 1		Article 2		Article 3		Article 4
	f	%	f	%	f	%	f	%
9. Should additional information or procedures be made available to enable me to evaluate the sensibility of these arguments? Is something missing?	3	33	6	66	6	66	6	66
10. Are there alternative interpretations for the meaning of the findings or different explanations for what caused them?	2	22	4	44	3	33	5	55

Table 11 demonstrates that nearly all of the students gave no consideration to the critical question “should additional information or procedures be made available to enable me to evaluate the sensibility of these arguments?” in relation to articles 2, 3, and 4; and 3 of the students considered this question in relation to article 1. On the other hand, nearly half of the students considered the critical question “are there alternative interpretations for the meaning of the findings or different explanations for what caused them?” in relation to article 2, 3, and 4, while 2 students took this into account in article 1. Some examples from the student responses are presented below:

S1: … they could have made a chart for this article … (Question 9—Article 1)

S2: … in the cities where more students are using [public transport], for instance, there could be a district where fewer students are living, as a much more private sector. I would ask about the number of recorded plates; I would ask about the number of schools in the district … (Question 10—Article 1)

S3: … we were 13th in the general ranking, but how … how did they rank this, that data (criterion) is not clear … (Question 9—Article 2)

S9: …I guess that too many burglaries happen in London, because I assume that they referred to all kinds of comfort when they said [that], but the criterion is missing. There is shopping in London, but no chance to roam. The department stores in Istanbul are more comfortable; presumably they are not crowded … (Question 10—Article 2)

S7: … I would add the district in which they were conducted and the number of banks … (Question 9—Article 3)

S5: … I could do a comparison, such as writing 2009–2010–2011, and show the increase in the number of smart phone usage up to this … (Question 10—Article 4)

In consideration of Table 11 and the related student responses, it can be seen that the students did not think that the given data were enough to explain the results. Furthermore, the additional information was also evaluated; and the students inquired about alternatives for the explanations.

4. Discussion and conclusion

In this study, the opinions of 8th graders about newspaper articles including statistical information were questioned according to the dimensions of consistency, sampling, data collection, data analysis, data presentation, and data support. In the consistency dimension, it was seen that most of students considered the context of the article, while it was found that less than half of the students inquired into whether the study was consistent in itself. With respect to the consistency of the study, whether the situations presented in the articles were supported by the data was neglected by the most of the students. It was seen that the rate of students interested in the consistency of the articles varied from one article to another. For this reason, it may be that the form of presentation of the articles had an effect on the consideration of critical situations by the students. The situations presented in the articles were seen as logical and accepted without any inquiry; the students did not demonstrate a critical stance on this point. In this respect, it was believed that the students made decisions based on their personal experiences while investigating the statistical data, as Cerrito (1999) likewise concluded.

When the findings concerning the sampling dimension were investigated, it was seen that the students mainly questioned how the sample was formed and how many people participated; in other words, they investigated the sample size. However, whether or not the sample size was large enough and generalizable was not questioned by most of the students. Although the students were aware of the concept of sample, they did not effectively question the sample size or the relationship between the sample and the target population. Watson and Moritz (2000) revealed similar results in their research. One reason for this situation may relate to the emerging picture derived from the results-based exam systems in Turkey, the construction of nationwide multiple-choice exams, and students’ lack of familiarity with implementations such as critical evaluation of statistical data in newspapers. As a result, it was believed that the students were unable to construct a relationship between the sample and the population, although they were able to handle these two concepts in different dimensions.

Under the dimension of data collection, it was concluded that the most of the students did not question how the data were collected or what kinds of data collection tools were used. Although the objective “Create research questions for a problem, identify an appropriate sample, and collect data” is included in the curriculum for 6th graders, it was seen that the students neglected the data collection dimension during their evaluations of the articles. The reason for this may be that students were not exposed to real-life experiences; in other words, they did not engage in activities such as collecting data and solving prepared questions. As a result, the students were not experienced in statistical processes. Because of the well-known effects of student attitudes toward statistics on their achievement, such experiences should be provided at the elementary education level (Mills, 2004).

When the results related to the data analysis dimension were investigated, it was seen that only a limited number of students questioned the articles with respect to the distribution of data. The issue of whether the statistics presented in the articles were appropriate for the data and the possibility of the news being falsified were neglected by most of the students. Similar results were also seen under the data presentation dimension. In the articles including data representation such as graphics, it was seen that a limited number of students considered whether these graphics were drawn properly. In this respect, the results obtained in this research were parallel to the study by Pfannkuch (2005). As the reason for these results, it may be construed that placing misleading graphics in teaching programs, but giving little attention to these, especially with pictorial and figure graphs, may lead to misinterpretations. Moreover, it was noted that none of the students investigated the statistical situations. Teaching that the possibility for misleading information exists and is part of statistics, without relaying this both in teaching programs and the related teaching activities conducted in schools (Gal, 2004b) could be a reason for this issue.

When the data support dimension was investigated, with the exception of article 1, it was seen that most of the students questioned whether there was a need for additional information or calculations to establish the reasonability of the evidence. In the first article, it was noted that the students did not examine this dimension closely, due to some of the included demographic features of research sample. When the student responses were investigated from the aspect of the data support dimension, it was seen that the responses included the calculations presented for increasing the credibility of the articles in general. Most of the students failed to note whether there was a need for additional information and explanations in order to understand the data. In this respect, it was observed that the students did not question the given data closely; they only recommended some additional calculations that could be made to increase the credibility of the evidence.

In general, the students reported that they had never experienced an implementation like this before, where they were asked to examine newspaper articles including statistical data. In reviewing the middle school teaching program, it was seen that there were 15 objectives related to statistics that were mainly coincident with the critical questions delineated in Gal’s (2004a) model. Although many of the objectives do not directly refer to statistical literacy (see Table 2), they are open to interpretation and teachers are expected to make connections between the statistical concepts to provide both, statistical literacy and critically thinking. However, it is believed that the content and the applications given in schools may be restricted by teacher beliefs concerning the greater time requirement of these kinds of implementations (Duru & Korkmaz, 2010), as well as on the need to present questions related to comprehension and application (see Krathwohl, 2002) related to the level determination examination for high school entrance (İskenderoğlu, Erkan, & Serbest, 2013). Similar results were obtained by Yolcu (2012). During observation of classroom activities, she has noted that teachers spent more time for application of statistical ideas while a small amount of time was dedicated to develop conceptual understanding and critical evaluation. This situation may prevent students from improving their statistical thinking (Wild & Pfannkuch, 1999). Instead, a student-centered learning environment should be created which gives opportunities students to develop their statistical conceptual understanding and critical thinking as they carry out statistical studies themselves which connect to real-life examples and situations.

4.1. Limitations

There were limitations associated with the methodology. First of all, although case studies are not designed for large groups and researchers tried to provide variation of the students with respect to their achievement, the number of participants in this study prevents generalizing results. Second limitation of the study concerns the data collection tools. As instrument, we selected newspaper articles to determine their statistical literacy. However, it was restricted in the context of these articles. Finally, statistical literacy of the students was investigated considering only Gal’s (2004a) critical questions.

4.2. Educational implications

Today, in the age of information, many researchers agree on the necessity of statistical literacy (e.g. Gal, 2004a, 2004b; Wallman, 1993). As a consequence of this study, it was concluded that the students did not perform well during their critical analysis of the newspaper articles including research data due to their lack of related experience. For this reason, in addition to exposing students to theoretical knowledge and concepts such as central tendency and distribution, an instructional environment should be constructed for statistics education using original data from real-life situations. In this way, instruction may enable a change of students’ views toward statistics and may allow them to develop a critical standpoint toward statistical data. In order to reach this aim, policy-makers and curriculum designers should identify the aim of statistics teaching clearly. In-service teacher training programs should also support teacher development in relation to statistical-pedagogical content knowledge.

Additional information

Funding

The authors received no direct funding for this research.

Notes on contributors

Mustafa Guler

Mustafa Guler and Kadir Gursoy are research assistants at Karadeniz Technical University, Institute of Educational Sciences. Their research areas are: teacher training, mathematics learning and teaching for elementary school students. The two authors participated a national project aiming to explore the teaching knowledge, beliefs about mathematics and learning opportunities of Turkish pre-service elementary mathematics teachers and compare them to TEDS-M countries.

Bulent Guven

Bulent Guven is a professor at Education Faculty of the same university. In addition to elementary level, his research area includes secondary school mathematics teaching and learning as well as curriculum development. He participated in many national projects on mathematics curriculum development and textbook commissions.

Appendix 1

Selected article 1

Appendix 2

Selected article 2

Appendix 3

Selected article 3

Appendix 4

Selected article 4