A descriptive-evaluative study of a Saudi EFL textbook series

Abstract

This study employs checklists from Williams (1983) and Keban, Muhtar, and Zen (2012) to evaluate the Flying High for Saudi Arabia series, which is currently taught in high schools in the Kingdom of Saudi Arabia, to ensure it can achieve their pedagogical goals. Different results were obtained from each checklist. Keban et al.’s checklist showed the evaluators to be generally satisfied with the series, despite two shortcomings: the review section and vocabulary list/glossaries, and the inadequate quality of textbook paper and binding materials. Williams’ checklist indicated both strengths and weaknesses. Overall, vocabulary, reading skills, and technical aspects were evaluated positively, while speech, grammar, and writing were evaluated negatively. Thus, Keban et al.’s checklist is designed to evaluate language textbooks broadly, whereas Williams’ checklist offers an in-depth evaluation. The recommendations are intended for the Ministry of Education, the curriculum developers, researchers, and schoolteachers.

Keywords:

• textbook evaluation
• evaluation checklist
• school textbooks
• EFL textbook
• Saudi curriculum

Public Interest Statement

This paper evaluated an English series textbook using two evaluative checklists, i.e. Williams (1983) and Keban et al. (2012), to ensure that the textbook is suitable and capable of assisting teachers in achieving their educational goals. Different results were obtained from each checklist, indicating that different checklists might be designed in order to evaluate different language skills, goals, and aspects. Hence, teachers are encouraged to utilize different methods and checklists when evaluating and selecting textbooks.

1. Introduction

Hirst and Peters (1970, p. 19) describe education as “the development of desirable qualities in people.” Thus, the continuous development of education is of indisputable importance to any country, as it leads to achieving excellence in education. However, the most important aspect of attaining excellence in education is the quality of education. Seidel, Tishman, Winner, Hetland, and Palmer (2009, p. 5) believe that quality is a moving target insofar as it is always changing. Hence, quality should be considered as a key aspect of education in addition to other elements such as teachers, classrooms, resources and sources, materials, and textbooks. Among these elements, textbooks in general and Foreign Language (EFL) textbooks in particular have always played a pivotal role in the quality of education.

The roles and influence of textbooks have been the subject of research and debate for many years (e.g. Allwright, 1982; Alptekin, 1993; Brown, 1995; Cunningsworth, 1995; Hutchinson & Torres, 1994). Since the introduction of computers and the Internet, education requirements and policies and classroom methodologies have been changing. Undisputedly, these massive technological developments have changed the concept of education itself and introduced new concepts such as electronic education; yet, as Garinger (2002, p. 1) explains, “[d]emand for textbooks continues to grow, and the publishing industry responds with new series and textbooks each year.” In language courses and programs, the role of textbooks is more pivotal. They “are regularly published in order to meet the changing focuses of instruction and the changing language needs of learners” (Mobarakeh & Arani, 2012, p. 243). They promote effective and rapid ways of learning and teaching a foreign language (Cunningsworth, 1995) as well as materials to teach and save the teachers’ time to focus on the teaching–learning process (Edge & Wharton, 1998). In addition, they provide the necessary guidance for teachers to construct and deliver their lessons (Hutchinson & Torres, 1994) and “give cohesion to the language teaching and learning process by providing direction, support and specific language-based activities aimed at offering classroom practice for students” (Mares, 2003). Moreover, they improve language learners’ linguistic and communicative abilities and skills (Sheldon, 1987), and motivate and stimulate learners (Allwright, 1982; Lee, 1997; Skierso, 1991). Hence, it is necessary to select the proper textbook to meet the students’ needs and requirements. Accordingly, textbook evaluation has become a necessity.

1.1. Textbook evaluation

The topic of textbook evaluation has been highlighted by many researchers and educators. Its importance has been continuously supported through research literature (e.g. Allwright, 1982; Alptekin, 1993; Brown, 1995; Cunningsworth, 1995; Hutchinson & Torres, 1994). Textbook evaluation is a dynamic process that examines the different aspects of the textbook to improve its quality (Antic, Ivic, & Pešikan, 2013, p. 200) and ensure “quality assurance and enhancement,” which allows “ongoing improvement of learning opportunities” (Kiely, 2009, p. 100). It “involves measuring the value or (potential value) of a set of learning materials by making judgments about the effect of the materials on the people using them” (Tomlinson & Masuhara, 2004).

The available literature on textbook evaluation is not extensive (Hashemi & Borhani, 2015; Litz, 2005; Tok, 2010); however, there are three basic methods that can be discerned in the literature on textbook evaluation. They are, as McGrath (2002, p. 25) explains, the impressionistic, the checklist, and the in-depth method. The impressionistic method refers to the evaluation of a textbook based on a general impression. The impressionistic method by itself is not sufficient “but [it] could be combined with for example the second method, which is … the checklist method” (AbdelWahab, 2013, p. 56). Most researchers who have contributed to the area of textbook evaluation have used checklists, “based on supposedly generalizable criteria” (Hashemi & Borhani, 2015, p. 48). In the last 30 years, a number of checklists have been developed to evaluate current EFL textbooks. Checklists enable teachers to evaluate textbooks more accurately (Ellis, 1997; Sheldon, 1988; Tomlinson, 2003; Williams, 1983), as they allow them to conduct a systematic and cost efficient evaluation based on a set of agreed-on evaluative criteria (McGrath, 2002, p. 27). In addition, checklists enable researchers to record information in a convenient manner to conduct comparison, as checklists offer a common framework for decision-making.

Cunningsworth (1995) argues that a detailed checklist can perform an in-depth evaluation of any given textbook. In a number of studies, researchers and theorists such as Cunningsworth (1995), Sheldon (1988), and Williams (1983), have indicated that evaluative checklists should include some physical characteristics of textbooks such as layout and organization in addition to criteria related to language, functions, grammar, and skills content. The checklist method is also a means to examine and evaluate the content of the textbook. This kind of content analysis is also utilized to analyze qualitative data (Keban et al., 2012, p. 3). In the available literature, there are a number of checklists, e.g. Cunningsworth’s (1995) scheme, Davison’s (1975) scheme, Keban et al.’s (2012) scheme, Littlejohn’s (1998) scheme, Sheldon’s (1988) scheme, Tucker’s (1975) scheme, Ur’s (1996) scheme, and Williams’s (1983) scheme. Although no universal checklist has been agreed on, there have been a few attempts to draft universal characteristics of EFL/ESL textbooks (e.g. Ansary & Babayi, 2002; Mukundan & Nimehchisalem, 2012; Tucker, 1975; Williams, 1983). These characteristics are related to aspects such as approach, content presentation, physical make-up, and administration concerns.

1.1.1. EFL textbook evaluation in KSA

The importance of the English language in the Kingdom of Saudi Arabia (KSA) “is not restricted only to interests and purposes outside the country. Its importance has been increasingly felt as a catalyst of progress for the country” (Al-Zubeiry, 2012, pp. 17–18). Al-Zubeiry further explains that the Saudi government has shown more interest in teaching the English language and its culture to school students. This is obvious in a document issued by the English Department in the Directorate of Curriculum (DDC) at the Ministry of Education (MOE) in 2000, which renovates English language education by fostering both students’ ability to understand and express themselves in English. In so doing, the Saudi government allocates a large amount of funding to achieving such goals within the framework of The King Abdullah Bin Abdul Aziz Public Education Development Project (Tatweer). Thus, “qualified, competent, and committed foreign faculty members [were hired] to teach English” (Al-Zubeiry, 2012, p. 18).

As in many educational sectors, textbooks dominate the educational process in KSA schools. The educators in Saudi Arabia face a number of practical problems because of the lack of textbook guidance and extracurricular activities, the domination of traditional instructional methods, and the culture of resistance to change. However, “the [fundamental] problem lies in the curriculum” (Khashoggi, 2014). Khashoggi (2014) explains, “the real predicament lies in the quality and not the quantity.” Thus, the introduction of new EFL textbook series has offered a natural platform for evaluation, which is a key stage following the design of any new textbook series. Accordingly, a number of studies have been carried out to evaluate the EFL textbooks that are used in KSA schools and universities. Below is a brief review of some studies.

Faruk (2014) analyses three third-grade secondary school textbooks published and used over the last three decades, that is, Saudi Arabian Schools’ English used from 1982 to 1997, English for Saudi Arabia, used from 1998 to 2012, and Traveller 3, used from 2013 to present, with a focus on the cultural context of the textbooks. It was demonstrated that with reference to the cultural context, Traveller 3 (2013 to present) was the opposite of the other two textbooks. That is, from 1989 to 2012, English was introduced in a religious and Saudi cultural context and there was deliberate avoidance of western culture. On the other hand, English in Traveler 3 (2013 to present) was contextualized in terms of western cultural elements. Only two texts contained Islamic and Saudi elements represented in a western way, whereas the western culture appeared in 18 texts. These findings reflect the interest of MOE in teaching English in addition to its culture.

Alhamlan (2013) examined the Traveller 5 textbook, which is taught in a third of secondary Schools in Saudi Arabia, as part of the Saudi English Language Development Project (SELDP). A checklist was distributed to 500 students to explore their perceptions of the textbook. The students’ perceptions were positive except in certain areas. Thus, this study recommended improving certain aspects, such as increasing the difficulty of the material in the textbook, the opportunities for students to interact in the classroom, and the number of new vocabulary items in the textbook. With reference to culture, 59.65% of students believed that the cultural context was suitable.

Al Fraidan (2012) examined two English for specific purposes (ESP) textbooks using McDonough and Shaw’s (2003) criteria to evaluate the external and internal aspects. These two textbooks are taught to BA students at King Faisal University, KSA. The first textbook was Business Objectives (1996) by Vicki Hollett and the second was Business Studies, Second Edition (2002) by Alain Anderton. The study concluded that concerning language, authenticity, and appropriateness, Business Studies was preferable, as it contains much interactivity and opportunities for simulation and role-play. Additionally, the study recommended the implementation of both textbooks to fulfill the objectives of the course.

Al-sowat (2012) examined the English Language textbook for First Year Intermediate Grade, in Taif City, entitled “Say it in English” from the perspectives of the teachers. A checklist was distributed among 48 teachers to evaluate the textbook. The layout and physical appearance, the content of the textbook, and the objectives of the textbook were evaluated as “Good.” On the other hand, the language type, language skills, activities and tasks, structures and vocabulary, cultural values, and teachers’ needs were evaluated as “Poor.” Overall, the textbook was evaluated as “Poor.”

Habtoor (2012) examined First Class: English for Tourism, an ESP textbook taught to Tourism and Archaeology sophomore students at King Saud University, Riyadh, Saudi Arabia. A checklist was utilized and 20 teachers participated to evaluate this textbook in terms of components, practicality, activities, language, topics, basic language skills, and the sub-skills of grammar and vocabulary. Based on the evaluation of the teachers, the textbook was found to be suitable for learners’ needs (in the Saudi Arabian context) and it met the expectations of teachers.

Rahman (2012) examined Interactions 1 by Cheryl Pavlik and Margaret Keenan Sega, the English writing textbook for the preparatory year at Najran University, KSA. A checklist was distributed to 22 teachers of the preparatory year to evaluate aspects such as content, presentation, organization, and exercises. The participants evaluated the organization of the textbook positively as well as its format and font size. However, they pointed out a number of shortcomings: the chapters did not provide clear and comprehensive summaries, chapters were not designed based on students’ previous knowledge, exercises were not prepared to develop students’ communicative competencies, and no periodical revisions were provided for diagnostic purposes. This study recommended that some modification, addition, and subtraction be carried out by the textbook authors.

Alamri (2008) evaluated the sixth-grade English language textbook, as English had been introduced at the elementary stage for the first time by the Ministry of Education in 2004. He utilized a checklist that was distributed to 93 English language teachers and 11 supervisors in Riyadh Educational Zone, to examine 12 main categories: the general appearance, design and illustration, accompanying materials, objectives, topic appropriateness, learning components, socio-cultural contexts, skills development, teachability, flexibility, teaching methods, and practice and testing. The textbook was evaluated positively except for the teaching methods. The learning components received the highest score, whereas the teaching methods received the lowest score.

Al-Saif (2005) examined the unified textbook of English for sixth grade in Saudi elementary schools to reveal its strengths and weaknesses as well as its suitability for teachers and students. A checklist was prepared and distributed to 2931 teachers, supervisors, and trainers. It was concluded that this textbook was “moderately adequate,” and required continuous evaluation.

1.1.2. Flying high for Saudi Arabia series

With the assistance of the Saudi English Language Development Program (SELDP), Macmillan publishing house developed and published “custom made” EFL curricula, designed especially to meet the Islamic and cultural aspects of Saudi Arabia. The curricula have three different series: Get Ready for primary students, Lift Off for intermediate students, and Flying High for high school students. The Flying High for Saudi Arabia series, which is the series under examination, is a six-level topic-based course for high school students in Taif City (as well as Madinah city), in the KSA. This series offers an online training package for students and online teaching resources for teachers. It also includes a student’s book, a workbook, a teacher’s book, and a CD. The textbook consists of eight units, each comprising four lessons.

A textbook can be evaluated in three stages: pre-use, whilst-in-use, and post-use (Ellis, 1997; Grant, 1987; Mukundan, 2007). To make a concerted effort with the MOE, this study evaluates this series during the second stage (whilst-in-use) to ensure that it is suitable and capable of assisting teachers to achieve their pedagogical goals. The main goals of the current paper are to (1) evaluate the textbook series under investigation from the perspective of the schoolteachers who are teaching the current textbook, and (2) to provide a better understanding of how teachers may vary in their understanding of textbook quality. The current study might be useful in helping Saudi EFL teachers to evaluate their materials in order to improve their students’ English competence. Accordingly, this evaluation may help to develop and revise this textbook series, and/or design pedagogical training courses for the teachers to achieve the purposes of the textbook. Hence, hopefully, this study may bridge the gap between theoretical knowledge and practical knowledge. It can also provide the Department of Foreign Languages (DFL) at Taif University with a general idea about the English language educational background of the high school students who will join the department. Furthermore, this study could assist the MOE in developing and/or designing (new) EFL textbooks in the KSA. In so doing, the study addresses the following questions:

According to the schoolteachers,

(1)	to what extent does the series Flying High for Saudi Arabia meet the characteristics of a good EFL textbook in terms of objectives, design and organization, language skills, language content, topics, and practical considerations?
(2)	to what extent does the series Flying High for Saudi Arabia meet the characteristics of a good EFL textbook in terms of up-to-date teaching methodologies, guidance for non-native teachers, learner needs, and relevance to the socio-cultural environment of the learners?
(3)	what are the strengths and shortcomings of this textbook series?
(4)	what is the overall perception of the teachers of this textbook series?

2. Method

The Flying High for Saudi Arabia series for high school students is evaluated in this study by high school teachers who have been teaching this textbook for at least one year. This series was chosen because it is taught in the Taif new school project, as part of the pilot scheme of the King Abdullah bin Abdul Aziz Public Education Development Project for high school students (Tatweer) and this study is funded by Taif University. As noted earlier, this textbook series was evaluated by schoolteachers in order to:

(1)	evaluate the textbook from the perspective of the schoolteachers;
(2)	highlight the shortcomings and strengths of the textbooks from the perspective of the schoolteachers;

To meet the objectives of the current study, a quantitative method was used. Two checklist instruments were employed: Williams (1983) evaluative checklist and Keban et al.’s (2012) criteria.

2.1. Participants

The participants (evaluators) in this study are female high school English language schoolteachers from Taif City’s new High Schools project for girls. The participants’ consent was obtained before conducting the study (for participant consent form, see Appendix 1). In Taif City, there are four high schools, namely, the Third High School, the Fourth High School, the Tenth High School, and the Nineteenth High School. Forty (n = 40) invitation emails were sent to all schoolteachers whose emails could be obtained to invite them to participate in the current study to evaluate the textbooks. Only 25 (n = 25) responses were received indicating their willingness to participate voluntarily. However, five (n = 5) participants were excluded, because they did not complete the checklists. A total of 20 (n = 20) evaluators took part in this study. Of them, only 8 (n = 8) schoolteachers (40%) had participated in pedagogical training organized by the MOE for new high school projects, while 12 (n = 12) (60%) had no such training. It should be noted that such pedagogical training was not compulsory. This textbook series was given to schools, and non-compulsory training programs were organized to introduce it to the schoolteachers (Table 1).

Table 1. Duration of teaching experience of evaluators

Years of experience	Number of evaluators	Percentage (%)
1	2	10
2	3	15
3	2	10
4	4	20
5	2	10
More than 5	7	35

As noted earlier, the book series has six levels and the participants were given the choice to evaluate any level. Of these six levels, only the third level was not evaluated. Table 2 shows the number of participants as well as the percentage of evaluators for each level.

Table 2. Percentage of evaluators for each level

Level	Number of evaluators	Percentage (%)
1	7	35
2	3	15
3	0	0.00
4	7	35.7
5	1	5
6	2	10

2.2. Instruments

As mentioned above, the current study utilizes Williams (1983) checklist (see Appendix 3) and Keban et al.’s (2012) checklist (see Appendix 2). These checklists were chosen because of the clear terminologies and variables that are used. Nogova and Huttova (2006, p. 338) explain that “[i]n order to be able to declare that textbook evaluation is objective, we must be confident that all evaluators have understood the criteria in the same way.” This method of integrating both checklists can help in conducting a detailed descriptive-evaluative content analysis and pinpointing the problematic issues distinctly. Additionally, utilizing both checklists helps in evaluating the textbook more comprehensively.

Williams’ evaluative checklist, as he elucidates, “can be used to draw up a checklist of items relevant to second (or foreign) language teaching … [It] relates assumptions about teaching a second evaluation language to a set of linguistic, pedagogical, general, and technical variables. These are the four assumptions on which the scheme is based” (1983, p. 251). This checklist is limited to the following features: up-to-date teaching methodology, guidance for non-native teachers, needs of learners, and relevance to the socio-cultural environment of the learners. These features are examined in light of linguistic/pedagogical variables, such as general, speech, grammar, vocabulary, reading, writing, and technical. Each variable has four evaluative items (the features mentioned earlier). The evaluative system is a 5-point scale: 0–4 (4: to the greatest extent; 3: to a large extent; 2: to some extent; 1: just barely; 0: not at all) (see Appendix 3).

On the other hand, Keban et al.’s (2012) checklist is based on Cunningsworth (1995) and Litz’s (2005) theory. In this checklist, there are six variables, i.e. objectives, design and organization, language skills, language content, topics, and practical consideration. Each of these variables has at least three evaluative items (maximum five) to examine the suitability, appropriateness, and authenticity of the content. The evaluative system is a 100-point scale: 0–100 (0–25: poor, 26–50: fair, 51–75: good, 76–100: excellent) (see Appendix 4).

Regarding the reliability of the checklists, a reliability test was conducted using both checklists and a sample from the study participants. The checklists were piloted to 10 (n = 10) schoolteachers as a first test. Then, after two months, the checklists were readministered on the same sample group as a second test. The answers to both tests were compared and the correlation coefficient was then calculated. The reliability of the test was estimated, and a 0.76 inter-rater reliability was found, which can be considered as a reasonable indication of the reliability of the questionnaires.

2.3. Procedures

The data collection procedures consisted of four steps: (1) the textbook series was made available; (2) the content was viewed by the researcher to have a general idea of the book to modify, change, or delete any inapplicable parts of the checklists; (3) the checklists were distributed to the voluntary schoolteachers; (4) after receiving the responses and entering the data, the complete data-set was analyzed.

With reference to the procedures the evaluators follow, three major steps were proposed, i.e. (1) selecting the level of the textbook series, (2) using Keban et al.’s (2012) checklist, and (3) using Williams (1983) checklist. Keban et al.’s (2012) checklist was presented first and then Williams (1983) checklist, as the former evaluates the content of the textbook in a general sense while the latter evaluates the content individually. In the results section, frequency counts and percentage indices were reported for individual features that were listed in both checklists.

2.4. Limitations

The most obvious limitation of the study is related to the number of participants. Only 20 (n = 20) female high school teachers were willing to evaluate the textbook series. Needless to say, this small number of evaluators limits the study’s generalizability. Additionally, the evaluation was limited to one textbook series entitled Flying High for Saudi Arabia. As for the instrument of the study, as noted earlier, it is limited to two evaluative checklists. The limitations of the current study are acknowledged; thus, recommendations for further research are encouraged.

3. Results

The results of the participants’ evaluations are presented below. The results of Keban et al.’s (2012) checklist are presented first followed by the results of Williams (1983) checklist.

3.1. Keban et al.’s (2012) checklist

The first subcategory is related to the objectives of the textbook series. It has four variables related to the teaching program, its clarity, its organization, and the different teaching and learning styles. From Table 3, the first two variables, in addition to the final variables, were evaluated as “Good” (51–75) by the majority, that is, 55, 45, and 35%, respectively. On the other hand, the third criterion, which is related to the organization of the objectives, was mostly evaluated as “Excellent” (79–100) by the majority (35%). Overall, all these variables were rated as “Good” (see Appendix 4A).

Table 3. Objective variables

Variables	Checklist items	Score range
		Poor		Fair		Good		Excellent
		0–25		26–50		51–75		76–100
Objectives		No	%	No	%	No	%	No	%
1	The aims of the textbook correspond closely with the aims of the teaching program/curriculum.	3	15	0	0	11	55	6	30
2	The materials’ objectives are apparent to both the teachers and students.	3	15	3	15	9	45	5	25
3	The objectives are systematically organized.	3	15	6	30	4	20	7	35
4	The textbooks allow different teaching and learning styles.	6	30	2	10	7	35	5	25

The second subcategory is related to the design and organization of the textbook. From Table 4, nine (45%) participants gave the first, second, and fourth variable “Excellent” (76–100). In addition, seven (35%) participants gave the third criterion “Good” (51–75) and six (30%) participants gave it “Poor” (0–25). The final criterion, which is related to the review section, was given “Poor” (0–25) by eight (40%) participants. Statistically, the first two variables were rated as “Excellent,” whereas the second two variables were rated as “Good.” The final variable was rated as “Fair” (see Appendix 4B).

Table 4. Design and organization variables

Variables	Checklist items	Score range
		Poor		Fair		Good		Excellent
		0–25		26–50		51–75		76–100
Design and organization		No	%	No	%	No	%	No	%
1	The layout and design (in terms of pictures, color) is appropriate and clear.	3	15	0	0	8	40	9	45
2	The textbook is organized effectively (e.g. according to topics and skills, etc.).	3	15	1	5	7	35	9	45
3	The textbook contains guidance about how it can be used.	6	30	3	15	7	35	4	20
4	The textbook is accompanied by other teaching/learning materials (workbook, cassettes, and CD).	3	15	1	5	7	35	9	45
5	There was a review section, and vocabulary list or glossaries are included.	8	40	6	30	5	25	1	5

The third subcategory is related to the language skills of the textbook and it has five variables. From Table 5, all of these variables were rated as “Good” (51–75) by eight (40%) participants, whereas six (30%) participants gave the first, third, and fifth variable “Excellent” (76–100). On the other hand, the second and fourth variables were rated as “Poor” (0–25) by five (25%) participants. Overall, the variables were rated as “Good” (see Appendix 4C).

Table 5. Language skills variables

Variables	Checklist items	Score range
		Poor		Fair		Good		Excellent
		0–25		26–50		51–75		76–100
Language skills		No	%	No	%	No	%	No	%
1	All language skills (reading, writing, listening, and speaking) are balanced and integrated within each unit.	5	25	1	5	8	40	6	30
2	Suitability of listening skills to young learners.	5	25	3	15	8	40	4	20
3	Suitability of speaking skills to young learners.	3	15	3	15	8	40	6	30
4	The reading passages are sufficient and suitable for young learners’ level and ability.	5	25	3	15	8	40	5	25
5	Writing activities are suitable for young learners’ level.	3	15	3	15	8	40	6	30

Table 6 shows the fourth subcategory, which is related to the language content of the textbook and has three variables. With reference to the first variable, seven (35%) participants rated it as “Fair” (26–50) and six (30%) participants rated it as “Poor” (0–25). On the other hand, the majority of participants, that is, 12 (60%) participants and 8 (40%) participants, respectively, rated the grammar and vocabulary items as “Good” (51–75). All the variables were rated as “Good” (see Appendix 4D).

Table 6. Language content variables

Variables	Checklist items	Score range
		Poor		Fair		Good		Excellent
		0–25		26–50		51–75		76–100
Language content		No	%	No	%	No	%	No	%
1	The language used in the textbook is authentic.	6	30	7	35	5	25	2	10
2	The grammar items are appropriate for the young learners.	3	15	1	5	12	60	3	15
3	The vocabulary items are appropriate for the young learners.	4	20	5	25	8	40	3	15

The fifth subcategory concerns the topics of the textbook and has three variables, which are related to the learners’ needs, engagement degree, and variation. Table 7 shows that all of these variables were rated as “Good” (51–75) by eight (40%) participants. Statistically, all the variables were rated as “Good” (see Appendix 4E).

Table 7. Topics variables

Variables	Checklist items	Score range
		Poor		Fair		Good		Excellent
		0–25		26–50		51–75		76–100
Topics		No	%	No	%	No	%	No	%
1	The topics of the textbook are relevant to the young learners’ needs as English language learners.	3	15	3	15	8	40	6	30
2	The topics of the textbook are interesting, challenging, and motivating.	4	20	3	15	8	40	5	25
3	There is sufficient variety in the topics of the textbook.	2	10	5	25	8	40	5	25

The sixth subcategory is related to the practical considerations of the textbook. From Table 8, it is apparent that all of the evaluators rated its price as “Excellent” (76–100) whereas 10 (50%) of the participants rated it as “Fair” (26–59) in terms of durability. On the other hand, 10 (50%) participants rated it as “Excellent” (76–100) in terms of accessibility and recentness. Statistically, the first and the third variables were rated “Excellent,” whereas the second variable was “Fair” (see Appendix 4F).

Table 8. Practical consideration variables

Variables	Checklist items	Score range
		Poor		Fair		Good		Excellent
		0–25		26–50		51–75		76–100
Practical considerations		No	%	No	%	No	%	No	%
1	The price of the textbook is reasonable.	0	0	0	0	0	0	20	100
2	The book is durable.	5	25	10	50	5	25	0	0
3	The textbook is a recent publication and easily accessible.	0	0	1	5	9	45	10	50

In brief, Keban et al.’s (2012) checklist demonstrates that the evaluators generally are satisfied with the textbook series, as it meets all the criteria and variables mentioned in the checklist.

3.2. Williams (1983) checklist

The first category was general aspects of the EFL textbook. As indicated in Table 9, the first variable shows to what extent the textbook takes into consideration accepted methods of ESL/EFL teaching. For this variable, seven (35%) participants believed that it does to a “Great Extent.” Regarding the second variable, seven (35%) participants believed that the textbook gives guidance in the presentation of languages items to “Some Extent.” In addition, eight (40%) participants indicated that the textbook caters to individual differences in students’ home language backgrounds to “Some Extent.” Overall, the evaluation of the general variables was rated as to a “Large Extent,” except the final variable, i.e. whether the textbook series relates content to the learners’ culture and environment, which was rated as to “Some Extent,” that is, a negative evaluation (see Appendix 5G).

Table 9. General variables

Variables	Checklist items	Score range
		Great extent		Large extent		Some extent		Just barely		Not at all
		4		3		2		1		0
General		No	%	No	%	No	%	No	%	No	%
1	It takes into account currently accepted methods of ESL/EFL teaching.	7	35	5	25	5	25	3	15	0	0
2	It gives guidance in the presentation of language items.	4	20	4	20	7	35	0	0	5	25
3	It caters to individual differences in home language backgrounds.	4	20	3	15	8	40	1	5	4	20
4	It relates content to the learners’ culture and environment.	5	25	5	25	1	5	4	20	5	25

Table 10 shows the second subcategory regarding speech has four variables. Regarding the first three variables, seven (35%) participants believe that the textbook is to “Some Extent” based on a contrastive analysis of English and L1 sounds, suggests ways of demonstrating and practicing speech items, and includes speech situations relevant to the pupils’ background. Concerning the final variable, seven (35%) participants believed that the textbook “Just Barely” allows for variation in the accents of non-native speakers of English and five (25%) participants believed it does to “Some Extent” or “Not at All.” Overall, (see Appendix 5H), the first and fourth criteria were evaluated as to “Some Extent” or negative. On the other hand, the other two criteria were evaluated as to a “Large Extent.”

Table 10. Speech variables

Variables	Checklist items	Score range
		Great extent		Large extent		Some extent		Just barely		Not at all
		4		3		2		1		0
Speech		No	%	No	%	No	%	No	%	No	%
1	It is based on a contrastive analysis of English and L1 sound systems.	2	10	5	25	7	35	0	0	6	30
2	It suggests ways of demonstrating and practicing speech items.	3	15	5	25	7	35	0	0	5	25
3	It includes speech situations relevant to the pupils’ background.	5	25	5	25	7	35	3	15	0	0
4	It allows for variation in the accents of non-native speakers of English.	2	10	1	5	5	25	7	35	5	25

Table 11 shows the third subcategory that examines four variables related to grammar. The grammar of the textbook under investigation emphasizes communicative competence in teaching structural items to a “Great Extent” (five participants, 25%), to a “Large Extent” (five participants, 25%), and to “Some Extent” (five participants, 25%). For the second criterion, eight (40%) participants thought that the textbook series provides adequate models for the structures to be taught to a “Large Extent.” However, for the third criterion, five (25%) participants indicated that the textbook shows clearly the kinds of responses required in drills to a “Large Extent” and “Some Extent.” The fourth criterion describes the selection of the structures. More than half of the participants indicated that the structures are not selected with regard to differences between L1 and L2 cultures. It is apparent that all the criteria of the grammar variables (see Appendix 5I) were rated as to a “Large Extent,” except the final one, which was rated as to “Some Extent” or negative.

Table 11. Grammar variables

Variables	Checklist items	Score range
		Great extent		Large extent		Some extent		Just barely		Not at all
		4		3		2		1		0
Grammar		No	%	No	%	No	%	No	%	No	%
1	It stresses communicative competence in teaching structural items.	5	25	5	25	5	25	3	15	2	10
2	It provides adequate models featuring the structures to be taught.	3	15	8	40	4	20	3	15	2	10
3	It shows clearly the kinds of responses required in drills (e.g. substitution).	4	20	5	25	5	25	3	15	3	15
4	It selects structures with regard to differences between L1 and L2 cultures.	2	10	2	10	3	15	7	35	6	30

The fourth subcategory evaluates four variables regarding vocabulary. From Table 12, the first variable, namely the selection of vocabulary according to frequency and functional load, was scored as to a “Great Extent” by seven (35%) participants. On the other hand, five (25%) participants believed that the textbook series distinguishes between receptive and productive skills in vocabulary teaching “Just Barely” and “Not at All” (five participants each, 25%). On the other hand, half of the participants indicated that the textbook series distinguishes between receptive and productive skills in vocabulary teaching and focusing on problems of usage related to social background to a “Large Extent.” However, nine (45%) participants thought that the textbook series focuses on problems of usage related to social background “Just Barely.” Overall, (see Appendix 5J) the first and third variables were evaluated as to a “Large Extent,” which was positive. On the contrary, the second and fourth variables were rated as to “Some Extent” or negative.

Table 12. Vocabulary variables

Variables	Checklist items	Score range
		Great extent		Large extent		Some extent		Just barely		Not at all
		4		3		2		1		0
Vocabulary		No	%	No	%	No	%	No	%	No	%
1	It selects vocabulary according to frequency, functional load, etc.	7	35	0	0	4	20	2	10	2	10
2	It distinguishes between receptive and productive skills in vocabulary teaching.	2	10	4	20	4	20	5	25	5	25
3	It presents vocabulary in appropriate contexts and situations.	3	15	10	50	4	20	3	15	0	0
4	It focuses on problems of usage related to social background.	1	5	2	10	2	10	9	45	6	30

Table 13 shows the fifth subcategory is reading, and has four variables. Regarding whether the reading passages offer exercises for understanding the main and implied meaning, eight (40%) participants believed that the textbook does so to a “Large Extent.” In the same vein, eight (40%) participants indicated that they believed that the textbook series relates reading passages to the learners’ background. On the other hand, seven (35%) participants pointed out that the textbook series selects passages within the vocabulary range of the pupils, and five (25%) participants believe that it does so to a “Large Extent.” With reference to the final variable, the textbook series was thought to select passages reflecting a variety of styles of contemporary English to “Some Extent” by eight (40%) participants and “Just Barely” by seven (35%) participants. Overall, the first three criteria of the reading variables (see Appendix 5K) were rated as to a “Large Extent,” whereas the final criterion was rated as to “Some Extent,” which was negative.

Table 13. Reading variables

Variables	Checklist items	Score range
		Great extent		Large extent		Some extent		Just barely		Not at all
		4		3		2		1		0
Reading		No	%	No	%	No	%	No	%	No	%
1	It offers exercises for understanding the main and implied meaning.	4	20	8	40	4	20	2	10	2	10
2	It relates reading passages to the learners’ background.	2	10	8	40	5	25	3	15	2	10
3	It selects passages within the vocabulary range of the pupils.	3	15	5	25	7	35	3	15	2	10
4	It selects passages reflecting a variety of styles of contemporary English.	2	10	0	0	8	40	7	35	3	15

The sixth subcategory has four variables related to writing skills as shown in Table 14. With reference to the first variable, 10 (50%) participants believed that the written work is related to the structures and vocabulary practiced orally to a “Large Extent.” However, six (30%) participants believed that the textbook gives sufficient practice in controlled and guided composition in the early stages “Just Barely.” Regarding the third variable, seven (35%) participants believed that the textbook does not relate written work to the pupils’ age, interests, and environment at all. Similarly, the final criterion was rated by seven (35%) participants indicating that the textbook demonstrates techniques for handling aspects of composition teaching only to “Some Extent” and “Not at All.” Overall, the writing variables were evaluated negatively, to “Some Extent,” except for the first variable, which was evaluated positively, to a “Large Extent” (see Appendix 5L).

Table 14. Writing variables

Variables	Checklist items	Score range
		Great extent		Large extent		Some extent		Just barely		Not at all
		4		3		2		1		0
Writing		No	%	No	%	No	%	No	%	No	%
1	It relates written work to structures and vocabulary practiced orally.	2	10	10	50	5	25	3	15	0	0
2	It gives practice in controlled and guided composition in the early stages.	0	0	5	25	4	20	6	30	5	25
3	It relates written work to the pupils’ age, interests, and environment.	3	15	3	15	2	10	5	25	7	35
4	It demonstrates techniques for handling aspects of composition teaching.	2	10	2	10	7	35	2	10	7	35

Table 15 shows that the sixth subcategory evaluates technical aspects, and has four variables. With reference to the first variable, 10 (50%) participants believed that the textbook meets the four variables to a “Great Extent.” However, 9 (45%) participants thought that the textbook is up-to-date in the technical aspects only to “Some Extent,” and 10 (50%) participants believed that it is dated to “Some Extent.” In addition, seven (35) participants indicated that the textbook series has authenticity in terms of the language and style of writing to a “Great Extent.” Statistically, the evaluation of the technical variables was positive, to a “Large Extent,” with reference to all the different criteria (see Appendix 5M).

Table 15. Technical variables

Variables	Checklist items	Score range
		Great extent		Large extent		Some extent		Just barely		Not at all
		4		3		2		1		0
Technical		No	%	No	%	No	%	No	%	No	%
1	It is up-to-date in the technical aspects of textbook production and design.	5	25	10	50	2	10	0	0	3	15
2	It shows quality in editing and publishing (cover, typeface, illustrations, etc.).	3	15	8	40	9	45	0	0	0	0
3	It is dated.	4	20	6	30	10	50	0	0	0	0
4	It has authenticity in terms of the language and style of writing.	7	35	5	25	3	15	3	15	2	10

3.3. Keban et al.’s (2012) checklist vs. Williams (1983) checklist

Keban et al.’s (2012) checklist is designed to evaluate language textbooks; yet, it does so broadly. That is, it treats language skills as a single aspect to be evaluated. A general view of the results from this checklist is provided below to show the overall evaluation of the textbook (Tables 16 and 17).

Table 16. Mean score using Keban et al.’s (2012) checklist

Variable	Excellent (76–100)	Good (51–75)	Fair (26–50)	Poor (0–25)
Keban et al.’s (2012) checklist	74.0–68.8	68.7–63.5	63.4–58.3	58.2–53.0

Table 17. Schoolteachers’ evaluation for each subcategory

Items	Mean	Critical value (Md)	Result
Keban et al.’s (2012) checklist	63.1	57.5	Positive

From Tables 16 and 17, it is apparent that the textbook series under examination was generally evaluated by the 20 schoolteachers as “Fair,” because the mean of Keban et al.’s (2012) checklist (see Table 18), which is 63.1, is between the range 63.4–58.3. Additionally, the general evaluation of the textbook was positive. In order to provide an in-depth discussion of the schoolteachers’ evaluation, the sum of all the quantitative measures (scores) of the variables in each subcategory was computed to give each subcategory a general “score.” After this, the mean of the “scores” of the 20 schoolteachers was computed in each subcategory (see Table 18).

Table 18. Schoolteachers’ evaluation using Keban et al.’s (2012) checklist

Items	Mean	Critical value (Md)	Evaluation
Objectives	11.0	10.0	Positive^1
Design and organization	13.6	12.5	Positive
Language skills	13.3	12.5	Positive
Language content	7.6	7.5	Positive
Topics	8.2	7.5	Positive
Practical considerations	9.5	7.5	Positive

¹ If Mean ≥ Md, then the schoolteachers’ evaluation of the textbook Flying High for Saudi Arabia with respect to each subcategory is deemed a POSITIVE evaluation. Otherwise, it is deemed a NEGATIVE evaluation.

Table 18 shows that generally the schoolteachers have evaluated the textbook positively. Thus, according to them, the textbook met the criteria of a good EFL textbook in terms of the different variables. Accordingly, the schoolteachers are satisfied with the textbook.

Table 19 presents the evaluation of all the participants for each variable. In terms of objectives, language skills, language content, topics, and practical consideration, the schoolteachers have evaluated the textbook series as “Good.” However, in terms of design and organization, the textbook was deemed “Fair.”

Table 19. Score range for each variable

Variable	Score range
Objectives	Good (51–75)
Design and organization	Fair (26–50)
Language skills	Good (51–75)
Language content	Good (51–75)
Topics	Good (51–75)
Practical considerations	Good (51–75)

Williams’ checklist was also utilized to examine the different language skills separately and in-depth. Tables 20 and 21 offer an overview of Williams (1983) evaluative checklist.

Table 20. Rule of thumb in interpreting the mean score using Williams (1983) checklist

Variable	Great extent (4)	Large extent (3)	Some extent (2)	Just barely (1)	Not at all (0)
Williams (1983) checklist	69.0–64.4	64.3–59.8	59.7–55.2	55.1–50.6	50.5–46.0

Table 21. Schoolteachers’ evaluation with respect to each subcategory

Items	Mean	Critical value (Md)	Evaluation
Williams (1983) checklist	85.3	70.0	Positive

Tables 20 and 21 show that the textbook series is generally regarded as meeting the criteria of a good EFL textbook to “Some Extent.” The mean score of this checklist (85.3) falls between the range 86.7–82.2, which corresponds to “Some Extent.” Additionally, the general evaluation was positive. This finding echoes the findings of Keban et al.’s (2012) checklist. That is, the textbook was evaluated as “Fair” and it is evaluated as to “Some Extent,” which is a similar result. The sum of all quantitative measures (scores) of the variables in each subcategory was computed to give each subcategory a general “score.” After this, the mean of the “scores” of the 20 schoolteachers was computed in each subcategory (see Table 22).

Table 22. Schoolteachers’ evaluation using Williams (1983) checklist

Items	Mean	Critical value (Md)	Evaluation
General	13.0	10	Positive^1
Speech	11.9	10	Positive
Grammar	10.1	10	Positive
Vocabulary	12.1	10	Positive
Reading skills	12.5	10	Positive
Writing skills	11.0	10	Positive
Technical aspects	14.7	10	Positive

¹ If Mean ≥ Md, then the schoolteachers’ evaluation of the textbook Flying High for Saudi Arabia with respect to each subcategory is deemed a POSITIVE evaluation. Otherwise, it is deemed a NEGATIVE evaluation.

Table 22 shows that the textbook meets the criteria of a good EFL textbook with respect to the aforementioned criteria. Using Williams (1983) checklist, the variables of the textbook were evaluated positively.

From Table 23, it is obvious that in terms of general, speech, vocabulary, writing skills, and technical aspects, the schoolteachers have concluded that the textbook meets these criteria to “Some Extent,” while the textbook meets the criteria of grammar to a “Large Extent.”

Table 23. Score range for each variable

Variable	Score range
General	Some extent (2)
Speech	Some extent (2)
Grammar	Large extent (3)
Vocabulary	Some extent (2)
Reading skills	Some extent (2)
Writing skills	Some extent (2)
Technical aspects	Some extent (2)

Utilizing both checklists provided an in-depth analysis of the textbooks. However, it is important to examine the differences between the results each checklist offered. In Table 24, the difference between the mean scores in each checklist is presented.

Table 24. Difference between the mean scores in Keban et al.’s (2012) checklist and Williams (1983) checklist

Checklist	Mean	Test statistic	p-value	Decision
Keban et al.’s (2012) checklist	63.1	−9.732	0.000	Significant
Williams (1983) checklist	85.3

From Table 24, the results demonstrate that there was a significant difference (p = .001) in the mean scores between the two checklists. This suggests that using Keban et al.’s (2012) checklist showed that the textbook series met the criteria of a good EFL textbook more so than the results of Williams (1983) checklist did.

4. Discussion and recommendations

Employing the former two checklists revealed a number of findings regarding the quality of the textbook under examination from the perspective of the schoolteachers. The textbook evaluation is generally positive; however, both checklists disclosed some aspects that could be viewed as shortcomings. The shortcomings of the textbook contents are related to the learners’ culture and environment, the structures with regard to differences between L1 and L2 cultures, and problems of usage related to social background. Al-sowat’s (2012) study demonstrated the same problem regarding dissatisfaction with the cultural and environmental aspects of the textbook. As Faruk’s (2014) notes, the cultural context of the English textbooks in the KSA has changed and the evaluators of the present study were not happy with the cultural and social background of the textbook series. With reference to the vocabulary section, the shortcomings were related to the insufficiency of the vocabulary lists or glossaries and the failure to distinguish between receptive and productive skills in vocabulary teaching. With reference to the shortcomings of the speech variables, they are related to the English and L1 sound systems and the variation in the accents of non-native speakers of English. I do believe that the contrastive analysis of English and Arabic is not useful because of the differences between the two languages. Regarding non-native speakers’ accents, from my perspective as a second language learner, learners should be exposed to the native speakers’ accent. In addition, the evaluators believed that the review sections of the chapters were insufficient. Rahman’s (2012) study demonstrated almost the same result and the teachers are not satisfied with the textbook summaries. The summary and review sections should be investigated further to form a clearer picture of the problem. The review section might be adequate but the teachers would like to have a more comprehensive review that the students can refer to instead of the chapters themselves, a behavior that should be discouraged. Moreover, the evaluators indicated that the reading passages do not reflect a variety of styles of contemporary English. With reference to the written section, it has a few shortcomings related to practice, especially in terms of controlled and guided composition in the early stages, and the demonstration of different techniques for handling aspects of composition teaching. In addition, the written work is related to the pupils’ age, interests, and environment. Again, failure to accommodate the cultural and environmental differences between Saudi Arabia and the West was rated negatively. Al-Seghayer (2013) notes that there have been many complaints regarding the dominance of Western cultural elements in a number of University English textbooks; however, the academics’ response was that it is inevitable for western culture to be introduced in teaching a western language such as English. Regarding the writing section, the evaluators demonstrated that the series lacks exercises in controlled and guided composition in the early stages, the written work was suitable to the students’ age, interests, and environment, and that it lacks demonstration of techniques for handling aspects of composition teaching. Finally, the evaluation demonstrated that the evaluators were not satisfied with the quality of the paper and the binding materials of the textbook. The other criteria of both checklists were considered as points of strength.

4.1. Recommendations

In light of the analysis and discussion provided above, a number of recommendations can be suggested to improve the quality of the current textbook series and English language education in the KSA. These recommendations are intended primarily for use by the MOE, the curriculum developers/designers of the current textbook, and researchers, and the DFL at Taif University.

It is recommended that the MOE revise and re-evaluate the current textbook series in order to enhance its quality by addressing the different variables suggested by the checklists that were employed. Because textbook evaluation is “fundamentally a subjective” activity (Sheldon, 1988, p. 245), the findings of the current study should be cautiously viewed as evidence that further development and studies are required to ensure quality EFL education in the KSA. Hence, this study could be utilized to expand research on textbook evaluation in the KSA. Regarding the recommendations to the publisher, the textbook developer committee may need to review and revise the textbook in light of the current results.

The current study could be used to guide future research in several ways. First, more focus should be directed on the parties involved in the educational process. That is, the evaluation should involve stakeholders such as curriculum designers, teachers, and learners. The involvement of these parties can provide a broad view on the textbook and offer more than an in-depth evaluation. Additionally, this approach can reveal the understandings and perceptions of these parties, particularly the teachers and learners whose perceptions and attitudes are essential to the educational process. Evaluation by teachers and students can guide curriculum designers and development teams to develop a textbook that meets their requirements, needs, and expectations, which vary from one society to another and from one context to another. Finally, it is hoped that the findings of the present study could be used as a reference for FLD at Taif University when choosing textbooks related to the English language, specifically textbooks for first-year students.

5. Conclusion

The current study aimed at evaluating the textbook series of Flying High for Saudi Arabia, developed by Macmillan publishing house. This series is the English language textbook for high schools in Taif City and has been taught for over five years now. The participants who were requested to evaluate the textbook were schoolteachers who have been teaching the textbook for at least one year. To evaluate the textbook, the participants used the checklists of Keban et al. (2012) and Williams (1983). The evaluation of the textbook series has shown several important factors necessary to enhance the quality of the textbook and in turn enhance the quality of EFL education in the KSA. Textbook evaluation is a subjective practice (Angell, DuBravac, & Gonglewski, 2008); hence, in order to bring more objectivity to the current evaluation, experts should also conduct reviews of the curriculum and textbook, conducting further studies using different instruments.

Acknowledgments

Ahlam Alharbi would like to thank the anonymous reviewers whose valuable comments and suggestions improved the quality of this manuscript.

Additional information

Funding

This research is funded by Taif University, KSA [grant number 1-435-3288].

Notes on contributors

Ahlam Alharbi

Ahlam Alharbi is an assistant professor at the Department of Foreign Languages at Taif University, Saudi Arabia. She holds a doctoral degree in Linguistics from Monash University, Australia and a BA in English with TEFL. One of her research interests focuses on teaching English to non-native speakers as well as teaching linguistics. The Department of Foreign Languages has been contributing to the quality of education offered by Taif University through a major effort at curriculum reform. This paper is closely related to the effort exerted by the Department of Foreign Languages and to the interest of the author. This study provides the Department of Foreign Languages (DFL) at Taif University with a general idea about the English language educational background of the high school students who will join the department.

Appendix 1

PARTICIPANT CONSENT FORM

Project title: A descriptive-evaluative study of a Saudi textbook series

Researcher’s name: Ahlam Alharbi

•	I have read the Participant Information Sheet and the nature and purpose of the research project has been explained to me. I understand and agree to take part.
•	I understand the purpose of the research project and my involvement in it.
•	I understand that I may withdraw from the research project at any stage and that this will not affect my status now or in the future.
•	I understand that while information gained during the study may be published, I will not be identified and my personal results will remain confidential.
•	I understand that I may contact the researchers if I require further information about the research, and that I may contact Taif University, if I wish to make a complaint relating to my involvement in the research.

Print name…………………………………… Date……………………………

Signed……………………………………….

Appendix 2

Keban, Muhtar, and Zen’s checklist

Table

Variables	Checklist items	Score range
		Poor	Fair	Good	Excellent
		0–25	26–50	51–75	76–100
Objectives
1	The aims of the textbook correspond closely with the aims of the teaching program/curriculum.
2	The materials’ objectives are apparent to both the teachers and students.
3	The objectives are systematically organized.
4	The textbooks allow different teaching and learning styles.
Design and organization
1	The layout and design (in term of pictures, color) is appropriate and clear.
2	The textbooks are organized effectively (e.g. according to topics and skills, etc.).
3	The textbooks contain guidance about how they can be used.
4	The textbooks are accompanied with other teaching/learning materials (workbook, cassettes, and CD).
5	There was a review section and vocabulary list or glossaries included.
Language skills
1	All language skills (reading, writing, listening, and speaking) are balanced and integrated within each unit.
2	Suitability of listening skills to young learners.
3	Suitability of speaking skills to young learners.
4	The reading passages are sufficient and suitable for young learners’ level and ability.
5	Writing activities are suitable for Young learner’s level.
Language content
1	The language used in the textbook is authentic, i.e. like real-life English.
2	The grammar items are appropriate for the young learners.
3	The vocabulary items are appropriate for the young learners.
Topics
1	The topics of the textbook are relevant to the young learners’ needs as English language learners.
2	The topics of the textbook are interesting, challenging, and motivating.
3	There is sufficient variety in the topics of the textbook.
Practical considerations
1	The price of the textbook is reasonable.
2	The book is durable.
3	The textbook is a recent publication and easily accessible.

Appendix 3

Williams’ checklist

Table

Variables	Checklist items	Score range
		Great extent	Large extent	Some extent	Just barely	Not at all
		4	3	2	1	0
General
1	It takes into account currently accepted methods of ESL/EFL teaching.
2	It gives guidance in the presentation of language items.
3	It caters to individual differences in home language background.
4	It relates content to the learners’ culture and environment.
Speech
1	It is based on a contrastive analysis of English and L1 sound systems.
2	It suggests ways of demonstrating and practicing speech items.
3	It includes speech situations relevant to the pupils’ background.
4	It allows for variation in the accents of non-native speakers of English.
Grammar
1	It stresses communicative competence in teaching structural items.
2	It provides adequate models featuring the structures to be taught.
3	It shows clearly the kinds of responses required in drills (e.g. substitution).
4	It selects structures with regard to differences between L1 and L2 cultures.
Vocabulary
1	It selects vocabulary on the basis of frequency, functional load, etc.
2	It distinguishes between receptive and productive skills in vocabulary teaching.
3	It presents vocabulary in appropriate contexts and situations.
4	It focuses on problems of usage related to social background.
Reading
1	It offers exercises for understanding the main and implied meaning.
2	It relates reading passages to the learners’ background.
3	It selects passages within the vocabulary range of the pupils.
4	It selects passages reflecting a variety of styles of contemporary English.
Writing
1	It relates written work to the structures and vocabulary practiced orally.
2	It gives practice in controlled and guided composition in the early stages.
3	It relates written work to the pupils’ age, interests, and environment.
4	It demonstrates techniques for handling aspects of composition teaching.
Technical
1	It is up-to-date in the technical aspects of textbook production and design
2	It shows quality in editing and publishing (cover, typeface, illustrations, etc.)
3	It is dated.
4	It has authenticity in terms of the language and style of writing.

Appendix 4

Results of Keban, Muhtar, and Zen’s (2012) checklist

Table A: Evaluation of objectives variables

Table

Variables	Items	Mean	Critical value (Md)	Evaluation	Rating
Objectives
1	The aims of the textbook correspond closely with the aims of the teaching program/curriculum.	3.0	2.5	POSITIVE	Good (51–75)
2	The materials’ objectives are apparent to both the teachers and students.	2.8	2.5	POSITIVE	Good (51–75)
3	The objectives are systematically organized.	2.8	2.5	POSITIVE	Good (51–75)
4	The textbooks allow different teaching and learning styles.	2.5	2.5	POSITIVE	Good (51–75)

Table B: Evaluation of design and organization variables

Table

Variables	Items	Mean	Critical value (Md)	Evaluation	Rating
Design and organization
1	The layout and design (in terms of pictures, color) is appropriate and clear.	3.2	2.5	POSITIVE	Excellent (76–100)
2	The textbooks are organized effectively (e.g. according to topics and skills, etc.).	3.1	2.5	POSITIVE	Excellent (76–100)
3	The textbooks contain guidance about how they can be used.	2.5	2.5	POSITIVE	Good (51–75)
4	The textbook is accompanied with other teaching/learning materials (workbook, cassettes, and CD).	3.0	2.5	POSITIVE	Good (51–75)
5	There was a review section and vocabulary list or glossaries included.	1.9	2.5	NEGATIVE	Fair (26–50)

Table C: Evaluation of language skills variables

Table

Variables	Items	Mean	Critical value (Md)	Evaluation	Rating
Language skills
1	All language skills (reading, writing, listening, and speaking) are balanced and integrated within each unit.	2.8	2.5	POSITIVE	Good (51–75)
2	Suitability of listening skills to young learners.	2.9	2.5	POSITIVE	Good (51–75)
3	Suitability of speaking skills to young learners.	2.7	2.5	POSITIVE	Good (51–75)
4	The reading passages are sufficient and suitable for young learners’ level and ability.	2.6	2.5	POSITIVE	Good (51–75)
5	Writing activities are suitable for young learners’ level.	2.7	2.5	POSITIVE	Good (51–75)

Table D: Evaluation of language content variables

Table

Variables	Items	Mean	Critical value (Md)	Evaluation	Rating
Language content
1	The language used in the textbook is authentic, i.e. like real-life English.	2.6	2.5	POSITIVE	Good (51–75)
2	The grammar items are appropriate for the young learners.	2.8	2.5	POSITIVE	Good (51–75)
3	The vocabulary items are appropriate for the young learners.	2.7	2.5	POSITIVE	Good (51–75)

Table E: Evaluation of topics variables

Table

Variables	Items	Mean	Critical value (Md)	Evaluation	Rating
Topics
1	The topics of the textbook are relevant to the young learners’ needs as English language learners.	2.9	2.5	POSITIVE	Good (51–75)
2	The topics of the textbook are interesting, challenging, and motivating.	2.7	2.5	POSITIVE	Good (51–75)
3	There is sufficient variety in the topic of the textbook.	2.7	2.5	POSITIVE	Good (51–75)

Table F: Evaluation of practical considerations variables

Table

Variables	Items	Mean	Critical value (Md)	Evaluation	Rating
Practical considerations
1	The price of the textbook is reasonable.	4.0	2.5	POSITIVE	Excellent (76–100)
2	The book is durable.	2.0	2.5	NEGATIVE	Fair (26–50)
3	The textbook is a recent publication and easily accessible.	3.5	2.5	POSITIVE	Excellent (76–100)

Appendix 5

Results of Williams’s (1983) checklist

Table G: Evaluation of general variables

Table

Variables	Items	Mean	Critical value (Md)	Evaluation	Rating
General
1	It takes into account currently accepted methods of ESL/EFL teaching.	3.8	3.0	POSITIVE	Large extent (4)
2	It gives guidance in the presentation of language items.	3.1	3.0	POSITIVE	Large extent (4)
3	It caters to individual differences in home language background.	3.1	3.0	POSITIVE	Large extent (4)
4	It relates content to the learners’ culture and environment.	2.9	3.0	NEGATIVE	Some extent (3)

Table H: Evaluation of speech variables

Table

Variables	Items	Mean	Critical value (Md)	Evaluation	Rating
Speech
1	It is based on a contrastive analysis of English and L1 sound systems.	2.9	3.0	NEGATIVE	Some extent (3)
2	It suggests ways of demonstrating and practicing speech items.	3.1	3.0	POSITIVE	Large extent (4)
3	It includes speech situations relevant to the pupils’ background.	3.6	3.0	POSITIVE	Large extent (4)
4	It allows for variation in the accents of non-native speakers of English.	2.4	3.0	NEGATIVE	Some extent (3)

Table I: Evaluation of grammar variables

Table

Variables	Items	Mean	Critical Value (Md)	Evaluation	Rating
Grammar
1	It stresses communicative competence in teaching structural items.	3.4	3.0	POSITIVE	Large extent (4)
2	It provides adequate models featuring the structures to be taught.	3.4	3.0	POSITIVE	Large extent (4)
3	It shows clearly the kinds of responses required in drills (e.g. substitution).	3.2	3.0	POSITIVE	Large extent (4)
4	It selects structures with regard to differences between L1 and L2 cultures.	2.9	3.0	NEGATIVE	Some extent (3)

Table J: Evaluation of vocabulary variables

Table

Variables	Items	Mean	Critical value (Md)	Evaluation	Rating
Vocabulary
1	It selects vocabulary on the basis of frequency, functional load, etc.	3.7	3.0	POSITIVE	Large extent (4)
2	It distinguishes between receptive and productive skills in vocabulary teaching.	2.7	3.0	NEGATIVE	Some extent (3)
3	It presents vocabulary in appropriate contexts and situations.	3.7	3.0	POSITIVE	Large extent (4)
4	It focuses on problems of usage related to social background.	2.2	3.0	NEGATIVE	Some extent (3)

Table K: Evaluation of reading variables

Table

Variables	Items	Mean	Critical value (Md)	Evaluation	Rating
Reading
1	It offers exercises for understanding the main and implied meaning.	3.5	3.0	POSITIVE	Large extent (4)
2	It relates reading passages to the learners’ background.	3.3	3.0	POSITIVE	Large extent (4)
3	It selects passages within the vocabulary range of the pupils.	3.2	3.0	POSITIVE	Large extent (4)
4	It selects passages reflecting a variety of styles of contemporary English.	2.6	3.0	NEGATIVE	Some extent (3)

Table L: Evaluation of writing variables

Table

Variables	Items	Mean	Critical value (Md)	Evaluation	Rating
Writing
1	It relates written work to structures and vocabulary practiced orally.	3.6	3.0	POSITIVE	Large extent (4)
2	It gives practice in controlled and guided composition in the early stages.	2.5	3.0	NEGATIVE	Some extent (3)
3	It relates written work to the pupils’ age, interests, and environment.	2.5	3.0	NEGATIVE	Some extent (3)
4	It demonstrates techniques for handling aspects of composition teaching.	2.5	3.0	NEGATIVE	Some extent (3)

Table M: Evaluation of technical variables

Table

Variables	Items	Mean	Critical value (Md)	Evaluation	Rating
Technical
1	It is up-to-date in the technical aspects of textbook production and design	3.7	3.0	POSITIVE	Large extent (4)
2	It shows quality in editing and publishing (cover, typeface, illustrations, etc.)	3.7	3.0	POSITIVE	Large extent (4)
3	It is dated.	3.7	3.0	POSITIVE	Large extent (4)
4	It has authenticity in terms of the language and style of writing	3.6	3.0	POSITIVE	Large extent (4)