Analysing the impact of external examination on teaching and learning of English at the secondary level education

Abstract

This study assesses the influence of public examination on teaching and learning in higher-secondary education. The focus is on the subject of English, taught and assessed at grades 10 and 12 in Pakistan. Using the case of the BISE Sukkur, this study attempts to evaluate the English examination papers from the last 10 years in the first phase. Using Bloom’s learning taxonomy as a theoretical framework, each item in the question papers was analysed to determine, (1) the cognitive complexities required by the students to respond to the item, (2) the variety of topics and the frequency with which the items are repeated in the different years, and (3) the chapters from which these items were selected. In so doing, the study highlighted (a) the patterns in which the high and low order learning are prioritised in the examination papers, (b) the frequency and variety in which topics are repeated in different years, and (c) the frequency in which the chapters from the prescribed textbooks are priorities or ignored in the examinations. To determine the ways in which this examination pattern influences teaching and learning practices in the schools affiliated with the BISE Sukkur, teachers and students from selected schools were interviewed in the second phase of the study. The findings show a strong correlation between examination patterns and teaching practice and students’ learning approaches.

Keywords:

• assessment
• public examination
• higher-secondary education
• high- and low-order learning
• Bloom’s taxonomy

Public Interest Statement

This study shows the influence of external examination on English teaching and learning at the higher secondary level education. The study is conducted in two phases. In the first phase, the quality of external examination is assessed. Using the case of Board of Intermediate and Secondary Education (BISE) Sukkur Pakistan, this study analysed last 10 years examination papers of English subject from grade ten and twelve to (1) analyse each item for its cognitive complexity using Blooms Taxonomy, (2) determine the repetition trend of items in last 10 years, and (3) determine the chapter of prescribed book from which these items are selected. In the second phase, the influence of these external examinations on the way teachers teach and students learn in the affiliated schools were studied. The findings from both phases show a strong correlation between examination patterns and teaching practice and students’ learning approaches.

1. Introduction

Assessment plays a vital role in the development of students. It is also used to determine whether or not a student is ready to move up to the next grade (Borghouts, Slingerland, & Haerens, 2017). In order to achieve a certain level of standardisation, governments worldwide draw to public examination systems to assess the abilities of students for the next level of education (Goldstein & Leckie, 2016; Lau & Tam, 2017). Despite their many benefits, public examinations are criticised for promoting examination-oriented pedagogies as the results of these examinations determine the quality and standards of the schools (Islam, 2016; Kirkpatrick & Zang, 2014). Research suggests that those institutions where examinations are conducted by external bodies and whose performance and reputation are associated with the results, usually promote examination-oriented teaching practices (Adegoke, 2010; Aftab, Qureshi, & William, 2014; Kirkpatrick & Zang, 2014; Rehmani, 2003). Hence, it can be assumed that if examination promotes low-order learning—i.e. remembering, understanding or applying knowledge, or high-order learning—i.e. deploying analytical, evaluative and creative skills—then teachers and students would also focus more on low or high order learning respectively. Likewise, if certain topics or particular chapters from prescribed books are frequently covered or ignored in the examinations; then, teachers and students would also prioritise or ignore them accordingly.

Public examination has flourished in developed countries, mainly because of continuous scrutiny of their standards through research. In Pakistan, however, very little systematic and comprehension research can be found to determine the quality of public examination at secondary and higher secondary levels and its effects on teaching and learning. This research attempts to fill this gap by conducting a comprehensive study of examinations conducted by Board of Intermediate and Secondary Education (BISE) Sukkur, and the effect of these on the teaching of English and on students’ learning at the affiliated schools and colleges. Using the case of BISE Sukkur, this study seeks to determine:

• the level of learning being assessed in examinations set by BISE Sukkur and the ways in which it shapes teachers’ and students’ teaching and learning approaches in English;

• how frequently topics are repeated in the examinations and how this influences teachers’ and students’ decision to focus specifically on certain topic or ignore other topics; and

• how frequently topics are selected from the chapters of the prescribed textbooks, and the ways in which these influence students’ and teachers’ decisions to focus on or ignore certain chapters from prescribed textbooks.

2. English in secondary and higher secondary education in Pakistan

English has become the language of international research, trade, commerce and communication in Pakistan. Therefore, many government policies acknowledge its indispensable role in education. More specifically, it has become a compulsory subject for students in grades 1 to 12, and is used as a medium of instruction across the curriculum for various other subjects. Realising the importance of English, the government planned and implemented a number of initiatives, including the development of the National Curriculum 2006 for English. The Curriculum outlines the five competencies (referred to as C) required of students in grades 1 to 12. These competencies include (C1) reading & thinking skills, (C2) writing skills, (C3) oral communication skills, (C4) formal & lexical aspects of language, and (C5) appropriate ethical & social development (Curricullum, 2006, p. 3). The Curriculum also sets specific learning standards (referred as S) for each competencies. For example, the reading & thinking skills has two standards: “S1—All students will search for, discover and understand a variety of text types through tasks which require multiple reading and thinking strategies for comprehension, fluency and enjoyment”, and “S2—All students will read and analyse literary text to seek information, ideas, enjoyment, and to relate their own experiences to those of common humanity as depicted in literature” (Curricullum, 2006, p. 7). Benchmarks (referred as B) are set against each standard according to the grade. These are informed by Bloom’s taxonomy (Anderson et al., 2001); the level of the benchmark, therefore, increases gradually for the advanced grades. For example, there are five benchmarks of C1S1 for Grades 1–12 (see Table 1).

Table 1. Benchmarks for C1S1

	Grades I–II	Grades III–V	Grades VI–VIII	Grades IX–X	Grades XI–XII
B1	Use reading readiness strategies; recognise words and sentences as meaningful units of expression and paragraphs as graphical units of expression.	Identify digraphs, silent letters and inflections in words; comprehend words, sentences and paragraphs as meaningful units of expression.	Analyse patterns of text organisation, and function of various devices used in a paragraph.	Analyse patterns of text organisation, and function of various devices used within and beyond a paragraph in a text.	Evaluate patterns of text organisation, and the function of various devices used within and beyond a paragraph in a text.
B2	Identify factual information applying reading comprehension and thinking strategies.	Interpret factual information, new processes and procedures, personal-, school- and public-related information, applying reading comprehension and thinking strategies.	Extend conceptual understanding of processes, procedures, events and issues, applying reading comprehension and thinking strategies.	Analyse complex processes, procedures, events, issues and various viewpoints, applying reading comprehension and thinking strategies.	Analyse, synthesis and evaluate events, issues, ideas and viewpoints, applying reading comprehension and thinking strategies.
B3	Locate information from a visual cue or a graphic organiser and express the information verbally.	Comprehend information from a visual cue or a graphic organiser to describe positions, directions, events and sequences, and to show comparison and contrast.	Interpret information from a visual cue or a graphic organiser to give directions and describe positions, simple processes and procedures and cause-and-effect relationships.	Analyse information from a visual cue or a graphic organiser to show complex processes, procedures, comparisons, contrasts, and cause-and-effect relationships.	Analyse and synthesis information from a visual cue or a graphic organiser to summarise, highlighting the key areas and main trends.
B4	Locate information for specific purposes using various aids and study skills.	Gather and use information for a variety of purposes using various aids and study skills.	Gather, analyse and use information for a variety of purposes using various aids and study skills.	Gather, analyse, evaluate and synthesis information to use for a variety of purposes including a research project using various aids and study skills.	Gather, analyse, evaluate and synthesis information to use for a variety of purposes including a research project using various aids and study skills.

Adopted from Curriculum (2006, p. 11).

The review of benchmarks in Table 1 suggests that students in grades I–II are expected to “remember”, “recognise”, “identify”, and “locate” different aspects of C1S1, whereas those in grades III–V are expected to “comprehend”, “interpret”, and “apply” the different aspects. Likewise, students in grades VI–XII are expected to “analyse”, “extend conceptual understanding”, “analyse complex procedures”, “compare”, “contrast”, “evaluate”, and “synthesise” the different aspects of C1S1. Each benchmark is further divided into measurable student-learning objectives (SLOs) to ensure its implementation.

In order to facilitate the work of teachers, a detailed teaching methodology for each competency has also been proposed in the National Curriculum (Curricullum, 2006, pp. 150–152), along with a very specific assessment strategies (Curricullum, 2006, pp. 154–157). The proposed teaching and assessment strategies ensure that each SLO is properly implemented as well as measured.

3. The examination in secondary and higher-secondary education in Sindh, Pakistan

Secondary and higher secondary education in Sindh is regulated by the government’s Education and Literacy Department (ELD). The ELD arranges and manages grant scholarships, executes different projects or schemes through donor coordination, administers human resources, undertakes capacity building for teachers, promotes research, and maintains supply and demand-side interventions (GoS, 2014). Before 2006, Sindh, like other provinces, had the provincially regulated Board of Curriculum and Extension Wing (BCEW) — responsible for developing and designing of curricula for primary, middle, secondary and higher secondary education; and the Sindh Text Book Board (STBB) — responsible for publishing books following the curriculum recommended by BCEW. In 2006, however, the Curriculum Wing of the Ministry of Education (CWMoE) recommended a new National Curriculum 2006. Simultaneously, the National Textbooks and Learning Materials Policy and Plan of Action was introduced in 2007 to produce quality textbooks through a competitive process between private sector publishers. However, due to a number of factors, provincial textbooks for the new curriculum could not be introduced until the academic year 2012–2013. In 2010, the federal government devolved the Federal Ministry of Education to the provincial level, empowering the ELD in Sindh and its allied departments — particularly the BCEW — to innovate in the implementation of the National Curriculum 2006. The examination of secondary and higher secondary education is conducted by five Boards of Intermediate and Secondary Education (BISEs) operating in Karachi, Hyderabad, Sukkur, Mirpurkhas and Larkana. These BISEs are autonomous bodies, governed by the Chief Minister, supported by a chairperson, a secretary, a controller and board members.

In summary, secondary and higher secondary education is regulated by the EDL, its curriculum is designed by the BCEW, its syllabi are published by the STBB and taught in government as well as private schools, and all examinations are conducted by BISEs. However, there is no apparent official mode of communication between BISEs and the BCEW or the STBB before, during or after examinations. BISEs prepare examination papers, administrate examinations, assess and grade papers, and disseminate the results independently.

4. Public examination system and English language teaching and learning

Public or external examinations are indicators of achievement which are critical to students’ success in their further studies (Madaus, 1991). The public examination system provides an opportunity for fair comparisons across the various schools and their students in a particular region (Madaus & Kellaghan, 1991). In Ireland, the public examination system prevails because, if the assessment of students is dependent on schools autonomously, it may lead to legal accountability for marks awarded by teachers, which may be subject to distortion due to parental pressure or that originating from the competition between local schools. Some argue that the public examination system gives students extrinsic motivation per se but weakens their motivation to put in any effort and learn. Instead, “preparation for high stakes tests often emphasises rote memorisation and cramming of students and drill and practice teaching methods” (Madaus, 1991, p. 7)

Most importantly, public examination guides teaching and learning in schools (Peterson, 2007). A number of studies on public examination suggest that those institutions where examinations are conducted by external examination boards and whose performance is associated with the results, usually promote examination-oriented teaching practices (Adegoke, 2010; Aftab et al., 2014; Aworanti, 2011; Rehmani, 2003).

In the context of the English language, the impact of examination on teaching and learning has been dubbed as backwash or washback (Alderson & Wall, 1993; Bachman & Palmer, 1996; Biggs, 1995; Cheng, 1997; Gu & Saville, 2016; Messick, 1996; Pan, 2016; Spolsky, 1994). Spolsky (1994, p. 2) defines backwash as a concept that “deals with the unforeseen side-effects of testing and not to the intended effects when the primary goal of the examination is the control of curricula”. Likewise, Biggs (1995, p. 3) maintains that “backwash refers to the fact that testing controls not only the curriculum but also teaching methods and students” learning strategies’. Washback is defined as “the extent to which the introduction and the use of a test influences language and teachers to do things they would not otherwise do that promote or inhibit language learning” (Messick, 1996, p. 4). Similarly, Alderson and Wall (1993, p. 1) argue that washback compels “teachers and learners to do things they would not necessarily otherwise do because of the test”. Pearson (1988, p. 7) argues that “public examinations influence the attitudes, behaviours, and motivation of teachers, learners, and parents, and because examinations often come at the end of a course, this influence is seen working in a backward direction, hence the term, washback”.

Washback or backwash in relation to teachers yields exam-oriented pedagogies. Although these latter can have both positive and negative effects — referred as the positive or negative washback effect — on educational practices (Hughes, 2007), these are usually negatively portrayed in the mainstream literature, mainly because of the quality of the examination. For example, if the emphasis in an exam is on lower order learning items, then teaching practice may also focus on low order learning (Kirkpatrick & Zang, 2011; Qi, 2004).

Bloom defined low and high order learning in his learning taxonomy (Bloom, Krathwohl, & Masia, 1984). Bloom categorises learning in a hierarchal order according to its cognitive complexities on six levels. He named the first level Knowledge, where students memorise facts and figures. The second level is termed as Comprehension, where students understand and interpret facts in their own way. The next level is Application, where students apply their knowledge. These first three levels fall in the low order learning category. The fourth level is Analysis, and here students analyse different situations in order to draw their conclusions. The fifth level is termed as Evaluation where students make judgments after a comprehensive learning process. The sixth and final level is called as Creation, where students are expected to produce new knowledge. These learning levels are also commonly used by course designers to set benchmarks and students’ learning objectives for specific courses. Usually, there is an alignment between students’ learning objectives, their course material, their teachers’ strategies, and assessment strategies.

Likewise, if an examination follows a specific pattern by repeating questions year after year for a considerable period; then, it is very probable that students will realise this and pass the exam without going through the whole syllabus. In the same way, if an examination ignores certain chapters from a prescribed course, and then the students and teachers may also ignore these chapters (Kirkpatrick & Zang, 2011; Qi, 2004). Fullilove (1992, p. 139) refers to such examinations as “little more than cloners of past exam papers”.

Pakistani public examination has long been criticised for putting too much pressure on students, who are then forced to have private tuition in addition to formal schooling in preparation for their examinations; students are encouraged to refer to past exam papers and mostly to memorise the concept for better results (Aftab et al., 2014). Moreover, the reliability and validity of the examination are also questionable as most critics claim that it does not cover the curriculum and that topics are frequently repeated (Rehmani, 2003).

In summary, exams play a vital role in the development of students as they are also used to determine whether or not a student is ready to move up to the next grade. In order to achieve a certain level of standardisation, governments worldwide draw on public examination systems to assess the ability of students for the next level of education. Despite a lot of criticism, public examination flourishes in developed countries mainly because of continuous scrutiny through research. In Pakistan, however, very little systematic and comprehension research can be found to determine the quality of public examination at secondary and higher secondary levels and its effect on teaching and learning. This research attempts to fill this gap by conducting a comprehensive research on examination conducted by BISE Sukkur, and its impact on teaching practices and students’ learning of English at the affiliated schools.

5. Methodology

This study was conducted in two phases. The focus of the first phase is to review the BISE Sukkur examination papers in English using a desk review approach. Examination papers for English grades 10 and 12 from 2006–2016 were thoroughly reviewed. The exam papers used variety of methods to test students’ knowledge, including multiple choice questions (MCQs), constructed response questions (CRQs) and extended response questions (ERQs). A grade 10 paper usually includes 15 MCQs, 12 CRQs and 5 ERQs. Likewise, a grade 12 paper would have 20 MCQs, 13 CRQs and 4 ERQs. In this way, 320 MCQs, 250 CRQs, and 90 ERQs were analysed in the first phase of this study. Each item was given three codes. The first code determined the cognitive complexities required by the students in responding to the items. If the items only assess students’ memory it was coded as L1—referred to the first level of learning according to Bloom’s taxonomy. The second code determined the variety of, and the frequency with which the items were repeated each year. If the item was repeated word for word then it was coded as A; if it was repeated with minor changes then it was coded as A1; if it was repeated with significant changes then it was coded as A2. The final code determined the chapter from which the item was selected. For example, if the item was selected from Chapter 1 then it was coded as Ch.1. The coded items were compiled in MS Excel, and the findings set out in graphs and charts for further interpretation.

The second phase of the study focused on understanding the teaching practice and students’ learning approaches in BISE Sukkur-affiliated schools. Eleven teachers and 21 students of different genders, ethnicity and background were selected from five of these schools. Semi-structured interviews were used as the main tool for data collection. Interview guides were strategically prepared separately for teachers and students. Firstly, examination papers were carefully reviewed in order to identify the different kinds of question used in the paper. For example, in the grade 12 English exam paper, it was found that three kinds of question were frequently given—MCQs, CRQs, including reading comprehension, and ERQs, which include explaining poems, describing characters and writing long essays. Student interviewees were asked four main questions: How did your teacher teach you X [the subject]? How did you learn X? How did your teacher prepare you for X in the exam? How did you prepare for X in the exam? Although the interviews were semi-structured, participants were given full freedom to express their opinion in a variety of ways. The data were analysed using NVIVO 9. The findings from Phase 2 were further analysed in the light of findings from Phase 1.

6. Findings and discussion

6.1. Assessment of the learning level in grades 10 and 12

In reviewing grade 10 examination papers, it found that about 41% of the items required students to memorise, 42% assessed students’ comprehension/understanding and 18% required students to apply their knowledge to respond to the items. None were found that assessed students’ analytical, evaluative or creative skills. The findings from the Grade 12 examination were also similar, where 29% of the items were at the level of “knowledge”, 52% were “comprehension” items and 19% required “application”. There were no items that assessed students’ analytical, evaluative or creative skills (see Figure 1).

Figure 1. Levels of learning in English at grades 10 and 12.

6.2. Repetition of items

Looking at the trend towards item repetition in Grades 10 and 12 in the examination papers, it was found that a significant number of items were frequently repeated in the ten-year period. In the Grade 10 English paper, around 26% were repeated as is, with no change, 13% with minor changes and only 12% with significant changes. Overall 52% of items were repeated year after year. The same applied to Grade 12 English, with nearly 19%, 21% and 7% repetition, respectively. Overall, 47% of the items had been repeated over the previous 10 years (see Figure 2).

Figure 2. Repetition trend in English of grades 10 and 12.

6.3. Syllabus coverage in the examination

In analysing for inclusion in the examination papers items based on the different chapters of the prescribed Grade 10 syllabus, it was found that there were two chapters which were completely overlooked, one from which only 5% or 6% of items were selected, six chapters from which 3% to 4% were included and 17 from which only 1% or 2% of were selected in the 10 years of examination papers covered in this study (see Figure 3).

Figure 3. Percentage of items selected in the examination from different chapters: English grade 10.

The situation is quite similar to that for grade 12, where it was found that only one chapter from which more than 10% of the items were drawn in 10-years’ worth of examination papers. Between 5% and 6% of the items were selected from one chapter, 10 chapters from which 3% to 4% from 10 chapters and 1–2% from four chapters; three chapters were totally ignored.

6.4. Teaching and learning trends in BISE Sukkur-affiliated schools

As mentioned above, in order to understand the impact of examination on teachers’ practices and students’ learning approaches, students and teachers were both interviewed. The findings, which are summarised in Table 2, suggest that teachers adopt examination-orientated pedagogies which mostly promote L1 and L2 learning among students. In the following section, teachers’ strategies and students’ learning approaches in relationship to MCQs, CRQs and ERQs are discussed in detail.

Table 2. Teachers’ teaching and students’ learning approaches in grade 12 English at BISE Sukkur

Item type	How teachers teach	Teaching methods	How students learn	How teachers prepare for exams	How students prepare for exams
MCQ	Reading; Emphasis on remembering (L1)	Lectures; Students’ reading separately; Practice tests; Note-taking; Emphasis is on important topics and chapters.	Remembering (L1)	Notes provided for remembering	Remember/cram notes (L1)
CRQ	Reading; Emphasis on remembering (L1)		Remembering (L1)	Notes provided for remembering	Remember/cram notes (L1)
ERQ			Remembering and understanding (L1/L2)	Notes provided for remembering	Remember/cram notes; Reading for understanding (L1/L2)
Explaining the poems	Reading; Emphasis on remembering (L1/L2)
Describe characters	Reading; Emphasis on remembering (L1)		Remembering (L1)	Notes provided for remembering	Remember/cram notes (L1)
Essays	Ignored		From guides and grammar books; Practice (L2/L3)	Sample essays	Remember some essays; Depends on their understanding (L2/L3)

The findings from the first phase of the study suggest that the majority of MCQs are constructed in a way that enables assessment of students’ L1 and L2 learning. Most MCQs require students to remember facts and figures from their lessons, the names, ages or status of characters from a novel they have studied in school, synonyms/antonyms and so on. This significantly influences the way in which teachers deliver the different lessons in school. It was found that teachers usually put the emphasis on reading the text followed by practice tests which are based on the BISE Sukkur format. Teachers prepare a number of MCQs from lessons, most of which are based on L1 and L2. Students are provided with these notes, which they are required to learn by heart and remember for their exams.

The findings of the first phase also suggest that the majority of CRQs are based on L1 and L2. CRQs require students to recall events from the lessons, the features of characters from novels, themes of the poems studied, quotations by characters, the usage of given idioms in sentences, and be able to convert direct speech into indirect, active-voice sentences into passive voice and so on. Teachers were found to adopt the same strategy—i.e., they prepare lectures on the lessons, mostly to narrate the story in their own words. They prepare notes which are based on the “important” topics. A topic is dubbed as “important” by both teachers and students if it is repeated frequently in the exam papers of the previous five years. Students are required to remember the “important” questions in their exam preparation. Grammar is mostly taught separately—students learn the basic rules in class, followed by practice tests, most of which require students to convert isolated sentences from direct into indirect speech, or active-voice sentences into passive. Idioms and vocabulary are taught in isolation as well, and students are given a long list of “important” idioms and vocabulary to remember.

There are four types of ERQ which usually appear in exam papers, including the description of characters, the events from novels studied, an explanation/description of a poem, and the writing of essays.

The findings from first phase suggest that the way in which questions asking for the description of characters or events from a novel or the explanation of poem are put in the exam papers usually triggers students’ L1 or L2 cognitive abilities. The following two items appeared in the 2016 exam paper:

Narrate in detail the incident of the ‘Summer House’ in the novel The Prisoner of Zenda (BISE Sukkur English Grade 12, 2013).

Write the character sketch of any one the following: (i) Roudolf Raseendyll, (ii) Princess Flavia, (iii) Col. Sapt (BISE Sukkur English Grade 12, 2016).

We found that teachers always keep the exams in mind when teaching a novel. For example, the teacher narrates the story of the whole novel in his or her words for the students, emphasising the “important events”. Once again an event in the novel is dubbed as important because it has usually appeared in previous examination papers. It was not surprising to learn that majority of the students do not even read the text of the novel. They remember the summarised story and important events narrated by teachers. Considering that the novel has been part of the syllabus for almost 15 years now, a variety of well-written novel character sketches are available on the market, which teachers refer students to. Students are found to study in depth three characters out of a possible 11, as one of these three characters has always been repeated in the previous 10 years’ exam papers.

Similarly, poems are covered in the exam in a way that promotes students’ L1 and L2 learning. The following is an example of such a question from the 2015 exam:

Write a critical appreciation of the poem ‘The Seven Ages of Man’ or ‘The lost Star’ (BISE Sukkur English Grade 12, 2015).

Although fancy phrases like “critical appreciation” are also frequently used in questions about poems in exam papers, the way in which the question is posed does not trigger students’ analytical skills. It was found that teachers only taught those poems which are frequently repeated in the examination papers, and that the way in which they teach poetry reflects the influence of the exams. Considering that these exams assess students’ L1 and L2 learning only, teachers prepare long monologues to provide a simple explanation of a poem without making any connection between it and its author, any reference to the context in which it was written or to the ways in which the poem may reflect or have any connection to students’ lives. Most of the teachers were unable to explain the term “critical appreciation” appropriately when asked during interview, and confused the term with “extended summary”. Teachers usually highlight important poems which may come up in the examination and suggest that students use the readily available well-written summaries of the poems on the market. Most of the students memorise the notes. However, some students were found to be rewriting these notes to give some variety in their answers. However, these were individual efforts with little support from the teachers.

We also found that majority of the teachers did not teach essay writing in class at all. A few of those who did spent only one or two 50 min lectures explaining the basic structure of an essay. In both cases, the main strategy for preparing students for exams was to present a list of “important titles”—again, important titles are those which are repeated frequently in previous exams. Some teachers suggested that students remember those titles which offer a kind of “flexibility” in the examination, which here refers to the opportunity for the students to use the material of one title in many other titles. One such example is “Allama Iqbal—the national poet of Pakistan”. This title offers “flexibility” to students in the way that the material of this title can fit into many other titles such as “national poet”, “favourite personality”, “role model”, etc. Although essays can effectively be used to promote students’ L4 and L5 learning, the way in which short titles are frequently repeated in the examinations encourages students to use L1 and L2 learning.

7. Conclusion and implications

The findings of this paper suggest that BISE Sukkur examinations prioritise low-order learning and mostly assesses students’ memory and comprehension skills. Even the application-level questions are developed in a way that assesses students’ procedural knowledge instead of their application skills. There were hardly any analytical questions and no evaluative or creative-level questions in the examination papers of the 10 years covered by our study. These findings contradict the standards required for the National Curriculum for English (Curricullum, 2006), which sets a high benchmark for students in Grades 6–12. According to the National Curriculum 2009, students in these grades are expected to develop analytical, synthesis and evaluative skills in different competencies of English. However, the exams only focus on memory and comprehension, which we found to have serious washback effects on students and teachers. Teachers are found to have adopted those teaching strategies which promotes memorisation and comprehension. These findings complement the findings of Tayeb, Aziz, Ismail, and Khan (2014) who conducted a similar study in the context of English examinations at the secondary education at Yemen, and found that teachers’ teaching strategies are highly influenced by the nature of the items in the examination. Sukyadi and Mardiani (2011) also found that teacher’s choice of material and resources at the secondary education in Indonesia are highly influenced by the nature of questions asked in the examination. In an informal meeting with the representatives of the BISE Sukkur, when they were asked their reasons for using low order learning items in the examinations, they blame the poor education system, which according to them, does not prepare students to solve high order learning items. According to them, if they use the high order learning items in the examinations, and then the overall percentage of passing students may significantly decrease. They also emphasised that the entrance in the medical and engineering universities heavily depend on the results of examinations conducted by BISEs. In Pakistan, parents show great enthusiasm in getting their children admitted in medical and engineering universities and can go to great lengths in making it possible (). Since the admission in aforementioned universities depends 40% on the results of BISEs and 60% on the results of entrance test, parents and students take BISEs results very seriously. Introducing high order learning items in the examinations conducted by BISE would significantly reduce the percentage of high graders, thus may result in an outcry by parents and community members. This excuse was frequently quoted by the representatives of the BISE Sukkur to defend their use of low order learning items in the examination.

The findings of the current study also revealed that not a single question was included in the exam papers which would have assessed students’ evaluative and creative skills. It is very difficult to assess students’ evaluative and creative skills using unseen assessment due to this limitation (Falchikov, 2013). In order to evaluate something, students need to gain information from different sources on the topic and analyse it before making their evaluative judgement. In this process, some students end up creating new knowledge. However, all this requires time and resources. Thus, seen or open-book assessment is considered to be appropriate when the aim of the teachers is to assess students’ evaluative and creative skills (Falchikov, 2013). However, BISE Sukkur has no provision for seen or open-book assessment. Although, the representatives of BISE Sukkur acknowledge the importance of the open-book assessment in assessing students’ high order learning skills, the logistic issues were frequently quoted as an excuse not to introduce it. More specifically, the large number of candidates for examination, limited time for checking and grading (as they get deadlines from Chief Minister Office to announce results within 90 days), and availability of trained assessors were the issues they frequently mentioned. With all these issues, BISE Sukkur ignores the high order learning items in the examinations, and our findings show that it has a very strong washback effect on teachers’ teaching approaches, where analytical, evaluative and creative skills are ignored.

Moreover, it was found that questions were frequently repeated “as is” in the exam papers of the 10 years covered by our study, and are likely to continue to be repeated, as the same syllabus has been used for a while. However, when the questions are repeated word-for-word every year, it suggests that those setting the papers lack creative and innovative ways to develop questions and therefore simply copy-paste those from previous years.

This trend also influences teachers’ and students’ choice of the topics to cover or ignore in schools. Students and teachers dub topics “important” if these are frequently repeated in the examination and therefore emphasis is placed on these topics only. Teachers were found to review the previous examination papers, prepare a “Guess Paper”— a guide which a list of expected questions to be repeated in the coming examination, and provide pre-prepared notes to the students (Rehmani, 2003). Even the local publishers get the advantage of this situation and publish guides that are easy to use both for memorisation or use to cheat in the examination (Khattak, 2012).

Finally, it was found that some chapters from prescribed textbooks were completely overlooked in the exam papers of the last 10 years, while other chapters were given significant importance, with many questions being frequently selected from them in the examination papers. This has implications for teaching as well. Teachers were found to not cover those chapters which were ignored in the previous exam paper, considering that they were not important from an examination point of view. The research of Rind (2015) found the same trend in high education as well where students and teachers ignore those chapters from the core books which are not frequently repeated. As curricula of different grades are developed with students’ levels of education in mind (Stein, Remillard, & Smith, 2007), it is important for a student to go through the complete syllabus of Grade 11, for example, in order to perform better in Grade 12, as the concepts taught at the latter grade are usually based on those taught in the former. Teachers adopting exam-orientated practices ignore all those chapters which were not included in the previous year’s examination. Thus, students may get good grades, but their foundation is weak for the next grade (Nomaan, Hanif, & Rehna, 2016). All these findings highlight the importance of the external examination conducted by the BISEs in Pakistan. Since there are high stakes associated with the results of the examinations conducted by BISEs, the patterns of examination highly influence the teaching and learning in the affiliated schools.

8. Future research

The next phases of the study will focus on the impact of both examinations on teaching practices in other subjects and on exam-oriented pedagogies at higher-secondary level on the students’ progress in higher education.

Additional information

Funding

The authors received no direct funding for this research.

Notes on contributors

Irfan Ahmed Rind

Dr. Irfan Ahmed Rind is currently working as Head of Education Department at Sukkur IBA University. He has got his PhD. in Education and Master in Applied Linguistics from University of Sussex, UK. More recently, he got his post-doctorate professional certification from Harvard University. Apart from teaching and administration Dr. Rind involves in different research projects sponsored by British Council, USAID, RSU, CIDA, World Bank and HEC. Currently, he is leading a USAID & Govt of Sindh sponsored (worth PKR.700 million) project titled “Public Private Partnership: Education Management Organization for the Government schools”. In last two years, he won two prestigious research awards by Higher Education Commission Pakistan under the (1) National Research Award for Universities (NRPU) in 2016 and (2) Thematic Research Grant in 2017. Dr. Rind enjoys teaching and recently was awarded with a national level Best University Teacher Award by Higher Education Commission, Pakistan in 2018.