Identifying difficulties and best practices in catering to diversity in CLIL: instrument design and validation

ABSTRACT

This paper makes available to the broader educational community the instruments which have been originally designed and validated within the European project CLIL for all: Attention to diversity in bilingual education (ADiBE) to determine how diversity is being catered to across a broad array of CLIL contexts in European Secondary Education (Austria, Finland, Germany, Italy, Spain, and the United Kingdom). They include three sets of questionnaires, interviews, and observation protocols and are qualitative and quantitative instruments whose design has been based on the latest research and which have undergone a carefully controlled double-fold pilot process for their validation (external ratings approach and pilot phase with a representative sample of 264 subjects). The questions included in the three sets of instruments are initially characterized, together with their format and main categories. The paper then details the steps undertaken for their research-based design and the double-fold pilot process followed for their validation. The questionnaires and interview and observation protocols are then presented in a format which is directly applicable in any CLIL classroom in order to determine the accessibility of bilingual programs for all types of achievers and to identify the chief difficulties and best practices in promoting inclusion in bilingual education.

KEYWORDS:

• CLIL
• diversity
• questionnaire
• interview
• observation
• validity

Introduction

After a quarter of a century of implementation, Content and Language Integrated Learning (CLIL) has been gathering ‘a momentum unlikely to be reversed in the foreseeable future’ (Dearden and Macaro 2016, 456) and is now largely considered ‘an unstoppable train which has already left the station’ (Macaro 2019, 232). However, after a lengthy period of positive appraisal and unbridled enthusiasm (Paran 2013), the criticism leveled at CLIL has been growing conspicuously particularly over the course of the past decade (Pérez Cañado 2020a). If there is one aspect which has found traction within the skepticism aroused by this approach to bilingual education that is the charge of elitism and segregation (Gortázar and Taberner 2020). Indeed, strong claims have been made for the exclusive gatekeeping inherent in CLIL. As Bruton (2011, 523) underscores, ‘there is every reason to believe some students may be prejudiced by CLIL’, as ‘rather than increasing the equality of opportunity, CLIL in certain contexts is subtly selecting students out’ (Bruton 2013, 593). In this sense, Paran (2013, 331) upholds that CLIL ‘probably works best in elite contexts’ and that ‘Implicitly, CLIL is likely to be elitist and cream off certain students’ (Bruton 2013, 595).

However, a growing body of robust research has recently shot down the belief that CLIL selects students out (Ainsworth and Shepherd 2017; Madrid and Barrios 2018; Pavón Vázquez 2018; Rascón Moreno and Bretones Callejas 2018; Pérez Cañado 2020a, 2020b). Indeed, three main findings have been salient in these recent investigations. To begin with, CLIL and non-CLIL students have been found to be increasingly homogeneous in hundreds of randomly chosen schools where both types of streams co-existed. Secondly, CLIL has been found to work very successfully even in the most disenfranchised settings: in rural contexts, public schools, with low socioeconomic status (SES), and with minority ethnicities. Finally, CLIL has been found to have a leveler effect, since, while differences in terms of setting, SES, or type of school are sustained in non-bilingual groups, they phase out in CLIL branches. This purported leveling effect can be explained by the fact that CLIL streams have a stronger sense of belongingness to a group with a clear-cut identity and are thus more responsible, that they are helped less by their parents due to the language barrier and are thus more autonomous, and that they are provided with greater language-sensitive teaching which helps them all access the language more equally (Halbach and Iwaniec 2020).

Thus, the hotly debated assumption that CLIL is elitist needs to be dismantled on the basis of the most recent empirical evidence and a shift in focus need to be operated towards catering for diversity; that is, setting in place measures that will allow CLIL to be accessible for over- and under-achievers alike. Course-correction is required in this respect, particularly now that CLIL is increasingly being mainstreamed school-wide in contexts where there was traditionally self-selection (cf. article by Siepmann et al. in this issue).

This need dovetails with another to priority on the current CLIL agenda (Pérez Cañado 2020c): replicating studies in different contexts in order to base decisions regarding CLIL programs on real and pertinent needs. The one-size-fits-all model no longer fits the bill in CLIL scenarios (cf. article by Pérez Cañado in this issue) and greater contextualization and situatedness are thus necessary in order to attune CLIL to context-specific realities (Pérez Cañado 2020a). Finally, a further area which warrants research is that pertaining to stakeholder perspectives. The self-reported perceptions of key players in CLIL programs indeed come across as a particularly relevant niche to be filled in our field:

However, some areas of CLIL remain in need of further exploration. One such area is that of stakeholders’ perceptions of CLIL, the interest of which lies in the fact that their interpretations and beliefs are crucial to understand how the CLIL programme is socially viewed, understood and constructed, and the expectations it raises. (Barrios Espinosa 2019, 1)

Despite having mapped out these future pathways for progression, we currently lack instruments which conflate the three afore-mentioned burning issues on the CLIL agenda. We do not have carefully designed and validated questionnaires, interview, or observation protocols which concomitantly tap into catering for diversity in CLIL programs, are adapted to different scenarios for further iterations, and factor in stakeholder perspectives. This is precisely the niche which the present investigation seeks to address. It presents, for the first time, three originally designed questionnaires, interviews, and observation protocols to determine how attention to diversity in CLIL is being tackled at all curricular and organizational levels. They have been carefully validated for their application to teachers, students, and parents and made culturally and linguistically a responsive to the CLIL reality in six different countries spanning northern, southern, and central Europe for their replication in different contexts. They are the first instruments on the topic of diversity and inclusion in CLIL programs to be designed taking into account the latest research and following a double-fold validation process (external ratings approach and pilot phase with a representative sample). They are also distinctive in following a largely parallel structure and in considering a series of identification variables which allow for both within- and across-cohort comparisons. As they have been developed within four broader governmentally-funded research projects (cf. Funding), they have been successfully applied in six different European countries (cf. remaining articles in this issue) and thus allow for replicability in different contexts (location triangulation): Primary and Secondary Education, private and public schools, and rural and urban contexts. They thus enable a comprehensive program evaluation of diversity-sensitive CLIL implementation according to its frontline stakeholders.

The article begins by expounding on the format and typology of questions included in the surveys and then outlines the steps undertaken for their research-based design and fleshes out the double-fold pilot process followed for their validation. The questionnaires and interview and observation protocols are then presented in a format which can directly be applied in any CLIL classroom in order to make decisions which are fine-tuned and relevant to specific and diverse contexts of CLIL implementation.

Objective

The broad objective of this part of the ADiBE project (www.adibeproject.com) is to design and validate three sets of parallel questionnaires (for teachers, students, and parents) in order to gauge the accessibility of bilingual programs for all types of achievers and to identify the chief difficulties and best practices in catering to diversity in intercultural and content integrated language learning and teaching across Europe via data triangulation (language teachers, non-linguistic area teachers, teaching assistants, students, and parents). They will also be complemented with interview and observation protocols.

Instrument characterization

Three sets of questionnaires, interviews, and observation protocols have been created within the ADiBE project, one for each of the cohorts. The versions of the three questionnaires have been originally designed and validated in English, and then linguistically and culturally adapted to determine how diversity is being catered for across a broad array of CLIL contexts in European Secondary Education (Austria, Finland, Germany, Italy, Spain, and the United Kingdom; in urban and rural settings; in public and private schools; with teachers, students, and parents). This kind of instrument, which was self-administered and group-administered, is subsumed within survey tools by Brown (2001).

The questionnaires include 6–12 demographic or background questions to elicit information from teachers, students, and parents. These identification variables allow for within-cohort comparison. Then, the surveys comprise opinion or value questions to probe the perceptions of respondents vis-à-vis attention to diversity in CLIL. They are exemplified in the form of 53, 40, or 38 items within the teacher, the student, and the parent questionnaire, respectively. Thus, Patton’s (1987) question types are followed. The former type of questions are fill-in and short-answer ones (in line with Brown’s 2001 typology) and the latter, Likert-scale ones to express the level of agreement with the statements included. A six-option scale has been chosen (strongly disagree, disagree, partly disagree, partly agree, agree, and strongly agree) to avoid the central tendency error.

Thus, closed-response items predominate, for ease and speed of applicability. Specification is required only in the case of a positive answer to one of the items in the teacher and student surveys. Moreover, in the teacher collaboration and development block, the three cohorts are given the possibility of selecting the ‘does not apply’ option, and parents can tick ‘I don't know’ throughout the entire questionnaire. The three different questionnaires deal with practically the same information, although the teacher survey also includes a final section for respondents to rank the three most important aspects for them in a list of six difficulties, eight best practices and eight development needs from 1 (most important) to 3.

The contents of the questionnaires are grounded on recent research findings and are defined by the following blocks: linguistic aspects (9 items for the teacher questionnaire, 5 for students, and 4 for the parents); methodology and types of groupings (12 items for the teachers, students, and parents); materials and resources (7 items for the teacher questionnaire, 5 for students, and 3 for parents); assessment (10 items for teachers and 11 for both students and parents); and, finally, teacher collaboration and development (15 items in the teacher questionnaire, 7 in the student survey, and 8 in that corresponding to parents). The parent survey only consists of four blocks because the items relating to materials and resources were merged into the methodology and types of groupings section owing to the statistical reason that will be reported on below when describing the validation process.

On the basis of these surveys, the interview and observation protocols were designed for methodological triangulation, following a parallel structure for comparability across instruments and contexts. The only noteworthy differences between them lie in that the interviews allow teachers and students to elaborate also on their overall appraisal of catering for different abilities in the CLIL classroom, and that the observation protocols include a Likertscale of four points. Thanks to these three sets of qualitative and quantitative instruments, a complete picture of how attention to diversity in CLIL is working across Europe can be drawn.

Instrument design

In designing the questionnaires, Brown’s (2001) indications for drawing up survey items were followed in terms of form, meaning, level of respondents, order, and clarity (cf. Pérez Cañado 2016a for a detailed rendering of these aspects). The questions were grouped into related thematic blocks, accompanied by instructions, and edited following a process which will be explained in the subsequent validation section.

The expertise of the researchers from the participating universities was crucial to determine the contents of the instruments and the latest literature on catering to diversity in CLIL was canvassed in order to establish the items to be included in the questionnaires and interview and observation protocols. To begin with, linguistic aspects are of prime importance in catering for diversity and thus constitute the initial block of the instruments designed. Coping with diverse ability levels, both linguistically and content-wise, surfaces as a major hurdle for CLIL teachers in the specialized literature (cf. article by Siepmann et al. in this issue) and it is thus an initial issue into which this section taps. It also probes scaffolding techniques to step up to this challenge, together with the role of the L1 in providing strategic support for pedagogical translanguaging (Pavón Vázquez and Ramos Ordóñez 2019). Finally, the teacher's specialized knowledge of BICS and CALP is also harnessed, as it has comes across as a pervading deficiency in CLIL teacher development (Pérez Cañado 2016b).

In addition to language-related aspects, methodology becomes a cornerstone for successful attention to diversity in CLIL programs. CLIL has been considered ‘a bid to bring innovation into the classroom, realigning teaching with modern pedagogical practices’ (Pérez Cañado 2016b, 7). It is held to entail a thorough overhaul of the way in which language and content teaching are envisaged in moving from a transmissive model of education to meaningful, communicative teaching, based on constructivism and learning by doing. Student-centered methodologies and variegated types of groupings acquire a particularly sharp relief in promoting inclusive bilingual education. Indeed, this was mentioned as the key for the sustainability of CLIL programs by the teachers polled in prior studies (Pérez Cañado 2018). However, a rift has been documented between theoretical potential and practical implementation of pedagogically-related aspects in CLIL (Cabezas Cabello 2010; Breidbach and Viebrock 2012) and this second heading thus probes whether student-centered options (such as project-based learning, task-based language teaching, cooperative learning, or multiple intelligence theory) are actually incorporated at the grassroots level to cater for diversity. The teacher-frontedness or student-led nature of the CLIL classroom is also gauged, as is the use of different types of groupings and layouts, peer-assisted strategies, or newcomer classes.

However, if there is an issue which comes across as particularly daunting in catering for diversity in CLIL classrooms that is materials and resources. This is why a third block of contents has been included in the instruments designed in order diagnose chief needs in this area. Indeed, numerous studies (Pena Díaz and Porto Requejo 2008; Fernández and Halbach 2011; Roiha 2014; Pérez Cañado 2016b, 2016c, 2018) have signposted the scarcity of materials as one of the major roadblocks to successful differentiation in CLIL. Teachers feel deterred by the challenge inherent in having to find, create, design, or adapt diversity-sensitive teaching materials and this consequently becomes a major niche to be filled. The stakeholders’ experiences with access to and creation/adaption of materials are thus explored in this next section, together with the role of ICTs within them and the multimodal nature of materials.

Assessment is no less taxing. Indeed, a systemic-functional approach to evaluation (Otto 2018) is now favored in CLIL, in line with a pluriliteracies approach (Meyer et al. 2015) which develops cognitive discourse functions (Dalton-Puffer 2013). It favors the conflation of BICS and CALP and challenges the division of both concepts, strengthening the relationship between the conceptual and communicative continua and making the transition towards a more integrated type of assessment (San Isidro 2019), where language-related issues are measured in relation to content objectives. This novel approach to evaluation considers not only if a language form is grammatically correct, but whether it is used appropriately to convey meaning in functional contexts, taking into account communicative intent in terms of language functions rather than linguistic accuracy or grammatical correction. Thus, assessment now needs to consider not only the ability to use linguistic forms correctly, but to use the appropriate form to express meaning in a particular academic context. On these grounds, a fourth block is directed to gathering opinions on the main strategies deployed in the CLIL classroom to attune formative, summative, self-, and peer-assessment to varying ability levels.

Finally, for a success-prone implementation of CLIL with all types of learning paces and ability levels, a multi-tiered system of support needs to be set in place (Scanlan 2011; Grieve and Haining 2011), where collaboration among teachers and support from authorities, parents, and multi-professional teams need to be stepped up. This is precisely what the final block of contents examines, together with the chief areas for teacher development which need to be sharpened and refined in making a diversity push in the CLIL classroom.

Thus, the questionnaires have been designed taking into account recent research outcomes in order to paint a comprehensive picture of how catering for diversity in CLIL programs is playing out on all curricular and organizational fronts. The semi-structured interviews incorporate clear-cut questions parallel in structure to the major blocks of the questionnaires, but always with a view to allowing further elaboration on each of the areas of concern included in the surveys. As for the observation protocols, they involve the more objective items in the questionnaires. Thus, the structure of the three sets of instruments is parallel in order to allow the more qualitative insights of the interviews to enrich the quantitative dataset provided by the surveys and to determine whether the latter is actually present at the grassroots level via the observation protocols. This mixed research design thus follows the parameters which Hellekjaer and Wilkinson (2001, 405) establish for measuring the quality of CLIL programs since there is a comparison of CLIL schemes across institutions, the quality of CLIL initiatives is assessed via teacher and student perceptions, and staff development options are gauged.

Instrument validation

Following this careful process of instrument design, a double-fold pilot procedure was then adopted for validation. It firstly entailed the expert ratings approach and, subsequently, a pilot phase with a representative sample of respondents. Regarding phase one, the versions of the questionnaires that had been drafted by the ADiBE team were subjected to the critical scrutiny of three CLIL teachers and two CLIL researchers/teacher educators per participating country. A total of 30 external experts across the six countries were asked to rate each item on a six-point Likert-type scale in terms of its clarity, precision, and relevance. Items with a mean lower than 3 were eliminated and those with means between 3 and 4 or a standard deviation higher than 0.95 were revised (quantitative analysis). Open questions on item deletion, addition, or modification were also included for the reviewers to make further detailed comments (qualitative analysis).

Then, in the second phase, the questionnaires were piloted with a representative sample of 132 students, 55 teachers, and 47 parents in Spain with exactly the same traits as the target respondents who would subsequently be surveyed with the final questionnaire. The teacher questionnaire was set up online through Survey Monkey, whereas the parent and student questionnaires were translated into Spanish and administered face-to-face in schools different from those who were going to participate in the final study. Their responses enabled the calculation of Cronbach alpha for each of its thematic blocks and for the survey as a whole in order to guarantee its reliability or internal consistency, which was ascertained by means of the high coefficients obtained for the three questionnaires: 0.853 for the teacher one, 0.871 for the student equivalent, and 0.945 for the parent survey.

Decisions on deleting, modifying, and adding items were made based on the qualitative and quantitative data stemming for this double-fold pilot process. General ones involved the deletion of items in relation to ‘taking into account student diversity when organizing pair work and group work in the CLIL classroom’, as experts considered them to be encompassed within the previous item enquiring about ‘the use of different types of groupings / mixed-ability groups in the CLIL classroom’. Other items that were deleted in the three tools had to do with assessment: ‘It is difficult to assess diverse students fairly with the established assessment criteria’ and ‘in my evaluation/assessment, I take into account different levels of achievement among students’. The referees objected to them because of the unclear meaning of ‘fairly’, presupposing the existence of ‘established assessment criteria’, and the length of the block, in the case of the former item, and since they were considered unnecessary, as other items refer to continuous and final assessment separately, in the case of the latter. Apart from this, the resulting Cronbach alpha of the block and the general Cronbach alpha recommended their deletion.

Modifications affecting the three surveys also entailed the rewording of expressions such as ‘who are diverse in their academic achievement’ to ‘who have different levels of ability’, ‘different types of learners’ to ‘learners with different abilities’, ‘cater to diversity’ to ‘cater for diversity’, ‘appeal to multiple intelligences’ to ‘plan for multiple intelligences’, and ‘evaluation and assessment’ to just ‘assessment’. Adaptations for homogeneity were also advised, such as changing ‘cover parts’ to ‘repeat parts’ and including the term ‘CLIL’ in some items where it was missing. Vis-à-vis the addition of items, a statement on self-evaluation was incorporated following the suggestion of a few experts, mainly from Spain.

Specifically for the teacher questionnaire, another four items were deleted following the calculation of the Cronbach alpha (2) or the experts’ comments (2). The qualitative analysis of the experts’ evaluation also implied some changes, for example ‘nationality’ to ‘mother tongue’ in identification variable 5, a few rewordings, and the elimination of two ‘difficulties’ in the section of the instrument on ranking aspects, being six and not eight the final number.

Relating to the student questionnaire, for the sake of the reliability coefficient, items 1 and 2 were eliminated (‘I find it challenging to learn in classes with students who have different levels of the foreign language/of academic achievement’), together with 40 (‘My CLIL teachers coordinate/collaborate with each other in order to cater to diversity’). In line with the experts’ opinions, ‘nationality’ was replaced with ‘language(s) spoken at home’ in identification variable 5 (which was also altered in the parent survey), percentages were provided in identification variable 7, the name of a subject was given as an example in item 5, and the full form of ICT preceded the acronym in item 21.

The most important alterations to the parent survey consisted exclusively in the deletion of three items. One of them stated that there was not enough time to attend to different types of learners in their child's CLIL lessons, and it was deleted following the experts’ qualitative suggestions. The other two dealt with teachers’ use of ICT to cater for diversity and their child's use of multimodal instructional materials in the CLIL classroom, and they were suppressed due to the low reliability of the block on materials and resources. Consequently, that section only contained three items. Thus, given its small size, it was merged with the previous one on methodology and types of groupings, which in turn led to an increased the reliability of the resulting block (cf. Table 1). This was the questionnaire that was reduced most considerably (in nine items, from 47 to 38), whereas the teacher and student surveys lost seven (from 60 to 53 and from 47 to 40, respectively).

Table 1. Parent questionnaire pilot study.

Items	Cronbach alpha of the block before their deletion	Cronbach alpha of the block after their deletion	Cronbach alpha of the old block 2 before merging it with the old block 3	Cronbach alpha of the new (merged) block 2
Old items 26 and 27	0.660	0.710	0.882	0.893

To finish with the validation analysis, the homogeneity of the items that were chosen was studied through the homogeneity index (IH), from which it can be concluded that the selected items had an index greater than 0.3, so they were adequate. The quantitative and qualitative analyses conducted through this double-fold pilot procedure allowed us to draw up a final version of the questionnaires for teachers (foreign language teachers, content teachers, and language assistants), students, and parents. They were subsequently translated into Spanish, German (with an Austrian and German version), Italian, and Finnish. The surveys were also slightly adapted to the context of the UK. They were then typeset for printing and also set up online on Survey Monkey. The refining of these instruments in the ways explained above also affected the final versions of the interview and observation protocols, which were modified accordingly.

The validity and reliability of the questionnaires can thus now be claimed. Their content validity is grounded on the expertise of the researchers who designed them and on findings of previous studies, as mentioned above. They also closely observe the four main requirements mentioned by authors like Denzin (1994) and Brown (2001) for qualitative research: credibility, transferability, dependability, and confirmability. Therefore, these instruments can be directly applied in any CLIL classroom in order to gauge the accessibility of bilingual programs for all types of achievers and to identify the chief difficulties and best practices in promoting inclusion in bilingual education. The actual six research tools in English can be found in appendices 1–6.

Conclusion

This article has fleshed out the design and validation process of three separate questionnaires, each in line with the specific characteristics of teacher, student, and parent cohorts, to identify their corresponding perspectives on attention to diversity in CLIL programs. On the basis of these surveys, interview and observation protocols have been designed for methodological triangulation. The content of the surveys encompasses five main aspects, based on recent research outcomes: linguistic aspects; methodology and types of groupings; materials and resources (merged with the previous one in the parent survey); assessment; and, finally, teacher collaboration and development. The teacher questionnaire also includes a final section for respondents to rank the three most important difficulties, best practices and development needs within a list provided.

A two-fold validation process has been followed, beginning with the experts ratings approach, whereby the instruments have been subjected to the critical scrutiny of 30 external reviewers across Europe with expertise in the topic at hand. Their on-point suggestions have allowed us to introduce substantial modifications pertaining to elimination, addition, or rewording of items, homogeneity adaptations, and closed-response percentages, examples or the full form of a term. Subsequently, a pilot study has been conducted with a representative sample of 234 students, teachers, and parents with the same traits as the target respondents to further sharpen and refine the instruments and to calculate Cronbach alpha. In this sense, the extremely high coefficients obtained for the questionnaires have allowed us to confirm their internal consistency and reliability.

The application of these instruments across diverse contexts will hopefully enable us to paint a comprehensive and empirically valid picture of where fully bilingual schemes stand in monolingual contexts across Europe, drawing a precise description of the way in which CLIL is working with different types of achievers. The other papers in this special issue already show findings derived from the administration of these research tools in various northern, central, and southern European countries. The ultimate aim is to contribute to the integration of all students, regardless of their socioeconomic status, educational background, or achievement level and to contribute to making CLIL accessible to all. The transnational nature of the enterprise is absolutely pivotal, as learning form the best practices of others is key to continue advancing on this front.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the European Union under Grant 2018-1-ES01-KA201-050356; the Spanish Ministerio de Ciencia, Innovación y Universidades under Grant RTI2018-093390-B-I00; and the Junta de Andalucía under Grants 1263559 and P18-RT-1513.

Notes on contributors

María Luisa Pérez Cañado

Dr María Luisa Pérez Cañado is Full Professor at the Department of English Philology of the University of Jaén, Spain, where she is also Rector's Delegate for European Universities and Language Policy. Her research interests are in Applied Linguistics, bilingual education, and new technologies in language teaching.

Diego Rascón Moreno

Dr Diego Rascón Moreno is Associate Professor at the Department of English Philology of the University of Jaén, Spain. His research interests are in Applied Linguistics, content and language integrated learning (CLIL), values education, and new trends in language teaching.

Valentina Cueva López

Valentina Cueva López is Interim Substitute Professor at the Department of Statistics and Operative Research at the University of Jaén. She holds a BA in Statistical Sciences and Techniques and an MA in Applied Statistics, both from the University of Granada, and is currently a PhD candidate at the University of Jaén.