English medium instruction learners’ self-efficacy, engagement, and satisfaction; developing a measurement instrument

ABSTRACT

This paper describes the development of a survey instrument to measure learners’ self-efficacy, engagement, and satisfaction within the context of English Medium Instruction (EMI) classes, with reference to both language issues and the discipline being taught. The paper presents details of analysis of the instrument using the Rasch model. The survey was administered to 287 tertiary students enrolled in EMI classes in Japan. Results of a Rasch Principal Components Analysis of Residuals showed the three intended constructs to be largely unidimensional. Item-level analysis, as well as reliability and separation measures and an investigation of Wright maps, provided further evidence that the items are working adequately to measure each of the three constructs and to allow differentiation between levels of each construct among the respondents. In addition, the effectiveness of the 6-point Likert scale used was confirmed with an analysis of category structure functioning. Finally, a correlation analysis with a subset (n = 97) of the participants for which English proficiency test scores were available was conducted. Results showed a small, statistically significant and positive relationship between reading proficiency and each of the three constructs.

KEYWORDS:

• English Medium Instruction
• measurement
• Rasch analysis
• self-efficacy
• engagement

Background

The drive to internationalize tertiary institutions around the world has led to a situation where many previously non-Anglosphere universities now provide instruction for different subjects in English, and this is usually referred to as English Medium Instruction (EMI). The students in EMI classes are usually (although not exclusively) second language (L2) English speakers, and often the teachers are too. This approach can be contrasted with “education in the disciplines” (Macaro & Aizawa, 2022), that is classes in universities in primarily English-speaking countries which focus on a given discipline. A class in international relations taught in English at an Australian university is part of “education in the disciplines”, while the same course taught at a university in Japan or Turkey is most probably EMI.

While learners in EMI classes may not speak English as an L1, the sole and original goal of EMI is the teaching of a particular disciplinary area (Airey, 2016), rather than a chance for learners to improve their English. Yet along the way, EMI has become inextricably linked with the teaching of English as a foreign/second language (EFL/ESL), and in some ways has even been co-opted by the language teaching profession, so that EMI is now viewed by many as a language learning approach. It is not uncommon to see discussion of EMI in relation to the potential for language proficiency gains, reflective in statements such as “English-medium education refers to curricula using English as a medium of instruction for basic and advanced courses to improve students’ academic English proficiency” (Taguchi, 2014, p. 89). In a meta-analysis of 154 journal articles on the topic of EMI, Macaro and Aizawa (2022) found that 91.6% had been written by applied linguistics specialists, compared to only 1.9% by discipline specialists, (with 6.5% a collaboration by both). Further, they note that 80% of these papers were published in the applied linguistics literature (their meta-analysis itself being published in such a journal) rather than higher education, or subject-related journals.

It is not surprising therefore that the focus of many studies concerning EMI has often been on language learning outcomes, or the L2 abilities (and inabilities) of students to successfully navigate a course completely in English, and in which can be sensed the strong belief among many EMI proponents that in these courses only English should be used (Sahan et al., 2022). Indeed, in many ways it seems that the original goal of EMI – to teach knowledge of a discipline – is sometimes forgotten.

The view of EMI as an approach to language learning has probably helped (or at least not hindered) its adoption in many contexts where the potential success of pure EMI is questionable at best. For example, Japan is a context in which the number of EMI courses has been rapidly increasing in recent years (Harris & Strefford, 2022). “Internationalization” has been a key driver, but the very strong EFL industry has arguably helped to promote it. It may be that EMI gives language specialists a chance to move away from pure language teaching (Yuan, 2021), a role that some may view as limiting or perhaps somehow “less valuable”.

From what has been outlined above, it can be strongly argued that EMI is presently “owned” by the applied linguists, and in discussing this idea, Macaro and Aizawa (2022) call for greater collaboration between language specialists and “teachers of the disciplines” in order to redress this imbalance. This does not of course mean ignoring language completely, and in contexts like Japan, language factors are indeed important and must be taken into account. EMI was originally developed in comparatively high English proficiency contexts, and there are special considerations that need to be made for lower-proficiency contexts like Japan. Therefore, it is important for researchers to take into consideration both language and discipline-related factors, and this became the key rationale behind the present study, creating a measurement instrument which takes both of these areas into consideration.

This paper outlines the development of a survey instrument to measure Japanese EMI learners’ satisfaction, engagement, and self-efficacy, all important areas which heavily affect success (or otherwise) in EMI classes in contexts such as Japan. For each of the three factors we inquire after both subject and language concerns. Furthermore, Rasch analysis was used to confirm validity and reliability of this new instrument, addressing a problem in previous EMI research of inadequately tested measurement instruments (Curle & Derakhshan, 2021). It also provides the entire instrument, which we hope may be used or further adapted by other researchers in the field, addressing a major issue in social sciences that such instruments are not always made freely available (Rammstedt & Blumke, 2019). Finally, the instrument was tested in a correlation study with English listening and reading proficiency scores from a subset of the participants. This avoided the “questionnaire curse”, a common situation in which all variables in a study, even dependent variables, derive from self-reported measures (Al-Hoorie et al., 2021).

Literature review

In previous EMI research focusing on the learner, outcomes fall overwhelmingly in the area of L2 ability (e.g., Rose et al., 2020), features of language use in the classroom (e.g., Chou, 2018), or language improvement (e.g., Coxhead & Boutorwick, 2018). However, as Thompson et al. (2019) point out, other factors besides language ability are important and have generally been neglected in EMI research, and this is despite the high potential for such factors to be instrumental in achieving successful learning outcomes.

Nonetheless, previous studies which have attempted to measure various aspects of learner experience, in addition to our combined professional experience of over 40 years of teaching, helped us to identify three key areas of interest: self-efficacy, engagement, and satisfaction. Below, we outline these studies, and provide a rationale for the development of an instrument that can accurately, and comprehensively, measure them. First however, we provide an outline of these three key concepts.

Self-efficacy (SE) is part of Albert Bandura’s Social Cognitive Theory (Bandura, 1997) and concerns the beliefs that a person holds regarding their ability to achieve a certain goal. While it has been argued that in some cases SE can be general, it is more commonly thought to be domain specific (high levels of SE concerning one’s golf swing will not necessarily carry over to tennis swing). It has been shown to be an important self-belief across various domains from sport to health to business (Schunk & DiBenedetto, 2016). It has also been shown to be a strong predictor of actual performance, perhaps more than any other self-belief (Graham & Weiner, 1996). For teachers, it should be considered an extremely important variable because people with high levels of SE tend to show greater persistence in the face of difficult tasks, are more intrinsically interested in tasks, will give more effort, and are likely to be less anxious (Prat-Sala & Redford, 2010). All of these are qualities that teachers seek to nurture in their learners and, because teachers can strongly influence the development of SE, it is an area where they can have a large and positive impact.

The term “engagement” describes a learner state that can change with time and context, (thus it is not treated as a trait), referring specifically to action (Lawson & Lawson, 2013). This distinguishes it from motivation, which it is held, is an antecedent of engagement (Reschly & Christenson, 2022). Therefore, unlike SE, engagement is less a psychological variable, but context- (and task-) specific. While it is not always clear exactly what engagement entails, it is generally thought to consist of three main dimensions: affective (for example signs of enjoyment or interest in a given class), cognitive (such as verbal and non-verbal markers of interaction and how learners connect new knowledge and relate it to their existing knowledge), and behavioural (aspects such as time spent on a given task or total amount of output). In this study, we tried to include elements of each of these in the items in the scale. However, due to our need to create a concise measurement instrument that includes reference to SE, engagement, and satisfaction concerning both the discipline and language in the same instrument, we treated engagement in a more general sense. Overall, engagement is an important variable in education, because, like SE, positive engagement has been shown to be a major contributor in learning success (Hattie, 2009; Skinner et al., 2008), and like SE, the teacher can have a great deal of influence on learner engagement. Importantly, measurement of engagement should be tailored to the specific domain because engagement in a learner can change drastically depending on context. As far as we are aware, there is not yet a domain specific measurement instrument for engagement in EMI, and especially not for contexts in which English proficiency is limited, and one that includes both content and language in its scope.

Learner satisfaction is a factor that has been the subject of extensive research in education (Weerasinghe & Fernando, 2017), aiming to capture levels of fulfilment or contentment. Learner satisfaction is tied to perseverance and academic achievement. Not only does satisfaction have direct implications for learner SE and engagement, but identifying lack of satisfaction can help teachers and institutions to improve. For EMI, we argue that it is an important variable to be measured. If, for example, expectations are met, then this can create something akin to a positive feedback loop in which SE and engagement rise, leading to positive learning outcomes.

SE, engagement, and satisfaction are three important variables in any area of education. As previously stated, these variables have not been extensively researched within the context of EMI. However, this does not mean there has been a total neglect, and so it is to an overview of studies which have investigated these variables within the context of EMI classes that we now turn.

In response to the early 21^st century expansion of EMI around the world, some studies have begun to appear which investigate learner perceptions and their importance to successful learning outcomes in EMI classes. Thompson et al. (2019) found that along with L2 proficiency, SE was a predictor of “success” (for which they used mid-term and end-of-term test scores as an outcome variable) of 139 second-year students in a business management EMI course in Japan, finding links between positive self-beliefs and perseverance. They employed a self-beliefs survey and supported it with interviews with seven of this main cohort. From the latter they found that language support was very important for these learners, both before and during the course. Through a multiple linear regression, the authors found that SE was indeed a predictor of success and this was backed up with interview data. One drawback of the paper is that the full survey is not included, but rather only example items such as “I usually get good marks in English” (Thompson et al., 2019, p. 209). In addition, the items that are included suggest a focus on language outcomes rather than including reference to the discipline in question.

Zhang and Pladevall-Ballester (2021) investigated the attitudes and perceptions of learners in three different kinds of EMI class: international trade (n = 96), film production (n = 45), and project management (n = 29) over one semester (September to December) in China. Pre- and post-course surveys were augmented with classroom observations held three times over the course. Overall, they found that learners became “less positive” towards the end of courses. They suggest that this may be related to language proficiency issues and conclude that translations might be used for scaffolding purposes when needed, or as extra materials given as out-of-class work. Once again, the full survey is not included so it is difficult to know exactly what was being asked of participants, but the authors refer to “perceptions, expectations, and attitudes” (p. 204). Also, the examples provided suggest a focus on language, such as “Can’t understand the lecture due to poor listening skills and limited vocabulary range” (Zhang & Pladevall-Ballester, 2021, p. 209).

Closely related to the present study, Le and Nguyen (2022) investigated the interaction between learner motivation, engagement, and satisfaction with questionnaire data from 437 tertiary students in Vietnam. Satisfaction was measured based on a survey instrument developed by Tseng et al. (2020), and this became the dependent variable with data from an engagement and motivation scale as the independent and mediating variables (however, once again the items for these latter constructs are not provided in the published paper). They found that while these students were generally positive in assessing their satisfaction with the course, cognitive and emotional engagement has a mediating effect between motivation and satisfaction.

In another study investigating learner satisfaction, Kym and Kym (2014) measured perceptions of both language and discipline knowledge of 364 students in 11 separate business administration EMI courses at a university in Korea. They developed an instrument which among other items aiming to measure self-perceived comprehension and “general perception” (p. 42), included five items measuring satisfaction. The authors performed a correlation study and an ANOVA with the resulting data and self-reported test scores of about half of the participants. They found that satisfaction did not correlate with proficiency, but it is important to note that these were self-reported scores of varying language proficiency tests.

As the review of previous research in this area above highlights, while researchers have begun to investigate these variables (SE, engagement, and satisfaction) highlighting their salience for EMI courses, there are, as far as we can see, two main gaps that need to be addressed. First, previous inquiry into these areas appears to focus heavily on the linguistic side. For example, a survey investigating SE may inquire after students’ perceived confidence in the language required to complete a task rather than SE related to knowledge of the discipline. Second, and this is an issue that is not only related to the EMI literature, but common to many studies in the applied linguistics field in which they are situated, is the lack of either assessing, or at least publishing details of validity and reliability of the scales employed (Al-Hoorie & Vitta, 2019), or indeed even publishing the survey itself (Rammstedt & Blumke, 2019).

The intention of this paper is to address these two issues by describing the design and psychometric testing of a concise (24-item) instrument that can accurately measure the SE, engagement, and satisfaction of EMI learners. While it is very difficult to parse out student’s attitudes towards language and content, we have tried to redress the balance by inquiring after both. The process of the development of the instrument and initial results are described below. Following this, we outline a correlation study of the three variables with language proficiency scores. First however, we provide the two research questions that framed this study.

Research questions

There are two main research questions that we seek to address in this study.

1. How can three key factors for student success in EMI (self-efficacy, engagement, and satisfaction) be measured accurately?

2. How do these three factors interact with language proficiency?

Methodology

Participants

The participants were first-, second-, and third-year university students in an International Relations faculty at a large private university in Japan. We chose students from across different grade levels in order to test the instrument with a variety of learners who were in differing, though connected, EMI class contexts. The participants were not language majors, although, in accordance with the ongoing internationalization of higher education in Japan, all students took a number of English language classes, and in-English classes (“EMI-like”) in their first and second year of study, and also a compulsory EMI class on international relations in their third year. Therefore, all of the students took the survey in, and in relation to EMI, or “EMI-like” classes. Participation was voluntary. Students were provided with a link to an online survey and were informed that they could opt out if they preferred. The data collection took place over one week towards the end of the academic year. In total, 287 students completed the survey. From this main cohort, 106 students had also taken an institutional version of the Test of English for International Communication Listening and Reading (TOEIC L&R, Educational Testing Service, 2008). After removing nine outliers from this group, the test scores for the remaining 97 students were used in bivariate correlations with the three survey variables.

Instruments

We created a battery of 24 questions, eight for each of the three intended constructs (SE, engagement, and satisfaction). All items were original, and were created based on the authors’ experiences with EMI classes. This experiential knowledge enabled us to pinpoint factors that may significantly impact learners, and these factors are interrelated.

A key issue is that what students in EMI classes are expected to do is very difficult for many of them. The gap between how they learnt in high-school, and what they are expected to do in EMI classes is large. Specifically, many students have limited language skills, and have limited experience participating in, and contributing to, a classroom environment in which they are expected to produce something. This naturally results in a lack of SE. As SE is a predictor of motivation (McGeown et al., 2014), and because motivation is essential for academic success (Schneider & Preckel, 2017), it is an extremely important factor to investigate. Regarding engagement, if a student is engaged in a task, then it is logical to conclude that motivation is higher, resulting in positive learning outcomes. SE is also positively linked to satisfaction (Kryshko et al., 2022) and most EMI classes are one component of a wider curriculum, and therefore satisfaction in one class is assumed to have a high potential to have a spill-over effect on other classes.

For each potential construct (SE, engagement, and satisfaction), items related to both language and content were included. The items were created first in English, and then translated into Japanese by the authors. This was then checked by a Japanese native speaker familiar with the project and small adaptions were made. In line with Krosnick and Fabrigar’s (1997) recommendation for keeping Likert scales between five and seven points, we opted for a six-point scale in order to avoid a situation where learners might rely on the “neutral” middle category. The final set of questions were uploaded to a Google Form which was then made available to the participants via QR code.

Analysis and results

Rasch analyses

In order to investigate the performance of the survey, Rasch analysis (Rasch, 1960) was conducted. Rasch is a mathematical model that is useful for survey design because it allows researchers to ascertain the unidimensionality of constructs in a potential scale. It provides information about the connections between “person” abilities (or in this case “level of a given attitude” in survey participants) and “item” difficulty (in this case “difficulty to agree or disagree with an item”). It also provides measures of reliability for both persons and items. An advantage of using Rasch analysis to refine survey instruments lies in the way it can help to identify differences in Likert scale categories. A fundamental problem with Likert scales is that resulting data gained from them are ordinal (Embertson & Reise, 2000), and as such, an assumption is that the gaps between the numbers are identical across the entire scale (for example, it is assumed that the gap between “1” and “2” on the scale is identical in strength to the gap between “2” and “3”), but this may not necessarily be the case. Rasch provides logit scores which can then be used in other statistical procedures.

To investigate the unidimensionality of each of the three constructs in the instrument, a Principal Components Analysis of Residuals (PCAR) was conducted with Winsteps software (Linacre, 2020b). In order to confirm that each construct is unidimensional, it is commonly held that variance explained by measures should be over 50% and eigenvalues in the first contrast should sit below 2.0 (Linacre, 2020a; Wright, 1996). Figures outside of this are potential indicators of other dimensions in the scale. As can be seen in Table 1, each of the three constructs in the instrument meets these requirements, with the exception of the engagement measure, which has a slightly higher eigenvalue in the first contrast. Linacre (2020a) however explains that these values are no more than indicators, and that in the same way that figures falling inside these recommendations do not instantly mean a construct is definitely unidimensional, figures outside of these parameters do not necessarily mean a construct contains other dimensions. Therefore, in order to gain a better understanding of the instrument’s performance, individual items were also investigated.

Table 1. PCAR results for the three scales.

Component	Variance explained as percentage	Variance explained as eigenvalue	First contrast explained variance	First contrast explained variance as eigenvalue
Self-efficacy	57.8%	10.95	9.9%	1.87
Engagement	60.9%	12.47	11.7%	2.39
Satisfaction	54.7%	9.68	9.1%	1.60

Rasch provides individual item information as both mean-square (MNSQ) and standardized (ZSTD) scores, but the latter are sometimes sensitive to large sample sizes (Bond et al., 2021), so for the present study, MNSQ scores are provided. For each item, infit and outfit values are produced. A score of 1.0 reflects “fit” to the expected model, and it has been suggested that values between 0.6 and 1.4 can be considered acceptable (Wolfe & Smith, 2007). Items outside of this recommendation suggest potential item “disturbance” (Bond et al., 2021; Wright & Linacre, 1994). A high MNSQ suggests that a response is not as predictable as would be expected by the model. On the other hand, low MNSQ values suggest the opposite – that an item response is very predictable to the point of overfitting the model, which means an item may not be efficient in yielding any new information. Outfit values provide information about outliers which are possibly the result of small issues like careless mistakes. High infit values are considered more problematic, indicating for example, issues with item wording (Apple & Neff, 2012). Once again, these are just guidelines, but upon reviewing flagged items, it may be the case that rewording or removal is deemed appropriate. As the data in Table 2 indicates, the vast majority of items in each construct show acceptable fit to the model. Only two items had higher than recommended infit values. In the SE construct, item 20 “I was able to complete all homework assignments for this class adequately” had an infit MNSQ of 1.65. A possible reason for this may be that it is the only item related to out-of-class work. While a student may be consistent in thinking about their ability to complete various kinds of tasks in class, homework may be thought of differently. It is also possible that levels of homework given might have varied depending on the teacher, making responses to this item more inconsistent. However, as the results of the PCAR showed the SE construct to be unidimensional, this item was retained. The other item displaying a higher than recommended infit MNSQ value (1.77) was item 5 from the satisfaction construct, “I need to have a high level of English competency for my future career”. A possible reason for this misfit might be that this is the only item in this construct referring to future career. As many of the participants were only first- or second-year students, for them, their careers may seem far into the future, and connections between what they are learning now, and what they will need for their career, may be hard to imagine. This has been an issue with other scales using career-related items with tertiary students, such as motivation for learning English (Leeming & Harris, 2024). Once again though, the PCAR showed the satisfaction construct to be unidimensional and therefore, it was decided to retain it.

Table 2. Item statistics for the three scales.

Item	Measure	Model S.E.	Infit MNSQ	Outfit MNSQ	Pt-Measure correlation
Self-efficacy
24	.44	.08	1.13	1.13	.54
21	.43	.08	1.10	1.10	.61
18	.16	.08	.91	.91	.82
17	−.01	.08	.97	.99	.80
19	−.08	.08	1.04	1.03	.85
22	−.15	.08	.59	.57	.80
23	−.26	.08	.57	.57	.81
20	−.54	.08	1.65	1.63	.76
Engagement
12	1.12	.07	1.07	1.05	.78
15	.88	.07	1.04	1.04	.77
11	.00	.07	.78	.79	.77
9	−.10	.07	.99	.98	.74
14	−.11	.07	.69	.66	.80
16	−.30	.07	.97	.91	.75
10	−.74	.08	1.09	1.12	.70
13	−.75	.08	1.37	1.44	.64
Satisfaction
8	.69	.07	.89	.87	.77
2	.43	.08	.66	.66	.79
4	.39	.08	.78	.80	.77
1	.32	.08	.83	.82	.75
3	.22	.08	.61	.62	.79
7	.12	.08	1.39	1.46	.62
6	−.90	.09	1.38	1.20	.65
5	−1.28	.09	1.77	1.79	.53

Bold type signifies items that are overfitting (<.6) or underfitting (>.1.6) the Rasch model.

Rasch also allows for investigation of reliability and separation for both persons and items. Person and item reliability figures are similar to Cronbach’s alpha in that those over 0.8 can be considered reliable (Bond et al., 2021). As Table 3 shows, all three constructs meet this benchmark. Person separation shows the level to which the scale is separating out high and low performers (in this case, different levels of the various constructs among respondents). A minimum of 2.0 means that it is adequately separating the participants into two distinct groups (Linacre, 2020a). Again, all three constructs meet this requirement. The item reliability figures for all three constructs support the replicability of the instrument. If the same set of items were administered to a separate group of participants, it is highly likely that the same item hierarchy would result. Item separation is a way of ascertaining item hierarchy, and figures above 3.0 in conjunction with item reliability above 0.9 strongly supports both the construct validity and replicability of the scales (Linacre, 2020a).

Table 3. Person and item separation and reliability for the three constructs.

Scale	Person Separation	Person Reliability	Item Separation	Item Reliability
Self-efficacy	2.71	.88	3.68	.93
Engagement Satisfaction	2.61 2.25	.87 .84	8.35 7.35	.99 .98

Another useful tool that Winsteps affords is the Wright map which provides a visual representation of the level to which items are endorsed by participants, and can help to highlight redundant items (Knoch & McNamara, 2015). Persons and items are placed on logit scales on the map. Persons are represented by the symbols on the left of the map (each # represents two people and each dot is one) and the higher a respondent, the more of a particular construct they exhibit in their answers (i.e., are more “self-efficacious”, “engaged”, or “satisfied”).

Items are displayed on the right within each of the six Likert categories. The positioning of the items on the map shows endorsement difficulty. The higher items are those that are more difficult for respondents to endorse. For example, in the engagement construct, item 12, “I sometimes forgot time in class because I was so involved in the topic” was the most difficult for respondents to agree with. Indeed, this item was created to try to tease out those highly engaged students and to provide a harder item for them to agree with. While certain tasks may be interesting for students, it would perhaps take extreme levels of engagement for them to “forget time”.

The mean values are represented by the letter “M”, and ideally the mean for persons and items should be parallel. The Wright maps (Figures 1–3) show that the person means for all three constructs are higher than those for items, and this, along with a ceiling effect evident from the collection of # symbols on the top left, which was particularly prominent for satisfaction (Figure 3) provides a visual depiction of how the respondents were generally positive in their responses.

Figure 1. Wright map for self-efficacy.

Figure 1. (Continued).

Figure 2. Wright map for engagement.

Figure 2. (Continued)

Figure 3. Wright map for satisfaction.

The items for SE (Figure 1) show a good level of spread with very little redundancy. For engagement (Figure 2) and satisfaction (Figure 3) however, there is some overlap suggesting a level of redundancy in certain items. Nonetheless, the spread of items as shown by these maps suggests that the questionnaire as a whole is differentiating between learners with higher and lower levels of each construct, as supported by the results from the other analyses outlined in this paper. With all of this taken into consideration, it was decided to retain all eight items for each construct.

Finally, an analysis of category structure functioning was conducted in order to test the effectiveness of the six-point Likert scale. According to criteria suggested by Wolfe and Smith (2007), each category should have at least 10 responses, the average measure for each category should be higher than those before it, outfit MNSQ figures should ideally be below 2.0, and the gap between difficulty levels should be between .59 and 5 logits. As can be seen in Table 4, with the exception of a slightly larger than 2.0 (2.12) outfit MNSQ for the “strongly disagree” category in the satisfaction construct, all of these criteria have been met, and this one divergence is slight. As can be seen in the “count” column, it was indeed difficult for learners to “strongly disagree” with the satisfaction items, and along with the heavy weighting towards the “agree” end of the scale for this construct, this suggests one of two possibilities. It could be that these learners are either actually very satisfied with the EMI course, or it could be a sign of acquiescence bias, a common problem with student evaluation data (Graeff, 2005) and which especially may be the case here given that the survey was administered in the faculty that ran the course which the respondents were requested to rate for satisfaction.

Table 4. Category structure functioning for the three constructs.

Category	Count	(%)	Infit MNSQ	Outfit MNSQ	Structure measure	Category measure
Self-efficacy
1 Strongly disagree	51	2	1.64	1.63	None	(−4.41)
2 Disagree	221	10	1.13	1.19	−3.20	−2.41
3 Slightly disagree	488	21	.77	.78	−1.43	−.79
4 Slightly agree	728	32	.76	.76	−.13	.73
5 Agree	536	23	.81	.82	1.52	2.45
6 Strongly Agree	272	12	1.40	1.32	3.24	(4.45)
Engagement
1 Strongly disagree	99	4	1.21	1.23	None	(−3.60)
2 Disagree	210	9	1.11	1.12	−2.27	−1.97
3 Slightly disagree	451	20	.81	.79	−1.36	−.72
4 Slightly agree	626	27	.96	.99	−.14	.53
5 Agree	534	23	.84	.89	1.15	2.00
6 Strongly Agree	376	116	1.18	1.13	2.63	(3.87)
Satisfaction
1 Strongly disagree	28	1	1.96	2.12	None	(−3.80)
2 Disagree	87	4	1.27	1.60	−2.50	−2.11
3 Slightly disagree	297	13	.91	.96	−1.45	−.77
4 Slightly agree	618	27	.84	.88	−.15	.60
5 Agree	689	30	.89	.86	1.30	2.14
6 Strongly Agree	577	25	1.00	1.00	2.80	(4.04)

The results of the analyses provided above suggest that the three constructs (SE, engagement, and satisfaction) are unidimensional, and with a few exceptions, the individual items are working well to separate out varying levels of the respective constructs in the participants. Strong item separation figures further support construct validity. High reliability figures suggest that this survey will provide similar results if administered again to a similar population.

Correlation studies

In addition to investigating the validity and reliability of the instrument, logit scores derived from Rasch were used for bivariate correlations, to investigate the relationship between the three constructs and a separate variable, language proficiency scores. TOEIC L&R scores were available for a small subset of the students (n = 106) who completed the survey. Scores for the 97 of these students were used for a correlation study using JASP 0.16.3 (after outliers from the main cohort were removed). Correlation studies are often employed when comparing various attitudinal constructs (Ahmadian & Ghasemi, 2017; Han & Wang, 2021; Yuan et al., 2023) and because the number of students for whom test data was available was relatively small, correlation was chosen over regression. This also addresses a common issue with many studies dealing with measurement, the “questionnaire curse” (Al-Hoorie et al., 2021, p. 6), whereby all variables, including the dependent variable in a study, come from self-reported measures.

First, logit scores derived from the three survey constructs were exported from Rasch. Outliers were then investigated using boxplots. Across the three variables, a total of nine outliers were identified and removed, leaving a total of 97 participants. Descriptive statistics for the three scales within the survey can be seen in Table 5. As can be seen from that table, skew and kurtosis are both well within the recommended range of −2 and + 2 (George & Mallery, 2010), which suggests that the data is normally distributed.

Table 5. Descriptive statistics for the three constructs of the survey (n = 97).

	Self-efficacy	Engagement	Satisfaction
Mean	0.440	0.377	1.617
Std. Error	0.152	0.108	0.130
Std. Deviation	1.494	1.060	1.278
Skewness	−0.282	−0.004	−0.213
Std Err. Of Skewness	0.245	0.245	0.245
Kurtosis	0.249	0.039	−0.005

The TOEIC L&R is a well-known test in parts of Asia including Japan, and has been shown to be a reliable and generalizable test of language proficiency (Zhang, 2006). In addition, scores from the TOEIC L&R have previously been used in studies showing correlation with variables such as SE (Powers & Powers, 2015). Descriptive statistics for the 97 participants are provided in Table 6 and show that these students performed substantially better in listening than reading, even though both skills are vitally important in EMI classes. As can been seen in Table 6, there were rather large differences in ability between some of the students, with the lowest score of 340, and the highest of 830.

Table 6. Descriptive statistics for the TOEIC listening and reading.

	Listening	Reading	Total
Mean	313.711	216.031	529.742
Std. Error	4.933	5.593	9.456
Std. Deviation	48.586	55.080	93.132
Skewness	−0.169	0.467	0.404
Std Err. Of Skewness	0.245	0.245	0.245
Kurtosis	−0.171	−0.049	0.067
Std Err. Of Kurtosis	0.485	0.485	0.485
Range	240	280	490
Minimum	200	110	340
Maximum	440	390	830

After checking assumptions, Pearson correlations were run on the data. The results showed small, statistically significant, positive correlations between TOEIC Reading scores and SE (r = .303, n = 97, p = 0.003), engagement (r = .218, n = 97, p = 0.032), and satisfaction (r = .267, n = 97, p = 0.008), suggesting that reading proficiency in particular may have a relationship with the attitudes investigated with this scale (Table 7). On the other hand, there were no statistically significant correlations with TOEIC listening scores. A potential reason for this relationship between reading ability and the three variables in question may lie in the fact that reading is an important part of the EMI classes of these learners (and indeed in many similar EMI programmes). EMI classes will, by necessity, include readings that are more difficult than those found in regular EFL classes. This is because EMI classes are primarily focused on building knowledge about the academic content, as opposed to pure language development, and therefore reading skills are particularly important in EMI classes (Kang et al., 2023).

Table 7. Correlations between proficiency and self-efficacy, engagement, and satisfaction (n = 97).

	Pearson’s r	p	n
TOEICL – Self-efficacy	0.175	0.086	97
TOEICL – Engagement	0.134	0.192	97
TOEICL – Satisfaction	0.197	0.053	97
TOEICR – Self-efficacy	0.303	0.003	97
TOEICR – Engagement	0.218	0.032	97
TOEICR – Satisfaction	0.267	0.008	97

Conclusion

As the number EMI programmes continues to grow in contexts where English is not an official language, and where English language proficiency and motivation may be low, the need to investigate the effects of EMI programmes on pedagogical outcomes, both positive and negative, becomes extremely important. Such programmes cost a great deal in terms of both money and time (for institutions, teachers, and students alike) and it is imperative to maximize the outcomes, in terms of discipline-related knowledge, and language development. Also, because the EMI research that has looked at learner issues has hitherto focused primarily on language learning concerns, there is a gap in studies looking at other areas such as psychological variables. This paper described the creation of a survey instrument to address this. It was designed to measure EMI learners’ levels of SE, engagement, and satisfaction concerning aspects of both the subject being taught and the language in which it is taught (English). The survey was administered to 287 Japanese tertiary students in EMI or EMI-like classes.

The results of a Rasch PCAR in conjunction with item-level analysis were used to closely investigate the functioning of the instrument. The three constructs were found to be largely unidimensional, with most of the items working well to discriminate varying levels of SE, engagement, and satisfaction among participants. The engagement construct had a slightly higher than recommended eigenvalue in the first contrast, but investigation of individual items suggests that they are functioning adequately to measure engagement of learners in EMI classes. One item in the SE construct and one item in the satisfaction construct had slightly higher than recommended infit scores, but the results of the PCAR along with data from Wright maps and acceptable person and item reliability and separation, all suggest that items in the instrument are working effectively to measure the three constructs with a range of tertiary learners. The same instrument used with a different population should theoretically bring about similar results.

In addition to these analyses, Wright maps and an investigation of category structure functioning revealed a noticeable ceiling effect for the satisfaction construct (Figure 3). This is possibly due to acquiescence bias, a common problem with student evaluations (Graeff, 2005). Nonetheless, for future reiterations, inclusion of one or two more “difficult to endorse” items might be useful. After taking into consideration the results of the PCAR, separation and reliability data, and individual item analyses, it was decided that the items in the section are still able to adequately function to gain valid and reliable data on learner satisfaction in EMI courses.

In previous EMI studies, there appears to have been an underreporting of instruments themselves. In providing the final instrument with all items included (see Appendix), we hope that other researchers might be able to use or further refine it, and that it can be employed in future studies of learners in various EMI contexts, as a concise, reliable, and valid measurement tool. Practically, it means that these important areas might become better understood by teachers and programme coordinators.

Finally, a correlation study was conducted with logit scores for the three constructs derived from Rasch and TOEIC L&R test scores for a subset of the main cohort (n = 97). While there were no statistically significant correlations with the Listening scores, there were small, but significant, positive correlations between all constructs and TOEIC Reading scores. This suggests that there is a positive connection between English proficiency and the three variables measured here. Students are required to do a considerable amount of challenging reading as preparation for each class and therefore students’ reading proficiency level possibly impacts these factors.

Future studies should seek to further improve on measurement instruments for EMI learners, addressing both language and the discipline being taught. In doing this, calls for greater collaboration between disciplines experts and applied linguistics will be met, which could ultimately lead to more positive outcomes in EMI programmes. We hope this paper provides an example of what might be achieved with such collaboration.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Notes on contributors

Justin Harris

Justin Harris is a professor in the Faculty of Economics at Kindai University in Osaka, Japan. His research interests center around methodologies that encourage language motivation including task-based language teaching (TBLT). Justin is cofounder of the JALT TBLT special interest group which is focused on bridging the gap between TBLT research and classroom implementation. He has a particular interest in measurement, and has published extensively on the develop of instruments for use within applied linguistics and the greater field of education.

Patrick Strefford

Patrick Strefford is Professor of International Relations at Kyoto Sangyo University, Japan where he teaches courses on International Relations and International Development. His research focuses on Myanmar’s foreign relations, particularly aid donors’ policies and practices towards Myanmar, focusing especially on that of Japan’s. In 2013, he was awarded a Japan Society for the Promotion of Sciences grant to support research into international aid to support the transition in Myanmar. He is a co-editor of a 2020 volume on Myanmar’s transition.

Appendix

EMI Satisfaction, Engagement, Self-Efficacy Survey

Satisfaction

1. I think that this course has taught me what I want to know about the main topic.

2. Through this course, I was able to develop my general English ability.

3. Through this course, I was able to develop my English related to the main topic.

4. I think that I was able to become more of an international person through this course. (L)

5. I need to have a high level of English competency for my future career.

6. Choosing this faculty was a good idea.

7. I think that my classmates were motivated to work hard in this class.

8. I think that the course materials for this class met my needs).

Engagement

• (9) The topics we discussed in class made me want to learn more about the topics.

• (10) It was very interesting listening to the teacher talk about the main topics of this class.

• (11) The textbook inspired me to learn more about the topic.

• (12) I sometimes forgot time in class because I was so involved in the topic.

• (13) When we did group discussions, I became very involved in them.

• (14) When we did tasks in class, I became involved in them.

• (15) Doing extra work out of class for this course is not hard because it was interesting for me.

• (16) I enjoyed talking to my classmates about the topics of this class.

Self-efficacy

• (17) The tasks that we did in class were easy for me to do.

• (18) I could understand everything that the teacher talked about.

• (19) I always contributed a lot to group discussions.

• (20) I was able to complete all homework assignments for this class adequately.

• (21) My English ability was adequate for this class.

• (22) I could understand the content of the textbook for this class.

• (23) I was at least as good as most of my classmates at understanding the topic.

• (24) I had enough knowledge of the main topic of this class that I could discuss it in my own language.