The development and validation of a scale for measuring L2 learning experience: the underappreciated element of the L2 motivational self system

ABSTRACT

The current study developed and validated a holistic and domain-specific scale for measuring the second language (L2) learning experience within the framework of the L2 Motivational Self System. Underpinned by positive psychology theories, a preliminary item pool was generated through combining insights from extant measurements, interviews with Chinese learners of various L2s (e.g. Japanese, French, German, Spanish), as well as literature L2 motivation. This L2 learning experience scale (LLES) was firstly piloted with 302 Chinese language-major undergraduates and further validated with a sample of 304 students from similar backgrounds. Both exploratory factor analysis as well as confirmatory factor analysis were conducted respectively to unravel its factorial structure and examine the model fit, validating a 28-item LLES with six dimensions (e.g. Positive emotions; Negative emotions; Engagement; Relationship; Meaning; Accomplishment). Validity and reliability analyses further established the convergent and divergent validity of the scale, exhibiting robust psychometric properties. The results revealed that the LLES is a sound scale for measuring learners’ multifaceted L2 learning experience and could add new insights into learners’ L2 motivational development.

KEYWORDS:

• Language learning motivation
• L2 learning experience
• Positive psychology
• PERMA
• L2 learning experience scale (LLES)

Introduction

Second language (L2) learning experience is to the L2 Motivational Self System (L2MSS) what emotion is to second language acquisition (SLA): both are ‘elephants in the room’ within its own kind (Prior 2019). While the role of L2 learning experience has been widely recognised and emphasised for decades (Csizer and Kormos 2009; Csizér and Kálmán 2019a), this third constituent of the L2MSS has been largely absent from L2 motivation literature (Al-Hoorie 2018; Ushioda 2011). As empirical inquiries have evidenced (e.g. Huang 2019; Lamb 2012; Papi and Teimouri 2012; You and Dörnyei 2016), L2 learning experience can be the most powerful predictor of motivated behaviour, as well as a strong predictor of other criterion measures such as intended efforts and language achievement. In this case, its marginalised status within the realm of L2 motivation is unjustified.

Ignorance about the L2 learning experience, which is as significant, if not more so than, motivational self guides, might be attributed to two factors. Firstly, this notion has long been ill-defined and under-theorised (Csizér and Kálmán 2019a). Following Dörnyei’s (2009) encompassing yet slightly ambiguous definition, researchers have struggled to determine what the construct of L2 learning experience should entail, both theoretically and empirically. The decision of whether to focus solely on immediate classroom-related factors or to broaden the scope to include past out-of-school experiences continues to be a topic of debate (Asker 2012). Secondly, its measurement is problematic. As an add-on component to the L2MSS, the operationalisation of L2 learning experience suffers from a jangle fallacy, a situation in which multiple terms are used to refer to the same construct of L2 learning experience. Oftentimes, researchers have aimed to investigate learners’ L2 learning experience in the literature review while employing items assessing ‘attitudes towards language learning’ in its instrument (e.g. Taguchi, Magid, and Papi 2009; You and Dörnyei 2016). Other terms commonly used in research underpinned by the L2MSS include ‘L2 learning attitude’ (Kormos, Kiddle, and Csizer 2011) and ‘English learning experience’ (Papi 2010), with the latter unconsciously equating the L2 with English. In general, regardless of the terminology used in studies, L2 learning experience is usually assessed with a brief scale (Dörnyei 2019) comprising items addressing only a few aspects of learners’ experience, such as learners’ attitudes toward the foreign language or their enjoyment of the classroom setting. While such scales may focus on some important aspects, their inability to fully capture the complexity of the L2 learning experience necessitates the development and utilisation of more extensive and diversified scales.

Therefore, I sought to fill this research gap by developing and validating a multi-dimensional scale to uncover the nuances of the L2 learning experience. Specifically, I employed positive psychology (PP) principles as my theoretical base, and I targeted learners of languages other than English (LOTEs) to address the prevalent monolingual bias in SLA (Dörnyei and Al-Hoorie 2017). As a first attempt to develop a comprehensive yet parsimonious scale to measure the L2 learning experience, this study will expand the current viewpoints of the conceptual and empirical inquiry of L2 motivation.

Literature review

Conceptualisation of the L2 learning experience

Distinct from the other two future self guides in the L2MSS, namely the ideal L2 self and the ought-to L2 self, L2 learning experience refers to the executive motives of learning contexts, including the impact of the teacher, curriculum, peer group, and experience of success (Dörnyei, 2009), aspects which can exert profound influences on L2 learning motivation (Wang and Liu 2020). In parallel with the motivational renaissance that occurred in the 1990s, which brought the impacts of the classroom learning situation to the fore (Mahmoodi and Yousefi 2021), L2 learning experience was originally included as an add-on constituent to the L2MSS to acknowledge those situation-specific factors in the immediate classroom. However, convinced by the potential of the novel self approach, motivational scholars have generally focused on the two future selves, so little elaboration has been provided on the conceptualisation of the L2 learning experience (Dörnyei and Ryan 2015), leaving it as ‘the Cinderella of the L2 Motivational Self System’ (Dörnyei 2019, 22).

The unspecified nature of the L2 learning experience is observable in extant L2 motivation research in which scholars have tended to develop their own interpretations of this umbrella term (Hiver et al. 2019). In You and Dörnyei’s (2016) large-scale survey of English learning motivation in China, language learning experience was operationalised as attitudes toward L2 learning and used items such as ‘I find learning English really interesting’ to measure this construct (501). This working definition of L2 learning experience, however, is more concerned with students’ appraisal of the language than their situation-specific motives, as proposed in the L2MSS. Contrastingly, Lamb (2012) broadened the concept in their research on Indonesian English as a foreign language students’ motivation. Taking a more inclusive perspective, he not only examined classroom L2 learning experience but also included out-of-school learning experience. Similarly, Csizér and Kálmán (2019a) recognised the complexity of language learning experience and advocated that it entails both concurrent and retrospective aspects. Underpinned by Douglas Fir Group’s (2016) framework, they mapped the L2 learning experience on the micro, meso, and macro levels, with components ranging from interactional resources and the impacts of school and family to cultural and economic values.

More recent attempts delving into the theorisation of L2 learning experience have mirrored the general trends in SLA. One prominent focus has been on the dynamic development of people situated in real contexts (Hiver and Al-Hoorie 2016). For instance, Hiver et al. (2019) innovatively employed the complex dynamic systems theory to reframe the L2 learning experience. Specifically, they operationalised this construct more organically as a ‘relational and soft-assembled complex system’ (Hiver et al. 2019, 87), suggesting that L2 learning experience comprises four prototypical scenes: initiating language learning, sparking interest, overcoming difficulties, and fostering human connections. Notably, their results foregrounded emotional loading in learners’ narrative interviews, which plays a key role in their motivated behaviours. Concomitant with recent research interests in learner engagement (Guo, Xu, and Chen 2022; Sinatra, Heddy, and Lombardi 2015), Dörnyei (2019) critically reflected on his original definition of the L2 learning experience and employed an engagement-specific perspective to revisit this concept. In his new proposal, the L2 learning experience is defined as ‘the perceived quality of the learners’ engagement with various aspects of the language learning process’ (Dörnyei 2019, 25). He also spelled out an array of specific facets for learner engagement, such as the school context, peers, learning tasks, and teachers.

Despite these initial efforts to retheorise the L2 learning experience, several issues should be addressed before elucidating its crucial role in learners’ motivational development. Firstly, while there is no consensus for what components comprise the construct of L2 learning experience, it is certainly a multi-dimensional conglomerate (Csizér and Kálmán 2019b). Diachronically, it should not be confined to immediate classroom experience; rather, retrospective L2 learning experiences from the past should also be included. Recurrent dimensions that scholars have mentioned entail teacher–student rapport, experience of success, positive affect, and classroom-relevant factors (e.g. Csizér and Kálmán 2019a; Dörnyei 2019; Hiver et al. 2019). However, these facets are rather fragmented, presenting a challenge for researchers who seek to investigate this construct systematically. An appropriate theoretical framework is therefore needed to determine these facets so that a holistic understanding of learners’ L2 learning experience can be developed (Asker 2012; Dörnyei 2019). Secondly, the measurement of the L2 learning experience is problematic. As mentioned earlier, although the L2 learning experience is characterised by its multidimensionality, researchers have tended to simplify the instrument, using only a few items to assess learners’ attitudes toward the language. To explore the underlying mechanism of L2 motivational development, it is thus crucial to develop and validate a multi-faceted scale that can measure the complex L2 learning experience.

Measuring L2 learning experience through a positive psychology lens

One frequently used instrument to gauge L2 learning experience is derived from Taguchi, Magid, and Papi’s (2009) comparative study in which they examined whether the L2MSS can be generalised to other cultural contexts. Four items measuring learners’ attitudes toward learning English were validated in both the Japanese and Chinese contexts, and six items were utilised in Iran. However, given the multidimensionality of the L2 learning experience (Csizér and Kálmán 2019a), a prerequisite is to identify a new theoretical framework that would allow for a more detailed and elaborate measurement (Dörnyei 2019).

Echoing recent ‘emotional turns’ in the field of SLA (Dewaele and Li 2020), I aimed to employ a model based on PP to develop a new scale explaining the L2 learning experience. Contrary to the predominant emphasis on hindrances to learners’ language development, PP prioritises what may enrich learners and help them to flourish (Dewaele et al. 2019; MacIntyre and Gregersen 2012). As such, incorporating PP principles into L2 motivation research facilitates a holistic understanding of learners’ complex experiences by creating a balance between positive and negative contextual factors. Among several notable PP frameworks such as Seligman’s (2011) PERMA model, Oxford’s (2016) EMPATHICS construct, and the most recent E4MC model of language learner well-being (Alrabai and Dewaele 2023), I chose Seligman’s (2011) PERMA construct to guide my measurement of L2 learning experience. This five-dimensional model, which assesses learners’ positive emotions (P), engagement (E), relationships (R), meaning (M), and accomplishment (A), serves as an appropriate framework for interpreting L2 learning experience due to its theoretical alignment with L2 motivation and its widely validated measurement approach.

Firstly, PERMA has great relevance and explanatory potential for learners’ L2 learning experience and motivation. Central to the PERMA model is positive emotions, which include ‘positively valenced emotions such as joy, contentment, and excitement’ (Schwartz et al. 2016, 518). The positive emotions that L2 learners experience can play a key role in their language learning process given its ‘broaden (momentary thought–action repertories) and build (enduring personal resources)’ features (Fredrickson 2003; 2013). Specifically, emotions can help learners cope with future opportunities and challenges, producing a motivating force to act (MacIntyre and Gregersen 2012; Reeve 2005). Engagement refers to involvement and participation in an activity (Seligman 2011) and includes multiple aspects of students’ L2 learning experience, such as their engagement with the course materials and cognitive effort to improve their learning proficiency. Mercer and Dörnyei (2020) suggested that the essence of the engagement construct resides in the behavioural aspect, which mirrors active participation and serves as a crucial foundation for subsequent motivation. The third element of PERMA is relationships, which relate to people’s interactions with others and features prominently in learners’ L2 learning experience. Interpersonal connections, including the teacher–student relationship (e.g. Goetz et al. 2022; Henry and Thorsen 2018), peer dynamics (e.g. Machin 2020), and relationships with significant others (e.g. role models) (e.g. Muir, Dörnyei, and Adolphs 2021), can have immediate influences on students’ motivated behaviours. The fourth dimension of PERMA, meaning, is associated with people’s beliefs and values (Seligman 2011). From the perspective of language acquisition, this facet can be succinctly interpreted as an answer to the inquiry, ‘Why is learning a language meaningful?’ The last element of Seligman’s (2011) PERMA model, accomplishment, embodies a learner’s advancement toward their personal goals and their capabilities to undertake relevant tasks. Gregersen (2019) pioneered the explicit application of the PERMA model to explain learners’ L2 motivation, positing that both engagement and accomplishment correlate with students’ sustained efforts in L2 learning, while relationships and meaning hold more relevance to students’ initial L2 motivation. Another relevant study was conducted by Oxford and Cuéllar (2014), who examined Chinese as a foreign language learners’ narratives using the PERMA construct and found that all five facets manifested in students’ L2 learning experience. Similarly, Li and Liu (2023) applied the PERMA model to conceptualise language learning experience among international students of Chinese. Their research indicated that the L2 learning experience is a multi-faceted construct that involves five interconnected elements: emotions, engagement, relationships, meaning, and accomplishment. These elements, crucially, are not static but continually interact throughout the L2 learning process, influencing learners’ motivation in both positive and negative directions.

In addition, I selected PERMA as an overarching framework since it has a well-established self-report measurement, which can lay the foundation for future adaptations. Underpinned by Seligman’s (2011) theory of PERMA, Butler and Kern (2016) undertook a large-scale quantitative study and developed a 15-item PERMA Profiler to measure people’s overall well-being. In this scale, additional eight items were added specifically to assess individuals’ negative emotions, loneliness, and general physical health. This corresponds well with the assertion that PP does not ignore the negative aspects of life but, instead, aims to utilise positive experiences in addressing these problems (Seligman and Csikszentmihalyi 2000). As a prominent well-being measure, the multi-dimensional PERMA Profiler has been applied in various educational contexts and has demonstrated solid psychometric properties (e.g. Coffey et al. 2016; Umucu et al. 2020; Wammerl et al. 2019). However, despite its usefulness as a domain-general measure for people’s flourishing, no study within the realm of L2 research has employed the PERMA Profiler to measure the L2 learning experience. As such, I aim to develop and validate a PERMA-informed L2 learning experience scale (LLES) to capture the nuances of the L2 learning experience. To do so, I pose the following research questions:

RQ1: What is the factorial structure of the L2 learning experience?

RQ2: To what extent is the L2 learning experience scale valid and reliable?

Methodology

Research context

This research was part of a larger investigation into Chinese learners’ LOTE motivation, and the ultimate goal was to discover how L2 learning experience contributes to LOTE motivational dynamics over time. Therefore, the current research was designed as the first stage of this investigation, with a focus on the development and validation of a PP-informed scale to measure L2 learning experience, which can be used alongside other motivational instruments to track learners’ motivational trajectories. As previously noted, existing scales for assessing L2 learning experience were primarily developed within the context of English language learning. To counteract this monolingual bias, I undertook the present study within the setting of LOTE learning, involving learners from language degree programmes in ordinary universities in China (not as elite as China’s ‘Double First Class’ universities, a list that includes some of the top universities in the country such as Tsinghua University and Peking University). Specifically, data were collected from both English majors, who were required to learn a LOTE as part of their curriculum, and LOTE majors, who pursued an undergraduate degree in a LOTE. In China, LOTE learning is a compulsory part of the English major curriculum, and students can choose a LOTE to study from the available options. After four semesters (approximately 256 credit hours) of LOTE learning, they must reach the level of a second-year student in the corresponding language (Zhao and Li 2014). As such, these English majors represent a significant proportion of LOTE learners in China and are thus worth investigating. Specifically, I recruited two samples of Chinese language-major undergraduates for developing and validating the scale. Table 1 presents the demographics of the participants.

Table 1. Demographics of the participants.

	Sample 1 (n = 294)		Sample 2 (n = 304)
Type	English major	LOTE major	English major	LOTE major
	208 (70.7%)	86 (29.3%)	221 (72.7%)	83 (27.3%)
Age	Mean = 20.84 (SD = 2.04)		Mean = 20.14 (SD = 1.54)
Gender	Male = 17 (5.8%)		Male = 46 (15.1%)
	Female = 258 (87.8%)		Female = 238 (78.3%)
	Prefer not to say = 19 (6.4%)		Prefer not to say = 20 (6.6%)
Year	Year 1 = 33 (11.2%)		Year 1 = 60 (19.7%)
	Year 2 = 30 (10.2%)		Year 2 = 51 (16.8%)
	Year 3 = 91 (31.0%)		Year 3 = 115 (37.8%)
	Year 4 = 140 (47.6%)		Year 4 = 78 (25.7%)
LOTE	Japanese = 131 (44.6%)		Japanese = 117 (38.5%)
	French = 89 (30.3%)		French = 64 (21.1%)
	German = 21 (7.1%)		German = 47 (15.5%)
	Spanish = 7 (2.4%)		Spanish = 39 (12.8%)
	Russian = 16 (5.4%)		Russian = 11 (3.6%)
	Korean = 10 (3.4%)		Korean = 5 (1.6%)
	Other language = 20 (6.8%)		Other language = 21 (6.9%)

Research design

Following Boateng et al.’s (2018) guidelines for validating a scale, I employed a three-phase research design including item development, scale development, and scale evaluation. During the first stage, I conducted 12 semi-structured interviews with both English majors and LOTE majors to gain a nuanced understanding of their L2 learning experience.¹ Subsequently, I synthesised the insights from the interviews, existing measurements, and pertinent literature to construct a model of L2 learning experience. This led to a preliminary pool of items, all in Chinese, which was the native language of the target participants. As a second step, these items were subjected to a series of validity and reliability tests that involved five SLA experts (for face validity) and 294 Chinese LOTE learners (for exploratory factor analysis, EFA). Corresponding revisions were made, and I developed a finer scale measuring various facets of the L2 learning experience. During the third and last phase, I conducted confirmatory factor analysis with another sample of 304 Chinese LOTE learners to evaluate the scale’s psychometric properties.

Scale construction

As mentioned in the above section, the scale items used for further analysis were derived from three sources: established measurements available in mainstream psychology, theoretical input from SLA literature, and interviewees’ opinions. To start with, I chose Butler and Kern’s (2016) PERMA Profiler, a widely validated questionnaire within the realm of PP, to serve as a foundation for item exploration. Specifically, each dimension of the PERMA construct, namely individuals’ positive emotions, engagement, relationships, meaning, and accomplishment, is assessed with three items. However, since these items are relatively generalised, I made several revisions to render them suitable for the language learning domain. For instance, I changed the item under the engagement dimension, ‘How often do you become absorbed in what you are doing?’ to ‘How often do you become absorbed in L2 learning?’

Having determined the core dimensions and elements of the scale, I also examined recent scholarship at the intersection of PP and SLA, particularly concerning emotions and engagement, to garner more information and inspiration. Compared to the original PERMA Profiler, which focuses on a limited set of positive emotions (contentment and joy), I expanded the item repertoire to encompass a broader variety of positive emotions, such as interest, hope, and pride. In addition, considering the ambivalence in SLA, namely the co-occurrence of negative and positive emotions (Mercer and MacIntyre 2014), I added another dimension, negative emotions, to capture the complexity embedded in the L2 learning experience. I similarly expanded and revised the items of the other dimensions. For instance, due to the multi-faceted nature of engagement (Zhou, Hiver, and Al-Hoorie 2021), I included multiple items to evaluate students’ behavioural, cognitive, and social engagement in L2 learning.

During the last stage of item pool generation, I drew on the information obtained from the 12 Chinese LOTE learners (six English majors and six LOTE majors) via the semi-structured interviews to extend and complement the scale items. That is, I asked overarching questions addressing why students chose to learn an L2, various aspects of their L2 learning experience, and how this learning experience contributed to their motivational development. As expected, their responses confirmed the prevalent existence of both negative and positive emotions when learning an L2 and added valuable supplements to the item pool.

Overall, with the combined expertise from the three aforementioned sources, a preliminary six-dimension, 58-item scale was generated to measure learners’ L2 learning experience (Figure 1). For this scale, respondents are required to indicate to what extent they agree with the statements about their L2 learning experience on a six-point Likert scale, ranging from 1 (strongly disagree) to 6 (strongly agree). Notably, I adopted an even number rating scale to prevent respondents from choosing the middle option without making a real choice (Dörnyei 2003). To ensure clear and accurate expressions and understanding, all the scale items are in Chinese.

Figure 1. The preliminary model of language learning experience.

Procedure and data collection

Prior to conducting an array of statistical validation tests to confirm the psychometric properties of the scale, two groups of people first reviewed the newly developed LLES to evaluate its content validity (Boateng et al. 2018). Five expert judges (SLA researchers) were invited to scrutinise the items and offer advice on the following issues: i) whether each item measures the targeted aspect of language learning experience; ii) whether the statement is ambiguous or leads to confusion. Considering their suggestions, I made corresponding revisions including deleting, rewording, and adding certain questions, leaving 56 items in total for further analysis. Then, a second round of content validity checking was conducted with the help of 15 LOTE learners from an average university in China, who were asked to read through and answer all the items. Incorporating their comments with the completed questionnaire, I refined the wording of a few statements, finalising a 56-item scale for subsequent validation.

Having finished the first stage of item pool generation, I employed a convenience sampling strategy to recruit samples in order to proceed with the next two phases of scale development and evaluation. I uploaded all the initial scales to www.wjx.cn, an online platform for survey design and administration, and QR codes were generated and inserted in a ‘call for participants’ poster. Then, I contacted my friends who were English or LOTE majors and teachers who taught English or LOTE majors and asked them to distribute the posters to my target population. I strictly followed ethical guidelines throughout, and informed consent was required prior to participation in the research.

Results

RQ1: what is the factorial structure of the L2 learning experience?

To reveal the underlying structure of the newly developed 56-item LLES, I conducted EFA to address the first research question.

Preliminary analyses

I examined initial EFA statistical requirements by conducting item-level descriptive analyses, including normality tests and overall reliability coefficient tests (Phakiti 2018). Firstly, as Hair et al. (2010) and Byrne (2010) stated, data are considered normally distributed if the skewness falls between −2 to +2 and if kurtosis falls between −7 to +7. Following these criteria, the skewness and kurtosis of all 56 items in the scale were within the range of −1.987 to 1.249 and −1.214 to 6.905, respectively, demonstrating that the univariate normalities of the item data were supported. To continue with an EFA approach, I also calculated the Cronbach’s alpha coefficient of the 56-item questionnaire, which was 0.941, well above the cut-off value of 0.70 (Phakiti 2018).

Exploratory factor analysis of the LLES

To determine the number of latent constructs of the LLES and its factor structure and to obtain a parsimonious scale, I conducted EFA using IBM SPSS Statistics 28.0 software. The Kaiser–Meyer–Olkin (KMO) measure indicated a KMO of 0.945, larger than 0.60, indicating data sampling adequacy (Fabrigar and Wegener 2012). Bartlett’s test of sphericity was also desirable (p < 0.001), confirming strong inter-item correlations and suggesting that the scale is factorable (Osborne 2014). Principal component analysis with varimax rotation was incorporated to extract the underlying dimensions. Specifically, while the scale was considered to consist of six dimensions, this limit was not imposed during the extraction process. Item and factor retention decisions were determined by the following criteria: i) factors with an eigenvalue greater than 1 (Kaiser criterion), ii) the scree plot, iii) factor loadings greater than 0.50, and iv) substantive interpretability and theoretical consistency of the factor structure (Hair et al. 2010; Phakiti 2018). Notably, a general cut-off value for an item to be retained is 0.30 or 0.40 (e.g. Field 2018; Osborne 2014), but I chose a minimum factor loading of 0.50 so that at least 25% of the shared variance could be extracted (Phakiti 2018). As a result, a total of 18 items (5c, 5 h, 6d, 6 g, 6j, 7f, 7 g, 7 h, 7i, 7j, 8a, 8b, 8e, 8f, 8 g, 8i, 9a, 9 h) were eliminated, leaving a 38-item LLES (see Table 2). Contrary to the initial proposal of six dimensions, a total of seven factors were extracted, each with an eigenvalue greater than 1. These seven factors collectively accounted for 72.35% of the variance, and the individual contributions of these factors to the overall variance can be seen in Table 3.

Table 2. Results of exploratory factor analysis: Factors, items, and loadings for 38 items.

Factor 1	Factor 2	Factor 3	Factor 4	Factor 5	Factor 6	Factor 7
4c 0.750	5f 0.843	6a 0.848	6f 0.736	7b 0.818	8c 0.863	9f 0.724
4d 0.709	5e 0.785	6c 0.803	6h 0.711	7a 0.779	8h 0.841	9e 0.695
4i 0.708	5g 0.769	6b 0.735	6i 0.682	7c 0.764	8d 0.778	9d 0.680
4e 0.695	5b 0.712		6e 0.617	7d 0.749	8j 0.545	9c 0.661
4b 0.669	5a 0.707			7e 0.701		9g 0.654
4a 0.618	5i 0.701					9b 0.581
4h 0.579	5d 0.638
4g 0.568
4f 0.530

Table 3. Total variance explained (seven-factor extraction).

Total variance explained
	Initial eigenvalue			Extraction sum of the squared loading
Factor	Total	% of variance	Cumulative %	Total	% of variance	Cumulative %
1	14.814	41.614	41.614	14.814	41.614	41.614
2	3.847	10.123	41.738	3.847	10.123	41.738
3	2.426	6.384	48.122	2.426	6.384	48.122
4	1.494	4.194	62.314	1.494	4.194	62.314
5	1.449	3.840	66.144	1.449	3.840	66.144
6	1.324	3.484	69.641	1.324	3.484	69.641
7	1.030	2.709	72.340	1.030	2.709	72.340

Note: Factor 1 = Positive emotions; Factor 2 = Negative emotions; Factor 3 = Behavioural engagement; Factor 4 = Cognitive engagement; Factor 5 = Relationships; Factor 6 = Meaning; Factor 7 = Accomplishment.

Factor 1 included nine items assessing an array of positive emotions that students might experience during their language learning process, including enjoyment, interest, hope, pride, and contentment (MacIntyre and Gregersen 2012; MacIntyre and Vincze 2017). Therefore, this factor was labelled ‘positive emotions.’ One sample item was, ‘When I do well in L2 learning, my heart pounds with pride.’ Factor 2 assessed multiple negative emotions and, when combined with factor 1, constituted essential components of the language learning process (Dewaele and MacIntyre 2016). In addition to the well-researched negative emotion of anxiety, the seven items for factor 2 assessed students’ boredom, shame, frustration, diffidence, and guilt, providing an overview of their emotion-charged experience. One sample item was, ‘I feel a sense of shame when performing badly in L2 exams.’ The hypothesised third dimension of engagement, however, was divided into two factors, with factor 3 labelled ‘behavioural engagement’ and factor 4 labelled ‘cognitive engagement.’ This categorisation agrees with previous work stating that engagement is a multi-faceted construct that can manifest as cognitive, emotional, social, and behavioural facets (Hiver et al. 2021; Philp and Duchesne 2016). Of relevance to the study, three items were included to measure learners’ in-class participation or behavioural engagement (Reschly and Christenson 2012). One sample item was, ‘I pay attention to what teachers have said in class.’ Factor 4 comprised four items, reflecting the mental processes and cognitive efforts of students, such as asking questions, providing feedback, and using strategies (Svalberg 2018). One sample question was, ‘I adopt various strategies to learn the L2.’ Factor 5 was associated with students’ interpersonal interactions established during the learning process and was labelled ‘relationships.’ Altogether, five items were included to reflect the influence of the teacher–student relationship on students’ motivated behaviours (Henry and Thorsen 2018). One sample question was, ‘I receive help and support from my teachers when learning the L2.’ Admittedly, other forms of relationships such as peer interactions or relationships with significant others are also important features of students’ language learning experience (Busse and Williams 2010; Zhang and Tsung 2021), but items assessing these aspects were removed from the scale due to their insufficient factor loadings. As such, only teacher–student relationships were assessed. Factor 6 contained four items and was labelled ‘meaning,’ examining in what way learning an L2 is meaningful to students. One sample item was, ‘Language skills offer a competitive edge in the global job market.’ The last factor of the LLES was ‘accomplishment,’ and as per Seligman’s (2011) conceptualisation, this dimension entailed making progress toward goals and realising them. Notably, it did not only refer to objective achievements such as test scores or awards but also encompassed the feeling of being capable of executing certain behaviours. Factor 7 comprised six items, one of which was, ‘I often achieve the important goals I have set for myself in learning the L2.’

Having determined the factors and items retained, I then performed reliability analyses to examine the internal consistency of the refined scale. Table 4 summarises the names of the newly extracted factors and their corresponding Cronbach’s alpha coefficients. As shown in the table, the Cronbach’s alpha coefficients for both the individual factors and overall LLES were well above the threshold of 0.70 (Phakiti 2018), suggesting high reliability.

Table 4. The seven factors extracted, the number of items, and Cronbach’s alpha coefficients.

	Item	Number of the item	Cronbach’s alpha
1. Positive emotions	4c; 4i; 4d; 4e; 4b; 4a; 4h, 4g, 4f	9	0.938
2. Negative emotions	5f; 5e; 5 g; 5b; 5a; 5i; 5d	7	0.888
3. Behavioural engagement	6a; 6b; 6c	3	0.911
4. Cognitive engagement	6f; 6h; 6i; 6e	4	0.848
5. Relationships	7b; 7a; 7c; 7d; 7e	5	0.918
6. Meaning	8c; 8h; 8d; 8j	4	0.870
7. Accomplishment	9f; 9e; 9d; 9c; 9g; 9b	6	0.891
Overall		38	0.910

RQ2: to what extent is the L2 learning experience scale valid and reliable?

The final stage of scale evaluation was conducted to further examine whether the hypothesised factors fit the collected data (Boateng et al. 2018). Specifically, the seven-dimension, 38-item LLES derived from the EFA was subjected to confirmatory factor analysis (CFA) to assess its construct validity (Brown 2006). I also tested other psychometric properties, including convergent validity, discriminant validity, and reliability, to validate this newly developed measurement.

Preliminary analyses

Prior to proceeding with CFA to confirm the factorial structure obtained via the EFA, I conducted several tests to ensure that the data were multivariate normal and that there were no outliers (Byrne 2010). As a prerequisite to the assessment of multivariate normality (DeCarlo 1997), I first conducted univariate normality tests. The skewness and kurtosis of all 38 items ranged from −1.329 to 0.323 and −0.971 to 2.334, respectively, indicating univariate normal distribution. However, the value of Mardia’s (1974) normalised estimate of multivariate kurtosis was 55.131, suggestive of multivariate non-normality in the sample. While this evidence for multivariate kurtosis may be problematic for interpretations based on the usual maximum likelihood estimation (MLE) (Byrne 2010), if all individual variables exhibit univariate normality, departures from multivariate normality are inconsequential (Hair et al. 2010). As such, MLE was still chosen as the estimation method for subsequent analysis. The squared Mahalanobis distance (D²) for each variable was computed to identify multivariate outliers (Byrne 2010), and since no variable showed a D² that was substantially different from that of the other variables, no outliers were removed from the dataset.

Confirmatory factor analysis of the LLES

Having dealt with the critical requirements and statistical assumptions, as mentioned earlier, I undertook CFA with MLE using AMOS 28.0 to confirm the validity of the seven-factor LLES. I employed multiple commonly applied fit indices to assess the overall fitness, including chi-squared fit statistics/degree of freedom (CMIN/DF), the comparative fit index (CFI), the Tucker–Lewis index (TLI), the root mean square error of approximation (RMSEA), and the standardised root mean residual (SRMR). Following the goodness-of-fit criteria that Hu and Bentler (1999) and Kline (2016) provided, the initial CFA results revealed that the model fit was not satisfactory (see Table 5). While the values of the CMIN/DF (2.596 < 3), SRMR (0.073 < 0.08), and RMSEA (0.073 < 0.08) indicated a good fit, both the CFI and TLI failed to meet the criteria, with 0.861 and 0.848, respectively. This lack of fitness might be attributed to the complexity of the proposed model, with not only seven factors but also up to nine items for each factor. Hence, I deleted several items to improve the model fit and obtain a parsimonious model. As Hair et al. (2010) suggested, the standardised loading estimates should be at least 0.5 and ideally over 0.7. In this case, four items (4i, 5 g, 5i, 7b) with factor loadings of less than 0.5 were removed from the dataset. Another four items (6e, 6f, 4e, 4f) were also deleted due to their high modification indices (over 30). At this stage, only two items (6 h, 6i) remained to measure factor 4 (cognitive engagement), which violated the rule that each latent construct should be indicated by at least three items (Hair et al. 2010). Taking this into consideration, I removed factor 4 and its remaining two items while ensuring that it was theoretically justified to do so. Finally, a 28-item LLES with six factors was subjected to a second round of CFA, and this revised model fit the data adequately, with CMIN/DF = 1.925, CFI = 0.946, TLI = 0.939, RMSEA = 0.055, and SRMR = 0.054 (see Table 5 and Figure 2). In addition, standardised factor loadings all exceeded the 0.50 threshold, suggesting that the items were strongly associated with their corresponding constructs, which is a fundamental sign of construct validity (Hair et al. 2010).

Figure 2. Graphical representation of the six-factor LLES and factor loadings. All modelled correlations and path coefficients are significant (p < .001).

Table 5. Results of the two-round confirmatory factory analysis (N = 304).

	CMIN/DF	CFI	TLI	SRMR	RMSEA	90% CI
Benchmark	< 3	> 0.90	> 0.90	< 0.08	< 0.08
38-item	2.596	0.861	0.848	0.073	0.073	0.068–0.077
28-item	1.925	0.946	0.939	0.054	0.055	0.049–0.062

The convergent validity, discriminant validity, and reliability of the LLES

In addition to checking construct validity via CFA, several items had to be calculated to establish the convergent and discriminant validity of the LLES (Urbach and Ahlemann 2010). Specifically, convergent validity assesses whether scores on items measure the same underlying construct, and this is examined through their factor loadings, composite reliability (CR), and average variance extracted (AVE) (Fornell and Larcker 1981). According to the commonly applied criterion, the minimum requirement for item loadings is 0.5 in CFA. Additionally, as a less-biased estimate of reliability than Cronbach’s alpha, CR anticipates a threshold value of 0.7. For AVE, which measures the amount of variance captured by a construct versus the amount of variance caused by a measurement error, a level of 0.5 is acceptable (Fornell and Larcker 1981). As shown in Table 6, the construct of language learning experience displayed good convergent validity. To assess discriminant validity, the square root of the AVE was used, indicating whether the constructs differed from each other. As Barclay, Higgins, and Thompson (1995) indicated, this attribute for each construct should be larger than the inter-construct correlation. Another guideline that Brown (2006) provided is that inter-construct correlations should be less than 0.85. Taking these requirements into account, the discriminant validity of the LLES could be considered satisfactory (Table 7).

Table 6. The convergent validity of each sub-scale.

		Convergent validity
Sub-scale	Item	λ	p	AVE	CR
Positive emotions	4a	0.890	0.00	0.639	0.913
	4b	0.929	0.00
	4c	0.790	0.00
	4d	0.709	0.00
	4g	0.792	0.00
	4h	0.650	0.00
Negative emotions	5a	0.844	0.00	0.638	0.898
	5b	0.783	0.00
	5d	0.855	0.00
	5e	0.739	0.00
	5f	0.765	0.00
Engagement	6a	0.935	0.00	0.796	0.921
	6b	0.875	0.00
	6c	0.864	0.00
Relationships	7a	0.936	0.00	0.669	0.888
	7c	0.907	0.00
	7d	0.687	0.00
	7e	0.710	0.00
Meaning	8c	0.890	0.00	0.617	0.863
	8d	0.891	0.00
	8h	0.666	0.00
	8j	0.662	0.00
Accomplishment	9b	0.765	0.00	0.637	0.913
	9c	0.785	0.00
	9d	0.814	0.00
	9e	0.830	0.00
	9f	0.766	0.00
	9g	0.825	0.00

Table 7. The discriminant validity and inter-construct correlations of each sub-scale

Scale	1	2	3	4	5	6	AVE
Factor 1	0.799						0.639
Factor 2	−0.477***	0.799					0.638
Factor 3	0.284***	−0.337***	0.892				0.796
Factor 4	0.298***	−0.245***	0.376***	0.818			0.669
Factor 5	0.365***	−0.220***	0.235***	0.243***	0.785		0.617
Factor 6	0.390***	−0.322***	0.347***	0.344***	0.223***	0.798	0.637

Note: Values in the diagonal are the square roots of their respective average variance extracted.

Factor 1 = Positive emotions; Factor 2 = Negative emotions; Factor 3 = Engagement; Factor 4 = Relationships; Factor 5 = Meaning; Factor 6 = Accomplishment

***p < .001

In assessing the reliability of the LLES, I calculated two statistics, including Cronbach’s alpha and CR. Firstly, the Cronbach’s alphas for the whole scale and its six subscales were 0.839, 0.911, 0.898, 0.920, 0.887, 0.860, and 0.913. Similarly, as presented below in Table 6, the CR values for each sub-scale were 0.913, 0.898, 0.921, 0.888, 0.863, and 0.913, respectively. Overall, these coefficients were above the cut-off value of 0.70 and were indicative of satisfactory internal consistency (Hair et al. 2010).

Discussion

Employing Boateng et al.’s (2018) three-phase research design for validating a scale, the current study involved developing a self-report inventory, the LLES, for evaluating learners’ L2 learning experience. Following the three-step process of item generation, scale development, and scale evaluation, the 28-item LLES was established. This scale, featuring the six dimensions of positive emotions, negative emotions, engagement, relationships, meaning, and accomplishment, was found to be a valid and reliable scale with desirable psychometric properties.

The first research question aimed to uncover the factorial structure of the proposed six-factor LLES. The initial scale facets and core items were generated based, in part, on Butler and Kern’s (2016) PERMA Profiler, complemented by recent scholarship at the intersection of PP and SLA and insights from interviewees’ accounts of their language learning experience. Altogether, seven factors were extracted from the EFA, which was slightly different from the original six-factor proposal. Specifically, the proposed ‘engagement’ sub-scale was divided into two dimensions. Per the definitions that Reschly and Christenson (2012) and Svalberg (2018) provided, these two factors were labelled ‘behavioural engagement’ and ‘cognitive engagement.’ This emergence of the sub-categories was consistent with findings from earlier studies indicating that engagement is a multi-dimensional concept with both cognitive and behavioural facets (Hiver et al. 2021).

Furthermore, CFA was performed to address the second research question and finally develop a vigorous and psychometrically sound measurement of language learning experience within the realm of SLA. This 28-item LLES features six factors, namely positive emotions, negative emotions, engagement, relationships, meaning, and accomplishment. Notably, while the EFA results indicated seven factors, as mentioned earlier, the factor of ‘cognitive engagement’ was removed after considering the high modification indices of its items and the rule of ‘at least three items for one sub-scale’ (Hair et al. 2010). As such, only the sub-scale of ‘behavioural engagement’ was kept in the construct, and to render it more conceptually inclusive, I relabelled it ‘engagement.’ This manifestation of ‘behavioural engagement’ from the dataset supports Mercer and Dörnyei’s (2020) statement that the behavioural facet lies at the core of the engagement construct and accords well with the empirical findings from Guo, Xu, and Chen (2022) showing that language learners report the highest level of behavioural engagement in interactive classrooms.

Overall, after the deletion and refinement of the factors and items, a 28-item LLES with six factors was finally established and validated, indicating desirable construct validity, convergent validity, discriminant validity, and reliability (Boateng et al. 2018). To address the conceptual fuzziness surrounding the notion of language learning experience, this LLES scale indicates a broader theorisation that extends beyond the confines of context (within versus outside the classroom) and temporality (immediate versus past learning experience). By capturing the nuances embedded in students’ language learning process from multiple angles, this scale offers a comprehensive and operational conceptualisation of language learning experience that can be utilised in future research. The current LLES also broadens the scope of the conventional operationalisation of this component, which typically utilises a few items measuring learners’ attitudes toward L2 learning (e.g. Kormos, Kiddle, and Csizer 2011; Taguchi, Magid, and Papi 2009), by including the emotional, behavioural, and social dimensions of language learning experience. Specifically, in response to MacIntyre, Gregersen, and Mercer’s (2019) call for a more holistic view of L2 learners’ emotions, an appropriate number of items was included to assess learners’ positive and negative emotional states when learning an L2. This integration of two opposing emotional types allows for a nuanced understanding of their joint effects on the language learning process, which would otherwise be incompletely examined (Dewaele and Dewaele 2017; MacIntyre and Vincze 2017).

The behavioural aspect of language learning experience is also featured in the scale and mainly addresses students’ behavioural engagement in the classroom. This addition of an engagement-relevant factor is congruent with the proposal of Dörnyei (2019), who recently defined L2 learning experience as ‘the perceived quality of the learners’ engagement with various aspects of the language learning process’ (25). He further detailed multiple facets of learners’ L2 learning process and environment that can be engaged with, including the school context, teaching materials, learning tasks, peers, and teachers. This retheorisation can be considered an expansion of the original definition of L2 learning experience in the L2MSS, which refers to the executive motives of learning situations, such as the impact of the teacher, curriculum, peer group, and experience of success (Dörnyei, 2009). However, while this engagement-related specification offers a promising way to capture the complexity of the language learning process, the current LLES prioritises behavioural engagement, which is the ‘core construct and most prototypical of engagement’ (Skinner et al. 2008, 778).

The scale also considers the social aspect of L2 learning experience, especially the teacher–student relationship. The need for relatedness or the feeling of being supported can offer great help when learners need to deal with demanding tasks and can boost their motivation (Henry and Thorsen 2019). Notably, while the factor of ‘relationships’ should theoretically cover a range of social interactions such as teacher–student relationships, peer relationships, and the role of significant others (Williams and Burden 1997), only the teacher–student relationship is included in the scale. This is mainly because Chinese LOTE learners predominantly accumulate their L2 learning experiences within the classroom setting. With the transition to online learning during the COVID-19 pandemic, the teacher–student relationship assumed greater significance, while opportunities for peer dynamics were significantly limited. As a result, items designed to measure peer relationships in the LLES scale exhibited low factor loadings and were consequently removed during the development process.

Research limitations, future directions, and implications

Although theorising language learning experience and developing a corresponding measurement indicate progress in the field, this cross-sectional study had limitations. Firstly, while this PERMA-informed LLES captures certain aspects of language learning experience, not all are captured. For instance, it does not assess curriculum-related facets such as course goals and materials, which, as Dörnyei (2019) noted, can also be integral to language learning experience. Similarly, as the LLES was deliberately designed as a comprehensive (in terms of dimensions encompassed) yet parsimonious (in terms of items for each factor) scale, certain nuances might become lost during operationalisation. This is particularly the case since the factor of engagement is a meta-construct that includes not only behavioural engagement but also emotional and cognitive engagement (Lin and Huang 2018). Hence, focusing only on students’ behavioural participation in language courses may limit the overall view of their language learning process, so it would be advantageous for future researchers to take various aspects of engagement into account when developing the measurement. In addition, only one-time cross-sectional data were collected to develop and evaluate the data. It is well-advised to establish test–retest reliability by administering the scale to the same cohort of participants later (DeVellis and Thorpe 2021). In future research, it would also be valuable to confirm the criterion validity of the LLES by comparing it to other established scales assessing L2 learning experience. Additionally, administering the LLES at different time points throughout a language course and correlating scores with objective measures of language proficiency would provide valuable information about the scale’s longitudinal validity and ability to predict language learning outcomes.

Nevertheless, both theoretical and pedagogical implications can be drawn from the current research. Echoing Csizér and Kálmán’s (2019a) call for examining the internal structure of foreign language learning experience, this study makes an initial attempt to develop a PP-informed, multi-dimensional scale to measure this marginalised component in the L2MSS. By assessing students’ emotional states, behavioural engagement, relationships, beliefs (meaning), and accomplishment throughout their language learning process, this scale unpacks the subtleties embedded in their learning experience. As such, it not only facilitates a more holistic understanding of the ill-defined conceptualisation of language learning experience but also offers a psychometrically sound measurement to operationalise this construct in research practice. Specifically, this scale can be utilised in conjunction with other motivational measures of, for instance, goal orientation and self-efficacy, enriching the array of instruments available. In addition, as the fundamental principles of motivation in language learning are not exclusive to a single language, this scale can be readily adjusted and applied across various language learning contexts, facilitating large-scale quantitative data collection in L2 motivation research.

Aside from its theoretical contributions, this LLES is also of pedagogical significance since it offers a specific instrument for teachers to understand learners’ L2 learning experience from various perspectives. This is particularly helpful for those LOTE learners who might encounter distinct challenges in a nuanced sociocultural context in which English learning is oftentimes prioritised (Dörnyei and Al-Hoorie 2017). Notably, the linguistic knowledge of learners who learn a LOTE as an L2 depends heavily on instructed learning, which necessitates teachers’ cooperation and strategy use. By administering this scale, teachers can gain insights into their students’ multi-layered language learning experience, even if they lack advanced research skills. This nuanced knowledge will enable them to tailor their instruction, classroom activities, and feedback to effectively promote individual motivation when appropriate. Specifically, these PP-informed principles can also be incorporated into pedagogical innovations or interventions to boost or sustain learners’ L2 learning motivation.

Acknowledgements

I would like to thank all the participants for their cooperation and support in my project. Great thanks also go to all my friends who helped distribute the questionnaire.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 The term ‘L2 learning’ broadly refers to learning any additional language, and in the context of this research, it is specifically employed to signify the participants’ experience in learning LOTEs rather than English.

Appendix

The L2 learning experience scale (LLES)

Table

Subscales	《二语学习经历量表》中文版	The L2 learning experience scale (LLES)
Positive emotion	1. 对我来说学习二语很享受也很有趣	1. Learning a L2 is enjoyable and fun for me
	2. 我学习二语能获得满足感	2. I have a sense of fulfilment when learning a L2
	3. 当我掌握该语言知识时，我会有成就感	3. I have a sense of achievement when I master the language skill
	4. 与该语言相关的文化很吸引我	4. The culture behind the language holds great appeal for me
	5. 我对二语学习持乐观态度	5. I hold favourable attitudes towards learning a L2
	6. 我为近期在二语学习上取得的进步感到骄傲	6. I feel proud because I have made progress in L2 learning
Negative emotion	1. 学习二语很累并且让人焦虑	1. It is exhausting and anxious to learn a L2
	2. 因为缺乏明确的二语学习目标，我感到沮丧	2. I become frustrated in L2 learning due to a lack of clear goals
	3. 我对二语学习没有信心	3. I feel diffident when learning the L2
	4.太多的测试和考试让我紧张	4. Too many tests and assessments make me feel stressful
	5. 我很沮丧因为看不到自己的进步	5. The feeling of making no progress makes me feel frustrated
Engagement	1. 我上课时仔细听讲积极参与	1. I am actively involved in the language class
	2. 我努力学习新的知识	2. I try hard to learn new knowledge in the class
	3. 我上课时注意力很集中	3. I stay focused in the classroom
Relationship	1. 当我学习二语的时候能获得老师的帮助	1. I receive help and support from my teachers when learn the L2
	2. 老师对我们二语学习持鼓励态度	2. The teacher is encouraging in our L2 learning process
	3. 老师自身的二语学习经历（留学）给我留下深刻印象	3. Teacher’s L2 learning history (overseas experience) leaves me a deep impression
	4. 因为我和老师关系很融洽，所以很有动力	4. The positive teacher-student interaction is motivating in learning the language
Meaning	1. 学习二语能给我在求职市场中带来优势	1. L2 skills offer a competitive edge in a global job market
	2. 学习二语能给我带来很多潜在的机会和可能性	2. Learning a L2 brings me more opportunities and possibilities
	3. 语言能力可以帮助我找到更好的工作	3. Language proficiency helps me land a better job
	4. 掌握一门二语很酷，可以帮助自己塑造一个好的形象	4. Mastering a L2 is cool and helps me make a good impression
Accomplishment	1. 我可以完成自己制定的有关二语学习的目标	1. I often achieve the important goals I have set for myself in L2 learning
	2. 我可以完成别人制定的有关学习二语的目标	2. I often achieve the important goals that others have set for me in L2 learning
	3. 我能在二语考试中取得好的成绩	3. I get good grades in my L2 proficiency exams
	4. 我能够完成我的二语学习任务	4. I am able to handle my L2 learning tasks
	5. 我在学习其他语言方面表现很好	5. I am proficient in another foreign language
	6. 我相信我有学习好二语的能力	6. I believe that I have the ability to master a L2