Effects of tool mediation on tertiary level EFL students’ reading comprehension and vocabulary learning skills: a case for a cloud computing environment

Abstract

Underpinned by constructivist learning theory (Feuerstein’s theory of mediated learning experience) and the cloud computing model, an experiment was conducted to examine the differences between the control and experimental groups regarding the linear combinations of tool mediation on students’ reading comprehension and vocabulary learning skills. The participants of the study were three intact classes of first-year students. A non-equivalent (pretest and posttest) comparison group quasi-experimental design and an explanatory sequential design type of mixed-methods design were employed. Questionnaires, tests, and diaries were used for the data collection procedures. Both quantitative and qualitative data analysis techniques were used. After several assumptions were checked, the quantitative analysis contained within mean, standard deviation, t-test, one-way MANOVA, Tukey HSD, Difference in Difference (DD), Propensity Score Matching (PSM), Average Treatment Effect (ATE), and Average Treatment Effect on the Treated (ATET). To explore students’ experiences and support the quantitative data, a latent content analysis was also used. The results from the quantitative data analyses indicated that there were statistically significant differences between the post-tests achievements of the study groups and the control group for reading comprehension and vocabulary learning skills (Wilk’s Λ = 0.7565, p < 0.05). After the post-mediation intervention, the multipartite treatment-effects estimation (DD, ATE, and ATET) also indicated that the experimental groups outdid the control group in overall post-mediation tests. The qualitative data results also showed that TM had a positive impact on students’ reading comprehension and vocabulary learning skills.

Keywords:

• Achievement
• glossing
• mediated learning experience
• moodle and cloud computing

REVIEWING EDITOR:

• Maria Popescu
• Strategic Communication
• Carol I National Defense University
• Bucharest
• Romania

SUBJECTS:

• Technology in Education
• Educational Technology
• Language Teaching & Learning

1. Introduction

Educational language instruction plays a crucial role in skilled staffing due to advancements in science and technology, enhancing global competition in the age of technological humanities. The rapid advancement of technologies has significantly influenced the changes in teaching reading and instructional methods in the English as a Foreign Language (EFL) context (Lin et al., 2017). In this thriving world of instructional technology and the increasing number of software apps and materials (Rashid & Asghar, 2016; Teng & Wang, 2021), educators and pupils are required to enhance their expertise in computers and make use of technological tools for learning and instruction to read (Ismail et al., 2012). Educators are necessitated to understand how such tools enhance their educational experience. Despite recent advancements in educational technology study, it still falls short of actual changes on the ground (Akram et al., 2022). However, there was splendid potential for utilizing technology or for extending readily available educational tools and procedures during the COVID-19 pandemic (Bozkurt & Sharma, 2020; Mengistie, 2020).

According to Lucas and Vicente (2023), for example, the epidemic that has spread all over the globe has culminated in an educational and instructional alter defined by an epidemic of unforeseen digitalization. Before the epidemic, literati suggested and stressed that it was time for shifting methods of instruction to incorporate instructional technology concepts (Huang, 2019). However, some vacillated about adapting their methods of instruction to the situation (Kerres, 2020). Consequently, during the outbreak, most educators overwhelmingly simulated their methods of teaching in colleges and universities using technological tools (Divjak et al., 2022; Kerres & Buchner, 2022).

Internationally, over the last two decades, technological advancements have begun to make their way into the educational framework for enhancing the learning experience (Binhomran & Altalhab, 2021) because students and instructors are increasingly engaging through digital devices and texts. This prompts questions about how the potential of apps or technologies (Reich et al., 2019) facilitates instruction and the learning of languages. Considering the plodding move to on-screen reading in English language learning and instruction, modern student reading patterns have also been profoundly shaped by the recent upsurge in electronic texts (Sidabutar et al., 2022). For example, the most current amalgamation of technology-based evaluations (online examinations) of comprehension of texts in lieu of worldwide or national evaluation of reading abilities (Mullis et al., 2017) indicates that students are now required to actively engage in and understand texts that are offered on digital platforms (Lenhard et al., 2017; Mullis et al., 2017). Therefore, it is vital to figure out whether and how this change impacts the ability to comprehend text (Halamish & Elbaz, 2020).

In Ethiopian contexts, numerous learners have been going through complications acquiring comprehension abilities that inhibit more thoroughly accomplishment in their perpetual educational achievement. The learners’ comprehension of texts also appears to be at a lower stage (Tsegaw, 2022), and it is worsening over time at the Ethiopian secondary and tertiary levels (Mesfin, 2008). This problem existed due to a lack of quality in learning or teaching, which could be an obstacle to understanding any text (Tsegaw, 2022). Tsegaw continued to say that, particularly in Ethiopian educational institutions, the challenge of mastering reading skills has become a means of obstruction from the primary to tertiary level, as educational quality has remained a concern. According to the Ethiopian Ministry of Education (2023), in the 2022 examination, 896,520 students took an exam in both natural and social science streams, with only 6.67% (29,909) passing and enrolling in university.

Likewise, at Bahir Dar University (BDU), students do not engage themselves in reading tasks since they have diminutive experience on campus. Consequently, this could be attributed to students’ low reading proficiency. As an illustration, when the researchers appraised first-year students’ mid- and final examination reading results, they scored below 42 out of 100. Similarly, 157 graduate students took reading proficiency tests for graduate assistant positions. Across each faculty, the students scored below 45 on average. The average score of all faculty students out of 100 was 39.44. This average result has not seen a better change yet. To present as evidence, the researchers collected and analyzed the mid- and final exams (a total of about 465 students’ papers) of the first-year communicative English language skills courses given at different times in 2021 and 2022. The mean score obtained was 41.42. Therefore, students’ reading scores have remained stable (below 42) in Ethiopian tertiary-level EFL contexts.

Accordingly, individuals had better find ways to improve their reading comprehension skills. Among these ways, a variety of educational settings have implemented information and communication technologies (ICTs) around the world to become skilled at language and other learning back issues (Buabeng-Andoh, 2012; Ulfa et al., 2022). ICTs provide advantages to English language students such as pupil autonomy (Frith, 2005), drive improvement (Fauzi, 2017; Schoepp & Erogul, 2001), and skill getting (Chen et al., 2018). Hence, incorporating newly developed technologies when instructing reading can benefit (Dumanig et al., 2011) both the instructor and students because technology turns out to be a tool for reflective learning, especially for those who have thought about and used it as a TM (Guerrero, 2007; Lin, 2021). Despite the fact that research in the arena of technologically-mediated education has clearly established in the past two decades (Chapelle, 2019), there show up to have been few research investigations into the challenges, issues, and potential remedies associated with tools (Liang, 2021).

Concerning this, an advanced expert who operates in a new technical advancement setting is most probably to have intricacy acquiring effectually, like a child with a disorder who has a problem acquiring somewhat fast (Feuerstein, 2000). Feuerstein continues to say that individuals like such children can be comparable to normal but culturally disadvantaged children (i.e., in this study context, the technological milieu) who have a mediational lack and acquire learning little by little defectively.

This kind of condition may also occur in a university student (Latva-Karjanmaa, 2001). This essential sight is built on Vygotsky’s (1978) and Feuerstein’s (1990, 1991) mediation theories, which are mutually reinforcing. Particularly, the theories of Feuerstein propound a wide-ranging theoretic backdrop to look at the two main scholars’ theoretic principles (mediated learning perspectives) for investigating mediation tools to examine students’ language skills on cloud computing tools during reading and vocabulary lessons. If Feuerstein theories are effectively implemented, they can be used to create an empowering learning environment using Cloud Computing Tools (CCTs) because it is unclear what constitutes an optimal reading environment (Rouet & Puustinen, 2009), i.e., a reading app and its features that display and integrate a digital reading task or text for students (Freund et al., 2016).

In an era where the internet is connecting everything, education confronts multifaceted opportunities and challenges. For instance, English language learning and teaching breach the edge of the classroom by extending educational activities without time constraints through technological models or cloud computing, which is emerging and impacting the teaching-learning process (Cheng, 2020; Constantinou, 2018). To apply the potential benefits of this, instructors ought to broaden the conventional classroom, backed by cloud computing and big data for instruction (Liu, 2021). Cloud computing is the use of computing resources (e.g., servers, storage, networks, services, and applications) that are delivered as a service (on-demand service) over a network. For this study, on-demand services refer to interactive teaching and learning methods (in reading and vocabulary tasks) using tool mediators that serve tertiary-level EFL students via the Internet from a specialized data center and that are not installed on users’ devices. As a result, in the context of this study, TM (CCTs) refers to the application of ICT tools in systematic language learning and teaching services (SaaS—Software as a Service) by fostering meaningful interaction in the classroom and creating a positive environment. With its dynamic scaling and use of virtualized resources in cloud computing language classrooms, CCTs are acknowledged as a new paradigm in computing resources.

Technology-mediated glossing has emerged as a new strategy to improve text comprehension and vocabulary learning for this reason (Yoshii, 2014). Recent research has focused on the effects of using electronic glosses in various modes and media to gloss individual words (Liu, 2009, 2021). Even if a few studies are a solid place to start when examining the usefulness of digital glosses in language learning, there is still a need for more empirical research on the effects of hypermedia glosses on vocabulary learning and reading comprehension skills (Bursali & Yilmaz, 2019; Park & Lee, 2021; Singer & Alexander, 2017; Wang, 2023). What is more, considering that mediation is a tool for learners’ affective and cognitive development (Zimmerman & Tsikalas, 2005), examining their experience of contentment (Bursali & Yilmaz, 2019) or reflection with mediational tools within web-based educational settings remains an important aspect of the present investigation. To put this into action, instructors may have significant personal experience with electronic resources, but they rarely utilize them when they are in the classroom (Burnett, 2011). Because the views of learners and educators have an important impact on implementing technologically mediated instruction (Melinis, 2011), higher education institutions should enhance their accessibility and utilization of technology-mediated lessons, as well as their teacher preparation (Parker, 1996), to make virtual instruction plainly apparent in higher education institutions (Alabdulakareem & Jamjoom, 2020).

Despite this, it is hardly enough to use technological tools as the sole mode of instruction at BDU, especially for the instruction of language skills. In passing, the researchers strongly argue that reading intervention studies are required to investigate the effectiveness of cloud computing and their particular features in the context of university-level EFL language learners and whether or not these tools have an effect on students’ language skills-reading and vocabulary learning skills. Thenceforth, the researchers conducted an experiment for 25 hours to overlook an effective technological aspect of mediation in BDU EFL learners’ context. For the purpose of the present research, the teacher provided the students of the experimental groups with adequate access to the Modular Object-Oriented Dynamic Learning Environment (Moodle) as a cloud-based tool both in and out of the classroom. Accordingly, with all the above insights, the researchers found it very alluring to investigate the subsequent research hypotheses:

H₀₁: There is no significant difference between the control and experimental groups regarding the linear combinations of TM on students’ reading comprehension and vocabulary learning achievement.

Ho₂: There is no significant difference between the control and experimental groups with reference to the univariate main effect of TM on students’ reading comprehension achievements.

Ho₃: There is no significant difference between the control and experimental groups with reference to the univariate main effect of TM on students’ vocabulary learning achievement.

Ha₁: There is a significant difference between the control and experimental groups regarding the linear combinations of TM on students’ reading comprehension and vocabulary learning achievement.

Ha₂: There is a significant difference between the control and experimental groups with reference to the univariate main effect of TM on students’ reading comprehension achievement.

Ha₃: There is a significant difference between the control and experimental groups with reference to the univariate main effect of TM on students’ vocabulary learning achievement.

2. Literature review

2.1. Mediation theories and technology

The focus of this study was what happens when humans interact with tools and one another through the mediation of digital devices, programs, tasks, and networks in the pursuit of reading skills. This essential view is built on Vygotsky’s (1978) and Feuerstein’s (1990, 1991) theories of mediation. Particularly, structural cognitive modifiability (SCM) and mediated learning experience (MLE) were used as comprehensive theoretical backgrounds in this study. Culturally deprived individuals are people who live in exceptional circumstances and who either have not been exposed to or have not been able to benefit from exposure to mediated learning experiences. This lack of cultural transmission or mediation results in rigidity and a low level of adaptability for the individual and the group. Through his observations, Feuerstein formulated the view that the role of cultural transmission and mediation is central to the formation of the propensity or ability to learn. Culturally deprived children had to "learn to learn" via mediation (Feüerstein et al., 1991, pp. 3–52) since it is used in deprived milieus. The role of culture in the form of human transmission is a central factor in the learning process.

Lack of cultural mediation can be caused by insufficient mediational tools. Feuerstein noticed that culturally deprived children showed significant gaps in their ability to learn. During his work, Feuerstein developed SCM and MLE theories. At the heart of Feuerstein’s Theory of SCM is the belief in the plasticity and modifiability of cognition. In brief, the concept of modifiability is of prime importance in SCM. Feuerstein (1990) argued that a person’s capacity to learn is not solely determined by one’s genetic endowment. Cognition can be improved or modified, irrespective of a person’s age and stage of development. In SCM theory, a child (or even an adolescent) who has cognitive deficiencies has every chance of positive change and development through mediation. In his SCM theory, Feuerstein (1990) argues that the human being can develop his or her cognitive capacities continuously. Mediation is the means for developing these capacities. The development of individual capacities takes place through mediated learning. The cornerstone of mediated learning is the MLE, which is described as "a breakthrough in the theory of cognitive development" (Feüerstein et al., 1991, pp. 3–19).

A MLE is the building block of SCM that human beings can learn to learn how to learn. The human is capable of modifying the underlying structure of his or her cognition. MLEs are products of a unique quality of interaction during which "the human organism is subject to the intervention of a mediator" (Feüerstein et al., 1991, pp. 3–19). Feuerstein (1990) views intelligence as a process, which means that intelligence is adaptive. The modifiability of individuals varies greatly, which Feuerstein (1990, p. 74) explains as "the consequence of mediated interactions that make mental schemata plastic and modifiable." To put it differently, the basic motivation of Feuerstein to fight the deprived learning situation can be applied to many human learning situations when studying the theory of MLE or when the learner is confronted with novel learning technology and virtual learning environments. Thus, the researcher tried to examine the CCTs that can be regarded in view of Feuerstein’s analysis of culturally different and culturally deprived individuals at BDU though mediational tools cannot be avoided in the learning situations of today.

2.2. Technology-mediated learning environment: a constructive perspective

Mediation is inherent in constructivism. From a constructivist viewpoint, reading is viewed as a social interactional practice and an active constructor of students’ learning environments to foster the development of students’ learning skills effectively. Mediation is the basic construction of learning that can be an act by a peer, teacher, or parent (Feuerstein, 1990) or a mediative tool, such as a toy or a text (Vygotsky, 1978). In a learning environment, the presence of symbolic mediators, humans, or tools present in interaction with the social context facilitates the development of the human being’s cognitive and metacognitive processes (Latva-Karjanmaa, 2015). Therefore, in the current study, the focus is on the various theories (constructivism) to conduct MLE in cloud-based learning environments. Because there is a great demand for an understanding of what kind of learning support or intervention is needed in web-based learning, particularly in EFL classrooms, it is necessary to investigate constructivist-learning environments in MLE to foster the development of students’ learning skills effectively.

Technology has changed the way we teach and learn. Many learning theories can be used to apply and integrate technology more effectively. There is a close relationship between technology and constructivism, with the implementation of each benefiting the other. Constructivism states that learning takes place in contexts, while technology refers to the designs and environments that engage learners. As Gilakjani et al. (2013) described, much of the early research and development with technologies considered the enhanced learning that could be achieved when computers played an important and key role in delivering content and creating learning opportunities to help students make meaning and develop understanding. Thus, recent efforts to integrate technology in the classroom have been within the context of a constructivist framework.

The more opportunistic and effective uses of technologies in classrooms are those where learning is achieved with the aid of technology, and the resulting environment is one where the technology supports and scaffolds the learning rather than being the object or derivative of the learning (Jonassen, 1991). In a constructivist-learning environment, technology plays a purposeful role in day-to-day activities, but it does not become the object of instruction. When tools are used in a constructivist manner, students use technologies to: a) manipulate data; b) explore relationships; c) intentionally and actively process information; d) construct personal and socially shared meaning; and e) reflect on the learning process (Jonassen et al., 1999).

This seeks to design contemporary pedagogical practices, or pedagogy. In addition, this study lets users (the targeted group) freely interact and actively construct their own knowledge and meaning from their experiences. This can be translated into the cloud medium infused with constructivist theory in classrooms that view learning as a process of construction so that there will be multiple constructions and perspectives, learning in contexts that are relevant to the learner, and learning mediated by tools or technology (Wood, 2010; Vygotsky, 1978).

3. Research method

3.1. Research design

A mixed-methods design was employed to investigate the impact of TM on tertiary-level EFL learners language learning– reading comprehension and vocabulary learning skills. This multi-methodological research was guided by a pragmatic worldview. Particularly, through the most common between-group designs (experimental type and quasi-experimental) in educational research, a pre-test-post-test comparison was conducted for this study. Participants were pretested prior to the commencement of the treatment, and a post-meditation test was given at the end of the treatment period. Furthermore, to avoid external and internal validity setbacks, all subjects were given the same pre- and post-tests and were taught the same content by the same teacher (Creswell, 2012). In other words, the researcher used two mutually conclusive research designs. The first one was between group pre-test-post-test quasi-experimental designs in order to explain the overall experimentation of the process of the study. The second one was an explanatory sequential design that aimed to explain the overall data collection and data interpretation process.

3.2. Research participants

This study’s participants were first-year students from pre-existing sections at BDU. Three groups were formed in this study. Based on the campus’s placement criteria (alphabetically or academically), the students were considered in the respective sections they were assigned by the program coordinators. Three groups were formed in this study. Students were assigned to a group that received the intervention or to a group that did not receive the intervention. Among these groups, two of them received different kinds of interventions in the multi-level treatment groups, whereas one group was involved in the comparison group. Thus, a sample of 69 treated students (Experimental Group A [EG-A] = 35, Experimental Group B [EG-B] = 34) were involved in two pre-existing classes for the experimental groups from the total population of students who enrolled in freshman programs. Similarly, 46 students were selected for the control group (CG), which has the same characteristics except for the treatment in the targeted population. Hence, an experiment was carried out to see the differences in students’ reading and vocabulary learning with and without TM between randomly selected experimental (N = 69) and control (N = 46) groups. Since quasi-experimental design takes intact classes into account, three sections of university-level first-year students were randomly chosen using lotto. Altogether, a total of 115 students partook in all three groups for investigational purposes. Simple random sampling (lotto) was employed to select the participants. In assigning the group, the first treatment group received TM (treatment) with L2 glossing for the reading section; the second treatment group received the treatment without glossing. That is to say, the two experimental groups (A and B) were divided into groups according to two different conditions: no gloss with CCTs and L2 glosses with CCTs. Notwithstanding, the comparison group received conventional instruction.

Students in experimental groups completed weekly reading tasks on Moodle sites with a user manual for practice. Grades were automatically or manually given, with progress reminders and response histories. The experimental group A was provided with glossing in reading tasks through a glossing activity module. A database of key terms was created, with individual entries linked to words appearing throughout reading tasks, allowing alphabetical browsing or keyword search. Students used hyperlinked words and glossaries to understand reading texts and engaged in vocabulary exercises like finding synonyms, matching, and filling gaps. In contrast, the control group accessed equivalent reading text and vocabulary in a module.

3.3. Instruments

3.3.1. Tests

Testing was used as one of the main tools to gather the data required for this study. In educational research, achievement tests are most commonly used. Among those types, criterion-referenced tests were administered to collect the data. A criterion-referenced test refers to a particular kind of test with an unconventional approach to the measurement of educational achievement. Based on this, two types of instruments (vocabulary and reading comprehension tests) were used to collect the data that were extracted and adapted from different leading English sources for higher education.

3.3.1.1. Procedures for conducting the reading comprehension tests

To administer the comprehension abilities of the students in the EG and the CG at the beginning of the experiment, all students were required to take part in the pre-test. The pre-test was provided to look at the similarity of the experiment’s participants, who were assigned to the three distinct groups. The reading comprehension tests (pretest and post-test) were designed based on the test construction schemes of the Council of Europe (2011), Brown and Abeywickrama (2010), and Association of Language Testers in Europe-ALTE (2005), along the lines of the educators’ comments. Thus, the content and the face validity of the reading tests were ensured before the tests were administered in the main study. In like manner, the post-mediation test was set in precisely the same way as the pre-test after the three sections were divided into treatment and comparison groups for the treatment. Hence, the pre-mediation and post-mediation tests were similar, with the equivalent accompanying reading texts, orderliness, number of itemized sections, instructions, allotted time, questions, value, and weighing up.

3.3.1.2. Procedures for conducting the vocabulary tests

Teacher-made vocabulary tests were given to determine students’ vocabulary learning achievement and to examine the effect of CCTs (L2 glossed and no-glossed texts) on students’ vocabulary learning achievement. In this study, 115 freshman students at BDU participated in the study before the intervention started. Then, a pre-test on vocabulary items was conducted to the sampled population. The words chosen for the pre-vocabulary test were from the module for the course Communicative English Language Skills I. A total of about 39 words were identified in this module. 20 of these words were divided into five sections (synonym, antonym, gap filling, matching, and meaning from context) and given as a pre-test to participants. After the experiment, everyone who took part in the condition of experimentation and the control group took posttests that followed the same pattern as the pre-mediation vocabulary test. The words selected for the post-vocabulary test were from the module for the course Communicative English Language Skills II, and 50 words were selected from all units for the test. Similarly, like the pre-vocabulary test (with the same organization), 20 of the 50 words were taken in the role of post-vocabulary tests. The post-test comprised of questions similar to the pre-test, not exactly alike. Afterward, the participants then took another unexpected vocabulary test a few weeks later. For both tests, the words were selected based on two considerations. The first was the degree of difficulty. The second consideration was the degree of importance.

Moreover, to determine whether students recalled the vocabulary (glossed items) from the experiment conducted four weeks later, the delayed post-test comprised a vocabulary recognition test. In order to look into effectual glossing for vocabulary recall or retention, three vocabulary tests (a pretest, an immediate and a delayed posttest) were administered. Four weeks later, the participants took a delayed vocabulary test.

3.3.2. Questionnaires

A post-mediation questionnaire was used for experimental groups (EGs) to find out more information about their experiences, and reflections in accordance with Moodle tools after the allotted or end sessions of the treatment or the immediate post-tests. On a five-point scale, post-meditation questionnaires were distributed and collected at the completion of all interventions. The students’ responses to these questionnaires were analyzed to explore the students’ experiences with the mediation tools: CCTs as a treatment for reading skills (15 items) and CCTs-glossing as a treatment for vocabulary learning skills (16 items). Before being distributed, the questionnaires were reviewed by experts. The internal consistency reliability coefficient was calculated for each section of the questionnaire’s items. The Cronbach alpha reliability coefficient for all items was acceptable (α = .83).

3.3.3. Validity and reliability

To eliminate or at least minimize threats to reliability and validity issues, it is important to think carefully through the design of data collection instruments. Thus, the content and face validity of reading and vocabulary tests were reviewed and assessed by three experts in the field. For both test items, eighteen close-ended (e.g., appropriateness, item variety, instructions, order, time) and five open-ended touchstones (e.g., content, relevance, practicality, task equivalence) were adapted from these English language testing sources and given to those three experts (Assistant Professors of TEFL and educational psychology at BDU) to rate and comment on the test items based on the set criteria. Concerning face validity, from the point of view or agreement of all three experts, the tests were appropriate and plausible to test the target skills. The researcher also gained useful insights from these experts, especially about the content validity (clarity of instructions, order of items, allotted time) in both tests. Based on the comments received, the researcher revised and made modifications before both tests were used in the main study. For example, the vocabulary test instruction IV (matching), the reading test instruction III (reference) order of items, and a distractor in reading instruction IV were revised. Based on the pre-experimentation processes, observations, and literati comments, the time allotted for the reading test improved from 70 minutes to 90 minutes, and the vocabulary tests were amended from 30 minutes to 40 minutes. The reading texts and vocabulary items were selected and prepared for first-year students based on Communicative English Language Skills I and II., The experts also ascertained that the items were designed based on the rules of exam preparation (relevance, format, and skills associated with reading comprehension.

Another serious design issue, reliability as test equivalence, was inspected by the researcher’s supervisors and experts. They ensured the comparability of tests through a careful evaluation of the equivalence between pre-tests and post-tests. Consequently, the EG and CG students had an equivalent set of test items on the pre-test and the remaining items on the post-test because a fixed group of items for all tests was established before instruments were administered for a trial run. A demanding pre-test with an easier post-test would make it more likely for enhanced performance to be apparent after a treatment, or the reverse scenario would make it more likely for no progress to be evident resulting from a treatment (Mackey & Gass, 2021). As touched on earlier, eighteen items were provided in order to look at the raters’ agreement on both reading and vocabulary tests. Before the researcher conducted a major study, the internal test reliability was computed to determine the inter-raters’ reliability of the instruments. Thus, all the observed alpha coefficients in the tests of inter-rater agreement responses or indices, being greater than.80, indicated that the instruments had an acceptable degree of agreement among raters.

Holistically, through piloting, the researcher checked the adequacy of the tests, the items, and the practical aspects of administering the data collection tools, including the time required to test and administer the instrument, the clarity of instructions, and inconsistencies. An attempt was made to identify potential threats to the internal (experimental procedures, treatments, or experiences of the participants) and external validity (the uniqueness of settings, situations, and the timing of the experiment) or reliability of the experiments. Inch by inch, some problems were identified and resolved because of the pilot study. Furthermore, lessons gained from the pilot study were considered to correct and improve the instruments for the final study. Finally, the refined instruments were employed to gather data for the main study.

3.3.4. Diary

A diary is one of the most research instruments and important ways of understanding and reflecting on the feelings, thoughts, and daily experiences of research participants. The diary method was applied to gather qualitative data pertaining to students’ feelings, thoughts, experiences, and so on in reading classes for the main study. In this regard, the students were given instructions to write about their own diary texts (lived experiences) or to keep on writing their own daily thoughts and reflections about the treatments after each treatment. That is, students in EG were informed to keep their daily records after they learned or practiced reading comprehension and vocabulary tasks. Based on this, students in the treatment groups were given training to record their learning progress during the mediation, particularly within CCTs. Also, some guiding or open-ended questions were provided for students to write down their essential experience after each session or treatment. Thus, the student diaries were brought together and examined after the end of the intervention for triangulation purposes.

3.4. Data analysis

Pertaining to vocabulary and reading tests, after several assumptions (univariate outliers and normality, multivariate outliers and normality, homogeneity of variance-covariance matrices, multicollinearity and singularity, and linearity) were checked, the pre-mediation and post-mediation univariate and multivariate tests (One-way-Multivariate Analysis of Variance [MANOVA]) were analyzed and compared among the control and experimental groups in terms of their means on the two dependent variables (reading comprehension and vocabulary achievement). In addition, to evaluate pairwise differences among group means, post hoc comparison and Tukey HSD were performed to determine where the significant differences lie. Stata 17 was used to analyze the quantitative data gathered from tests and questionnaires.

Turning now to this study as another investigative matter, to make causal claims based on a quasi-experimental investigation, the impacts of the initial group differences should be evaluated (Dornyei, 2007). It is vital to minimize pre-test discrepancies between the control and treatment groups as much as feasible in order to improve the quasi-experimental design. To achieve this, the researchers believes that DD and matching methods are efficacious methods for this study because they are powerful statistical tools when there is no randomized assignment (Gertler et al., 2016).

Thus, we can consider any differences (counterfactual change) between the control and experimental groups that remain stable throughout time using DD method (Gertler et al., 2016). The estimate of the counterfactual, according to Gertler et al., was simply determined as the difference between two differences (differences in outcomes for the comparison group as compared to the two counterfeit counterfactuals): $$\text{DD impact}=\left(\mathrm{B}‐\mathrm{A}\right)–\left(\mathrm{D}‐\mathrm{C}\right)$$

Additionally, the researchers used PSM with the characteristics of the gathered data. Following that, participants and non-participants were matched based on this probability, or propensity score. The mean difference in outcomes between the comparison and treatment groups was then calculated to ascertain the ATE of the treatment. Treatment D (TM) was a binary variable (D = 0 [CG observations] and D = 1 [EG observations]) that indicated whether an observation had received the treatment or not.

It is essential to discuss the two methodologies in order to discuss the PSM model. First, as previously mentioned, the observations were divided into two groups (the control and the treated group). Second, the researchers estimated a model for the propensity of observations to belong to the treatment group. The PS model was a probit model as a result. $$\mathrm{p}\left(\mathrm{x}\right)=\text{prob}\left(\text{D}=\text{1}|\mathrm{x}\right)=\text{E}\left(\mathrm{D}|\mathrm{x}\right)$$

On the basis of the propensity score, various methods were also used to match participants and nonparticipants. The researchers computed the TM effect following the matches (NN and Kernel Matching). The first step in this process was to compute the ATE (change between the findings of treatment and comparison observations). $$\Delta ={\mathrm{y}}_1-{\mathrm{y}}_0$$ $$\text{ ATE + E}(\Delta )=\text{E}\left({\mathrm{y}}_1|\mathrm{x},\text{D}=\text{1}\right)-\text{E}\left({\mathrm{y}}_0|\mathrm{x},\text{D}=0\right)$$

However, ATET was employed in experimental research for greater comparability. ATET is the difference between the outcomes of the treated observations and the outcomes of the treated observations if they had not been treated. Likewise, the researchers compared the outcomes of treatment and comparison observations after matching propensity scores. $$\text{ATET}=\text{E}(\Delta \left|\text{D}=\text{1})=\text{E}({\mathrm{y}}_1\right|\mathrm{x},\text{D}=\text{1})-\text{E}\left({\mathrm{y}}_0|\mathrm{x},\text{D}=\text{1}\right).$$ $$\text{ATET}=\text{E}(\Delta \left|\mathrm{p}\left(\mathrm{x}\right),\text{D}=\text{1})=\text{E}({\mathrm{y}}_1\right|\mathrm{p}\left(\mathrm{x}\right),\text{D}=\text{1})-\text{E}\left({\mathrm{y}}_0|\mathrm{p}\left(\mathrm{x}\right),\text{D}=0\right)$$

At last, the quantitative data collected from questionnaires were analyzed through descriptive (M, SD) and inferential statistics (t-test).

4. Results

4.1. Pre-mediation multivariate tests

Table 1 includes the summary statistics for the dependent variables (pre-mediation vocabulary and reading scores) disaggregated by the independent variable, TM, as well as the overall score. As can be seen from the vocabulary result, EG-A's mean score was 50.29 (SD = 19.99), while EG-B and CG-C's mean scores were 49.85 (SD = 19.40) and 45.21 (SD = 20.10), respectively. In addition, the variation of the pre-mediation vocabulary score was similar (SD = 19.99, 19.40, and 20.10) in each group of the independent variable. As the table shows, in the pre-mediation reading scores, the mean scores of EG-A and CG-C were 33.8 and 33.47, respectively. These results were quite similar in magnitude though the EG-B pre-mediation reading score was 3 points higher than groups A and C. Looking at their overall average score, the vocabulary score of all 3 groups was 48.13 (SD = 19.84), and their reading score was 34.53 (SD = 10.32).

Table 1. Descriptive statistics for pre-mediation: N, M, SD by TM (1 = EG-A, 2 = EG-B, 3 = CG-C).

Tool-mediation	Pre-mediation vocabulary scores	Pre-mediation reading scores
1	35	35
	50.29	33.8
	19.99	11.37
2	34	34
	49.85	36.70
	19.40	9.97
3	46	46
	45.21	33.47
	20.10	9.70
Total	115	115
	48.13	34.53
	19.84	10.32

Table 2 contained the results of the four pre-mediation multivariate tests (P, W, H, and R) performed to test for the statistically significant differences of the variations among the TM groups on the linear combination of the two dependent variables: pre-mediation vocabulary and reading comprehension achievements. Results of the multivariate F(Λ) yielded that no statistically significant difference existed between the three groups, experimental group A (treatment with L2 gloss), experimental group B (treatment with no gloss), and control group C (conventional), based on the dependent variables: Wilks’ Λ = 0.9693, F(4, 222) = 0.87, p = 0.4815.

Table 2. Pre-mediation multivariate tests.

	obs = 115
Source	Stat	df	F(df1,	df2) =	F	Prob > F
Tool_Mediation	W 0.9693	2	4.0	222.0	0.87	0.4815 e
	P 0.0309		4.0	224.0	0.88	0.4771 a
	L 0.0314		4.0	220.0	0.86	0.4859 a
	R 0.0204		2.0	112.0	1.14	0.3221 u
Residual		112
Total		114

W = Wilks’ lambda, L = Hotelling trace, R = Roy’s largest root, P = Pillai’s trace.

4.2. Pre-mediation univariate tests

To examine the TM groups separately for significant differences across the dependent variables, univariate tests were checked from the output of TBSEs. The TBSEs table indicated (see Table 3) that there were nonsignificant main effects of TM on pre-mediation vocabulary (F(2, 112) = 0.828, p = 0.440) and reading tests (F(2, 112) = 1.084, p = 0.342). These results showed that both dependent variables were similar before the study participants started the treatment.

Table 3. Pre-mediation univariate tests.

Tests of between-subjects effects (TBSEs)
Src	Dep. var	Sum of squar. (Type III)	df	Mean square	F	Sig
Tool_Mediation	Pre-mediation vocabulary score	653.810	2	326.905	.828	.440
	Pre-mediation reading score	230.506	2	115.253	1.084	.342
a. R Squ = .015 (Adj. R Squared = −.003)
b. R Squ = .019 (Adj. R Squared = .001)

4.3. Post-mediation multivariate tests

The group statistics displayed in Table 4 showed that the post-mediation vocabulary mean score for experimental group A was 65.00 (SD = 13.67). For control students (group C), students scored 46.20 (SD = 14.50), whereas the treatment group (group B) was 54.11 (SD = 19.34). Regarding post-mediation reading scores, experimental group A scored (M = 43.09) higher than the average score of the remaining groups, and the average score of the experimental group B students (M = 41.41) was comparable to the average score of group A (M = 43.09). In contrast, the mean score of comparison group C (M = 34.87) was 7 to 9 points lower than the mean score of groups A and B. As it can be seen from the table (see Table 4), the overall post-mediation vocabulary and reading mean scores of the groups were 54.35 (SD = 17.57) and 39.30 (SD = 12.83), respectively.

Table 4. Descriptive statistics for post-mediation: N, M, SD by TM (1 = EG-A, 2 = EG-B, 3 = CG-C).

Tool-mediation	Post-mediation vocabulary scores	Post-mediation reading scores
1	35	35
	65.00	43.09
	13.67	13.61
2	34	34
	54.11	41.41
	19.34	12.52
3	46	46
	46.20	34.87
	14.50	11.27
Total	115	115
	54.35	39.30
	17.57	12.83

A one-way MANOVA was used to examine if there were any changes in the post-mediation reading comprehension and vocabulary tests across the three TM groups, as shown in Table 5. The multivariate F results showed a significant difference with Wilk’s Λ = 0.7565, F (4, 222) = 8.31, p = .0000, and multivariate η² = 0.123; Thus, on both dependent variables, a statistically significant difference existed between the comparison and the treatment groups.

Table 5. Post-mediation multivariate tests.

	obs = 115
Source	Stat	df	F(df1,	df2) =	F	Prob > F	Eta-squared
Tool_Mediation	W 0.7565	2	4.0	222.0	8.31	0.0000 e	0.123
	P 0.2458		4.0	224.0	7.85	0.0000 a	0.130
	L 0.3189		4.0	220.0	8.77	0.0000 a	0.138
	R 0.3090		2.0	112.0	17.31	0.0000 u	0.236
Residual		112
Total		114

The immediate vocabulary post-mediation mean of 65.00 (see Table 4) decreased to 62.16 (see Table 6) at the delayed vocabulary posttest, indicating that there was still substantial variation between the two groups (Wilk’s = 0.7739, F (2, 59) = 8.62, p = .0005). The L2 gloss group (experimental group A) continued to outperform the no gloss group though both groups scored less than in the immediate vocabulary posttest, with a mean score of 62.16 and 49.17 in turn.

Table 6. Post-mediation multivariate tests (delayed vocabulary post-test).

TM	N	M	Stat	df	F(df1,	df2) =	F	Prob > F
L2 gloss No gloss	32	62.16	W 0.7739	1	2.0	59.0	8.62	0.0005
	30	49.17

4.4. Post-mediation univariate tests

Since significant results were obtained on this multivariate test of significance, univariate tests allowed further investigation in relation to each of the dependent variables. Using the Bonferroni method, each follow-up univariate test was tested at a .025(.05/2) alpha level. The tests of between-subject effects indicated that there were significant main effects of TM on both dependent variables, with medium to large effect sizes singly. For example, the effect of TM on post-mediation vocabulary achievements was large (η² = 0.200), and the effect of TM on students’ post-mediation reading scores was medium (η² = 0.083).

4.5. Post-mediation multiple comparison tests

As noted earlier about the post-mediation multivariate tests (see Table 5) and univariate tests (see Table 7), a significant main effect indicated that all of the group differences as well as each of the dependent variables were significant enough to be taken into account statistically. Additionally, it should be noted that not all group differences may be significant if the main effect is significant (Hair et al., 2013). To ascertain where the significant differences lie (i.e., which specific level of the independent variable significantly differs from another in each group), Tukey HSD post hoc comparisons were used. Table 8 showed the post hoc multiple comparisons or contrasts method (Tukey HSD) that was applied to both dependent variables across the three levels of TM. When the researchers examined the post-mediation vocabulary score, it can be seen that the two groups main effects were significant (CG-C versus EG-A and EG-B versus EG-A) even though the differences between these neighboring groups were not constant. The difference between those in group 2 and those in group 1 was -10.588. Likewise, group 3 had the lower value (-18.804), analogized to group 1. When the researchers inspected the group difference between groups 3 and 2, however, the difference was reduced to -8.216, which indicated that the difference was statistically nonsignificant (p = 0.061). When the researchers observed the post hoc comparisons for the post-mediation reading score, the differences between the first and last two groups (CG-C versus EG-B and EG-B versus EG-A) tests indicated that the two groups were not different in their mean scores. In contrast, The following Tukey post hoc test comparison (CG-C versus EG-A) revealed considerable pairwise mean differences between the two groups (p = 0.011).

Table 7. Post-mediation univariate tests (TBSEs).

TBSEs
Source	Dep. var	Sum of squa. (Type III)	df	M Squ	F	Sig	Partial eta squared
Tool_Mediation	Post-mediation vocabulary score	7028.613	2	3514.306	13.984	.000	.200
	Post-mediation reading score	1556.152	2	778.076	5.065	.008	.083
a. R Squ. = .200 (Adj. R Squared = .186).
b. R Squ. = .083 (Adj. R Squared = .067).

Table 8. Post hoc pairwise comparisons of marginal linear predictions for individual group differences on post-mediation vocabulary and reading means across groups of TM.

			Tukey		Tukey
	Contrast	Std. err	t	P>|t|	[95% conf	interval]
Post-mediation vocabulary score Tool_Mediation
2 vs 1	−10.588	3.817	−2.770	0.018	−19.655	−1.521
3 vs 1	−18.804	3.556	−5.290	0.000	−27.250	−10.358
3 vs 2	−8.216	3.585	−2.290	0.061	−16.732	0.300
Post-mediation reading score
2 vs 1	−1.674	2.984	−0.560	0.841	−8.763	5.415
3 vs 1	−8.216	2.780	−2.960	0.011	−14.819	−1.613
3 vs 2	−6.542	2.803	−2.330	0.055	−13.200	0.116

Group 1 = EG-A, Group 2 = EG-B, Group 3 = CG-C.

4.6. Post-mediation mean difference outcomes

Table 9 showed the mean outcome for each group as well as the group’s single difference to answer the difference between results in the treatment group with treatment versus without treatment. By way of explanation, what would the treatment group’s outcome be post-treatment if treatment had not occurred? As Table 9 indicated, the outcomes of group 2 (EG-B) versus group 1 (EG-A) were computed for the treatment group before and after the mediation (pre- and post-mediation vocabulary tests). In all groups after the intervention, the statistical inference results (among each of the three groups) showed significance, but the DD treatment effects were different. The impact estimate (DD) for Group 2 versus 1 was 10.155, with a non-significant difference (p > 0.05). In the same way, the DD treatment effect for group 3 contrasted with group 2 was 3.581, which indicated statistically nonsignificant (p = 0.542). However, for experimental group 1 and comparison group 3, the calculated outcome of the difference in the outcome (Diff (T-C)) for these groups was 13.736 (p < 0.05).

Table 9. Difference-in-differences (DD) estimation results: post-mediation vocabulary tests.

Outcome var (vocabulary)
		Group		Diff (T-C)	S. Err	|t|	P>|t|
		2	1
EG-B vs EG-A	Before	49.853	50.286	0.433	4.398	0.10	0.922
	After	54.412	65.000	10.588	4.398	2.41	0.017**
	Diff-in-diff	10.155			6.220	1.63	0.105
		3	1
CG-C vs EG-A	Before	45.217	50.286	5.068	3.893	1.30	0.195
	After	46.196	65.000	18.804	3.893	4.83	0.000***
	Diff-in-diff			13.736	5.506	2.49	0.014**
		3	2
CG-C vs EG-B	Before	45.217	49.853	4.636	4.146	1.12	0.265
	After	46.196	54.412	8.216	4.146	1.98	0.049**
	Diff-in-diff	3.581			5.863	0.61	0.542

** Inference: ***p < 0.01; **p < 0.05; *p < 0.1.

Table 10 disentangled the components of the DD estimates for post-mediation reading tests. Between the comparison of all three groups (CG-C vs. EG-B, CG-C vs. EG-A, and EG-B vs. EG-A), the result of the reading test outcome means (Diff-in-Diff) showed 4.580, 7.894, and 3.315 for each combined group, respectively. Apart from this, except for Group 3 vs. 1 (p = 0.031), the other p-values are nonsignificant.

Table 10. Difference-in-differences estimation results: post-mediation reading tests.

Outcome var (reading)
		Group		Diff (T-C)	S. Err	|t|	P>|t|
		2	1
Group 2 vs 1	Before	36.706	33.800	−2.906	2.879	−1.01	0.315
	After	41.412	43.086	1.674	2.879	0.58	0.562
	Diff-in-diff	4.580			4.072	1.12	0.263
		3	1
Group 3 vs 1	Before	33.478	33.800	0.322	2.564	0.13	0.900
	After	34.870	43.086	8.216	2.564	3.20	0.002***
	Diff-in-diff	7.894			3.626	2.18	0.031**
		3	2
Group 3 vs 2	Before	33.478	36.706	3.228	2.457	1.31	0.191
	After	34.870	41.412	6.542	2.457	2.66	0.009***
	Diff-in-diff	3.315			3.475	0.95	0.342

**Inference: ***p < 0.01; **p < 0.05; *p < 0.1.

4.7. Post-mediation average and multifaceted treatment effects Estimation

As can be seen in Table 11, the difference between the experimental groups (A and B) after matching (actual nearest neighbor matches) was 9.71. Table 11 also surmised what the experimental groups’ post-treatment outcomes would be if treatment had not occurred. After matching control (Group C) and treated participants, the effects of the TM on students’ vocabulary results were higher by 6.914 (p = .012) using the post-mediation data. In the last part of Table 11, although the estimated ATE between experimental group B and group C was 2.5, it did not show a significant difference (p = .386).

Table 11. Average treatment-effects estimation (post-mediation vocabulary tests).

VocabularyScore		Coef	St.Err	t-value	p-value	[95% Conf	Interval]
Treatment (A vs B)	ATE (nn)	9.71	3.117	3.11	.002***	3.6	15.82
	ATET	9.71	3.124	3.11	.002***	3.587	15.834
Treatment (A vs C)	ATE (nn)	6.914	2.763	2.50	.012**	1.499	12.329
	ATET	6.914	2.795	2.47	.013**	1.436	12.392
Treatment (B vs C)	ATE (nn)	2.5	2.884	0.87	.386	−3.152	8.152
	ATET	2.5	2.868	0.87	.383	−3.121	8.121

nn: nearest neighbor. **Inference: ***p < 0.01; **p < 0.05; *p < 0.1.

Table 12 indicated that the students will take 7.029 more scores when they are involved in a treatment. Thus, students who are involved in the treated population will take 4.802 more scores than they would if they did not take the treatment in reading sessions at tertiary levels. On the other hand, the estimated ATE and ATET of reading scores on the controlled (Group C) and treated (Group B) groups were 2.8, with non-significant differences (p > 0.05).

Table 12. Average treatment-effects estimation (post-mediation reading tests).

ReadingScore		Coef	St.Err	t	p	[95% Conf	Interval]
Treatment (A vs B)	ATE (nn)	7.029	2.03	3.46	.001***	3.05	11.008
	ATET	7.029	2.025	3.47	.001***	3.061	10.997
Treatment (A vs C)	ATE (nn)	4.802	1.812	2.65	.008***	1.251	8.354
	ATET	4.802	1.826	2.63	.009***	1.223	8.382
Treatment (B vs C)	ATE (nn)	2.8	1.712	1.64	.102	−.555	6.155
	ATET	2.8	1.705	1.64	.101	−.542	6.142

Inference: ***p < 0.01; **p < 0.05; *p < 0.1.

Table 13 showed the estimated average difference in the potential outcomes among the treatment levels (ATET), 7.71 (CI [–.214, 15.626], p = .057) and 18.36 (CI [11.807, 24.919], p = .000). Experimental group A (treatment level 2) had an 18.4 score increase in students’ vocabulary achievements relative to if they had not been involved in the treatment program. If the control group C (treatment level 0) were engaged in treatments 1 and 2 instead, the outcome of the students’ mean score would increase by an average of 7.71 and 18.4, respectively.

Table 13. Multifaceted treatment-effects estimation (post-mediation vocabulary tests).

Vocabulary score ABC	Coef	St.Err	t-value	p-value	[95% Conf	Interval]
ATET
Treat_Level
(1 vs 0)	7.706	4.041	1.91	.057	−.214	15.626
(2 vs 0)	18.363	3.345	5.49	.000***	11.807	24.919
POmean
Treat_Level 0
	46.706	2.426	19.25	.000***	41.951	51.46

***p < .01; Treat_Level: 2 = tool_L2gloss, 1 = tool_Nogloss, 0 = No_tool_gloss.

The output revealed that the estimated mean reading score increased as the treatment level went from 0 (control group) to 1 (group B) and to 2 (group A). Likewise, the estimated potential outcome mean (POM) of the control level of group C was 35.23. The estimated ATET of going from level 0 to level 1 treatment was 6.15, and the estimated ATET of going from level 0 to level 2 was 8.26. So overall, all these effects were highly significant at each treatment level (Table 14).

Table 14. Multifaceted treatment-effects estimation (post-mediation reading tests).

Reading score ABC	Coef	St.Err	t	p	[95% Conf	Interval]	Sig
ATET
Treat_Level
(1 vs 0)	6.152	2.879	2.14	.033**	.51	11.795	**
(2 vs 0)	8.264	3.756	2.20	.028**	.903	15.625	**
POmean
Treat_Level 0	35.259	1.987	17.74	.000***	31.365	39.154	***

***p < .01, **p < .05; Treat_Level: 2 = tool_L2gloss, 1 = tool_Nogloss, 0 = No_tool_gloss.

As shown in Table 15, there is a significant difference between the participants’ response mean score (4.63) and the test value (3) at the stated P < .05 level, t(66) = 59.12, P < .05, d = .226, 95% CI [4.57, 4.69].

Table 15. Mean test for students’ response to mediation tools (moodle tools as a treatment for reading skills).

One-sample t test
Variable	Obs mean	Std. err. Std. dev	[95% conf. interval]
ToolRV	67 4.63	.026 .226	4.57 4.69
Mean (Mediation_Tools_Reading)			t = 59.12
H0: mean = 3		df = 66
		Ha: mean != 3
		Pr(T > t) = 0.0000

As shown in Table 16, there is a difference of statistical significance between the expected mean (3) and the students’ observed average score (4.38) at the specified P < .05 level, t(34) = 24.19, P < .05, d = .338, 95% CI [4.26, 4.50].

Table 16. Mean test for students’ response to mediation tools (glossing as a treatment for vocabulary learning).

One-sample t-test
Variable	Obs mean	Std. err. Std. dev	[95% conf. interval]
ToolPe ∼ n	35 4.38	.057 .338	4.26 4.50
Mean = mean (Mediation_Tools_Vocabs)			t = 24.19
H0: mean = 3		df = 34
		Ha: mean! = 3
		Pr(T > t) = 0.0000

5. Discussions

To answer the first, second, and third research hypotheses, a one-way MANOVA was used to see if there was any substantial variation between the experimental and control observations regarding the effects of linear combinations of TM on students’ reading comprehension and vocabulary learning achievement. Two dependent variables were used: reading comprehension and vocabulary learning achievement. The independent variable was TM, with three levels of treatment (tool_gloss, tool_no gloss, and no tool_gloss). Preliminary assumption testing was performed on variance-covariance matrices to check for univariate outliers and normality, multivariate outliers and normality, linearity, multicollinearity, singularity, and homogeneity. The results of the evaluation of these assumptions were satisfactory, with no serious violations noted. Significant multivariate effects were found for the main effects of TM on the two dependent variables with the use of Wilk’s criterion: F (4, 222) = 8.31, p = .0000; Λ = 0.7565; multivariate η² = 0.123 (see Table 5).

When the results for the dependent variables were examined separately, they revealed a substantial difference in post-mediation vocabulary and reading comprehension achievements, F(2, 112) = 13.984, p = 0.000, partial η² = 0.200, and F(2, 112) = 5.065, p = 0.008, partial η² = 0.083, respectively (see Table 7). The strength of the relationship between TM and post-mediation vocabulary score was strong, with the type of group accounting for 20.0% of the variance of the dependent variable. Also, the effect sizes, η² = 0.200 and η² = 0.083, indicated that students enrolled in the TM had significantly higher learning vocabulary and reading skills than students enrolled in conventional lessons.

The results of the delayed vocabulary test showed that the students of the glossed group still had similar scores (p = 0.0005) on the vocabulary test given after one month as they did immediately after treatment (see Tables 4 and 6) although the L2 gloss group showed an average decrease of 2.84 and the no gloss group 4.84 during the delayed test.

To examine pairwise differences among the three group means, post hoc comparisons (Tukey’s HSD) were conducted. Examination of the post hoc comparisons revealed that the significant change for the first outcome variable (vocabulary scores) was non-significant only for group 3 contrasted with group 2, p = 0.061. Upon inspection of the second outcome variable (reading scores), only the results of post hoc comparisons between group 3 (control group C) and group 1 (experimental group A) indicated a significant difference at a p value of 0.011 (see Table 8).

Additionally, an inspection of the mean difference outcomes (Diff-in-diff, Diff (T-C)) indicated that group 3 (M = 13.736) and group 2 (M = 10.155) reported considerably higher levels of mean scores (see Table 9) contrasted with group 1 (A). In like manner, the DD treatment effects confirmed that the treatment improved the treated group’s (A) vocabulary scores (Diff (T-C) = 13.736, p = 0.014) (see Table 9) and reading scores (Diff (T-C) = 7.894, p = 0.031) (see Table 10). On the contrary, the DD estimation results revealed that students participating in a TM class without glossing (groups B and C) did not significantly differ on either dependent variable compared to students involved in conventional reading or vocabulary lessons.

Tables 10 and 11 show that the results of ATE and ATET are similar; this shows that both ATE and ATET coincide when they meet the assumptions of multicollinearity and normality. As indicated in Table 11 above, when group A was contrasted with groups B and C, the average treatment effects estimation of 9.71 and 6.91 was the effect that showed someone involved in the treatment (TM) would increase with 9.71 and 6.91, at a p-value of.002 and.012, respectively. This means that students who received the TM scored more than those who did not receive the glossing (see Table 11).

Using two sets of data for the outcome variable (the difference between after and before treatments), the results show that students (Group A) who received tools with glossing increased their reading scores by 4.802 to 7.029, with p values of.008 and.001, respectively, compared to other groups (see Table 12). The multipartite treatment-effects estimation (ATE and ATET) also confirmed that the students (experimental group A) improved on their pre-mediation vocabulary and reading scores by 18.4 (see Table 13) and 8.3 (see Table 14), respectively. This pointed to the fact that the receiving treatment (TM) increased the outcome of experimental group A by an average reading score of 8.3 relative to receiving conventional reading lessons.

These findings are consistent with prior study findings by Al-Bukhari and Dewey (2023), Azizi et al. (2022), Gürkan, (2019), Holubnycha et al. (2019), Ko (2012), Pratiwi and Ubaedillah (2021), and Savuran and Elibol (2015). All these scholars’ analyses showed that the TM or glossed conditions effectively improved students’ (students in the experimental group’s) learning vocabulary achievement. For instance, Holubnycha et al. (2019) examined the effect of cloud computing on vocabulary development for university students learning foreign languages, and their findings showed that the experimental group’s pre-test and final test results demonstrated the effectiveness of cloud computing for this purpose. Additionally, they pointed out that using cloud computing in the classroom today can be seen as a useful teaching tool that fosters vocabulary growth, raises students’ interest in and motivation for learning, and creates new opportunities for both students and teachers.

In terms of studies, Gürkan, (2019) examined whether the use of annotations affected students’ vocabulary recall and retention. The findings suggested that students who used multimedia annotations did a better job of remembering and retaining vocabulary than students who used paper-based annotations or who received no instruction at all. An increasing body of pertinent studies has also shown that glossing provides students with numerous crucial additional cues for understanding vocabularies and texts; as a result, applying glosses is a helpful technique for increasing vocabulary learning and word retention (Chen, 2016; Ghahari & Heidarolad, 2015; Khezrlou et al., 2017; Rassaei, 2018; Varol & Erçetin, 2016; Yoshii, 2014). Reviewing the current study’s findings reveals that vocabulary item processing may have boosted learners’ vocabulary learning and achievement. This is because different modalities of the glosses resulted in improved word processing for the glossing words.

When taking into account the reading comprehension results, graded-reading websites or applications gradually improved the experimental groups’ reading comprehension in short time. (Alghizzi & Elyas, 2022). Alghizzi and Elyas continued by saying that the type of specialized technology and the applied learning environments limit the impact of technology on undergraduates’ reading comprehension. Similar to that, as confirmed by the results of the study of Miranda et al. (2023), the beneficial connection between ICT mediation and motivational reading processes, including textual relationships linked to text interpretation, highlights the value of digital educational resources in comprehensive processes. Reading comprehension should therefore be incorporated into online learning. (Akbari et al., 2021) because students can benefit from annotation and highlighted features, which allow them to readily comprehend and organize information from readings when dealing with reading activities during online learning (Azmuddin et al., 2020). Particularly, the instructional computational resources imply that the 21st-century education system needs to adopt constructivist classrooms (Misra & Misra 2020), focusing on active learning and new teaching approaches. Research suggests constructivist approaches (5E learning model, experiential learning, collaborative learning using collaborative tools, etc.) improve academic achievement and teaching effectiveness, so teachers should understand these concepts and play an active role in implementing these approaches (Misra, 2019; Misra & Misra 2020; Sharma, 2006).

Then again, since students’ reactions to an application of Moodle tools can greatly impact their learning experience (Chung & Ackerman, 2015), the TM students’ responses were analyzed as well. The students’ responses (as it is made plain in terms of Moodle tools as a treatment for prolonged and repeated practices of vocabulary learning and reading comprehension skills) to the implementation of TM for both skills were received auspiciously. In light of this, the following excerpts of students’ reflections in their diaries also substantiated the effectiveness of TM (Moodle tools) for reading comprehension and vocabulary learning skills.

SD (Student Diary 14: (…) these reading tasks are very good for improving our language skills. In my opinion, this method of teaching is the leading method. (…) it gives us a chance to practice more, to increase our performance, and to enhance the habit of practicing things like these tasks in the future.

SD2: (…) this plat form is very good for us students in terms of time and convenience; this plat form made me practice reading vocabulary and references.

SD30: (…) To finalize, I had a great experience and learned lifelong lessons regarding how to answer reading questions.

SD32: I will recommend that this learning system be implemented for the next students because it is easier to learn and increases students’ communication with technology. It is the best experience for an online learning system.

SD13: Today, I was so happy because I scored a good grade on an online exam. I have significantly improved in my reading and learning; I have noticed this improvement several times as I worked (…).

SD3: (…) the online teaching helps me learn how to read the passage and a lot of vocabulary meanings.

SD31: (…) I know the meaning of words that I did not know before, and help me improve my English skills (…) Overall, the passage helped me improve my vocabulary and reference skills.

SD22: Today was my second day to attend an online class. I get some vocabulary words collected from the reading passage with their meanings. I tried to memorize(…)

SD9: I was practicing new words and increasing my vocabulary knowledge related to the passage (…) it is very important to me because it will not be forgotten when I practice and train myself daily.

SD38: (…)This helped me a lot to understand that and gave me a good experience. Even if it was difficult, it was enjoyable. I also improved my reading and vocabulary skills through this program.

All in all, students feel that cloud computing tool (Moodle) is useful in an overall way (perceived usefulness). This result is aligned with several studies (Chung & Ackerman, 2015; Gudkova et al., 2021; Truong, 2021).

6. Conclusions

After data were gathered and analyzed thoroughly from the reading and vocabulary achievement tests, the results obtained from the pre-mediation univariate and multivariate tests showed that the three groups (CG-C, EG-A, and EG-B) were similar in their mean scores of the pre-mediation reading and vocabulary achievement tests. The quantitative results, multivariate F(Λ), indicated that there was no significant difference among the three groups before the study participants started the treatment. Likewise, the pre-mediation univariate test results revealed that the univariate differences among the three groups had statistically non-significant main effects on students’ vocabulary and reading achievements. Thus, based on the findings of the pre-mediation multivariate and univariate tests, it can be concluded that there was no significant difference in the means of the three groups’ pre-mediation reading and vocabulary scores.

Following the pretest posttest comparisons, post-tests were given to all groups, and the multivariate and univariate tests, post-hoc pairwise comparisons, DD, ATE, and ATET estimation were used for statistical analysis of the post-test results. Accordingly, the presence of the TM treatments caused statistically significant differences in the post-test results among the three groups, as demonstrated by the multivariate F results and the univariate tests. The post-mediation post-hoc pairwise comparisons strengthened the conclusion that the use of the treatment (TM) improved students’ reading and vocabulary achievement across each group, but the significant change for vocabulary achievement was non-significant only for group 3 contrasted with group 2.

The DD treatment effects also firmed up that the TM improved students’ vocabulary and reading scores. Overall, the Tukey’s HSD and the DD treatment effects demonstrated that there was adequate evidence to reject the post-mediation reading null hypothesis (Ho₂) and the vocabulary null hypothesis (Ho₃). In addition, the results of the multivalued or average treatment-effects estimation (ATE and ATET) indicated that if students learned reading comprehension and vocabulary learning sessions using TM (cloud-based tools), they would get much better results. Also, it can be concluded from the delayed vocabulary results that student learning by combining reading with glossed conditions had a significant contribution in students’ incidental vocabulary learning; the delayed vocabulary test showed glossing for efficacious vocabulary recall to language use in tertiary-level EFL freshman classrooms. Based on these results, evidence was sufficient to reject the null hypotheses (Ho₁₎ regarding the linear combinations or students’ univariate main effect of TM on students’ reading comprehension and vocabulary learning achievement.

The responses made by the experimental group students to the questionnaires (students’ reactions to Moodle tools as treatment) in general showed that the treatment (TM) had a positive impact on university-level freshman students’ vocabulary learning and reading comprehension skills. Thus, based on these results, it can be concluded that the treatment (TM) had the most likely effect on students’ language skills.

Finally, the digital spurt has brought the transition of reading skills from in-print reading to digital reading. If this transition (digital interaction) continues at its current pace, there might be a generational shift in the benefits of digital reading because the posterity has grown up with the more and more prevalent human-computer interaction. Based on this concern, the intimidating tasks that educators face to make the digital learning and teaching experience better will be constricted if students are assembled and set up in their real learning environment. Thus, for English language teachers and students, real training, retraining, and grooming sessions should be provided at the university or faculty level on how to integrate English language reading and vocabulary skills with the regular curricula through CCTs. For teachers to work effectively around CCT or educational software technology, a university-level (chair-based information technology) team should also be set up to give practicing exercises, assignments, tests, and instructions, individually or in groups, and be implemented differently from the conventional classroom. For example, for this model, the researcher has developed a new application called BDU_vocab_app for targeted word lists of Communicative English Language Skills courses (vocabulary learning skills), and this app can be used by students at least. Additionally, constructivist virtual learning environments are underutilized in academic institutions, requiring more effective teaching methods. Teachers should be encouraged to use the social construction of knowledge, focusing on discussion, collaboration, negotiation, and shared meanings in technology-mediated learning. Educational institutions should organize and create contextual professional development programs, design instructional strategies, plan curriculum materials, and redesign pre-service and in-service programs to accommodate constructivist approaches in virtual learning environments. At last, since this study was conducted at the level of a higher educational institution, the results of this study did not take into account primary and secondary schools according to their grade levels.

Data availability statements

Due to privacy issues, the datasets obtained and/or processed during the present research are not accessible to the public, but can be accessed from the corresponding author upon a justifiable request.

Disclosure statement

The authors declare no competing interests

Additional information

Notes on contributors

Sisay Ayalew Tsegaw

Sisay Ayalew Tsegaw is a lecturer and PhD candidate (TEFL) at Bahir Dar University. He earned his TEFL Master’s degree from BDU. He also received his second Master’s degree in project planning and management from Debre Markos University jointly with YOM. He also earned a BSc in Computer Science from Bahir Dar University, Bahir Dar Institute of Technology. His research interests include computing education, language teaching and learning, teaching methodologies, and educational technology.

Abiy Yigzaw Filate

Abiy Yigzaw Filate is a professor in the department of English Language and Literature at the faculty of humanities, Bahir Dar University. He specializes in TEFL. His research interests include Teaching English as a Foreign Language, classroom research, mediation (scaffolding) reading research, etc. https://bdu.edu.et/fh/?q=node/419

Mulugeta Teka Kahsay

Mulugeta Teka Kahsay is an associate professor in the department of English Language and Literature at the faculty of humanities, Bahir Dar University. He specializes in TEFL. His research interests include pedagogy, professional development, teaching and learning, mentoring, curriculum development, collaborative learning, pedagogy and education, academic writing, cooperative learning, and constructivism