![Sample our Politics & International Relations journals, sign in here to start your FREE access for 14 days]({"type":"image" , "src" : "/sda/5947/TOC-Subject-Sample-2021-Politics-International-Relations.jpg"})
Abstract
Recent attention has been placed on whether computer assisted learning (CAL) can effectively improve learning outcomes. However, the empirical evidence of its impact is mixed. Previous studies suggest that the lack of an impact in developed countries may be attributable to substitution of effort/time away from productive, in-school activities. However, there is little empirical evidence on how effective an in-school programme may be in developing countries. To explore the impact of an in-school CAL programme, we conducted a clustered randomised experiment involving over 4000 third and fifth grade students in 72 rural schools in China. Our results indicate that the in-school CAL programme has significantly improved the overall math scores by 0.16 standard deviations. Both the third graders and the fifth graders benefited from the programme.
Notes
1. The study underwent and successfully passed ethical review by Stanford University’s Internal Review Board for non-medical human subjects research with an IRB number of 19341.
2. In terms of educational achievement, Shaanxi Province is at about the national average. However, there is huge inequality within the province (NBSC Citation2011). For example, in relatively rich areas such as Guanzhong (in the central part of the province), 14.4 per cent of its population received a college education (higher than the national average of 8.9%). In contrast, in Ankang Prefecture only 4.8 per cent of the population holds a college degree.
3. We only included wanxiao (or ‘complete schools’) with six full grades in our sample because the programme requires that third grade and fifth grade students stay in the same school during the programme period (one year and a half). In rural China, there are other elementary schools with only two, three and four grades. These are often small schools (several students per grade) in remote rural villages. In Chinese these are called jiaoxuedian, or ‘teaching point schools’. The schools that were not complete schools could not be included in the programme because students often transfer to other schools when they reach the third or fourth grades. Teaching point schools also are being shut down and merged into larger complete schools. In either case, it would be impossible for students to continue to attend the CAL sessions. It would also be difficult for us to follow the students. Therefore, non-wanxiao schools were excluded from the sampling frame.
4. In selecting the teacher-supervisors, we were guided by two principles. First: we wanted to choose the teacher-supervisor rather than the school principal. We also did not want to select a teacher-supervisor who was also a math teacher. With these principles in mind, we excluded from our selection the math teachers or homeroom teachers of the third and fifth grade students. We then created a list of teachers that were available. We then randomly chose the teacher.
5. In terms of teacher training, all teacher-supervisors of the 36 treatment schools participated in a two-day mandatory training programme. During the training, the teachers were introduced to our programme protocol and the two pieces of software. The teachers also underwent hands-on session where they practiced with the software and asked questions. At the end of the training session, randomly selected teachers gave mock classes to all the other teachers who pretended to be students.
6. Both the third and fifth grade CAL software packages consisted of two individual pieces of software. The first piece of software was a commercial, game-based math-learning software programme that we obtained through donation. The software provided remedial tutoring material (both animated reviews and remedial questions) in math for the third and fifth grade students in keeping with the national uniform math curriculum. We developed the second piece of software ourselves. The package (henceforth, the CAL software) was designed to provide the students with a large number of remedial questions.
7. The students were not allowed to discuss math questions with the supervising teacher because the goal of the study is to test whether a CAL programme can improve learning of the underserved students in rural schools. We are interested in knowing whether the programme can benefit students in the poorest schools with little teaching resources. Therefore, we would like to isolate the programme impact from teacher instruction. In other words, teachers were not allowed to intervene in the classes to affect the programme impact. In fact, this is policy relevant given a scenario in which the CAL sessions were run in-school during computer class sessions. The computer teacher would not be an expert in the field and would likely be busy managing the technology and curriculum and not focused on teaching students the material that other teachers were supposed to be teaching. Likewise, the students were not allowed to discuss with other teams (students using a different computer) to limit the influence of student interaction on programme impact. It also makes it easier for teachers to manage the classes without having to organise the group discussion or other activities. According to our observation, students typically had no time to discuss with each other because the sessions were so intense that the students were almost always exclusively focused on their computers.
8. The test questions for the standardised math exam were chosen from the TIMSS test data bank. Drafts of the tests were screened by a set of rural elementary teachers in Shaanxi province. We then rigorously tested the questions in a pilot survey. We then made adjustment to the test by eliminating the questions that were too difficult (almost no one got them right) and the questions that were too easy (almost everyone got them right).
9. The standardised test scores are normalised using the distribution of test scores of the control group students within the same grade and on the same subject.
10. This is important to show because the original analysis for the out-of-school treatment effects for CAL was conducted for boarding school students only. If we show that the effects are the same for boarding and non-boarding students (as we do), then the analysis can really focus on differences between in-school effects (as reported from this study) and out-of-school effects (as shown in Lai et al. Citation2011).
11. We have also conducted an analysis of heterogeneous effects by student baseline math score, student gender, family wealth and their starting grade. The questions that the tests are intended to address are whether poorer performing students benefit more from the programme than better performing students, whether boys or girls benefit more from the programme, whether poorer or richer students improve more after the programme and whether starting grade (third or fifth grade) makes a difference in how much student learning can be improved. The results show that none of the tests detect any significant difference in programme impact among the subgroups. By following the Bonferroni approach to adjust multiple hypotheses, we divide the significance level of all the correlated outcomes of heterogeneous effects, 0.1, by the number of hypotheses we tested (that is, 4 different types of heterogeneous effects of the CAL programme). By doing this, we obtain the adjusted p-values for each individual null hypothesis of heterogeneous effects: 0.1/4 = 0.025. Since none of the heterogeneous effects are significant at the 0.1 level, they do not meet the 0.025 adjusted significance requirement either. In other words, with or without adjusting for multiple hypotheses testing, we cannot reject the null hypotheses that there are no heterogeneous effects between the poorer and better performing students, between girls and boys, between the richer and poorer students, and between the third grade and fifth grade students.
12. Although no re-randomisation was done to reassign treatment and control schools in the in-school programme, we did conduct a balance test before the start of the programme to ensure the students in the two groups were balanced. As shown in Appendix 2, the key variables of the treatment and control groups are balanced at the baseline.
13. The result table is available upon request.
14. We also tested whether the programme had any crowding-out effect on Chinese learning. Based on the regression results using Equation (1), the out-of-school programme does not seem to have crowded out student learning in Chinese (Appendix 5). The coefficient of the treatment variable is not significant for either the whole sample (third and fifth grades) or each grade separately. The magnitudes of the coefficients are small and positive.
15. Using our data on the computer class activities, we conducted a test on whether the treatment effect differs for schools where students learn basic computer skills in computer classes and the schools where students do not learn these skills. We included an interaction term between the treatment variable and a variable indicating whether the students learn basic computer operations in the regression that estimates the treatment effect (using Equation (2)). We can only run such a test among fifth grade students because there are too few third grade students who have these activities. The result table is available upon request.
16. Cost-effectiveness analysis suggests that the programme has low cost per unit of improvement in student learning. From the perspective of China’s policymakers considering to upscale the programme, computer hardware itself is already a sunk cost given that the government is installing computer labs in every rural elementary school as part of its Twelfth Five Year Plan. The marginal costs that are needed to execute the programme include teacher training, administration costs and allowance for CAL teacher-supervisors. Using the method suggested by Dhaliwal et al. (Citation2012), we calculate the total cost of the programme in our project area to be 9439 USD (in 2011, the project year) and 10,035 USD (in 2014, after taking inflation into account). We then divide the total cost by total impact (total impact = average programme effect multiplied by the total number of students attending CAL sessions): 10,035 USD/(0.17 SD * 2435 students) = 24.2 USD/SD. The cost-effectiveness of our programme is comparable to the CAL programme conducted in India. According to the estimates provided by Banerjee et al. (Citation2007), the CAL programme in India costs 21.4 USD/SD (in 2002) and 28.2 USD/SD (in 2014) – also excluding the costs of computers.