Digital device use and scientific literacy: an examination using Programme for International Student assessment (PISA) 2015 data

ABSTRACT

This paper uses data from the OECD’s 2015 PISA and an endogenous treatment effects model to investigate the impact of different intensities of digital device use for academic purposes on science learning outcomes. When we do not differentiate the location of device use, we find that greater use can help students improve their science scores in most of the countries. When we consider school and outside-of-school use separately, we find the above positive results are driven by outside-of-school digital device use and that there are more negative results of increased device use at school.

KEYWORDS:

• Digital devices
• information and communications technology (ICT)
• learning outcome
• PISA
• endogenous treatment effects model

JEL CLASSIFICATIONS:

• I20
• I24
• I28
• C01

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 While we estimated our model separately for each of the 47 countries which participated in the ICT Familiarity Questionnaire, we could not obtain estimation results for a subset of 12 countries due to a “convergence not achieved” maximum likelihood estimation problem.

2 For example, Beltran, Das, and Fairlie (2010) models the student’s decision to graduate from high school, where home computer use has different effects on the utility of graduating versus not graduating, and where computer use is assumed to impact expected earnings.

3 There are 46 participating countries excluding Spain (Regions) according to the OECD website. The term “country” is used throughout to also include the economies of B-S-J-G (Chinese participating cities and/or provinces: Beijing, Shanghai, Jiangsu, and Guangdong), Hong Kong and Macao (China’s Special Administrative Regions), and Spain (regions).

4 See online Appendix B for the pairwise correlation coefficient matrices between ict and the three age-related instrumental variables; icthomeuse and the three age-related instrumental variables, and ictschluse and the three age-related instrumental variables. Overall, the statistics indicate that ict, icthomeuse, or ictschluse are negatively and significantly related to the age variables, albeit sometimes numerically small, implying that our intuition is correct; that is, the earlier the respondents use a digital device, computer, or internet, the more likely that they use digital devices more intensively now.

5 To further buttress our use of the age-of-first-use variables as instrumental variables, we note that other research on the relationship between ICT use and academic achievement using PISA 2012 (where the ICT Familiarity Questionnaire first introduced questions on age of first use of computers and accessing the Internet) and PISA 2015 do not include these age variables among their control variables (for PISA 2012, see, e.g., Agasisti, Gil-Izquierdo, and Han (2020), Petko, Cantieni, and Prasse (2017), and Skryabin et al. (2015); for PISA 2015, see, e.g., Hu et al. (2018), Juhaňák et al. (2019), and Odell, Galovan, and Cutumisu (2020)).

6 We did not use the derived variables HOMESCH and USESCH which measure students’ ICT use at home and at school for schoolwork respectively because those are to be explained by our model and have been captured by our endogenous ICT use dosage or intensity variables, ict, icthomeuse, and ictschluse.

7 We do not include the estimated regression coefficients for each country because our focused interest is on the average treatment effects on the treated (ATET); however, both the computer code and the regression results are available from the authors upon request.

8 As already noted, PISA uses a probabilistic, stratified and clustered sample design; in some countries schools or students are oversampled, requiring the use of appropriate weights. As per Jerrim et al. (2017), we applied final student (or sampling) weights, which scale the sample up to the size of the population within each country.

9 As is common in psychometric studies, given the amount of test material, not every student is asked every test question. PISA 2015 reports ten student test scores or “plausible values”, which are multiple imputations based on students’ randomly assigned test questions and background and school information. Jerrim et al. (2017) supports the use of a single plausible value in regression analysis: it provides both unbiased point and sampling variance estimates and “The only aspect that using a single plausible value misses is the ‘imputation error’– uncertainty that should be added to the standard error to reflect that multiple imputation is used to generate the science, reading and mathematics proficiency scores. Yet, in practice, this additional imputation error is almost always of negligible magnitude (as per the Lavy example), with key conclusions continuing to hold if it is simply ignored.” Also, ‘using one plausible value or five plausible values does not really make a substantial difference on large samples’ (OECD 2009, 46).

10 As noted earlier, results for all estimated coefficients related to Table 3 for every country are available from the authors upon request. Similarly, results for all estimated coefficients related to Tables 4 and 5 for every country are also available from the authors upon request.