Effects of Presentation Mode and Computer Familiarity on Summarization of Extended Texts

Abstract

Comparability studies on computer- and paper-based reading tests have focused on short texts and selected-response items via almost exclusively statistical modeling of test performance. The psychological effects of presentation mode and computer familiarity on individual students are underresearched. In this study, 157 students read extended English texts, presented on computer or in print, and then wrote summaries on paper in English and Chinese. Besides summarization performance, the students' perception of such effects was collected through postsummarization questionnaire and interviews. The statistical analyses found that the only significant main effect of presentation mode was on the length of Chinese summaries, and computer familiarity did not affect summarization performance. The qualitative interview data concur with the statistical findings, but the students also articulated some minute physical and psychological differences between the two delivery modes and their potential effects on summarization. They indicated further that the effects of computer familiarity might be more expected than actually experienced. Although the effects of presentation mode and computer familiarity seemed to be mixed and minimal on test performance, the perceptual data demonstrated the psychological side of such effects and the importance of exploring multiple parameters and the voices of students when investigating comparability of delivery modes.

ACKNOWLEDGMENTS

I would like to thank Professor Pauline Rea-Dickins for supervising my PhD project, on which this paper is based. The comments by Dr. Richard Kiely, Dr James Purpura, and three anonymous reviewers helped improve this paper. However, any error remains mine. The financial support of the ORS Award Scheme of the United Kingdom and the University of Bristol Postgraduate Research Scholarship are also much appreciated. An earlier draft of this paper was presented at the 30th Language Testing Research Colloquium in Hangzhou, China (June 2008).

Notes

¹Only 12 of the 24 interviewees actually read the computer-presented texts, but the other 12 students who read the print texts were also asked, during the interviews with them for other research aims, to comment on the effects of presentation mode and computer familiarity, imaging if they had been requested to read the computer-presented texts.

²By the sample size (around 170 according to Baggaley, 1982), the Kaiser-Meyer-Olkin measure of sampling adequacy (MSA = .81, above .60 the minimum value for a good factor analysis; Kaiser, 1970, 1974; Tabachnick & Fidell, 2003), and Bartlett's (1954) test of sphericity, approx. χ²(496) = 1,977.6, p < .001, in the initial solutions. Furthermore, the anti-image correlation matrices were also examined to see whether there were variables that did not fit with the structure of the other variables. Q5 (.38) and Q30 (.47) were found to have MSA values below .50 and were therefore removed in further factor analyses. Finally, the Pearson correlation matrix for the remaining 30 questions showed that a significant number of correlations were greater than 0.3.

³The use of the variables/questions having loadings of 0.3 or more of the first factor in the two-factor varimax rotations did not make much difference in students' computer familiarity scores; therefore only the results from the promax rotations were reported and used to create the scale.

⁴This is because the effects of language order were controlled through research design and it would reduce the number of cases in the interaction terms if language order were included as the third fixed factor in the full-factorial models. However, we added language order as the third fixed factor using the main-effects models, and it was found that language order did not exert any significant effects on summarization performance except the length of English summaries (F = 6.24, p < .05). English summaries were consistently longer when written in order of English then Chinese than Chinese then English. When modeling the effects of computer familiarity, the similar significant main effects of language order was observed (F = 5.90, p < .05).

⁵Although the total variances explained were quite small in the four models, it is interesting to note that text type had significant effects on all the indicators of summarization performance except for the length of English summaries and that the language ability covariates had significant effects only on CHS.

⁶Pseudonyms, not necessarily indicative of the students' actual gender.