Combining Multiple Measures of Students' Opportunities to Develop Analytic, Text-Based Writing Skills

Abstract

Guided by evidence that teachers contribute to student achievement outcomes, researchers have been reexamining how to study instruction and the classroom opportunities teachers create for students. We describe our experience measuring students' opportunities to develop analytic, text-based writing skills. Utilizing multiple methods of data collection—writing assignment tasks, daily logs, and an annual survey—we generated a composite that was used in prediction models to examine multivariate outcomes, including scores on a state accountability test and a project-developed response-to-text assessment. Our findings demonstrate that students' opportunities to develop analytic, text-based writing skills predicted classroom performance on the project-developed response-to-text assessment. We discuss the importance of considering the measure(s) of learning when examining teaching–learning associations as well as implications for combining multiple measures for purposes of better construct representation.

Notes

^a n = 426.

^b n = 18.

¹Only slight differences in frequency are noted between this sample of teachers and the much larger sample from the Study of Instructional Improvement using nearly identical language arts logs. The one main difference between the two studies was the type of log administration which was paper and pencil in the Study of Instructional Improvement but online in the current study.

^aItem is part of literature-based scale that is contrasted in measurement model with all other log items.

²We created cut points around our standardized writing scale such that the bottom tier consists of teachers whose scores were between ¹/₂ standard deviation below the mean and 1¹/₂ standard deviations below the mean, and the top tier consists of teachers whose scores were between ¹/₂ standard deviation above the mean and 1¹/₂ standard deviations above the mean; and the middle tier were in between the two.

³Omits highest scoring teacher because their score was higher than 1¹/₂ standard deviations above the mean.

^† p < .10.

*p < .05.

**p < .01.

***p < .001.

⁴Examination of the scree plot and the ratio of first to second eigenvalues versus second to third eigenvalues confirms a single factor solution is preferred for the six-item composite.

⁵We expected grade to have an effect on the MSA but not the RTA due to the scale score construction of the MSA and the inability to equate the two different forms of the RTA administered at different grade levels.

^† p < .10.

*p < .05.

**p < .01.

***p < .001.

^a% Var. explained from null.

⁶The percent of variance explained was calculated using the following formula (τ_{β0background} − τ_{β0ExtendedWriting})/τ_{β0background}.

⁷The assignments, in particular, should be most aligned to the RTA because we asked teachers to turn in challenging assignments and because this represented students' opportunities to practice the skills required to do well on the RTA.

^aPercent variance explained is calculated from model immediately prior to the inclusion of the single covariate measuring instruction and is thus calculated from variance remaining after adjusting for all student and teacher characteristics.

^† p < .10.

*p < .05.

**p < .01.

***p < .001.

^aPercent variance explained is calculated from model immediately prior to the inclusion of the single covariate measuring instruction and is thus calculated from variance remaining after adjusting for all student and teacher characteristics.

^bComposites were made up of the following individual covariates: (a) Time for extended writing, (b) Text discussions with emphasis on writing, (c) Log writing scale, (d) Integration of comprehension with Writing, (e) Task cognitive demand, (f) Percentage of tasks with open response, and (g) Length of students' written responses, such that 1 = abcde; 2 = abcdg; 3 = abcdf; 4 = acdeg; 5 = acdef; 6 = acdfg; 7 = bcdeg; 8 = bcdef; 9 = bcdfg; 10 = abceg; 11 = abcef; 12 = abcfg; 13 = abdeg; 14 = abdef; 15 = abdfg.

^† p < .10.

*p < .05.

**p < .01.

***p < .001.