Modelling Pedagogy in Australian School Reform

Abstract

This article presents a discussion of the technical development and statistical results of one of Australia's most widely recognised models of pedagogy designed to research school improvement. This is the first public reporting of the statistical results of the productive pedagogy research. Although the modelling of classroom practice from which the productive pedagogy model was drawn clearly supports the hypothesis that pedagogy needs to be seen as multidimensional, not all of the theoretical dimensions outlined in the productive pedagogy model are empirically defensible. Of the four dimensions theoretically proposed, Intellectual Quality, Support Learning Environment, and Connectedness were sufficiently measured for sound empirical examination. Although the notion of there being a dimension of pedagogy related to Recognition of Difference may well have theoretical justification, the productive pedagogy research cannot offer empirical substantiation of hypotheses related to this construct. Consequently, this account of the development of productive pedagogy stands as one example of the risk of professional development interests that run too far ahead of research.

Notes

¹I should note that the plural form of this label, productive pedagogies, has also been used publicly to denote variability in theoretically desirable forms of classroom practice. I maintain the singular in this analysis because, as will be shown, the research on which the label is based does not in itself support the denotation of the plural form. Additionally, to my knowledge, formally, pedagogy has no plural form.

²For a more sceptical view of Education Queensland's intent and a more elaborated historical account of these events, see Lingard, Hayes, and Mills (2002). There is an irony in the way Education Queensland wedded authentic pedagogy to SBM in that the work done by CORS actually had demonstrated that SBM was unrelated to improvements in pedagogy and student performance, especially in terms of authentic pedagogy (see Newmann, 1993).

³Thus the initial design of the QSRLS was largely a replication and extension of core studies in Newmann's centre. On the one hand, detailed, multimethod data were to be gathered from a total of 24 schools over 3 years—8 schools per year. Although survey data was gathered from all students in target year-levels and all teachers in these schools, the proposed 24 schools were the sites of direct classroom observations; interviews with teachers, school leaders, and key “change agents”; and the source of samples of student work and teacher assessment tasks. Taken together, these schools became known as the QSRLS field study schools. In addition to this set of data, a broader survey of 150 schools was to be conducted annually, in a cascading longitudinal design over the 3-year life of the study. This second set of data was known simply as the annual survey. Finally, as a third set of data, Education Queensland committed itself to providing student-level prior achievement data and school-level demographic and student performance data. All of these data sets were to be brought together in a multilevel analysis of school restructuring in a manner similar to CORS, covering four levels of phenomenon: student performance and experience, classroom practice, school organisational capacity, and external “supports” (cf. Newmann & Wehlage, 1995).Although this design was intended as a direct response to Queensland's interest in authentic pedagogy and SBM, it was also helpful that the designer and author of the initial proposal (and of this analysis) was a member of the CORS team that had developed the instruments used in authentic pedagogy and school restructuring, and had been part of the conducting of the CORS studies prior to emigrating to Australia. Thus, access to all levels of CORS information (including the unreported material) was direct for the QSRLS.

⁴It should also be noted that many educational commentators have suggested that the academic focus of authentic pedagogy was itself a problem, arguing that “some” students just don't benefit from an academic concentration. Without opening the debate about the normative ends of schooling, I would note that this view clearly has not taken into account the long recognised positive impact of academic press for students from all socioeconomic and racial backgrounds in the United States (see Lee & Smith, 1999; Lee et al., 1997).

⁵If this line of reasoning isn't clear, it may help to consider the simple proposition that there are many ways a teacher might produce classroom practice that would score highly on the authentic pedagogy measure. Some of these differences were documented in the reports on authentic pedagogy, others were known firsthand to the author, who had conduct some of the fieldwork on which CORS analyses were based.

⁶The goodness of fit indexes for the CFA model of authentic pedagogy also yielded mixed results. Although overall measures were quite good (GFI = .99, AGFI = .99, etc.) the chi-square and root mean square error of approximation were less than optimal (.04 and .07, respectively). See Schumacker and Lomax (1996), for one explanation of these fit statistics.

⁷The GFI = .99 and AGFI = .98, while the chi-square p was .09 and RMSEA = .05.

⁸GFI and AGFI = .99, chi-square p = .25, and RMSEA = .04.

⁹GFI = .99, AGFI = .97, chi-square p = .29, and RMSEA = .03.

¹⁰One generous rule of thumb holds 3 as an upper boundary for skew and two of these items were way beyond that (see Hoyle, 1995). Kurtosis is more debated among statisticians and of particular concern for analyses of covariance.

¹¹GFI = .94, AGFI = .83, chi-square p = .00, and RMSEA = .06.

¹²To follow the concordance from the item means to the rubric descriptors, public access to a large portion of the rubrics can be found via Education Queensland's Web site: http://education.qld.gov.au/public_media/reports/curriculum-framework/productive-pedagogies/html/manual.html In that public version, the wording for codes of a 1, 3, and 5 were included—without referring to the numbers. A full version of the productive pedagogy observation manual has also been distributed widely in Queensland and NSW but never published. Full copies can be obtained from the Professional Learning in Education Research Group, School of Education, The University of Newcastle, Callaghan, NSW, 2308.

¹³Another measurement concern not noted earlier included interrater reliability. To monitor this, approximately 20% of the lessons observed were independently coded by two observers and their scores analysed for (1) the percentage of exact agreement and the percentage of agreement within one rating and (2) the Pearson correlation coefficient. In general, the items of the first three dimensions, Intellectual Quality, Connectedness and Socially Supportive Classroom Environment, obtained levels of interrater reliability comparable to those reported by Newmann for Authentic Instruction (see Newmann & Associates, 1996; Appendix A). That is, agreements occurred generally around 90% of the time (some items higher, none lower than 85%), and the correlations were generally above .65 (although strongly skewed items had more exact agreement and lower correlations due to the skew). Three items did not obtain these levels of interrater reliability: Metalanguage, Narrative, and Inclusivity. The specification of these items' rubrics was slightly refined after the 1st year of the study to improve reliability. By the end of the study, the reliability of Metalanguage and Narrative were up to the levels of the other items. Although definitional changes led to improvements in reliability, Inclusivity never obtained highly defensible levels of reliability.