Data-Informed Decision Making on High-Impact Strategies: Developing and Validating an Instrument for Principals

Abstract

In this study, the authors connect 3 streams of literature to develop an instrument for measuring the degree to which principals engage in data-informed decision making on high-impact strategies that are empirically associated with higher student achievement. The 3 literature streams are (a) the importance of data-informed decision making, (b) the role of principals in school effectiveness, and (c) high-impact strategies for raising student achievement. The authors used Marzano's (2003) 11 high-impact strategies as a framework to develop items and collected data from 256 principals in Michigan to test the reliability (internal consistency) and (construct and factorial) validities of the instrument. The instrument, “Data-Informed Decision-Making on High-Impact Strategies: An Inventory for Principals,” demonstrates good psychometric properties. Theoretical and practical implications are discussed.

Keywords::

• data-informed decision-making
• high-impact strategies
• leadership measurement and development
• principal leadership

Notes

1. From a survey research perspective, we used Cochran's (1977) formulas to evaluate our response rate of 8.4%. With the alpha level set a priori at .05, a 4-point scale, a level of acceptable error at 3%, and a standard deviation of the scale as 1.00 (see Bartlett, Kotrlik, & Higgins, 2001), the minimum sample size from a population of 3,061 is 246. Our actual sample size of 256 is, therefore, adequate from a survey research perspective.

2. Statistical research has shed little light on the issue of a minimum level of sample size for factor analysis in behavioral sciences (Guadagnoli & Velicer, 1988; MacCallum et al., 1999). In a review of 60 factor analyses published in four journals (Educational and Psychological Measurement; Journal of Educational Psychology; Personality and Individual Differences; and Psychological Assessment), Henson and Roberts (2006) reported a minimum sample size of 42. MacCallum et al. (1999) conducted a Monte Carlo analysis on sample size effects in which an excellent recovery of population factor structure is achieved with a minimum sample size of 60.

3. ANOVA is not a primary means to address the issue of invariance of the instrument across principals of different demographic characteristics. One strategy for future studies to investigate this issue is to perform multigroup confirmatory factor analysis as a way to demonstrate that the items measure the same number of constructs and with the same degree of precision across multiple group memberships.

4. According to Thompson (2004), pattern and structure coefficients are helpful to appreciate a selected model. For the economics of space, these results are omitted but available from the authors.

5. For the concern of space, the authors reported only statistically significant ANOVA results. Full results are available from the authors.

6. In general, self surveys among principals tend to show relatively positive behaviors. It would not surprise us if data from principals show some departure from a normal distribution. We would be concerned about a serious departure that could indicate some problems in item construction. The near-normal distribution of our data alleviated this concern considerably.