Testing the predictive power of an instrument titled “Orientation to School Renewal”

ABSTRACT

Policy makers, school practitioners, and scholars around the world have been searching for better school improvement models. The purpose of this study was to understand how an instrument we developed, Orientation to School Renewal, can be used to predict school-level academic achievement. We used the instrument to predict the academic performance of 83 schools as measured by the Michigan Student Test of Educational Progress (M-STEP) and the Scholastic Achievement Test (SAT) for mathematics and language. We found our instrument was more sensitive to M-STEP. We found that school renewal efforts were able to predict school academic performance with M-STEP in both mathematics and language. The three leading dimensions for predicting achievement on M-STEP were (a) focus on students and their achievement, (b) internal responsibility, and (c) continuous improvement. The renewal model provides a new perspective on school improvement, and future studies in other countries and international settings are recommended.

KEYWORDS:

• School reform
• instrument development
• school renewal

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Jianping Shen is the John E. Sandberg Professor of Education and the Gwen Frostic Endowed Chair in Innovation and Research in the Department of Educational Leadership, Research and Technology at Western Michigan University. He earned a PhD in educational leadership and policy studies from the University of Washington. He teaches, among other courses, leadership theory, policy analysis, research methods, and dissertation seminar. His research interests include leadership theory, data-informed decision making, teacher retention and attrition, alternative certification, systemic change, and others, using both quantitative and qualitative methodologies. Shen has directed or co-directed several large, externally funded projects.

Xin Ma is a professor at the Department of Educational, School, and Counseling Psychology at the University of Kentucky. He is a Spencer Fellow of the (U.S.) National Academy of Education, a recipient of the Early Career Contribution Award from the American Educational Research Association, (former) Canada Research Chair, and founder and (former) Director of the Canadian Center for Advanced Studies of National Databases.

Nancy Barnes Mansberger is an associate professor of educational leadership, research and technology at Western Michigan University, specializing in educational leadership. She earned an EdD in educational leadership, an MA in music and a BA in music from Western Michigan University. She teaches, among other courses, data-informed decision-making research and evaluation, school law and ethics, and school–community relations and cultural competence. Her research has focused on both the process and outcome evaluation of school reform initiatives using a mixed-method approach.

Xingyuan Gao is an assistant professor at the Department of Education at East China Normal University. He earned a PhD degree in Educational Leadership from the Western Michigan University. He has published six journal articles and three book chapters in the area of K–12 educational leadership.

Louann Bierlein Palmer is a professor of educational leadership, research and technology at Western Michigan University, specializing in educational leadership. She earned an EdD in educational administration from Northern Arizona University. Her research interests include a broad array of K–12 and higher education reform and policy issues.

Walter Burt is an associate professor of educational leadership, research and technology at Western Michigan University, specializing in educational leadership. He earned a PhD in educational leadership from the University of Michigan. He teaches, among other courses, advanced systems thinking, data-informed decision making, research and evaluation.

Robert Leneway is an expert on educational technology. He was a faculty member in the Department of Educational Leadership, Research and Technology at Western Michigan University.

Dennis McCrumb is a faculty specialist II of educational leadership, research and technology at Western Michigan University, specializing in educational leadership and K–12. He earned an EdD in educational administration from Indiana University. His research interests include national teacher retirement systems, using a mixed-method approach.

Sue Poppink is an associate professor of educational leadership, research and technology at Western Michigan university, specializing in K–12 policy and practice. She earned a PhD in Curriculum, Teaching and Policy and qualitative methods from Michigan State University. Her research interests include education policy implementation, particularly through the lens of team and teacher learning.

Patricia Reeves is a professor of educational leadership, research and technology at Western Michigan University, specializing in educational leadership and evaluation, measurement and research. Her research interests include school district and superintendent leadership, the development and credentialing of school leaders, educator performance assessment and evaluation, and education policy: school and school systems redesigned for the 21st century using qualitative methods.

Elizabeth Whitten is a professor of Special Education and Literacy Studies at Western Michigan University, specializing in special education.

Notes

1 When a school contains more than one grade level, some researchers may immediately think of hierarchical linear modeling (HLM) as the analytical technique. In such an HLM model in our case, grade levels (Level 1) are nested within schools (Level 2). At Level 1, the proportion of students who are in the proficient and advanced categories at a grade level is the outcome measure, and there are no variables descriptive of grade-level characteristics. The Level 1 model produces an average proportion for each school (adjusted for an error term) which becomes the outcome measure at Level 2. School renewal efforts and school contextual characteristics are school-level independent variables to account for the variance in the school average proportion. We did not adopt the HLM approach for the same two reasons that we discussed earlier. The average proportion produced by the Level 1 model brings us back to our first concern. The major modeling activities happen at the school level where there are only 83 schools, which brings us back to our second concern. Because of these complications, we sought multiple regression as a more parsimonious model for data analysis, with limitations to be discussed later on.

2 When the SAT scores were used to produce the (school-level) outcome measures, the analytical situation became simpler because each school had only one outcome measure in, say, mathematics. The issue here pertained to our second concern; that is, the small sample size. In this case, we had only 21 schools even though the multiple regression approach was obviously the appropriate choice. The same concern held true for the use of (school-level) college readiness as the (school-level) outcome measure.

3 For the concern of space, other descriptive information such as correlations among the seven dimensions of the instrument is available from the authors.

4 We used the term effect for convenience of interpretation, even though an effect was simply a (unstandardized) regression coefficient. There was no implication for causality because regression analyzes association. Neither did the term imply a hierarchical (multiple) regression where blocks of variables are entered into the model for the examination of added values for each block. We simply used the term as a way to emphasize the predictive nature of the dimensions of the instrument to school performance in the present study. For the concern of space, we presented essential regression results only. Other regression information such as confidence intervals for regression coefficients is available from the authors.

5 With caution, we included an alternative alpha level of .07 based on the fact that this significance level is so close to .05. It is equivalent to making wrong decisions seven out of 100 times versus five out of 100 times. We labeled relevant effects marginally significant, with the intention to inform policy makers, administrators, and educators about their potential effects.