Developing and validating an instrument measuring school renewal: testing the factorial validity and reliability

ABSTRACT

For this paper, we developed and validated the Orientation to School Renewal instrument,a 21-item instrument, based on seven factors, which allows schools to measure their school renewal efforts. The research is based on Goodlad’s notion of school renewal. Through an extensive literature review and our work over an eight-year period engaged in discussions with over 110 principals and 300 teachers, we gradually developed dimensions of the construct of school renewal. We then obtained survey data from a sample of 1,195 teachers in 83 schools, and validated the instrument using confirmatory factor analysis and other procedures. The seven factors identified from the literature and discussions with K-12 educators and confirmed via the validation process included: (a) focus on students and their achievements, (b) continuous school improvement, (c) balance between the internal and external influences, (d) the dialogue, decision, action and evaluation (DDAE) process, (e) implementation integrity, (f) implementers as active developers, and (g) internal responsibility and professionalism. Researchers and practitioners can use the instrument to collect valid and reliable data to measure, evaluate, and understand school renewal.

Disclosure of potential conflicts of interest

No potential conflict of interest was reported by the author(s).

Supplemental data

Supplemental data for this article can be accessed here.

Notes

1. There are two approaches to conduct empirical studies. The data-driven approach is common when conducting research in a field that is largely uncharted theoretically (i.e. not well researched). The lack of theoretical guidance allows data to ‘make decision’ concerning the building of a statistical model. In the case of instrument validation, exploratory factor analysis (EFA) is often appropriate in studies taking the data-driven approach. On the other hand, the theory-driven approach is common when conducting research in a field that is relatively more established with theoretical guidance to specify a statistical model (i.e. theories ‘make decision’ concerning the building of the statistical model). In the case of instrument validation, CFA is often appropriate in studies taking the theory-driven approach. This is exactly the case for our study because the school renewal framework is relatively established which dictates the building of our factor structure. This is also the reason why we did not attempt EFA in combination with CFA in that there is no need to let data make decision when the school renewal framework is mature enough to dictate how we should build the factor structure for the instrument.

2. Again, following the same logic as in our previous footnote (i.e. 1 above), because the theory-driven approach was appropriate for our study (given the status of the relevant literature), we did not compare factor structures in any exploratory manner. Instead, our focus was on the seven-factor structure as dictated by the school renewal framework. We basically compared this factor structure with the overall one factor structure and the higher-order one mega factor structure. Other factor structures were deemed not optimal by the school renewal framework. Thus, our main attention was to examine if the seven-factor structure can ‘hold water’ (i.e. withstand the test) in the presence of standard references (e.g. the overall one factor structure).

3. Based on a significance level of .05, these χ² results were all statistically significant. It is well known that χ² is sensitive to sample size, and as such the attention is often paid to the drop in χ² (e.g. Hu & Bentler, 1999). The drops we reported were statistically significant, indicating important improvement in model-data-fit. In addition, to overcome the sensitivity of χ² to sample size, we used other model-data-fit indices independent of sample sizes such as CFI and TLI.

4. For the (final) seven-factor model, Table 3 presents the main structural results. Meanwhile, Appendix C presents the estimated relationships among the seven factors, given that these factors were allowed to correlate.

5. Some researchers may also want to know whether any of the items also loads highly on other factors. We did not examine the possibilities because we were pursuing a theory-based instrument validation (i.e. to which factor each item loads is determined by theories). We were confident of our factor structure given our content validity via multiple ways (e.g. expert panel, literature review).

6. High reliability can occur when the same question is answered multiple times or there is an acquiescence bias (i.e. respondents answer yes to every item). We believed that neither case occurred in our study. We paid close attention to the content validity, making sure that each item addressed only one factor and was distinguishable from all other items (even those measuring the same factor). Our pilot study reinforced this claim. Although we did not have reverse-coded items to check against acquiescence bias, we found very few returned questionnaires with the same response across all items. In addition, the completion of the questionnaire (instrument) was both anonymous and not timed (giving teachers no reason to rush through the questionnaire). It is our belief that we safeguarded against acquiescence bias in an adequate way.

Additional information

Notes on contributors

Jianping Shen

Jianping Shen received his Ph.D. in educational leadership and policy studies from the University of Washington in Seattle. He is currently the John E. Sandberg Professor of Education and the Gwen Frostic Endowed Chair in Research and Innovation at Western Michigan University. He published extensively on educational leadership, policy analysis, and research methods.

Xin Ma

Xin Ma is a professor at the Department of Educational, School, and Counseling Psychology at the University of Kentucky. He is a Spencer Fellow of the (U.S.) National Academy of Education, a recipient of the Early Career Contribution Award from the American Educational Research Association, (former) Canada Research Chair, and founder and (former) Director of the Canadian Center for Advanced Studies of National Databases.

Nancy Mansberger

Nancy Barnes Mansberger is an associate professor of educational leadership, research and technology at Western Michigan University, specializing in educational leadership. She earned an EdD in educational leadership, an MA in music and a BA in music from Western Michigan University. She teaches, among other courses, data-informed decision-making research and evaluation, school law and ethics, and school–community relations and cultural competence. Her research has focused on both the process and outcome evaluation of school reform initiatives using a mixed method approach.

Louann Bierlein Palmer

Louann Bierlein Palmer is a professor of educational leadership, research and technology at Western Michigan University, specializing in educational leadership. She earned an EdD in educational administration from Northern Arizona University. Her research interests include a broad array of K–12 and higher education reform and policy issues.

Walter Burt

Walter Burt is an associate professor of educational leadership, research and technology at Western Michigan University, specializing in educational leadership. He earned a PhD in educational leadership from the University of Michigan. He teaches, among other courses, advanced systems thinking, data-informed decision making, research and evaluation.

Robert Leneway

Robert Leneway is an expert on educational technology. He was a faculty member in the Department of Educational Leadership, Research and Technology at Western Michigan University.

Patricia Reeves

Patricia Reeves is a professor in the Educational Leadership, Research and Technology at Western Michigan University, specializing in educational leadership and evaluation, measurement and research. She teaches, among other courses, superintendency, instructional leadership and supervision, dissertation seminar, qualitative research methods and qualitative research practicum. Her research interests include school district and superintendent leadership, the development and credentialing of school leaders, educator performance assessment and evaluation, and education policy: school and school systems redesigned for the 21st century using qualitative methods.

Sue Poppink

Sue Poppink is an associate professor of educational leadership, research and technology at Western Michigan university, specializing in K–12 policy and practice. She earned a PhD in Curriculum, Teaching and Policy and qualitative methods from Michigan State University. Her research interests include education policy implementation, particularly through the lens of team and teacher learning.

Dennis McCrumb

Dennis McCrumb is a faculty specialist II of educational leadership, research and technology at Western Michigan University, specializing in educational leadership and K–12. He earned an EdD in educational administration from Indiana University. His research interests include national teacher retirement systems, using a mixed-method approach.

Elizabeth Whitten

Elizabeth Whitten is a professor of Special Education and Literacy Studies at Western Michigan University, specializing in special education.

Xingyuan Gao

Xingyuan Gao is an assistant professor in the Department of Education at East China Normal University. He earned his Ph.D. in educational leadership from Western Michigan University. His areas of research include educational change, organizational theories, and data-informed decision making.

Huang Wu

Huang Wu is a post-doctoral fellow at Western Michigan University. He earned his Ph.D. in educational leadership from Western Michigan University. His areas of research include principal leadership, teacher leadership, and social justice leadership.