An iterative adaptive developmental evaluation framework: case study of the TREE semester learning community

Abstract

Program evaluation frameworks that assess organizations as complex adaptive systems can open our eyes to new approaches and understandings. The proposed iterative and adaptive developmental program evaluation framework interrelates developmental evaluation principles with adaptive action approaches. We explain and demonstrate the framework with a mixed methods case study of the Teaching and Researching Environmental Education (TREE) Semester, an undergraduate pre-service environmental and sustainability educator leadership program. To illustrate developmental evaluation rigor, we describe the collection of TREE students’ end-of-program quantitative and qualitative self-assessment responses to factors that helped their learning and promoted their learning gains. Analyzing these responses allowed us to continually make adaptive changes to develop a thriving learning community that fosters environmental and sustainability education leadership development. The iterative and adaptive developmental program evaluation framework improved our understanding of TREE’s learning community as a complex adaptive system by elucidating five simple, emergent, pattern-forming rules. The rules describe activity selection, enhanced strategies for communication and support, integrating staff and student learning, and program pace. This demonstration of a developmental program evaluation provides a model framework for environmental and sustainability education program evaluators, especially those who work with new, innovative, complex, and re-imagined programs.

Keywords:

• Developmental program evaluation
• adaptive program evaluation
• pre-service teachers
• undergraduate students
• learning schools

Acknowledgements

I acknowledge Dr. Elaine Seymour for creating and supporting SALG and providing comments on a draft manuscript, Laurel Sullivan for sorting and compiling much of the SALG quantitative data, Julie Francis for listening to numerous versions and providing stimulating ideas for the manuscript, Drs. Kathleen Greene and Becca Franzen for insightful comments, an anonymous reviewer whose comprehensive understanding of developmental evaluation provided critical insight, and TREE Stewards and Fellows for living all the experiences and providing so much detailed feedback.

Disclosure statement

Although I serve as volunteer Board President and treasurer of the non-profit Catamount Center for Environmental Science and Education where TREE takes place, I have no financial interest in the organization and there is no conflict of interest. The work in this research did not receive any project-specific funding. Colorado College pays my salary to direct the TREE Semester as part of my faculty position.

Notes

1 The term Steward is used at TREE and throughout this report to differentiate college students from their K-12 students and to recognize their role at TREE beyond just students.

2 The course titles of the required education classes are: Environmental Education Practicum, Critical Foundations of Environmental & Sustainability Education (theory), Developing Environmental & Sustainability Education Curriculum. The course titles of the elective classes are Aquatic & Terrestrial Ecosystems and Environmental Inquiry.

3 TREE staff include 2-3 professors, an education director, site manager, a full-time seasonal chef as well as teacher observers who observe and help Stewards improve their teaching during five-hour inquiry classes. The TREE program director served as lead professor for all classes except the practicum in all years of this case study. For the first four years of TREE, the education director and site manager were residential TREE program staff.

4 The Colorado Alliance for Environmental Education (CAEE) certifies individual portfolios at certified and master certified levels. In 2014, NAAEE endorsed all CAEE master-level portfolios (https://naaee.org/our-work/programs/certification) based on a thorough review of the CAEE’s potfolio alignment with the NAAEE (2019) professional development guidelines.

5 The NAAEE distinguished program (https://naaee.org/our-work/programs/higher-education-accreditation) follows the Professional Development of Environmental Educators: Guidelines for Excellence and awards distinction based on how well a program meets these guidelines.

6 Westhorp, Stevens, and Rogers (2016) provide a realist action research model with characteristics that overlap FIA. Their model, which, like ours, uses an iterative cycle of plan, act, observe and reflect, differs from our developmental approach by starting with a realist hypothesis rather than a sensitizing question and though not a method, is, according to Westhorp, ‘methodological.’

7 In this sense, FIA is more like Westhorp’s (2016) action research approach (Note 6). Patton (2011) suggests that developmental evaluators are like bricoleurs, creative people who use what is available and needed rather than being constrained by a given methodology.

8 The Colorado College IRB review board granted approval for this project in October 2015. TREE students sign a non-coercive consent form allowing use of their required class assignments for research and publication. The PI receives a report from a TREE staff member acknowledging the receipt but not the identity of those who complete consent forms. In all five years of this study all students provided consent.

9 The SALG instrument is free and available at: https://www.salgsite.net/. Currently, the only version is in English, but one study (Lu 2018) noted translating SALG to Chinese.

10 Parsing the essays help scale into a reflective essay help scale and a theory-based essay help scale showed no significant differences by factor and reliability analyses, so we retained essays as a single help scale. The synthesis essay in Year 5 combined three theory-based essays into one extended essay.

11 Only one Steward who completed the TREE Semester did not return SALG (Year 1). As noted in Table 2, two Stewards did not complete coursework at TREE. Thus, our survey return rate is 94% for all Stewards who attended TREE and 98% for those who completed TREE. This high response rate represents Stewards’ interest in the program and that 5% of their final grade in the Practicum course is awarded for completing the survey by the due date.

Additional information

Notes on contributors

Howard Drossman

Howard Drossman is Professor of Environmental Education and TREE Semester Program Director at Colorado College since 2014, where he served previously as Environmental Studies Program Director and as Professor of Bioanalytical Chemistry. Since testing SALG in its initial 1996–1998 development, Howard has used the instrument to help evaluate ∼100 classes and programs in ∼20 different courses. Howard has been a member of the CAEE certification review board since 2009 when he certified as a master-level environmental educator. With his wife Julie Francis, he co-founded two local environmental non-profits in 1997, Catamount Center for Environmental Science and Education and The Catamount Institute. His current research interests include program evaluation, pre-service teacher education, and adult learning and development.