Collaborative capacity-building for collective evaluation: a case study with informal science education centers

References

• Allen, S., Campbell, P. B., Dierking, L. D., Flagg, B. N., Friedman, A. J., Garibay, C., & Ucko, D. A. (2008, February). Framework for evaluating impacts of informal science education projects. In Report from a National Science Foundation Workshop. The National Science Foundation, Division of Research on Learning in Formal and Informal Settings.
   Google Scholar
• Allen, S., & Peterman, K. (2019). Evaluating informal STEM education: Issues and challenges in context. New Directions for Evaluation, 2019(161), 17–33. https://doi.org/10.1002/ev.20354
   Google Scholar
• Ardoin, N. M., Bowers, A. W., Roth, N. W., & Holthuis, N. (2018). Environmental education and K-12 student outcomes: A review and analysis of research. The Journal of Environmental Education, 49(1), 1–17. https://doi.org/10.1080/00958964.2017.1366155
   Web of Science ®Google Scholar
• Ardoin, N. M., Clark, C., & Kelsey, E. (2013). An exploration of future trends in environmental education research. Environmental Education Research, 19(4), 499–520. https://doi.org/10.1080/13504622.2012.709823
   Web of Science ®Google Scholar
• Ash, D, & Lombana, J. (2013). Reculturing museums: Working toward diversity in informal settings. Journal of Museum Education, 38(1), 69–80.
   Google Scholar
• Atkinson, D. D., Wilson, M., & Avula, D. (2005). A participatory approach to building capacity of treatment programs to engage in evaluation. Evaluation and Program Planning, 28(3), 329–334. https://doi.org/10.1016/j.evalprogplan.2005.04.012
   Web of Science ®Google Scholar
• Bequette, M., Cardiel, C. L. B., Cohn, S., Kollmann, E. K., & Lawrenz, F. (2019). Evaluation capacity building for informal STEMeducation: Working for success across the field. New Directions for Evaluation, 161, 107–123.
   Google Scholar
• Bevan, B, Calabrese Barton, A, & Garibay, C. (2020). Broadening perspectives on broadening participation: Professional learning tools for more expansive and equitable science communication. Frontiers in Communication, 5, 52.
   Google Scholar
• Biggar, M., Ardoin, N. M., & Morris, J. (2017). Collective impact on the ground. Stanford Social Innovation Review. https://ssir.org/articles/entry/collective_impact_on_the_ground
   Google Scholar
• Borgatti, S. P., Everett, M. G., & Freeman, L. C. (2002). Ucinet for windows: Software for social network analysis. Analytic Technologies.
   Google Scholar
• Borgatti, S. P., Everett, M. G., & Johnson, J. C. (2018). Analyzing social networks. Sage Publications.
   Google Scholar
• Brody, M., Bangert, A., & Dillon, J. (2007). Assessing learning in informal science contexts. Commissioned Paper for the National Research Council, Science Learning in Informal Environments Committee.
   Google Scholar
• Busch, K., Chesnut, L., Ayala, R., Stevenson, K., Larson, L., & Yelton, C. (2022). Building capacity for collective evaluation across ISE centers: A tested model for a collaborative approach [Paper presentation]. National Association for Research in Science Teaching (NARST) Annual International Conference, Vancouver, Canada.
   Google Scholar
• Busch, K., Kudumu, M., & Park, S. (2023). Pedagogical content knowledge for informal science educators: Development of the ISE-PCK framework. Research in Science Education, 53(2), 253–274. https://doi.org/10.1007/s11165-022-10055-9
   Web of Science ®Google Scholar
• Carleton-Hug, A., & Hug, J. W. (2010). Challenges and opportunities for evaluating environmental education programs. Evaluation and Program Planning, 33(2), 159–164. https://doi.org/10.1016/j.evalprogplan.2009.07.005
   PubMed Web of Science ®Google Scholar
• Center for Advancement of Informal Science Education. (2011). Principal investigator’s guide: Managing evaluation in informal STEM education projects. http://informalscience.org/evaluation/evaluation-resources/pi-guide
   Google Scholar
• Center for the Advancement of Informal Science Education. (2017). Learn more about evaluation. http://informalscience.org/design-evaluation/learn-more-about-evaluation
   Google Scholar
• Chaudhary, A. K., Diaz, J., Jayaratne, K. S. U., & Assan, E. (2020). Evaluation capacity building in the nonformal education context: Challenges and strategies. Evaluation and Program Planning, 79, 101768. https://doi.org/10.1016/j.evalprogplan.2019.101768
   PubMed Web of Science ®Google Scholar
• Clark, C., Heimlich, J., Ardoin, N. M., & Braus, J. (2015). Describing the landscape of the field of environmental education. North American Association for Environmental Education Annual Conference, San Diego, CA.
   Google Scholar
• Coburn, C. E., & Penuel, W. R. (2016). Research–practice partnerships in education: Outcomes, dynamics, and open questions. Educational Researcher, 45(1), 48–54. https://doi.org/10.3102/0013189X16631750
   Web of Science ®Google Scholar
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates, Publishers.
   Google Scholar
• Collaboration for Ongoing Visitor Experience Studies. (2018). Understanding our visitors: Multi-institutional science center study (July 2017–June 2018). http://www.understandingvisitors.org/wp-content/uploads/2018/10/COVES-FY18-Aggregate-Report_spreads.pdf
   Google Scholar
• Collaboration for Ongoing Visitor Experience Studies website. (n.a.) http://www.understandingvisitors.org/
   Google Scholar
• Cousins, J. B., Goh, S. C., Elliott, C. J., & Bourgeois, I. (2014). Framing the capacity to do and use evaluation. In J. B. Cousins, & I. Bourgeois (Eds.), Organizational capacity to do and use evaluation. New directions for evaluation (Vol. 141, pp. 7–23). https://doi.org/10.1002/ev.20076
   Google Scholar
• Dawson, E. (2014). Not designed for us”: How science museums and science centers socially exclude low-income, minority ethnic groups. Science education, 98(6), 981–1008.
   PubMed Web of Science ®Google Scholar
• DeWitt, J., & Archer, L. (2017). Participation in informal science learning experiences: The rich get richer? International Journal of Science Education, Part B, 7(4), 356–373. https://doi.org/10.1080/21548455.2017.1360531
   Web of Science ®Google Scholar
• Ellenbogen, K. (2014). Summary of the CAISE convening on building capacity for evaluation in informal science, technology, engineering and math (STEM) education. Center for Advancement of Informal Science Education (CAISE). http://informalscience.org/research/ic-000-000-010-034/ECB_Convening_Summary
   Google Scholar
• Falk, J. H., & Dierking, L. D. (2010). The 95 percent solution. American Scientist, 98(6), 486–493. https://doi.org/10.1511/2010.87.486
   Web of Science ®Google Scholar
• Falk, J. H., Dierking, L. D., Swanger, L. P., Staus, N., Back, M., Barriault, C., Catalao, C., Chambers, C., Chew, L.-L., Dahl, S. A., Gorecki, B., Lau, T.-C., Lloyd, A., Martin, J., Santer, J., Singer, S., Solli, A., Trepanier, G., Tyystjärvi, K., … Falla, S. (2016). Correlating science center use with adult science literacy: An international, cross-institutional study. Science Education, 100(5), 849–876. https://doi.org/10.1002/sce.21225
   Web of Science ®Google Scholar
• Falk, J. H., & Needham, M. D. (2010). Measuring the impact of a science center on its community. Journal of Research in Science Teaching, 48(1), 1–12. https://doi.org/10.1002/tea.20394
   Web of Science ®Google Scholar
• Falk, J. H., Randol, S., & Dierking, L. D. (2012). Mapping the informal science education landscape: An exploratory study. Public Understanding of Science, 21(7), 865–874. https://doi.org/10.1177/0963662510393606
   PubMed Web of Science ®Google Scholar
• Feinstein, N W, & Meshoulam, D. (2014). Science for what public? Addressing equity in American science museums and science centers. Journal of Research in Science Teaching, 51(3), 368–394.
   Web of Science ®Google Scholar
• Fisher, R. J. (1993). Social desirability bias and the validity of indirect questioning. Journal of Consumer Research, 20(2), 303–315. https://doi.org/10.1086/209351
   Web of Science ®Google Scholar
• Fleming, M. L., & Easton, J. (2010). Building environmental educators’ evaluation capacity through distance education. Evaluation and Program Planning, 33(2), 172–177. https://doi.org/10.1016/j.evalprogplan.2009.07.007
   PubMed Web of Science ®Google Scholar
• Garibay, C, & Teasdale, R M. (2019). Equity and evaluation in informal STEM education. New Directions for Evaluation, (161), 87–106.
   Google Scholar
• Glaser, B. G. (1965). The constant comparative method of qualitative analysis. Social Problems, 12(4), 436–445. https://doi.org/10.2307/798843
   Web of Science ®Google Scholar
• Grack Nelson, A., Goeke, M., Auster, R., Peterman, K., & Lussenhop, A. (2019). Shared measures for evaluating common outcomes of informal STEM education experiences. New Directions for Evaluation, 2019(161), 59–86. https://doi.org/10.1002/ev.20353
   Google Scholar
• Hargraves, M., Buckley, J., Urban, J. B., Linver, M. R., Chauveron, L. M., Samtani, S., Archibald, T., & Moorman, L. (2021). Resonance, stickiness, and the value propositions of evaluation capacity building: Key takeaways and future directions. New Directions for Evaluation, 2021(169), 97–116. https://doi.org/10.1002/ev.20442
   Google Scholar
• Heimlich, J. E. (2010). Environmental education evaluation: Reinterpreting education as a strategy for meeting mission. Evaluation and Program Planning, 33(2), 180–185. https://doi.org/10.1016/j.evalprogplan.2009.07.009
   PubMed Web of Science ®Google Scholar
• Kania, J., & Kramer, M. (2011). Collective impact. Stanford Social Innovation Review, 36–41.https://ssir.org/articles/entry/collective_impact
   Google Scholar
• King, J. A.. (2002). Building the evaluation capacity of a school district. New Directions for Evaluation, 93, 63–80.
   Google Scholar
• King, J. A., & Volkov, B. (2005). A framework for building evaluation capacity based on the experiences of three organizations. CURA Reporter, 10–16.
   Google Scholar
• Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33(3), 307–338. https://doi.org/10.1177/1098214011434608
   Web of Science ®Google Scholar
• Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge University Press.
   Google Scholar
• Lawrenz, F., Kollmann, E. K., King, J. A., Bequette, M., Pattison, S., Nelson, A. G., Cohn, S., Cardiel, C. L. B., Iacovelli, S., Eliou, G. O., Goss, J., Causey, L., Sinkey, A., Beyer, M., & Francisco, M. (2018). Promoting evaluation capacity building in a complex adaptive system. Evaluation and Program Planning, 69, 53–60. https://doi.org/10.1016/j.evalprogplan.2018.04.005
   PubMed Web of Science ®Google Scholar
• Li, Y, & Krasny, M E. (2021). Relationship between professional networks and practice change in environmental education. The Journal of Environmental Education, 52(3), 174–189.
   Web of Science ®Google Scholar
• Merriam, S. B., & Tisdell, E. J. (2015). Qualitative research: A guide to design and implementation. John Wiley & Sons.
   Google Scholar
• Merritt, E., Stern, M. J., Powell, R. B., & Frensley, B. T. (2022). A case study of an evidence-based learning network for environmental educators [Unpublished manuscript].
   Google Scholar
• Milstein, B., Chapel, T. J., Wetterhall, S. F., & Cotton, D. A. (2002). Building capacity for program evaluation at the centers for disease control and prevention. New Directions for Evaluation, 2002(93), 27–46. https://doi.org/10.1002/ev.40
   Google Scholar
• Naccarella, L., Pirkis, J., Kohn, F., Morley, B., Burgess, P., & Blashki, G. (2007). Building evaluation capacity: Definitional and practical implications from an Australian case study. Evaluation and Program Planning, 30(3), 231–236. https://doi.org/10.1016/j.evalprogplan.2007.05.001
   PubMed Web of Science ®Google Scholar
• National Informal STEM Education Network. (n.a.) https://www.nisenet.org/
   Google Scholar
• National Research Council. (2009). Learning science in informal environments: People, places, and pursuits. The National Academies Press.
   Google Scholar
• National Science Foundation. (2008). Framework for evaluating impacts of informal science education projects. http://www.informalscience.org/documents/Eval_Framework.pdf
   Google Scholar
• National Science Foundation (NSF). (2010). The 2010 user-friendly handbook for project evaluation. Author.
   Google Scholar
• Norton, S., Milat, A., Edwards, B., & Giffin, M. (2016). Narrative review of strategies by organizations for building evaluation capacity. Evaluation and Program Planning, 58, 1–19. https://doi.org/10.1016/j.evalprogplan.2016.04.004
   PubMed Web of Science ®Google Scholar
• Phillips, T., Porticella, N., Bonney, R., & Grack-Nelson, A. (2015). DEVISE. Cornell Lab of Ornithology.
   Google Scholar
• Phillips, T., Porticella, N., Constas, M., & Bonney, R. (2018). A framework for articulating and measuring individual learning outcomesfrom participation in citizen science. Citizen Science: Theory and Practice, 3(2), 3. https://doi.org/10.5334/cstp.126
   Google Scholar
• Powell, R. B., Stern, M. J., & Ardoin, N. (2006). A sustainable evaluation framework and its application. Applied Environmental Education & Communication, 5(4), 231–241. https://doi.org/10.1080/15330150601059290
   Google Scholar
• Powell, R. B., Stern, M. J., & Frensley, B. T. (2023). Which approaches are associated with better outcomes? Evidence from a national study of environmental education field trip programs for adolescent youth in the United States. Environmental Education Research, 29(3), 331–356. https://doi.org/10.1080/13504622.2022.2145270
   Web of Science ®Google Scholar
• Powell, R. B., Stern, M. J., Frensley, B. T., & Moore, D. (2019). Identifying and developing crosscutting environmental education outcomes for adolescents in the twenty-first century (EE21). Environmental Education Research, 25(9), 1281–1299. https://doi.org/10.1080/13504622.2019.1607259
   Web of Science ®Google Scholar
• Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4), 443–459. https://doi.org/10.1177/1098214008324182
   Web of Science ®Google Scholar
• Reich, C., Pattison, S., Olney, V., Bequette, M., & Cohn, S. (2019). Nise Net: Team-based inquiry. In L. U. T. L. W Martin, L. U Tran, & D. Ash (Eds.), The reflective museum practitioner (pp. 53–63). Routledge: Taylor & Francis.
   Google Scholar
• Stern & Powell EE21 Study. (n.a.) https://frec.vt.edu/people/Stern/stern_docs.html
   Google Scholar
• Stocklmayer, S. M., Rennie, L. J., & Gilbert, J. K. (2010). The roles of the formal and informal sectors in the provision of effective science education. Studies in Science Education, 46(1), 1–44. https://doi.org/10.1080/03057260903562284
   Web of Science ®Google Scholar
• Wenger, E. (2011). Communities of practice: A brief introduction.
   Google Scholar
• Wenger, E., McDermott, R. A., & Snyder, W. (2010). Cultivating communities of practice: A guide to managing knowledge. Harvard Business School Press.
   Google Scholar