Can ChatGPT Evaluate Plans?

References

• American Planning Association (APA). (2023). Unleashing the potential of ChatGPT in planning practice. https://www.planning.org/learn/course/9274131/#:∼:text=Utilize%20ChatGPT%20to%20augment%20different,a%20city’s%20real%2Dlife%20experience
   Google Scholar
• Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023). Evaluating the performance of ChatGPT in ophthalmology: An analysis of its successes and shortcomings. Ophthalmology Science, 3(4), 100324. https://doi.org/10.1016/j.xops.2023.100324
   PubMedGoogle Scholar
• Bell, J. B. (2004). Managing evaluation projects. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (pp. 571–603). Wiley.
   Google Scholar
• Berke, P., & Godschalk, D. (2009). Searching for the good plan: A meta-analysis of plan quality studies. Journal of Planning Literature, 23(3), 227–240. https://doi.org/10.1177/0885412208327014
   Web of Science ®Google Scholar
• Berke, P., Smith, G., & Lyles, W. (2012). Planning for resiliency: Evaluation of state hazard mitigation plans under the disaster mitigation act. Natural Hazards Review, 13(2), 139–149. https://doi.org/10.1061/(ASCE)NH.1527-6996.0000063
   Web of Science ®Google Scholar
• Brinkley, C., & Stahmer, C. (2021). What is in a plan? Using natural language processing to read 461 California city general plans. Journal of Planning Education and Research. Advance online publication. https://doi.org/10.1177/0739456X21995890
   PubMed Web of Science ®Google Scholar
• Brinkley, C., & Wagner, J. (2022). Who is planning for environmental justice—And how? Journal of the American Planning Association. Advance online publication. https://doi.org/10.1080/01944363.2022.2118155
   Web of Science ®Google Scholar
• Brody, S. D., & Highfield, W. E. (2005). Does planning work? Testing the implementation of local environmental planning in Florida. Journal of the American Planning Association, 71(2), 159–175. https://doi.org/10.1080/01944360508976690
   Web of Science ®Google Scholar
• Butler, W. H., Deyle, R. E., & Mutnansky, C. (2016). Low-regrets incrementalism: Land use planning adaptation to accelerating sea level rise. Journal of Planning Education and Research, 36(3), 319–332. https://doi.org/10.1177/0739456X16647161
   Web of Science ®Google Scholar
• Chalkidis, I. (2023). ChatGPT may pass the bar exam soon, but has a long way to go for the LexGLUE benchmark. arXiv Preprint, arXiv:2304.12202. https://doi.org/10.48550/arXiv.2304.12202
   Google Scholar
• Chen, L., Zaharia, M., & Zou, J. (2023). How is ChatGPT's behavior changing over time? arXiv Preprint, arXiv:2307.09009. https://doi.org/10.48550/arXiv.2307.09009
   Google Scholar
• Chula Vista. (2011). City of Chula Vista climate adaptation strategies. https://www.chulavistaca.gov/home/showdocument?id=5443
   Google Scholar
• Dalton, L. C., & Burby, R. J. (1994). Mandates, plans, and planners: Building local commitment to development management. Journal of the American Planning Association, 60(4), 444–461. https://doi.org/10.1080/01944369408975604
   Web of Science ®Google Scholar
• Daniel, C. (2023). ChatGPT: Implications for planning. PAS QuickNotes. https://planning-org-uploaded-media.s3.amazonaws.com/publication/download_pdf/PAS-QuickNotes-101.pdf
   Google Scholar
• Denver. (2014). City and county of Denver climate adaptation plan. https://www.denvergov.org/files/assets/public/v/1/climate-action/documents/climate_adaptation_final-with-letter.pdf
   Google Scholar
• Floridi, L., & Chiriatti, M. (2020). GPT-3: Its nature, scope, limits, and consequences. Minds and Machines, 30(4), 681–694. https://doi.org/10.1007/s11023-020-09548-1
   Web of Science ®Google Scholar
• Forsyth, A. (2022). What is planning? A guide for submitting authors. Journal of the American Planning Association, 88(1), 1–2. https://doi.org/10.1080/01944363.2021.1995286
   Google Scholar
• Forsyth, A. (2023). What tense is a plan: Current framework, future vision, or creature of its time? Journal of the American Planning Association, 89(1), 1. https://doi.org/10.1080/01944363.2022.2144700
   Google Scholar
• Fu, X., Gomaa, M., Deng, Y., & Peng, Z. R. (2017). Adaptation planning for sea level rise: A study of US coastal cities. Journal of Environmental Planning and Management, 60(2), 249–265. https://doi.org/10.1080/09640568.2016.1151771
   Web of Science ®Google Scholar
• Fu, X., & Li, C. (2022). How resilient are localities planning for climate change? An evaluation of 50 plans in the United States. Journal of Environmental Management, 318, 115493. https://doi.org/10.1016/j.jenvman.2022.115493
   PubMed Web of Science ®Google Scholar
• Fu, X., Li, C., & Zhai, W. (2023). Using natural language processing to read plans: A study of 78 resilience plans from the 100 resilient cities network. Journal of the American Planning Association, 89(1), 107–119. https://doi.org/10.1080/01944363.2022.2038659
   Web of Science ®Google Scholar
• Fu, X., Sun, B., Frank, K., & Peng, Z. R. (2019). Evaluating sea-level rise vulnerability assessments in the USA. Climatic Change, 155(3), 393–415. https://doi.org/10.1007/s10584-019-02488-5
   Web of Science ®Google Scholar
• Guyadeen, D. (2018). Do practicing planners value plan quality? Insights from a survey of planning professionals in Ontario, Canada. Journal of the American Planning Association, 84(1), 21–32. https://doi.org/10.1080/01944363.2017.1404486
   Web of Science ®Google Scholar
• Guyadeen, D., & Seasons, M. (2016). Plan evaluation: Challenges and directions for future research. Planning Practice & Research, 31(2), 215–228. https://doi.org/10.1080/02697459.2015.1081335
   Web of Science ®Google Scholar
• Hess, D. J., & McKane, R. G. (2021). Making sustainability plans more equitable: An analysis of 50 US cities. Local Environment, 26(4), 461–476. https://doi.org/10.1080/13549839.2021.1892047
   Web of Science ®Google Scholar
• Hoch, C. J. (2002). Evaluating plans pragmatically. Planning Theory, 1(1), 53–75. https://doi.org/10.1177/147309520200100104
   Google Scholar
• Kaiser, E. J., & Godschalk, D. R. (1995). Twentieth century land use planning: A stalwart family tree. Journal of the American Planning Association, 61(3), 365–385. https://doi.org/10.1080/01944369508975648
   Web of Science ®Google Scholar
• Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274
   Web of Science ®Google Scholar
• Khakee, A. (2000). Reading plans as an exercise in evaluation. Evaluation, 6(2), 119–136. https://doi.org/10.1177/13563890022209172
   Google Scholar
• Kim, H., & Marcouiller, D. W. (2018). Mitigating flood risk and enhancing community resilience to natural disasters: Plan quality matters. Environmental Hazards, 17(5), 397–417. https://doi.org/10.1080/17477891.2017.1407743
   Web of Science ®Google Scholar
• Knaap, G. J., Ding, C., & Hopkins, L. D. (2001). Do plans matter? The effects of light rail plans on land values in station areas. Journal of Planning Education and Research, 21(1), 32–39. https://doi.org/10.1177/0739456X0102100103
   Web of Science ®Google Scholar
• Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35, 22199–22213. https://proceedings.neurips.cc/paper_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html
   Google Scholar
• Krizek, K., Forysth, A., & Slotterback, C. S. (2009). Is there a role for evidence-based practice in urban planning and policy? Planning Theory & Practice, 10(4), 459–478. https://doi.org/10.1080/14649350903417241
   Google Scholar
• Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health, 2(2), e0000198. https://doi.org/10.1371/journal.pdig.0000198
   PubMedGoogle Scholar
• Laurian, L., Crawford, J., Day, M., Kouwenhoven, P., Mason, G., Ericksen, N., & Beattie, L. (2010). Evaluating the outcomes of plans: Theory, practice, and methodology. Environment and Planning B: Planning and Design, 37(4), 740–757. https://doi.org/10.1068/b35051
   Web of Science ®Google Scholar
• Laurian, L., Day, M., Berke, P., Ericksen, N., Backhurst, M., Crawford, J., & Dixon, J. (2004). Evaluating plan implementation: A conformance-based methodology. Journal of the American Planning Association, 70(4), 471–480. https://doi.org/10.1080/01944360408976395
   Web of Science ®Google Scholar
• Li, C., & Song, Y. (2016). Government response to climate change in China: A study of provincial and municipal plans. Journal of Environmental Planning and Management, 59(9), 1679–1710. https://doi.org/10.1080/09640568.2015.1085840
   Web of Science ®Google Scholar
• Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2021). GPT understands, too. AI Open. https://doi.org/10.48550/arXiv.2103.10385
   Google Scholar
• Loh, C. G., & Kim, R. (2021). Are we planning for equity? Equity goals and recommendations in local comprehensive plans. Journal of the American Planning Association, 87(2), 181–196. https://doi.org/10.1080/01944363.2020.1829498
   Web of Science ®Google Scholar
• Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023). ChatGPT and a new academic reality: Artificial intelligence‐written research papers and the ethics of the large language models in scholarly publishing. Journal of the Association for Information Science and Technology, 74(5), 570–581. https://doi.org/10.1002/asi.24750
   Web of Science ®Google Scholar
• Lyles, W., & Stevens, M. (2014). Plan quality evaluation 1994–2012: Growth and contributions, limitations, and new directions. Journal of Planning Education and Research, 34(4), 433–450. https://doi.org/10.1177/0739456X14549752
   Web of Science ®Google Scholar
• Neuman, M. (1998). Does planning need the plan? Journal of the American Planning Association, 64(2), 208–220. https://doi.org/10.1080/01944369808975976
   Google Scholar
• OpenAI. (2022). OpenAI ChatGPT. https://openai.com/blog/chatgpt
   Google Scholar
• Polak, M. P., & Morgan, D. (2023). Extracting accurate materials data from research papers with conversational language models and prompt engineering. arXiv Preprint, arXiv:2303.05352. https://doi.org/10.48550/arXiv.2303.05352
   Google Scholar
• Reckien, D., Flacke, J., Dawson, R. J., Heidrich, O., Olazabal, M., Foley, A., Hamann, J. J.-P., Orru, H., Salvia, M., De Gregorio Hurtado, S., Geneletti, D., & Pietrapertosa, F. (2014). Climate change response in Europe: What’s the reality? Analysis of adaptation and mitigation plans from 200 urban areas in 11 countries. Climatic Change, 122(1–2), 331–340. https://doi.org/10.1007/s10584-013-0989-8
   Web of Science ®Google Scholar
• Ryan, B. D. (2011). Reading through a plan: A visual interpretation of what plans mean and how they innovate. Journal of the American Planning Association, 77(4), 309–327. https://doi.org/10.1080/01944363.2011.616995
   Web of Science ®Google Scholar
• Shi, L., Chu, E., & Debats, J. (2015). Explaining progress in climate adaptation planning across 156 US municipalities. Journal of the American Planning Association, 81(3), 191–202. https://doi.org/10.1080/01944363.2015.1074526
   Web of Science ®Google Scholar
• Stevens, M. R., Lyles, W., & Berke, P. R. (2014). Measuring and reporting intercoder reliability in plan quality evaluation research. Journal of Planning Education and Research, 34(1), 77–93. https://doi.org/10.1177/0739456X13513614
   Web of Science ®Google Scholar
• Talen, E. (1996). Do plans get implemented? A review of evaluation in planning. Journal of Planning Literature, 10(3), 248–259. https://doi.org/10.1177/088541229601000302
   Google Scholar
• Teebagy, S., Colwell, L., Wood, E., Yaghy, A., & Faustina, M. (2023). Improved performance of ChatGPT-4 on the OKAP exam: A comparative study with ChatGPT-3.5. medRxiv 2023-04. https://doi.org/10.1101/2023.04.03.23287957
   Google Scholar
• Wang, J., Shi, E., Yu, S., Wu, Z., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., Hu, H., & Yue, C. (2023). Prompt engineering for healthcare: Methodologies and applications. arXiv Preprint, arXiv:2304.14670. https://doi.org/10.48550/arXiv.2304.14670
   Google Scholar
• Wheeler, S. M. (2008). State and municipal climate change plans: The first generation. Journal of the American Planning Association, 74(4), 481–496. https://doi.org/10.1080/01944360802377973
   Web of Science ®Google Scholar
• White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023). A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv Preprint, arXiv:2302.11382. https://doi.org/10.48550/arXiv.2302.11382
   Google Scholar
• Wray, S. (2023). Cities explore the potential of ChatGPT. CitiesToday. https://cities-today.com/cities-explore-the- potential-of-chatgpt/
   Google Scholar
• Zhong, Q., Ding, L., Liu, J., Du, B., & Tao, D. (2023). Can ChatGPT understand too? A comparative study on ChatGPT and fine-tuned BERT. arXiv Preprint, arXiv:2302.10198. https://doi.org/10.48550/arXiv.2302.10198
   PubMedGoogle Scholar