Inter-rater reliability of the ‘Tool for assessing determinants of health in public space’ in a co-creative urban design process with care home residents in Barcelona: a Health CASCADE study

ABSTRACT

This study aims to evaluate the inter-rater reliability of the Tool for assessing determinants of health in public space, a methodology developed to assess the potential health impact of public space improvement actions. The study involved a participatory evaluation conducted during an urban design co-creation process with older adults and researchers from various disciplines. Ten older adults from a care home in Barcelona and five researchers participated in the evaluation of five selected public spaces using the tool. An analysis using Intra-class Correlation Coefficient (ICC) was conducted to assess inter-rater reliability. A descriptive comparison of the results was performed based on the scores given to each determinant. The findings provide insights into the tool’s reliability and its potential for capturing diverse perspectives. The Tool does not appear to have high reliability when implemented in a community setting. The ICC values for most of the determinants are relatively low, indicating poor or moderate levels of agreement among the raters/judges. The study highlights the importance of involving end-users and researchers in the evaluation process, emphasizing the need for user-centered design and co-creation in urban planning. The results contribute to improving public health outcomes and promoting age-friendly environments in urban spaces.

KEYWORDS:

• Co-creation
• co-evaluation
• urban design
• aging
• older adults
• urban health

Introduction

Population aging is becoming a major challenge in many countries. According to European Union data, in 2060, about 30% of people will be aged older than 65, and 12% will be 80 years or older. Regarding the urban context, governments, authorities, politicians, and economists must change their approach in relation to cities’ development and management, especially public spaces (Jablonska and Trocka-Leszczynska 2020). Age-friendly cities are those that prioritize the well-being and inclusivity of their aging populations, creating environments where older adults can lead fulfilling lives. These cities recognize the unique needs and contributions of older citizens and adapt their urban planning to ensure accessibility, safety, and social engagement for this segment of the population (Rémillard-Boilard et al. 2021). Age-friendly cities recognize that population aging is not a burden but an opportunity. They understand that older adults are a valuable resource with a wealth of knowledge, skills, and experience to offer. By creating inclusive and supportive urban environments, these cities ensure that people of all ages can thrive and continue to contribute to their communities, fostering a more vibrant and sustainable future for everyone (World Health Organization 2007).

Vulnerable groups such as older adults, are more likely to experience difficulties accessing spaces for physical activity and interaction. These difficulties are even more pronounced in older adults who live in long-term care settings (World Health Organization 2007). During the aging process, physical activity levels decrease (Suryadinata et al. 2020), thereby increasing the likelihood of individuals developing metabolic disorders and other chronic diseases, such as cancer, diabetes, and cerebrovascular and cardiovascular diseases. While physical activity gradually declines with age, muscle mass, and strength decrease. Studies showed that an increase in physical activity tends to lower the risk of having cognitive disorders and improves overall well-being and quality of life (Notthoff et al. 2017, Gopinath et al. 2018). High levels of physical activity have positive effects on the physical, social, and emotional dimensions of the older adult population (Kirk-Sanchez and McGough 2014). Recent studies have also shown that older adults are the most sedentary segment of the population (Giné-Garriga et al. 2020).

Inclusive urban spaces are essential for achieving age-friendly cities (World Health Organization 2018). Ensuring more opportunities to participate in public spaces actively can provide pathways for all age groups to maintain a healthy lifestyle according to their ability (World Health Organization 2021). Several studies have highlighted the importance of including older adults in co-creating and evaluating urban spaces in the contexts of well-being perception, health promotion, physical activity, and social interaction. Baquero Larriva and Higueras (2020) emphazise the need to include older adults in the design and evaluation of urban spaces to mitigate the effects of climate change and urbanization on health. Baquero Larriva & Higueras’ paper highlights the importance of their perspectives, preferences, and needs to create age-friendly environments. Rémillard-Boilard et al. (2021) also bring attention to the significance of collaborative governance and involving older adults in the co-design of age-friendly cities.

In urban planning practice, we frequently encounter barriers to community engagement, such as a tendency to use consultation methods over very short time frames (Fudge et al. 2020), which endanger meaningful participation. In this evolving landscape, tools for assessing public spaces become essential instruments to ensure that urban environments are age-friendly and promote health and well-being (Dempsey et al. 2011), allowing for a comprehensive evaluation of the quality and usability of these spaces. There is a need for tools that can capture the unique experiences and insights of older adults. Sixsmith et al. (2017) explore the role of older adults in co-designing urban spaces to age-in-place and discuss the importance of tools and approaches that engage older adults in evaluating and transforming urban environments to meet their specific needs.

Several tools have been developed for evaluating urban spaces and their impact on health. These include: (a) Health Impact Assessment (HIA) which evaluates and predicts health impacts, and informs decision-making for healthier urban projects (Kemm 2013); (b) Public Space Public Life (PSPS) methodology that assesses urban spaces, aims to improve the quality of life and guides inclusive design (Gehl and Svarre 2013); (c) The Urban HEART tool, developed by the World Health Organization, aims to assess and address health inequities in urban areas by evaluating the social determinants of health and guiding targeted interventions for achieving urban health equity (Novoa et al. 2018); (d) Placemaking Standard Tool that enhances urban spaces through comprehensive standards and guidelines and provides an intuitive method for a variety of users (Hasler and Howie 2020); and (e) Healthy Cities Tool that Supports planning and policies for healthier urban environments, promoting well-being (Healthy Cities 2023). Overall, the existence of multiple tools contributes to a more nuanced understanding of urban health and enables evidence-based decision-making and interventions to create healthier and more equitable cities.

The aforementioned tools have limitations and a potential lack of reliability. A common problem includes relying on available urban data which may be limited or outdated, potentially affecting the accuracy and validity of the evaluation. Choosing the appropriate tool can vary depending on the local context, demographics, and cultural factors. Most tools use subjective assessments, reflecting to a certain extent the priorities of the users (e.g. perceived safety or perceived green space quality), which can introduce biases and affect the reliability and objectivity of the evaluation (Nguyen et al. 2021). In addition, the complexity and technical nature of these tools may limit their accessibility and understanding for the general population, hindering their widespread use and participation. Thus, the technical expertise, data interpretation skills, and resources required to use these tools can create barriers to meaningful engagement and participation from the public.

Cities may also choose to create their own tools for evaluating urban spaces and their impact on health to customize the evaluation process according to their specific needs, engage local stakeholders, address unique urban health challenges, and build upon existing research and knowledge, while adopting internationally recognized standards. This is the case of the ‘Tool for assessing determinants of health in public space’, originally designed for experts but with an aim to provide the possibility to be extended to communities. Our decision to test it with community members, particularly elderly users, warrants a deeper exploration of the circumstances and motivations. In the context of the co-creation project: EngAGE4Change, conducted over 12 months, a group of ten older adults have been discussing and evaluating a set of five public spaces in the Dreta de l’Eixample neighborhood in Barcelona. As part of this process the co-creators, along with a group of five researchers implemented the Tool for assessing determinants of health in public space for the selected spaces. This initiative emerged from a confluence of factors. Firstly, the traditional top-down urban planning approaches often neglect the unique needs and preferences of older adults, who may face barriers to accessing and utilizing public spaces (Jablonska and Trocka-Leszczynska 2020). Secondly, the prevalence of physical inactivity among older adults poses serious health challenges (Suryadinata et al. 2020). Thirdly, previous tools designed for expert use had limitations in engaging and involving community members effectively (Nguyen et al. 2021).

In response to these challenges, our study aims to test the tool in a community setting while bridging the gap by involving older adults in the evaluation of public spaces. This approach aligns with our goal of evaluating the tool’s potential for broader community use and its suitability for capturing diverse perspectives, even without specific adaptations for non-expert users. It also aligns with the principles of co-creation and user-centered design, which emphasize active involvement and empowerment of end-users in shaping their environments (Leask et al. 2019). It is important to critically evaluate and interpret the findings from these tools while considering the unique characteristics and needs of each urban area and target population to ensure a comprehensive understanding of the relationship between urban spaces and health. Moreover, it is critical to include communities and stakeholders as active agents in the evaluation process, within a co-creation paradigm (Leask et al. 2019).

The Tool for assessing determinants of health in public space, developed by Diputació de Barcelona in partnership with the Barcelona Institute for Global Health (IS Global), is the first local tool developed to assess complex urban dimensions with a focus on the determinants of health. The tool can be applied to existing spaces as well as projects, allowing the user to evaluate the current determinants of health or to predict them to iterate a proposal accordingly. The user guide of the tool (About the Tool | Espai Públic i Salut n.d.) states that its use can be extended to communities, although it was originally planned to be used by individuals, technicians, and experts in the urban planning field.

The tool’s reliability has not yet been assessed and considering the intention stated in the user guide, the growing importance of community involvement in urban planning and public health, and the specific challenges faced by older adults, we decided to explore the tool’s applicability beyond its original scope. By testing the tool for use with community members, including older adults, and conducting a co-creation process with researchers from various disciplines, we aim to ensure a more inclusive and comprehensive evaluation of public spaces from the perspective of those who directly use and benefit from these environments. Additionally, using a local tool is of paramount importance because it ensures that the assessment of determinants of health is context-specific and tailored to the unique needs and characteristics of the community in question. Local tools are designed with a better understanding of the local environment, culture, and challenges, which makes them more relevant and accurate in capturing the intricacies of public health in that particular area.

Incorporating the community in the usage of this tool can significantly enhance its effectiveness and impact. Local residents possess firsthand knowledge and experiences of their surroundings, offering invaluable insights that may not be fully grasped by technicians alone. By involving the community, the tool becomes a shared resource, fostering a sense of ownership and empowerment among the residents, thus encouraging active participation in improving their public spaces and overall health outcomes. Moreover, a co-creation process involving end-users (i.e. older adults) and multidisciplinary researchers (i.e. urban planning, architecture, aging, physical activity, evaluation, and implementation) brings diverse perspectives and expertise to the table. This collaborative approach allows for diverse perspectives and user-centered design (Leask et al. 2019), creating a dynamic environment to test the tool’s reliability. This approach enhances the tool’s quality and applicability, ultimately improving public health outcomes. As a result, the tool’s functionalities and applicability can be refined and expanded, ensuring it captures a comprehensive range of determinants of health, making it a more comprehensive and robust instrument.

This article aims to assess the inter-rater reliability of the Tool for assessing determinants of health in public space measuring the degree of agreement among independent observers in a participatory evaluation, involving researchers from several disciplines and end-users, carried out during an urban design co-creation process.

Methods

For this study, the ‘Tool for assessing determinants of health in public space’, originally designed for experts, was used in its original form. We conducted a participatory evaluation involving community members from a care home in Barcelona, and five researchers from various disciplines. The decision to use the tool in its original form allowed us to assess its inter-rater reliability under real-world community conditions. This approach aimed to provide insights into the tool’s performance without alterations for non-expert users and its potential for capturing diverse perspectives within the community. This study is embedded within a year-long co-creation process aimed at fostering collaborative insights and shared ownership amongst varied stakeholders and community members. Co-creation not only accommodate diverse timeframes and nonlinear processes (Leask et al. 2019, van Vries et al. 2020) but also ensures a detailed, comprehensive evaluation (Lazo-Porras et al. 2020), leveraging the synergy of community experiences and academic insights to refine and optimize the tool for real-world applicability (Messiha et al. 2023).

Sample

Ten older adults (10 older adults − 8 female and 2 male, mean age 85 years (sd 7.25)) from one selected care home in the Dreta de l’Eixample neighborhood in Barcelona. The selected care home participates in the EngAGE4Change – Health CASCADE, a co-creation project focused on urban design with the older adult population. This care home was recruited after conducting a spatial care home distribution study across Barcelona municipal boundaries in the first stage of the project. All residents were eligible and the only exclusion criteria were having severe cognitive impairment, end-stage disease, or reporting discontinuation of their participation for a 9-month period. A group of 10 residents that volunteered to participate as co-creators in EngAGE4Change were invited to an informative session. The study was approved by the Ethics Committee of the Faculty of Psychology, Education and Sport Sciences Blanquerna and all participants signed an informed consent prior to their participation. In November 2022, as part of the EngAGE4Change project we conducted one community walk audit, and several participatory workshops to identify and generate discussion about existing public spaces in the Dreta de l’Eixample neighborhood that concluded in the selected spaces for this study. The group identified public spaces within a 10-minute walking radius from their residence (care home) and selected different typologies to explore, document, and evaluate (Figure 1).

Figure 1. EngAGE4Change-Health CASCADE project community walk audit map and selected public spaces for evaluation in the Dreta de l’Eixample neighborhood in Barcelona. Data: Open Data BCN, 2022. Map: J. R. Zapata-Restrepo.

The identified public spaces and typologies were:

• Jardins de Jaume Perich (Center-block garden/plaza)

• Jardins del Carlit (Center-block garden/plaza)

• Tetuan Square – Jardins del Doctor Robert (Neighborhood-scale park)

• Carretera Antiga d’Horta (Inner block path/plaza)

• Passeig Sant Joan (Green corridor)

For this inter-reliability study, the participants included: 2 groups of 5 care home residents each (8 female and 2 male) to be able to apply the tool in a small setting with the assistance of one researcher; 4 researchers (3 female) from the Fundació Blanquerna-URL who participated in the community walk held with profiles oriented towards physical activity, sports, and co-creation; and 1 researcher (female) who did not participate in the walk and whose profile focuses on architecture, urbanism, and sports.

A small, localized group like the one in this study allows for deeper, more nuanced discussions and insights. It provides the opportunity to delve into the specific challenges, desires, and unique characteristics of a given community or demographic group. This tailored approach (Leask et al. 2019) aims to create more effective, contextually appropriate interventions or recommendations, even if those findings may not be as easily generalizable to other settings.Therefore, while the sample size may limit the broad applicability of the study’s findings, it is conducive to the highly specialized, context-sensitive goals of the EngAGE4Change project.

This smaller, more focused approach can provide invaluable, in-depth knowledge that could be lost in a larger, more generalized study. Furthermore, perception of public spaces is a highly subjective experience influenced by a multitude of factors (Ho and Au 2020). These can range from personal preferences and past experiences to cultural norms and even physiological conditions such as mobility constraints or sensory limitations (Levy-Storms et al. 2018). As the study is focused on a specific subset of older adults in a particular locale, the perspectives and evaluations they bring to the table are inherently shaped by these diverse factors. These may include local cultural norms about public spaces, as well as individual factors like health status, prior experiences, and personal preferences.

Tool for assessing determinants of health in public space

The tool is based on the objective of evaluating how healthy the public spaces of our cities are and what conditions we should avoid or promote in order to generate an urban environment with a vision of health. The focus on the determinants of health is a key factor to be considered in urban transformation processes, which was an interesting and necessary approach to be included in our project’s co-creation process.

The User Guide of the tool (About the Tool | Espai Públic i Salut n.d.) presents a methodology developed to assess the potential health impact of public space improvement actions. It highlights the importance of the built environment and its influence on people’s well-being. The methodology was developed by the Public Health Service and the Public Space and Facilities Service of the Barcelona Provincial Council. It draws inspiration from the Place Standard Tool used in Scotland (The Place Standard Tool Is a Way of Assessing Places. | Our Place n.d.) and combines perspectives on health and a scientific approach based on literature highlighting which elements of public space have a direct impact on health. The methodology categorizes determinants of health-related to public spaces, such as environmental exposure, community, comfort and attractiveness, access and connectivity, safety, maintenance and sanitation, and uses (Figure 2). It provides a questionnaire format with multiple-choice questions to assess the degree of compliance with each determinant and condition.

Figure 2. Development of a methodology to assess the potential health impact of actions and improvements to a public space. Source: Diputació de Barcelona., Laura Hidalgo, and ISGlobal (Tool for Assessing Determinants of Health in Public Space | Espai Públic i Salut n.d.).

The tool can be used in various phases of public space improvement projects and allows for comparisons over time. The results are presented in spider diagrams to provide a general understanding of the space’s current state and identify areas that require improvement to maximize health benefits (Figure 3).

Figure 3. Example of tool-generated spider diagrams showing the relations among the determinants of health. This diagram corresponds to one of the researchers that used the tool in the study to evaluate the same set of public spaces. Data: Tool for assessing determinants of health in public space (Questionnaire | Espai Públic i Salut n.d.).

The motivation behind this methodology stems from the understanding that people’s health is influenced not only by individual factors but also by broader environmental, economic, cultural, and social determinants. Based on the World Health Organization’s definition of health as not merely the absence of disease but the complete physical, mental, and social well-being, it is crucial to ensure that the urban environment facilitates the achievement of this well-being (Health and Well-Being n.d.). The methodology employs a questionnaire format that includes a total of 107 questions (see Supplementary Appendix – Questionnaire), designed to assess public spaces such as parks, green areas, and squares. These questions address each determinant of health and its associated conditions, enabling planners to gauge the extent to which a given space promotes health benefits. The questionnaire reflects the complexity of urban reality and includes diverse question types to encompass varying aspects of space assessment (About the Tool | Espai Públic i Salut n.d.). The questions utilize a scoring system, typically on a scale of 1, 3, and 5, with higher scores indicating greater alignment with health-enhancing criteria. Some questions also offer optional scores of 0 or additional information to account for the complexity of urban realities. The user guide of the tool provides an explanation of the scoring system, question types and assessment of the questionnaire (see Supplementary Appendix – Scoring System).

Procedure

• Establish two (2) groups of co-evaluators made up of 5 residents of the selected care home who conducted a community walk through five (5) public spaces in the Dreta de l’Eixample (Figure 1).

• Use the tool in groups to rate each public space (Figure 4). The decision of conducting the application of the tool with the residents in two groups comes from facilitating the understanding of the questions due to the technical language. In addition, most residents are older adults with different levels of hearing impairment so facilitating the questionnaire in a small group setting is key to assure understanding and facilitating discussion among the residents.

  Figure 4. Set of evaluators that applied the tool for assessing determinants of health in public space in the context of the EngAGE4Change-Health CASCADE project. Graphic: J. R. Zapata-Restrepo.

  

• Recruit four (4) researchers from Fundació Blanquerna-Ramon Llull University who attended the community walk to use the tool to rate each public space.

• Recruit one (1) Fundació Blanquerna-Ramon Llull University researcher who was not involved in the community walk to make use of the tool to think about the spaces we visited from an external perspective.

• Collect the reports generated when completing the use of the tool, and make a descriptive comparison of the results taking into account the final scores of the determinants and the scores given to each of the variables of each determinant (Figure 3).

Data analysis

Statistical analysis was performed using JASP free statistical software (version 0.17.2.1). The results are expressed as mean and standard deviation or proportions. Inter-rater reliability was examined by calculating the Intra-class Correlation Coefficient (ICC) of the scores of each determinant and subcategory of the ‘Tool for assessing determinants of health in public space’.

The ICC is a value between 0 and 1, where values below 0.5 indicate poor reliability, between 0.5 and 0.75 moderate reliability, between 0.75 and 0.9 good reliability, and any value above 0.9 indicates excellent reliability (Koo and Li 2016).

ICC 3,1 type corresponds to a two-way mixed single measure that evaluates consistency and absolute agreement as referenced by (Shrout and Fleiss 1979). This ICC type was used for the scores assigned to each of the Subcategories under the Determinants of Health (Table 1).

Table 1. Intraclass Correlation (ICC), Tool for assessing determinants of health in public space.

Determinants of Health	Number of Subcategories	ICC 3,1 Subcategories	Lower 95% CI	Upper 95% CI	Inter-rater Reliability (Koo and Li 2016)
Environmental Exposure	8	0.244	0.034	0.650	Poor
Safety	7	0.164	−0.022	0.610	Poor
Maintenance and Sanitation	6	0.287	0.036	0.765	Poor
Access and connectivity	2	0.723	0.357	0.975	Moderate
Uses	7	0.427	0.178	0.764	Poor
Community	3	0.231	−0.012	0.784	Poor
Comfort and Attractiveness	5	0.460	0.177	0.830	Poor

Results

Table 1 presents the Intraclass Correlation (ICC) coefficients representing the inter-rater reliability of a tool designed for assessing various determinants of health in public spaces. These determinants, integral in understanding public health, are categorized into seven distinct aspects, namely: Environmental Exposure, Safety, Maintenance and Sanitation, Access and Connectivity, Uses, Community, and Comfort and Attractiveness. Each determinant consists of different numbers of subcategories and is assessed for reliability using ICC (3,1), with results ranging from poor to moderate as classified by Koo and Li (2016). The lower and upper 95% Confidence Intervals (CI) for each determinant are also provided to depict the precision of the ICC estimates.

The results highlight inconsistent reliability across different categories of the Tool for assessing determinants of health in public spaces. Importantly, only the Access and Connectivity category reached a moderate level of inter-rater reliability (see Table 1), suggesting that raters found this area easier to evaluate consistently. In stark contrast, most other categories demonstrated poor agreement among raters, raising concerns about the tool’s overall reliability and effectiveness when used by non experts.

Determinants with the highest agreement (highest ICC values)

Access and connectivity

This category had the highest ICC values of 0.723 (ICC 3,1). It indicates a moderate level of agreement among the raters/judges regarding the determinants of health-related to access and connectivity in public spaces.

Comfort and attractiveness

The Comfort and Attractiveness category showed ICC values of 0.460 (ICC 3,1). Although, it suggests a poor level of agreement among the raters/judges regarding the determinants of health-related to comfort and attractiveness in the evaluated public spaces.

Determinants with the lowest agreement (lowest ICC values)

Safety

The Safety category had relatively low ICC values of 0.164 (ICC 3,1). It indicates a poor level of agreement among the raters/judges when assessing the determinants of health-related to safety in public spaces.

Community

The Community category also exhibited lower ICC values of 0.231 (ICC 3,1). It suggests a poor level of agreement among the raters/judges regarding the determinants of health-related to community aspects in the evaluated public spaces.

Discussion

The present study shows that the agreement between raters using the Tool for assessing determinants of health in public space was overall poor for most of the determinants analyzed. The intricacy of the tool’s interface presented potential challenges for both community members and older adults, potentially impeding their ability to effectively navigate and utilize its functionalities. Consequently, the reliance on external assistance may have introduced inconsistencies in interpretation and usage, ultimately contributing to reduced levels of consensus among users. To address these challenges and enhance the tool’s usability in community settings, particularly for older adults, it is imperative to contemplate the development of a more user-friendly iteration of the tool. Such an enhanced version should aspire to alleviate cognitive burdens, eliminate potential participation barriers, and ensure that the tool is readily comprehensible to community members.

To achieve this goal, it is crucial to incorporate the perspectives of both experts and community members. Establishing a common understanding of the tool’s purpose and usage can pave the way for more inclusive and age-friendly public space design and policies. The creation of a user-friendly version of the tool is pivotal for its practical application in real-world practice and policy-making, enabling a broader spectrum of stakeholders, including community members and older adults, to actively partake in the assessment of public spaces. It is imperative not only to amalgamate insights from professionals and local inhabitants but also to meticulously consider varying timeframes to ensure the relevance and applicability of the tool in diverse contexts, potentially accommodating evolving community needs (van Vries et al. 2020).

For Environmental Exposure, Community, Safety, Maintenance and Sanitation, Comfort and Attractiveness, and Uses, the agreement among raters was poor. This aligns with findings from previous studies by (Phillips et al. 2013) and (Hand et al. 2020), which also noted inconsistencies in the assessment of similar determinants in public spaces. Conversely, the determinant of Access and Connectivity witnessed moderate agreement among raters, which highlight this category as an intuitive subject for users to evaluate, prioritizing experience over knowledge. Studies such as (Barnett et al. 2020) and (Noreau et al. 2004) suggest that provision of good access to neighbourhood destinations is important for older adults’ neighbourhood satisfaction and, thus, their health.

There is a need to improve the tool’s reliability and establish clear guidelines for consistent assessment, particularly when using the tool in a community setting to achieve higher agreement scores. This will ensure more accurate and representative data that can inform decision-making processes related to urban planning and the creation of age-friendly cities. This discussion section highlights the differences in the perspectives of the groups of older adults and researchers that ultimately affected the scores given to each of the topics presented by the tool.

Perspectives differed among participants in the Environmental Exposure determinant. Environmental exposure refers to non-tangible elements in the surrounding air that can affect health, such as thermal comfort, noise, odors, air quality, lighting, and radiation. A study conducted by (Gobbens and Van Assen 2018) showed that multiple environmental factors are associated with quality of life in older adults. When approaching this topic, academic researchers focused on objective measures of environmental exposure, such as air pollution levels or noise levels, and their potential health effects. In contrast, our study found that older adults in the sample perceived and prioritized environmental exposure based on their subjective experiences and memories. They were more attuned to issues like excessive noise, poor air quality, or inadequate lighting, which directly impact their daily comfort and well-being. As pointed by (Baquero Larriva and Higueras 2020) air temperature is the most related to older people’s thermal comfort. Chiarini et al. (2020) highlights that subjective perceptions can be employed jointly with objective indicators to obtain full and comprehensive knowledge of the quality of urban environments.

The determinant of Safety presented differences from the way researchers and older adults approached the topic. Brenner and Clarke (2019) point out physical condition and safety of the immediate outdoor home and neighborhood environment as critical aspects for maintaining independence and well-being for older adults. Safety in public spaces includes factors such as lighting, risk management, the perception of safety, materials, and vegetation, and the distance to traffic. Academic researchers discussed design features that deter criminal activity, and the impact of safety perceptions on public space utilization. Older adults were more concerned about personal safety, similar to Kimic and Polko (2022) study, they prioritized specific needs and preferences regarding safety in public spaces, such as well-lit areas, clear visibility, the absence of physical hazards or barriers, and an overall sense of security to feel comfortable and confident while using the space.

Maintenance and Sanitation were perceived differently among the raters. Several studies indicate the importance of well-mantained neighborhood environment for older people health and functioning (Yen et al. 2009, Akinci et al. 2021). Maintenance and sanitation entail the cleanliness, upkeep, and functionality of public spaces, ensuring that they are safe and free from health hazards. Academic researchers focused on the impact of maintenance standards, waste management practices, and cleanliness on user satisfaction and health outcomes. Older adults placed greater importance on the cleanliness and hygiene of public spaces. They valued well-maintained facilities, regular cleaning, proper waste disposal, and accessibility to restrooms (Barnett et al. 2020). observational study suggests that provision of good access to neighbourhood destinations is important for older adults’ health.

Both academic researchers and older adults found common ground in their agreement on the importance of the Accessibility and Connectivity determinant. Access and connectivity encompassed the physical and social dimensions of how public spaces were linked to their surroundings and the extent to which people could easily enter and move within them. Academic researchers recognized the significance of inclusive design and the need for public spaces to be easily accessible to people of all abilities. They emphasized factors such as proximity to public transportation, ensuring walkability, and the availability of ramps and elevators to enable seamless movement for everyone. Similarly, older adults, who often face mobility limitations, placed a high priority on accessibility when it came to public spaces. Similar to the results observed in Phillips et al. (2013) older pedestrians’ study, they expressed the need for well-maintained pathways that were free from obstacles, allowing them to navigate with ease. Benches strategically placed along the pathways provided resting spots for older adults to catch their breath or take a moment to enjoy the surroundings. Clear signage was crucial for them to navigate and find their way without confusion, ensuring a stress-free experience.

There was disagreement about the determinant of Uses, focusing on the appropriateness of public spaces for varied activities and user groups. Researchers focused on the diversity of programmed activities in public space, assessing their inclusivity and adaptability for different needs and ages. Older adults favored spaces allowing socialization and relaxation in shaded, quiet areas with tailored amenities. A contention in defining Uses was balancing active and passive recreational activities. Researchers argued for prioritizing active pursuits like sports to enhance physical health, emphasizing the need for related facilities, which aligns with international standards and recommendations for age-friendly cities (World Health Organization 2018). Similar to the needs and preferences highlighted by Levy-Storms et al. (2018), older adults advocated for spaces promoting relaxation and connection with nature, emphasizing the significance of passive activities for mental well-being.

Views were diverse regarding the Community determinant. The community aspect of public spaces emphasizes the involvement of local people and the social function of these spaces. Academic researchers focused on looking at factors like social cohesion, sense of belonging, and opportunities for interaction. Older adults, with their accumulated life experiences, expressed a deep understanding of the importance of community in public spaces. They valued spaces that foster social connections, encourage intergenerational interactions, and provide opportunities for active participation and engagement within their communities. These aspects are social determinants for older adult engagement in urban design, as pointed out by Brüchert et al. (2021). The group of older adults expressed that there has been a lack of effort from decision-makers to involve older adults in urban transformation processes in the neighborhood. The absence of their involvement not only hindered the potential for intergenerational collaboration and mutual learning but also hindered the chance to tap into their wisdom and valuable perspectives (Dabelko-Schoeny et al. 2020).

Priorities regarding the Comfort and Attractiveness determinant varied among the group of researchers and older adults. Comfort and attractiveness of public spaces encompass design and integration with the environment, overall aesthetic appeal, educational aspects, and the presence of natural elements and heritage (Akinci et al. 2021). Academic researchers studied elements of urban design, landscape architecture, and aesthetics to understand how these factors impacted the users’ experience and well-being. They explored concepts like urban livability, visual preferences, and the impact of natural environments on mental health. Older adults’ perceptions revolved around the availability of seating, shade, cleanliness, accessibility, and a sense of tranquility, as well as the significance of the space. The latter was related to personal memories, as well as historical and cultural values tied to the space. Subjective perception, as pointed out by Ho and Au (2020) is integral as it molds the priorities and assessments of different individuals or groups, leading to a multifaceted understanding of what constitutes comfort and attractiveness in public spaces. The diverse perceptions contribute to a richer, more nuanced comprehensive evaluation of public space design and usage (Hand et al. 2020), accommodating a broader range of needs and preferences.

Strengths and limitations

Strengths

This study is a critical endeavor, offering nuanced insights into the varying perspectives between academic researchers and older adults, paving the way for user-centric refinements to the tool. The detailed exploration provided by this study serves as a foundational base for enhancing the inclusivity and relevancy of the tool, emphasizing the importance of embracing diverse user perspectives, particularly from older adults, in assessing public spaces. The delineation of varied priorities and considerations across different determinants underscores the need for a comprehensive and multifaceted approach to public space assessment, promoting a balanced integration of experiential and objective evaluations.

The utilization of a small sample size in this study is particularly relevant and important. It allowed for a more in-depth, qualitative understanding of the perspectives and experiences of the participants, enabling the collection of rich, detailed data. This approach is crucial when exploring subjective experiences and perceptions, as it facilitates a closer examination of the nuances and complexities inherent in individual perspectives, contributing to the development of a more refined and user-centered assessment tool.

Limitations

The overarching limitation stems from the discerned low inter-rater reliability across several crucial determinants. This raises crucial concerns about the tool’s comprehensiveness and precision, pointing to the immediate need for clearer guidelines and addressing the subjective dimensions in evaluations like safety and community engagement. The current complexity and the potential ambiguities in the tool’s guidelines have manifested in inconsistencies and varied interpretations, underlining the importance of enhancing the tool’s clarity and coherence.

The low inter-rater reliability in key areas like Safety and Community is particularly concerning. These are critical dimensions for assessing public spaces, especially from the perspective of older adults and vulnerable populations. The results collectively point to a need for further refinement of the tool, especially in areas that showed poor inter-rater reliability. The inconsistencies may be due to a variety of factors, including ambiguous guidelines for assessment in the community setting or the complexity and subjectivity involved in evaluating determinants like safety and community engagement. Such unreliable outcomes signal an urgent need for tool improvement for its application in a community setting, but it also open pathways for meaningful discussion within co-creation processes to find common ground over the different perspectives of the raters.

The tool has the potential to be a valuable asset for evaluating the health determinants of public spaces, its current form appears to lack the reliability required for dependable application. The study underscores the need for modifications to the tool, potentially involving clearer guidelines, training for raters, or even an overhaul of the categories in question. While the complexity of the tool may have influenced the results, this study marks a significant starting point in recognizing the importance of user-friendliness and clarity in co-creation processes. Future endeavors aimed at refining the tool and enhancing its accessibility to community members have the potential to yield more dependable and representative assessments of public spaces.

Conclusion

Overall, the ICC scores provide insights into the level of agreement among the raters/judges for each determinant of health in public spaces in the Dreta de l’Eixample neighborhood. While some determinants demonstrate higher levels of consistency and agreement (e.g. Access and Connectivity), others show more variability (e.g. Safety). These scores suggest that there is a gap in how academic researchers, technicians, and older adults perceive and assess certain determinants. This gap may be influenced by the original design of the tool, which was anticipated to be used primarily by experts and technicians, potentially affecting the interpretational congruence and consensus among diverse user groups.

Academic researchers and older adults posed different perspectives and priorities when it comes to the determinants of health in public spaces. While researchers and experts focused on objective measurements and broader societal benefits, older adults often emphasized their subjective experiences, comfort, and immediate well-being. Recognizing and incorporating the perspectives of both groups can lead to more inclusive and age-friendly public space design and policies. Additionally, extending the evaluation process to the community leads to a more comprehensive understanding of public space design and the factors influencing health outcomes.

The findings of the study contribute to the understanding of the inter-rater reliability of the Tool for assessing determinants of health in public space. The results of the study can inform the future use and improvement of the tool, enhancing its quality and applicability in promoting public health outcomes. There is a need to explore and develop ways to facilitate the application of the tool in a community setting in order to ensure the understanding of the users and reduce the potential for unclear answers or unrepresentative data.

One of the contributing factors to the observed low agreement in tool usage is the perceived lack of user-friendliness. This issue is particularly prominent among older adults or other community members who may encounter difficulties in effectively using the tool without substantial assistance and interpretation from the researcher. The complexity of the tool interface may impede their ability to navigate and utilize its functionalities accurately. Consequently, the reliance on external support may introduce inconsistencies in interpretation and usage, resulting in diminished agreement levels among users. To address this challenge it may be important to develop a more user-friendly version of the tool that is easily comprehended by community members, including older adults. By reducing the cognitive burden and eliminating potential barriers to participation, a community-oriented version of the tool could increase the likelihood of obtaining meaningful and representative data from the community, accommodate shorter timeframes for its application, as well as complement the professionals’ individual assessment to obtain a complete picture of the public spaces to be evaluated. This development would be crucial for the practical application of the tool in real-world practice and policy-making. It would enable a broader range of stakeholders, including community members and older adults, to actively participate in the assessment of public spaces.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplemental data

Supplemental data for this article can be accessed online at https://doi.org/10.1080/23748834.2023.2286724

Additional information

Funding

This work was developed under the Health CASCADE project. Health CASCADE is a Marie Skłodowska-Curie Innovative Training Network funded by the European Union’s Horizon 2020 research and innovation program under Marie Skłodowska-Curie grant agreement n°956501.

Notes on contributors

Jorge R. Zapata-Restrepo

The authors are PhD fellows and supervisors within the Health CASCADE consortium. The project consists of 7 beneficiaries and 14 partner organizations from 7 European countries – led by 15 Early Stage Researchers investing and testing different aspects of co-creation. The aim of Health CASCADE is to foster the next generation of highly-trained research leaders to develop evidence-based guiding principles, novel tools, and new technologies to make co-creation an effective tool to fight complex public health problems through a European Joint Doctoral Programme.