Strengths and weaknesses of accessibility instruments in planning practice: technological rules based on experiential workshops

ABSTRACT

Accessibility instruments can play a valuable role in urban planning practice by providing a practical framework for exploring and testing relationships between land use and transport infrastructure. Despite many available accessibility instruments, they are still not widely used in planning practice. This paper explores the background of this problem by examining the findings of a EU-funded study on the usability and usefulness of existing accessibility instruments. The study applied 16 instruments in local planning contexts according to a standardized process protocol. The outcomes of these so-called experiential workshops were analysed through a standardized measurement protocol, which included participant observation along with pre- and post-workshop practitioner questionnaires. This broad investigation presents a rich analytical tool for understanding how different types of accessibility measures, spatial resolutions of output and levels of comprehensiveness affect usability and usefulness. Based on this we propose 10 technological rules that (a) can be used directly in practice to improve usability of accessibility instruments and (b) can provide hypotheses to be examined in further academic studies. Our results suggest that instead of striving for the ultimate accessibility measure, it would be more effective to identify which measures could successfully serve different user needs in accessibility planning.

KEYWORDS:

• Accessibility instruments
• urban planning
• experiential case study
• technological rules
• usability
• usefulness

Introduction

Accessibility concepts are increasingly acknowledged as fundamental for understanding the functioning of cities and urban regions (Axhausen, 2008; Bertolini, le Clercq, & Kapoen, 2005; Ferreira & Batey, 2007; Ferreira, Beukers, & te Brömmelstroet, 2012; Geurs & van Wee, 2004; Handy, 2002; Straatemeier & Bertolini, 2007). While many planning strategies include planning goals aimed at improving accessibility, planners are still struggling to articulate the accessibility concept in practice (Curtis, 2011; Curtis, Scheurer, & Burke, 2013). Accessibility instruments are designed to measure the extent to which the transport network provides access to various opportunities in urban agglomerations so that people can meet their daily needs. The transport network can be assessed for different transport modes. The measurements can include variously distance, time, transport transfers and cost. The measurements are made between origins and destinations of various opportunities. The outputs are often expressed visually in the form of maps. While accessibility measurement is not new, the past decade has seen a new generation of accessibility tools emerge designed to assist in decision-making for transport and development (Curtis & Scheurer, 2015) in a way that differs from conventional transport modelling tools which focus on traffic generation and distribution (usually for vehicular traffic).

Accessibility instruments can play a significant role by providing a framework for understanding the reciprocal relationships between land use and mobility, which can be used in developing strategies that improve accessibility. Despite this large potential and the numerous tools already developed, accessibility instruments are not widely used to support urban planning practices (a fate shared with other types of planning support instruments) (Hull, Silva, & Bertolini, 2012; te Brömmelstroet, Silva, & Bertolini, 2014). The wider academic discussion on planning support systems (PSS) identified a persistent disconnect between the worlds in which PSS are developed and the worlds in which they need to be applied. This disconnect, manifested in different languages, incentives and working processes, breeds a reciprocally antagonistic attitude (Brail, 2008; Geertman & Stillwell, 2003; Geertman, Toppen, & Stillwell, 2013; Vonk, 2006). As a result, the existing PSS rarely fit the dynamic, wicked and fluid contexts of most planning processes.

This paper will further inform the ongoing academic debate by presenting the results of experiential workshops with local planning practitioners, held in 15 different European countries and Australia. The practitioners applied different accessibility instruments on concrete planning tasks, and were then asked to reflect on the experience. The different accessibility instruments were tailored to the particular planning context. Through these context-rich experiences, we sought to identify the strengths and weaknesses of accessibility instruments in bridging the implementation gap. How do the characteristics of each instrument affect its usability and usefulness in planning practice?

We begin the paper with a discussion on the current implementation gap of accessibility instruments, followed by a description of the research design, workshop methodology and methods for data collection and analysis. Then a discursive exploration of strengths and weaknesses of different accessibility instruments in different contexts is provided. This exploration does not aim for generalization, seeking instead to distil a number of technological rules that can be tested in future research (following van Aken, 2004; Pawson & Tilley, 1997; Straatemeier, Bertolini, te Brömmelstroet, & Hoetjes, 2010). In closing we discuss the experience-based case study methodology and provide input for improving accessibility instruments, in order to facilitate their adoption as a mainstream support technology in planning practice.

Background

The urban transport problem and the role of accessibility planning

Travel behaviour, transport networks and spatial patterns have changed significantly in many European cities over the last decades, resulting in increasing average travel distances and increasing levels of car dependence (Banister, 2005; Handy, 2002; Jeekel, 2011; Lutz & Lutz Fernandez, 2010). North American and Australian cities show similar unsustainable trends, and rapidly developing Asian cities will face the same fate unless strong accessibility measures are taken. These developments have resulted in a crucial policy dilemma (Bertolini, 2012): Mobility has become an important element in the functioning of individuals, cities and regions; however, it has also produced a wide range of problems that adversely affect the economy and general well-being.

The numerous policies, strategies and implemented projects have failed to curb these negative trends. Often these interventions arise from a particular policy sector (land transport, public transport, land use and urban planning, non-motorized transport, etc.) and do not link up with initiatives from the other sectors. Each sector has a different view of the planning problem and a specialized professional language and process protocols (Bertolini, le Clercq, & Straatemeier, 2008; te Brömmelstroet & Bertolini, 2010; te Brömmelstroet & Bertolini, 2011; Curtis & Low, 2012; Straatemeier & Bertolini, 2007; Straatemeier & Bertolini, 2008). Because of lack of integration, these strategies and projects are sometimes even downright contradictory, which severely hampers cities and regions in addressing their mobility dilemmas (Bertolini et al., 2008; te Brömmelstroet & Bertolini, 2010; Stead, Geerlings, & Meijers, 2004; Straatemeier & Bertolini, 2008).

The concept of accessibility offers a suitable framework to support the development of such integrated strategies (see, for example, Curtis & Scheurer, 2010; Ferreira et al., 2012; Geurs & Ritsema van Eck, 2001; Geurs & van Wee, 2004; Halden, 2003; Handy, 2002; Handy & Niemeier, 1997; Makrí, 2001; Nuzzolo, Coppola, Crisalli, & Comi, 2010; Nuzzolo, Coppola, & Papa, 2014; Silva, 2008). In light of the ongoing debates on its definition, in this report we define accessibility as ‘an expression of the potential number of relevant activities that are located within acceptable reach' (or travel time, see, e.g. Milakis, Cervero, & Maat, 2014) ‘of a given place or people in acceptable reach of an activity'. Following this definition, accessibility (1) makes the overall goal of the land use and transport system explicit (i.e. supporting interactions between individuals and activities); (2) is relatively easy to model, interpret and understand; (3) correlates closely with real-life behaviour of individuals and companies; (4) offers a direct link with the characteristics of flows (i.e. speed and travel time) and (5) offers a direct link with the characteristics of place (i.e. the number of relevant activities or people in a given area). Thus, it offers a potentially powerful guide that planning practitioners can employ to develop and test effective strategies for sustainable cities (Straatemeier & Bertolini, 2008). They can learn about the effectiveness of different types of strategies in addressing the mobility dilemma.

Limited usability of accessibility instruments in planning practice

Although both the concept of accessibility and its potential contribution to urban planning practice have been extensively discussed, these concepts are yet to be successfully translated into effective and usable planning instruments (te Brömmelstroet, 2010; te Brömmelstroet & Bertolini, 2011) used in everyday planning practice. In urban planning practice, the concept of accessibility is often misunderstood and the support instruments are often seen as complex, inflexible, incomprehensible and rigid black boxes.

This antagonistic attitude towards accessibility instruments is mirrored in the debates on PSS. Planners see them as far too generic, complex, technology oriented (rather than problem oriented), narrowly focused on strict technical rationality and incompatible with the unpredictable/flexible nature of most planning tasks and information needs (te Brömmelstroet, 2012, 2013; Geertman, 2006; Gudmundsson, 2011; Klosterman, 2001; Lee, 1973, 1994; Vonk, Geertman, & Schot, 2005). Although we have seen significant progress in computational power and in PSS capabilities, the ‘implementation gap' remains wide open (Vonk, 2006).

One of the underlying fundamental problems seems to be a persistent disconnect between the worlds of instrument developers (who aim for scientific rigor and have an abstract understanding of the planning problem and process) and potential users (who aim for direct relevance, with the complexity of the real world as a starting point, and often are antagonistic towards sophisticated external technologies). Each group fails to see and appreciate the perspective of the other. As a result, developers produce technologies based on a distant and abstract idea—rather than a clear, shared understanding of the needs and demands of specific planning contexts. Planners, on the other hand, often hold unrealistic expectations of what the technology can offer; the inevitable disappointment with the provided support leads to antagonistic attitudes towards new technologies (Meadows & Robinsons, 2002; Vonk et al., 2005). Bringing these two worlds together could help bridge the implementation gap and address some of the most pressing urban mobility dilemmas.

However, improving PSS usability is a ‘means' instead of a goal. Instead, the academic and practical advances in improving usability have the ultimate goal to make the instruments ‘useful' for planning practice (Nielsen, 1994). Accessibility instruments often aim to offer a shared language, bring different professions together and support a shared enlightenment process. Even so these attributes are also not goals, but rather by offering these attributes the assumption is that the quality of planning (processes and/or outcomes) will be improved.

Research design

Epistemological concerns

Our research design seeks to develop knowledge through grounded theory. The wide sample of cases (neither random nor representative) allows us to gain rich insights into how the characteristics of accessibility instruments interact with different settings and practitioners. Also, as part of a ‘realistic evaluation’, we refrain from generalizable statements. Instead we develop context-sensitive rules that illustrate the complex relations between interventions, contexts and outcomes. These so-called technologic rules follow the CIMO logic: ‘In a certain context (C), an intervention (I) through specific mechanisms (M) leads to a certain outcome (O)’ (Denyer, Tranfield, & van Aken, 2008; Pawson & Tilley, 1997; Straatemeier et al., 2010; Van Aken, 2004). Building on the pragmatist and critical realist traditions, technological rules form a bridge between two ways of knowing: the academic, where these rules are tested as hypotheses, and practical knowledge (or phronesis, see Flyvbjerg, 2001), where the outputs are directly used to guide real-life interventions.

Research questions

From the discussion in Section 2, several points become evident: (a) cities should be planned in such a way so that improvements to accessibility resolve mobility problems; (b) many accessibility instruments are available; (c) the potential usefulness of these instruments is not utilized in mainstream planning practice and (d) this shortcoming appears to be caused by a disconnect between the instrument developers and their potential users. By examining a sample of context-rich cases where different accessibility instruments were applied, we aim to develop technologic rules on the usability and usefulness of accessibility instruments. Our analysis is structured along the main research question: How do different mechanisms (M) of accessibility instruments (I) influence their usability and usefulness (O) in different planning contexts (C). The following three mechanisms are derived from academic debate:

• Differences according to the specific accessibility measure (space syntax, gravity-based measures, etc.); the literature suggests that this is a relevant variable for its usefulness in practice (Geurs & Ritsema van Eck, 2001).

• Differences in the spatial resolution and structure of the output.

• The level of comprehensiveness of the instrument (infrastructure focus vs. integrated land use and transport; single mode vs. multimodal) (Straatemeier, 2008).

The research explores how differences in context and mechanism variables affect usability and usefulness of accessibility instruments in planning practice. Perceived ‘usability' is defined as ‘the degree to which a person believes that using a particular system would be free from effort' (Keil, Beranek, & Konsynski, 1995, p. 76). It refers to the planners’ hands-on experiences, that is, how easy and intuitive it is to work with the instrument. The literature on the PSS implementation gap has developed a wide range of important usability variables (te Brömmelstroet, 2010; Vonk, 2006). In this research we explore the most relevant ones in the field of land use and transport strategy-making (after Te Brömmelstroet, 2010):

• Transparency—How easy is it to understand the assumptions, algorithms and outputs?

• Communication value—Can the output be used to support interaction among professionals?

• User -friendliness—How easy is it to use the instrument?

• Interactivity—The ability to interact with the instrument during application in a workshop.

• Flexibility/Rigidity—The ability to adjust the instrument during application in a workshop.

• Calculation time—The time needed to run analyses for interventions.

• Specificity—The level of detail of the planning issue presented by the instrument.

• Credibility—Are the instrument and its results ‘sound' in relation to the planners’ context?

• Integration capacity—The degree to which an instrument can be used as a platform to integrate different planning specializations.

‘Usefulness' of accessibility instruments goes a step further by asking the following question: Does the instrument bring added value to the planning task? In other words, an instrument can be usable, but not useful (and vice versa). Although planning tasks vary widely, there are some common usefulness characteristics of accessibility instruments. These relate to such qualities of the planning process as enlightenment (Amara, Ouimet, & Landry, 2004; Gudmundsson, 2011) or a shared learning environment (Rouwette, Vennix, & van Mullekom, 2002). These concepts have been translated into the PSS realm and are extensively discussed elsewhere (te Brömmelstroet, 2013). Here we focus on the specific added values hypothesized in the above-mentioned academic literature on accessibility:

• Individual reaction (enthusiasm),

• Increased insight in the problem,

• Development of a shared professional language,

• Communication (shared vision),

• Prospects for planning practice (analysis),

• Prospects for planning practice (strategy development),

• Efficiency (of the group work),

• Cohesion (sense of being part of a group).

Data gathering and analysis methods

A standardized process and measurement protocol enabled us to compare the workshop observations and findings and to develop an understanding of the usability characteristics of accessibility instruments. The combination of the experiential approach and the strengths of the classical multiple case study design allows for a comparison between cases (Bryman, 2008; Gerring & McDermott, 2007; Yin, 1994).

The evaluation was designed to provide information on the participants’ current use of accessibility instruments, their view of the instruments prior to the workshop intervention as well as an in-depth examination of their reaction to the workshop experience. The evaluation package comprised a mix of respondent questionnaires (pre- and post-workshop), a semi-structured focus group with the participants after the workshop and a panel assessment involving only the workshop organizing team (i.e. accessibility instrument developers). All workshop organizing teams were also asked to describe their complete experience in a written pre-structured report covering the instrument, the context, the workshop and lessons on usability (i.e. how well the planning practitioners were able to work with the instrument and its output).

Based on a recent publication on the same sample, we could divide the instruments along their variations on the above-mentioned mechanisms (Hull et al., 2012). For each of these variations, we are interested in any systematic effects on (perceived) usability and usefulness. The first aspect is analysed through NVivo coding of the pre-structured reports prepared by the workshop organizing teams. This discursive analysis aims to uncover general patterns in the usability of accessibility instruments. For the second aspect, we use the responses from the post-workshop evaluation questionnaire where the participants were asked (among other questions) to assess the added value of the accessibility instrument on the seven usefulness dimensions addressed above, using a Likert scale rating from 1 ‘strongly disagree' to 5 ‘strongly agree’.

In total, 16 workshops were organized (see Table 1—15 in Europe and 1 in Australia) and 13 complete evaluation packages were returned to us comprising responses of 80 participants. Most of the participants were male (69%), young (31–45 years old, 46%) and middle-aged (46–60 years old, 44%), with various professional backgrounds: transport planners (43%), urban planners (26%), architects (8%), urban and transport planners (6%), regional planners (4%) and others (lawyers, surveying engineers and housing developers; 14%). Professionals from public organizations were the majority in the workshops (78%), while 17% worked in the private sector and 5% were affiliated with universities or non-governmental organizations.

Table 1. Participating partners and accessibility instruments.

Country	COST partner	City of application	Accessibility instrument^a
Australia	Curtin University	Adelaide	SNAMUTS
Cyprus	University of Cyprus	Limassol	ASAMeD
Finland	Aalto University	Helsinki	Himmeli
France	Université Paris-Est	Paris	Contactability
Germany	TU Munich	Munich	Erreichbarkeitsatlas
Greece	University of Thessaly	Volos	SpatialistLines (MoSC)
Italy 1	University of Rome	Rome	GraBAM
Italy 2	Politecnico di Torino	Turin	InViTo
Netherlands	University of Amsterdam	Breda	Joint-Accessibility Design
Poland	University of Technology	Krakow	GDATI
Portugal	TU Oporto	Porto	SAL
Slovenia	University of Ljubljana	Ljubljana	ATI
Spain	University of Madrid	Madrid	IMaFa
Sweden	University of Gothenburg	Gothenburg	Accessibility Atlas
Turkey	Çukurova University	Izmir	Cittaslow
UK	Herriot Watt University	Edinburgh	SNAPTA

^aFor the more details regarding these accessibility instruments, see Hull et al. (2012) and http://www.accessibilityplanning.eu/accessibility-instruments-list/

Selection and organization of cases

The research design uses elements of a classical multiple case study, whereby each accessibility instrument is used and analysed within one local planning setting (see Figure 1). Bearing in mind that there is no ‘best' or ‘most used' accessibility instrument and that each available instrument had generally been designed to answer a specific type of planning question, this approach offered a valuable way to test the range of accessibility instruments and the range of different planning and institutional contexts. It is both fluid enough to cater to this variability and robust enough to enable comparison of results.

Figure 1. Combining multiple case studies with experiential case study elements.

As discussed above, the sample is self-selected, based on the network of participating countries and European Cooperation in Science and Technology (COST) project eligibility requirements. We did not seek to generalize across an entire population of accessibility instruments. Confirming that an implementation gap exists, we found that only in 4 out of 13 case studies that returned a complete evaluation package, some of the participants reported that their organization used an accessibility instrument in practice.

Experiential workshop

The combination of experiential cases in a multiple case study design leads to a number of specific characteristics that affect the research and its findings. In an experiential case study design, the researcher does not only observe, but also actively intervenes in planning practice.

Fuelled by theoretical understandings on how practice can be improved, the researcher develops an intervention, applies it in a case, reflects on its effectiveness and (if needed) improves both the theoretical understanding and the intervention itself. As illustrated in the example above (Figure 2), three meetings were held with two three-week periods in between for the implementation of workshop results, allowing for a flexible and interactive approach. An experiential case is not real planning practice. In an ideal situation an intervention is tested in real-life practices, but because of the distinct focus on reflection by both researcher and participants some distance is necessary. Nevertheless, we sought to replicate real-life planning as close as possible. The workshop protocol had a four-step structure that mirrored the procedure during a typical planning exercise (Table 2). The aim was not to develop strategies but to play with the instrument in a next-to-real exercise. For a description of all case workshops, see te Brömmelstroet et al. (2014).

Figure 2. Example of interactive workshop organization and planning questions in the Gothenburg case (see details of steps in Table 2).

Table 2. Four-step workshop.

Steps	Objective	Action
1	Formulating economic, social and spatial planning goals and defining accessibility criteria	The planning actors agreed on a planning question and discussed how the accessibility instrument indicators could support them in exploring this planning question. The aim is to translate individual and group thinking on a particular planning question into a shared accessibility language
2	Collectively mapping, measuring, interpreting and analysing the current level of accessibility	The instrument developers produced and presented the desired accessibility output (e.g. maps, tables and numbers). The aim was to develop a shared understanding of the current accessibility situation and of possible interventions
3	Developing intervention strategies to understand the sensitivity of accessibility	The planning actors developed planning interventions to improve or maintain the accessibility situation as presented in step 2. The instrument developers then presented the effects of these interventions on accessibility. The aim was to develop a shared understanding of the sensitivity of the accessibility situation towards planning interventions
4	Scanning/evaluating the effects of intervention strategies in order to designing integrated planning solutions	Based on the shared understanding of the accessibility situation and sensitivity to planning interventions, the group could then agree on interventions for the planning question in step 1. The aim was for end users to internalize the accessibility language in their individual understanding of the planning question

Below, we discuss the findings for the three mechanisms separately. For each, we first identify the range of instruments and then continue by discussing the differences in usability and usefulness.

How different accessibility measures influence usability and usefulness

Accessibility instruments use different measures of accessibility. Drawing on the categorization of Curtis and Scheurer (2010), we have identified accessibility measures available across the range of instruments used in the workshops. A distinction is made between the measures most commonly used in the workshops and others available but used less or not at all. ‘Spatial separation' refers to measures of travel impediment between origin and destination, such as distance or travel time. ‘Gravity measures' are typically used to analyse opportunity by defining catchment areas based on a travel impediment on a continuous scale as opposed to ‘Contour measures' where one or more static travel time contours are drawn around a node. ‘Network measures' capture different aspects of connectivity and centrality across movement networks such as roads or public transport. If accessibility takes into account predefined time constraints such as traveller's time-budgets, we refer to ‘Time–space measures'. Where instruments use a combination of measures to create a summary measure or index, a new ‘synthetic measure' was introduced. The main reason to make a distinction between measures is the fact that they illustrate significantly different approaches to accessibility in terms of planning questions and problems. These measures can also differ in terms of data resources and computational skills they require and in terms of visualization opportunities they offer. Understanding if and how these differences relate to usability is a key aim of this research. The following examples illustrate differences between the represented measures:

• Spatial separation: How long does it take (from the place of residence of potential students) to travel to the nearest university that offers a degree in urban planning?

• Gravity measures: How many potential students in a country can commute (based on their place of residence) daily to a university that offers a degree in urban planning? The opportunity is weighted by travel time, the number of students admitted each year and the national ranking of the university.

• Contour measures: How many potential study positions in urban planning at the university level can a student (based on his or her place of residence) reach within 30 minutes, 60 minutes, 120 minutes and above 120 minutes travel?

• Network measures: How many changes are needed for a student (based on their place of residence) to reach, by public transport before 09.00 on a weekday morning, the nearest university that offers a degree in urban planning?

• Time–space measures: How many potential study positions in urban planning at university level can a student (based on their place of residence and the location of their children's school) reach by bicycle, car or public transport respectively given that they need to leave two children at their school at 09.00 and pick them up at 16.00?

Figure 3 illustrates the range of measures available for each instrument and the main measure used during the workshop. Very few instruments are one-dimensional. In cases where two or more types of measures were used separately, we selected the most significant one in the context of the workshop.

Figure 3. Categorization of instruments according to type of accessibility measure and case city.

Usability

The most important usability measure according to the participants was integration capacity, a view supported by the literature, especially for spatial separation instruments. One important explanation for the high degree of integration capacity of spatial separation could be that it is simple and intuitive. Practitioners from diverse backgrounds can interpret the map output relatively easily and exchange information. In the Polish workshop, for example, participants underlined that the instrument's ability to visualize measures made the potential options for public transport modification and modernization understandable and acceptable not only for planners but also for the clients of real estate developers and the public.

In workshops where other types of instruments were used (gravity, network or synthetic), integration capacity was not emphasized as a positive usability factor. This could be explained by the fact that scientifically more rigorous measures can be a barrier between the transport planners familiar with traffic models and the less model-oriented planners. In an interactive session, minimizing cognitive distance between participants is key for the instruments’ usefulness as an integration tool. Next to minimizing the distance between participants, there is also the need to be aware of the intrinsic cognitive distance between participants and developers of accessibility instruments (Bertolini et al., 2005; Lucas, 2006).

The workshops’ outcomes make it clear that specific accessibility measures provide only part of the contribution to usability. Of equal, and sometimes even greater, importance is the way in which the respective instrument represents accessibility, that is, the degree of interactivity provided, the visual appearance of maps (in terms of their communication value in both the output and the planning process) and the user-friendliness. This can be attributed to the output of relatively simple maps that facilitate dialogue with non-experts; however, opinions on its usefulness differ strongly between professional groups. Gravity instruments provide an illustrative case. Transport planners often query the measures themselves, while urban/regional planners see the mapping of transport and land use interaction as a very valuable complementing feature for their planning problems, without fully understanding the measure.

The most striking impact in relation to the usability dimension of calculation time can be attributed to the experienced lack of real-time scenario testing capabilities. Only the instrument from Turin was designed for interactivity. This feature is the centrepiece of the instrument and was seen as very valuable for the facilitation of dialogue between different planning alternatives and their impact of accessibility. Non-experts appreciated the simple measure, but it was also recognized that other measures, such as time and monetary costs, would greatly enhance the usefulness of the instrument.

Usefulness

There are no clear differences in usefulness between instrument types (Figure 4), although instrument developers observed that the spatial separation measures seemed to generate more positive reactions on usefulness (not reflected in the post-workshop participant survey). While the quantitative outcome shows only small differences, we assert that it is the qualitative outcomes that offer clear and valuable insights (this is also the case in respect of further analysis reported below). In general, participants reported that the instruments provide a very good platform for collective reasoning and problem-solving between different planning disciplines as well as with outside actors:

It showed the relations between traffic congestion and the economic attractiveness of a place. Even with some travel time delay on the network, the economic potential can be high. This shared understanding gives insight where to develop. (Participant from the Netherlands workshop)

Figure 4. Usefulness by type of accessibility measure (average of Likert scale: 1 – strongly disagree to 5 – strongly agree).

The different accessibility measures performed similarly in relation to the dimensions of enthusiasm, communication, efficiency and cohesion, but there are differences between accessibility instruments for other dimensions. Gravity and network measures of accessibility seem to offer better insights into the processes that play a role in the problem and are perceived as more useful for analysis and strategy development. However, gravity measures performed worse than network measures with respect to development of a shared language among planning actors.

The perspective of instrument developers (as observers of the workshops) provides an additional assessment of usability. The instruments using the spatial separation measure generate the most positive responses, both for its usability as an integrator and as a basis for dialogue (increased insight):

It also gives a basis for understanding the effects of other aspects outside of transport and infrastructure, which we known in principle but now can see in detail. A bit of an ‘aha' experience to look at sectors like we have. I have not done this before. (Swedish public transport planner)

Yet the data in Figure 4 show that spatial separation measures performed slightly lower than all other types of instruments with respect to insight. Both developers and users reported that gravity-based instruments created new insights, mainly related to the ability to model and to evaluate scenarios and planning alternatives. We can conclude that different instruments create different insights depending on the specifics of the measure. While this may not be a surprising result, it illustrates that accessibility planning can be supported by a range of different measures. Instead of striving for an ultimate accessibility measure, it seems more logical to highlight the specific strengths of each measure.

Spatial separation measures also score slightly lower on efficiency. Again the experience is not shared in all workshops or by developers and users alike. Two primary efficiency hurdles were identified: the complexity of the instruments in terms of data handling and the lack of more complex or realistic measures (e.g. cumulative). Some practitioners see spatial separation measures as an ideal base because it is simple to use and to understand, while others point to its wide range of advanced analytical possibilities.

This analysis does not take account of the specific conditions at each workshop. In some cases, instruments with similar measures had very different degrees of technical development, which likely influenced both the usability and usefulness experience of the participants. An example is the synthetic index type of measure used in the Australian case; in spite of its relative complexity, it proved to be very useful in the analysis of new potential land-use alternatives. This can be explained by the fact that the instrument (SNAMUTS) has a very well-developed user interface with indicators in standardized colours and symbols. The use of less developed user interfaces and expert systems applied in a non-expert environment poses the risk of devoting substantial time on unproductive discussions about technical details rather than the planning problem at hand.

How different spatial resolutions and structures effect usability and usefulness

Usability and usefulness are closely linked with how the result is presented to the end user. One important dimension of the communication interface between calculation and interpretation is the spatial resolution (the level of detail of the output) and structure of the output (i.e. the type of spatial reference used to present results as maps). The following three categories are examined (Figure 5):

• Administrative areas: Using functionally defined pre-set areas, for example, transit areas.

• Network: Based on network links and nodes for presentation of output.

• Grid cells: Typically made up of squares covering the entire area of analysis.

Figure 5. Categorization of output resolution and structure by case city.

Usability

The workshop participants reported positive experiences with the network structure in terms of its communication value. The output in the form of predicted network flows or link connectivity was reported as easy to interpret by non-expert users. Another identified positive asset is that it allows for zooming without loss of accuracy. Three out of four network-based visualization applications used standard commercial software (TransCAD or Space Syntax), while the fourth (SNAMUTS) is a purpose-built instrument with a clear focus on accessibility indicators and measures.

Most of the comments on usability came from the SNAMUTS workshop, which provided a well-developed graphic interface combining colours, symbols and numeric values to show different indices of accessibility in the public transport network. In this case, the network alone managed to capture interests and supported communication in a very good way, despite the slow start of the workshop:

This [the combined index] is presented in a highly visual map using traffic light colours to depict accessibility across the network. This type of visual approach appeared to generate interest, and from that point onwards the discussion gained a lot of momentum. The questions asked during this process clearly indicated that the participants, after seeing the results, were convinced that this tool could be useful. (Instrument developer observation, Australia)

The grid cell and administrative visualization features also provided good usability by combining infrastructure and land-use data on the same map. The visualization power of maps was highlighted as a key usability feature in almost all workshops, but less pronounced in the network cases. This difference suggests that the network itself provides a good basis for accessibility visualization of network measures; however, cell and administrative principles work better for the more land-use-based measures. High-resolution administrative areas or grid cells provide a strong basis for a common understanding of the accessibility analysis:

The visualization power of the instrument using detailed maps based on the key small-area statistical geographical units in Scotland made the used accessibility language easy for the participants to interpret and communicate. (UK instrument developer)

Usefulness

Usefulness performed similarly for the dimensions of enthusiasm, strategy development and cohesion (Figure 6). In relation to communication, the instrument developers noted that grid cell-based tools were highly valued for their communication potential, a view supported by the post-workshop participant surveys. Still, they received a slightly lower score for helping generate a shared language among the planning actors. Thus, increased communication may not automatically imply the development of a shared professional language and vice versa.

Figure 6. Usefulness according to the resolution and structure of output (average of Likert scale: 1 – strongly disagree to 5 – strongly agree).

Network-based accessibility measures seem to perform better regarding insight, analysis and the development of a shared language. Administrative-based accessibility measures showed the lowest scores for insight, analysis and efficiency. Grid cell-based accessibility measures were perceived as very useful with respect to efficiency.

In relation to the instruments’ ability to increase problem insight, all output types scored well, with network scoring highest. The workshop comments highlighted the scale and spatial structure of the instruments. In a number of cases, the accessibility maps provided an explicit illustration of what planners already knew intuitively. However, in all cases the geographical scale of the visualization was detailed enough for planners to recognize their own practice-based knowledge about local specificities. Therefore, one main strength of the instruments, independently of the geographical structure of the output, is the use of below-municipal level spatial data to create a sense of trust between the place-based everyday knowledge of planners and the scientific rigour of accessibility analysis output. The following workshop evaluation reports illustrate the point:

The micro-data on labour markets and industrial location together with accessibility and travel time measures gave the participants new knowledge. They expressed that they used an ‘educated guess' or had a feeling about the situation before. (Grid cell-based instrument developer, Sweden)

The group commented that they had never thought that the southern side of the metropolitan area was so left out of public transport accessibility in the future. It was clear that these were new ideas to the group that caused considerable interest. We also provided the evidence base to confirm the group's understanding that some of the more peripheral urban land releases at the urban fringe would fail to achieve accessibility by public transport without substantial investment. (Network-based instrument developer, Australia)

There was growing awareness that accessibility as a concept in tandem with other planning instruments proved to be a very useful platform for interaction between experts and non-experts. This was the case across all types of spatial structures of output. Network-based instruments were not rated lower than other types; however, providing more than only network outputs was an important feature if visualization was to provide a useful discussion platform. This is echoed in some of the perceived limitations for usability and usefulness, where on the one hand we could see a demand for a wider set of specific accessibility indicators and accompanying socio-economic data, while on the other there was a request for aggregated indicators. These contradictory user-needs point to a general problem of balancing geographical details against technical capabilities and visualization possibilities.

How level of comprehensiveness influences usability and usefulness

Level of comprehensiveness classifies the instrument according to the modes of transport included in the accessibility analysis. The categorization presented in Figure 7 is based on a combination of instrument reports and case study reports for each instrument. It is important to point out that given the wide array of instruments in the study, there are potentials to include both ‘walk' and ‘bike' via the road network in most cases. However, we have checked if and how these two modes were used in the workshops and positioned them accordingly. Several of the instruments also include walking as part of an integrated public transport journey. This has not been taken into account as a specific mode. Again we would like to stress that the focus is on the actual and most important use in the workshops. Accordingly, no instrument explicitly measured biking as a mode per se and only in some cases (especially in Space Syntax) was walking identified as a specific mode of transport.

Figure 7. Categorization of inclusiveness of transport modes by case city. Note: One case is classed as ‘other’. It is referring to the context of accessibility to public utilities networks (such as electricity and water networks) from potential housing development sites. Since this case is not relevant for travel and mode choice, it is excluded from analysis in this section.

The majority of the instruments are mono-modal, with the combination of car and walking as the single most prominent group. The remaining five multimodal instruments all provide car and public transport comparison. Two of these have an integrated approach where also walking and cycling are explicitly involved in the measure.

Usability

Multimodal instruments were seen as less interactive. One explanation is that the maps had to be produced prior to the workshop due to the technical limitations of calculating speed and data management. As a result, changes (scenarios) could not be analysed in real time, a key weakness in the eyes of the workshop participants. The same experiences hold true for the mono-modal instruments regarding technical limitations, although this affected only two out of eight cases. Instead, low interactivity is seen as related more closely to the workshop methodology; due to time limitations, some teams had to skip the interactive session in order to cover all the planned steps.

The positive reports on interactivity relate to the mono-modal instruments, especially the interactive web-based design used in Turin. However, also the Space Syntax cases reported that interactivity was positive in the sense that changes could be mapped and discussed during the session. Some evidence exists that participants found it easier to interact with mono-modal instruments. This is partly due to the technical complexity to manage—and above all quickly change—large databases for integrated multimodal analysis. One further explanation may be that multimodal accessibility analysis requires users to work with two very different transport modes simultaneously. This may not be compatible with traditional planning that focus on one specific mode.

Usefulness

Participants perceived mono-modal accessibility instruments as more useful than multimodal instruments regarding insight, analysis, strategy development and efficiency (Figure 8). Multimodal accessibility instruments did not perform better than mono-modal instruments in any of the usefulness dimensions. Both categories showed similar scores for enthusiasm, shared language, communication and cohesion.

Figure 8. Usefulness according to the level of inclusiveness of accessibility instruments (average of Likert scale: 1 – strongly disagree to 5 – strongly agree).

There is a negligible difference between multimodal and mono-modal instruments regarding communication, which is first and foremost linked to the combination of visual presentation through maps and its capacity to simplify large data sets and complicated calculations:

The second issue refers to the visualisation of the outcomes, which seems to be its most important aspect—both for the success of the workshop and for the usability of the instrument. The angular segment analysis visual outputs are easy to understand, not only by professionals but also by ordinary citizens (as confirmed by participant comments). Therefore, it could be a useful and convincing tool with a broad application potential. (Greek instrument developer)

The mono-modal instruments received many negative comments on the specificity of the instruments, which could be seen as a downside of the higher degree of communication potential reported above. Half of the workshops with mono-modal instruments reported that usability would increase substantially if they were upgraded with new and broader measures, such as land use, costs or cumulative measures, including the inclusion of more transport modes in the analysis.

Mono-modal instruments are typically praised for their simplicity and communication potential with non-experts. The communication value of the multimodal ones, on the other hand, is related more to their ability to integrate transport and land-use planners. This observation supports the general conclusion that accessibility instruments can be useful for a wide array of different purposes depending of the context and objective. Categorizing all instruments into one accessibility instrument category may, therefore, severely impede their potential use in non-standardized settings.

Conclusion and discussions

Accessibility instruments hold great potential to provide much-needed support to integrating different domains of urban planning, an important prerequisite for solving pressing urban problems. Although consultants and academics have been developing a wide range of relevant instruments, implementation in daily planning practice is lagging behind. In this paper, we reported on our recent study on the usability and usefulness of accessibility instruments. The tools were used to assess local planning questions in sixteen context-rich experiential cases. The instrument developers and planning practitioners then reflected on their experience, focusing on usability (i.e. ease of use and/or user-friendliness) and usefulness (i.e. added value). The study did not aim to extract generalizable statements, but rather technological CIMO rules that can be used directly in practice and provide rich heuristics for further academic scrutiny. The following rules can be distilled from the research:

• For different accessibility measures:

  • Spatial separation measures seem to be more usable in strategic planning sessions in terms of integration capacity;

  • gravity and network measures appear to outmatch other types of accessibility measures with respect to analysis, strategy development and insights into the processes that play a role in the problem;

  • gravity measures lag compare to other measures in supporting the development of a shared language among planning actors.

• For spatial resolution and structure:

  • The output of network structure measures seems to have the highest communication value;

  • network flows or link connectivity is easy to interpret by non-expert users;

  • maps for visualization is a key usability feature;

  • network structure seems to perform better with respect to insight, analysis and development of a shared language.

• For level of comprehensiveness:

  • Multimodal instruments have lower interactivity;

  • mono-modal instruments are seen as too specific;

  • mono-modal instruments seem to have an advantage compared to multimodal instruments regarding insight, analysis, strategy development and efficiency.

These findings show that there are important trade-offs between usability and usefulness. While simple measures lead to a more inclusive debate that also engages non-experts, it is also seen as not comprehensive enough. Multimodal instruments address this problem, but suffer from lower interactivity due to longer computational times. Network measures seem to offer the highest potential for both usability and usefulness.

These 10 technological rules can be utilized in the development of accessibility instruments, and they also provide useful hypotheses for further research. Subsequent research efforts could increase the control over the research context. Each country has a different planning environment (15 in total in our study), and the participants also come from different backgrounds and bring different power relations and knowledge into the meetings. A more experimental set-up could counter this methodological shortcoming. Application in real-life contexts is another possibility. Our cases were next-to-real, in that they took practitioners outside of their working routine to experience the instrument. But would our findings be corroborated in actual planning practice? Additional sociological observation of real-life settings where accessibility instruments are deployed can enrich the discussion.

Next to these methodological advances, we should expand the CIMO logic by zooming in on the differences in the context (C). How valid and relevant are the ten rules when we zoom in on the difference between local settings? Which ones hold and which ones break down. Interactivity and communicative value of output are more important in communicative settings, but not all involved countries follow a strong communicative planning logic. Although the study shows that we are far from single, easy-to-use and useful accessibility instruments, it also raises the awareness that for each typical planning question and setting there might be different suitable instruments. This study added rich accounts that can be used to make such decisions properly.

Acknowledgements

Under the European Cooperation in Science and Technology Action (COST) TU1002 ‘Accessibility Instruments in Planning Practice’, the authors steered Working Group 3 in the design of the workshop protocol and evaluation instruments. The results and conclusions reported in this paper were made possible with the enthusiastic participation of the local COST action country team members and the respective workshop participants. The authors would like to gratefully acknowledge the language editing by Nikola Stalevski and the very helpful comments by the reviewers.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

The authors would like to gratefully acknowledge the financial support provided by the European Union and the Australian Academy of Science.