The design of civic technology: factors that influence public participation and impact

Abstract

Civic technology needs to be better understood in terms of the factors that promote representative public participation and impact. This paper reports on a mixed-methods study of a civic tech platform that enabled the public to provide feedback on public transport to the service providers. The overall aim of this research was to investigate the public’s use of a leading civic tech platform, FixMyTransport. The key findings were that: an effective and easy-to-use civic technology platform enables broad participation; data and process complexity need to be removed; factual information can be captured in situ with impacts, consequences and opinions added later; emotions (if important) need to be explicitly elicited; feedback to, and a ‘conversation’ with, the users is important for engagement, as is a feeling of being part of a community. These findings can contribute to the future design of civic technology platforms.

Practitioner Summary: There is a lack of understanding of how ‘civic tech’ platforms are used and how they may be designed for maximum effectiveness. Multiple data collection methods were used to investigate a well-developed example of civic tech. Effective civic tech can enable broad democratic participation to improve public services.

Keywords:

• Civic technology
• crowdsourcing
• citizen-sourcing
• public participation
• user impact
• user-centred design
• transport

1. Introduction

There is increasing interest in technologies that enable engagement with a population, and a number of terms are used to describe this broad emerging phenomenon. The most familiar – ‘crowdsourcing’ (Howe 2006) – normally refers to a problem offered to the crowd by a problem or system owner (Doan, Ramakrishnan, and Halevy 2011). ‘Citizen-sourcing’ has a focus on mass contribution within the public sector (Lukensmeyer and Torres 2008). The term ‘citizen science’ has also been increasingly prominent over the last few years, and refers to the use of the public to contribute large-scale data for scientific purposes (Silvertown 2009). Several authors, such as Goodchild (2007) and Lathia (2013) have described how humans can act as ‘sensors’ within either a scientific or broader context.

Civic technology or ‘civic tech’ (Steinberg 2014) is a term that is increasingly prevalent (and particularly in relation to innovation), and this is the term used in this article. The Knight Foundation (2013) describes an ongoing 20% annual growth rate in the launch of civic tech organisations, and has mapped out how civic tech can help citizens to more actively participate in democratic society, including: data access, visualisation, resident feedback, voting, public decision-making, neighbourhood forums, information crowdsourcing, crowdfunding and peer-to-peer sharing of goods and services. The growth in civic tech can be explained by technological advances enabling ubiquitous computing, as well as drivers for open data and greater democratic transparency and accountability.

Despite the rise of civic tech there are a number of potential issues of concern. One issue is how you assess their impact – both from a methodological and outcomes perspective. There are a number of impact-related questions which have had little attention within the literature. It is unclear how users interact with civic tech, how data can be usefully collected, and what the immediate and longer term impact of civic tech is within the context of use. A second concern relates to one of the key principles of the open government movement – participation – and in particular whether civic tech can enhance and broaden the democratic process, or whether it just provides another outlet for the already engaged and vociferous. A third area of concern is the lack of guidance for successful design of these platforms.

This study uses a multi-methods approach to tackle the above three concerns. It investigates the use of FixMyTransport by members of the public. FixMyTransport (hereafter termed FMT) was created by mySociety¹ in 2011 to allow individuals to report issues with public transport. Customer complaints have long been recognised as key to developing relevant service quality attributes and improving levels of service (Friman, Edvardsson, and Gärling 1998) and data relating to Rail in the UK (for which there are centrally collected statistics) show that the use of email/webforms has overtaken letters as the predominant means of customers complaining to the transport operator (Office of Rail and Road 2015). Gonçalves et al. (2014) describe how service providers can benefit from ‘apps’ that allow customers to report problems. Civic tech, such as FMT, provides a potentially more interactive and fulfilling method for persuasive messages from the public and engagement with transport operators in order to bring about service improvement.

FMT contained data on over 300,000 transport routes and stops, sourced from public transport data-sets including: National Public Transport Access Nodes (NaPTAN), National Public Transport Gazetteer (NPTG) and the National Public Transport Data Repository (NPTDR).² Aided by location, a user would select (1) a stop or station, or (2) a journey or route and could submit a free text title and problem description. An individual could either report a single issue, or start a campaign, or look for others having already reported the same or similar issues. The key feature of FMT was that it automatically linked to the responsible body or operator, so that, for example, a complaint about a particular bus stop was routed through to the owner of that asset, or an issue with a train service was sent directly to that train operator. Figure 1 shows part of the home page of the service. Within the broad range of crowdsourcing systems, FMT is a ‘user feedback-based crowdsourcing system’ as categorised by Misra et al. (2014).

Figure 1. Initial interface to FixMyTransport.

The overall aim of this research was to investigate the public’s use of a leading civic tech platform, FMT. The specific objectives were to (1) investigate whether civic tech was able to support democratic participation in society, (2) understand the context in which it is used, (3) identify the nature of the data gathered and (4) assess user perceptions of impact and intentions to use. As an outcome, the intention was to use the findings to provide guidance for the design of future civic tech platforms.

2. Related work

Most reports on civic tech or related initiatives have emerged since around 2010. In proposing a framework that can be used specifically for investigating citizen-sourcing platforms, Nam (2012) sets out three basic categories of (1) design evaluation, (2) process evaluation and (3) outcome evaluation. Design evaluation refers to sociotechnical, functional, procedural and policy design. Process evaluation refers to the act of participation and communication that the citizen-sourcing platform enables – and therefore this includes criteria such as transparency, participation and diversity. Outcome evaluation relates to effectiveness and impact – i.e. whether the initiative produces the results that are wanted.

Investigations into civic tech have tended to focus on data quality, rather than the human component within the system, or a broader view on the impact and added value of the civic tech proposition. The diversity of projects that come under the broad banner of citizen-sourcing or civic tech results in a lack of standard assessment criteria and metrics, particularly in relation to the impact of the initiative. In relation to the general phenomena of humans acting as sensors or data providers, and evaluating their worth, Lathia (2013) describes how methodologies ‘remain elusive’. Similarly, even in a relatively established field such as volunteered geographic data, a key issue is the lack of standardised methods and metrics for evaluation (Antoniou and Skopeliti 2015). In a specific study related to citizen science, Cox et al. (2015) developed a set of outcome-based metrics for the Zooniverse projects. The two key criteria used were (1) contribution to science, and (2) public engagement, and these were then broken down into measures and proxy metrics.

Prest (2012) identified five key elements that underpin crowdsourcing initiatives (and that could therefore be used as a basis for impact assessment): (1) Transparency refers to openness of data, but also how contributions within a citizen-sourcing process are moderated, collated and then used. (2) Participation is described as key, and in particular the involvement of diverse, representative groups. (3) Collaboration is important, either between citizens and other citizens, or in the case of e-government, between citizens and government. (4) Deliberation is particularly relevant for citizen-sourcing involving problem solving, and it is claimed that this builds trust, capacity and eventually greater participation. (5) Responsiveness is important because ‘citizens will continue to participate and engage in government initiatives only if they feel they are being listened to’.

Impact assessments specifically related to civic tech are just emerging. Uppström and Lönn (2013) describe an evaluation of a mobile platform for complaint and problem reporting targeted at Swedish municipalities and the local inhabitants. They use the framework of Nam (2012), but were unable to perform an outcome assessment due to the designed platform ‘not yet being in a productive state’. Lee, Almirall, and Wareham (2016) studied the broad impact that civic apps had made, particularly following open data initiatives and attempts to incorporate information into applications and services for public benefit. Although the methodology is unclear, their conclusion was ‘applications that had real impact for citizens or government were few’. Misra et al. (2014) discuss crowdsourcing specifically in the context of transport planning, and conclude that although there are issues with participation and data quality, well-designed projects and platforms have great potential for helping to resolve transport issues.

In relation to representation and democratic participation, Escher (2011a, 2011b) studied two civic tech sites developed by mySociety and found that a significant proportion of users were first-time users who might not otherwise have engaged in civic action. Forty per cent of those using WriteToThem³ had never previously contacted their Member of Parliament, and 60% of the users of TheyWorkforYou⁴ had never looked up information on their Member of Parliament. Similarly, Lee, Almirall, and Wareham (2016) investigated a range of civic applications, and also concluded that FixMyStreet⁵ (a similar concept to FMT, enabling reporting or discussing of local problems) was able to reach out to a ‘new demographic that would have been less likely to report through traditional channels’. In contrast, however, Cantijoch, Galandini, and Gibson (2016) concluded that civic tech (and specifically other mySociety sites) attracts individuals who are already engaged in civic engagement, and use online platforms as a means of supplementing and deepening their levels of civic or community engagement. Cantijoch also found that users were not representative of the general population, being older and predominantly male.

The need to better design civic tech was recognised by Foth and Parra Agudelo (2013). They seek an agenda ‘to design the next generation of “digital soapboxes”’. Two particular points they make are (1) the need for interaction designers to create better tools for connection, exchange and dialogue and (2) the need to exploit the capabilities of new technologies. A study by Baruch, May, and Yu (2016) identified some key enablers for success in relation to a specific example of civic tech: a satellite imaging crowdsourcing platform. They found that the following design features need to be incorporated into civic tech: the need for transparency with a platform (i.e. being able to see what others are doing); the ability of contributors to discuss with others; giving contributors control over what they do; generating a feeling of cooperation between contributors; a need to disseminate the outcomes with tangible well-communicated outputs; and increased interaction between platform/campaign coordinators and volunteers.

3. Study design

3.1. Mixed methods approach

An explicit mixed methods approach within user research is increasingly promoted – e.g. Onwuegbuzie and Leech (2006). With a mixed methods ethos, this study sought to generate representative response rates and also analyse causal mechanism and affective responses (Harrits 2011). Although most studies of civic tech have employed a single method of data collection, a notable example of a mixed method study in this field is that of Cantijoch, Galandini, and Gibson (2016). They used an online survey which was additionally used to recruit individuals for a longitudinal online diary, resulting in ‘a richer and more contextualised understanding’.

The current study used three main sources of data: (1) a data log of 10287 complaints generated when individuals reported transport issues using the FMT platform: (2) survey data from a subset of N = 140 of the individuals reporting problems; (3) follow-up interviews with a subset of N = 28 of the survey respondents. These data sources are described in more detail below.

3.2. Logged data

Each reporting of a transport problem by a member of the public using the FMT platform generated a log entry. Each log entry (N = 10287) included the following fields: unique identifiers, free text description of the problem, mode of travel, location when reported, organisation, problem status, date/time stamps, number of supporters (if it had been set up as a campaign). To preserve anonymity, no demographic data were collected via these logs, and although individuals could include their names and other personal information, problem reports could be completely anonymous if desired. FMT encouraged complainants to revisit and update their original complaint, prompted by email reminders. A link to a survey was included in these email reminders and the original complainant invited to take part in the short survey (described below). By clicking through to the survey, a unique problem identifier was transferred which allowed the linking of the survey response to the original problem log. There was therefore no explicit experimental control over the generation of the logged data.

3.3. Survey

The online questionnaire was implemented using BOS.⁶ It was limited to 20 short questions and designed to be completed in less than 5 min in order to maximise the click through and response rate. The aims of the questionnaire were to find out more about the demographics, attitudes and behaviours of the complainants, their interaction with the FMT platform, and to recruit individuals for follow-up. Respondents (N = 140) could opt for entry into a prize draw, and/or the opportunity for taking part in a short follow-up telephone interview. Since the completion of the online survey was based on who followed the link provided via FMT, there was no direct experimental control over who completed the questionnaire, or who chose to be available for follow-up interviews.

3.4. Interviews

Just less than 50% of the N = 140 survey respondents were willing to take part in a follow-up, and short structured interviews (of about 15 min duration) were undertaken with 28 of these respondents. The aim of these interviews was to better understand the answers given in the survey, and to investigate the motivations for using FMT. The interviews comprised the following sections: how they felt at the time, why they were prompted to make a complaint, the support they received and its impact, comparison of FMT with other ways of making complaints about public transport. The interviews also determined participants’ reasons for the following survey responses: usefulness of FMT, device used to report the issue, the time they reported it, whether they would use FMT again and/or recommend it to others.

3.5. Sampling strategy

The telephone interviews (described above) provided the only opportunity for purposeful sampling based on the responses provided during the prior survey phase. Interviewees were selected according to a cross-sectional strategy, based on key factors that would influence attitudes and behaviour relevant to this study: technology acceptance (Rogers 2003), using an adapted scale used previously (May, Bayer, and Ross 2007); previous reporting history of transport-related problems; stated usability influencing attitudes and use (Szajna 1996). It was not possible to generate a fully balanced sample based on the survey responses, and therefore the interviews were completed with individuals who represented diversity in relation to: (1) whether they indicated that they were relatively early or late adopters of technology; (2) whether they had previously made transport-related complaints; and (3) the degree to which they had found FMT to be useful.

3.6. Procedure

Data from the complaint logs and surveys were processed and collated on an ongoing basis. Resulting survey responses were categorised according to whether they stated they were willing to complete a follow up-interview, and how they scored on the three main sampling criteria (described above). The telephone interview was undertaken with chosen participants as soon as possible after their survey entry. This was audio recorded.

3.7. Data verification

Duplicate reports were deleted (based on consecutive problem IDs referring to the same problem) – to remove subsequent, repeated reports of the same issue with either the same name or aliases. This resulted in 73 duplicates (0.7%) being removed. Note that multiple reporting of the same problem at different times (non-consecutive problem IDs) was retained as indicative of an ongoing problem.

For the survey respondents, there was a potential incentive for multiple responses due to entry into a prize draw. Deliberate or inadvertent multiple survey submissions were checked based on the unique numeric code that was transferred from the problem log to the survey, and by checking for identical consecutive responses and duplicate email addresses. There was one ‘disguised’ multiple entry to the prize draw where a different name, email address and phone number were used (with all other fields identical). For the telephone interviews, the main concern was to maximise the accuracy of recall of the issue, and this was done by completing the interviews on a rolling basis, as soon as possible after the survey response. The interviews also aided recall by initially summarising the problem as stated in the data log, and referring back to each of the survey responses that had been provided.

4. Results

4.1. Profile of participation

No demographic data were available from the log data. Of the survey respondents (N = 140), 55% were male, 43% were female and 2% did not answer. The age distribution of the FMT respondents (survey) is shown in Figure 2, together with the UK population age distribution (ONS 2015). Attitudes to technology adoption, based on the adapted criteria described above, are shown in Table 1.

Figure 2. Age distribution of FMT respondents (N = 138).

Table 1. Responses to adapted technology acceptance scale (N = 140).

Technology adoption category	Frequency (%)
I like to be the first with new technology	19.3
I like to try out new technology’	37.1
I like to wait and see with new technology	38.6
I am slightly sceptical about new technology	3.6
I do not like new technology	1.4

One of the key aims of this study was to investigate the extent to which FMT was enabling broad participation, from both those who were, and were not accustomed to making travel complaints. Four categories of complaint behaviour were investigated: (1) use of FMT, (2) formal complaints (using recognised channels such as web forms, telephone contact, report forms), (3) informal complaints (such as directly to a member of staff present, where a formal response is not normally requested), (4) web post (e.g. on a forum or using social media, where it is not directed to a specific individual). Figure 3 shows how often, over the previous 12 months, the respondent had used FMT, made a formal complaint through standard channels, an informal complaint or made a transport-related post on social media.

Figure 3. Number of times respondents had used FMT, made an informal or formal complaint, or commented on social media, in the previous 12 months (N = 140).

4.2. Context of use

Few of the interview participants accessed FMT directly or searched specifically for it; instead it appeared prominently after searching for contact details in order to lodge a complaint. A small proportion recognised the branding since it was similar to other relatively well-known sites developed by mySociety such as FixMyStreet.

The FMT website was designed to be fully mobile compatible. However, in 82% of cases, FMT was accessed using a desktop PC or laptop, with only 9% using a tablet and a further 9% using mobile phone with web access. This is consistent with when the problem was reported, with only 14% reporting it at the time it happened, 1% later during the journey, 17% after they had arrived at their destination and 67% reporting the issue later on during the day or at a later date. Where a mobile phone was used to access FMT, in the majority of the cases, this was in order to report it at the time it occurred. In comparison, the majority of the tablet users reported the problem later on that day or at a later date, i.e. were more similar to the desktop or laptop users. Ease of use of FMT was excellent, with 95% of respondents (survey, N = 138) stating it was ‘quite’ or ‘very’ easy to report a problem using FMT. There was no statistical difference in stated ease of use when comparing different device types, or whether an individual had, or hadn’t, used FMT before.

4.3. Nature of the data gathered

To assess the extent to which users contributed free text of sufficient detail, a frequency distribution was undertaken of the number of characters used by customers to report the issue using the FMT platform (N = 10287). The mode response category was between 100 and 150 characters, and the median 441 characters, reflecting the long tail of the distribution. Of note is that there were 30 instances of problem reports comprising more than 4000 characters, including one that was 14795 characters long. The boxplot in Figure 4 shows the impact of device type on the number of characters used to submit the problem reports. The median values (solid horizontal line) are desktop/laptop: 620; tablet: 314; mobile phone: 398. The box represents the interquartile range (IQ) based on the 25th and 75th percentiles, being the central 50% of the distribution. The whiskers extending from the end of each box show the largest and smallest observed values that are not statistical outliers. Outliers (at a distance of between 1.5IQ and 3IQ from the end of the box) are shown with a circle.

Figure 4. Impact of device type on the number of characters used to report issues.

To understand to what extent the free text in the logged data was capturing experiential or emotional data, respondents to the survey (N = 140) were asked to provide a single word or short phrase that represented how they (recalled they) felt when they encountered the transport issue. The responses, with size of word representing frequency count, are shown below in Figure 5. In addition, based on the survey data, 81% stated they felt ‘strongly’ or ‘very strongly’ about the issue.

Figure 5. Words used to report passenger’s feelings in survey (increasing size according to frequency).

Tracing these same respondents back to their logged data, their original problem reports submitted using FMT, and their survey responses, were coded for affective and cognitive components of subjective well-being based on the theoretical framework of Ettema et al. (2010). The ‘affective’ component refers to an individual’s emotional state whilst ‘cognitive’ is a more evaluative judgement of how things are. The results of this analysis are shown in Figure 6.

Figure 6. Frequency of affective and cognitive responses expressed in matched logged data and survey responses.

4.4. Resulting impact and future intentions

4.4.1. Generating a response

A key feature of the FMT platform (and how it differed from more standard email and web form reporting) was that it enabled responses from other passengers, volunteers at FMT or the specific transport operator to be linked to the initial complaint. In addition, the customer was able to mark their issue as resolved if they felt it had been addressed satisfactorily. Table 2 shows the nature of the responses obtained through the platform. Note that the percentages are based on the total number who completed the survey (N = 140) and do not sum to 100% as typically more than one response was obtained in relation to each reported issue.

Table 2. Nature of responses received by complainants (survey, N = 140).

Type of response	%
Received a response (of any type)	79
Had supporters join the campaign	13
Had support from FMT	29
Had a standard response from the transport operator	36
Had a personalised response from the transport operator	27
Had been told the issue has been or will be tackled	25
Stated they hadn’t checked	4
Did not answer this question	16

The interviews (N = 28) showed that the input from FMT was encouraging for the complainants, for example: ‘it was nice to get an e-mail [from FMT] asking if the problem had been resolved’; however, the actual responses from the transport operators were mixed. In some cases, there was no response, a generic response or a reply stating that the operator would not respond through the website. The percentage of cases that were actually formally logged as resolved, by the original complainant going back and updating their complaint was 14% (based on the log data, N = 10287).

Figure 7(a) shows the number of comments that individual problem reports generated. In 49% of all problem reports (N = 10287), individuals chose to start a campaign – i.e. to enable others to support their complaint. The number of supporters per campaign is shown in Figure 7(b).

Figure 7. Frequency of number of comments (a), and supporters of ‘campaigns’ (b).

It is interesting that the above-mentioned figures include an extreme outlier with 118 comments and 262 supporters for one specific campaign (a long-standing and well-organised campaign for more bicycle parking at Cambridge Rail Station).

4.4.2. Influences on perceived usefulness

74.5% of respondents (survey, N = 140) judged FMT as ‘quite’ or ‘really’ useful. There was no statistical difference in perceived usefulness between those who had or hadn’t, in the last year: (1) made a formal complaint (using an alternative means); (2) made an informal complaint; (3) posted on a transport-related web forum. However, FMT was judged as significantly less useful by those who had reported more than one problem using it in the last 12 months (U = 1536, p = .017). The difference in these two groups is shown in Figure 8.

Figure 8. Impact of previous usage on perceived usefulness.

Based on the interviews, FMT was judged as ‘really useful’ if: the complainant received a quick response from the transport operator (even if only to acknowledge the complaint); the problem was resolved as result of the complaint; the underlying motivation was to make themselves heard; or previously, it had been difficult identifying how to complain.

FMT was judged as ‘quite useful’ by respondents where: they understood that the complaint was being directed to the appropriate person; they were relieved to have found a single place to address transport issues; they felt they were able to make their voice heard (but were not sure what had happened as a result); they were offered moral support by other passengers; they valued being able to state their case in a public forum; it was an avenue for venting frustration; it was seen as a means of taking positive action and/or seeking address; they had received only limited feedback or an automated reply from the transport operator; where they felt similar issues had not been connected; or where they were aware that FMT itself had no power to bring about a resolution to the problem.

The negative attitudes that resulted in FMT being judged to be ‘of no use’ or ‘not very useful’ were influenced by the following: if they were sceptical that the transport operators would do anything, due to a lack of legal obligation; if the operators refused to engage with the site; if they received no feedback; if they didn’t know it existed and stumbled up it by accident; if nothing had changed as a result of the complaint; or if they felt their complaint was reported as a one-off, with no support from others.

4.4.3. Intention to use it again

68.4% of respondents (survey, N = 140) stated it was ‘quite’ or ‘very’ likely that they would personally use FMT again. Those who had made one or more formal complaints (using alternative traditional channels) in the last year were significantly more likely to use it again than those who hadn’t (U = 1667, p = .002). This is shown in Figure 9 .

Figure 9. Impact of other formal complaints behaviour on FMT usage intentions.

Passengers were quite or very likely to use it again if: they were heavily reliant on public transport; there was direct feedback in response to a comment; there had been successful prior use of FMT (even though other avenues may have been also used). Passengers stated they were not likely to, or undecided about using it again where: they felt it didn’t work by being too micro and small scale; it was unclear how effective it had been or would be in terms of resolving a complaint; it only acted as a means of venting frustration; or they were unsure of the geographical scope of FMT.

4.4.4. Providing future recommendations

68.4% stated they would be ‘quite’ or ‘very’ likely to personally recommend FMT to a friend in a similar position. Those who had made at least one formal complaint (using alternative channels) in the past year were significantly more likely to recommend FMT than those who had not (U = 1881.5, p = .038), see Figure 10.

Figure 10. Impact of other formal complaints behaviour on FMT recommendation intentions.

In addition to the enablers highlighted above, passengers were more likely to recommend it to a friend when they thought (erroneously) that it was part of an official website, or valued the independent nature of it; found it easy to find due to a catchy name and prominence in web searches; wanted to help empower other individuals; wanted to build a critical mass of users; or had friends or colleagues who were interested in the same issue.

5. Discussion

5.1. Profile of participation

The demographics data show that the users of FMT represented a broad and fairly representative cross section of the public. The gender split was balanced, and the age profile is broadly consistent with that of the UK population (Figure 2). This is in direct contrast with Cantijoch, Galandini, and Gibson (2016) who observed an older and more male-biased sample for users of civic tech. Note that the UK population figures in Figure 2 exclude those <18 and 75 + as the upper and lower bounds of the FMT sample are unknown. The FMT sample slightly under-represents the younger adult and slightly over-represents those around UK state retirement age. It is likely that this is influenced by the degree of free time available, particularly given that most complaints were reported some time after the issue was encountered. The attitudes to technology adoption (Table 1) indicate a sample slightly biased towards early adoption, as would be expected given the online reporting medium, when compared to the more symmetrical distribution described by Rogers (2003).

Figure 3 shows that the individuals who used FMT were generally not engaged in regular transport commentary – i.e. would not be termed ‘activists’ or ‘campaigners’. This is consistent with Escher (2011a, 2011b) and Lee, Almirall, and Wareham (2016) but again in contrast to the findings of Cantijoch, Galandini, and Gibson (2016). It is likely therefore that FMT was used more as a reporting tool than as a campaigning tool; this is supported by the general low rate of support of ‘campaigns’ instigated by others (Figure 7), discussed in more detail in Sims, Ross, and May (2013).

5.2. Context of use

Although the FMT site was fully mobile accessible, a surprising result was the relatively low rate of reporting using a mobile device at the time the incident actually occurred. Reporting a problem directly after it has occurred has many benefits including the ability to capture feelings as close as possible to their occurrence, rather than relying on a late recall (Bolger, Davis, and Rafaeli 2003). In addition, this can prevent retention degradation (Reis and Gable 2000) and minimise limitations of delayed recall such as the availability bias (high-impact incidents become more salient in memory), the reconstruction process – memories of facts and events may depart from how they actually occurred – (French, Maule, and Papamichail 2009) and the distortions inherent in recollecting emotions (Reis and Gable 2000). Instead, most individuals waited until they had reached their destination or returned home before using FMT, and some described how they had actually submitted the complaint one or more days later. The data do not show the extent to which delayed reporting came about as a necessity rather than through choice. However, some stated that they had chosen to seize the moment: ‘if I’d left it you know till the next day, I probably wouldn’t have bothered, I’d have calmed down’ – a result also found in the study of Gonçalves et al. (2014) of SMS-based feedback on public transport. The interviews clearly showed that the use of FMT was not merely about real-time problem reporting at the time it occurred. It was additionally to do with responding to a situation that was perceived as unjust and unnecessary (‘this is still playing on my mind’), and looking for support and corroboration from others.

5.3. Nature of the data gathered

The design of data contribution for civic tech is particularly challenging where there is (1) an inherently mobile context of use (e.g. transport) and (2) wide potential diversity in content (such that form-based or cascading categorisation would not provide sufficient descriptive power). Free text response offers the greatest flexibility for data contribution, and has been shown to be the most intuitive means of entering data with a mobile device (Dai, Lutters, and Bower 2005). However, it was unclear the extent to which users would be willing to contribute free text responses. Within this study, a typical problem description ranged from 20 to 80 words (consistent with the familiar 140 character limit of Twitter). Figure 4 also shows a very long tail distribution, indicating the potential for unconstrained free text entry.

Parker, May, and Mitchell (2014) highlighted how volunteers using an online platform will contribute not just objective data, but also experiential and perceptual (or emotively derived) data. The frequency with which components of subjective well-being were present in the free text of the logged data differed from those provided via the survey to describe how the passenger felt (Figure 6). It was clear that enabling a free text response to describe the problem elicited fewer indications of the affective component of subjective well-being than a direct request of feelings. This is important when attempting to capture passenger experience. Carreira et al. (2014), in a study with 1226 bus passengers, determined that ‘experience components’ (emotions, value and satisfaction) have a significant impact on customer loyalty. These components are close to the affective and cognitive components of subjective well-being used to analyse the data in this study – after Ettema et al.’s (2010) Satisfaction with Travel Scale. If transport providers are to understand the experiences of their passengers then gleaning affective and cognitive responses is important if they wish to retain their loyalty. This is particularly pertinent considering that the majority of issues prompting the negative affective responses indicated by the words in Figure 5 are within the control of the transport operator.

5.4. Resulting impact and future intentions

Table 2 shows that in the majority of cases complainants did receive a response comprising one or more of: other travellers, variety of response from the transport operators and/or from FMT admin who made efforts to contribute to postings. The percentage who did not receive any response is likely to be around 20%, although it is not possible to distinguish between those who had or had not specifically checked the website in order to report a lack of response. It is noticeable that the most frequent response was the transport operator sending a standard response to the individual, and in many of these cases this response asked the complainant to use the standard complaints process channels. Although a quarter received a response stating that the problem had been or was going to be resolved, it appeared that only about half of these individuals subsequently logged on the website to check that the issue had been resolved. Within crowdsourcing platforms, feedback has been shown to be a key enabler for garnering and maintaining participant interest (Baruch, May, and Yu 2016). If a ‘campaign’ was started by a complainant, there was considerable variability in the support these garnered. Sims, Ross, and May (2013) found that the more successful campaigns instigated using FMT were characterised by calm, factual and evidence-based reporting.

FMT was reported as being significantly less useful by those who had used it previously. Corroboration from the interviews suggests that first-time users were influenced positively by its functionality in terms of reporting issues, whereas more experienced users tended to judge it more negatively in relation to ultimate impact. This is consistent with the usability literature – e.g. Kujala et al. (2011) – that shows that usability criteria and associated judgement depend on the system and domain experience of the user. Those with some track record of making transport complaints were significantly more likely to use it again and more likely to recommend it to a friend. This contrasts somewhat with those who had used FMT previously judging it relatively less useful. However, this indicates that individuals with more experience of reporting transport problems had a desire to use an effective reporting mechanism, but also understood that reporting the issues may not necessarily bring about change.

5.5. Limitations

The main limitations of this study relate to the sampling biases which were introduced at varying stages in the study, and an inability to collect demographic data from the logged data in particular, in order to accurately assess the representativeness of the large log data sample. However, the distribution of transport modes from the complaint logs (bus: 76.9%, train: 19.9%, tram/metro: 2.7%, coach: 0.4%, ferry: 0.1%, N = 10287) is broadly in line with the GB distribution of passenger journeys across modes of transport, albeit slightly biased towards bus journeys.

The self-selecting nature of the online questionnaire would have introduced additional response bias. The transport mode distribution from the log data (N = 10287) was used to calculate expected values for the questionnaire (N = 140) and interview (N = 28) data. A Chi-Square test showed that the mode distributions in these questionnaire and interview data-sets were not significantly different to the logged data. Other data from the questionnaire showed that these respondents did not appear to be biased in terms of age, gender or previous complaints history. Of course, a key limitation is that this study does not provide insight into those individuals who did not use FMT, either through lack of awareness, or through choice – and therefore does not highlight the issues to do with non-use of civic tech.

6. Conclusion

The conclusions of the study were that FMT was a successful initiative that enabled easy and effective reporting of transport issues. However, its ultimate impact was more mixed as although the reporting and support aspects worked well, the degree of engagement from transport operators and their resources to actually address problems resulted in more limited ultimate resolution of issues.

The participation was ‘democratic’ i.e. contributors were not niche in terms of age, gender, attitude to technology or degree of activism in relation to transport issues. This suggests that easy-to-use web-based interfaces are an effective way of enabling wider participation from the public.

Free text entry (as opposed to a categorisation approach) was effective for users, and enabled a wide range of customer responses. Many individuals chose to postpone their problem reporting, and subsequently provided detailed descriptions where appropriate. FMT was clearly much more than just a reporting tool, and was valued as a means of ‘making yourself heard’, being part of a larger group interested in an issue and trying to right a wrong. Feedback to the customer was very important, and the contributions from other passengers and from FMT were valued, even if there was only a generic response from the transport operators.

Several key findings can be generalised to guide the design of other large-scale civic tech projects as follows: (1) No assumptions should be made about the types of people who will contribute, and an effective and easy-to-use platform that addresses everyday issues can appeal to a wide audience. (2) A platform should strip away as much of the data and process-related complexity as possible and present a simple concept to the end user. (3) The degree to which real-time (as opposed to post-event) data capture occurs will be influenced by contextual factors that will constrain in situ and/or encourage subsequent reporting. Platforms should be designed so that the ‘facts’ can be reported (or at least recorded) at the time of occurrence. However, impact, consequences and considered opinion may emerge later. (4) Explicitly eliciting emotions is important as they will not necessarily emerge through unsolicited free text responses and are important as indicators of the experience of the service and impact on customer loyalty. (5) Feedback to the contributors is very important, from the various stakeholders, so platforms should enable conversations built on information, rather than mere transactions with data. (6) Community is empowering, and platforms should aim to both make the individual feel integral to a community, and also harness the power of communities to bring about positive change within society.

Disclosure statement

No potential conflict of interest was reported by the authors.

Funding

This study was supported by the UK Government via the Engineering and Physical Sciences Research Council [Grant Ref. EP/F005172/1], the Technology Strategy Board [Grant Ref. 400050] and the Department for Transport.

Acknowledgements

This study was initiated during the lifetime of the Ideas In Transit project (www.ideasintransit.org) supported by the UK Government via the Engineering and Physical Sciences Research Council under Grant Ref. EP/F005172/1, the Technology Strategy Board under Grant Ref. 400050 and the Department for Transport. Thanks are due to the study participants and the staff at mySociety who provided valuable assistance throughout. Please feel free to contact the authors for further details of the study.

Notes

1. mySociety is a project of UK Citizens Online Democracy, a registered charity.

2. Available at: https://data.gov.uk/data-set/naptan, https://data.gov.uk/data-set/nptg, https://data.gov.uk/data-set/nptdr/ respectively.

3. www.writetothem.com – provides a way of messaging your local or national political representative.

4. www.theyworkforyou.com – enables you to find out about your Member of Parliament.

5. www.fixmystreet.com

6. https://www.onlinesurveys.ac.uk/