How was it for you? Experiences of participatory design in the UK health service

Abstract

Improving co-design methods implies that we need to understand those methods, paying attention to not only the effect of method choices on design outcomes, but also how methods affect the people involved in co-design. In this article, we explore participants' experiences from a year-long participatory health service design project to develop ‘Better Outpatient Services for Older People’. The project followed a defined method called experience-based design (EBD), which represented the state of the art in participatory service design within the UK National Health Service. A sample of participants in the project took part in semi-structured interviews reflecting on their involvement in and their feelings about the project. Our findings suggest that the EBD method that we employed was successful in establishing positive working relationships among the different groups of stakeholders (staff, patients, carers, advocates and design researchers), although conflicts remained throughout the project. Participants' experiences highlighted issues of wider relevance in such participatory design: cost versus benefit, sense of project momentum, locus of control, and assumptions about how change takes place in a complex environment. We propose tactics for dealing with these issues that inform the future development of techniques in user-centred healthcare design.

Keywords::

• participatory design
• experience
• service design
• health
• user-centred healthcare design

Introduction

The UK National Health Service (NHS) is the largest single provider of healthcare services in the world. Innovation in NHS services is being driven by an ageing population, often with multiple health problems; a rising incidence of chronic health conditions and subsequent need for self-care; raised expectations from service-users accustomed to efficient, customer-centred commercial services; and a desire from some to be active and informed participants in decisions about, and the delivery of, their healthcare. Methods and principles from design, in particular service design, are increasingly being seen as providing a strategy for such health service innovation (Mager 2009).

Healthcare services are often complex, relying on interactions among multiple stakeholders. Stakeholder participation has been shown to be a beneficial component of service design, leading to innovation, a closer fit to user needs and improved service experiences (Steen, Manschot, and De Konig 2011). However, the benefits can be even wider reaching. Sangiorgi (2011) argues that service design is evolving into a form of design for transformation, delivering not only service changes but also increased capacity and supporting resources for organisations and communities to drive change themselves. Participatory service design then becomes ‘an engine for wider societal transformations’ (Sangiorgi 2011, 30). Manzini and Rizzo (2011) have also argued that integrated participatory design (PD) initiatives can contribute to wider visions of social transformation.

The project discussed in this article is part of a wider research programme on user-centred healthcare design (UCHD) to understand and develop participatory methods for health service design for the NHS with a goal of transformation. As Winschiers-Theophilus, Bidwell, and Blake (2012) argue, PD methods and practices must always be adapted to the cultural setting in which they are applied, and health services in general and NHS hospitals in particular have distinctive cultural attributes. Particiular features are strong professional hierarchies, high degrees of specialisation which can lead to working in silos, and an (understandable) aversion to risk. Owing to these factors, there is a preference for techniques that have been afforded some internal legitimacy. The NHS Institute for Innovation and Improvement, in collaboration with academic researchers, has developed an approach named experience-based design (EBD), which builds on established methods of service improvement within the NHS, has received recognition from NHS ethical governance bodies and is being widely promoted within the NHS (Bate and Robert 2007). As the ‘state of the art’ for service design and improvement in the NHS, EBD was adopted as the focus for a baseline service design case study with the goal of understanding its strengths and weaknesses before seeking to develop improved methods for UCHD.

We have reported elsewhere on our own experiences and reflections of using EBD as designers and healthcare practitioners (Bowen et al.2010, 2011; Wolstenholme et al.2010),through which we developed a view on the value and limitations of the approach. In this article, we explore the experiences of other stakeholders in the same service improvement project to enrich our understanding and develop a clearer assessment of the strengths, weaknesses, barriers and facilitators when using EBD. The insights from this evaluation are informing the creation of implementable methods in our wider research programme. Our aim in this article is to explore how the project reveals issues of wider relevance to participatory health service design and to suggest general tactics for dealing with them.

Related work

Like Bossen, Dindler, and Iversen (2012), we found very few studies in the literature examining the outcomes of PD projects and the experiences of participants. Hirschheim (1983) presents a review of ‘participative systems design’, which concludes that little conclusive evidence is provided of benefits. But, aside from the age of this study, the evaluation is framed solely in terms of qualities of the resultant systems without attention to participants' own experiences, sense of agency and capabilities. Pilemalm and Timpka (2008) report on possible obstacles to PD in a large-scale information systems project, and suggest a ‘3rd generation PD’ framework for dealing with them. Their report is based on participant observation in a participatory action research project, together with post-project interviews exploring the barriers and the new framework. Sjöberg and Timpka (1998) use Grounded Theory to analyse recordings of design meetings in a medical informatics project to explore the structure of discourses within the design process and their effect on the project outcomes, but not participants' experiences of the activity. Lin et al. (2011) explore the implementation of previously co-designed health service ideas, rather than experiences of the co-design process itself. Their evidence is centred on designers' experiences, with other stakeholders' perspectives presented third hand via designers' reflection. In this article, we foreground other stakeholders' own accounts of their participation and use these as a basis for discussion.

Perhaps the closest comparable studies are those of Bossen, Dindler, and Iversen (2010, 2012), who conducted and analysed semi-structured interviews with project participants to explore what participants ‘gained’ from PD activities. In our case, we focus less on notions of individual gain in the PD process, and instead attend to participants' experiences of engaging in participatory processes, understanding how participants valued EBD and the different activities within it, and how they felt about the progress and outcomes of the project, and compare these with our own experiences and reflections. Like Bossen, Dindler, and Iversen (2010, 2012), we used semi-structured post-project interviews and thematic analysis.

Structure of this article

In the next section we provide some background on EBD, the service improvement project in which we used it, and our previous reflections on the approach. We then describe our evaluation approach based on participant interviews and thematic analysis. In the Findings section we present the resultant themes in two broad areas relating to participants' accounts of using EBD and their reflections on EBD in general. This analysis includes an ‘emotional map’ of their experiences of the process. Finally, we discuss the strengths and weaknesses of EBD as identified by this evaluation, and the wider implications for PD in health services.

Background

Experience-based design

In discussing EBD as an answer to the NHS's aim for patient-led care (Department of Health 2004), Bate and Robert (2007) propose ‘co-designing services with the patient’ and look to PD (citing Schuler and Namoika 1993), human-centred design and related subdisciplines as a relevant community of practice. The common principle they adopt from this diverse field is an ‘unswerving commitment to the direct involvement of users in designing their own experiences’ (Bate and Robert 2007, 19).

EBD is structured as a four-phase process of patients, carers and healthcare staff capturing and then understanding their lived experiences of healthcare services, working together to improve the service based on this understanding, and then measuring the effects of changes (Bate and Robert 2007; NHS Institute for Innovation and Improvement 2012). EBD is presented such that healthcare staff can lead the process without professional designers, and online and printed resources provide methods and materials for undertaking each phase.

In the capture phase, participants are encouraged to record their personal stories of using services, and then staff and patients participate in separate story-sharing events.

In the understand phase, participants analyse their experiences by plotting elements of their stories on emotional maps (Figure 1). These maps identify the touchpoints with the service (interactions such as a letter, telephone call or in-person contact) and explore feelings associated with participants' encounters with those touchpoints, as evident in the stories. Emotion words drawn from the stories are written on Post-it notes, which are then placed in line with the identified touchpoint. Positive emotional reactions appear higher on the map and negative emotions lower, thus creating a visual representation of the emotional journey. Clusters of negative responses indicate problems to address. Two emotional maps are created, one compiled by patients and the other by staff. The final part of the understand phase is sharing these maps and collaborative discussion of priorities for improvement.

Figure 1 The emotional map constructed from stories about using the outpatient service.

For the improve phase, EBD establishes a set of co-design teams each involving both patients and staff to explore and implement service improvements in different areas. However, it provides relatively few specific methods for doing so. It does mention using a worksheet to encourage participants to identify ‘rules of thumb’ that appear to produce positive experiences elsewhere in this service, or in other services. Reminder cards and record sheets are also used to note participants' specific suggestions and actions. The rules of thumb idea is based on the concept of design pattern languages (Alexander 1977, 1979; see also Dearden and Finlay 2006) and is mentioned in the academic publication about the method (Bate and Robert 2007) but not in the ‘user guide and tools’ booklet for practitioners.

Other than proposing a shared evaluation between participants, the EBD resources similarly provide limited specific guidance about the measure phase.

Better Outpatient Services for Older People

Better Outpatient Services for Older People (BOSOP) was a 12-month service improvement project focused on the medical outpatient service for older people at the Royal Hallamshire Hospital (Sheffield, UK), with the intention that some of its outputs would be generalisable to other outpatient departments in the hospital and other hospitals in the wider healthcare trust (henceforth, the Trust).

Twenty-one patients, carers and healthcare staff formed the core project group, taking part in EBD activities (although attendance varied for individual events). The eight staff were two nurses, a ward sister, a health support worker, a clerical worker, a receptionist, an ambulance dispatcher and a hospital volunteers' coordinator. All were involved in front-line work with patients, with the ward sister and volunteers' coordinator also managing other staff's work. Thirteen participants represented users of the outpatient service: three older patients (plus one older patient's daughter/carer) recruited by the hospital department, four older patients recruited via a local charity that supports older people in hospital attendance, and six staff from the same charity who acted as advocates.

A steering group was established to oversee the project, consisting of managers and patient representatives, including the outpatient department's matron and nurse director, a Trust practice development coordinator, a representative from the hospital's patients' forum, the charity's director acting as an advocate for patients' interests, one of the participating nurses and four representatives from the research team facilitating the project. In addition, specific expertise was brought to the project during the improve phase, including two postgraduate graphic design students, a Trust estates manager and a local authority road planner/engineer.

Following setting up and recruitment to the project, we undertook the capture, understand and improve phases of EBD within 12 months, with the measure phase taking place later. Some additional activities took place, including a final celebration event, where a training resource DVD of patient stories was launched, and presentation of key outputs to stakeholder groups within the Trust.

The principal project outputs were: a new template for patient appointment letters, design proposals for new way-finding materials (signage and maps), a proposal for layout of roads surrounding the outpatient building, a video of patient stories distributed to all staff in the Trust to highlight older people's experiences, and a forum theatre training event to improve outpatient staff awareness of customer care. The new letter template was adopted by several outpatient clinics immediately following the project, and several elements of the design proposals have been implemented in new maps and signage and improved street design outside the outpatient building, but these changes were implemented two years after the end of BOSOP and interviewees were not aware of them at the time of interview.

Evaluating experience-based design as design researchers

We have previously described the activities and outputs of BOSOP (Bowen et al.2010, 2011; Wolstenholme et al.2010), and developed initial reflections on the strengths and weaknesses of the approach from our own perspectives. We reported our view that the sharing of stories and emotional maps helped to build trust and create alliances for change between patients and staff (Bowen et al.2011). In the improve phase, we noted EBD's relative lack of ideation tools, and our view that the co-design teams tended to converge early on simple ‘quick fix’ solutions without sufficient divergent thinking; that EBD does not highlight the value of making to explore complex problems (Bowen et al.2010); that EBD's reliance on pattern languages as a tactic for ideation is problematic owing to the lack of a pattern language for this context; and that involving trained designers who are not direct stakeholders can help to generate novel ideas, but can adversely affect core participants' ownership of the change process (Bowen et al.2011). We also noted that storytelling in EBD means that the focus of service improvement can extend much further than the clinical encounter or even the single hospital visit (Wolstenholme et al.2010). This is an advantage in offering a holistic understanding of patients' experience, but can be problematic because effective change is then dependent on engaging and persuading agents outside the project to take action (Bowen et al.2011). This finding highlighted that the institution does not deliver care in the form of a coherent journey, but rather that care is delivered in silos (Wolstenholme et al.2010).

Overall, we were concerned that there had been only limited tangible service improvement at the end of the project. This article is an effort to test our previous hypotheses against the viewpoints of other stakeholders and to refine our understanding.

Evaluation method

The post-project evaluation consisted of a series of interviews with a sample of 11 project participants, nine via telephone, lasting on average for 28 minutes (range 12–72 minutes, 308 minutes in total). Interviewees were selected to reflect the composition of the project group, and comprised two Older Patients, one Older Person's Advocate, four front-line outpatient staff (Nurse, Sister, Receptionist, Clerical Worker), one middle manager (Matron), one senior manager (Nurse Director), one external expert (Estates Manager) and one Graphic Designer. Interviewees were given £10 shopping vouchers in recognition of their time. One older patient was unable to participate because of ill-health, so another patient participant was interviewed in her place. Two management staff and one front-line staff member declined to be interviewed, as they did not perceive themselves to be fully involved in the project and felt they would have nothing to contribute.

Two of the current authors, who were not members of the original BOSOP project team, conducted the interviews face to face or by telephone. Each interview followed the same semi-structured format, using an interview schedule consisting of open questions prompting participants to provide their own account of each stage of their project. In addition, drawing on the traditions of realistic evaluation (Pawson and Tilley 1997) and recognising the importance of testing the design researchers' previous hypotheses, participants were asked to respond to statements that declared the design team's perspectives and beliefs about specific questions. The interviews were transcribed in full for analysis.

Data analysis took a thematic approach (e.g. Boyatzis 1998) and each transcript was coded and cross-checked by both interviewers. Analysis involved an initial stage of data familiarisation to gain an overview of the data set, and each interview was summarised to ‘map’ commonalities in accounts. This was followed by the development of a coding framework, combining both inductive and deductive approaches (e.g. Fereday and Muir-Cochrane 2006), identifying themes emerging from participants' accounts, but also investigating topics of specific interest to the design research team. The coding framework development involved a series of stages of cross-checking and modification, and was carried out by both interviewer/analysts to ensure reliability. The framework was also developed in regular consultation with the design team, who were able to contextualise some parts of the data to support coding. As the evaluation was concerned with participants' experiences of participating, the analysis paid particular attention to any evaluative statements or descriptions of emotional states in their accounts. Overarching themes were identified, which are discussed below.

In addition to the overarching themes, the analysis developed a sense of each participant's experience (or ‘participant journey’) through the project. To this end, the positive and negative evaluations identified during coding were used to produce a ‘map’, similar to the emotional maps employed during the BOSOP project, to provide a visual overview of participants' experiences of the project.

Findings

The initial analysis classified low-level themes into five broad areas: story-sharing, emotional mapping, the design process, project constraints and group dynamics. In further exploring subthemes it became clear that some themes relate to interviewees' experiences of the particular techniques and methods used in EBD, and others relate to interviewees' reflections on their emergent experiences of the whole project. We follow this division in presenting the findings below.

‘Doing’ experience-based design

Sharing stories and emotional mapping

The distinction between storytelling, story-sharing and emotional mapping was not clear to interviewees, and several needed reminding of the mapping activity. In practice, the activities overlapped because constructing and presenting emotional maps required that participants share their stories. However, similar themes appeared in participants' discussion of both activities.

Several interviewees discussed the benefits of making their perspective understood by others, and enabling them to appreciate others' perspectives: ‘Tell[ing] our stories helped members of staff to see into situations a little more deeply. … talking about the way in which it felt like to be a patient’ (Older Patient) and ‘… the patients listen because I don't think they realise what hoops we have to jump through to get them seen’ (Staff Member).

Participants recognised that sharing stories and constructing emotional maps had enabled common themes and shared concerns to be identified: ‘The patients brought it up and we brought it up … Everybody put it on the little stickers. The ambulance people did, the girls who work on the ambulance reception did, the nurses did. So it wasn't just an issue for the nurses … it was an issue for everybody’ (Staff Member).

Co-designing

All members of the project group were involved in the co-design activities; however, the interviewees' perception was that they were not ‘doing’ the designing. One Staff Member suggested that the emotional mapping ‘was for the design team to see, if we put something down they might pick on it and take it further’, another's view was that ‘We didn't design [the service improvements], we just told them what might work and they designed it’ and one Older Patient replied that she was not involved in any design meetings when they were described to her.

The Graphic Designer who became involved in the later stages of the project (the improve phase) also describes a process of consultation rather than direct participation: ‘they gave us the ideas and we were to visualise them in some kind of graphic form’.

Although not seeing themselves as designers, some participants did discuss generating ideas; for example, ‘we split up in rooms to come up with these ideas’ (Staff Member) and ‘… We had scale plans of the place and tried to move things around … and we made several suggestions about how the traffic could be better managed’ (Older Patient).

Experience-based design processes and outcomes

Participants discussed the value they perceived in story-sharing, emotional mapping and co-designing. Although they responded to questions about specific activities, their perspectives relate to EBD in general because they tended not to make clear distinctions between activities.

Most interviewees' experiences were positive: ‘It was good, it felt as though [the hospital] and the professional staff were really interested … There was a comradeship if that's the right word …’ (Older Patient) and ‘Because the patients are involved, normally it's just a man in a suit comes round … Whereas this time, they've actually asked us what might work … they're listening to us which is a first’ (Staff Member). A manager noted that ‘very often … a lot of time and effort talks about things that we all know are wrong, but … it's going straight to a pragmatic solution that's more useful’.

Although participants were positive about the activities, there was doubt about how effective the intervention had been as a whole. An Older Patient was ‘not sure if it has had any effect’. The staff agreed, ‘in some areas, the area has not been improved’, another noting ‘… we haven't had anything done about signage’ and another, ‘There's a few changes … [but] … there was some really serious things … they were highlighted, there was no resolution’. A staff member who chose to stop participating directly in workshops early in the project, preferring a more ad-hoc consultant role, provided the most negative assessment: ‘Not a lot has changed’; ‘as an exercise of getting people all around the table and chatting about experiences … it was very good [but] in the effect of change, not very good’; ‘I think a lot of it was waffle and people that didn't understand the process wanted to get involved from the University … I think that if they just left the patients and the staff to it we could have … made more of an effect, I don't know how much people were paid … but yeah for the output it seemed that too many people got involved, it was too long winded’.

Improvements to signage and road layout, including speed reduction measures, walkways, crossings, lighting, and street furniture to improve separation between pedestrians and vehicles, did occur subsequent to this evaluation exercise, and some 18–24 months after the project ended. In addition, outputs from the project were used to support a successful funding bid to redevelop the waiting area. However, these views raise issues about participants' expectations about change, our own initial expectations as designers, and the project's efforts to manage and communicate about these expectations.

An emotional map of participants' accounts

To provide a broad overview of the data, the interviewers constructed a form of emotional map of each participant's involvement using excerpts highlighted by the thematic analysis. The excerpts were grouped chronologically against the project timeline. Comments were given broad ratings of positive (green), negative (red) or neutral (grey). Although this rating does not differentiate the strength of comments so the resulting map only displays a quantity, it gives a visual summary of participants' feelings about each stage (Table 1).

Table 1 An emotional map of interviewees' participation in Better Outpatient Services for Older People (BOSOP).

In general, participants began with mostly negative attitudes and expectations, found the storytelling and emotional mapping positive, had mixed feelings about the design activities, and were more negative in their overall evaluation of the project and the outcomes. It is interesting to note that the middle and senior managers are entirely positive about the project outcomes, where other interviewees are more mixed or negative.

Reflecting on experience-based design and Better Outpatient Services for Older People

In discussing their experiences, interviewees also reflected more deeply on the project and what they perceived to be the issues in using such approaches for health service improvement.

Empathy, cohesion and conflict

Despite concerns about activities being ‘a bit daunting at first’, participants supported our perception that sharing experiences via storytelling and emotional mapping helped to build empathy and cohesion in the project group: ‘… You know there was … a readiness to share and meet each others’ difficulties'; ‘it helped [staff] to begin to understand and articulate how they felt about patients’ (Older Patient); ‘We have a good banter …’ (Staff Member).

An Older Patient described how initial tension was alleviated in the story-sharing: ‘staff were a bit on the defensive … When we began to speak about our experiences, then it began to resonate with them, and … people began to open up a bit’. The Advocate noted: ‘definitely people felt able to speak out …’ and empathised with others: ‘Sometimes the staff must have felt judged by [local charity's] thinking, “who are they coming along here criticising what we're doing” and “we're doing our best job we can”’. An Older Patient reported that staff soon realised ‘we weren't there to actually shoot them down … but we were genuinely interested in making our points of view’.

Recognising commonality of experiences contributed to group cohesion: ‘… really the two groups began to gel together because many of the experiences were virtually the same emotionally’ (Older Patient). Patients' familiarity with staff also affected group dynamics: ‘it's funny that the patients knew you, because they come to the desk … They recognise you, which is good’ (Receptionist). Although sharing experiences contributed to group cohesion, most participants' transcripts distinguish clearly between ‘the staff’ and ‘the patients’.

However, group cohesion did not grow in a simple linear way and occasional conflict remained. In the story-sharing one Older Person felt disrespected: ‘I felt that when I spoke about my experiences I thought [staff member] was a little bit aggressive as if she thought that couldn't happen …’. The Advocate reported that ‘one member of staff decided she didn't want to be involved any more so made it difficult for all the others …’. Relationships in the project cannot be discussed without also reviewing relationships with researchers. An Advocate was upset by the behaviour of one researcher: ‘[Researcher] was talking to one of the volunteers … and then made a face behind this volunteer as they were walking away … Blow it, you know it was horrible, and I know we all do things afterwards and I know he was a bit fed up at the time but this volunteer was doing her best’.

Assumptions about change

There was dissatisfaction with the effectiveness of the project, but analysis suggested that interviewees' expectations about changes reflected broader assumptions about what, how and when change can happen. It is impossible to tell from post-project interviews whether these views pre-date the project, but they clearly interact with interviewees' accounts.

There was a view that the ageing physical infrastructure and lack of financial resources would severely limit what could be implemented: ‘One of the biggest things … was the parking … I think we knew from the outset that there isn't very much that can be done’ (Advocate); ‘Obviously there are some things that they can't do and there's nothing we can do about it’ (Staff Member). ‘At the end of the day, we know, because we keep on being told, that there's no money…’ (Older Patient).

Responses also suggested recognition of the hospital's complexity, implying that change is a difficult and lengthy process: ‘[The researchers] hadn't realised some of the complex logistics of things’ (Staff Member); ‘… when I was confronted with the vastness of it, it was never going to be like that’ (Advocate); ‘… even the easy sorts still needed time. [Improvement is] not cut and dry, yeah, everything's not in black and white’ (Staff Member).

One response to the perceived difficulty of change-making was that expectations should therefore be modest and pragmatic: ‘… but it's not an ideal world is it, so you have to make do with what you've got and then enlighten it from that’ (Staff Member). However, viewpoints varied between moderating expectations and more pessimistic positions: ‘When I told people that we were doing this and I think most people said, “oh nothing's going to happen, nothing's going to change”’ (Staff Member).

A manager suggested that people's previous experiences colour their perceptions: ‘[Cynicism] was probably a result of people who have been around a long time, been involved in various initiatives in the past, perhaps not seeing a great deal of change … right at the beginning it was a bit of an “oh dear another new initiative” sort of thing’. Another staff member highlighted long-standing disagreements about the way the service is provided: ‘… issues were identified … like a receptionist on A floor, which is crucial to the whole operation. And I've been highlighting it for two years … and it's still not happened …’. Patients recognised these doubts: ‘I think there was that same kind of cynicism among the members of staff too. Because they kept saying, is it going to make any difference? Because there's no money [laughs]’ (Older Patient).

Institutional constraints and commitment

Participants' beliefs about what and how change can happen blurred with external constraints, as well as perceptions of institutional commitment.

The Graphic Designer describes working around the legacy of several signage systems, and attending to relevant NHS guidelines and regulations. An Older Patient had similar experiences: ‘Because we came up with several possibilities … but somebody said you can't use that space because …’. However, participants did not perceive all of the bounding constraints to be externally determined. Two participants described aspects of the service and hospital as being off-limits for the project: ‘[The Trust's project sponsor] came in and said, ‘oh well you can't do that bit of it’, so immediately [researcher] went up to the board and scrubbed that out … I made a very loud comment because I just thought what is the point?’ (Advocate). One Older Patient questioned the Trust's commitment: ‘There wasn't really enough positive belief that it should be altered’; ‘We kept getting feedback … that, yeah the Trusts were interested … [but] …. I think others too left with this question mark … about whether the whole project was taken seriously …’.

Costs, benefits and value for money

In an environment where resources are often very constrained, participants' perceptions of resource use within the project (time, people and money) were a key issue. Many participants raised the question of value versus cost: ‘… was it really worth all the time and effort?’ (Older Patient); ‘to take people out of their work environment … there's got to be something at the end of it’ (Staff Member). The Advocate questioned the costs of transport and refreshments at events. Another staff member questioned the extra value in the activities: ‘there's a lot of work involved [finding out the problems] and I think that sometimes you can just go and identify them anyway’.

The costs of a change project are not only financial. Staff involvement costs more than just paying for a temporary replacement: ‘Perhaps [they] saw it as … taking staff away from the front line … we had funding in order to backfill people … It isn't always … money, it's about having available people to backfill easily’ (Manager). The Advocate recognised the problem: ‘… I think from the staff … perspective they were finding it very hard to come’; ‘there was a Doctor … and he came … he was really keen but he couldn't come to any of the other meetings … because he was so busy’.

Project logistics

The logistical difficulties of the project were seen as problematic. ‘It always felt that there wasn't much time … problems about where could we meet … How we were going to get people there … The logistics of it did seem to be very difficult’ (Advocate). Several interviewees were concerned about a lack of continuity that they perceived as due to transport difficulties for patients, workload, and insufficient staff cover to enable attendance and holidays.

On the other hand, one Older Patient felt that the project: ‘Seemed to go on for a long time … and I don't think we were properly prepared for that … my initial impression was that it would be perhaps a couple of interviews, and a discussion’. Again, this raises the issue of clarifying both participants' and designers' expectations about the complexity of the task and likely timescales for impact.

Locus of control

Participants' descriptions of how they became involved in the project differed. The older patients felt invited to participate and were keen to get involved: ‘they asked me if I wanted to participate in the scheme and I thought, yes’. In contrast, front-line staff described being strongly encouraged by their superiors to participate: ‘I was just told by my line manager’; ‘It was just put through by the supervisors at work’. Only the middle manager suggested that she had some choice ‘[a manager asked] “do you want to take part?”’

Senior staff viewed themselves as outside the core team: one described involvement as part of her management responsibility, while another attended steering meetings ‘to be kept in the loop’, but not the EBD activities as ‘that wasn't for me to do’, apart from one session where ‘[I] was there as a sort of onlooker’.

Discussion

In evaluating EBD (or any other PD method) there are many aspects and dimensions that could be examined, from multiple perspectives; for example, the quality of designed outputs, the ‘gains’ for participants (Bossen, Dindler, and Iversen 2012), the levels of engagement of participants at different stages, changes in the outlook of people involved and transformations to the organisational setting (Sangiorgi 2011). In this project, following the focus of EBD on lived experience, we were interested in understanding participants' experiences and their perspectives on the value and limitations of EBD as a means of reviewing our own reflections.

In interpreting the findings, we should be aware of the limitations of the research design, which was developed after the main parts of the BOSOP project had been completed. The interviews prompted participants to reflect on the project outcomes as well as the processes, and their view of outcomes will have affected their perception of their experiences. In developing new techniques and methods, we are integrating participants' reflection on experiences and expectations into the main flow of the project (which also enables attention to be paid to their concerns during the project). However, even with this limitation, the BOSOP participants' responses provide valuable insight into the current state of EBD and into issues that are relevant to participatory health service design in general.

Prior to this evaluation, we felt that BOSOP showed that EBD's strengths were in building trust and rapport using a focus on lived experience via stories and emotional mapping, but that the modest service improvements that resulted may be due to the specific structuring of participation and the limited ideation tools in EBD. Having analysed our interviewees' reflections, we can now enrich our understanding of EBD and discuss issues that influence its effectiveness, which have general relevance to participatory health service design. We can also suggest some ways that they might be addressed.

Participants' positive accounts of the story-sharing and emotional mapping support our previous observation that EBD's techniques are useful for building trust and rapport (Bowen et al.2011; Wolstenholme et al.2010), and these activities provide additional value to approaches where lived experience is used as a design resource. However, this should not be over-idealised, and BOSOP involved initial tensions, disagreements and conflicts throughout the project among all the different stakeholders, including the researchers.

Most participants perceived themselves as giving input to a design process but did not see themselves as ‘doing’ the designing (despite our best efforts to encourage their active involvement), which was disappointing to us as participatory designers as we wanted them to feel that they were co-creators. This may be related to the fact that the patients were outsiders from the normal hospital decision-making processes, while front-line staff referred to an external locus of control in describing how they joined the project. The participants' responses are consistent with our previous discussion of the tension between the need to engage specialists and persuade people outside the project to take action, and how this can adversely affect participants' sense of ownership of the change process (Bowen et al.2011). This perception of the designing as being something that was done by others may be related to the lack of (participatory) ideation tools provided in EBD, which we have previously highlighted (Bowen et al.2011), but this must remain as a conjecture for now. The fact that much of the recording in design sessions was done by the researchers (it was difficult for our primary participants to move around to write on flip charts and stick up Post-it notes) may also have inadvertently contributed to this perception. To foster ownership of the change process and consequently increase the likelihood of transformation, we suggest that EBD may need enhancing with specific participatory ideation activities. We are placing strong emphasis on developing participatory creative activities in our current work (see Sustar et al.2013 for a report on a more recent project).

Participants' assumptions and expectations can affect their readiness to explore new ideas, and can affect their perceptions of the project. In BOSOP, it is difficult to attribute their pre-existence from post-project interviews. But such assumptions could set up forms of self-fulfilling prophecies that a project is unlikely to make significant change. As we saw in BOSOP, such assumptions are often informed by past experience (the repeated requests about the reception desk). There is also a blurred distinction between assumptions about the institution and its resources and concrete institutional constraints, which may be of limited significance (an obstacle is an obstacle, real or imagined). A more pragmatic tactic in such projects is to encourage participants to critically question all perceived constraints so that they do not limit their own ideas and actions.

The issue of cost versus benefit also had a significant impact on interviewees' opinion of project outcomes. Many participants felt that the project outcomes did not represent good value for the resources used (time and money). This highlights a key challenge to designers: making service change readily visible to those directly using or providing the service, and being careful to match resource inputs to the institutional context. The idea of securing (and publicising) early successes can be an important factor to maintain morale and engagement when dealing with more complex changes that take longer to deliver.

Our own expectations (as participatory designers) about trajectories of change can also be naive when working in unfamiliar and complex organisational contexts. The slow (and uneven) progress from ideas to implementation, and the way that project proposals have been adapted and fused with other inputs to stimulate the actual changes, challenged our own morale and confidence about the impact of the work. It is clearly helpful to establish realistic expectations from the start of a project, but the uncertainties of action in complex environments mean that the ‘realism’ of expectations is always hard to assess. It is telling that the Senior Manager had a more optimistic view of project outcomes. One principal output was a report on the project for the Trust, which was distributed on CD-ROM along with the supporting design proposals. Although this report was made available and distributed at the final celebration event, for many participants the format was less meaningful to them (nurses spend little of their time at desks or computers). A report is more amenable to the Senior Manager's work practices and her experience of management processes may give her greater confidence in the report being used to promote change. The progress of proposals and institution-wide changes therefore needs to be communicated via more accessible means, to ensure that front-line staff are aware of initiatives that produce less immediate tangible change.

There remains a challenge to deliver more change in the timescales that participants might reasonably expect. Improvements need someone within the service to actively drive the implementation of ideas. As design researchers working as facilitators in BOSOP we could push ideas, but implementing sustainable change needs managers and staff to choose to pull those ideas forward. Much has been written about the distribution of decision-making power in participatory projects (Arnstein 1969; Floyd et al.1989; Kanstrup and Christiansen 2005; Dearden and Rizvi 2008) and it might be argued that part of the reason that the immediate outcomes were disappointing for participants was that patients had insufficient decision-making power in the project. ‘Patient and Public Involvement’ in healthcare (Department of Health 2004) has received considerable attention in recent years. In a complex institution such as the NHS, there is a wide variety of mechanisms for public accountability ranging from the national government, through formal representation of patients at various levels in hospital governance, to individual patients' decision making about their own care. Various authors have highlighted the limitations of Arnstein's one-dimensional metaphor of a ladder of participation (Tritter and McCallum 2006; Morrison and Dearden 2013). Gärtner and Wagner (1996) and DePaula (2004) discuss the need for projects to engage with different ‘arenas of participation’.

Within BOSOP, patients were represented on the project steering committee by both the charity director and a representative from the hospital's patients' forum, while the steering group also included senior managers, who might be regarded as very powerful actors able to effect changes. However, a notable feature of BOSOP was that many design suggestions implied working with other stakeholders beyond the scope of the outpatient department (e.g. the estates manager) and were sometimes constrained by national policies (e.g. the NHS has extensive national guidelines and standards with regard to signage in hospitals).

Even within the local arena, the staff who deliver the outpatient service do not all belong to a common management hierarchy (e.g. the receptionists, ambulance staff, nurses and doctors all have separate lines of accountability), and implementing local changes depended not only on decisions of the staff involved in the project, but also on their colleagues and middle managers.

The literature presenting EBD provides little guidance about power relations or the challenges of intervening in multiple arenas. EBD is intended as a tool that can be used by NHS managers (without requiring the assistance of professional designers), and many of the case studies of EBD (see http://www.institute.nhs.uk/quality_and_value/experienced_based_design/case_studies.html) have been initiated and led by local managers of particular services. Such arrangements give rise to particular configurations of power both in relation to the performance of co-design and in the implementation of changes. The configurations of power within and surrounding a project, and the commitments of specific actors, will have a significant influence on project outcomes. The effectiveness of any methodology, tool or technique will always depend upon the people applying it and the enactment of power in context. Key challenges for health service (re-)design remain: ensuring that participants have strong ownership of change processes, ensuring that key decision makers (including both senior and middle managers) are fully engaged, and developing stronger institutional cultures of participation.

Conclusion

Effective design of healthcare services relies on a collaboration among front-line staff, service users, designers, decision makers and managers. Analysis of stakeholders' accounts of their participation in our BOSOP project has enabled us to develop a richer understanding of the strengths and weaknesses of EBD as a PD methodology within the context for which it was developed. It has supported our initial view that EBD is effective in building collaborations between service users and service providers and identifying areas for improvement by focusing on lived experience. And it has reinforced our view that the ideation and implementation phases would benefit from further support.

Our analysis also indicates some issues affecting EBD that have relevance to other participatory approaches, and we suggest tactics for dealing with them to promote transformative design. We highlight stakeholders' ownership, ongoing engagement, and the active pulling of service improvement: inclusion in ideation, maintaining momentum, setting expectations, encouraging a critical attitude to (perceived or actual) constraints and communicating change accessibly.

In highlighting loci of control, pre-existing assumptions, perceived cost versus benefit and momentum, the evaluation enriches our understanding of how project outcomes and effectiveness are affected by the nature of participation. This evaluation, however, does not validate our claim that poor outcomes reflect the lack of ideation tools provided within EBD, only that participants tended not to see themselves as ‘doing’ the designing, and that there are challenges in managing expectations.

In a subsequent case study within our wider research programme we are exploring these factors in more detail (Sustar et al.2013). By engaging in more creative activities and increasing external input to participatory workshops, we are developing a point of comparison to the ideation methods in EBD and BOSOP. There are many techniques to support collaborative ideation in the co-design and participatory design literature that could be incorporated. Our revised approach also places a stronger emphasis on proactively engaging key decision makers. Our findings from these and further case studies will then help us to develop a methodology for participatory, human-centred health service design.

Acknowledgements

With thanks to all those who participated in BOSOP, in particular those who agreed to be interviewed for this evaluation.

This article presents independent research by the Collaborations for Leadership in Applied Health Research and Care for South Yorkshire (CLAHRC SY). CLAHRC SY acknowledges funding from the National Institute for Health Research (NIHR). The views and opinions expressed are those of the authors, and not necessarily those of the NHS, the NIHR or the Department of Health.

CLAHRC SY would also like to acknowledge the participation and resources of our partner organisations. Further details can be found at www.clahrc-sy.nihr.ac.uk.

Ethical approval for this work was obtained from the ethics committee of Sheffield Hallam University Faculty of ACES. All participants consented to their involvement in the BOSOP project and to the recording and transcription of the subsequent interviews and the use of extracts in publications.

Additional information

Funding

Funding

The BOSOP service improvement project was funded by Sheffield NHS Primary Care Trust and our research was funded by the UK's National Institute for Health Research (NIHR) as part of User-Centred Healthcare Design in the CLAHRC for South Yorkshire.