Participatory theme elicitation: open card sorting for user led qualitative data analysis

ABSTRACT

Participatory Theme Elicitation (PTE) is a new approach to increasing user-involvement in qualitative data analysis. In this five-step method, participants independently identify themes in the data using open card sorting, and are supported in their development of a set of themes that reflect all participant views. We describe and evaluate PTE with a case study, where nine lay researchers identified five core themes in qualitative data from an online intervention with depressed adults. Total participation time was approximately 6 hours. Evaluation data revealed high levels of satisfaction with the PTE process and the themes generated. The strong commitment to public engagement in social science research and limited guidance on methods for effective participatory analysis of qualitative data highlight the value and growing need for robust standardised approaches such as PTE.

KEYWORDS:

• Participatory data analysis
• public-patient involvement
• co-production
• user involvement
• participatory theme elicitation
• card sorting

Introduction

User involvement and collaboration, particularly within social science research, have been building momentum for a number of years. It is now a widely accepted pre-requisite for research funding, and effective involvement of users is recognised as a critical issue for research topics as diverse as health services, child protection and environmental management (Garfield et al., 2016; Ives, Damery, Redwod et al. 2012; Cornwall & Jewkes, 1995; Van Bijleveld et al., 2015). Involving users can generate new knowledge and insight which enriches the quality of research findings (Gamble et al., 2014; Hawkins et al., 2017) Moreover, the rise of ‘co-production’ as a recognised approach has led to more workable solutions to crucial and complex social issues (Hawkins et al., 2017; Osborne, Radnor and Strokosch; Osborne et al., 2016; Boyle & Harris, 2009). Multiple guidelines have been developed (Involve, 2018; Sten. et al., 2006).

In practice, engagement is more common during planning and design (Concannon et al., 2014) despite emerging research suggesting that involving lay partners within data analysis can be equally beneficial as for other research steps (Garfield et al., 2016). Some argue the reasons for this are mainly practical due to the time and financial implications associated with participatory analysis methods (Campbell & Vanderhoven, 2016). In addition, there may also be a reluctance on behalf of researchers to share power at such a crucial stage of the process (Julkunen & Ruch, 2016).

Common approaches to participatory analysis of qualitative data include some form of team-based coding activity (Flicker & Nixon, 2014; Jackson, 2008; Stevenson & Taylor, 2017; Sweeney et al., 2013). In general, lay researchers are trained in the coding process and teams are structured so that everyone has an equal voice regarding analysis and interpretation (Flicker & Nixon, 2014; referred to as ‘multiple coding’). Other approaches (e.g., Stevenson & Taylor, 2017) do not formally code the data, but instead conduct various activities with users that are designed to organise and interpret qualitative data, such as role plays, handouts and posing specific questions. Supported by careful facilitation, these approaches have enabled lay researchers to contribute differing interpretations that have led to more nuanced analysis. While this demonstrates the value of including lay researchers in qualitative analysis, the barriers surrounding time, cost and the practicalities of training lay individuals remain. In addition, adopting the methods of the academic researchers may automatically privilege the academic perspective within teams comprised of both experienced and newly trained coders.

This paper describes an innovative approach to participatory qualitative data analyses known as Participatory Theme Elicitation (PTE). This approach has five key steps and has been developed to overcome the barriers associated with team-based coding, but still access the different perspectives of lay researchers. PTE uses open card sorting, an established method to collaboratively organise information. In an open sort, the items to be organised are each listed on separate cards. Several sorters each place cards with similar items into piles (see PTE step 3 below for more detail). The piles from each sorter are combined into an overall set of piles (see PTE step 4). While card sorting is most commonly used in information architecture to, for example, organise software menus into categories that are meaningful for intended users (Righi et al., 2013), there is little restriction on the type of information or items that can be organised. Of most relevance to qualitative data analysis, open sorts are used to investigate conceptual similarity in psychology (Miller, 1969) and as one element of cultural domain analysis in anthropology (Bernard et al 2016, chapter 18).

PTE extends the idea of open sorting in two ways. The first is to use network methods to combine the individual sorts rather than the standard approaches of hierarchical clustering or multi-dimensional scaling with spatial clustering. The second is to embed the sorting in a participatory process so that the lay researchers can modify the themes that emerge from the sort and then guide the interpretation of the agreed themes (see full process, but particularly step 5).

Previous work (Best et al., 2017, 2019; Yap et al., 2020) has demonstrated proof-of-concept, that PTE is an accessible and practical method that can be used with lay researchers to organise qualitative data into themes that may differ from those identified by academic-led analysis. This study develops and evaluates the participatory process of PTE. We use a case study to ground the description, working with a panel of nine lay researchers analysing qualitative data taken from a feasibility randomised controlled trial for adults with depression. The full qualitative analysis, including the PTE outcomes, is described elsewhere (Best et al., 2019). The objectives of this study are to:

1. describe the method of PTE in sufficient detail that it can be adopted by other researchers;

2. assess whether the PTE method supports the generation of user-developed themes; and

3. evaluate the acceptability of the PTE method among lay researchers (extending the previous proof-of-concept study that focused on acceptability from the perspective of the academic researchers).

This study does not compare network methods to the existing approaches for combining sorts. Such a comparison requires multiple cases and is therefore deferred until further studies have used PTE. However, the accessibility and acceptability of network methods is examined as part of objective 3. The case study presented uses qualitative data obtained in health research, but the method is independent of the data content and is expected to be useful in any broadly defined social science with an interest in user participation.

Context: The DES Study

Qualitative data used in this project were collected as part of the DES study (Trial Number: NCT03288506). The study was an 8-week online peer-led support group for depression to test the feasibility of video-conferencing software (Skype) to deliver a group-based intervention for depression. Participants were recruited via online registration and the service was promoted through local radio, newspaper and social media.

The data analysed by the lay researchers in this PTE study were gathered at the end of the intervention (Week 8). Two online groups completed the 8-week intervention with semi-structured interviews conducted with five participants and three group facilitators across both groups (n = 8). Interviews were conducted either face-to-face or online (via Skype) and lasted between 20 mins – 45 mins. Audio recordings were taken and these were transcribed verbatim and anonymised.

Methods

The methods described in this section relate to the PTE elements only. Ethical approval for the current study was approved by the University Ethics Committee. PTE is designed to fit within conventional research designs, with little modification needed to support the supplementary components of the approach. The qualitative analysis of the intervention is presented elsewhere (Best et al., 2019).

Recruitment

A convenience sample approach (Bryman, 2004) was used to recruit lay researchers to undertake the PTE method. This was achieved through contacts at two local mental health charities. Information packs and consent forms were provided to interested individuals. Inclusion criteria for lay researchers were having previous experience of using services for depression and no prior knowledge of the intervention. The academic research team liaised closely with staff in those charity organisations to ensure that potential lay researchers were not actively suicidal and their mental health had been relatively stable for a period of six months or more. Nine lay researchers signed up to take part (n = 9) (see Appendix A). Ages ranged from 20 to 63 years old, with three males and six females. All participants completed secondary level education with three having University degrees.

The PTE process has five key steps (explained below) with lay researchers involved three of the steps, conducted in two workshops. All nine lay researchers participated in the first workshop, and six returned for the second workshop (see Appendix A).

Participatory theme elicitation

PTE implementation involves five key steps delivered in the following sequence: (1) data selection; (2) capacity building; (3) open sorting; (4) data grouping and (5) analysis and interpretation.

Step one: data selection

Recordings from the eight interviews were transcribed verbatim. One member of the academic research team [TMC] initially reviewed each transcript and selected excerpts that could be easily understood as standalone statements. The mechanics of this approach are similar to Lincoln and Guba (1985) ‘cutting and sorting’ method, but no formal coding of data took place during this activity and no conscious effort was made to group data together or select excerpts based on potential significance. The only terms of reference were that the excerpts were broadly representative of the larger dataset. This produced an initial list of 170 anonymised excerpts. This list was then given to a lay project advisor [CC] with lived experience of depression and, together with TMC, they reviewed and narrowed this list to a final set of 49 excerpts (see Appendix B for full list of excerpts, with reference codes ID01 – ID49). The decision to limit the number of excerpts was based on several factors, including previous user feedback regarding PTE sorting and the potential for participant fatigue.

Step two: capacity building

The academic research team developed two, 60 minute training sessions including (1) an overview of the online intervention and research project; (2) a brief overview of qualitative and quantitative research; (3) an overview of PTE and user involvement within research and (4) sorting techniques used to group data together. These sessions were delivered back-to-back with a short break. Lay researchers were not all available on the same day and therefore this session was conducted on three occasions (two group-based sessions and one individual session).

Training sessions were designed to be interactive and included short videos and other media. Images as well as excerpts were used to demonstrate possible sorting techniques for the analysis of qualitative data. For example, lay researchers were provided with images of different products and asked to sort these into groups by similarity; some categorised by colour, and others by food groups such as fruit or vegetable. After completion of sample exercises, discussion took place regarding the subjective nature of qualitative research and analysis.

It is also important to note the differences between capacity building activities within PTE and formal research methods training. The latter has been associated with what Ives et al. (2013) refer to as the ‘professionalization paradox’ in mind. This is a process whereby the lay researcher achieves some level of professional socialisation through in-depth training. As such, their position as a lay researcher is undermined. In contrast, for PTE, the training is limited, explicitly focusing on the validity of different approaches to sorting to promote confidence in the sorting task (step 3) and participation in the discussion (step 5).

Step three: open sorting

In the third step, lay researchers independently sort the excerpts into piles (see Figure 1). This step is partially informed by other qualitative open sorting and grouping methods (see Jackson, 2008; Lincoln & Guba, 1985; Rosenberg & Park, 1975; Sayles et al., 2007). The 49 excerpts chosen as part of the data selection process were provided in information packs given to lay researchers when they arrived. The packs also contained a project information sheet, consent form, an evaluation form and blank sheets of paper to create pile labels. Instructions were to sort the excerpts into piles, based on similarity, using whatever criteria they found relevant (Rosenberg & Park, 1975) with a minimum of two piles. Participants were further instructed that excerpts that cannot be grouped (placed) with others are to be placed in their own pile rather than create a ‘miscellaneous’ pile. No further instructions were given as researchers were cautious that over instruction may limit or constrain analysis (Garfield et al., 2016). Lay researchers sorted independently, with no influence on each other’s sorting. Members of the academic research team were available to clarify any questions regarding excerpts but did not offer opinions or interpretation. A PowerPoint slide containing task instructions was visible throughout. This task was completed by the lay researchers in the same workshop as their capacity building.

Figure 1. Network diagram: The network of excerpts (shaded circles), with the thickness of the edge indicating the number of so-researchers who sorted the pair of excerpts into the same pile. The colour indicates groups identified in Step 3: Group 1 (light blue); Group 2 (green); Group 3 (pink); Group 4 (red); Group 5 (yellow) and Group 6 (orange).

After sorting was complete, each lay researcher had between 3 and 12 piles of excerpts. These piles were labelled by the lay researcher who conducted the particular sort and stapled together (see Appendix C).

Step four: data grouping

The objective of the data grouping step is to construct a set of piles that, to the extent possible, reflects the consensus between the separate sorts in step 3. Piles from each member were recorded and uploaded into a Microsoft Excel spreadsheet by PB who was not involved in initial data selection. The spreadsheet columns are (1) anonymised lay researcher; (2) the excerpt ID reference code and (3) the pile (an example is provided in Appendix D).

A similarity score was calculated for each pair of excerpts that simply counts the lay researchers (out of 9) who had placed the pair of excerpts into the same pile. A network was constructed with nodes to represent excerpts and edges for any non-zero similarity score (that is, at least one lay researcher placed the two excerpts in the same pile). The similarity score was converted into a distance weight by taking its inverse.

In network science, community detection algorithms are established methods to identify clusters within the network where there is a relatively high edge density, with relatively few edges between two clusters (Fortunato, 2010). For the purposes of PTE, such clusters indicate agreement between lay researchers that some group of excerpts are sufficiently similar to be placed in the same pile and different from excerpts placed in other piles. For this application, we used algorithms that are able to take into account the distance weight for each edge (referred to as weighted community detection in the network science literature) so that the excerpts that are placed together by more lay researchers are more likely to be placed together in the proposed groups. In addition, the excerpt that is ‘most close’ to all others was identified for each group. This is the excerpt with the shortest average (weighted) path to all other excerpts and is a common measure of centrality. All network analysis was conducted using the igraph package in R (Csardi & Nepusz., 2006; R Core Team, 2015).

Initially, the Louvain algorithm (Blondel et al., 2008) identified four communities. However, one of the communities was relatively large and the edge betweenness approach (Girvan & Newman, 2002) was used to break it into three smaller groups. This resulted in six groups, displayed at displayed at figure 2. The number of excerpts contained with each group ranged from 4-13 (see Appendix E).

Step five: data analysis and interpretation

Two weeks after data sorting was complete, six (of the original nine) lay researchers returned to undertake the final part of the PTE process, development of themes in the qualitative data excerpts. The three remaining lay researchers could not attend for the following reasons – family-caring responsibilities (n = 1), illness (n = 1) and work commitments (n = 1). On arrival, each lay researcher was provided with a copy of the network diagram, an information pack containing the six network groupings, a pen and a notepad. After describing the network information, academic research team members read out all of the excerpts contained within each group and asked if any further clarity was needed on the analysis task. Flipchart paper was also provided and the session was audio recorded.

The network information comprised:

• A proposed set of six core groupings based on the combined results from lay researcher pile sorting during Step Three;

• Identification of the central excerpt within each grouping; and

• Visual representation of relationships between excerpts (see figure 1)

As PTE is designed as a user-based approach to qualitative data analyses, numerical data regarding exact distances excerpts and other detailed information were not provided. However, lay researchers were advised that this information could be interpreted (visually) from the network diagram itself e.g., strength of relationship between excerpts could be extrapolated through their proximity to each other and the thickness of lines between excerpts. The strength of groups could be interpreted through compactness of the excerpts within a group and separation from excerpts in other groups. Themes were described to lay researchers as something ‘important about the data in relation to the research question, and represents some level of patterned responses or meaning’ (Braun & Clarke, 2006, p. 82).

Lay researchers proceeded to discuss relationships and possible themes within each individual grouping using the network diagram and groups as a starting point. Academic research team members recorded key points, codes and developing themes on flipchart paper as well as any other information lay researchers felt were relevant, but did not provide opinions or interpretation. Lay researchers reviewed and discussed each group separately in numerical order (which is arbitrary). Once this was completed, they debated possible outliers and refined themes further. No time limit was set for data analysis and interpretation, which was completed over a two-hour period. This was followed by a final 15 minute group interview discussing the acceptability/value of this approach.

Evaluation of participatory theme elicitation

For the evaluation of PTE, semi-structured group interviews (10–15 minutes) were conducted at two separate time points during the PTE method (post Step 3 and Step 5). As the capacity building and data sorting sessions were run on three separate occasions, there were three separate post Step 3 interviews. During these interviews, feedback was sought on the sorting task and training provided e.g., (1) general understanding of sorting task; (2) suitability of excerpts; (3) number of excerpts and; (4) content of training. The final group interview (n = 1) took place immediately after data analysis (Step 5) and included all six lay researchers who had participated in the data analysis step. Feedback taken here related to the data analysis process (length, acceptability, usefulness of network diagram and groupings) as well as the PTE method in its entirety.

A short 10-item questionnaire was also administered to supplement interview data following Step 3. The questionnaire sought specific information regarding the training materials, sorting task and background of lay researchers and included questions relating to – (1) duration of training, (2) length of excerpts, (3) number of excerpts, (4) understanding of sorting task and (5) information provided (see Appendix F).

Results

User Generated Themes

Lay researchers produced five core themes about the depression intervention using the PTE method. These included; (1) challenges of using the online service; (2) benefits of accessing an online service; (3) role of technology in facilitating help-seeking; (4) therapeutic processes and outcomes within an online context and; (5) service improvements. As a result, thematic relationships were identified within five of the six groups produced via network analysis techniques during Step Four. Appendix G has a brief summary of the excerpts used to develop themes within each grouping.

Lay researchers developed multiple themes and sub-themes within Groups 1, 2, 3, 4 and 6 however, no thematic relationship was identified within Group 5. Instead, Group 5 excerpts (if relevant) were used to support themes in other groups. Lay researchers also felt that some of the excerpts contained within Group 3 (ID04, ID11 and ID30) were equally relevant to themes developed within Group 2. As such, while the final groups largely reflect the proposed groupings (produced at Step Four), some changes were made by lay researchers. This indicates that lay researchers felt comfortable using PTE to support rather than dictate the analysis process. Full discussion of the thematic content generated during this task and its contribution to knowledge of online services for depression is outside the remit of this article and is presented elsewhere (Best et al., 2019). However, the following example, taken from Group 1, highlights the processes through which lay researchers developed themes using PTE.

The theme development (Group 1 example)

Lay researchers began by reviewing the network diagram (light blue dots) as academic research team members read out Group 1 excerpts (Table 1). Network data revealed ID03 was identified as the most central. Each lay researcher spent a few minutes working independently identifying patterns and generating ideas based on the excerpts contained within Group 1 and cross-referencing this with the network diagram. Group discussion then took place organically with little prompting from academic research team members.

Table 1. List of group 1 excerpts

Number	Excerpt Text	Central
Group 1
ID01	I think, when it started, there were one or two just technical things that … weren’t a big deal but, in some way, kind of did interrupt the flow of the conversation. So, that was quite distracting.
ID03	the technology was just a wee bit of an issue, but … I think resolved itself because when [we] moved into an office in our building where we have a good system	X
ID08	So, I think it’s less easy on a videoconferencing call to actually pick up any non-verbals of the other people who are not talking.
ID20	In terms of attendances and stuff, it was quite low, so … that sort of hindered me getting the full experience from what an online support group could be.
ID21	I really found that extremely difficult … in terms of syncing the voice when we were talking … trying to be sensitive, pick up on tones and moods … and even looking at the people and their body language
ID34	I thought it would have been nice to maybe have a few more men that I would be able to relate to, because people like myself, obviously, we question ourselves … I think we suffer in silence, that’s the problem.
ID46	The only like bad thing was some people … you would talk over each other sometimes, but that even happens face-to-face, but that’s about it. It was grand otherwise.

It was noted early on that ID20 and ID34 were (visually) the closest together within the group and that they both appeared to relate to group attendance. This gave lay researchers early confidence in the usefulness of the network diagram. A single, over-arching theme (challenges of using the online service) was generated fairly quickly and confirmed through the central excerpt (ID03) which stated ‘the technology was just a wee bit of an issue, but … I think resolved itself because when [we] moved into an office in our building where we have a good system’. Lay researchers used all three pieces of network data available (i.e. grouping, diagram and central excerpt) to develop this theme. Later on, lay researchers added an additional excerpt under this theme (ID10) that was contained in another grouping (Group 6). This was justified mostly through the content of the excerpt itself e.g., ‘I just think that the technology got in the way a little bit because, if you’d be talking to someone, there’d be a little bit of feedback … when you’re not there to pick up on non-verbal cues as easily, it can be a little bit challenging’ (ID10). However, its inclusion was also supported through the network diagram which showed a direct relationship (edge) between ID10 and ID21, which was located in Group 1. The thickness of the line between excerpts was also noted and discussed by lay researchers as further justification for its inclusion.

It is important to note that while PTE is designed to offer a hybrid approach to the user analysis of qualitative data, the extent to which the network data available is used to inform theme generation is at the discretion of the lay researchers. In most cases, lay researchers relied heavily on this for guidance but as confidence grew this was not always the case. For example, later in the session, lay researchers returned to Group 1 and developed three specific sub-themes – (a) technical issues of physically using the service (b) barriers to communication (in-group) and; (c) increased sense of self-awareness. The latter sub-theme (increased sense of self-awareness), was revisited on several occasions and caused much discussion among the group. The debate mainly centred upon whether this sub-theme could be discredited or confirmed using the network diagram and whether, as lay researchers, they should interpret the excerpts based on surface-level meaning (explicit) or whether they should ‘read between the lines’ (interpretative). Members of the academic research team did not offer any opinion on this issue apart from acknowledging that the group were free to interpret the data in whatever way they saw fit. As such, this provides an interesting, real life, example of theory generation supported by, but not dictated by, the PTE method. This sub-theme was developed from three main excerpts – ID21, ID34, ID46.

Initially, the group decided to code this as ‘group composition’ but suggested that they return to it later in the session. When this occurred, one lay researcher discussed how the cluster of excerpts within this sub-theme reminded him of the individuality and sense of ‘presence’ that one must feel when taking part in an online group (particularly given the geographical distance between users). This observation generated more discussion, whereby lay researchers reflected upon their own experiences of group therapy and related it to the qualitative data. There was a sense that one could feel very much ‘under the spotlight’ when it was their turn to speak as they knew that others would have to watch intently to pick up non-verbal cues (ID21). It was postulated that it might also cause some group users to look for relatable features in others in order to feel comfort (e.g., males – ID34).

Interestingly, rather than theorising the wider implications in relation to the literature (e.g., social presence theory etc.), (Short, Short et al., 1976) lay researchers reflected upon their own experiences of group therapy and thus provided a level of ‘insider’ knowledge that would have been otherwise absent. There was some acknowledgement that additional data were needed to fully support this finding – particularly as the network diagram placed these excerpts furthest apart from each other (suggesting a weaker relationship). It was also noted that the links between ID46 and the others appeared to be moderated through ID03 and therefore the network diagram could not fully support this conclusion. Nonetheless, they were keen for its inclusion within the analysis and this is perhaps indicative of the confidence generated from taking part in earlier PTE steps (capacity building and sorting).

Evaluation of the PTE method

Feedback on the PTE method was positive with interview data suggesting that as lay researchers progressed through the various steps, their interest in the DES study increased, for example, ‘I now understand more about what you guys are doing … .I’m now even more interested in this’ [lay researcher 3]. One lay researcher also stated, “I found it very intense but was surprised how engaged I remained throughout [lay researcher 5]. A sense of ownership also began to emerge as the lay researchers saw value in what they were doing – ‘I was looking forward to coming back and seeing how it all came together’ [lay researcher 2]. It was noted that by presenting excerpts in smaller groups it felt less ‘daunting’ as it ‘was easier to review 6 or 7 quotes together than try to keep all 49 in your head’ [lay researcher 5]

Questionnaire data revealed that lay researchers felt the training (Step 2) was pitched at an appropriate level (Appendix B). Six lay researchers rated the information and training provided as ‘excellent’ with three rating it as ‘very good’. The number of excerpts presented during the sorting task was also rated as ‘just right’ with qualitative data revealing that lay researchers were cautious regarding increasing this number further. In terms of improvements, it was suggested that some additional examples of the sorting task using ‘test data’ would be beneficial. There was also a sense that the sorting task itself was a key part of the training and this increased confidence in the quality of data analysis during Step 5. For example, one lay researcher stated, ‘it would have been very difficult to do this bit [analysis] if we hadn’t had done the sorting’ [lay researcher 3]. It was also suggested by some that the entire PTE process could be performed within a single day. The rationale was that the data would be ‘fresher’ in their minds.

The network diagram and groupings given to lay researchers at the beginning of Step Five provided a useful starting point for user-led analysis. Moreover, it prevented a single individual from monopolising the group or imposing their views on others. One lay researcher commented that ‘if we hadn’t of done that I probably would have just sat and agreed’ [laughing] [lay researcher 4]. Interestingly, lay researchers felt that the group discussion greatly improved the quality of analysis. One lay researcher noted that ‘the second part (discussion) has helped to make the final themes better’ [lay researcher 2]. This was supported by others e.g., ‘sub-themes are more defined than when I completed the sorting task’ [lay researcher 7]. Reflecting upon the added value of group analysis, one lay researcher noted how his views and ultimately the themes changed and evolved as time went on,

“it’s really interesting when you’re getting other people’s perspectives, rather than just sitting in your own wee bubble … when you hear someone’s perspective it can make the quote speak to you in a different way than you hadn’t thought of” [lay researcher 3].

Views on the facilitation provided by members of the academic research team were also positive. Lay researchers felt that it helped keep the session ‘on track’ and provided independent moderation of discussion. There was no (reported) sense that academic research team members influenced the generation of themes – “you need a bit of leadership otherwise it would have taken much longer … but I feel like we came up with the themes ourselves [lay researcher 3].

Discussion

Objective 1: describe the method of PTE in sufficient detail that it can be adopted by other researchers

The authors have described all five steps contained within the PTE method. A case example of qualitative data analysed using PTE has been presented as well as feedback taken from lay researchers involved throughout the process. No concerns were expressed by lay researchers or by members of the academic research team regarding the implementation of open sorting or network analysis within a wider participatory process. Appendix H includes further details of the PTE protocol.

Most of the steps are straightforward. However, the grouping (step 4) requires specialist skills in network analysis. In order to make the method accessible to any researcher, the authors have developed an online platform to easily produce the network diagram (available at www.ptegroups.net/main). Researchers with no experience of network analyses can upload a spreadsheet in the appropriate format, and the website will construct the weighted network and apply the selected community detection algorithm. The network diagram and groups can then be downloaded.

Objective 2: to assess whether the pte method supports the generation of user-developed themes

The example given above demonstrates the process through which lay researchers, using the PTE method, were supported to develop their own themes. Broadly speaking, the thematic content developed using PTE was similar to previous and current literature in the field (Griffiths et al., 2012; Mallow et al., 2016), supporting the validity of this approach. Moreover, there were some interesting observations and unique insights confirming the ‘added value’ of user involvement in data analysis. Core themes suggested that participants who took part in the online intervention found it largely beneficial. This was mainly due to ease of access, quality of facilitation and a reduced sense of anxiety from being able to link in from a familiar environment. Themes were developed from five of the six groups presented to lay researchers suggesting that network analysis techniques applied during Step 4 produced relevant groups.

At the beginning, lay researchers relied heavily on the network groupings in order to develop thematic content. It was noted that having a smaller number of excerpts narrowed the focus of discussion and prevented lay researchers from feeling overwhelmed. The network data provided lay researchers with a framework with which to generate discussion based on observable data. The diagram offered further opportunity to debate possible relationships and discuss the strength of connections between excerpts (e.g., distance apart or thickness of adjoining line). Lay researchers were informed at the beginning of Step 5 that the diagram represents the best ‘mathematical fit’ following the sorting task and that there was no expectation that they form themes from within these groups. That being said, the network analyses did, in fact, produce some common thematic groupings (e.g., themes developed within five of the six groups and only a small number of excerpts were re-allocated). Together with the six network groupings, the information provided enabled members of the academic research team to take a ‘back seat’ as lay researchers had sufficient information in which to generate discussion without researcher prompting.

It is also important to note that the network diagram delivered a mixed picture whereby groups appeared scattered and dispersed – perhaps indicating level of disagreement among lay researchers during sorting. Nonetheless, lay researchers were still able to use the visual information provided to confirm/challenge potential relationships within groups (e.g., sub-themes). In the example given above, lay researchers ultimately chose to rely more on their own experiences to interpret the meaning behind the generated theme (increased sense of self-awareness); however, the network diagram was a focal point of this discussion wherein they debated the validity of this theme (e.g., no direct connection within the group – ID21 to ID46). As time progressed and confidence grew, lay researchers used the network data to confirm their decisions regarding a possible theme rather than dictate it. On previous occasions (Best et al., 2017) the network diagram has produced more defined groups; however, the extent to which this has influenced the development (and quality) of generated themes is unclear.

A key goal of PTE is to provide a ‘robust, accessible and reproducible approach to user involvement in data analyses’ (Best et al., 2017, p. 18). However, it is also important that PTE provides sufficient room for flexibility so as not to directly influence outcomes. Lay researchers in this study adopted a variety of approaches in order to make sense of the data. The most interesting of which occurred when they used excerpts across the different groups to support themes. This meant that generated themes had more ‘evidence’ in which to support their existence. Often, initial ideas were developed and expanded through group discussion. This provided a useful layer of scrutiny for each new theme.

While PTE is informed by the literature on qualitative data analysis (Saldana, 2016), it does not seek to replicate complex methodological procedures and apply them to end-user populations. Rather the goal is to provide a specific framework for data analysis (Morse, 2015) in which themes can be developed in a manner accessible to those with little or no formal training. However, key stages and techniques are still identifiable with the PTE method when one examines the process more closely. For example, capacity building during Step 2 provided the context for coding and theme generation during Steps 3 and 5 by giving a detailed overview of the research study. In addition, descriptive coding (Saldana, 2016) is evident during Step 3 when lay researchers are encouraged to identify ‘similarities and differences in the data’ (Castleberry & Nolen, 2018: 808) and assign labels to excerpts as a process of organisation (see figure 1). Finally, Step 5 provided an opportunity for the data (e.g., groups of codes) to be thematically organised and contextualised through shared discussion and debate among those with lived experienced. While PTE simplifies the analysis process for end-user populations, the extent to which the loss of complexity is compensated by the insider knowledge of lay researchers is unknown and subject to further study. It may well be that PTE is more suitable for particular types of studies, such as intervention-based process evaluations where the focus is geared toward information that can be garnered at a semantic, rather than latent level of coding (Braun & Clarke, 2006).

Objective 3: to evaluate the acceptability of the pte method among lay researchers

Findings suggest that PTE is both an acceptable and credible approach to user-led participatory analysis. Lay researchers did not report feeling overburdened during the process and there is some evidence to suggest that motivation actually increased as time went on. The network diagram and initial groupings provided a useful starting point that increased focus and reduced reliance on researcher input or facilitation. Moreover, it lessened the impact of a smaller number of individuals imposing their views on others.

There is also some preliminary evidence to suggest that PTE facilitated lay researchers to go beyond semantic level descriptions by reflecting upon their own experiences. This is an important point as the specificity behind this, often touted, benefit of user participation is seldom demonstrated within the wider literature. Of course, this also introduces a risk that the themes reflect lay researcher experience rather than arising from an analysis of the data. Evaluating acceptability of the PTE method also highlighted the following potential outcomes -

1. Reduced researcher influence over the participatory analysis process

2. A good starting point for individuals with no training or background in qualitative data analyses

3. Provides support for user-led theme generation

4. An accessible and time-limited alternative to other participatory analyses methods

Overall, feedback suggests the five-step PTE approach was acceptable for lay researchers. The number of excerpts used and length of time set aside for capacity building, sorting and analysis was also found to be appropriate. Flexibility should be encouraged within the data analysis stage and lay researchers should be made aware than the network groupings and diagrams are for guidance purposes only.

Strengths and limitations

While card sorting methods have been used in participatory qualitative data analysis, the practice is not widespread. One existing method (Stevenson [REF]) has the lay researchers sort and group through discussion. The individual sorts of PTE and network analysis construction of initial groups ensure those groups represent the perspectives of all lay researchers equally. It is also considered that the process could easily be adapted for virtual delivery, thus increasing its accessibility. As such, this study is both novel and innovative. Nonetheless, there are several potential limitations.

The study provided a relatively limited number of excerpts to lay researchers and this may have unintentionally narrowed the number of themes and limited the depth of analysis. Efforts were made to mitigate this by involving a lay project advisor (and service user) in data selection and selecting diverse excerpts.

Further, one selection criterion for excerpts was that they could stand alone and, when used in the sorting, these excerpts were presented as independent sentences. This approach separates the excerpts from their context and raises the possibility that the lay researcher interpretation of an excerpt is different than what was intended in the source material.

The number of lay researchers involved could be expanded. However, one must consider the impact that increased numbers may have on data analysis (Step 5) in regards to timing, group facilitation etc. Only six individuals who completed Step 3 (sorting) were available at Step 5 (analysis). As such, all views may not have been fully represented. Moreover, the suitability of PTE within non-intervention-based research designs is yet to be tested.

Finally, the network-based community detection algorithms used in this study were selected as they are popular in the social networks literature. However, there are many algorithms available and the network of excerpts represents a different type of relationship than is typical in social networks. Further research is required to compare the clusters generated by different algorithms and assess whether specific algorithms are more or less appropriate for PTE grouping.

Conclusion

In conclusion, lay researchers described the PTE method as accessible and enjoyable with participation time approximately 5–6 hours in total. The method should therefore appeal to a wide range of researchers seeking to improve or introduce user-led analyses and perspectives within their work. This is particularly true of qualitative data generated through intervention-based evaluations as opposed to research exploring individuals’ lived experiences, where the use of excerpts could remove important contextual information. The data presented here also demonstrate how open card sorting and network analysis methods can be successfully incorporated into a participatory framework. Further iterations of PTE would do well to include some additional modelling to strengthen the robustness of the community detection algorithms. This process would also benefit from an increased number of users applying the PTE method. It would also be useful to assess the acceptability of PTE from a researcher/academic perspective – particularly in regard to accessibility, cost, implementation and value.

However, we do not seek to turn this important process into a mechanical approach that is overly manualised, process-driven and ultimately meaningless. We are also conscious of those who are wary of involving ‘machines’ in qualitative-based research and are reminded of the words of Bernard and Ryan (2010: 73) – ‘ultimately, we [humans] are responsible for all analysis … computer based content … [will] just be another tool that will (1) make the analysis of qualitative data easier; and (2) lead to much wider use and appreciation of qualitative data in all the social sciences’. The main goal of PTE is to provide a practical and flexible approach to increasing user-involvement in qualitative data analysis that can be easily incorporated into current research designs (principally those that are intervention-based) and reduces the potential for researchers to influence the process. Moreover, PTE adds value by enabling a broader range of perspectives on the data collected, thereby enriching the analysis and contributing to the rigour of the research. While the study presented here was conducted in the context of health research, the method is sufficiently general to be used in other social science disciplines seeking to involve lay researchers in the qualitative analysis process.

Disclosure statement

There are no relevant financial or non-financial competing interests to report.

Additional information

Funding

The original feasibility study was funded by an HSC R&D Enabling Research Award (Public Health Agency Northern Ireland).

Notes on contributors

Paul Best

Dr Paul Best is the founder of the Immersive Technologies and Digital Mental Health Network within the School of Social Sciences, Education and Social Work at Queen’s University Belfast (QUB). He is a qualified social worker, cognitive behavioural therapist and Programme Director (UGR) for Social Work at Queen’s University, Belfast. His primary research interests relate to the use of immersive technologies in the treatment of mental health conditions as well as participatory approaches to data analysis. Email: [email protected]

Jennifer Badham

Dr Jennifer Badham is an applied mathematician with a background in health policy. Her research uses agent-based modelling, social network analysis and other complex systems methodologies to understand the ways in which individual decisions, social structure and social phenomena mutually influence and constrain each other. Email: [email protected]

Tracey McConnell

Dr Tracey McConnell is a Marie Curie Senior Research Fellow, School of Nursing and Midwifery at Queen’s University Belfast. Her research has focused on improving mental health outcomes and improving palliative and end-of-life care for patients and their families. Tracey’s methodological expertise includes mixed-method realistic reviews/evaluation to help identify what works, for whom, and in what circumstances when designing, implementing and evaluating complex healthcare interventions. Tracey also has methodological expertise in Randomised Controlled Trials. Email: [email protected]

Ruth F Hunter

Dr Ruth F Hunter is a Reader (Associate Professor) in Public Health at the Centre for Public Health at Queen’s University Belfast. Her research has focused on health behaviour change and social network analysis. Email: [email protected]

Appendix A.

Recruitment of Co-researchers

Participant	Gender	Step 2 Capacity Building	Step 3 Data Sorting	Step 5 Data Analysis
Co-researcher 1	Male	X	X	X
Co-researcher 2	Male	X	X	X
Co-researcher 3	Female	X	X	X
Co-researcher 4	Female	X	X	X
Co-researcher 5	Female	X	X	X
Co-researcher 6	Female	X	X
Co-researcher 7	Male	X	X	X
Co-researcher 8	Female	X	X
Co-researcher 9	Female	X	X

Appendix B

Full list of excerpts and groups produced during Step Four

Number	Excerpt Text	Central
Group 1
ID01	I think, when it started, there were one or two just technical things that… weren’t a big deal but, in some way, kind of did interrupt the flow of the conversation. So, that was quite distracting.
ID03	the technology was just a wee bit of an issue, but… I think resolved itself because when [we] moved into an office in our building where we have a good system	X
ID08	So, I think it’s less easy on a videoconferencing call to actually pick up any non-verbals of the other people who are not talking.
ID20	In terms of attendances and stuff, it was quite low, so… that sort of hindered me getting the full experience from what an online support group could be.
ID21	I really found that extremely difficult… in terms of syncing the voice when we were talking… trying to be sensitive, pick up on tones and moods… and even looking at the people and their body language
ID34	I thought it would have been nice to maybe have a few more men that I would be able to relate to, because people like myself, obviously, we question ourselves… I think we suffer in silence, that’s the problem.
ID46	The only like bad thing was some people… you would talk over each other sometimes, but that even happens face-to-face, but that’s about it. It was grand otherwise.
Group 2
ID06	I think my part in the group was more conversational than it might be in a face-to-face group, where I probably would have said far less, largely because the face-to-face groups tend to be bigger.	X
ID07	When you’re in a group (face-to-face), you’re very much… in someone else’s building and it probably isn’t as easy to feel as comfortable to maybe be as conversational as maybe we were in the… on the online group.
ID16	There may be people who may feel more comfortable with the teleconferencing… the fact that they’re in their own home, rather than having the anxiety and… sometimes the discomfort that you can find when you have to take yourself out of that safe space and you have to go in to another space.
ID24	I think the number one reason (wanted online support) is that I do get a little bit anxious in a group of people that I don’t know.
ID26	I felt like I could open up a lot more than I would do in a room full of people. It felt a lot more comfortable for me.
ID32	Whenever sometimes you feel very down, you just don’t really feel like… getting yourself all dressed up and having to go out. You kind of just want to be in the comfort of your own home, but also talk to someone and feel close to someone… and you do feel so much better after it. So, I’d definitely recommend it.	X
ID36	I would recommend it (online support) to people. One benefit of the online session of course would be that you can just link in from your phone and you don’t have to anywhere… you just don’t have to travel.
ID38	I mainly wanted online because I don’t do very good in public, in groups… online I could do it from the comfort of my own home.	X
ID39	It helped me even more. The fact it was online, I was probably able to be more open with them (group facilitators)… if I’d been in a group, I probably would have sat there and kept my mouth shut.
ID45	I went to the doctor and obviously the waiting list was so long, so I turned [to online]… You don’t have to get like ready to go anywhere or anything, don’t have to face getting up and going somewhere, so – it’s better, I think, to do it online.
ID48	Yes, definitely (would recommend online support) because I know there’s lots of people who do find it hard to get up and get out, so doing it from the comfort of your own house and sitting just doing it online can definitely be beneficial for some people.
ID49	I do need more help and I thought a group would be a good idea, and the online one, I thought would suit my lifestyle better… and also less personal.
Group 3
ID04	I think it’s good that we see each other, and people who have been on the calls, not all of them have been used to Skype and yet it’s been a system that people have got used to very quickly.
ID05	It seems to work with most settings, you can just use it from your phone.
ID11	I actually found it quite easy to engage with the users, and I found it quite easy to communicate with them, and we didn’t seem to have any problems going back and forth.
ID22	The protocol (facilitator handbook) did help… it would be very supportive.
ID27	I didn’t have a problem with the questions at all (when registering to take part). I don’t think they did (take long to complete).
ID30	It was no problem for me (using technology). I’ve used it in the past. Sometimes I used my phone to get on and chat, so I don’t have to drag the laptop out all the time, so it’s handy.	X
ID40	Now that I’ve used it, because that was the very first time I’d ever used Skype, I would use it again, didn’t see any problem with it then once it got up and running.
Group 4
ID02	Now, it has been a small group, but even at that, people listen well and have been really good at taking their turn, so it’s been very positive
ID09	I’ve found that whenever we were able to get people engaged and connected to the sessions, the sessions went very well.
ID14	So, I think the best thing to do is just to… focus on the positives of the service and promote that, because, from what I did see from talking to the members that were using the service regularly, I could see and I could hear a difference in them, and they did seem to be benefiting from it.
ID18	Overall, I found it a very interesting and a very enjoyable experience (facilitating an online group).	X
ID19	When I was actually talking to the people on the online support group, I found… it ran very much like a normal group in terms of how you would support people.
ID25	So, it’s been really good for me, and it’s brought me a little bit closer to other people, even though it’s online. I can feel like a wee bit closer, and I don’t feel so left out.
ID28	No, I think, (facilitator) is great… She actually comes in and she says, well, I can relate, and this is what happened to me”. I think she’s amazing.
ID31	Absolutely (been a benefit). It’s been really nice to [sighing]… to have people that listen… when other people are chatting, and I can offer maybe some advice – this is just some techniques or something that you could do that I do. It’s something that I do look forward to.
ID35	The facilitator was great. I don’t know how difficult it would be for (facilitator) to manage a bigger group, but certainly, for the numbers that we were talking about, nobody felt that they didn’t get a chance to share how they were feeling.
ID41	One of the benefits was, just being able to open up to somebody, with my problems, that would listen to me and gave me some great advice.. it’s actually got me out of an abusive relationship, for a start… so that’s a very, very positive thing that came out of it.
ID42	Oh God, yeah, definitely (would recommend to others). I’ve already mentioned to a couple of people about it, and I’ve said how brilliant it was, how very, very good and how helpful it was.	X
ID44	An interesting finding is that an interviewee had mentioned … .that she is now looking at joining the other services offered by us.	X
ID47	I like somewhere that I can actually speak openly about stuff… there’s people there that are going through the same thing and they can understand and relate.
Group 5
ID13	So, I think with the face-to-face group, there’s a bit of an onus on the individual to make it to the meeting, whereas, with an online support session, the user may just think, ‘You know something, I don’t really feel up to this today’, and it may not feel as real and as concrete for them because they may feel it’s more of a virtual session rather than a real session.
ID15	I think in terms… if you’re looking to deliver a support service, it’s important to have a good support model in place (referring to facilitator handbook).	X
ID33	Obviously, the bigger the group, the more interaction, the less time there is perhaps for individual participation, but at the same time, maybe you would hear from a lot more people, from a broader range.
ID43	I thought more people would have actually jumped at the chance to get into it. Unless putting a testimonial… about somebody being afraid of using Skype but then found it so beneficial or so easy to use in the end… because I think that might be what the stumbling block is.
Group 6
ID10	I just think that the technology got in the way a little bit because, if you’d be talking to someone, there’d be a little bit of feedback… when you’re not there to pick up on non-verbal cues as easily, it can be a little bit challenging.
ID12	Maybe a little bit of training around Skype might be useful, just to be fully familiar with it, and then just to explore, small things that might make the service a little bit better or might ensure that things are smoother.
ID17	I would say… if this was going forward, perhaps we have a designated space where online facilitation happens, and if it was just AWARE’s own logo types and… information there… this isn’t just a facilitator sitting in his house doing this.
ID23	Whenever you’re signing up to register, it’s easy to do that because there’s no big commitment there and you’re not paying any money… so as your motivation or your mood would change, then that could be a barrier to coming along, or maybe it was just easy to go and click a button and say, oh, that looks interesting, like I would do with a lot of stuff online.
ID29	I think the only thing is that sometimes an hour is not long enough [laughing]!
ID37	But I do feel though that you need to be aware of the technology, how it works – and there’s a lot of elderly men and women, they’re horrified at the prospect of having to learn something new or to maybe manage something technically.

Appendix C

Photo of Co-researchers undertaking the sorting task.

Table 500.

Spreadsheet Input during Step Four

Excerpt	Participant	Pile/Group Label
ID1	P1	Gp6
ID2	P1	Gp3
ID3	P1	Gp6
ID4	P1	Gp1
ID1	P2	Gp2
ID2	P2	Gp5

Appendix E.

Number of Excerpts within each group

Group 1	Group 2	Group 3	Group 4	Group 5	Group 6
7	12	7	13	4	6

aGroup 6 included the remaining excerpts that didn’t fall into other groupings

Appendix F:

Training Evaluation Form

Participatory Theme Elicitation (PTE) Training Evaluation Form

We would appreciate if you could take a few minutes to share your opinions with us so we can improve on this training.

Please return this form to the organizer at the end of the training. Thank you.

Date:________________________

1. The duration of training was: a. Too short b. Just right c. Too long

2. The length of quotes were a. Too short b. Just right c. Too long

3. The number of quotes were a. Too short b. Just right c. Too long

StronglyStrongly

agreedisagree

1. I understood the sorting task12345

2. Please rate the following:

ExcellentVery GoodGoodFairPoor

1. Information provided

2. The training overall

1. What did you most appreciate/enjoy/think was best about the training day?

________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

1. Any suggestions for improvement?

________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Your Background

1. Gender a. Male b. Female9. Age ____________

10. Educational Background N/A GCSE A-Level NVQ Degree

Other (please specify ____________________________________)

Thank you!

Please return this form to the organizer at the end of the training session.

Appendix G.

Summary of themes developed across groups

Group Number	Proposed groups	Final Group	Theme Label
1	ID01, ID03, ID08, ID20, ID21, ID34, ID46	ID01, ID03, ID08, ID10a, ID21, ID34, ID46	Challenges of using the online service
2	ID06, ID07, ID16, ID24, ID26, ID32, ID36, ID38, ID39, ID45, ID48, ID49	ID04a, ID06, ID07, ID11a ID16, ID24, ID26, ID30a ID32, ID36, ID38, ID39 ID45, ID48, ID49	Benefits of using the online service
3	ID04, ID05, ID11, ID22, ID27, ID30, ID40	ID04, ID05, ID11, ID27, ID30, ID40	Role of technology in facilitating help-seeking
4	ID02, ID09, ID14, ID18, ID19, ID25, ID28, ID31, ID35, ID41, ID42, ID44, ID47	ID02, ID09, ID14, ID18, ID19, ID25, ID31, ID41, ID42, ID44, ID47	Therapeutic processes and outcomes within an online context
5	ID13, ID15, ID33, ID43	No thematic pattern identified	N/A
6	ID10, ID12, ID17, ID23, ID29, ID37	ID12, ID17, ID29, ID37, ID43a	Service Improvements

aMoved from original group or used across multiple groups/themes

Appendix H:

Protocol for PTE

Participatory Theme Elicitation Protocol

Step 1: Data Selection

• Identify qualitative dataset

• Select excerpts for inclusion

• Cut excerpts into individual strips or segments

Step 2: Capacity Building

• Provide an overview of the project

• Discuss different types of research data (qualitative, quantitative etc.)

• Describe the Participatory Theme Elicitation method

• Demonstrate the subjective nature of qualitative analysis through sorting activities (e.g., images of fruits and vegetables)

Step 3: Data Sorting

• Provide each co-researcher with a set of excerpts

• Each co-researcher is to group their excerpts into piles, with at least two piles, no minimum pile size, and no ‘miscellaneous’ pile.

• Emphasize that this is an independent activity

Step 4: Data Grouping

• Create a weighted network of excerpts, with an edge wherever at least one co-researcher placed the pair of excerpts in the same pile and the edge weight given by the number of co-researchers who placed the excerpts together.

• Apply a community detection algorithm to the network to create groups of excerpts.

• Note that step 4 can be completed online, see [ANONYMISED LINK]

Step 5: Data Analysis

• Gather co-researchers together

• Audio record session for accuracy

• Present network diagram and groupings

• Emphasize that the network data are only for guidance purposes

• Discuss the relationships between excerpts contained within each group

• Examine possible outliers or excerpts that might be better placed elsewhere