Methods for think-aloud interviews in health-related resource-use research: the PECUNIA RUM instrument

ABSTRACT

Background

The think-aloud (TA) approach is a qualitative research method that allows for gaining insight into thoughts and cognitive processes. It can be used to incorporate a respondent’s perspective when developing resource-use measurement (RUM) instruments. Currently, the application of TA methods in RUM research is limited, and so is the guidance on how to use them. Transparent publication of TA methods for RUM in health economics studies, which is the aim of this paper, can contribute to reducing the aforementioned gap.

Methods

Methods for conducting TA interviews were iteratively developed by a multi-national working group of health economists and additional qualitative research expertise was sought. TA interviews were conducted in four countries to support this process. A ten-step process was outlined in three parts: Part A ‘before the interview’ (including translation, recruitment, training), Part B ‘during the interview’ (including setting, opening, completing the instrument, open-ended questions, closing), and part C ‘after the interview’ (including transcription and data analysis, trustworthiness).

Conclusions

This manuscript describes the step-by-step approach for conducting multi-national TA interviews with potential respondents of the PECUNIA RUM instrument. It increases the methodological transparency in RUM development and reduces the knowledge gap of using qualitative research methods in health economics.

KEYWORDS:

• Instrument development
• patient involvement
• qualitative research methods
• resource-use measurement
• think-aloud

1. Introduction

Self-reported resource-use information is key in capturing costs in economic evaluations, in particular concerning data that are not represented in administrative sources, such as costs in sectors other than the healthcare sector [1–4]. Resource-use obtained from RUM instruments may not always be in line with actual resource use [5–7]. This can be partially explained by the problems that individuals may encounter when completing RUM instruments for health economics research purposes [2]. Such problems may include, for example, difficulties in understanding questions and answer options [8], difficulties in distinguishing between types of care providers [9], or recall difficulties arising from either too short or too long recall periods [10,11]. Furthermore, some underlying constructs (e.g. stigmas) might cause respondents to report a more socially desirable answer [8]. Despite these problems, RUM instruments are often developed without consulting potential respondents, among other relevant stakeholders, throughout the development process [11]. If these problems are not taken into account during the development of RUM instruments, then using such instruments in practice could lead to biased results.

A potential solution to this problem is the timely involvement of respondents in the RUM instrument development process [11]. By employing qualitative methods, insights into respondents’ thoughts, thinking processes, and difficulties when completing the instrument can be gained [12,13]. These insights can be used to tailor the RUM instrument to the potential respondents. However, given the existing stigmas on certain types of resource-use (e.g. those on mental health care use [14]), individual resource-use could be a sensitive topic. Qualitative research methods conducted in one-on-one settings, such as think-aloud (TA) interviews, a form of cognitive interviewing (CI) [15], allow sensitive topics to be treated more confidentially compared to research methods conducted in group settings [12,13]. During TA interviews, cognitive processes are captured while individuals are encouraged to think aloud during the individual performance of a task [16,17]. TA is a widely used method for deriving information on questionnaire design [18,19]. Yet, so far, it has rarely been used for the development of health-related RUM instruments [20], hence there is little guidance on how to do this [20,21].

Methodological papers describing the application of the TA method are only available in other fields [17,22], and transferability of these is unknown. Therefore, to further support the use of TA methodology in RUM-related research, the aim of the manuscript is to describe the methods used for conducting TA interviews in the development process of the PECUNIA (ProgrammE in Costing, resource use measurement and outcome valuation for Use in multi-sectoral National and International health economic evaluAtions) RUM instrument. Although the methods were initially developed to ensure a transparent and uniform method for multi-national data collection with multiple interviewers as part of the PECUNIA project, the paper may also serve as a guide for researchers who want to conduct TA studies in the field of health economics. Transparent publication of qualitative research methods for health economics research may contribute to the use of TA methodology. Inherently, such methods would add value by bridging the current knowledge gap in the predominantly quantitative field of health economics [23]. For those unfamiliar with the methods of TA we explained the pros and cons of TA compared to other validation methods in the paragraph ‘Cognitive interviewing and think aloud.’

1.1. Cognitive interviewing and think aloud

There has been increased emphasis on building quality into questionnaire design through pretesting [24]. One of the key pretesting methods is CI. CI is an evidence-based qualitative methods which is specifically designed to whether a survey question satisfies its intended purpose [15]. CI pretests are typically used relatively early in survey development [25]. Its aims are to determine whether respondents understand the items as intended, whether they are able to recall information they need, and whether they are able to select an appropriate response. CI ideally precedes behavior coding or respondent debriefing techniques and, other than these alternative techniques, does not necessarily need to represent the usual or field survey context. Furthermore, CI only requires a small number of number of interviews – regularly twenty or fewer – to obtain valuable information, and interviewees can be purposively selected [24,25].

CI is conducted using two key procedures. The first is verbal probing, which is an active form of data collection in which a cognitive interviewer asks a series of questions designed to elicit information beyond that normally provided by respondents. Verbal probing techniques such as concurrent probing require training and skill on the part of the interviewer. If not done carefully, it could lead to reactivity effects. The alternative to verbal probing is TA, which is a technique wherein survey respondents are asked to actively verbalize their thoughts as they attempt to answer the evaluated survey items [15,24]. TA is intended to capture the first information available to the respondent. TA requires very little training demands on the interviewer, and avoids being too directive in ways that could bias responses [15]. A potential weakness of TA is that it could place a burden on respondents, as the process could be experienced as unnatural and difficult. However, given the early stage in the development process of the RUM, the skill level of the interviewers, and the aim of minimizing bias in just in time responses, TA was considered the preferred choice over other qualitative pretesting methods in this study.

2. Methods

2.1. Context

The TA interviews were conducted as part of the project PECUNIA (Grant Agreement no. 779,292) [26]. PECUNIA aims to ‘tackle the healthcare challenges of an ever-growing and rapidly ageing population in Europe by developing new standardised, harmonised, and validated methods and tools for the assessment of costs and outcomes in European healthcare systems’ [26]. The PECUNIA Consortium is an international collaboration of health economists from ten academic European institutes in six European countries. One of the objectives of the PECUNIA project is to develop a standardized generic multi-national, multi-sectoral RUM instrument, i.e. the PECUNIA RUM which is also harmonized with the other PECUNIA costing tools. TA interviews were conducted as part of a larger process to establish the validity of the draft PECUNIA RUM instrument. More details on the development process and content of the instrument are discussed elsewhere [27]. The final version of the PECUNIA RUM instrument can be accessed online [28]. Country-specific National Medical Ethical approval for this study was sought and given.

2.2. Methods development

In this manuscript, the developed TA approach is referred to as ‘methods.’ The methods for the TA interviews are a written document, originally developed in the English language by a multi-national working group of the PECUNIA Consortium. The steps are described in chronological order including a description of the steps before (translation, recruitment, training), during (setting, opening, completing the instrument, open-ended questions, closing), and after the interview (transcription, data analysis) (Figure 1). The methods were developed in four online meetings, and additional expertise was sought by involving a qualitative research expert and studying literature on qualitative research methods [29]. The aim of the first two meetings was to decide on a method that would allow for the aim of the study. The aim of the latter two meetings was to decide on the operationalization process, given the European scope of the PECUNIA project. The involved expert oversaw the quality of the protocol, and paid attention to limiting the power imbalance between respondent and interviewer (see B1).

Figure 1. Schematic representation of steps in the study protocol.

3. Before the interview

3.1. Translation

To overcome language barriers for respondents, participating partners translated the PECUNIA RUM instrument from English to their national language. Partners pilot-tested the translated version of the instrument among two or three colleagues for quality assurance prior to use in the think-aloud study.

3.1.1. Recruitment of respondents

3.1.2. In- and exclusion criteria

Partners aimed to recruit five to eight respondents per participating country. Respondents were either former mental health care users or current informal caregivers. In this TA protocol, we did not intend to reflect the full spectrum of patient experiences, but a specific sample group was selected, which was likely to struggle with resources use in several domains. Therefore, we selected for this protocol former MH patients and informal caregivers of MH patients, which we consider a relevant group using resources in and outside the healthcare sector. These persons were likely to have interacted with healthcare systems and be aware of the broader consequences of a disorder. With regard to their vulnerability, current mental health care users were excluded. Former mental health care users were defined as ‘people who used mental health services in the past five years, but did not use it in the past twelve months.’ In this study, informal caregivers were defined as ‘people who currently take care of a mentally ill relative, friend or neighbour or did so in the past five years.’ An equal division between former mental health care users and current informal caregivers was preferred, but not required, as both types of respondents were believed to offer equally relevant perspectives. To ensure a broad perspective, eligibility criteria were formulated to be as inclusive as possible within the practical boundaries of the study, without exclusion due to characteristics such as age, sex, socioeconomic status, education level or former diagnosis.

3.1.3. Recruitment approach

Each partner was responsible for the recruitment of potential respondents within their country and was free to adopt a recruitment approach that best suited the respective cultural and organizational setting. Contacting patient representation groups and the use of social media were suggested.

3.1.4. Information provision

Potential respondents were informed in lay national language about the aim and the purpose of the study, their rights, and what participation entailed (see English example in SF1). Once a person agreed to participate, the relevant partner sent a confirmation letter to the respondent with explanation of their rights to withdraw. Written consent was sought before the start of the interview (see also B2 – opening).

3.2. Training

An online training session for interviewers was organized to harmonize the practical aspects of the interview process and to practise the interviewing skills of the researchers (See SF2). In this meeting, firstly, qualitative research theories and data analysis plans were presented (as explained in section C2). Secondly, partners received information about all the phases of the interviews (see Section B). Lastly, interviewers practised their TA interviewing skills by conducting short rounds of TA interviews in pairs followed by rounds of feedback and reflection on each other’s performances. By playing the role of the respondent, the researcher experienced the difficulty of the task and the potential vulnerability of the interviewee.

4. During the interviews

4.1. Setting

Given the COVID-19 pandemic at the time of this study, there has been a shift from face-to-face toward online research and universities have developed their own policies to comply with national guidelines. Both face-to-face and online possibilities were accepted in the current approach, so that partners had the possibility to opt for the most suitable approach, taking into account the national/regional circumstances and policies. If face-to-face interviews could be organized in a safe way, this was preferred, as it better allows for non-verbal communication of the respondents to be observed by the interviewer.

The location of face-to-face interviews needed to be chosen carefully as it could influence the respondent’s level of comfort. It was recommended to offer the respondent the possibility to meet at a location of his/her preference, although crowded places should be avoided due to privacy considerations (e.g. the risk of unintentionally recording bystanders without consent). This could be either 1) at their home, 2) at a neutral place (e.g. public library), 3) at the university. The latter option was least preferred as the university might be an overwhelming environment for some respondents. Interviewers were advised to always take into account the latest national COVID-19 physical distancing rules. Researchers should abide by a ‘lone worker’ policy, such as the one adopted by [XX], a guide that lays out the risks and sets out the rules of working alone [30].

Online TA interviews could be held via video conferencing services approved by the partner’s university. A video connection is necessary to capture non-verbal communication. It was recommended to send a hard copy of the PECUNIA RUM instrument to the respondent in advance and have them complete the pen-and-paper version during the interview. The alternative was to use the ‘share screen’ option, when the respondent shares the screen with the interviewer and then goes over the questionnaire online. While a video connection is necessary to notice non-verbal behavior, audio recording only was decided sufficient for the analysis of the online interviews. Participating in interviews online might be more difficult for persons who are less familiar or less comfortable with using computers, therefore, participants who felt the need for this were allowed to invite someone to assist them.

The interviews were audio-recorded with an encrypted audio recorder. Respondents were first informed about this when they received the confirmation letter. To minimize the power imbalance between the interviewer and the respondent, the interviews were conducted individually, with one researcher present per interview. Each recruitment site was allowed to divide the workload between a maximum of two researchers per country.

The interview duration was restricted to one hour; otherwise, the findings might be affected by fatigue [31]. Survey fatigue may prevent respondents from fully executing the exercise and simultaneously thinking-aloud, and may cause them to choose more convenient answers. If a respondent could consequently only complete the instrument partially, the next respondent was asked to start from the subsequent module to guarantee that all RUM modules were equally covered.

4.2. Opening

The interviewer started the meetings by welcoming the respondent, introducing themselves and explaining the process of ‘thinking aloud.’ The respondents were informed that their feedback was essential to improve the questionnaire, ergo there were no right or wrong answers and their actual resource-use was not important. Respondents had to fill in their true resource-use as recalled, but the purpose of the exercise was to see if they understood the questionnaire, rather than discussing their resource-use. When respondents identify themselves as working with the research team instead of identifying themselves as a study subject, they might be more willing to share personal experiences. Furthermore, the interviewer explained the informal caregivers that they had to complete the PECUNIA RUM instrument on behalf of the person they cared for. Finally, the interviewer mentioned that it was possible to stop at any given time without having to give any explanation. Informed consent was explained and obtained before starting with the TA exercise. One existing question of the PECUNIA RUM instrument, or an unrelated random question, was completed jointly with an imaginary example to illustrate the task for the respondent. Afterward, audio-recording started.

4.3. Completing the PECUNIA RUM instrument

During the TA interviews, the level of interference from the interviewer was kept to a minimum and served only to encourage the respondent to verbalize thoughts, for example by asking ‘how did you arrive at this number?.’ The interviewer was advised to encourage the respondent to circle words perceived as difficult or unclear with a pencil, or mark the word/sentence when filling in the RUM online. Content-related questions from respondents (e.g. ‘Does this type of resource-use also include online calls?’) were not answered by the interviewer, to simulate the task of completing the questionnaire when a researcher is absent. Interviewers were instructed during the training how to answer such questions (e.g. ‘I may not answer this question right now, what would you do if I weren’t here?’). This part of the interview was stopped after 40 minutes even if the PECUNIA RUM instrument was not fully completed, to maintain sufficient time for open-ended questions (part B4).

4.4. Open-ended questions

After filling in the PECUNIA RUM instrument, fifteen minutes were dedicated to semi-structured open-ended questions. The aim of this peer-debriefing was to gain more in-depth information on the overall experience of completing the PECUNIA RUM instrument by reflecting on the process. Even though some example questions were provided, it was more relevant for interviewers to have a natural conversation in which interest was shown in the respondent’s experiences and answers to the questions of the PECUNIA RUM instrument. Example questions touched upon their general experience (‘How did you find completing it?,’ ‘How did it make you feel?’), and the experienced difficulty (‘At which items did you experience confusion?,’ ‘To what extent are you able to remember your resource-use with a three month recall period?’).

4.5. Closing

Respondents were thanked at the end of the interview and received a small (monetary) gift afterward (e.g. a voucher or cash). Respondents were asked to give feedback on the interview itself and the interviewer’s performance. This evaluation was used to enhance the quality of subsequent interviews and to make the respondent feel more included as part of the study team. The interviewers wrote a brief summary of their subjective experience after each interview and sent it with the transcripts to the researcher who was analyzing the interviews. The researcher conducting the data analysis read the summary before reading the transcripts, as it gave more context to the transcript.

5. After the interviews

5.1. Transcription and data analysis

Transcripts were analyzed using both the model of the survey response process by Tourangeau et al. [32] and a phenomenology approach, which allows for respondent’s verbalized thoughts to be analyzed as lived experiences, for describing the respondent’s experience of completing the PECUNIA RUM instrument. The Tourangeau model describes four phases (comprehension, retrieval, judgment, reporting) that a respondent can go through when completing a survey. The thematic analysis method as described by Braun and Clarke [29] was used. In accordance with this method, firstly, the interview transcripts were read in their entirety to get a first grasp of the ambiance/context/setting during the interviews. Secondly, the transcripts were read more carefully and data were divided into segments (open coding). All segments that could reflect the same category or themes were clustered. Then, new connections between categories were made by exploring the context of the segments. Every segment was connected to one phase of the Tourangeau model.

Findings were shared with the PECUNIA Working Group and were used to adapt the instrument over joint online meetings with the Working Group. Data were stored in a secure online environment in accordance with the data management plan consolidated by the PECUNIA project.

5.2. Trustworthiness

Trustworthiness of the study results was assessed based on the concepts of credibility, transferability, dependability, and confirmability [33–35]. Credibility, ‘the extent to which the study’s findings are trustworthy and believable to others,’ was safeguarded by the recruitment of a heterogeneous group of respondents, and by having multiple interviewers. Transferability, ‘the extent to which the findings can be transferred or applied in different settings,’ was covered by the brief summary that the interviewers wrote about their experiences of each interview. Dependability, ‘the extent to which the findings are consistent in relation to the context in which they were generated,’ was checked by analyzing three interviews to check for data saturation. Confirmability, ‘the extent to which the findings are based on the study participants and settings instead of researchers’ biases’, was safeguarded by peer debriefing: respondents were asked about their experiences using open-ended questions after completing the PECUNIA RUM instrument.

5.2.1. Experiences of the interviewers

As mentioned in the methods each interviewer send a separated summary report to the coordinator about each interview and their own experiences. The interviewers indicated that they considered the TA protocol ‘easy to apply.’ Some of the interviewers mentioned that sometimes it was difficult to keep motivating the respondent to verbalize all their thoughts and how they got to a certain calculation, as this is a less natural process for respondents. As the experiences from the interviewers were comparable for the different interviewers coming from the various countries and overall suggest that the TA protocol is applicable during the interview, this did not lead to any changes in the TA protocol.

6. Discussion

This article describes the methods that were followed to conduct TA interviews for pilot-testing the PECUNIA RUM in four countries as part of the PECUNIA project. These interviews aimed to gain more insight into the experience of a respondent when completing the PECUNIA RUM. Results of the qualitative analysis were used for further improvement of the PECUNIA RUM instrument and will be published separately.

The developed TA approach was successfully applied to identify thoughts and experiences of the respondents when completing the PECUNIA RUM instrument. The TA method thus can be considered a suitable method for capturing thoughts and experiences when completing a RUM instrument as it mimics a ‘real-world-situation’ with only minor interruption from an interviewer present. Given the aim of the study, this is a strength of the TA approach compared to other one-on-one qualitative research methods, such as structured debriefing [36], which includes a deeper analysis of respondents’ thoughts by the respondent and the interviewer together.

The majority of the respondents expressed some discomfort at the beginning and became more relaxed during the interview. Therefore, we would recommend future researchers that will apply the TA approach to take more time at the beginning to make the respondent feel more included (e.g. by repeating that the actual resource-use is not relevant for this study). Furthermore, the interviewers noted that sometimes respondents wrote down zero resource-use for an item, as some relevant resource-use did not came to mind. For example, some individuals did initially not recall having any medical tests, and only did so when specifically asking whether they had COVID-19 related tests. Even though interruption should still be kept to a minimum, we recommend that future researchers encourage the respondents a bit more to dig deep (e.g. the interviewer could ask the respondent what professions he/she thinks are involved in the resource-use item).

Several limitations apply. Current mental health care users were excluded, and by doing so we excluded views of a potential target group of users. In addition, the eligibility criteria may have led to the exclusion of persons, as persons with mental health disorders that may be considered ‘more severe’ or prolonged were less likely to be included. Furthermore, only certain kinds of cognitive processes – not too complex and not too easy tasks – are accessible by using TA interviews [16], and even though the open questions during the interview created the possibility to gain more insight into respondents’ experiences, it remains uncertain to what extent the full experience was captured. Complementing the TA approach by additional qualitative methods (such as more in-depth interviews) may be a viable strategy to explore in future research.

7. Conclusions

In conclusion, capturing the viewpoint from respondents by using the TA approach allows for gaining insight into the thoughts and experiences of the respondents. It gives valuable information about the cognitive processes from reading a question to writing down the answer. This information is beneficial during the development process of a RUM, as it exposes which problems may arise. Using this information to adapt the RUM instrument leads to better-tailored RUM instruments. Publishing the methods in detail adds value to health economics studies as it increases the methodological transparency in RUM development, which allows researchers and policy makers to better interpret study results. Furthermore, it reduces the knowledge gap of qualitative research methods in health economics and facilitates the use of TA methods in the field.

Declaration of interest

This research is part of the ProgrammE in Costing, resource use measurement and outcome valuation for Use in multi-sectoral National and International health economic evaluAtions (PECUNIA) project, which aims to develop a multi-national multi-sectoral instrument for health economics research. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Previous presentation

Methods of this manuscript are presented at the iHEA2021 online conference.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/14737167.2023.2187379

Additional information

Funding

This project has received funding from the European Union´s Horizon 2020 research and innovation program under grant agreement No 779292.