Measuring health-related stigma: Exploring challenges and research priorities to improve assessment

ABSTRACT

Despite the advances in stigma research, measuring health-related stigma continues to be challenging and knowledge gaps remain. This study gained insight into challenges and research priorities related to the assessment of health-related stigma. Interviews were conducted with 14 stigma researchers, followed by a survey that was completed by 36 respondents. The findings showed a diverse range of research priorities. Among the top ranked priorities were the need for robust measurement properties of existing scales (content validity, responsiveness, validation across settings), exploration and assessment of subtle changes in stigma, and investigation on ways to assess actual behaviour and discrimination. Various challenges with the cross-cultural use of measures were identified, along with a research opportunity to shorten the cross-cultural validation process. Other identified research priorities related to: studying multi-level intersectional stigma; focusing on positive features that counter stigma; rephrasing negative and offending scale items; developing generic measures; and, the further development of practical tools to support researchers with scale implementation. The defined research priorities can guide future studies to advance stigma measurements and, as our ability to measure is critical for our understanding, enhance our knowledge about the complex stigma processes.

KEYWORDS:

• Stigma
• assess
• scale
• measure
• agenda

Introduction

Stigma has been shown to increase the burden of a wide variety of health conditions, including infectious diseases like HIV (Pantelic et al., 2017; Relf et al., 2021; Yang et al., 2021), tuberculosis (TB) (Oladele et al., 2020) and leprosy (Sermrittirong & Van Brakel, 2014; Van Brakel, Peters, et al., 2019), and non-communicable diseases (NCDs) like cancers and diabetes (Rai, Syurina, et al., 2020). This study adopts the definition of health-related stigma by Weiss and Ramakrishna (2006), which states it as ‘a social process or related personal experience characterised by exclusion, rejection, blame, or devaluation that results from experience or reasonable anticipation of an adverse social judgment about a person or group identified with a particular health problem’ (p. 536). Drivers of health-related stigma may vary across conditions and cultures, and can include factors like fear of infection, perceived unattractiveness, lack of awareness and cultural values and beliefs (Sermrittirong & Van Brakel, 2014; Stangl et al., 2019). The consequences of health-related stigma can be far-reaching and affect various areas of the affected person’s or their family’s life, including but not limited to marriage, social relationships, education, and careers (Link & Phelan, 2001; Van Brakel, Peters, et al., 2019). Culture can profoundly shape stigma experiences, and, in turn, impact the way we understand and assess stigma. For example, whereas reduced work participation may be felt as the most harmful impact of stigma in one cultural context, having children for whom to provide and maintaining social relationships may be core achievements in other contexts (Misra et al., 2021; Yang et al., 2007; Yang, Chen, et al., 2014). Health-related stigma is a complex global health issue with a dynamic nature and, as complex as it is, can be even more complex to measure.

A wide variety of tools have been developed to measure different facets of health-related stigma, focusing on diverse health conditions like HIV (Berger et al., 2001) or leprosy (Dadun et al., 2017); stigma types (Stevelink, Wu, et al., 2012; van Brakel, 2006); and populations, such as affected persons, community members (Bogardus, 1933; Weiss et al., 1992), children and adolescents (de Groot et al., 2020; Gavan et al., 2022), or healthcare workers (HCWs) (Srinivas et al., 2018; Wouters et al., 2017). The majority of existing measures are disease-specific, but previous research has demonstrated that some measures are suitable across various conditions (van Brakel, 2006; Van Brakel, Cataldo, et al., 2019). Measurement tools are important for understanding the stigma process, capturing the degree of stigma in a certain context, evaluating changes in stigma over time, and assessing the effectiveness of stigma reduction interventions (Heijnders & Meij, 2006; van Brakel, 2006).

Despite the advances in stigma research over the past years, measuring health-related stigma remains complex and challenges and knowledge gaps remain. For example, self-reported measures can induce social desirability bias (Smith et al., 2020), assessing intersectional stigma (existence of multiple stigmatised identities) remains difficult due to the interrelations between stigmas (Fox et al., 2018; Turan et al., 2019), and some measures lack proper psychometric validation (Fox et al., 2018; Gavan et al., 2022; Stevelink, Wu, et al., 2012). Furthermore, many stigma measures have been developed in North America and Europe, and may not be able to capture local stigma dynamics in other contexts (Link et al., 2004; Yang, Thornicroft, et al., 2014). Besides these examples, more challenges and gaps persist (e.g. selecting the appropriate measure; the complexity of cross-cultural validation). This can complicate measurement approaches taken by stigma researchers, thereby making their perspectives crucial in gaining more understanding of current shortcomings and to better understand how we can improve the way we measure.

Flake and Fried (2020) explored questionable measurement practices in psychological science by recentring the researcher and their decisions that impact the validity of measurements and the results, and thus the validity of the conclusions of the study. While previous studies evaluating the state of stigma measurement have been primarily based on literature reviews (Fox et al., 2018; Link et al., 2004; Nyblade, 2006; Relf et al., 2021; Turan et al., 2019; van Brakel, 2006), few studies focus on researchers in evaluating health-related stigma measurement approaches. The current study takes a different approach by examining stigma measurement approaches from the perspective of stigma researchers. In doing so, the perspectives and experiences of researchers can be taken into account, while taking a non-siloed perspective on health-related stigma to stimulate more uniformity and cross-disciplinary collaboration in the stigma field, and further promote generic measurement approaches (Van Brakel, Cataldo, et al., 2019) that can facilitate cross-comparisons and the development of common stigma reduction interventions. Thus, this study aimed to gain insight into challenges and research priorities related to the assessment of health-related stigma, by exploring the experiences of stigma researchers and their perspectives on existing measurement approaches and research priorities.

Methods

Study design

This study followed an exploratory mixed-methods design. Semi-structured interviews were conducted to explore challenges and research priorities associated with health-related stigma measurements. Based on the identified research opportunities, a short questionnaire was developed, aiming to further examine the perspectives of a larger group of stigma researchers and to investigate which research priorities were perceived as the most relevant to address.

Study population

The target population for both the interviews and the survey were purposively selected, which included researchers in the field of health-related stigma with diverse backgrounds (e.g. working on diverse conditions and in diverse contexts and cultures), who were eligible if they: (1) had experience working on health-related stigma; (2) validated or used a health-related stigma measure; (3) were aged ≥ 18; (4) had access to a computer/smartphone with internet connection, (5) were able to speak English.

Study participants were identified via professional networks of co-authors as well as via scoping peer-reviewed literature (published 2011-2021) on health-related stigma measurement broadly. Corresponding authors of relevant published studies, focusing on the use or validation of health-related stigma measures, were contacted for either an interview or the survey. Stigma researchers were prioritised for an interview if the author was assumed to have high expertise on the topic, if the study was recently published, or if it focused on the cross-cultural adaptation of a measure. We aimed to recruit interviewees working on diverse conditions and in diverse cultural contexts, to ensure applicability of findings to a broad range of settings. Interviewees were also asked to share contact references of colleagues working in the stigma field. Other authors of selected studies, and additional contacts obtained from interviewees, were contacted for the questionnaire. Interviewees were also invited to complete the questionnaire, as we wanted to include their perspectives on the priorities.

Semi-structured interviews

Fourteen semi-structured interviews were conducted with stigma researchers in order to gain an in-depth insight into their experiences and perspectives on challenges and research priorities related to stigma measurements. Interviews were conducted by the first author (MM) via an online video platform and recorded with the interviewee’s permission. Interviewees were not compensated for their participation. Interview questions were guided by a cross-cutting framework on the construct of health-related stigma (Stangl et al., 2019), a theory on instruments’ measurement properties (Terwee et al., 2007), and a model on cross-cultural equivalence (Herdman et al., 1998; Stevelink & van Brakel, 2013). The resulting interview guide consisted of the following main categories: (1) General questions; (2) Measuring different aspects of the stigma process, at different socio-ecological levels and across conditions; (3) Measurement properties; and (4) The role of culture in measuring stigma.

After the interview, a summary of the interview was sent to the interviewee for a member check. Interviews were transcribed and transcripts were coded by the first author (MM) using ATLAS.ti (Version 9). A thematic analysis approach was used to code the qualitative data. All identified challenges and research priorities were thematically coded, and coding patterns, frequently occurring themes and connections between themes were explored. A codebook was developed after analysing the first few interviews, which was further refined while analysing the remaining interviews and complemented with new emerging codes. Qualitative content analysis was performed wherein the first author reviewed all codes and classified them into overarching categories and related subcategories. These themes were based on the theories used to guide the interviews (as described above), as well as new emerging themes. Upon this review core themes emerged, which were discussed with one of the co-authors (RP), and key quotes were extracted. The qualitative analysis resulted in defining core research priorities. Despite being close to data saturation, complete saturation was not reached as a few new research priorities were identified during the final interviews. Changes were made to the questionnaire because of this (see the next paragraph).

Questionnaire

The interview results were used to develop a short online questionnaire (Appendix) (Qualtrics, Provo, UT; distributed via e-mail to the respondents) to broaden the reach among researchers and diversify the pool of study participants. This questionnaire was used to prioritise the research recommendations previously identified through interviews based on their level of significance to researchers. As complete data saturation was not reached in the interview phase, the new respondents (i.e. not the interviewees) were first asked to share their ideas for further research. These responses were analysed to see whether they corresponded to the interview findings. Furthermore, all respondents were asked to rank the research priorities that were identified through the interviews according to their perceived relevance. The priorities identified in the interviews were categorised into two ranking questions based on the following groups: priorities for further research (16) and priorities for the development of practical tools that can support measurements (4). The priorities were ranked according to their relevance, with low scores indicating that respondents perceived it as more relevant to address (1 = most relevant). The responses were analysed using descriptive statistics (minimum, maximum, mean, standard deviations) (IBM SPSS Statistics, Version 27.0).

Ethical considerations

An online self-check for research ethics, managed by the university’s research ethics review committee (BETCHIE, Vrije Universiteit Amsterdam), was completed prior to the study. An exemption for full ethical review was given. Written informed consent was obtained from interviewees, personal information in the transcribed interviews was removed, and the questionnaire was anonymous. Data was stored on a secure server managed by the university, in a protected folder with password that could only be accessed by the authors of this manuscript.

Results

Characteristics of the participants

The demographics of the interviewees (N = 14) and respondents of the questionnaire (N = 36; 10 interviewees and 26 other respondents) are displayed in Table 1. The participants had diverse backgrounds, and worked on stigma related to infectious diseases (e.g. Neglected Tropical Diseases (NTDs), HIV/AIDS) as well as non-communicable diseases (e.g. mental health). They were located in different parts of the world, although the majority was located in Europe and North America.

Table 1. Characteristics of interviewees and questionnaire respondents.

	Interviewees (N = 14)	Questionnaire respondents (N = 36)
Gender
Male	4	7 (19.4%)
Female	10	28 (77.8%)
Prefer not to say	–	1 (2.8%)
Location
Europe	7	16 (44.4%)
North America	4	12 (33.3%)
Africa	2	3 (8.3%)
Asia	1	3 (8.3%)
South America	–	1 (2.8%)
Australia	–	1 (2.8%)
Age
25–34	–	8 (22.2%)
35–44	–	16 (44.4%)
45–54	–	4 (11.1%)
55–64	–	7 (19.4%)
Prefer not to say	–	1 (2.8%)
Organisation
NGO	4	–
Academic institution	10	–
Health topic(s)*
NTDs	7	12 (33.3%)
HIV	7	19 (52.8%)
Tuberculosis	3	7 (19.4%)
Mental health	6	13 (36.1%)
Overweight/obesity	–	2 (5.6%)
Cancer	–	2 (5.6%)
Other	7	8 (22.2%)

Note: NGO = Non-governmental organisation; NTDs = Neglected Tropical Diseases; HIV = Human Immunodeficiency Virus.

*Many participants worked on multiple health topics.

Priorities for further research

A variety of priorities for further research were identified in the interviews. An overview of research priorities was established by integrating the results of the interviews and questionnaire (Table 2). These research priorities, along with identified challenges, will be presented in order of importance as ranked by the respondents of the questionnaire.

Table 2. List of research priorities that resulted from the questionnaire (N = 36).

	Research priorities	Minimum	Maximum	Mean	SD
1	Refine measures using recent conceptualisations to ensure content validity	1	12	4.44	3.00
2	Assess the responsiveness of measures by comparing scores to an external criterion	1	12	5.22	3.52
3	Investigate how we can capture subtle changes in stigma levels	1	16	6.39	4.16
4	Validation studies of measures across cultures using datasets	1	16	6.56	4.03
5	Investigate how we can capture actual behaviour and discrimination	1	15	6.92	3.68
6*	Select the most essential aspects of the cross-cultural validation process	1	14	6.97	2.96
7*	Investigate how we can study and measure multi-level intersectional stigma	1	16	6.97	5.16
8	Investigate how we can combine quantitative and qualitative methods in one stigma measure	1	15	7.50	4.44
9	Reformulate negatively phrased stigma items to make them more positive	2	16	8.81	4.26
10	Investigate how we can use more experimental studies (e.g. the IAT)	1	16	10.00	4.34
11	Investigate how we can incorporate visual graphics in measures	2	16	10.17	3.97
12*	Generic measure: compare conditions to find common and condition-specific elements	1	16	10.75	4.20
13*	Investigate how we can include more positive features that counter stigma in measurements (e.g. resilience)	1	16	10.75	4.42
14	Develop new or improve existing stigma measures that target HCWs	1	16	11.00	4.41
15	Investigate how we can include the effect of gender in existing measures	4	16	11.53	3.48
16	Develop more stigma measures that target close friends or family members specifically	3	16	12.03	3.93

Notes: Minimum (lowest ranking score), maximum (highest ranking score), mean and standard deviation (SD) of 16 opportunities for research studies ranked by the respondents. The answers are ordered by the mean (low to high); a lower score indicates that respondents perceived it as more relevant to address (1 = most relevant; 16 = least relevant). IAT = Implicit Association Test; HCWs = health care workers.

*In case of equal means, answers were ordered by the SD (low to high).

Priorities 1–4: The need for enhanced measurement properties

Four research priorities that were perceived as the most relevant by the respondents of the questionnaire related to the measurement properties of stigma measures. The first, most prioritised, opportunity was to use recent conceptualisations of stigma to compare and refine existing measures to ensure their content validity. During the interviews, three researchers argued that some measures are not supported by a definition or conceptual framework, do not capture all the relevant stigma dimensions, or contain items that are not categorised within the correct domain. The second and third research priorities were focused on the assessment of changes in stigma levels. One interviewee pointed out the lack of studies investigating the responsiveness of measures, and suggested that potential future responsiveness studies should compare stigma scores with external criteria to draw conclusions about the measured changes. Additionally, two other interviewees argued that stigma scales are not sensitive enough to capture subtle changes in stigma levels, and recommended to investigate ways to capture this. The measurement properties were also highlighted in the fourth priority, which focused on the validation of measures across datasets. This was suggested by one interviewee, who explained that validating frequently used measures, such as the Explanatory Model Interview Catalogue (EMIC), across datasets collected in diverse settings could strengthen the validity and reliability of these tools.

Priority 5: Measurement of differential treatment

Another research priority that was considered important by the respondents of the questionnaire related to the assessment of actual behaviour and discrimination. Three interviewees indicated that we currently lack the ability to directly measure differential treatment, instances of discrimination and actual behaviour, as existing stigma measures focus on attitudes, or they assess enacted stigma in an indirect manner by measuring reported behaviour. One of them explained the complexity of measuring discrimination:

(…), you could ask persons affected by leprosy: ‘Do you know someone who has gone through a divorce because of leprosy?’ and people say ‘Yes, I have heard about that’, then you never know whether everyone has heard about the same person in that community, whether that happened 10 years ago, or whether that happens frequently. This problem has led to the fact that it is, as far as I know, still not possible to measure discrimination or instances of discrimination directly. (Interviewee 3, male, mainly working on NTDs)

Priority 6: Cross-cultural validation: Challenges and opportunities

The sixth research priority was related to the process of cross-cultural validation, as one interviewee suggested to host a Delphi panel to select the most essential aspects of this process to address in instances where researchers are short of resources. The importance and complexity of cross-cultural validation was observed in the stories of several interviewees, who encountered various difficulties when using measures across settings. Interviewees provided examples of questions which were not applicable in certain cultures, including the item ‘would you rent a room to this person?’, questions related to sharing food, or work-related items. Additionally, translation of measures to another language posed challenges, as it was sometimes difficult to retain the meaning of certain concepts, response categories or health conditions. For example, one interviewee reported:

… there were statements [in the scale] that ‘some patients feel lonely’, when you translate that in Swahili, I would have to put the explanation that they avoid other people, but once I put the avoid other people, I found another statement that actually is asking ‘some TB patients avoid others’, while in my translation, I meant feeling lonely is actually avoiding others. (Interviewee 5, female, working on TB)

Other cross-cultural challenges were related to the format of stigma measures. Interviewees reported difficulties with interviewer-administrated first-person statements, as respondents did not understand whether the statement referred to the interviewee or interviewer, as well as with various response scales. One interviewee reported to deal with confusion regarding response options like ‘probably’, ‘uncertain’, and ‘don’t know’, while another interviewee’s target population had difficulties with comprehending frequency Likert scales (‘always’, ‘often’, etc.).

Priority 7: Complexity in measuring intersectional stigma

The ongoing challenge of understanding and measuring intersectional stigma was recognised by many interviewees. The next research opportunity was related to this complex topic, and focused on measuring intersectional stigma across socio-ecological levels. One interviewee stressed intersectional stigma research to have been mainly focused on the individual level, and suggested to further study multi-level intersectional stigma and associated assessment methods:

We’ve done a lot of work in, you know, multidimensional scales over the past decade, and we've done a lot of work in structural stigma and measures for structural stigma. (…) I think that … that is a next challenge for folks, just to think about how do we study this multidimensional aspect of stigma in a multi-level way. (Interviewee 10, female, mainly working on HIV)

Priorities 8–9: Qualitative approaches and phrasing of scale items

Many interviewees stressed the importance of using qualitative methods and involving the target population in stigma research, for example by engaging affected individuals in scale development and adaptation processes. The eighth research priority, as unpacked by two interviewees, was to investigate the possibility of developing a mixed-methods stigma measure. One of the suggested ways to do this was to conduct a validation study of the original EMIC for affected persons. This scale alternates quantitative and qualitative items (each quantitative item is followed by asking ‘why’), thereby providing explanations to why someone has given a specific response. Since the qualitative responses differ across respondents, these might exert an unpredictable influence on the response to the subsequent quantitative question and ultimately the stigma score. These effects could be studied in a validation study.

The ninth research priority related to the reformulation of negatively phrased items in stigma measures. Several interviewees expressed concerns about the negative and sometimes offending phrasing of questions (e.g. ‘do people consider affected people disgusting?’), as this may increase the burden on respondents and administrators and reinforce stigma. Researchers working on TB argued that TB scales are less upsetting due to the third-person phrasing (e.g. ‘some people who have TB … ’).

Priorities 10–11: Alternative approaches to stigma assessment

The tenth research priority was to investigate how we can use more experimental study designs to assess stigma. One interviewee explained this may provide a new approach of studying stigma and overcome limitations of other approaches. This interviewee illustrated this opportunity with the example of the Implicit Association Test (IAT), which can capture implicit stigma and overcome social desirability bias. The eleventh research priority was to further investigate how we can incorporate visual graphics in existing stigma measures (e.g. vignettes with graphics or emoticons in response options).

Priority 12: Generic stigma measures

The twelfth research priority focused on the comparisons of diverse conditions for the development of generic measurement approaches. Interviewees’ opinions differed as to whether measures could be used across contexts, as some (N = 5) stressed the advantages of comparing across settings and reducing the resources needed for scale validation, while two others had their doubts about a standardised ‘one size fits all’ approach. Concerns were expressed regarding capturing detailed information and condition- or culture-specific aspects with generic approaches, but several researchers suggested that a menu with add-ons could tackle this. Two interviewees suggested future studies may compare and categorise various conditions in diverse cultures, for example by using Jones et al.’s (1984) dimensions and Pachankis et al.’s (2018) classification or by contrasting drivers, facilitators, manifestations and outcomes. Two other interviewees recommended conducting qualitative research to investigate what stigma features are relevant to different groups of persons affected:

It could be then interesting to, again, show them [potential generic measures] to participants, and say: which ones do you think are the core elements of being HIV positive, or experiences of being HIV positive? What are the core elements of being a cancer patient? And then maybe cross checking all of these and say: can we really find a common element? (Interviewee 11, male, working on HIV and mental health)

Priority 13: Focus on countering negative outcomes

The thirteenth research priority was to further investigate how we can increase our focus on outcomes countering the negative effects of stigma (e.g. empowerment-related features, resilience, interpersonal support) and incorporate these in measurement approaches. This suggestion was addressed by two interviewees, who expressed stigma measurements tend to only focus on deficits and negative outcomes of stigma.

Priorities 14–16: Enhancing measurements for specific target groups

The last, least prioritised, research priorities were related to the assessment of stigma among specific target groups, viz. to develop new or improve existing HCWs measures to ensure proper psychometric properties; to investigate how we can incorporate gender-sensitive effects on stigma in existing measures, as stigma experiences may differ between men and women; and, to develop specific measures for the interpersonal level. This latter opportunity was reported by one interviewee, who explained that the EMIC Community Stigma Scale (EMIC-CSS) may be less suitable for individuals’ close friends and family members. This interviewee recommended to have a separate measure targeting this level specifically.

Priorities for the development of supportive practical tools

The interview findings also highlighted four suggestions for the development of practical tools to support researchers with stigma measurements. These were subsequently ranked by the respondents of the questionnaire (Table 3). The opportunity considered the most relevant was to develop new or improve existing toolkits to be used across conditions. Two interviewees explained these toolkits could stimulate comparisons across contexts, and allow the assessment of other aspects associated with stigma (e.g. barriers to healthcare). The second suggestion for practical tools engaged with support in selecting an appropriate measure. This appeared to be challenging for some interviewees, due to the overabundance of measures and the complexity of finding a scale that captures what researchers set out to capture. To address this issue, two interviewees suggested to develop an online database or digital application containing well validated and frequently used measures and their translations. This was followed by a recommendation to develop an online platform for support regarding adapting measures to the local context and phrasing and classifying items, which was reported by one interviewee:

Some instructions on scales in general, like how to write good versus bad questions. And then also, around like, this is what counts as anticipated stigma and this is what counts as internalized stigma. (…) You can envision something where there is materials online, like a toolkit, but then there’s also a human, or maybe a group of humans involved. (Interviewee 10, female, mainly working on HIV)

The last recommendation for practical tools was focused on the (further) development of extensive training toolkits. This was suggested by one interviewee, who explained that training materials regarding scale administration and prompting could increase the reliability and validity of studies.

Table 3. Results of ranking of 4 suggestions for practical tools (N = 32; four respondents did not complete this question).

	Practical tools	Minimum	Maximum	Mean	SD
1	Develop new or improve existing toolkits that can be used across conditions	1	4	1.81	1.15
2	Develop a digital application or database containing frequently used measures and translations	1	4	2.50	1.05
3	Develop an online platform supported by a team of experts, for support on how to adapt measures and formulate and classify items	1	4	2.81	1.00
4	Develop training toolkits on scale administration to ensure consistency across studies	1	4	2.87	1.00

Notes: Minimum (lowest ranking score), maximum (highest ranking score), mean and standard deviation (SD) of the responses. The answers are ordered by the mean (low to high), a lower score indicates that respondents perceived it as more relevant to address (1 = most relevant; 4 = least relevant).

Discussion

This mixed-methods study explored the current state of the field of health-related stigma measurements and provided recommendations for the way forward. Although progress has been made, the substantial number of research priorities identified in this study highlights the great amount of work that is yet to be done. The priorities identified in this study were related to: robust measurement properties; capturing subtle changes in stigma; assessing actual behaviour; the cross-cultural validation process; multi-level intersectional stigma; increased focus on positive features that counter stigma; changing the negative phrasing of scale items; mixed-methods measurement approaches; experimental study designs; visual graphics in measurements; generic measures; measurements for specific target groups, and; improved practical support for stigma researchers.

First of all, the results of this study underlined the need to refine existing stigma measures. Scales may only capture the complexity of stigma to a limited extent, and are not always an accurate reflection of stigma theories (Gavan et al., 2022). Comparing and refining existing measures may be useful, as our understanding of stigma has grown extensively and recent conceptualisations and frameworks (Stangl et al., 2019) may expose shortcomings in measures. Another identified research priority was to strengthen the validity and reliability of measures by reviewing previously collected data across settings. Similar studies have been conducted for the Internalised Stigma of Mental Illness (ISMI) scale (Boyd et al., 2014) and the Participation-scale (Stevelink, Hoekstra, et al., 2012), and future studies may focus on other scales that have been shown to be suitable across conditions, such as the EMIC or Social Distance Scale.

Additionally, the results indicated that further research should investigate how we can capture subtle, but meaningful, changes in stigma and assess the responsiveness of measures. This is in line with reviews showing that the assessment of responsiveness is often lacking (Brohan et al., 2010; Sastre-Rus et al., 2019; Stevelink, Wu, et al., 2012; Wei et al., 2017). The work of Dale et al. (2021), who used ecological monetary assessments (EMA; measurement of real-time experiences and feelings) combined with microaggression measures, presents a promising approach to capture daily changes in subtle forms of stigma. In addition, Moore et al. (2016) demonstrated that EMA was more sensitive to change when assessing mindfulness and depression compared to traditional measures. Subtle forms of stigma, such as microaggressions, are often not captured by the traditional stigma scales (Eaton et al., 2020), while these can be harmful due to their cumulative nature (Sue, 2010). Whereas explicit forms of stigma may have become less socially acceptable, deeper or more subtle forms may continue to harm individuals (Stier & Hinshaw, 2007; Young et al., 2019). As scholars have also called for more longitudinal stigma research (Fox et al., 2018; Kane et al., 2019; Katz et al., 2013; Pantelic et al., 2015; Rai, Syurina, et al., 2020), future longitudinal studies could compare the responsiveness of traditional scales to other approaches (e.g. microaggression measures, EMA, qualitative methods, observations) to investigate their sensitivity to capture (subtle) changes in stigma.

Furthermore, this study found a limitation within current stigma measures, namely their inadequate capability to measure actual behaviour and discrimination. Nyblade (2006) stressed that individuals are reluctant to admit they are engaging in stigmatising behaviour. Additionally, surveys measuring witnessed enacted stigma may not reflect the actual situation, as responses may depend on the respondent’s interpretation of discriminatory behaviour and the size of one’s social network (Stahlman et al., 2017; Straetemans et al., 2017). As traditional retrospective questionnaires are unable to assess actual behaviour, other approaches are needed. A promising approach in this field is the work of Varas-Díaz et al. (2017, 2019), who analysed stigmatising behaviour in clinical settings using simulations and observation techniques through an experimental study design. However, even though studying actual discriminatory acts is important, just the perception of discriminatory acts or the possibility of being treated differently in the future may be enough for individuals to perceive stigma, with or without the actual act of discrimination. Since these perceptions of stigma may affect one’s (mental) wellbeing, these are just as important to capture as actual experiences of stigma.

Further, measurement approaches targeting one disease or characteristic in isolation neglect the interactions of different stigmas (Bauer, 2014). Focusing on one’s entire identity is crucial if we want to gain a complete understanding of one’s lived experience. As illustrated in a study by Rai, Peters, et al. (2020), the intersectional experience of health-related stigma manifests across all socio-ecological levels. This supports the research priority found in this study, namely to further study multi-level intersectional stigma and ways to assess this. Turan et al. (2019) provided an overview of different measurement approaches for capturing intersectionality, among which multilevel modelling. This modelling approach can capture associations between manifestations of stigmas at different levels, such as internalised stigma at the individual level, experiences of discrimination at the interpersonal level, or stigmatising laws or policies at the macrolevel (Earnshaw et al., 2022; Turan et al., 2019). Future research may further investigate how to apply these methods to study interconnections between intersecting stigmas across levels. This could support the implementation and evaluation of multi-level interventions, which may effectively reduce stigma and promote resilience (Gottert et al., 2020)

The challenges with the cross-cultural use of stigma measures that were found in this study highlight the significant influence that culture has on stigma assessment. Many measures have been developed in North America and Europe (Yang, Thornicroft, et al., 2014), and conceptualisations of constructs, the applicability of items and manifestations of stigma vary across cultures (Herdman et al., 1998; Koschorke et al., 2017). Yet, studies have shown that this cross-cultural validation process is often insufficiently addressed (Bowden & Fox-Rushby, 2003; Gavan et al., 2022; Stevelink & van Brakel, 2013). Considering this significant influence of culture on stigma, scholars have engaged with a new culture-sensitive approach by incorporating local cultural mechanisms in stigma measurements (Yang, Chen, et al., 2014; Yang et al., 2021).

Moreover, this study underscored the importance of engaging and empowering the target population in stigma research. Participatory and qualitative research methods can be a way to reduce power imbalances, promote community engagement, and trigger changes through leadership and agency within the community (Sprague et al., 2019; Stutterheim & Ratcliffe, 2021). Further, the results indicated that there is a need to reduce the negative focus within stigma research. The predominant negative phrasing of scale items may reinforce stigma perceived and internalised by persons affected, as well as, when targeting community members or HCWs, stigmatising attitudes and prejudice. As pointed out by Reeves et al. (2021), focusing solely on the negative aspects of stigma induces paternalistic attitudes, whereas incorporating resilience-related features may lead to more respectful and comprehensive views of individuals. Although scales for the assessment of diverse strength-based features (e.g. resilience, empowerment, social support) exist (Bakker & van Brakel, 2012; Gjesfjeld et al., 2008; Windle et al., 2011), literature on stigma is commonly focused on the harmful impact of stigma (Shih, 2004). For example, a review by Logie et al. (2021) showed that less than half of the quantitative studies on intersectional stigma considered empowerment-based factors. While it is important to understand the negative consequences of stigmas, this can provide a limited view of one’s experience when an individual becomes more empowered through successful coping strategies (Shih, 2004). Peters et al. (2014) showed affected individuals may use different sources to cope with the negative effects of stigma, such as interpersonal support and self-confidence. Researchers may place more emphasis on these positive factors, as resilience resources can mitigate the detrimental outcomes of stigma and may therefore be promising intervention targets (Dulin et al., 2018; Earnshaw et al., 2013; Rai et al., 2022).

Based on the current study’s findings and supported by the literature discussed above, we suggest several important recommendations for research to advance the field and enhance measurement approaches (Table 4).

Table 4. Recommendations for further research to improve measurement approaches.

Research suggestions to be addressed in future studies
Measurement properties	Evaluate and refine the content of frequently used measures using recently developed conceptualisations Assess the robustness of frequently used measures across settings
Changes in stigma	Longitudinal research to investigate the sensitivity of stigma measures to capture (subtle) changes in stigma and compare this to other measurement approaches
Actual behaviour	Further research on ways to capture actual behaviour and discrimination
Generic measurement approaches	Further research on the development and application of generic measurement approaches, by comparing and contrasting stigma features across conditions
Cultural sensitivity	Further) development and application of culture-sensitive measurement approaches Evaluation of the most important steps of the cross-cultural validation process to facilitate increased cultural adaptation of measures
Reduce the negative focus of stigma research	Development of neutrally or positively phrased stigma measures, by reformulating negatively phrased scale items More research focusing on positive features that counter stigma
Intersectional stigma	Further research on the assessment of multi-level intersectional stigma, e.g. through multilevel modelling

For the measurement of many psychological constructs, including stigma, there is generally not a standard universally accepted measure used by all researchers (Flake & Fried, 2020). Reviews on stigma measurement approaches (Brohan et al., 2010; Ferguson et al., 2022; Fox et al., 2018; Gavan et al., 2022; Relf et al., 2021; Stevelink, Wu, et al., 2012; van Brakel, 2006) reveal the great amount of stigma scales that are being used in the field, and new measures are still being developed. For example, in Fox and colleagues’ review on mental illness stigma (2018), they identified over 400 stigma measures, of which two-thirds were lacking proper psychometric validation. The use of such a variety of measurement approaches can complicate the selection of appropriate tools and the comparability of results across settings. Additionally, transparent reporting of measurement approaches is often lacking in scientific papers, while this is important for the reader to evaluate the study’s findings (Flake & Fried, 2020). Meier (2023) argued that researchers measuring psychological phenomena should evaluate any potential measurement issues and assess how this may have impacted the findings. The current study also revealed the need for more practical support in the implementation process of stigma measures, as interviewees acknowledged that people involved in implementing and administering stigma measures may affect the study results. Increased transparency about methodological decisions and rationales can promote learnings among stigma researchers across different fields, and thereby be a step in the right direction in improving measurement practices.

Strengths and limitations

Major strengths of this study were its mixed-methods approach, the in-depth perspectives of researchers, and the inclusion of researchers working on a diverse range of conditions. There are a few limitations to consider. First, stigma researchers may have different priorities for future research than other stakeholders, such as affected individuals. Second, the study methods lacked a consistently systematic approach to select study participants, thereby affecting the representability of the stigma field and generalisability of findings. The majority of the participants were located in Europe and North America whereas other continents were underrepresented, which has influenced the perspectives on research priorities. In addition, we did not specially enquire about the participants’ years of experience in this research area, so sufficient expertise cannot be ensured. There was no complete data saturation after fourteen interviews. We tried to accommodate for this by asking the respondents of the questionnaire for their research suggestions, through which we identified more new research priorities (e.g. validating measures for particular groups like children/adolescents), thereby confirming the lack of data saturation. Therefore, it is probable that there are more persisting knowledge gaps and opportunities, which should be further explored in future research. Another limitation is that the open question in the questionnaire focusing on research suggestions may have been unclear or respondents may have been uncomfortable sharing their ideas, as some answers were difficult to interpret and some respondents omitted this question. Additionally, the lengthy ranking question with sixteen opportunities may have been difficult to answer, but we kept this format as the remainder of the questionnaire was relatively short.

Implications of findings and further research

This study provided an overview of research priorities based on a small study population, and thereby it laid the foundation for future research into this topic. The findings of the questionnaire showed high variation and disagreement among the respondents on ranking of the research priorities, as each research priority was both highly (very relevant) and lowly (less relevant) ranked by the different respondents. This, along with the lack of complete data saturation, confirms the need for more research to further define what is needed to advance the field. In such future studies, the focus of the study population should be shifted to include more participants from continents other than North America and Europe, as well as people affected by various conditions who have experienced (or still experience) stigma. Also, more collaboration with researchers from non-health-related stigma fields may be needed to address the intersecting stigmas that many people are facing.

Conclusion

This mixed-method study presented an in-depth exploration of the current challenges and research priorities related to health-related stigma measurements, and showed that there is much work yet to be done. Lessons can be taken from researchers’ experiences with the implementation of stigma scales, and the defined research priorities may lay the basis for and guide future studies contributing to enhanced stigma assessment approaches. This may further advance our understanding of the complex and dynamic stigma process, and hopefully our ability to prevent and address it.

Acknowledgements

We would like to thank Ms. Kim Hartog for her assistance in the design of the study, and we wish to express our gratitude to all participants of this study for their contribution.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Appendix. Questionnaire