Effects of comparative information when communicating personalized risks of treatment outcomes: an experimental study

Abstract

Despite great promise of using personalized risks of treatment outcomes during shared decision-making, patients often experience difficulty evaluating and using them. We examined the effects of providing comparative information of the average person’s risk when discussing personalized risks on people’s cognitive, emotional, and behavioural responses. Participants (n = 1,807) from a representative sample of the Dutch population received personalized risks of treatment side-effects in three different health scenarios. Participants either received only their own personalized risk statistic, or with comparative data indicating that their risk was below or above average. Furthermore, we examined whether the effects would be influenced by message format (natural frequencies with or without icon arrays) and individual differences (subjective numeracy, health literacy, and graph literacy). Providing comparative information did not influence participants’ risk perceptions, affective evaluations, nor their treatment intention. However, participants who were told that their personalized risks were above average, estimated their own risk as lower than participants who received the same personalized risks that were below average or that were without any comparative data. Message format and individual differences did not influence people’s responses to comparative data. Healthcare professionals can consider providing comparative data for helping people make sense of their personalized risks.

Keywords:

• Comparative information
• context
• personalization
• risk communication
• risk perception

Introduction

Communicating probability information about risks and benefits of health interventions or treatment options to patients is becoming an essential part of modern-day health care and shared decision-making (Bonner et al. 2021; Elwyn et al. 2017). In light of the growing emphasis on personalized medicine and open access of ‘big health data’ (Chawla and Davis 2013; Lyles et al. 2020; Schilsky 2010), there has been rapid growth in the development of personalized risk information tools for patients in different health context (Trevena et al. 2021), ranging from cardiovascular diseases to the oncology setting (Thurtle et al. 2021; Vromans et al. 2022; Tillier et al. 2021). These tools are aimed at providing patients personalized risks and health statistics that take into account personal and clinical characteristics of unique patients. These personalized risks are—compared to generic risks—perceived as more personally relevant (Vromans et al. 2020), which increases the likelihood of being better processed and understood by patients (Lustria et al. 2016; Petty and Cacioppo 1986).

Even though personalized risk may be more useful and relevant for patients, a critical problem is that they still have low evaluability: they are often unfamiliar, lack inherent meaning, and therefore make it difficult to evaluate whether they are good or bad (Zikmund-Fisher 2019). Following theory on ‘information evaluability’ (Hsee 1996; Hsee and Zhang 2010), providing context may improve evaluability of personalized risks by helping patients to derive meaning from unknown risks (Zikmund-Fisher 2019), for instance by telling patients how good or bad a 70% risk is (Waters, Fagerlin, and Zikmund-Fisher 2016; Bonner et al. 2021). Another promising contextual strategy for improving evaluability of personalized risks is by adding comparative information of the average person’s risk to the patient’s individual risk estimate. Indeed, patients often report that they want to see their own risk score for a particular outcome in comparison with scores of other patients (Vromans et al. 2021), especially for determining whether they are above or below average (Oerlemans et al. 2017). When personalized risks are presented with other data of the average risk, people are better able to interpret even unfamiliar risks because both types of risks serve as a reference for each other, which enables people to evaluate the ‘goodness’ or ‘badness’ of the risks (Waters, Fagerlin, and Zikmund-Fisher 2016).

However, a much-debated question is whether patients should be provided with comparative risks at all. To some, comparative statements should not be communicated (or at least with caution), since such data could unintentionally influence patients’ risk perceptions, affective evaluation or levels of worry, and subsequent treatment intentions (Zikmund-Fisher, Fagerlin, and Ubel 2010; Fagerlin, Zikmund-Fisher, and Ubel 2007; French et al. 2004; Klein 2003; Han et al. 2011). For instance, people who are above average risk feel more worried and compelled to take a treatment, even though this may not always be the best option based. To others, however, patients should always be informed about whether their personalized risk is above or below average, since most patients will make such comparisons on their own anyway (Schwartz 2009, 2016). In this case, providing context would be useful for correcting inaccurate beliefs and risk perceptions. To date, little is known about interpretations that people derive from comparative information and whether this may depend on what type of comparative data are communicated and in what specific health decision context (Waters, Fagerlin, and Zikmund-Fisher 2016; Schmiege, Klein, and Bryan 2010). Therefore, the first aim of this study is to examine people’s responses to comparative information when communicating personalized risks of treatment side-effects. We hypothesize that (H1) people receiving personalized risks that are above average report higher (a) risk estimates, (b) risk perceptions, and (c) affective evaluations than participants viewing the same personalized risks that are below average or without any comparative data.

Yet, the communication of comparative information may be influenced by various factors, including the way in which comparative data is communicated and individual differences (Han et al. 2011). Arguably, the most common way to communicate risks in healthcare is through numbers such as percentages or natural frequencies (e.g. ‘5 out of 100 people’), but such formats are typically misinterpreted by less literate and numerate patients. Consistent with dual coding theory (Paivio 1986), adding visual representations of statistical data such as pictographs to numbers can influence people’s understanding and perceptions of risk, and might therefore be a better strategy for communicating personalized risks with comparative data (Garcia-Retamero and Galesic 2010; Lipkus 2007; Bonner et al. 2021). However, pictographs have also been shown to have the highest affective impact through automatic associations (e.g. the color red associates with danger) compared to numbers, which in turn may impact people’s perceptions and evaluations (Chua, Yates, and Shah 2006; Timmermans, Ockhuysen-Vermey, and Henneman 2008; Visschers et al. 2012). Especially in a situation when a patient’s personalized risk is above average, communicating this both numerically and visually may cause people to be unintentionally worried about the information. As such, the second aim of this study was to determine whether people’s responses to comparative information might be influenced by the message format. We hypothesize that (H2) people receiving personalized risks presented in a numerical + visual format will show higher (a) risk estimates, (b) risk perceptions, and (c) affective evaluations than participants viewing the same personalized risks presented in a numerical-only format, especially when people receive personalized risks that are above average (H3).

Finally, as an exploratory objective, we also wanted to examine the extent to which individual differences might influence people’s responses to comparative risk data. Indeed, a substantial body of research suggest that people may differ in their perceptions and interpretations of the same risk information. For instance, females tend to have higher risk perceptions than men for a wide range of risks (Slovic 1999; Xiao and McCright 2015; van der Linden 2015). In addition to gender, age (being older) and education (having limited formal education) also tend to be associated with increased risk perceptions (van der Linden 2015; Akerlof et al. 2013). Similarly, in all segments of society, people who are less numerate, health literate, or graph literate typically have higher risk perceptions and/or problems interpreting multiple risks compared to their counterparts (Peters 2020; Galesic and Garcia-Retamero 2011; van der Heide et al. 2013; Nelson et al. 2008). Therefore, a third aim of our study was to explore whether the effects differ for people with different sociodemographics, subjective numeracy, health literacy, and graph literacy skills. The study design, hypotheses, exploratory objectives, and analysis plan were pre-registered prior to data collection (https://tiu.nu/osf/q2dcz).

Materials and methods

Study design and sampling

In a large-scale experiment, participants were presented with three health scenarios and personalized risk of treatment side-effects, for which we adopted a 3 (comparative information: personalized-risk only, personalized risk above average, personalized risk below average) × 2 (message format: numerical-only, numerical + visual) between-subjects design. Ethical approval was granted by the Research Ethics and Data Management Committee of the Tilburg School of Humanities and Digital Sciences (ID.REDC.2019.26a).

A representative sample of the Dutch population (age ≥ 16) was recruited through CentERdata’s Longitudinal Internet Studies for the Social Sciences panel. This panel consists of 5,000 households in the Netherlands, comprising approximately 7,500 individuals, and represents a true probability sample of households drawn from the population register by Statistics Netherlands (Scherpenzeel 2011). Panel members complete online questionnaires every month for which they receive financial compensation. As part of the pre-registration, power calculations were conducted to determine our sample size using G*Power 3.1.9.2 (Faul et al. 2007). To detect a small effect for the primary measures with a 3 × 2 between-subject design, a sample of 1269 participants was calculated to be needed for the experiment (power = 0.9, α = .05). Given that we were interested in exploring individual difference measures, we aimed for a total of 1800 participants (300 per experimental condition).

Materials

Health scenarios and personalized risks

Participants read three health scenarios and treatment information in a randomized order: chemotherapy for advanced colon cancer, medication for increased cholesterol levels, and surgery for skin cancer. We selected these scenarios to represent various diseases, treatment options, and associated side-effects, as well as for variation in the actual probability estimate of experiencing side-effects. In each scenario, participants were asked to imagine being diagnosed with that specific disease and that they were given a website from their doctor including information about treatment and its associated side-effects. Participants received personalized risks of experiencing a treatment side-effect, which was always a fictitious number and held constant for each participant across experimental conditions. Participants were informed that their risks were personalized based on their reported gender, age, and specific information about the disease as stated in the scenario. For colon cancer, the risk of experiencing neuropathy after chemotherapy was 38%. For increased cholesterol, the risk of gastrointestinal complaints after medication was 25%. For skin cancer, the risk of inflammation of the skin surrounding the wound after surgery was 17%.

Comparative information

Participants either received their personalized risk without comparative data (personalized risk-only), or they received their personalized risk together with a lower generic risk (personalized risk above average) or together with a higher generic risk (personalized risk below average). A generic risk was defined as the risk for all people of all ages diagnosed with all forms of colon/skin cancer or all cholesterol levels. We varied the absolute risk difference between the personalized and generic risks in the three health scenarios. These absolute risk differences were arbitrarily chosen, and were ± 20% for the colon cancer scenario, ± 15% for the increased cholesterol scenario, and ± 10% for the skin cancer scenario.

Message format

Half of the participants received the risk in numbers (numerical-only format) using natural frequencies. The other half received the risk in a combination of natural frequencies and visual aids using icon arrays (numerical + visual format). As recommended by Zikmund-Fisher et al. (2014), the icons included a matrix of 100 restroom icons displayed in different colors to represent the number of individuals with (red) or without (grey) experiencing the side-effect (Figure 1).

Figure 1. Structure of stimulus material within the 3 [comparative risk information: personalized risk-only, personalized risk above average, and personalized risk below average] × 2 [format: numerical-only, numerical + visual) between-subject design, shown for male participants receiving risks about neuropathy after chemotherapy for colon cancer. Participants either received their personalized risk without comparative data, or they received their personalized risk together with a lower generic risk or together with a higher generic risk.

Measures

After participants had read the risk information, we measured their risk estimate, risk perception, and affective evaluation (primary measures) and their perceived personal relevance and treatment intention (secondary measure). The risk information was always visible when participants were answering the questions, to ensure that we tapped into their interpretations and assessment rather than their memory. This method was repeated for each health scenario, after which we assessed the individual difference measures, background questions, and the participants’ need for personalized and comparative data.

Primary measures

Risk estimates was measured using the question ‘What do you think is the probability you will experience this side-effect’ (measured as a percentage between 0 and 100) (Büchter et al. 2014; Vromans et al. 2020). Risk perception was measured with two items (‘How likely do you think you would be to experience this side-effect after treatment?’ and ‘How big do you think is the chance that you would be to experience this side-effect after treatment?’, Cronbach’s α = .85) measured on 5-point scales, with 1 as ‘not likely at all’/‘not big’ and 5 as ‘very likely’/‘very big’) (Büchter et al. 2014; Vromans et al. 2020). Affective evaluation of the risks was measured with three items (‘How frightening/worrisome/serious do you think the information about this side-effect was?’, Cronbach’s α = .90) measured on a 5-point scale, with 1 as ‘not frightening’/‘not worrisome’/‘not serious’ and 5 as ‘very frightening’/‘very worrisome’/‘very serious’ (Timmermans, Ockhuysen-Vermey, and Henneman 2008).

Secondary measures

Perceived personal relevance was measured using two items (‘The risk information about the side-effect was made personally for me’ and ‘The way how the risk information was being presented was relevant to me’, Cronbach’s α = .85) measured on 5-point scales, with 1 as ‘strongly disagree’ and 5 as ‘strongly agree’ (Lustria et al. 2016). Treatment intention was measured using the question ‘Based on the information that you just read, how likely it is that you will choose this treatment?’ measured on a 5-point scale, with 1 as ‘not likely at all’ and 5 as ‘very likely’ (Timmermans, Ockhuysen-Vermey, and Henneman 2008).

Individual difference measures

Subjective numeracy was assessed by the Subjective Numeracy Scale (SNS; Fagerlin, Zikmund-Fisher, and Ubel 2007), which is an 8-item self-assessment for determining participants’ quantitative ability and preferences for receiving numerical information (measured on a 6-point scale, with 1 as ‘least numerate’ and 6 as ‘most numerate’, α = .87). The SNS has proven to be a valid and reliable measure, and correlates strongly with objective numeracy measures (Zikmund-Fisher et al. 2007). For the current study, we used the Dutch version of the SNS (Heringa et al. 2018; Vromans et al. 2020). The mean subjective numeracy score was determined by computing the average score of the eight items, with higher scores indicating higher numeracy skills.

Health literacy was measured with the 13-item version of the Short Assessment of Health Literacy in Dutch (SAHL-D; Maat et al. 2014). Participants were exposed to multiple choice questions in a randomized order in which they had to select the accurate meaning of health-related words. Each answer was coded as 1 (correct) or 0 (incorrect). A sum score was calculated (range: 0 − 13) with higher scores representing higher health literacy skills.

Graph literacy was assessed with the 4-item version of the Graph Literacy Scale in Dutch (Galesic and Garcia-Retamero 2011; Okan et al. 2019; van Weert et al. 2021), presenting different types of graphs and questions (two open-ended and two multiple choice, in a randomized order) about understanding the information in the graphs. Each answer was scored 1 (correct) or 0 (incorrect). A sum score was calculated (range: 0–4) with higher scores representing higher graph literacy skills.

Background measures

Demographic data (i.e. age (groups), gender and educational level) were extracted from the LISSPANEL database. We also controlled for prior (indirect) experience with the three diseases and/or treatments and/or side-effects that were used in the experiment.

Need for personalized risks and comparative data

We assessed participants’ need for receiving their personalized risks of treatment outcomes by asking ‘Imagine that you’re being informed about potential risks of a treatment. In general, to what extent do you have a need for knowing your personalized risk of experiencing treatment side-effects?’ (measured on a 4-point scale, with 1 as ‘not at all’ and 4 as ‘very much’). Participants’ need for comparative data was assessed by asking ‘Imagine that you’re being informed about your personalized risks of experiencing side-effects after treatment. Would you also want to know what your personalized risk is compared to the average risk?’ (answer options: yes always, yes but only when my personalized risk is better than average, yes but only when my personalized risk is worse than average, no I don’t want to know).

Statistical analyses

We conducted three separate two-way multivariate analyses of covariance (one for each health scenario) with risk estimates, risk perception, and affective evaluation as dependent variables, and comparative information and message format as independent variables. We controlled for age, gender, education, and individual differences (numeracy, health literacy, and graph literacy) by entering them as covariates. If applicable, significant interaction effects were further analyzed by means of simple effect analyses. We used pre-specified contrasts to assess whether personalized risks above average yielded higher levels of (a) estimated risks, (b) perceived risks and (c) affective evaluations compared to the other two comparative strategies. We further conducted separate two-way ANCOVAs, with perceived personal relevance and treatment intention as dependent variables, and comparative information and message format as independent variables, and the same demographics and individual differences as covariates.

As an exploratory analysis, we converted numeracy into a binary variable consisting of less (SNS ≤ 4) and highly (SNS ≥ 4) numerate people (Zikmund-Fisher et al. 2014), and subsequently tested its association with specific probability estimates (e.g. fifty-fifty response) using chi-square tests for the different comparative risk strategies.

Data on demographics and individual differences for the (1) respondents and non-respondents, and for the (2) six experimental conditions were compared using chi-square tests for categorical variables and ANOVAs for continuous variables. All statistical analyses were performed using SPSS version 24.0 (IBM Corporation, Somers, NY, USA). Tests were two-sided and considered statistically significant at p <. 05.

Results

Sample characteristics

Out of 2,409 people who were invited to participate, 1,807 (75%) clicked the link to launch the survey. Of those, 1753 (97%) fully completed the survey (Figure 2). Completion rates were high across experimental conditions (range = 90–99%), and only completed cases were analyzed. Fifty-four percent of the sample was female, and the mean age of participants was 53.9 years (SD = 18.3, range = 16–95 years). Overall, this sample was largely representative of the Dutch population.¹ There were no differences between respondents and non-respondents when comparing gender and education (Table 1). However, non-respondents were on average younger (ps < .001). Finally, the participants in all six experimental conditions were comparable in terms of sociodemographic characteristics, numeracy, health literacy, and graph literacy skills (all p values >. 590, Table A1, Appendix A).

Figure 2. Flowchart of the data collection process.

Table 1. Sociodemographic characteristics of respondents and non-respondents.

Characteristics	Respondents (n = 1807)		Non-respondents (n = 602)		p-value
	n	%	n	%
Gender
Female	980	54	333	55
Male	827	46	269	45	.644
Age, mean (SD)	53.9 (18.3)		40.7 (17.0)		<.001
15–24 years	142	8	104	17
25–34 years	199	11	156	26
35–44 years	213	12	124	21
45–54 years	277	15	85	14
55–64 years	356	30	68	11
>65 years	620	34	65	11	<.001
Education^a
Lower^b	472	26	130	22
Medium^c	600	33	214	36
Higher^d	729	41	256	42	.084

Note. ^a = Missing data for nine participants; ^b = Primary education, preparatory secondary vocational education; ^c = higher secondary general education or pre-university education, secondary vocational education; ^d = higher vocational education, university; SD = standard deviation.

Table A1. Sociodemographic characteristics and individual difference scores by experimental condition.

	Personalized risk-only		Personalized risk above average		Personalized risk below average		p-value
	Numerical-only (n = 299)	Numerical + visual (n = 288)	Numerical-only (n = 297)	Numerical + visual (n = 312)	Numerical-only (n = 320)	Numerical + visual (n = 291)
Gender (%)
Female	157 (53)	154 (54)	170 (57)	169 (54)	178 (56)	152 (52)
Male	142 (47)	134 (46)	127 (43)	143 (46)	142 (44)	139 (48)	.817
Age, M (SD)	52.7 (18.5)	54.2 (17.7)	54.6 (18.7)	53.4 (18.4)	55.1 (18.3)	53.5 (18.0)	.662
15–24 years	29 (10)	21 (7)	24 (8)	23 (7)	22 (7)	23 (8)
25–34 years	35 (12)	25 (9)	35 (12)	41 (13)	32 (10)	31 (11)
35–44 years	35 (12)	38 (13)	26 (9)	37 (12)	42 (13)	34 (12)
45–54 years	45 (15)	42 (15)	46 (16)	49 (15)	43 (13)	52 (18)
55–64 years	56 (18)	71 (24)	55 (18)	54 (18)	70 (22)	50 (18)
>65 years	98 (33)	91 (32)	111 (47)	108 (35)	111 (35)	101 (35)	.802
Education (%)
Lower^a	80 (27)	81 (28)	73 (25)	80 (26)	80 (25)	78 (27)
Medium^b	100 (33)	85 (30)	102 (35)	105 (34)	108 (34)	100 (35)
Higher^c	119 (40)	121 (42)	121 (40)	126 (40)	131 (41)	111 (38)	.980
Numeracy^d, M (SD)	4.01 (1.12)	4.20 (1.01)	4.13 (1.10)	4.19 (1.10)	4.09 (1.04)	4.15 (1.07)	.688
Health literacy^d, M (SD)	9.56 (2.76)	9.38 (2.82)	9.59 (2.88)	9.53 (2.63)	9.57 (2.70)	9.62 (2.97)	.935
Graph literacy^d, M (SD)	2.01 (1.06)	1.98 (1.06)	2.06 (1.01)	2.01 (1.05)	2.10 (1.05)	2.09 (1.14)	.590

Note. ^a = Primary education, preparatory secondary vocational education; ^b = higher secondary general education or pre-university education, secondary vocational education; ^c = higher vocational education, university; ^d Missing data for twenty-five patients; M = Mean; SD = standard deviation.

Effects on outcomes in the colon cancer scenario

Descriptive statistics for all outcomes in the colon cancer scenario are shown in Table 2, and all test statistics can be found in Appendix B (Table B1). There was a significant effect of comparative information on people’s estimated risks, F(2, 1741) = 8.27, p < .001, η_p² = .009. Participants who received personalized risks that were above average reported significantly lower probability estimates than participants who received the same personalized risks that were below average (p < .001) or that were without any comparative information (p = .008) (Figure 3a). However, there were no significant main effects of comparative information on perceived risk, affective evaluations, perceived personal relevance, or treatment intention (all ps > .182). Overall, these (non-)effects of comparative information did not depend on the message format (all ps > .120).

Figure 3. Risk estimates as a function of comparative risk strategy, shown for all three health scenarios.

Table 2. Participants’ mean scores on the primary and secondary outcomes in the colon cancer scenario across all experimental conditions.

Outcomes	Personalized risk-only			Personalized risk above average			Personalized risk below average
	Numerical-only	Numerical + visual	Total	Numerical-only	Numerical + visual	Total	Numerical-only	Numerical + visual	Total
Primary outcomes
Risk estimates	42.4 (17.9)	40.7 (16.4)	41.5 (17.2)	38.0 (15.5)	39.9 (17.1)	39.0 (16.3)^a***	42.8 (16.6)	43.0 (17.0)	42.9 (16.8)
Risk perception	3.04 (0.81)	3.04 (0.82)	3.04 (0.81)	3.04 (0.85)	3.11 (0.81)	3.08 (0.83)	3.09 (0.76)	3.03 (0.82)	3.06 (0.79)
Affective evaluation	3.23 (0.96)	3.16 (0.94)	3.20 (0.95)	3.22 (0.97)	3.22 (0.96)	3.22 (0.96)	3.21 (0.94)	3.28 (0.92)	3.24 (0.93)
Secondary outcomes
Perceived personal relevance	2.97 (0.95)	2.92 (0.86)	2.95 (0.90)	2.97 (0.96)	3.09 (0.97)	3.04 (0.96)	3.07 (0.92)	3.00 (0.90)	3.04 (0.91)
Treatment intention	3.63 (1.13)	3.55 (1.03)^b*	3.59 (1.08)	3.53 (1.07)	3.47 (1.09)^b*	3.50 (1.08)	3.62 (1.10)	3.47 (1.03)^b*	3.55 (1.07)

Note. Standard deviations within parentheses; Control variables include age, gender, education, subjective numeracy, health literacy, and graph literacy; ^a Mean differs significantly compared to total personalized risk-only and personalized below average conditions; ^b Total numerical + visual format differs significantly compared to total numerical-only format; *p < .05, **p < .01, ***p < .001.

Table B1. Main and interaction effects on primary outcome measures resulting from a 3 × 2 MANCOVA and on the secondary outcome variables resulting from two separate 3 × 2 ANCOVAs for the colon cancer scenario.

Main/interaction effects	Primary outcome variables									Secondary outcome variables
	Risk estimates			Risk perception			Affective evaluation			Perceived personal relevance			Treatment intention
	F	p	ηp^2	F	p	ηp^2	F	p	ηp^2	F	p	ηp^2	F	p	ηp^2
Comparative risk information^a	8.27	<.001	.009	0.16	.854	.000	0.22	.801	.000	1.71	.182	.002	1.60	.202	.002
Message format^b	0.00	.952	.000	0.01	.925	.000	0.02	.896	.000	0.04	.834	.000	4.04	.045	.002
Comparative risk information × Message format^a	1.88	1.53	.002	0.94	.389	.001	0.90	.406	.001	2.12	.120	.002	0.32	.728	.000

Note. ^adf = 2, 1741; ^bdf = 1, 1741; Covariates include age, gender, education, subjective numeracy, health literacy, and graph literacy; Significant results are given in bold.

Furthermore, there was no effect of message format on people’s estimated risks, perceived risks, affective evaluations, or perceived personal relevance (all ps > .834). However, participants who received risk information in a numerical-only format were more likely to take the treatment compared to participants who received the information in a numerical + visual format (F(1, 1741) = 4.04, p = .045, η_p² = .002).

Effects on outcomes in the increased cholesterol scenario

Descriptive statistics for all outcomes in the increased cholesterol scenario are shown in Table 3, and all test statistics can be found in Table B2 (Appendix B). The pattern of results for the effects of comparative information are almost similar to that of the first health scenario. There was a significant effect of comparative information on people’s estimated risks, F(2, 1741) = 3.65, p < .026, η_p² = .004. Participants who received personalized risks that were above average reported significantly lower probability estimates than participants who received the same personalized risks that were below average (p < .010) or that were without any comparative information (p = .048) (Figure 3b). Again, there were no significant main effects of comparative information on perceived risk, affective evaluations, or treatment intention (all ps > .244). However, there was a significant main effect of comparative information on perceived personal relevance (F(2, 1741) = 3.73, p = .024, η_p² = .004), with participants perceiving personalized risks that were below average as more personally relevant than personalized risks without comparative data (p = .022). Similar to the previous scenario, these (non-)effects of comparative information did not depend on the message format (all ps > .083).

Table 3. Participants’ mean scores on the primary and secondary outcomes in the increased cholesterol scenario across all experimental conditions.

Outcomes	Personalized risk-only			Personalized risk above average			Personalized risk below average
	Numerical-only	Numerical + visual	Total	Numerical-only	Numerical + visual	Total	Numerical-only	Numerical + visual	Total
Primary outcomes
Risk estimates	31.4 (18.8)	29.6 (18.1)	30.5 (18.4)	27.7 (16.8)	29.2 (17.9)	28.4 (17.4)^a*	31.3 (18.6)	31.0 (18.8)	31.2 (18.7)
Risk perception	2.60 (0.88)	2.44 (0.85)^b*	2.52 (087)	2.51 (0.80)	2.50 (0.85)^b*	2.50 (0.83)	2.57 (0.88)	2.46 (0.90)^b*	2.52 (0.89)
Affective evaluation	2.41 (0.98)	2.33 (0.90)^b*	2.47 (1.01)	2.40 (0.98)	2.35 (0.96)^b*	2.37 (0.94)	2.50 (1.03)	2.43 (0.97)^b*	2.38 (0.97)
Secondary outcomes
Perceived personal relevance	2.80 (0.98)	2.72 (0.95)	2.76 (0.97)	2.79 (0.94)	2.93 (1.02)	2.86 (0.98)	2.95 (0.96)	2.87 (1.00)	2.91 (0.98)^c*
Treatment intention	3.38 (1.15)	3.46 (1.18)	3.42 (1.16)	3.34 (1.15)	3.27 (1.16)	3.31 (1.16)	3.32 (1.21)	3.34 (1.21)	3.33 (1.21)

Note. Standard deviations within parentheses; Control variables include age, gender, education, subjective numeracy, health literacy, and graph literacy; ^a Mean differs significantly compared to total personalized risk-only and personalized below average conditions; ^b Total numerical + visual format differs significantly compared to total numerical-only format; ^c Mean differs significantly compared to total personalized risk-only condition; *p < .05, **p < .01, ***p < .001.

Table B2. Main and interaction effects on primary outcome measures resulting from a 3 × 2 MANCOVA and on the secondary outcome variables resulting from two separate 3 × 2 ANCOVAs for the increased cholesterol scenario.

Main/interaction effects	Primary outcome variables									Secondary outcome variables
	Risk estimates			Risk perception			Affective evaluation			Perceived personal relevance			Treatment intention
	F	p	ηp^2	F	p	ηp^2	F	p	ηp^2	F	p	ηp^2	F	p	ηp^2
Comparative risk information^a	3.65	.026	.004	0.04	.960	.000	1.32	.267	.004	3.73	.024	.004	1.41	.244	.002
Message format^b	0.11	.741	.000	5.47	.019	.003	4.20	.041	.002	0.17	.681	.000	0.03	.854	.000
Comparative risk information × Message format^a	1.36	.258	.002	1.16	.314	.001	0.91	.404	.001	2.50	.083	.003	0.62	.540	.001

Note. ^adf = 2, 1741; ^bdf = 1, 1741; Covariates include age, gender, education, subjective numeracy, health literacy, and graph literacy; Significant results are given in bold.

Furthermore, there was a significant main effect of message format on risk perception (F(1, 1741) = 5.47, p = .019, η_p² = .003) and affective evaluation (F(1, 1741) = 4.20, p = .041, η_p² = .002). Participants in the numerical + visual format reported lower levels of perceived risk and affective impact of the risk information compared to participants who received the same personalized risks in a numerical-only format. There were no significant main effects of message format on risk estimates, perceived relevance, or treatment intention (all ps > .681).

Effects on outcomes in the skin cancer scenario

Descriptive statistics for all outcomes in the skin cancer scenario are shown in Table 4, and all test statistics can be found in Table B3 (Appendix B). The pattern of results for the effects of comparative information are almost similar to that of the other two health scenarios; there were again no significant main effects of comparative information on people’s perceived risks, affective evaluations, or treatment intentions (all ps > .272). However, this time, there was no effect of comparative information on participants’ estimated risks, F(2, 1741) = 1.28, p = .279, η_p² = .001. Furthermore, similar to the previous scenario, there was a significant main effect on perceived personal relevance (F(2, 1741) = 3.26, p = .039, η_p² = .004), with participants perceiving personalized risks that were below average as more personally relevant than personalized risks without comparative data (p = .034). Overall, these (non-)effects of comparative information did not depend on the message format (all ps > .099).

Table 4. Participants’ mean scores on the primary and secondary outcomes in the skin cancer scenario across all experimental conditions.

Outcomes	Personalized risk-only			Personalized risk above average			Personalized risk below average
	Numerical-only	Numerical + visual	Total	Numerical-only	Numerical + visual	Total	Numerical-only	Numerical + visual	Total
Primary outcomes
Risk estimates	24.4 (18.7)	22.3 (16.9)	23.4 (17.9)	22.0 (16.8)	24.0 (19.5)	23.0 (18.2)	24.1 (16.8)	25.0 (19.1)	24.5 (17.9)
Risk perception	2.16 (0.85)	2.06 (0.77)	2.11 (0.81)	2.18 (0.79)	2.09 (0.89)	2.13 (0.84)	2.14 (0.77)	2.13 (0.90)	2.14 (0.83)
Affective evaluation	2.45 (0.98)	2.25 (0.89)^c**	2.36 (0.94)	2.37 (0.99)	2.23 (0.99)^c**	2.30 (0.99)	2.44 (1.00)	2.35 (1.04)^c**	2.40 (1.02)
Secondary outcomes
Perceived personal relevance	2.71 (1.00)	2.59 (0.96)	2.65 (0.98)	2.68 (0.96)	2.79 (1.07)	2.73 (1.02)	2.80 (1.02)	2.78 (1.02)	2.79 (1.02)^a*
Treatment intention	3.74 (1.17)	3.82 (1.14)	3.78 (1.16)	3.69 (1.18)	3.72 (1.18)	3.70 (1.18)	3.69 (1.20)	3.76 (1.14)	3.72 (1.17)

Note. Standard deviations within parentheses; Control variables include age, gender, education, subjective numeracy, health literacy, and graph literacy; ^a Mean differs significantly compared to total personalized risk-only condition; ^b Total numerical + visual format differs significantly compared to total numerical-only format; *p <. 05, **p < .01, ***p < .001.

Table B3. Main and interaction effects on primary outcome measures resulting from a 3 × 2 MANCOVA and on the secondary outcome variables resulting from two separate 3 × 2 ANCOVAs for the skin cancer scenario.

Main/interaction effects	Primary outcome variables									Secondary outcome variables
	Risk estimates			Risk perception			Affective evaluation			Perceived personal relevance			Treatment intention
	F	p	ηp^2	F	p	ηp^2	F	p	ηp^2	F	p	ηp^2	F	p	ηp^2
Comparative risk information^a	1.28	.279	.001	0.22	.803	.000	1.30	.272	.001	3.26	.039	.004	1.05	.351	.001
Message format^b	0.03	.853	.000	3.50	.062	.002	10.15	.001	.006	0.20	.653	.000	1.03	.310	.001
Comparative risk information × Message format^a	2.32	.099	.003	0.35	.702	.000	0.47	.628	.001	1.60	.202	.002	0.04	.965	.000

Note. ^adf = 2, 1741; ^bdf = 1, 1741; Covariates include age, gender, education, subjective numeracy, health literacy, and graph literacy; Significant results are given in bold.

Furthermore, similar to the second scenario, there was a significant main effect of message format on affective evaluation (F(1, 1741) = 10.15, p = .001, η_p² = .006). Participants in the numerical + visual format reported lower levels of affective impact of the risk information compared to participants who received the same personalized risks in a numerical-only format. Again, there were no significant main effects of message format on risk estimates, risk perception, perceived relevance, or treatment intention (all ps > .062).

Exploratory findings: individual differences and need for personalized risks

We observed some important differences between people with different subjective numeracy skills on a range of different measures. First, regardless of message format or comparative risk strategy, less numerate people had less accurate risk estimates and were less likely to take over the personalized risk they were provided with compared to highly numerate participants in all three health scenarios (χ²_skincancer(1, 1784) = 153.25, p < .001; χ²_cholesterol(1, 1784) = 142.20, p < .001; χ²_coloncancer(1, 1784) = 158.08 p < .001). Moreover, less numerate people were more likely to estimate their risk as 50% percent compared to highly numerate people (χ²_skincancer(1, 1784) = 35.81, p < .001; χ²_cholesterol(1, 1784) = 44.59, p < .001; χ²_coloncancer(1, 1784) = 18.16, p < .001).

Even though our sample showed great interest in receiving personalized risk of treatment side-effects (70.6% reported to have quite a bit or very much need, 22.5% a little need, and 6.9% no need at all), less numerate people also showed less interest in wanting personalized risks (M = 2.72, SD = 0.99) than highly numerate people (M = 3.19, SD = 0.86); t(1673) = 10.43, p < .001, d = 0.51). If people were being provided with their personalized risk score, the majority also wanted to receive comparative information (73.5%), whereas some only wanted to receive comparative data when their personalized risks were better (7.4%) or worse (11.7%), and few did not want to receive comparative data (7.4%). However, both less (65%) and highly (78.1%) numerate people showed great need for receiving comparative information.

Discussion

In this experimental study, we found that comparative data of the average person’s risk can be used by people for estimating their own personalized risk, without negatively impacting their risk perceptions, affective evaluations, or treatment intentions. Processing of comparative information was not influenced by whether it was presented via numbers-only or combined with visual information, nor did it not differ for people with varying sociodemographic characteristics and different levels of subjective numeracy, health literacy, and graph literacy skills.

These findings are in contrast with our pre-registered hypothesis (H1) and previous research on this topic (Zikmund-Fisher, Fagerlin, and Ubel 2010; Fagerlin, Zikmund-Fisher, and Ubel 2007; French et al. 2004; Klein 2003; Han et al. 2011). People who received contextual information indicating that their personalized risk was above the generic risk did not report higher risk perceptions and did not evaluate the information as more frightening or worrisome compared to people who received the same personalized risks that were below average or without any comparative information. A possible explanation could be that we investigated the effects of comparative data in the context of personalized side-effect risks of treatment options, whereas previous research was carried out in the screening context, emphasizing the risk of getting a disease such as breast cancer (Fagerlin, Zikmund-Fisher, and Ubel 2007; Han et al. 2011). Moreover, given that we did not discuss any risk factors that contributed to the personalized risk score, it may also be that participants found it difficult to fully understand why exactly their personalized risk deviated from the average person’s risk (Vromans et al. 2022; Zikmund-Fisher 2019). However, our results do corroborate with researchers who are in favor of disclosing comparative information (Schwartz 2009, 2016), and are also strengthened by the fact that the vast majority of our sample reported a high need for receiving comparative data when being provided with personalized risks (Fagerlin, Zikmund-Fisher, and Ubel 2007).

One unanticipated finding was that, on average, people’s risk estimates were influenced by comparative information, but in the opposite direction of our hypothesis (H1). We expected that people’s risk estimates would—jointly with their risk perceptions and affective evaluations—increase when showing people that their personalized risk is above average. Generally, even when people are being provided with numerical risks, they tend to overestimate their own risk estimate (Knapp, Raynor, and Berry 2004; Knapp et al. 2010; Vromans et al. 2020). In two out of three health scenarios, people used the average risk for determining their own risk score by reporting lower risk estimates when being provided with relatively less favorable information, and higher risk estimates when being provided with more favorable risk information. These results suggest that people do not ignore comparative data when estimating their own risk. Additionally, distributions plots of people’s risk estimate showed that most people take over the personalized risk score they were provided with, thereby demonstrating accurate risk estimates. However, less numerate people were more likely to estimate their own risk as ‘fifty-fifty’ and less likely to take over their personalized risk score compared to highly numerate participants. These ‘fifty-fifty’ responses may represent answers such as ‘don’t know’, especially among less educated and numerate people (Bruine De Bruin and Carman 2012). Although people’s responses to comparative information did not depend on numeracy skills, the different risk estimates patterns of less numerate people may invite further research to examine whether providing comparative data may be beneficial or detrimental for less numerate people.

The results found for comparative information did not depend on whether risks were communicated via natural frequencies or combined with colored icon arrays. However, regardless of comparative risk strategy, in two out of three scenarios, numbers combined with visual information were evaluated as less affective compared to numerical risks without visual information. It has been suggested that some graphical formats, such as icon arrays or pictographs, may be affect-inducing formats through automatic associations (e.g. the color red may be associated with danger), which may have an impact on people’s risk perceptions and emotional responses (Chua, Yates, and Shah 2006; Visschers et al. 2012). Our result, which is the opposite of our expectation (H2), can thus possibly be explained by the fact that these health scenarios contained lower risk estimates, and therefore presented icon arrays displaying less people affected by the risks in red (and therefore more nonaffected people in grey). Overall, these findings did not depend on whether risks were presented with or without comparative information (H3), nor on people’s graphical literacy skills.

This study is an improvement over previous research, specifically by utilizing a large representative sample of the Dutch population. This allowed us to examine the effects of comparative information, while testing influential factors such as message format, and controlling for numeracy, health literacy, and graph literacy. Other strengths are that we tested the effects across three different health scenarios, and that we pre-registered our study before data collection. However, our findings are limited to the use of hypothetical treatment decision-making scenarios; even though this is a common research strategy, a general limitation of this method is that we cannot be sure whether results fully generalize to participants experiencing real consequences of their perceptions and decisions (Schmiege, Klein, and Bryan 2010). Whether our findings are similar or different in real-world clinical settings and decision-making contexts should be investigated in future research.

We believe that our study results have implications for both research and practice. First, the findings directly add to two recently published review articles on communicating outcome probabilities as part of the 2021 evidence update for the International Patient Decision Aids Standards Collaboration (Trevena et al. 2021; Bonner et al. 2021). These reviews particularly highlight (1) the need for more systematic research on the usage and effects of personalized risks in the context of treatment decision-making (Trevena et al. 2021), but also emphasize (2) the importance of communicating risk estimates in context and evaluative labels including strategies such as comparative data for helping people make sense of their health risks (Bonner et al. 2021). By experimentally testing different presentation format for conveying health risks that are personalized towards the user characteristics, and taking into account the variability in numeracy, health literacy, and graph literacy skills, the results of our study will make an important contribution to field of risk communication and shared decision-making. Second, our results may help decision aid developers who wish to personalize risk estimates within their tools (Vromans, Tenfelde et al., 2019; Vromans, van Eenbergen et al. 2019; Hommes et al. 2021). Knowing the conditions under which circumstances, for instance, comparative information and visual displays are most associated with risk perception, worry, or treatment decision-making might help the design of effective personalized risk information tools. This could help patients to become more involved in shared decision-making with their doctor, so that they can jointly make a well-informed decision about treatment.

Conclusion

Despite the ongoing debate, our results suggest that healthcare professionals can consider providing comparative data when communicating personalized risks of treatment outcomes. Comparative data can be used for especially estimating people’s own risk, without negatively impacting their risk perceptions, affective evaluations, or treatment intentions. Future research should confirm our findings in real-world decision-making scenarios, and explore which people are particularly influenced by comparative risk data.

Disclosure statement

This work was supported entirely by the Data Science Center Tilburg (DSC/t) and in part by the Netherlands Organisation for Scientific Research (NWO) (Grant No. 628.001.030).

Data availability statement

The data can be accessed via https://tiu.nu/osf/q2dcz.

Additional information

Funding

nederlandse organisatie voor wetenschappelijk onderzoek.

Notes

1 Our sample was slightly older than the mean age (49.5 years) of the Dutch population, Mdiff = 4.47, 95% CI [3.64, 5.32], t(1806) = 10.42, p < .001), and represented slightly more females (50.7%).

&

Appendix A

&

Appendix B