3,062
Views
13
CrossRef citations to date
0
Altmetric
Original Articles

Personalized risk communication for personalized risk assessment: Real world assessment of knowledge and motivation for six mortality risk measures from an online life expectancy calculator

, , , &

ABSTRACT

In the clinical setting, previous studies have shown personalized risk assessment and communication improves risk perception and motivation. We evaluated an online health calculator that estimated and presented six different measures of life expectancy/mortality based on a person’s sociodemographic and health behavior profile. Immediately after receiving calculator results, participants were invited to complete an online survey that asked how informative and motivating they found each risk measure, whether they would share their results and whether the calculator provided information they need to make lifestyle changes. Over 80% of the 317 survey respondents found at least one of six healthy living measures highly informative and motivating, but there was moderate heterogeneity regarding which measures respondents found most informative and motivating. Overall, health age was most informative and life expectancy most motivating. Approximately 40% of respondents would share the results with their clinician (44%) or social networks (38%), although the information they would share was often different from what they found informative or motivational. Online personalized risk assessment allows for a more personalized communication compared to historic paper-based risk assessment to maximize knowledge and motivation, and people should be provided a range of risk communication measures that reflect different risk perspectives.

Background

“Personalized risk assessment” is a process where an individual’s level of risk is calculated using multiple predictors that are specific for an individual. Implicit in the risk assessment is the belief that individualized information, more so than general information, will better inform people about the risk of negative health outcomes (Citation1Citation3). A critical component of personalized risk assessment is communication of that risk. Risk communication has been defined as “the open two-way exchange of information and opinion about harms and benefits…to improve understanding of risk and promote better decisions about clinical management” (Citation4). There is a growing body of research, which examines the impact of how risk information is presented, and the impact the presentation format has on patients’ risk perception, decision-making, and behavior change (Citation3,Citation5Citation7).

People perceive risk differently depending on their underlying knowledge and, importantly, the way in which the risk information is presented (Citation4,Citation8). A study by Fagerlin et al. found that providing people with comparative risk information changes risk perceptions: women, who were told that their hypothetical cancer risk was higher than the average women’s, were more compelled to endorse pharmaceutical treatment (Citation9). The framing of risk information has also been shown to have a significant effect on decisions made (Citation6). The largest effects are evident when relative risk information is presented, as compared with absolute risk data (Citation6,Citation10,Citation11). Additionally, there is evidence that “loss framing” had a larger impact on screening uptake than “gain framing” (Citation6), and surgery was preferred to other treatments when treatment efficacy was presented in a positive frame (survival) rather than a negative frame (mortality) (Citation11). In general, studies examining the impact of manipulations of risk and benefit information suggest that providing more information and expressing the information in more than one way are associated with improved patient knowledge and may provide a balanced view enabling more informed decisions (Citation6,Citation11Citation14).

Currently, there is a range of risk calculators available on the Internet, and there is a growing body of research examining the impact of the calculator risk estimates on users’ risk perception and motivation for health behavior change (Citation2,Citation15Citation20). In a study that evaluated patient understanding of “heart age” (i.e., Framingham risk calculations converted to heart age, where older heart age indicates modifiable risk factors), participants were motivated to consider lifestyle changes regardless of whether they received older, younger, or the same heart age results (Citation15). Another study evaluated the impact of heart age on intention to make lifestyle changes in head-to-head comparison with percentage risk formats (Citation20). Heart age was found to be superior to percent risk for risk perception and was more emotionally impactful for participants with higher CVD risk (e.g., older heart age). A more recent study randomly allocated subjects to conventional medical advice (control group), 10-year Framingham risk percentage score, or heart age (Citation17). Both the Framingham and heart age groups had significant decreases in risk scores 12 months post-intervention, compared to the control group; the magnitude of difference was greater for the heart age group. However, a study of adults who were at increased risk of CVD and were provided a personalized 10-year CVD risk estimate demonstrated no improvement in physical activity or estimated CVD risk over a one-month period (Citation19).

Other studies that evaluated the impact of online diabetes risk calculators found that health consumers who overestimated their risk showed larger improvements in their risk perceptions after exposure to the risk calculator than those who initially underestimated their risk (Citation2,Citation16). However, there was no effect in interest in increasing healthy behaviors (as assessed by interest in follow-up preventive health services) (Citation2).

While risk assessment is individualized, the same cannot be said of how risk assessment calculations are communicated to patients. We define personalized risk communication as a process where the results of an individual’s risk assessment are tailored to an individual’s preference and intended use. Historically, risk calculations were performed by clinicians and presented and discussed with patients using the same measure for all patients; this is still largely the case. However, with the advent of electronic risk calculation, it has become increasingly feasible to calculate different measures of risk. For example, for cardiovascular disease, the Framingham risk tool or other predictive algorithms are used to calculate a patient’s 5- or 10-year probability of heart disease (an absolute measure of risk) or their risk category (high, medium, low risk) (Citation15,Citation17Citation20). More recently, heart disease risk algorithms have been translated to heart age (a relative measure of risk) (Citation15,Citation17,Citation20).

While risk assessment and communication can be viewed as an intervention, they are best viewed as a component of decision-making. It is within the decision-making process where patients consider their options to engage in different interventions with the purpose of improving their health. For example, in the case of heart disease prevention, a decision aid could be used to help patients decide between starting a statin, modifying their lifestyle, or neither intervention. The purpose of a decision aid is to align a patient’s intervention choice with their values.

Risk measures are used within the decision-making context to help patients accurately gauge the benefit and harms from the alternative interventions. Relative measures and risk categories are often used with the implicit objective to motivate patients to uptake interventions, particularly for people who are categorized as “high” risk or have a (relative) risk that is higher than a reference population. Engagement of a person’s social networks has been shown to be a predictor of achieving health behavior modification and other health outcomes and has been identified as a key performance indicator for evaluation (Citation21,Citation22). However, it is unclear what type of risk assessment measures people are interested in sharing with others.

In the clinical setting, there is low use of clinical risk calculators as well as very little ascertainment of health behaviors. The omission is notable, given many clinical practice recommendations advocate the use of multivariable (personalized) risk tools. The approach to disease prevention and management changes when a personalized risk assessment is performed and includes measurement of health behaviors (Citation23). However, there is a growth in the number of online risk calculators, but there are scant empirical data on the use of online calculators by users in the community or their role in communicating risk.

The objective of our study was to investigate how informative and motivating are different life expectancy measures (i.e., different presentations of risk) in an online calculator that was developed for use by people, independent of their clinician. We were interested in determining which healthy living metrics are useful for risk assessment and risk communication—that is, do they improve knowledge or motivation—and whether there were differences in the performance characteristics (informative and motivating) across the different metrics. We also wanted to examine whether there were differences in the performance characteristics across sociodemographic and behavior profile groups. Lastly, we were interested in determining whether people were interested in sharing results of their risk assessment with their clinician and/or social networks.

Methods

Setting

An online life expectancy calculator (www.projectbiglife.ca) was developed as part of a knowledge translation strategy for a report, which examined the burden of health behaviors on life expectancy (Citation24). The sex-specific mortality risk algorithms, which were developed by a multidisciplinary team with substantive input from a policy advisory group, were based on population-based survey data with 568,997 person-years of follow-up and had excellent discrimination (c-statistic: 0.87 for both models) and accuracy (Citation24). The online life expectancy calculator that was developed from these algorithms had tremendous uptake—over 1 million risk calculations performed to date.

The calculator provided personalized life expectancy based on self-reported responses to health behavior (smoking status, alcohol consumption, diet, and leisure-time physical activity) and sociodemographic (age, sex, education level, and an area-based measure of neighborhood wealth/social support) questions. The calculator has 21 to 25 questions (response tree that varies depending on question responses) and takes 3 to 4 minutes to complete. Individualized life expectancy estimates are generated using a period life table approach and the probability of death based on respondents’ self-reported exposures.

In January 2015, we implemented five new risk measures to supplement the original life expectancy calculator (see for a hypothetical example of a 65-year-old male). We added the impact of individual risk factors (smoking, alcohol, diet, and physical activity) in the form of life expectancy loss due to the each behavior. We calculated a relative measure life expectancy, health age, which compared a user’s life expectancy to the average Canadian of the same age and sex (e.g., “health age 58.7 years”). When the Canadian life expectancy was less than the user’s life expectancy, the difference was subtracted from the user’s current age (thus, their “health age” was younger than their actual age); conversely, when the Canadian life expectancy was greater, the difference was added to the user’s current age. We added probability of future life event (“will you live to see it?”), which gave users the opportunity to identify any future, self-defined event and obtain the probability of being alive at that date. We added customized whimsical facts based on users’ life expectancy (e.g., how many tons of vegetables a user will eat over their lifetime). Lastly, we added a customizable detailed report that included recommended health behavior targets.

Figure 1. Life expectancy calculator outcome metrics at www.projectbiglife.ca.

Figure 1. Life expectancy calculator outcome metrics at www.projectbiglife.ca.

Study approach

This observational study that targeted community dwelling adults (aged 20 to 79 years) involved a convenience sample of voluntary participants. On February 13, 2015, we launched an open online survey in which users of the www.projectbiglife were invited to participate. New visitors to the website were alerted to the survey through a note on the landing page: “Update! Five new ways to present how your healthy living affects you. Please provide us with feedback—survey at the end of the report” and previous visitors who indicated that they were interested in receiving updates on the Big Life calculators were sent a newsletter with information about the update and online survey.

Survey development, which drew on tools previously developed for evaluating acceptability of patient decision aids (Citation25), focused on questions regarding how informative (Did you learn something new?) and motivating (Were you inspired to make a change?) respondents found the new risk information and how interested they would be in sharing the information with their healthcare professional (Would you share the information with your healthcare provider?) or with friends/family through social media (How likely would you be to share the information with others (e.g., Facebook, Twitter, e-mail, etc.)?). The questions were reviewed by a group of colleagues with expertise in web-based risk assessment, decision aid evaluation, and survey development. The survey, which was developed on www.surveygizmo.com and then imbedded into www.projectbiglife.ca, employed skip pattern logic (i.e., conditional display of items based on responses to other items) and allowed users to go back and change responses. The survey was piloted with an invited test group that included both novice and experienced users of www.projectbiglife.ca. Testers provided feedback on clarity and length of the survey as well as any technical issues encountered with either mobile or desktop use. Revisions were made based on testers’ feedback; Additional File 1 provides the final questions that were included in the survey.

Analysis

Data from the life expectancy calculator were categorized as follows. Smoking status was characterized as never, former, or current; collapsing heavy (≥ 1 pack per day) and light (<1 pack per day) for the latter two categories: Alcohol consumption was specified as heavy, moderate, and light/non using cut points for daily alcohol consumption and the presence of bingeing behavior. Heavy drinkers indicated that they consumed >21 (men) or 14 (women) drinks in the previous week or typically had five or more drinks on an occasion; moderate drinkers consumed 4 to 21 (men) or 3 to 14 (women) drinks in the previous week; and light- or nondrinkers reported consuming 0 to 3 (men) or 0 to 2 (women) drinks in the previous week. Using average metabolic equivalent of task (MET) per day derived from an aggregate list of light-, moderate-, and vigorous-intensity leisure-time physical activities, physical activity was categorized into three groups: inactive (0 to < 1.5 METs/day), moderately active (1.5 to <3 METs/day), and active (≥3 METs/day). A diet index score was developed for the mortality algorithm using an a priori approach that considered the possibility that different dietary components could be either protective (fruit and vegetable and carrot consumption) or harmful (high potato or fruit juice consumption) (Citation24). The index score varied between 0 and 10, with points for each daily serving of fruit or vegetables, and one minus point for high consumption of potatoes and/or juice. This score was categorized as very poor (<1), poor (1 to <3), fair (3 to <5), and adequate (≥5) diet. Descriptive statistics from users of the life expectancy calculator were summarized as frequencies.

The number of visitors to the website who viewed the first page of the survey divided by the number of users of the life expectancy calculator was defined as our view rate; the number of people submitting responses to the first survey page (“agreed to participate”) divided by the number of people who viewed the first page was defined as our participation rate; and the number of people submitting the last survey page divided by the number of people who agreed to participate was defined as our completion rate (Citation26). The analysis was restricted to respondents who completed the survey (including those who left questionnaire items blank). Data from the online survey were summarized as frequencies with 95% confidence intervals using the Clopper-Pearson (exact) binomial distribution. The responses to the survey were based on a six-point scale, with anchoring statements at either end of the scale. For the analysis, we identified any rating of five or six as highly supportive (e.g., highly informative or strongly supportive of sharing). If a survey respondent indicated that they had not reviewed a metric, they were excluded from the denominator of the relevant analysis.

Ethical approval for program evaluation of the online life expectancy calculator was obtained from the Ottawa Health Science Network Research Ethics Board (Protocol ID 20130138-01H). Consent to participate in the survey was implied by completing the online survey. No personal identifying information was retained (e.g., postal code, IP address).

Results

During a two-month period, February 13, 2015 to April 12, 2015, there were 9,823 users of the life expectancy calculator. The survey view rate was 3% (n = 325), the participation rate was 99% (n = 322), and our survey completion rate was 98% (n = 317). A larger proportion of survey respondents were Canadian and older than nonrespondents (). Additionally, survey respondents had a healthier profile with fewer respondents in the least healthy category for the four health behaviors and fewer respondents with a health age below the population average (). Overall, the leading health behavior risk was diet, followed by smoking and physical activity for both survey respondents and nonrespondents.

Table 1. Baseline characteristics of Project Big Life users, February 13 to April 12, 2015.

Are the healthy living metrics informative and/or motivating?

When survey respondents were asked whether they learned something new from the healthy living measures, 82% rated at least one of the six metrics (i.e., life expectancy, impact of risk factors, health age, whimsical facts based on users’ life expectancy, probability of a future life event, and the detailed report) as highly informative—that is, rated it as a five or six on the six-point scale (). Eighty percent of survey respondents found at least one of the healthy living measures highly motivating ().

Table 2. Percentage of survey respondents who found the risk measures informative or motivational.

To explore whether there was a single metric that respondents endorsed over others, we created a cross-tabulation of the proportion of respondents who found the risk communication metrics highly informative or highly motivating by the number of other risk communication metrics they found informative or motivational (). There were few respondents who rated a single metric as highly informative or motivational. Of those that did, health age had the highest proportion of respondents that solely identified it as highly informative and life expectancy had the highest proportion that solely identified it as highly motivational; however, the proportions were low in both cases (7.9% and 7.0%, respectively). It is notable that each metric is identified by a proportion of respondents as being the only metric that they found highly informative or highly motivating.

Table 3. Cross-tabulation of the percentage of respondents who found the risk communication metrics highly informative or highly motivating by the number of other risk communication metrics they found informative or motivational.

Are there differences by sociodemographic group or health behavior profile?

While there was variation in the individual metric respondents found most informative and motivating, there were no statistically significant differences by sociodemographic group or health behavior profile (, Additional File 2).

We compared endorsement of the metrics by health status, as measured by having a health age better or worse than the population average () and across the individual health behaviors (). The smaller number of respondents with worse health age and poorer health behaviors is reflected in the wider confidence intervals and limits our ability to make any conclusions regarding differences in ratings by health profile.

Figure 2. Percentage of respondents who rated the health metric as highly informative or highly motivating, by general health status and individual health behaviors.

Figure 2. Percentage of respondents who rated the health metric as highly informative or highly motivating, by general health status and individual health behaviors.

Is there potential for broader health promotion impact through intention to share with clinicians or social networks?

Our results show that many people were interested in sharing their assessment with their clinician and/or social networks: 44% indicated that they would be highly likely (five or six on the six-point scale) to share at least one of the health metrics with their healthcare provider and 38% indicated they would be highly likely to share through social media (). There was variation in the level of engagement (as measured by intention to share) across sociodemographic and health behavior characteristics ( and Additional File 2); however, in general, respondents indicated that they were more likely to share their detailed report followed by impact of the risk factors with their healthcare practitioner and more likely to share life expectancy followed by health age socially.

Table 4. Percentage of respondents interested in sharing any measure with others.

Discussion

Online personal health calculators are becoming increasingly common and used. However, to our knowledge, this is the first study to examine the performance of a multivariable calculator for communicating health risk. Furthermore, clinical risk assessment usually includes just one or two measures of risk. We examined six different measures of risk that included different metric types: absolute and relative measures, brief overall summary measures, a detailed report, and whimsical measures.

Most respondents reported that personalized assessment and communication of life expectancy (mortality) were helpful and motivating. The value of the risk assessment was seen across people of different ages, sociodemographic, and health profiles. That stated, there was a range of measures people found helpful, suggesting that people should be offered or provided a range of assessment measures. For example, healthy and unhealthy respondents (better or worse than population average health age) equally reported that at least one of the six risk measures assessed was informative and motivating; however, more healthy respondents reported that they gained knowledge from their health age measure and more unhealthy respondents found their detailed report most informative.

What this study adds and next steps

This study supports and builds on previous literature in the field of risk communication: particularly the findings on the need for multiple formats for presentation of risk information. The study data suggest that most people found personalized risk information, about the impact of their health behaviors on life expectancy, informative, and motivating. Many people are interested in sharing this information with their clinician and/or their social networks, and there may be differences in what information or summary measures people found informative, motivating, or were willing to share with others (i.e., their clinician or social network). Our results were not statistically significant, and further study is needed to explore potential differences. The large confidence limits noted for respondents with below average life expectancy likely reflect the smaller sample that we had for this group; however, future research could focus on exploring the hypothesis that individuals with poorer health status are more variable in their response to personalized risk communication. Lastly, given the growth in availability of online health apps and risk calculators, there is rising concern regarding their quality and safety particularly when these applications are not developed by experts and/or are not adequately evaluated (Citation27Citation32). Future evaluation of online risk assessment tools, like the Project Big Life life expectancy calculator, should focus on obtaining a high standard of evidence regarding benefit as well as freedom from unintended harms (Citation27).

Limitations

The main limitation of this study was survey participants that were generally healthier than other users of the life expectancy calculator, and therefore, the responses are potentially not reflective of the general population who use the Project Big Life life expectancy calculator. That said, this study was the evaluation of the real use of a life expectancy calculator, as opposed to the more common approach of evaluating risk assessment in an academic or clinical setting in a single or limited number of communities. The study has broad representation by age and other sociodemographic characteristics, including survey participation from a wide range of countries (30% of survey participants were outside of Canada and visitors to the website during the survey time frame represented nine additional countries). However, users of the Project Big Life life expectancy calculator do not represent the population at large (e.g., no representation of those who do not use technology, do not read English, and have literacy issues, etc.). Despite this, there has been an interest in the use of web-based surveys to overcome the growing challenge of recruiting subjects for in-person or telephone-based surveys (Citation33). Given the moderately high use of our life expectancy calculator, any future evaluation should consider the possibility of using a sample matching approach to overcome potential response bias (Citation34). Several studies that compared the sample matching approach have shown favorable results compared to traditional random digit dialing (Citation35).

Conclusions

Previous studies have found personalized risk assessment in the clinical setting improves risk perception and motivation. Our study shows that the benefit of personalized risk assessment can be extended beyond the clinical setting, where there is low use of risk calculators, to the community setting. There is a shift in health care toward more consumer/patient involvement in decision-making and active participation in care. Online personalized risk assessment allows people to be engaged in their health outside or prior to the clinic setting. Furthermore, online risk assessment is likely more accessible than the clinical setting and can be performed independently of clinician, thereby reducing healthcare resources needed for health risk assessment. The low use of multivariable risk assessment, which is recommended by many clinical guidelines, could potentially be improved with online calculations performed by people outside the clinic setting.

Consideration should be given to personalizing risk measures. Use and value of online calculators for informing and motivating people are likely improved when a range of measures is provided. Furthermore, there are differences between what measures people find informative and motivating and what they are willing to share with their clinician and social networks, further suggesting a personalized approach to communicating risk.

Key findings

What we know:

  • Personalized risk assessment is a process where an individual’s level of risk is calculated using multiple predictors.

  • Personalized risk communication is a process where the results of an individual’s risk assessment are tailored to an individual’s preference and intended use.

  • Personalized risk assessment is facilitated by online applications that can easily perform complex risk calculations and can summarize these complex calculations in many different ways.

What this study adds:

  • Most people find personalized risk assessment informative and motivating when presented as multiple risk measures. Many people are interested in sharing their assessment with their clinician and/or social networks.

  • There is likely variation in what information or summary measures people with different health profiles (e.g. smokers versus non-smokers) find informative, motivating, or are interested in sharing with others (i.e., their clinician or social network).

Implications:

Personalized risk communication is likely improved when people are provided a range of risk metrics as opposed everyone with a single or selected measure of risk.

Competing interests

The authors declare that they have no competing interests

Supplemental material

IMIF_A_1255632_Supplementary_material.zip

Download Zip (569.5 KB)

Acknowledgments

This independent study was funded by Canada Health Infoway Inc.

The opinions, results, and conclusions reported in this manuscript are those of the authors. No endorsement by the Ottawa Hospital Research Institute or Canada Health Infoway is intended or should be inferred.

Additional information

Notes on contributors

Douglas G. Manuel

DM conceived the study; KA and CB made substantial contributions to the design of the study. DM, KA, RP, SB, and CB contributed to acquisition and interpretation of data. CB drafted the manuscript and all authors contributed to revisions. All authors approved the final version of the manuscript to be published.

Kasim E. Abdulaziz

DM conceived the study; KA and CB made substantial contributions to the design of the study. DM, KA, RP, SB, and CB contributed to acquisition and interpretation of data. CB drafted the manuscript and all authors contributed to revisions. All authors approved the final version of the manuscript to be published.

Richard Perez

DM conceived the study; KA and CB made substantial contributions to the design of the study. DM, KA, RP, SB, and CB contributed to acquisition and interpretation of data. CB drafted the manuscript and all authors contributed to revisions. All authors approved the final version of the manuscript to be published.

Sarah Beach

DM conceived the study; KA and CB made substantial contributions to the design of the study. DM, KA, RP, SB, and CB contributed to acquisition and interpretation of data. CB drafted the manuscript and all authors contributed to revisions. All authors approved the final version of the manuscript to be published.

Carol Bennett

DM conceived the study; KA and CB made substantial contributions to the design of the study. DM, KA, RP, SB, and CB contributed to acquisition and interpretation of data. CB drafted the manuscript and all authors contributed to revisions. All authors approved the final version of the manuscript to be published.

References

  • Weinstein ND, Atwood K, Puleo E, Fletcher R, Colditz G, Emmons K. Colon cancer: risk perceptions and risk communication. Journal of Health Communication. 2004;9(1):53–65.
  • Harle CA, Downs JS, Padman R. Effectiveness of personalized and interactive health risk calculators: a randomized trial. Medical Decision Making. 2012;32(4):594–605.
  • Edwards AG, Naik G, Ahmed H, Elwyn GJ, Pickles T, Hood K, et al. Personalised risk communication for informed decision making about taking screening tests. The Cochrane Database of Systematic Reviews. 2013;2:Cd001865.
  • Ahmed H, Naik G, Willoughby H, Edwards AG. Communicating risk. BMJ. 2012;344:e3996.
  • Waldron CA, van der Weijden T, Ludt S, Gallacher J, Elwyn G. What are effective strategies to communicate cardiovascular risk information to patients? A systematic review. Patient Education Counseling. 2011;82(2):169–181.
  • Edwards A, Elwyn G, Covey J, Matthews E, Pill R. Presenting risk information–a review of the effects of “framing” and other manipulations on patient outcomes. Journal of Health Communication. 2001;6(1):61–82.
  • Zipkin DA, Umscheid CA, Keating NL, Allen E, Aung K, Beyth R, et al. Evidence-based risk communication: a systematic review. Annals of Internal Medicine. 2014;161(4):270–280.
  • Edwards A, Elwyn G. Understanding risk and lessons for clinical risk communication about treatment preferences. Quality in Health Care. 2001;10(suppl 1):i9–i13.
  • Fagerlin A, Zikmund-Fisher BJ, Ubel PA. “If I’m better than average, then I’m ok?”: comparative information influences beliefs about risk and benefits. Patient Education Counseling. 2007; 69 (1–3): 140–144.
  • Malenka DJ, Baron JA, Johansen S, Wahrenberger JW, Ross JM. The framing effect of relative and absolute risk. Journal of General Internal Medicine. 1993;8(10):543–548.
  • Moxey A, O’Connell D, McGettigan P, Henry D. Describing treatment effects to patients. Journal of General Internal Medicine. 2003;18(11):948–959.
  • Fagerlin A, Zikmund-Fisher BJ, Ubel PA. Helping patients decide: ten steps to better risk communication. Journal of the National Cancer Institute. 2011;103(19):1436–1443.
  • Trevena LJ, Zikmund-Fisher BJ, Edwards A, et al. Presenting quantitative information about decision outcomes: a risk communication primer for patient decision aid developers. BMC Medical Informatics and Decision Making.; 2013;13(Suppl 2):S7. doi:10.1186/1472-6947-13-S2-S7
  • Trevena LJ, Zikmund-Fisher BJ, Edwards A, Gaissmaier W, Galesic M, Han PK, et al. Presenting quantitative information about decision outcomes: a risk communication primer for patient decision aid developers. BMC Medical Informatics and Decision Making. 2013;13(Suppl 2):S7.
  • Bonner C, Jansen J, Newell BR, Irwig L, Glasziou P, Doust J, et al. I don’t believe it, but i’d better do something about it: patient experiences of online heart age risk calculators. Journal of Medical Internet Research. 2014;16(5):e120.
  • Harle CA, Downs JS, Padman R. A clustering approach to segmenting users of internet-based risk calculators. Methods of Information in Medicine. 2011;50(3):244–252.
  • Lopez-Gonzalez AA, Aguilo A, Frontera M, Bennasar-Veny M, Campos I, Vicente-Herrero T, et al. Effectiveness of the Heart Age tool for improving modifiable cardiovascular risk factors in a Southern European population: a randomized trial. European Journal of Preventive Cardiology 2015;22(3):389–96.
  • Nolan T, Dack C, Pal K, Ross J, Stevenson FA, Peacock R, et al. Patient reactions to a web-based cardiovascular risk calculator in type 2 diabetes: a qualitative study in primary care. The British Journal of General Practice: the Journal of the Royal College of General Practitioners. 2015;65(632):e152–e160.
  • Price HC, Griffin SJ, Holman RR. Impact of personalized cardiovascular disease risk estimates on physical activity-a randomized controlled trial. Diabetic Medicine: a Journal of the British Diabetic Association. 2011;28(3):363–372.
  • Soureti A, Hurling R, Murray P, van Mechelen W, Cobain M. Evaluation of a cardiovascular disease risk assessment tool for the promotion of healthier lifestyles. European journal of Cardiovascular Prevention and Rehabilitation: Official Journal of the European Society of Cardiology, Working Groups on Epidemiology & Prevention and Cardiac Rehabilitation and Exercise Physiology. 2010;17(5):519–523.
  • Neiger BL, Thackeray R, Burton SH, Giraud-Carrier CG, Fagen MC. Evaluating social media’s capacity to develop engaged audiences in health promotion settings: use of Twitter metrics as a case study. Health Promotion Practice. 2013;14(2):157–162.
  • Neiger BL, Thackeray R, Van Wagenen SA, Hanson CL, West JH, Barnes MD, et al. Use of social media in health promotion: purposes, key performance indicators, and evaluation metrics. Health Promotion Practice. 2012;13(2):159–164.
  • Schulte JM, Rothaus CS, Adler JN. Starting statins — polling results. New England Journal of Medicine. 2014;371(4):e6.
  • Manuel DG, Perez R, Bennett C, Rosella L, Taljaard M, Roberts M, et al. Seven more years: the impact of smoking, alcohol, diet, physical activity and stress on health and life expectancy in Ontario. Toronto: Institute for Clinical Evaluative Sciences and Public Health Ontario; 2012.
  • O’Connor AM, Cranney A. User manuel - acceptability. Ottawa: Ottawa Hospital Research Institute; 1996. p. 5.
  • Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). Journal of MEDICAL INTERNET RESEARCH. 2004;6(3):e34.
  • McCartney M. How do we know whether medical apps work? BMJ. 2013;346:f1811.
  • Marceglia S, Bonacina S, Zaccaria V, Pagliari C, Pinciroli F. How might the iPad change healthcare? Journal of the Royal Society of Medicine. 2012;105(6):233–241.
  • Huckvale K, Morrison C, Ouyang J, Ghaghda A, Car J. The evolution of mobile apps for asthma: an updated systematic assessment of content and tools. BMC Medicine. 2015;13:58.
  • Visvanathan A, Hamilton A, Brady RR. Smartphone apps in microbiology–is better regulation required? Clinical Microbiology and Infection. 2012;18(7):E218–E220.
  • Demidowich AP, Lu K, Tamler R, Bloomgarden Z. An evaluation of diabetes self-management applications for Android smartphones. Journal of Telemedicine and Telecare. 2012;18(4):235–238.
  • Bonacina S, Marceglia S, Pinciroli F. A pictorial schema for a comprehensive user-oriented identification of medical Apps. Methods of Information in Medicine. 2014;53(3):208–224.
  • Brick JM. The future of survey sampling. Public Opinion Quarterly. 2011;75(5):872–888.
  • Rivers D, editor Sampling for web surveys. Joint Statistical Meetings 2007. http://www.amstat.org/meetings/jsm/2007/
  • Vavreck L, D. Rivers The 2006 cooperative congressional election study. Journal of Elections, Public Opinion and Parties. 2008;18(4):355–366.