Abstract
Objective: To describe the quality of online information on testosterone replacement therapy (TRT) in men.
Methods: A quantitative content analysis was conducted on websites providing patient-directed information on TRT for the purpose of treating late onset hypogonadism (LOH). Websites were identified through Google in March 2017. The DISCERN instrument was used to determine the quality of health information.
Results: A total of 20 websites met inclusion criteria. Websites were primarily from the United States (45%), United Kingdom (25%), and Australia (15%). Sources of information were cited by 40% of websites. Several websites (40%) claimed that TRT had benefits, with 25% claiming that TRT was effective for treating LOH. TRT was described as a safe therapy by one website (5%), with gynecomastia (35%) and increased hematocrit (35%) representing the most commonly described side effects. Prostate specific antigen (35%) and serum testosterone monitoring (30%) were the most commonly described monitoring parameters. The mean DISCERN score was 46.4, indicating fair quality information. The Flesh–Kincaid Grade Level was 12.2.
Conclusion: Online TRT information is incomplete, often failing to describe important safety information and the need for regular monitoring.
Introduction
Classic male hypogonadism (often referred to as testosterone deficiency) develops when the testis fail to produce physiological levels of testosterone due to a disruption in the hypothalamic–pituitary–testicular axis (HPTA). Disruptions to the HPTA at the testicular level cause primary testosterone deficiency, whereas central defects in the hypothalamus or pituitary cause secondary testosterone deficiency [Citation1,Citation2]. Testosterone deficiency can also develop secondary to aging, with serum levels decreasing by 1–2% per year [Citation3,Citation4], often referred to as late-onset hypogonadism (LOH) [Citation5]. Other terms used to describe LOH include “low-T”, “andropause”, and “male menopause” [Citation2]. Testosterone deficiency is associated with low sex drive, erectile dysfunction, impotence, fatigue, reduced muscle mass and strength, loss of body hair, depression, osteoporosis, metabolic syndrome, and anemia [Citation1,Citation2,Citation5]. However, many of these signs and symptoms may also be due to other common comorbidities [Citation6]. The serum testosterone threshold required to cause deficiency symptoms is highly variable between men, further complicating the diagnosis [Citation7].
Administration of exogenous testosterone (known as testosterone replacement therapy or “TRT”) is considered the standard of care for classic male hypogonadism [Citation8]. TRT is associated with several side effects including gynecomastia, increased hematocrit, changes in lipid levels (increased LDL, lowered HDL), sleep apnea, and reduced fertility [Citation9]. However, testosterone supplementation for treatment of LOH is riddled with controversy and uncertainty. Recent systematic reviews have reported minimal to no clinically meaningful improvements in quality of life, erectile dysfunction, mood, physical function, and cognition with TRT [Citation10,Citation11]. Uncertainty around the cardiovascular safety of TRT has also sparked debate, with some studies showing an increase in cardiovascular events [Citation12–14], and others showing a neutral [Citation15–19] or protective effect [Citation20,Citation21]. The United States Food and Drug Administration approved TRT for testosterone deficiency due to disorders of the testicles, pituitary gland, or brain, outlining that a lack of definitive efficacy and safety data precludes its approval for LOH [Citation22]. Other regulatory agencies (Health Canada, and the European Medicines Agency) have taken a more liberal stance, approving TRT for men with clinical manifestations of testosterone deficiency and low testosterone levels [Citation23,Citation24].
The global sales of testosterone is estimated to have increased 12 fold, from $150 million to $1.8 billion, over an 11-year period (2000–2011) with the largest increase occurring in Canada [Citation25]. A recent review found that there were various patient, provider, and healthcare system factors that have increased prescribing, including aggressive marketing, establishment of specialty clinics, online sales, and ambiguous guidelines [Citation26]. Other possible contributors include direct-to-consumer advertising, creation of clinics dedicated to prescribing testosterone, mail-order internet pharmacies, and the improved ease of use with new transdermal formulations [Citation24].
It is estimated that 60% of adult internet users in North America and Europe seek out health information online [Citation27]. A recent survey of health information seekers found that the internet was the preferred source of health information [Citation28]. It is important that patients have access to accurate health information. The need for accurate information is especially important for TRT given the unclear efficacy and safety data with TRT, together with a rapid increase in testosterone sales. The objective of this study is to describe the quality of information on websites providing patient-directed information on testosterone supplementation in men.
Methods
Study design
A quantitative content analysis was conducted to assess the quality of health information about TRT found on websites.
Website selection
Websites were identified using Google from 10th March 2017 to 17th March 2017. Google was chosen to conduct the search as research suggests 65% of internet searches are performed using this search engine [Citation29]. A single researcher (CM) completed the search using the following search terms: “testosterone supplementation AND men”; “testosterone replacement therapy AND men”; “testosterone deficiency AND men.” Prior to conducting the search, the browser history and cookies were erased and all existing website data were removed to avoid generating personalized results. The first 3 pages of consecutive websites from each search term were screened with the inclusion and exclusion criteria. Previous studies have shown that 90% of internet users will not click on a webpage beyond the first 3 pages of results [Citation30].
To be included, websites had to target male patients and provide health information regarding TRT for the purpose of treating LOH, or associated terms. Associated terms included male menopause, andropause, low T, and low testosterone. Websites had to provide information on testosterone supplementation, and be written in the English language. Websites were excluded if they were duplicates, blogs, forums, required users to become members, discussed bioidentical or compounded testosterone therapy, targeted healthcare professionals or females, or if they provided information regarding the use of testosterone primarily for purposes other than for treatment of LOH.
Data abstraction
Data abstraction was completed by two investigators (CM, ES) independently; any discrepancies were resolved via consensus. If a consensus could not be reached, a third coder (CS) resolved the discrepancy.
The coding scheme was developed by the study investigators based on currently available clinical practice guidelines for TRT [Citation1,Citation2,Citation5,Citation31]. Data categories included: general website characteristics, the website’s description of testosterone supplementation, efficacy and safety claims, and monitoring parameters for patients using TRT. The presence of commonly used words to describe testosterone (e.g. vitality, energy, and vigor) was also abstracted.
General website characteristics that were captured included: the source of the website (e.g. commercial company, professional organization, nonprofit organizations, government website, or general information websites). Data regarding uniform resource locator (e.g. .com, .org, .gov, .ca, or .net), geographic region of publication, the use of citations, and the time of last update were also captured.
To assess the description of testosterone supplementation, websites were assessed for the presence/absence of a formal definition of TRT. This definition had to explain that TRT was the practice of taking exogenous testosterone via one or more routes for the purpose of treating male hypogonadism. Data regarding the type of TRT product and route of administration were also extracted.
Efficacy claims about testosterone supplementation included claims regarding treatment of LOH, increasing muscle mass, hair loss, erectile dysfunction, and low energy. Safety claims about testosterone supplementation included side effects, abnormal lab values, contraindications, common drug interactions and monitoring parameters.
Quality assessment
The quality of information on each webpage was analyzed using the DISCERN tool. The DISCERN tool is a 16-item validated instrument used to estimate the quality of health-related information by a lay-person [Citation32]. Each item is rated on a scale of 1 to 5, with a higher number being assigned to higher quality information. A score of 5 is assigned to items where the answer is a definite yes. A score of 1 is assigned to items where the answer is a definite no. A score of 2 to 4 is assigned if the webpage meets the criterion of the item to some extent. Total possible score for the DISCERN is 75. Scores were then categorized based on quality as excellent (a score between 63 and 75), good (51 to 62), fair (39 to 50), poor (27 to 38), and very poor (15 to 26) [Citation33,Citation34].
Readability assessment
The readability of each website was determined using the Flesch–Kincaid Grade Level (FKGL) formula which assigned a U.S. school-grade reading level that the average student in that grade could read [Citation35]. The Flesch reading-ease score (FRES) was also calculated for each website. From the FRES, websites were classified as very easy to read (a score between 90 and 100), easy to read (80 to 89), fairly easy to read (70 to 79), standard (60 to 69), fairly difficult to read (50 to 59), difficult to read (30 to 49), or very difficult (≤29). A score ≥60 is considered acceptable for most documents [Citation36].
Data analysis
Summary statistics were used to describe the data. The mean score and standard deviation were calculated for continuous data (DISCERN, FRES, and FKGL scores). All analysis was conducted with Microsoft Excel 2007.
Results
Search results
The search yielded 91 websites, of which 20 met the inclusion criteria. The reasons for exclusion of websites included duplicates (5%), directed at health professionals (30%), did not provide information on testosterone for treatment of hypogonadism (26%), discussion forums or blogs (9%), required membership or subscription (4%), website was not available (2%), and websites focused on bioidentical or compounded testosterone supplementation (1%). Search results are included in .
Figure 1. Inclusion and exclusion of websites identified through Google search.
General website characteristics
The majority of websites were general information websites (45%) and originated from the United States (45%) and United Kingdom (25%). The mean FRES was 40.37 (±10.37) and written at a Flesch–Kincaid Grade level of 12.16 (±1.99) indicating a grade 12 or higher reading level is required to read the text. Website characteristics are included in .
Table 1. Website characteristics.
Description of testosterone supplementation
Most websites provided general information on testosterone (80%), with 20% describing a commercial testosterone product (). With respect to route of administration, 90% of websites described intramuscular, while 85% identified transdermal administration as a potential option. Oral testosterone was discussed by 60% of websites, while 40% discussed other testosterone formulations including implanted pellets, buccal tablets or patches, and intranasal sprays.
Table 2. Testosterone description.
Efficacy of testosterone supplementation
Testosterone was claimed to have a general benefit in 40% of websites, while a benefit in LOH was mentioned in only 20% (). Specific efficacy claims made on websites included increased muscle mass (35%), increased energy (30%), erectile dysfunction (20%), and hair loss reversal (5%).
Table 3. Efficacy, safety, and monitoring claims of testosterone supplementation.
Safety of testosterone supplementation
Testosterone was claimed to be safe by only 1 website (5%) (). Specific risks and side effects described by websites included prostate cancer (75%), cardiovascular risks (65%), gynecomastia (35%), increased hematocrit (35%), and mood disturbances (15%). Absolute contraindications include prostate cancer (40%) and male breast cancer (20%). None of the websites discussed drug interactions with testosterone. Specific monitoring parameters mentioned included regular serum testosterone levels (30%), prostate specific antigen (PSA) (35%), digital rectal exam (DRE) (15%), and liver function tests (LFT) (5%).
Other claims
In addition to efficacy claims, the words energy, vitality, younger, and vigor were found on 65%, 30%, 25%, and 10% of websites, respectively (). Other words or phrases found on websites included happiness, bloom, second adolescence, and longevity.
Figure 2. Words used to describe testosterone supplementation. Other = happiness, bloom, second adolescence, longevity.
Quality of information
The mean overall DISCERN score was 46.4 ± 10.81 (). The highest mean score was for “does it describe the benefits of each treatment” (4.25 ± 1.29), and the lowest mean score was for “does it describe what would happen if no treatment is used” (1 ± 0.0). In this analysis, 10% of websites were of very poor quality, 10% were poor quality, 40% were fair quality, and 40% were good quality. None of the websites were excellent quality ().
Figure 3. Overall webpage rating based on DISCERN score.
Table 4. Discern instrument.
Discussion
Findings and implications
Our findings indicate that the majority (60%) of websites describing patient specific information on testosterone supplementation were “very poor” to “fair” quality. Although most websites provided information on the different routes of administration, our findings suggest that online information on TRT is inadequate, with minimal information on side effects, contraindications, drug interactions, or the need for monitoring. Recent studies indicate that aggressive direct-to-consumer advertising is associated with recent increases in testosterone use [Citation37–39]. Given these marketing strategies, it is important that men have access to quality information on TRT.
There are a few encouraging findings to take from this analysis. One positive finding was that only a few websites made general claims of testosterone being effective for late onset hypogonadism, with fewer websites claiming testosterone as being a generally safe treatment. Findings indicate the majority of websites made an effort to describe the uncertainties surrounding testosterone therapy, with few websites making unfounded general claims of efficacy or safety. However, although the information on the websites was accurate, there were many missing details to help men make informed decisions about testosterone therapy. Another encouraging finding was that almost all the websites mentioned intramuscular or transdermal routes of administration, which is appropriate given that research indicates these are the two most commonly used routes [Citation40,Citation41]. However, few websites went into detail regarding the specific pros and cons associated with each formulation.
The overall lack of higher quality websites is concerning, as only 40% of the websites were rated as “good” quality. Studies have shown that majority of patients who use the internet to find health information do not have the background to enable them to accurately discern the credibility of a website [Citation42–44]. Additionally, websites did not provide adequate safety information. Details outlining how testosterone may cause an increase in hematocrit and theoretical increased risk of venous thromboembolism were lacking on most websites [Citation45]. None of the websites mentioned common drug interactions. For example, testosterone can increase insulin sensitivity, leading to a decreased need for insulin. If insulin doses are not adjusted, there is an increased risk of hypoglycemia [Citation45]. Testosterone can also interact with anticoagulants such as warfarin [Citation45]. It is important that men are aware of these interactions.
Our results are consistent with other studies published looking at online information. McBride et al. found that the majority of websites providing patient-centered information on TRT were of poor or sub-optimal quality, as only 30% of websites described at least 2 of 3 undefined risks and most were written at a high reading grade level [Citation46]. Another study evaluating the quality of TRT focused patient information provided on healthcare provider websites found that less than half of the websites discussed the risks of TRT, and only 27% described any side effect [Citation47].
Internet seeking behaviour
One reason men may turn to the internet for health information is that studies have shown that men feel their gender identity is threatened by seeking help from healthcare professionals [Citation48]. Similarly, other research suggests patients feel empowered and more involved with their health decisions by seeking out information online [Citation49,Citation50]. A recent survey of 312 men with hypogonadism found that 82–89% of men obtained their health information from a healthcare professional, with ∼20–30% obtaining information from online blogs/chats [Citation51]. Although overall trends in health information seeking behavior show the internet as the most used source of information, hypogonadal men may be an exception. The use of the internet for disseminating information is a rational approach in engaging men, which is also supported by a recent study that used the internet in the UK to survey over 10,000 men internationally. They linked the survey regarding LOH symptoms to the Andropause Society and two other men’s health websites [Citation52]. The robust response indicates that men are accessing web-based information, and are willing to engage in online activities for research and to gather information.
Readability
The readability of the websites was at a much higher grade level (12.1) compared to what the American Medical Association and the National Institutes on Health recommend for patient health information (no higher than a sixth-grade reading level) [Citation53,Citation54]. This is due in part to the fact that the average American reads at a grade 8 level [Citation55,Citation56]. Strategies to improve readability include reducing the number of syllables per word, reducing the number of words per sentence, avoiding the use of medical jargon, and utilizing bullet points.
Limitations
Limitations of the study included a small sample size of websites (n = 20); however, these numbers are consistent with other studies of online health information [Citation46,Citation57]. Another limitation is the dynamic nature of the internet, as websites may have been updated or changed since the search was conducted. Our search strategy utilized the most popular search engine as well as simple search terms in an effort to emulate what a male would do when searching for TRT information; however, it is difficult to predict exactly how a man would search for this information. Although our study enabled us to capture what type of information is presented online, it does not speak to how this information affects men’s opinion of TRT. Additionally, our study does not speak to the accuracy of online TRT information as there is currently no validated tool available to facilitate such an assessment. The DISCERN tool was validated in the 1990s for written information, and does not take into account how information is presented or organized on a website. Additionally, one question (does it describe what would happen if no treatment was used?) is not well suited to medical conditions that do not have dichotomous outcomes. All websites scored low on this question which may have been due to the non-dichotomous nature of LOH rather than the quality of the websites. In addition, we did not review all information that would impact decision making, such as duration of treatment. Addressing the duration of treatment and continuous versus intermittent use is important, particularly in light of research demonstrating that those with continuous exposure to treatment fared far better in terms of generalized symptoms, prostate symptom scores, obesity, and bladder function [Citation58]. Future research is necessary to identify the ideal timing and duration of testosterone treatment in men with LOH. Future research efforts should assess how men may be impacted by online information on TRT.
Conclusion
Information about TRT on the internet is not comprehensive, often failing to adequately describe important safety information and the need for regular monitoring and follow up. Most websites do not describe the various routes of administration in enough detail to support patients with their decision making. Additionally, websites are written at a higher readability level than what is currently recommended for patient health information. More effort is required to improve the quality of patient focused online TRT information.
Disclosure statement
No potential conflict of interest was reported by the authors.
Additional information
Funding
Notes on contributors
Evan Sehn
Dr. Sehn is a recent graduate from the pharmacy program at the University of Alberta. He currently practices in community pharmacy, providing pharmaceutical care for hypogonadal men on a daily basis. He has an interest in geriatrics and men's health and has also coauthored evidence based-medicine summaries, answering common questions in primary care.
Cassidy Mozak
Cassidy Mozak is a recent graduate of the pharmacy program at the University of Alberta. She currently practices in the community pharmacy setting. She has an interest in research and men's and women's health.
Nese Yuksel
Nese Yuksel is a Professor at the University of Alberta and specializes in the area of women's health. She has a clinical practice in a mature women's health clinic, and conducts research on related topics, including osteoporosis, hormone therapy, and decision making for women. She also teaches on topics related to women's health (including hormonal contraception, menopause, menstrual related disorders, etc.). She has won numerous pharmacy practice and teaching awards in her career. She is also a NAMS Certified Menopause Practitioner (NCMP).
Cheryl A. Sadowski
Cheryl Sadowski is a Professor at the University of Alberta and specializes in the area of geriatrics. She has a clinical practice in an interprofessional geriatric outpatient assessment clinic. She conducts research on geriatric syndromes, interprofessional education, and pharmacy practice change. She teaches geriatrics, men's health, and topics related to vulnerable populations, including those with disabilities. She has won pharmacy practice awards for her work in geriatrics, and teaching awards for innovation and team-based teaching. She is also a board certified geriatrics pharmacist.
References
- Bhasin S, Cunningham GR, Hayes FJ, et al. Testosterone therapy in men with androgen deficiency syndromes: an Endocrine Society Clinical Practice Guideline. J Clin Endocrinol Metab. 2010;95:2536–2559.
- Dean JD, McMahon CG, Guay AT, et al. The International Society for Sexual Medicine’s process of care for the assessment and management of testosterone deficiency in adult men. J Sex Med. 2015;12:1660–1686.
- Feldman HA, Longcope C, Derby CA, et al. Age trends in the level of serum testosterone and other hormones in middle-aged men: longitudinal results from the Massachusetts male aging study. J Clin Endocrinol Metab. 2002;87:589–598.
- Harman SM, Metter EJ, Tobin JD, et al. Longitudinal effects of aging on serum total and free testosterone levels in healthy men. Baltimore Longitudinal Study of Aging. J Clin Endocrinol Metab. 2001;86:724–731.
- Lunenfeld B, Mskhalaya G, Zitzmann M, et al. Recommendations on the diagnosis, treatment and monitoring of hypogonadism in men. Aging Male. 2015;18:5–11.
- T'Sjoen G, Feyen E, De Kuyper P, et al. Self-referred patients in an aging male clinic: much more than androgen deficiency alone. Aging Male. 2003;6:157–165.
- Kelleher S, Conway AJ, Handelsman DJ. Blood testosterone threshold for androgen deficiency symptoms. J Clin Endocrinol Metab. 2004;89:3813–3817.
- Nguyen CP, Hirsch MS, Moeny D, et al. Testosterone and “age-related hypogonadism” – FDA concerns. N Engl J Med. 2015;373:689–691.
- AACE Hypogonadism TF. American Association of Clinical Endocrinologists medical guidelines for clinical practice for the evaluation and treatment of hypogonadism in adult male patients—2002 update. Endocrine Pract. 2002;8:439–456.
- Huo S, Scialli AR, McGarvey S, et al. Treatment of men for “low testosterone”: a systematic review. PLoS One. 2016;11:e0162480.
- Wells G, Elliott J, Kelly S, et al. Testosterone in the treatment of androgen deficiency. Ontario Drug Policy Research Network [Internet]. 2014 [cited 2017 Oct 8]. Available from: http://odprn.ca/wp-content/uploads/2014/12/114-TRT-systematic-review-final-Nov-17-2.pdf
- Basaria S, Coviello AD, Travison TG, et al. Adverse events associated with testosterone administration. N Engl J Med. 2010;363:109–122.
- Xu L, Freeman G, Cowling BL, et al. Testosterone therapy and cardiovascular events among men; a systematic review and meta-analysis of placebo-controlled randomized trials. BMC Med. 2013;11:108.
- Alexander GC, Iyer G, Lucas E, et al. Cardiovascular risks of exogenous testosterone use among men: a systematic review and meta-analysis. Am J Med. 2017;130:293–305.
- Calof OM, Singh AB, Lee ML, et al. Adverse events associated with testosterone replacement in middle-age and older men: a meta-analysis of randomized, placebo-controlled trials. J Gerontol A Biol Sci Med Sci. 2005;60:1451–1457.
- Haddad RM, Kennedy CC, Caples SM, et al. Testosterone and cardiovascular risk in men: a systematic review and meta-analysis of randomized placebo-controlled trials. Mayo Clin Proc. 2007;82:29–39.
- Fernandez-Balsells MM, Murad MH, Lane M, et al. Clinical review 1: adverse effects of testosterone therapy in adult men: a systematic review and meta-analysis. J Clin Endocrinol Metab. 2010;95:2560–2575.
- Borst SE, Shuster JJ, Zou B, et al. Cardiovascular risks and elevation of serum DHT vary by route of testosterone administration: a systematic review and meta-analysis. BMC Med. 2014;12:211.
- Corona G, Maseroli E, Rastrelli G, et al. Cardiovascular risk associated with testosterone-boosting medications: a systematic review and meta-analysis. Expert Opin Drug Saf. 2014;13:1327–1351.
- Shores MM, Smith NL, Forsberg CW, et al. Testosterone treatment and mortality in men with low testosterone levels. J Clin Endocrinol Metab. 2012;97:2050–2058.
- Sharma R, Oni OA, Gupta K, et al. Normalization of testosterone level is associated with reduced incidence of myocardial infarction and mortality in men. Eur Heart J. 2015;36:2706–2715.
- U.S. Food and Drug Administration. FDA drug safety communication: FDA cautions about using testosterone products for low testosterone due to aging; requires labeling change to inform of possible increased risk of heart attack and stroke with use [Internet]. U.S. Food and Drug Administration [cited 2017 Aug 27]. Available from: https://www.fda.gov/Drugs/DrugSafety/ucm436259.htm
- Health Canada. Summary safety review – testosterone replacement products cardiovascular risk [Internet]. Health Canada; 2014 [cited 2017 Aug 27]. Available from: http://www.hc-sc.gc.ca/dhp-mps/medeff/reviews-examens/testosterone-eng.php
- European Medicines Agency. No consistent evidence of an increased risk of heart problems with testosterone medicines [Internet]. European Medicine’s Agency [cited 2017 Aug 27]. Available from: http://www.ema.europa.eu/docs/en_GB/document_library/Press_release/2014/11/WC500177618.pdf
- Handelsman D. Global trends in testosterone prescribing, 2000-2011: expanding the spectrum of prescription drug misuse. Med J Aust. 2013;199:548–551.
- Medlock S, Eslami S, Askari M, et al. Health information-seeking behavior of seniors who use the internet: A survey. J Med Internet Res. 2015;17:e10.
- Jasuja GK, Bhasin S, Rose AJ. Patterns of testosterone prescription overuse. Curr Opin Endocrinol Diabetes Obes. 2017;24:240–245.
- Atkinson NL, Saperstein SL, Pleis J. Using the internet for health-related activities: findings from a national probability sample. J Med Internet Res. 2009;11:e4.
- Sullivan D. Google still world's most popular search engine by far, but share of unique searchers dips slightly [Internet]. Search engine land; 2013 [cited 2017 Sept 10]. Available from: http://searchengineland.com/google-worlds-most-popular-search-engine-148089
- iProspect. Search engine user behavior study: a report by iProspect and Jupiter Research [Internet]. [cited 2017 Sept 10]. Available from: http://district4.extension.ifas.ufl.edu/Tech/TechPubs/WhitePaper_2006_SearchEngineUserBehavior.pdf at: http://www.iprospect.com
- Morales A, Bebb RA, Manjoo P, et al. Diagnosis and management of testosterone deficiency syndrome in men: clinical practice guideline. CMAJ. 2015;187:1369–1377.
- Charnock D, editor. The DISCERN Handbook: Quality criteria for consumer health information on treatment choices [Internet]. Abingdon, U.K.: Radcliffe Medical Press; 1998 [cited 2017 Oct 6]. Available from: http://www.discern.org.uk/discern.pdf
- San Giorgi MRM, de Groot OSD, Dikkers FG. Quality and readability assessment of websites related to recurrent respiratory papillomatosis. Laryngoscope. 2017;127:2293–2297.
- Hargrave DR, Hargrave UA, Bouffet E. Quality of health information on the Internet in pediatric neuro-oncology. Neuro Oncol. 2006;8:175–182.
- Kincaid JP, Fishburne RP Jr, Rogers RL, et al. Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy-enlisted personnel. Research Branch Report 8-75, Millington, TN: Naval Technical Training, U.S. Naval Air Station; Memphis, TN; 1975.
- Flesch R. How to write plain English [Internet]. [cited 2017 Oct 3]. Available from: http://www.mang.canterbury.ac.nz/writing_guide/writing/flesch.shtml
- Busnelli A, Somigliana E, Vercellini P. “Forever young” – testosterone replacement therapy: a blockbuster drug despite flabby evidence and broken promises. Hum Reprod. 2017;32(4):719–724.
- Layton JB, Kim Y, Alexander GC, et al. Association between direct-to-consumer advertising and testosterone testing and initiation in the United States, 2009-2013. J Am Med Assoc. 2017;317:1159–1166.
- Kermode-Scott B. Canadian regulators dismiss complaint about campaign publicising low testosterone. BMJ. 2011;343:d5501.
- Piszczek J, Mamdani M, Antoniou T, et al. The impact of drug reimbursement policy on rates of testosterone replacement therapy among older men. PLoS One. 2014;9:e98003.
- Layton JB, Meier CR, Sharpless JL, et al. Comparative safety of testosterone dosage forms. JAMA Intern Med. 2015;175:1187.
- Ravitch S, Fleisher L, Torres S. What websites are patients using: results of a tracking study exploring patients use of websites at a multi-media patient education center. AMIA Annu Symp Proc. 2005;2005:1092.
- Sillence E, Briggs P, Harris PR, et al. How do patients evaluate and make use of online health information? Soc Sci Med. 2007;64:1853–1862.
- Sajid MS, Iftikhar M, Monteiro RS, et al. Internet information on colorectal cancer: commercialization and lack of quality control. Colorect Dis. 2008;10:352–356.
- Pfizer Canada Inc. Depo-Testosterone Monograph [Internet]. [cited 2017 Sept 6]. Available from: https://www.pfizer.ca/sites/g/files/g10028126/f/201601/DepoTestosterone_PM_189981_15Jan2016_E.pdf
- McBride JA, Carson CC, Coward RM. Readability, credibility, and quality of patient information for hypogonadism and testosterone replacement therapy on the Internet. Int J Impot Res. 2017;29:110–114.
- Oberlin DT, Masson P, Brannigan RE. Testosterone replacement therapy and the internet: an assessment of providers’ health-related web site information content. Urology. 2015;85:814–818.
- Hiebert B, Leipert B, Regan S, et al. Rural mens health, health information seeking, and gender identities: a conceptual theoretical review of the literature. Am J Mens Health. 2016 [cited 2017 Oct 4]. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27170674
- Broom A. Virtually healthy: the impact of internet use on disease experience and the doctor-patient relationship. Qual Health Res. 2005;15:325–345.
- Oh HJ, Lee B. The effect of computer-mediated social support in online communities on patient empowerment and doctor–patient communication. Health Commun. 2012;27:30–41.
- Rosen RC, Seftel AD, Ruff DD, et al. A pilot study using a web survey to identify characteristics that influence hypogonadal men to initiate testosterone replacement therapy. Am J Mens Health. 2018;12:567–574.
- Trinick TR, Feneley MR, Welford H, et al. International web survey shows high prevalence of symptomatic testosterone deficiency in men. Aging Male. 2011;14:10–15.
- Weiss BD. Health literacy and patient safety: Help patients understand manual for clinicians Second edition. American Medical Association, American Medical Foundation. [cited 2017 Oct 3]. Available from: http://med.fsu.edu/userFiles/file/ahec_health_clinicians_manual.pdf
- National Institutes of Health. How to Write Easy-to-Read Health Materials. MedlinePlus [Internet]. [cited 2017 Oct 3]. Available from: https://medlineplus.gov/etr.html
- Doak CC, Doak LG, Friedell GH, et al. Improving comprehension for cancer patients with low literacy skills: strategies for clinicians. CA Cancer J Clin. 1998;48:151–162.
- Doak CC, Doak LG, Root JH. Teaching patients with low literacy skills. 2nd ed. Philadelphia (PA): JB Lippincott; 1996.
- Ma Y, Yang AC, Duan Y, et al. Quality and readability of online information resources on insomnia. Front Med. 2017;11:423–431.
- Yassin A, Nettleship JE, Talib RA, et al. Effects of testosterone replacement therapy withdrawal and re-treatment in hypogonadal elderly men upon obesity, voiding function and prostate safety parameters. Aging Male. 2016;19:64–69.