https://orcid.org/0000-0002-1079-2764
ABSTRACT
Since 2019, three types of HPV vaccine have been approved for use in mainland China. High quality messages are crucial for vaccine acceptance, but little is known about the online information quality concerning HPV vaccine in China. “HPV vaccine” and “cervical cancer vaccine” in the form of Chinese were used as keywords through search engines from personal computer (PC), portable mobile device (PMD), and WeChat Public Accounts in 2019. Readability, information content, as well as DISCERN scores were evaluated for each message included. Characteristics associated with quality indicators of the messages were also analyzed. A total of 294 messages from PC engines (104, 35%), PMD engines (128, 43%) and WeChat (62, 21%) were assessed. Most (269, 91%) messages required at least undergraduate readability level. The most frequently reported theme was HPV vaccine and its function (273, 93%), while the least was information regarding quality, safety and side effects (129, 44%). The frequency of messages with at least one error was 132 (45%). The median of sum DISCERN scores was 42 (IQR = 14), and only one (< 1%) message showed good DISCERN quality. Messages retrieved from PC engines and those with pictures were of better overall quality. The overall quality of HPV vaccine-related online messages in Chinese websites was not optimal. Government and health professionals should promote information quality construction to improve the status of HPV vaccination messages online.
1. Introduction
Human papillomavirus (HPV) could cause six cancers and genital warts.Citation1 The HPV prevalence was 15.6% for women in China,Citation2 resulting in 99,253 cervical cancers cases nationwide in 2014.Citation3 HPV vaccine can prevent persistent HPV infections effectively, and the World Health Organization (WHO) recommends it to be included in the National Immunization Program.Citation1 According to the program, by 2030, 90% of girls should be fully vaccinated before the age of 15, the average age before the first sexual behavior.Citation4 In China, it is also encouraged by the National Health Commission to increase public accessibility and to promote HPV vaccination for target populations. According to the instructions released by the National Medical Products Administration in China, the authorities have approved a three-dose schedule (0, 1, or 6 months) for the bivalent vaccine for females aged 9–25, a three-dose schedule (0, 2, or 6 months) for the quadrivalent and the nonavalent vaccine for females aged 20–45 and 16–26 respectively. Meanwhile, males have not been eligible for the HPV vaccination by far.
However, current acceptance of the HPV vaccination is not ideal, as the proportions of groups at recommended ages (9–45) willing to take the vaccination were only 66.9% for junior school students aged 12–13,Citation5 70.6% for college students aged 16–25,Citation6 and 83.7% for women aged 20–35 in 2013 ~ 2017.Citation7 Ultimately, only 11.0% of female college students have been vaccinated until 2019.Citation8
Receiving high-quality recommendations can considerably promote people’s willingness of taking vaccination.Citation9 Apart from suggestions from healthcare providers, online messages have also become an important source for HPV vaccine-related informationCitation10 under the context of dotcom boom. In mainland China, the usage rate of Personal Computer (PC) and Personal Mobile Device (PMD) are 81.3% and 80.0% respectively in 2019.Citation11 Moreover, WeChat, a free instant messaging App released in 2011, has now become the most widely and frequently used social media platform.Citation12 These channels release a huge variety of health information every day,Citation13 including abundant information concerning HPV vaccine from health care agencies, news outlets and individual users.
However, people are confronting a plethora of health-related information of uneven quality. The openness of Internet also makes it easier for unqualified information to get spread,Citation14 while the questionable quality of online messages makes it hard for the public to identify and to receive accurate information, which will influence the coverage of immunization.Citation15 In China, no research had focused on the overall quality of online messages about HPV vaccine, so this study aims to conduct a comprehensive analysis on messages from PC and PMD engines, as well as WeChat Public Accounts in mainland China, and to explore the potential factors associated with the quality indicators of the messages.
2. Methods
2.1. Selection of websites and WeChat public accounts
Messages from PC and PMD engines were selected according to the 2019 annual market share of search engines in China. Four search engines, Baidu Search (67.09%), Sogou Search (18.75%), Shenma Search (6.84%), and Haosou (1.91%) were selected in this study, as these four occupied over 95% of the market share in mainland China in 2019 according to StatCounter.Citation16,Citation17 PC and PMD engines were searched independently, and Shenma Search was only included in PMD owing to its availability.
Meanwhile, for the PMD terminal, top 30 WeChat Public Accounts in medical field were also selected according to their subscription rankings available from a database for WeChat Public Accounts named xiguaji (http://data.xiguaji.com/).
2.2. Identification of messages
We used “HPV vaccine” and “cervical cancer vaccine” in the form of Chinese as keywords for online searching.Citation18-20 Messages of top 30 websites from each search engine through PC and PMD were included, and all pushes from top 30 WeChat Public Accounts were collected.Citation21 HPV vaccination messages were defined as texts and other communicating materials (pictures and animations) that could inform the public of HPV-related vaccination information. The exclusion criteria were: (1) not available (2) without text (e.g., only pictures, animations, videos, or audios), or news reports irrelevant to HPV vaccine; (3) textbooks, research papers or other professional information not for popularization of health knowledge; (4) advertisement, forums or posts; (5) directing links to other websites, or links in need of further registration or Apps-downloading to visit. (6) other messages without HPV-related educational materials. Websites were classified into three categories according to the suffix and their affiliation statement: commercial (e.g., sponsored sites, private medical sites), noncommercial (e.g., academic associations, hospitals, government agencies) and unknown (sources not clearly identified).
2.3. Readability, content and DISCERN score evaluation
2.3.1. Readability evaluation
Message length (the number of Chinese characters), sentence numbers and words numbersCitation22 were counted. Meanwhile, the words in messages were classified into different difficulty levels in ascending order (1, 2, 3, and 4) with natural language recognition programming, according to the General Standard Chinese Character Table and the Syllabus of Graded Words and Characters for Chinese Proficiency, readability databases used in Chinese linguistic researches.Citation21 Consistent with a previous study,Citation21 readability scores were calculated as Y = 24.345×(percentage of Word Level 2) + 10.153×(percentage of Word Level 3) + 25.150×(percentage of Word Level 4) + 0.091×(average character numbers in sentences). Lower score represented better readability. Messages were classified into ascending readability levels (primary, middle, high school, undergraduate and doctor) based on their readability scores (Appendix 1).Citation21
2.3.2. Content evaluation
A checklist was made based on WHO position paperCitation1 and HPV vaccine Q&As,Citation23-26 in which 94 information topics were extracted and organized in four major themes: (1) basic information of HPV and related diseases (26 topics), (2) HPV vaccine and its functions (32 topics), (3) inoculation and precautions of vaccination (21 topics), (4) quality, safety, and side effects (15 topics). Then it was translated into Chinese by authors (WW, JL and ML) with medical training of obstetrics and gynecology. We evaluated in pairs if each information topic was reported (yes/no) and if the reported topic was accurate (correct/wrong) according to the checklist above.Citation18 The report frequency and report rate of each major theme and information topic were calculated. Report rate was defined as the number of messages that reported a certain information topic (or a major theme) divided by 294 (the number of included messages). Correct rate of each information topic was calculated as the proportion of messages that reported the topic correctly. We also counted the error number for each message, defined as the number of wrong topics in it, and calculated the proportion of messages with at least one error.
2.3.3. DISCERN score evaluation
The DISCERN instrument includes three parts: publication reliability assessment (Items I to VIII, which helps consider whether it can be trusted as a source of information), quality assessment (Items IX to XV, which focuses on specific details of information) and overall quality ratings of the publication (Items XVI). Each item scored ranging from 1 to 5 points, with 1 meaning that the criterion has not been fulfilled at all, 2–4 meaning that it has partially been fulfilled, and 5 meaning complete fulfillment of the criterion. The sum score of all message was calculated and classified into poor (if the score was less than 40%), fair (in the range of 40% to 79%) and good (more than 79%) group to represent the overall quality.Citation27
2.3.4. Quality control
Selection and identification of the messages were done in pairs. A pilot study was conducted by WW, JL, and ML to test for the operability of evaluation, and the result showed a poor to fair interrater agreement according to the standard used by Pistoia et al.Citation28 Therefore, a detailed consensus was made by all author to control the interrater reliability of evaluation (Appendixes 2 and 3). Included messages were then divided into nine groups. Each of the three groups evaluated by YZ, ZX, and YC was assessed in parallel and double-checked by WW, JL, and ML, respectively. Weighted Cohen’s kappa statistic was used to test the interrater reliabilityCitation29 and all the inconsistent results was reevaluated and corrected under the supervision of corresponding author.
2.4. Statistical analysis
The characteristics of messages were described in following items: (1) message source (PC engines, PMD engines or WeChat); (2) websites classification (commercial, uncommercial and unknown); (3) forms of messages (texts, pictures, animations, videos and audios); and (4) target readers (professionals, age-eligible groups, parents, and caregivers). Age-eligible groups here refer to females of all ages (9–45 years) approved to get vaccination in China. Count data were reported as numbers and percentages. Readability scores and DISCERN scores were reported as median and interquartile range (IQR) due to their skewed distribution.
The associations between message characteristics and aforementioned quality indicators were also explored. First, readability scores and DISCERN scores were compared using Kruskal-Wallis test (among three message sources) or Wilcoxon rank sum test (between messages with or without pictures). Second, the proportions of each major theme and the proportions of messages with at least one error were compared using Pearson’s Chi-squared test. Third, Bonferroni correction was adopted in multiple comparisons to control family-wise error rate (FWER).
All statistical process and hypothesis-testing were performed using R (Version 3.6.1). Statistical significance was defined as two-sided P < 0.05.
The IRB at Peking University Health Science Center considers this study to be exempt.
3. Result
3.1. Basic features of the message
294 messages (104 from PC engines, 128 from PMD engines and 62 from WeChat) were finally included from a total of 741 webpages (). Most (92%) messages were from commercial websites, and only 22 (7%) were noncommercial. The top two forms of the messages were texts (100%), and pictures (55%), and most of the target audiences were age-eligible groups (87%).
Figure 1. Selection procedure of messages included in the evaluation
3.2. Readability of the messages
The median length (numbers of Chinese characters) of included messages was 1273 (IQR = 1225) (), with a median of 49 sentences (IQR = 49) and 703 words (IQR = 698) (Supplemental Table 1). Readability level of most messages (92%) were in the undergraduate and doctor categories ().
Table 1. Basic characteristics and readability for the HPV vaccination messages
3.3. Content coverage and accuracy of the messages
The interrater agreement of the evaluation was moderate to substantial (most weighted Cohen’s kappa in 0.50–0.79). HPV vaccine and its function was the most frequently reported theme (93%), while quality, safety and side effects mentioned least (44%) (). In terms of the specific topic level, “Three kinds of HPV vaccines are currently available in China” (64%), “Bivalent HPV vaccine is for HPV-16 and 18” (60%) and “Quadrivalent HPV vaccine is for HPV-6,11,16, and 18” (59%) were the top three reported. In contrast, “HPV infection is not a genetic disease” (< 1%), “HPV vaccine is effective for people whose family members do not have cervical cancer” (< 1%), and some safety issues (i.e. anaphylactic shock, behavior influence and safety tests) (< 1%) were the least reported (Supplemental Table 2). In addition, four information topics were not reported in all messages, including
Table 2. Frequency and accuracy (%) of information topics according to WHO position paper and HPV vaccine Q&As (subset)
“HPV vaccine is currently considered as one of the most effective vaccines”, “HPV vaccine is also effective to people who are sexually active”, “HPV vaccine should be vaccinated within the shelf-life” and “No evidence shows that vaccination can weaken immunity”.
As for topic accuracy, the most frequent errors were “There are no more than 200 types of HPV viruses” (77%), “HPV cannot be transmitted if people do not have sexual behaviors” (66%), and “HPV vaccine is not effective for people with genital warts or precancerous lesions” (46%) ( and Supplemental Table 2). As for message accuracy, the proportion of messages with at least one error was 45%, with most (71%) having only one error and few (2%) having four or more errors ().
Figure 2. Frequency distribution and constituent proportion of error numbers. (a) Distribution of error numbers in the messages included (n = 294). (b) Constituent proportion of error numbers in the error-existing messages (n = 132)
3.4. DISCERN scores of the messages
The median of DISCERN scores was 42 (IQR = 14) (), among which 50 (17%) of the messages showed poor quality, 243 (83%) showed fair quality and only one (< 1%) showed good quality. Quality scores were worse than publication reliability scores, with median scores of 17 (IQR = 7) and 22 (IQR = 6), respectively. For specific DISCERN items, “VIII. Reference to areas of uncertainty”, “XI. Risks of Treatment” and “IV. Sources of Information” performed the worst, with average scores of 1.79, 1.93, and 1.95, respectively.
Table 3. DISCERN score for the HPV vaccination messages
3.5. Message characteristics associated with the quality indicators
Message source and presence of pictures were found to be significantly associated with the quality indicators mentioned above. Significant differences were observed in readability scores (P < 0.01) and DISCERN scores (P = 0.01) among PC engines, PMD engines and WeChat channels (). Messages from WeChat had lowest readability score (Median = 10.70, IQR = 2.68), a relatively high DISCERN score (Median = 43.00, IQR = 10.00), but were the most likely to have errors (47%). In contrast, messages from PC engines presented the highest readability score (Median = 12.20, IQR = 2.01), the highest DISCERN score (Median = 44.00, IQR = 11.25), and the lowest proportion of existing errors (44%).
Table 4. Comparison of readability, content indexes and DISCERN scores of the HPV vaccination messages, grouped by message sources
Meanwhile, compared to messages with pictures, messages without pictures were significantly worse in report rate of major themes and lower in DISCERN scores (). Error number and its distribution were similar among three messages sources, and were also consistent between messages with or without pictures (Supplemental Figures 1 and 2).
Table 5. Comparison of readability, content indexes and DISCERN scores of the HPV vaccination messages, grouped by with or without pictures
4. Discussion
To our best knowledge, this study is the first to describe the overall quality of HPV vaccine related messages in China. Most of the included 294 websites (92%) were commercial. Over 90% of the messages were generally difficult to read, requiring an education level of at least undergraduate. Most messages introduced HPV vaccine and its function, yet only less than half of them reported its safety and side effects. Moreover, the overall status of content accuracy was not optimal, with only 55% of the included messages having no errors. In addition, the DISCERN scores were not satisfied with only one (< 1%) message reaching the score of good quality.
In this study, the vast majority of the messages related to HPV vaccine required at least undergraduate readability level, which was consistent with another study in China focusing on Crohn disease,Citation21 and higher than findings from other countries with required reading levels above Grade 9.Citation18,Citation27,Citation30 This difference was partly attributed to the seal of approval granted by HONcode, which contains eight criteria (authoritativeness, complementarity, privacy, attribution, justifiability, transparency financial disclosure, and sponsorship) for developers of health information websites to improve quality and readability (https://www.hon.ch/HONcode/) but was seldom used in China.Citation31 Studies have found that the increasing popularity of HONcode can promote clarity and readability of medical online messages.Citation22,Citation32,Citation33 Additionally, the limited application of real-time readability assessment tools could be another reason.Citation34,Citation35 And Chinese-expression could also explain this distinction, for Chinese tends to use words instead of affixes for professional materials.Citation36,Citation37 Previous studies reported that people with lower education level were more vulnerable to HPV infection,Citation38 and people with less knowledge about HPV were also a barrier to mass vaccination.Citation39 Therefore, the inappropriate readability of messages would be a serious challenge in conquering HPV. As China initiated the “Healthy China 2030” Plan in 2016 to encourage the promotion of scientific knowledge popularization and health literacy improvement,Citation40 this plan might be an opportunity to improve the readability of all kinds of health information.
The report rate of information topics and the accuracy of messages were also not optimal in our study. This result was consistent with a recent study in China, which focusing on Zhihu, a social media mainly targeting to users well educated.Citation20 Comparatively, our study mainly focused on search engines or social media (WeChat) with apparently broader range of users. We found that only 35% of the messages mentioned the cancer-preventing function of HPV vaccine, similar to a research on Instagram posts (35%).Citation41 This prevention role is actually endorsed as “best communication practices” of HPV vaccine,Citation18 and the disclosure of this point has been proved to enhance the confidence of the messageCitation41,Citation42 and reduce the possibility of vaccine refusal.Citation43 In addition, our results conveyed that information about the safety and side effects was the least mentioned. Given that worries regarding safety and high perceived harmsCitation43,Citation44 were two of the most common reasons for vaccination denial, the lack of those information would also be a barrier for public acceptance. Furthermore, only 55% of messages included in our study were without errors, worse than a similar study in Thailand, where 97.4% of the messages were reported to be accurate.Citation45 Even though the difference in evaluation criteria could not be neglected, the disparity still indicated a flawed situation of content accuracy. The main mistakes of the messages in our study included “HPV cannot be transmitted if people do not have sexual behaviors” and “HPV vaccine is not effective for people with genital warts or precancerous lesions”. A previous study also showed that the non-sexual transmission of HPV was ignored in Wikipedia messages.Citation46 We speculated that the insufficiency knowledge of HPV transmission route and the underestimation of the vaccine efficacy could not be excluded as a potential reason of the low HPV vaccination rate in China.Citation8 The increase of vaccination rate is indispensable for better control of cervical cancer,Citation47 and measures including reviewing and modifying the messages by professionals with clear criteria should be considered to promote content quality.Citation1
The overall DISCERN score of the messages on HPV vaccine was below expectation, as most messages did not include sufficient details on references and taking vaccination. Although no previous studies had investigated the DISCERN scores of HPV vaccination messages, our result was obviously lower than a similar study on oncology.Citation48 Apart from the aforementioned deficiency in regards to the seals of approval, another non-neglected reason could be the omission of published information sources and conflicts of interests.Citation49 Furthermore, this might also relate to the weak awareness of copyright protection and the unstandardized scientific creation.Citation21 In our analysis, the mean score of Item IV (Sources of Information) was 2.17, which was below the results of research on HIV in China (2.40)Citation31 and on cesarean section in Brazil (2.40).Citation50 Given that the quality was poor, references were particularly important to serve as a guarantee to obtain more detailsCitation51 and to improve the quality of messages. In addition, nearly two thirds of the messages never mentioned any risks of taking vaccination. As risk perceptions among most people are grounded in their past experiences rather than scientific data, which can be easily susceptible by irrational factors, informing people of the risks of taking vaccination can reduce the negative impact on acceptability of the vaccine from emotional cognition, social distortions and media bias.Citation52
Our results showed that messages from WeChat had significantly better readability, while higher DISCERN scores were seen in those from PC engines. WeChat depends more on its subscribers due to the little utilization of WeChat search engines and its subscriber-dependent information reception mode.Citation12 In this case, operators of WeChat public accounts might have more incentives to reduce reading difficulty to cater to the general public. Among these three message sources, PC seemed to be more suitable for the display of formal and authoritative messages, since PMD and WeChat were originally designed for fragmented information with the aim of convenience, which could explain the higher DISCERN scores in PC messages to some extent. In addition, higher report rate of major themes and higher DISCERN scores were found in messages with pictures implanted. We suspected it was partially because having pictures in a message suggested that the author paid more attention to the form of information and usually indicated a more abundant content, which could be proved by the message length to some extent. It is worth mentioning that prior studies pointed out encoding texts with pictures contributed to lower reading difficulty in health-related messages,Citation53,Citation54 but the results of our study showed no such significant association. Considering that the overall quality of online HPV vaccine-related messages cannot be improved within a single day, we suggest the public should start by using PCs to retrieve messages and firstly browse those with pictures implanted, which are more likely to have higher content quality and DISCERN scores.
This is the first study to comprehensively describe the overall quality of HPV vaccine-related online messages in mainland China based on validated and authoritative materials, yet several points of limitations should be noted. First, the findings in this study should be extrapolated cautiously considering the temporal changes and language distinctions of online materials, which is the common limitation of cross-sectional study in information epidemiology. Second, The formula we used was developed by a study focused on Crohn disease-related online information in China published in the year of 2019.Citation21 To our best knowledge, this is the first developed Chinese readability assessment tool to evaluate online medical messages in Chinese-language, and it still needs to be verified by further studies. However, the test was built based on official linguistic materials following procedures suitable for Chinese as an isolating-language.Citation37,Citation55 Third, subjectivity of DISCERN evaluation might have effects on the results. Nevertheless, each message was evaluated by two researchers according to our predefined explanatory file, and all the inconsistent evaluation had been modified.
High quality and audience-friendly online messages are important for the cognition and acceptance of HPV vaccine among the public. In this study, most of the online messages manifested problems of difficult readability, inadequacy and inaccuracy of content, as well as low DISCERN scores, especially for those retrieved from PMD engines and those without pictures. Government and health professionals should formulate online message paradigm, and strengthen quality construction of the messages on PMD engines to improve the overall quality of HPV vaccine-related online messages, for the sake of promoting mass vaccination campaigns and reducing the disease burden of cervical cancer.
Disclosure of Potential Conflicts of Interest
No potential conflicts of interest were disclosed
Contributors
Conception and design: SW, WW, JL, ML; Acquisition of data: ML, WW, JL, YZ, ZX, YC; Natural language recognition programming: JZ; Analysis and interpretation of data: WW, ML, JL, ZX; Drafting the article: WW, JL, ML; Final approval of the version to be submitted: All authors.
Disclaimer
The authors alone are responsible for the content and writing of the paper.
Supplemental Material
Download MS Word (2 MB)Acknowledgments
Thanks to Mengran Li for his kind assistance on the English writing of the manuscript.
Supplementary material
Supplemental data for this article can be accessed online at http://dx.doi.org/10.1080/21645515.2020.1814095.
Additional information
Funding
References
- World Health Organization. Human papillomavirus vaccines. WHO Position Paper, May. 2017;2017.
- Ma X, Wang Q, Ong JJ, Fairley CK, Su S, Peng P, Jing J, Wang L, Soe NN, Cheng F, et al. Prevalence of human papillomavirus by geographical regions, sexual orientation and HIV status in China: a systematic review and meta-analysis. Sex Transm Infect. 2018;94(6):434–42. doi:10.1136/sextrans-2017-053412.
- Duan R, Qiao Y, Clifford G, Zhao F. Cancer burden attributable to human papillomavirus infection by sex, cancer site, age, and geographical area in China. Cancer Med. 2020;9(1):374–84. doi:10.1002/cam4.2697.
- World Health Organization. Draft: global strategy towards the elimination of cervical cancer as a public health problem. 2019.
- Xue L, Hu W, Zhang H, Xie Z, Zhang X, Zhao F, Qiao Y, Gao L, Ma W. Awareness of and willingness to be vaccinated by human papillomavirus vaccine among junior middle school students in Jinan, China. Hum Vaccin Immunother. 2018;14(2):404–11. doi:10.1080/21645515.2017.1393132.
- Zou H, Wang W, Ma Y, Wang Y, Zhao F, Wang S, Zhang S, Ma W. How university students view human papillomavirus (HPV) vaccination: A cross-sectional study in Jinan, China. Hum Vaccin Immunother. 2016;12(1):39–46. doi:10.1080/21645515.2015.1072667.
- Chen L, Song Y, Ruan G, Zhang Q, Lin F, Zhang J, Wu T, An J, Dong B, Sun P, et al. Knowledge and attitudes regarding HPV and vaccination among chinese women aged 20 to 35 years in Fujian Province: a cross-sectional study. Cancer Control. 2018;25(1):1073274818775356. doi:10.1177/1073274818775356.
- You D, Han L, Li L, Hu J, Zimet GD, Alias H. Human Papillomavirus (HPV) vaccine uptake and the willingness to receive the HPV vaccination among female college students in China: a multicenter study. Vaccines. 2020;8(1):31.
- Dorell C, Yankey D, Kennedy A, Stokley S. Factors that influence parental vaccination decisions for adolescents, 13 to 17 years old: national immunization survey-teen, 2010. Clin Pediatr (Phila). 2013;52(2):162–70. doi:10.1177/0009922812468208.
- Patel PR, Berenson AB. The internet’s role in HPV vaccine education. Hum Vaccin Immunother. 2014;10(5):1166–70. doi:10.4161/hv.27967.
- China Internet Network Information Center. The 44th China Statistical Report on Internet Development. 2019.
- Zhang X, Wen D, Liang J, Lei J. How the public uses social media wechat to obtain health information in china: a survey study. BMC Med Inform Decis Mak. 2017;17(S2):66. doi:10.1186/s12911-017-0470-0.
- Peng Y, Wu X, Atkins S, Zwarentein M, Zhu M, Zhan XX, Zhang F, Ran P, Yan WR. Internet-based health education in China: a content analysis of websites. BMC Med Educ. 2014;14(1):16. doi:10.1186/1472-6920-14-16.
- Cline RJW, Haynes KM. Consumer health information seeking on the Internet: the state of the art. Health Educ Res. 2001;16(6):671-692.
- ALCORN N, MADHOK R. Health Information on the Internet. J Rheumatol. 2012;39(5):883–84. doi:10.3899/jrheum.120136.
- Wei L. Research on Google’s brand performance. J Serv Sci Res. 2016;8(2):161–75. doi:10.1007/s12927-016-0008-5.
- Crandall RW. The dubious antitrust argument for breaking up the internet giants. Rev Ind Organ. 2019;54(4):627–49. doi:10.1007/s11151-019-09680-y.
- Calo W, Gilkey M, Malo T, Robichaud M, Brewer N. A content analysis of HPV vaccination messages available online. Vaccine. 2018;36(49):7525–29. doi:10.1016/j.vaccine.2018.10.053.
- Bodemer N, Müller SM, Okan Y, Garcia-Retamero R, Neumeyer-Gromen A. Do the media provide transparent health information? A cross-cultural comparison of public information about the HPV vaccine. Vaccine. 2012;30(25):3747–56. doi:10.1016/j.vaccine.2012.03.005.
- Li J, Zheng H. Coverage of HPV-Related Information on Chinese Social Media: a Content Analysis of Articles in Zhihu. Hum Vaccin Immunother. 2020 Mar 11. [accessed 2018 May 1]:[1-7]. doi:10.1080/21645515.2020.1729028. PMID: 32159420.
- Bai XY, Zhang YW, Li J, Li Y, Qian JM. Online information on Crohn’s disease in Chinese: an evaluation of its quality and readability. J Dig Dis. 2019;20(11):596–601. doi:10.1111/1751-2980.12822.
- Kaicker J, Debono VB, Dang W, Buckley N, Thabane L. Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument. BMC Med. 2010;8(1):59. doi:10.1186/1741-7015-8-59.
- World Health Organization. HPV – facts about the virus, the vaccine and what this means for you-answers to common questions asked by adolescents and young adults 2017.
- World Health Organization. Questions and answers about HPV-information for health professionals. 2017.
- World Health Organization. Questions and answers about HPV vaccination-information for parents and caregivers. 2017.
- World Health Organization. Questions and answers about HPV-facts about the virus and the vaccine
- Cerminara C, Santarone ME, Casarelli L, Curatolo P, El Malhany N. Use of the DISCERN tool for evaluating web searches in childhood epilepsy. Epilepsy Behav. 2014;41:119–21. doi:10.1016/j.yebeh.2014.09.053.
- Pistoia F, Carolei A, Bodien YG, Greenfield S, Kaplan S, Sacco S, Pistarini C, Casalena A, De Tanti A, Cazzulani B. The comorbidities coma scale (CoCoS): psychometric properties and clinical usefulness in patients with disorders of consciousness. Front Neurol. 2019;10:1042. doi:10.3389/fneur.2019.01042.
- Robillard JM, Jun JH, Lai JA, Feng TL. The QUEST for quality online health information: validation of a short quantitative tool. BMC Med Inform Decis Mak. 2018;18(1):87. doi:10.1186/s12911-018-0668-9.
- Fu LY, Zook K, Spoehr-Labutta Z, Hu P, Joseph JG. Search engine ranking, quality, and content of web pages that are critical versus noncritical of human papillomavirus vaccine. J Adolesc Health. 2016;58(1):33–39. doi:10.1016/j.jadohealth.2015.09.016.
- Niu L, Luo D, Liu Y, Xiao S. The accessibility, usability, and reliability of Chinese web-based information on HIV/AIDS. Int J Environ Res Public Health. 2016;13(8):834.
- HON. Code of conduct for medical and health web sites. Am J Health-Syst Pharm. 2000;57(13):1283. doi:10.1093/ajhp/57.13.1283a.
- Alwani MM, Campiti VJ, Bandali EH, Nesemeier BR, Ting JY, Shipchandler TZ. Evaluation of the quality of printed online education materials in cosmetic facial plastic surgery. Facial Plast Surg Aesthet Med. 2020;22(4):255–61. doi:10.1089/fpsam.2019.0013.
- Beaunoyer E, Arsenault M, Lomanowska AM, Guitton MJ. Understanding online health information: evaluation, tools, and strategies. Patient Educ Couns. 2017;100(2):183–89. doi:10.1016/j.pec.2016.08.028.
- Yu L, Reichle ED. Chinese versus English: insights on cognition during reading. Trends Cogn Sci. 2017;21(10):721–24. doi:10.1016/j.tics.2017.06.004.
- Kuo L-J, Anderson RC. Morphological awareness and learning to read: a cross-language perspective. Educ Psychol. 2006;41(3):161–80. doi:10.1207/s15326985ep4103_3.
- Xue J, Jiang X. The developmental relationship between bilingual morphological awareness and reading for Chinese EFL adult learners: a longitudinal study. Read Writ. 2017;30(2):417–38. doi:10.1007/s11145-016-9683-3.
- Kops NL, Hohenberger GF, Bessel M, Correia Horvath JD, Domingues C, Kalume Maranhao AG, Alves de Souza FM, Benzaken A, Pereira GF, Wendland EM, et al. Knowledge about HPV and vaccination among young adult men and women: results of a national survey. Papillomavirus Res. 2019;7:123–28. doi:10.1016/j.pvr.2019.03.003.
- Carhart MY, Schminkey DL, Mitchell EM, Keim-Malpass J. Barriers and facilitators to improving virginia’s HPV vaccination rate: a stakeholder analysis with implications for pediatric nurses. J Pediatr Nurs. 2018;42:1–8. doi:10.1016/j.pedn.2018.05.008.
- Tan X, Zhang Y, Shao H. Healthy China 2030, a breakthrough for improving health. Glob Health Promot. 2019;26(4):96–99. doi:10.1177/1757975917743533.
- Basch Ch EDMPH, MacLean SBA. A content analysis of HPV related posts on instagram. Hum Vaccin Immunother. 2019;15(7–8):1476–78. doi:10.1080/21645515.2018.1560774.
- Shah PD, Calo WA, Gilkey MB, Boynton MH, Alton Dailey S, Todd KG. Questions and concerns about HPV vaccine: a communication experiment. Pediatrics. 2019;2143(2):e20181872. doi:10.1016/j.jpeds.2019.02.025.
- Gilkey MB, Calo WA, Marciniak MW, Brewer NT. Parents who refuse or delay HPV vaccine: differences in vaccination behavior, beliefs, and clinical communication preferences. Hum Vaccin Immunother. 2017;13(3):680–86. doi:10.1080/21645515.2016.1247134.
- Yu Y, Xu M, Sun J, Li R, Li M, Wang J, Zhang D, Xu A. Human papillomavirus infection and vaccination: awareness and knowledge of HPV and acceptability of HPV vaccine among mothers of teenage daughters in Weihai, Shandong, China. PLoS One. 2016;11(1):e0146741. doi:10.1371/journal.pone.0146741.
- Likitdee N, Jampathong N, Kietpeerakool C. Quality assessment of websites providing information on human papillomavirus vaccines in Thailand. Asian Pac J Cancer Prev. 2019;20(11):3473–77. doi:10.31557/APJCP.2019.20.11.3473.
- Lopez CS, Krauskopf E, Villota CE, Burzio LO, Villegas JE. Cervical cancer, human papillomavirus and vaccines: assessment of the information retrieved from general knowledge websites in Chile. Public Health. 2017;148:19–24. doi:10.1016/j.puhe.2017.02.017.
- Zhao F, Qiao Y. Cervical cancer prevention in China: a key to cancer control. The Lancet. 2019;393(10175):969–70. doi:10.1016/S0140-6736(18)32849-6.
- Marie D, Alexandra R, Willianne N, Antoinette A, Alfred R, Saskia D. Female fertility in the cancer setting: availability and quality of online health information. Hum Fertil (Camb). 2020;23(3):170-178.
- Frangos Z, Steffens M, Leask J. Water fluoridation and the quality of information available online. Int Dent J. 2018;68(4):253–61. doi:10.1111/idj.12377.
- Fioretti B, Reiter M, Betrán A, Torloni M. Googling caesarean section: a survey on the quality of the information available on the Internet. BJOG. 2015;122(5):731-739.
- Zhang Y, Sun Y, Xie B. Quality of health information for consumers on the web: A systematic review of indicators, criteria, tools, and evaluation results. J Assoc Inf Sci Technol. 2015;66(10):2071-2084.
- Julie B, Alexandra H. ‘Between the demands of truth and government’: health practitioners, trust and immunisation work. Soc Sci Med. 2006;62(2):433-443.
- Choi J, Bakken S. Web-based education for low-literate parents in neonatal intensive care unit: development of a website and heuristic evaluation and usability testing. Int J Med Inform. 2010;79(8):565–75. doi:10.1016/j.ijmedinf.2010.05.001.
- Eltorai AE, Han A, Truntzer J, Daniels AH. Readability of patient education materials on the American orthopaedic society for sports medicine website. Phys Sportsmed. 2014;42(4):125–30. doi:10.3810/psm.2014.11.2099.
- Arcodia GF, Basciano B. The construction morphology analysis of Chinese word formation. Booij G, editor. The construction of words: advances in construction morphology. Cham: Springer International Publishing, Cham, Switzerland; 2018. p. 219–53.