https://orcid.org/0000-0002-2613-4125
ABSTRACT
Human papillomavirus (HPV) is considered the second largest human carcinogen after tobacco and is responsible for 5% of all cancers, 10% of cancers in women, and 15% of all cancers in developing countries. Among these, cervical cancer is the most prevalent. An HPV vaccine has recently been developed to provide primary protection against the viral infection. In 2014, Brazil’s National Immunization Program (Programa Nacional de Imunizações, PNI) started making a quadrivalent vaccine available to the public. However, after 2014, the vaccine coverage dropped and did not reach the PNI’s targets. Among other factors, this low uptake was due to the quality of information on the Internet. Using Google Trends, the main search terms used to search for vaccine-related information on the Internet were identified. The content of the identified websites was analyzed using the DISCERN instrument and their reach was determined using their page authority score. Most of the texts analyzed were not of high quality. The data that most commonly reach the lay public are from sites that lack scientific rigor. We found a weak correlation between the DISCERN and page authority scores. Based on our analysis, we inferred that the information that reaches the user is not always the most accurate and can lead to harmful decisions on vaccination. The content that reaches the user most easily is not always of sound quality. New analyses are important, especially on the impact of social networks that present even fewer criteria in publications and are more easily accessible.
Introduction
Human papillomavirus (HPV) is considered the second largest human carcinogen after tobacco, accounting for 5% of all cancers, 10% of cancers in women, and 15% of all cancers in developing countries, with cervical cancer being the most common among them.Citation1,Citation2 HPV infection is highly prevalent and affects approximately 80% of the sexually active population. It is estimated that approximately 600 million people are infected worldwide.Citation3 According to the José Alencar Gomes da Silva National Cancer Institute (INCA), the estimated number of new cases of cervical cancer in Brazil in 2020 was 16 710, making it the third most frequent malignant neoplasm in women.Citation4
Vaccination against HPV has proved to be a fundamental pillar in the global strategy to control the spread of the virus and its consequences. The vaccine attempts to inactivate HPV by introducing antibodies in the intercellular fluid. HPV vaccines are highly immunogenic and have been shown to protect against cervical intraepithelial neoplasia grade 2 or higher (NIC2+), related to the vaccine type, in 100% of cases.Citation5 In 2014, Brazil’s National Immunization Program (Programa Nacional de Imunizações, PNI) made available a quadrivalent vaccine for girls aged 11 to 13. In 2015, the age brackets were expanded, to include girls aged between 9 and 14. The PNI provides an extended regimen (0 to 6 months) using only two doses. In 2017, male adolescents were included in the program, to improve vaccination coverage.Citation4
In the first year (2014) of the PNI’s vaccine implementation, the campaign was successful, achieving approximately 95% of the vaccination target.Citation3,Citation4 However, in subsequent years the national vaccination index dropped significantly to approximately 50% in girls and 25% in boys.Citation6 Understanding the reasons behind this drop requires a multifactorial analysis. Several reasons can be pointed out such as access, vaccination information, and campaigns.
The Internet has become an important global source of information and medium of communication. There are more than 4.6 billion Internet users worldwide, with Brazil claiming 140 million users. Of these, 94% use the Internet to search for health-related information. Although the Internet allows easy access to information, there is often no filter or quality control, with the result that the quality of information may not be controlled and evaluated in advance.Citation7 This can lead to the user having incorrect decisions with regard to getting vaccinated.
Therefore, it is important to investigate whether the information provided on the Internet has an influence on vaccination uptake, as was the Brazilian experience with HPV vaccination. Many websites may transmit technically incorrect information and contribute to alienating potential beneficiaries of vaccination campaigns. Misleading information, or fake news, can lead people to avoid being vaccinated, out of suspicion and fear.
The Internet influences all areas of knowledge, especially health-related knowledge.Citation6 However, there are few studies evaluating the quality of information provided on the Internet, especially regarding the HPV vaccine. This is an important gap in Internet and vaccine literature. Critical evaluation of Internet content can increase use of better-quality Internet sites for information and contribute to more effective strategies aimed at improving public health levels, such as increasing HPV vaccination coverage in Brazil.Citation7,Citation8 Considerable misinformation has been spread in Brazil regarding the vaccine, and there exists no instrument to assess the content of the information disclosed. Thus, this study aimed to analyze and evaluate the quality of publicly accessible content published on the Internet about the HPV vaccine.
Materials and methods
This was a cross-sectional observational study in which the need for ethical clearance was waived owing to the lack of human respondents in the study.
Specific search terms related to the HPV vaccination were selected. The selection was determined by the Google Trends platform, which showed the most common Portuguese Internet searches by users regarding HPV. This tool identifies the main terms used in Google search used by the user when using the search platform. The term HPV has been used, however it can evaluate any term
Websites were selected on 27 September 2019, using the following search tools: Google (http://www.google.com), Yahoo (http://www. yahoo.com), and Bing (http://www.bing.com). These are popular search engines on the World Wide Web, accounting for more than 90% of the searches performed.Citation7 The first 10 websites returned by each search engine were selected and amalgamated.
After selection, the sites were immediately downloaded using ScrapBook, a Firefox Internet browser extension (https://addons.mozilla.org/en-US/firefox/addon/scrapbook/), in order to save and annotate the sites in an offline format and to obviate potential updates during the review process.
The following exclusion criteria were applied when considering websites to be included in this study:
Directory, discussion, or forum websites
Websites that require a password
Websites that require payment
Websites with error messages
Video streaming websites
Websites on subjects beyond the scope of the research
Duplicate websites
These exclusion criteria were established because they envision the main accesses of the user and allow analyzing the contents presented
Data analysis
After applying the exclusion criteria, the remaining websites were selected. To analyze the general quality of the website content, the DISCERN tool was used.Citation9
The DISCERNCitation9instrument is designed to assess website content from the perspective of a health consumer or a patient. This tool is valid for any language to analyze the quality of published content It consists of 16 items rated on a continuous scale from 1 to 5 (1 = definitely no, 5 = definitely yes). A value between 1 and 5 (i.e., 2, 3, or 4) indicates that the element being assessed partially complies with the item.
The test is divided into two major dimensions: the reliability of a website and the quality of information focused on treatment options. The reliability dimension consists of 8 items, with a maximum score of 40. The quality of information dimension consists of 7 items, with a maximum score of 35. The total score therefore ranges from 0 to 75. Reliability tells as to whether the information is correct and quality of information if the text brings content of good scientific rigor and referenced
The test results are classified into three groups: low quality (scores between 15 to 30), moderate quality (31 to 74), and high quality (75). In addition, there is a 16th question that allows the evaluator to determine the overall quality of the site. lists the questions that are present in the DISCERN instrument.
Table 1. DISCERN questions.
The items from each test were applied to the selected sites. The frequency and percentage of the number of sites that served all items were also calculated.
The sites were evaluated independently by three experts, all of whom are specialists in obstetrics and gynecology. They had no contact with one another or knowledge of the grades assigned by others. The final grade was obtained by averaging the grades assigned by the three evaluators.
The sites were divided into four categories: laypersons, public institutions, private institutions, and scientific physicians. This division was determined according to specific common characteristics that allowed for comparison between them. The first group comprised texts on nonscientific platforms written by authors with no health-related training. The second group consisted of texts presented on public institution sites, such as the Ministry of Health, or State Health Departments. These publications were not specifically attributed to any healthcare professionals. In the third group, private institutions, publications from private laboratories or hospitals, for example, were included. These texts were not attributed to any specialist. The fourth group comprised medical-scientific sites. In this context, the published texts identified the health professional responsible for them and, in some cases, the references responsible for the construction of the site.
The data were analyzed using Pearson’s coefficient of variation. SPSS software (version 10.0) for Windows was used for data analysis. After the application of normality tests, the main differences between groups assessed by DISCERN were compared by analysis of variance (ANOVA). Statistical significance was set at P < .05.
Finally, a page authority (PA) tool was used. This tool establishes, from an equation on a logarithmic scale, the relative importance of the website in the context of the Internet. It is not related to the quality of the content posted, but to the number of times it is referenced, cited, or viewed. This scale is rated on a score from 0 to 100; a higher score indicates the relative importance or authority of the site, not the quality or correctness of the information. For survey purposes, access to sites with a higher PA is easier on the Internet. We sought to establish relationships between the DISCERN and PA scores to assess whether sites with greater access have better quality information.
Results
The five most used terms, as determined by Google Trends, were “HPV vaccine,” “What is the HPV vaccine for,” “Quadrivalent HPV vaccine,” “How many doses of HPV vaccine,” and “HPV vaccine reactions.”
We conducted searches for these five terms using the three search engines. After applying the exclusion criteria to the 150 sites returned, 64 remained for analysis, as shown in . The reasons for the exclusion of sites were as follows: 43 were duplicate sites, 22 were question forums, 18 sites entailed payment, and 3 were video streaming sites.
Figure 1. Flowchart showing division of sites according to inclusion and exclusion criteria.
shows the division of groups after we had evaluated the sites according to their specific and previously established characteristics.
Figure 2. Distribution by group of sites surveyed.
DISCERN
As illustrated in , the sites generally showed moderate reliability, quality, and total feature scores. Results for the medical-scientific websites across all dimensions were statistically superior, compared with that of the websites in the other groups (P < .01). When we compared the results of the layperson group, the public institutions group, and the private institutions group, similar scores were found and no statistically significant differences were found.
Figure 3. Average grades assigned to the reliability, quality, and total score dimensions.
PA
We used the PA score to rank the top 10 sites. These top 10 sites are shown in , whose last column shows the DISCERN score for each site.
Table 2. Top 10 sites ranked according to page authority.
DISCERN and PA correlation
Finally, the correlation between the DISCERN and PA scores was calculated. As shown in , the correlation was positive, but statistically weak.
Figure 4. Correlation analysis between sum of total DISCERN and PA scores.
Discussion
This study aimed to evaluate the quality of information published on the Internet in Portuguese regarding the HPV vaccine. Determining the main search terms and verifying whether the content has a sound basis can help lay users draw more accurate conclusions about the topic.
Three tools that are not commonly used in clinical practice in the medical community were used. They were Google Trends, the DISCERN website analysis instrument, and PA. The first helped to establish the behavior of lay users by verifying the main search terms used when the subject was the HPV vaccine. The second is a validated instrument that assesses the reliability and quality of the information in a text. PA is a ranking created from a logarithmic equation that determines the relative importance of a website. It does not assess the quality of information but reflects the relative authority and importance of the site, based on the number of times it is cited or referenced, among other factors. Using these tools in conjunction with traditional medical analysis allows exploring new dimensions in particular the content published outside sites that value technical rigor and scientificism.
It is interesting to note that one of the most used search terms according to Google Trends was “HPV vaccine reactions.” This suggests that the public, perhaps because of the dissemination of content that emphasized alleged adverse outcomes of inoculations, was very concerned about the side effects of the vaccine. During the search, cases of mass hysteria and blogs warning about the risk of autism after inoculation were encountered. An example of collective hysteria was a case that occurred in the Brazilian city Bertioga. A group of eleven girls became ill collectively after applying the vaccine in 2014. After investigations it was found that it would have no relation to the application of the immunization. This type of information confuses people and disrupts vaccination campaigns, not only for HPV, but also for the entire PNI. Using Google, Yahoo, and Bing allowed us to find a significant variety of sites that disseminated information in a more diversified way. An interesting balance was observed regarding the number of sites between the four proposed categories (laypersons, public institutions, private institutions, and medical-scientific). Each contributed approximately one-quarter of the total number of sites.
Overall, most texts contained moderate quality of information. Unfortunately, few high-quality texts were available. This shows that the general population lacks reliable sources of information regarding HPV. This lack of quality information can lead people to make incorrect immunization decisions. In comparing the four groups, as expected, the medical-scientific publications achieved a high DISCERN score. Most of these texts showed greater rigor in data and references and, consequently, contained more reliable information for the user. These publications showed statistically significant superior quality compared with the other three groups. In these other groups, lay people, public institutions, and private institutions showed lower quality of information according to the DISCERN scale. Their mean scores showed no statistically significant differences between them. The low score achieved by the lay publications can be attributed to the lack of scientific rigor in the texts and the absence of robust references. Regarding the category of public institutions, it is possible to infer that this lower average may be due to a lack of updates to the articles. Perhaps because of the difficulties inherent in public service, new information was posted less frequently. Finally, we also found low scores for the private institutions group. Analyzing these publications showed, for the most part, an economic bias. That is, they attempted to sell doses of the vaccine. This may have influenced the suppression of important and pertinent information on the topic.
Analyzing the 10 highest PA grades revealed interesting information. Most sites (6/10) were from the lay group. This demonstrates that most of the information that reaches the general public may not be the most accurate, since the authors lack the necessary scientific knowledge and rigor. Most of these texts do not contain references, or citations from experts. Furthermore, it was observed that two of these publications had significantly lower scores according to the DISCERN tool (33 and 23). These scores demonstrate that they were low-quality sites that might mislead the lay user, as with fake news sites. On the other hand, there were texts of much better quality in the top 10 PA-ranked sites. Three websites, two medical-scientific websites and one layman site, scored more than 50. This score indicates that these are more reliable sites that can transmit and provide consumers with more reliable information. In addition, it was possible to establish a correlation between the DISCERN and PA scores. Despite presenting a statistically significant correlation, the value of this correlation was weak. In other words, high-value sites, as per DISCERN, were not necessarily high value in terms of PA, showing that the best information does not always reach users.
There are very few studies on how the use of digital tools impacts public health quality. In Portuguese literature, there are no studies aimed at evaluating online information.
There however exists a differentiated analysis of the theme in the literature. In the studies by Dyda et al.Citation6 and Dunn et al.,Citation10 more general analyses of the Internet were undertaken. They did not, however, analyze the quality of the information itself, but correlated Twitter retweets by region, and compared the vaccination coverage of those locations. Regions that had a greater number of retweets tended to have better vaccination levels.
Rosen et al.Citation11 compared the medium of information dissemination and demonstrated its impact on adolescents’ knowledge. According to their study, the Internet served as a medium for the transmission of quality information.
This study’s main strength lies in the fact that it provides a method for analyzing existing information on the Internet and verifying its quality. This allows the determination of what information reaches the user more easily, and whether it helps them to make the correct decision which, in the case of this study, is to get vaccinated.
However, the study has some limitations, such as restricting itself to the Portuguese language, not establishing a global assessment of the topic, and not being able to assess the impact of social networks, such as Instagram and Twitter, on the topic. These platforms are also pervasive and the content they convey needs to be evaluated.
Nevertheless, this type of analysis has practical value. Establishing better information sites can help spread better-quality content and restrict the dissemination of lower-quality texts. Creating a tool that restricts access to low quality content, or even reduces the site’s PA rating, is possible. This study will, therefore, help users draw accurate conclusions on diverse topics.
An important research issue is determining how information impacts upon an individual’s decision making. The same text does not always lead to the same conclusions among different people. Understanding this process and using it in mass media allows the public to make better decisions about the mechanisms that involve public health and increase its levels in a given region. Furthermore, it is important to understand the influence of social networks and influencers in everyday life.
This study was an initial attempt at assessing the quality of Internet content and understanding how this information reaches the user. Strategies to improve vaccination coverage are multifactorial but establishing whether the information that reaches the average user is of quality is already an initial possibility to provide better content. Establish what has the best quality to read and what has not. More studies could help to assess the impact of social networks and how they influence the public. In addition, future studies need to create tools that favor access to superior-quality information on the Internet.
Conclusion
The HPV vaccine was the first and most effective immunobiological agent to protect against a type of cancer, i.e., cervical cancer. This neoplasm is quite prevalent and is responsible for the high morbidity and mortality in the female population of Brazil. Previous clinical trials have already shown this vaccine to be highly safe and immunogenic. It is ideal for mass vaccination. However, there are low levels of vaccination uptake in all age groups and genders, as was revealed by the PNI. The causes are multifactorial. Understanding them properly requires an analysis of several variables, including information received by the public through Internet websites.
Establishing the quality of information that reaches the user more easily helps to determine whether the public is provided with the correct data that help them make sound decisions, as in the present case of HPV vaccine. This is just one of the existing variables that help us understand low cooperation with an immunization program.
In general, from the analysis in this study, it can be concluded that, in most cases, the user has access to content that is not very scientific and could lead to wrong, or even dangerous conclusions and decisions.
Final considerations
This work was built on multidisciplinary cooperation, which should be increasingly encouraged. Such combination of knowledge allows growth in multifactorial analysis and thus generate new conclusions that contribute to the growth of knowledge and improvement in the quality of life.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Additional information
Funding
References
- Sasagawa T, Takagi H, Makinoda S. Immune responses against human papillomavirus (HPV) infection and evasion of host defense in cervical cancer. J Infect Chemother. 2012;18(6):1–6. doi:10.1007/s10156-012-0485-5.
- Winer RL, Lee S-K, Hughes JP, Adam DE, Kiviat NB, Koutsky LA. Genital human papillomavirus infection: Incidence and risk factors in a cohort of female university students. Am J Epidemiol. 2003;157(3):218–226. doi:10.1093/aje/kwf180.
- WHO/ICO Information Centre on HPV and Cervical Cancer (HPV Information Centre). Human papillomavirus and related cancers in world. Summary report 2018; [accessed 2021 Aug 20]. https://www.who.int/health-topics/cervical-cancer#tab=tab_1.
- Instituto Nacional de Câncer. Estimate 2020: incidence of cancer in Brazil/Instituto Nacional de Câncer José Alencar Gomes da Silva. Coordenação de Prevenção e Vigilância. Rio de Janeiro: INCA; 2017 [accessed 2021 Jun 5]. http://www1.inca.gov.br/inca/Arquivos/estimativa-2020.pdf
- Stanley M, Gissmann L, Nardelli-Haefliger D. Immunobiology of human papillomavirus infection and vaccination: implications for second-generation vaccines. Vaccine. 2008;26(10):K62–67. doi:10.1016/j.vaccine.2008.05.066.
- Dyda A, Shah Z, Surian D, Martin P, Coiera E, Dey A, Leask J, Dunn AG. HPV vaccine coverage in Australia and associations with HPV vaccine information exposure among Australian Twitter users. Hum Vaccin Immunother. 2019;15(7–8):1488–1495. doi:10.1080/21645515.2019.1596712.
- Do Nascimento NM, Souza HPS, Almentro EK, Jcsp L. Aplicando Transparência no Domínio de Informação Sobre Saúde na Internet: Uma Abordagem Visando o Cidadão. In: Workshop de transparência em sistemas (WTRANS), 6, 2018, Natal. Anais. Porto Alegre: Sociedade Brasileira de Computação; 2018. 10.5753/wtrans.2018.3097
- Ministério da Saúde. Informe técnico da ampliação da oferta das vacinas papilomavírus humano 6, 11, 16 e 18 (recombinante) – vacina HPV quadrivalente e meningocócica C (conjugada); [accessed 2021 May 10]. Informe-T–cnico-HPV-MENINGITE.pdfsaude.gov.br
- Kaicker J, Debono VB, Dang W, Buckley N, Thabane L. Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument. BMC Med. 2010;8(1):59. doi:10.1186/1741-7015-8-59.
- Dunn AG, Surian D, Leask J, Dey A, Mandl KD, Coiera E. Cervical cancer, human papillomavirus and vaccines: assessment of the information retrieved from general knowledge websites in Chile. Public Health. 2017 Feb 17;148:19–24.
- Rosen BL, Shew ML, Zimet GD, Ding L, Mullins TLK, Kahn JA. Human papillomavirus vaccine sources of information and adolescents’ knowledge and perceptions. Glob Pediatr Health. 2017;4:2333794X17743405. doi: 10.1177/2333794X17743405.