COVID-19 InfoVaccines: A WHO-supported educational project to promote COVID-19 vaccination information among professionals and the general population

ABSTRACT

COVID-19 vaccine uptake varied across countries, in part due to vaccine hesitancy fueled by a lack of trustworthy information. To help health workers provide evidence-based answers to common questions about COVID-19 vaccines and vaccination, and thereby, assist individuals´ decisions on vaccine acceptance, COVID-19 InfoVaccines, a joint WHO-EU project, was launched in February 2021 to support COVID-19 vaccine rollout in 6 Eastern European countries. COVID-19 InfoVaccines was made available in seven languages and shared on social media networks. A total of 262,592 users accessed COVID-19 InfoVaccines.com between February 11, 2021, and January 31^st, 2023. The users were most interested in: general questions; vaccine efficacy and duration of protection; vaccine safety; vaccine co-administration, and dose-interval and interchangeability; though the interest in a specific theme varied in function of the epidemiological situation. A total of 118,510 (45.1%) and 46,644 (17.7%) users scrolled up to 35% and 75% of the COVID-19 InfoVaccines webpage, respectively. The average engagement rate was 71.61%. The users accessed COVID-19 InfoVaccines from 231 countries and territories, but the majority were in Ukraine (N = 38,404; 14.6%), Spain (N = 23,327; 8.9%), and Argentina (N = 21,167; 8.1%). Older Facebook users were more interested in COVID-19 information than younger individuals (X² p-value < .0001). Two hundred twenty-eight videos were shared on YouTube. The average Click-Through-Rate on Facebook was 7.82%, and that on YouTube was 4.4%, with 60 videos having a Click-Through-Rate >5%, falling in the range of average YouTube video Click-Through-Rate (2% – 10%). As misinformation about vaccines and vaccination spreads easily and can negatively impact health-related decisions, initiatives like COVID-19 InfoVaccines are crucial to facilitate access to reliable information.

KEYWORDS:

• COVID-19
• COVID-19 InfoVaccines
• educational tool
• Europe
• misinformation
• World Health Organization
• vaccine
• vaccine hesitancy

Introduction

The COVID-19 pandemic has been a major threat to public health. By greatly reducing the risk of severe illness and death, vaccination against SARS-CoV-2 is the most powerful tool to combat the virus. Vaccination program success lies not only in making safe vaccines equitably available but also in ensuring their acceptance and uptake by the target population.

COVID-19 vaccine acceptance rates widely vary across countries. A meta-analysis of 172 studies from 50 countries estimated a pooled COVID-19 vaccine acceptance rate of 61%.¹ Determinants of vaccine hesitancy encompass, among others, socioeconomic, and psychosocial factors, including knowledge of the vaccine.² A shift in how patients seek health information has been reported for more than a decade, with more using the internet as a first source of information.^3,4 The quality of online information can vary from peer-reviewed online health information to personal blogs and opinions that are not fact-checked, raising the need for increasing the population’s awareness of trustful health information sources. Basch and colleagues concluded that most videos on vaccine information on YouTube are published by non-experts in vaccines, and that more than 65% of those videos discouraged vaccination.⁵ A study in the United States reported a significant correlation between the sentiments of social media (tweets) and vaccine acceptance.⁶

During the COVID-19 pandemic, social media has been particularly influential. A study undertaken in Philadelphia reported that individuals who use social media as the only source of vaccine information have double the chance of rejecting a COVID-19 vaccine.⁷ Anti-vaccination efforts on social media contribute to population beliefs on the safety of vaccinations. Disinformation campaigns are substantially associated with a drop in vaccination coverage over time, with each point shift upwards in a five-point disinformation scale associated with a two percentage point drop in mean vaccination coverage year over year.⁸ A randomized controlled trial in the United Kingdom and the United States found that recent misinformation lowers the intent to accept a COVID-19 vaccine by 6.2% points and 6.4% points in the United Kingdom and the United States, respectively, with “scientific-sounding” misinformation leading to a further decline in vaccination intent.⁹ Another study in the United States also found a negative association between misinformation and vaccination uptake rates.¹⁰ Information published through unscientific communications substantially influences socioeconomically vulnerable populations as they have difficulties finding and understanding health information.¹¹ Social media platforms vary in the degree of vaccine disinformation shared with the public.¹² If the information on social media originates from trusted sources, it can be a useful tool to increase and disseminate information on vaccines.⁷ A study undertaken in Thailand revealed that the quality of a health website affects the intention to use it, especially when users trust the website and perceive its usefulness.¹³ Accordingly, educational interventions, especially in increasing awareness of the benefits of vaccination and enhancing public confidence and trust in vaccination, play a fundamental role in minimizing vaccine hesitancy and thereby increasing uptake.²

In light of the speed in which evidence was developing on COVID-19 and the importance of making WHO evidence-based recommendations on vaccination available as widely as possible to both health professionals and the general public, the WHO Collaborating Center for Vaccine Safety at the University de Santiago de Compostela, together with the WHO Regional Office for Europe, developed Covid19infovaccines.com, a multimedia content website that offers extensive educational information in the form of scientific videos, podcasts, and text content related to COVID-19 vaccination and COVID-19 vaccines. COVID-19 InfoVaccines is partially funded through a European Union-WHO joint project to support COVID-19 vaccine deployment and vaccination in six countries in Eastern Europe.¹⁴ In this report, using data collected between February 2021 and January 2023, we aimed to analyze the adoption of COVID-19 Infovaccines as an educational tool. We also pretended to uncover the ways and means of reaching the COVID-19 Infovaccines website to help plan the dissemination of other educational tools in the future. Lastly, we examined COVID-19 Infovaccines content usefulness by exploring the users’ interest in a specific topic published in COVID-19 Infovaccines and the trend of interest in that specific topic in function of time and geographical location.

Methods

WHO collaborating center for vaccine safety

The WHO Collaborating Center for Vaccine Safety is located at the University Hospital Complex of Santiago de Compostela, Spain. It has been designated as a collaborating center since 2018, and its main function is to assist WHO in developing and updating training materials for health workers on vaccine safety, contraindications, and clinical management of adverse events following immunization; to contribute to WHO work in training of health workers on vaccine safety, contraindications, and introduction of new vaccines; and to contribute to WHO work in training national immunization technical advisory groups of experts (NITAGs) on developing recommendations on COVID-19 vaccines and vaccination strategies.¹⁵

COVID-19 InfoVaccines project creation

COVID-19 InfoVaccines was created by the WHO Collaboration Center for Vaccine Safety in collaboration with the WHO Regional Office for Europe primarily to improve the knowledge and increase the confidence of healthcare workers in advising their patients about COVID-19 vaccines in order to enhance vaccine acceptance and avoid missed opportunities to vaccinate patients due to unjustified contraindications. COVID-19 InfoVaccines secondary aim was to inform members of the general public looking for thorough evidence-based answers to common questions related to COVID-19 vaccines and associated WHO recommendations.

COVID-19 InfoVaccines was established on 11 February 2021 as a result of the need for comprehensive and up-to-date information for health professionals in the context of a rapidly evolving epidemiological situation and several new vaccines being developed and introduced globally. A working group was established with experts from the WHO Collaborating Center, the WHO Regional Office for Europe, several WHO country offices, and independent experts in various fields.

COVID-19 InfoVaccines working group consists of 15 multidisciplinary members from Czechia, Georgia, Romania, Russian Federation, Spain and Ukraine. The working group, co-led by an international expert in vaccines and infectious diseases (F M-T) from the WHO Collaborating Center, and a Regional Advisor of the Vaccine-preventable diseases and immunization program at the WHO Regional Office for Europe (S S-D), includes content creators, translators, dubbers, project coordinators, video editors, designers, website developers and communications professionals.

COVID-19 InfoVaccines content creation and update

The working group is responsible for selecting the evidence-based content most relevant to COVID-19 vaccination and that could be of interest to healthcare professionals and the general public.

By WHO request, the WHO Collaborating Center for Vaccine Safety trained hundreds of frontline medical workers from Azerbaijan and Georgia on all aspects of COVID-19 vaccines and vaccination via webinars in April, June and October 2021. In February 2022, the WHO Collaborating Center also conducted an online training on vaccine safety and contraindications against vaccination for leading clinicians and immunization program staff from Armenia, Azerbaijan, Belarus, Georgia, Kazakhstan, Kyrgyzstan, Republic of Moldova, Tajikistan, Ukraine and Uzbekistan. During the webinars, the most updated information on the value and safety of COVID-19 vaccines, and recommendations on contraindications against vaccination as well as the risk–benefit ratio of vaccinating patients with comorbidities and children were discussed. Questions raised by the participants were noted and informed in part by the selection of relevant content for COVID-19 InfoVaccines. Other sources of questions included media and individual requests received by the WHO Regional Office for Europe. All answers were developed based on available, robust scientific evidence and WHO recommendations, with input provided by all members of the working group as per their individual areas of expertise.

The content was made available in the form of questions and the answers were recorded in video format and provided as a downloadable PDF. In addition, the WHO Collaborating Center for Vaccine Safety collaborated with experts in hygiene, infectious disease prevention, immunization and vaccines to organize a series of expert opinion videos addressing common concerns about COVID-19 and vaccination. The expert opinion series was in the format of short videos, from 1 to 5 minutes in duration.

The process of content creation and publication is summarized in Figure 1.

Figure 1. Workflow of COVID-19 InfoVaccines content creation and diffusion.

COVID-19 InfoVaccines dissemination

To increase the utility and visibility of COVID-19 InfoVaccines, the website as well as the transcripts of each video were posted in seven languages: Czech, English, Georgian, Romanian, Russian, Spanish, and Ukrainian. The content was originally prepared in English and Spanish, and then translated and adapted to the other five languages. COVID-19 InfoVaccines was promoted within professional networks and on social media, namely via Instagram, Facebook, Twitter and Google Ads.

The English and Ukrainian versions of the videos were transmitted via specific channels on YouTube® (@covid19infovaccines and @covid-19infovaccinesUA respectively), while the Spanish and Russian versions were disseminated through Vimeo® (covid19infovaccines). The content was also made available as a guide entitled “Learn about COVID-19 vaccines and vaccination on the go” which can be downloaded in PDF format and printed. All videos can be shared by e-mail or via social media networks.

Data analysis

To explore the reach of COVID-19 InfoVaccines, metadata were drawn from Google Analytics, Google Data Studio, and YouTube Studio between 11 February 2021 and 31 January 2023. The data included the number of users over time; undertaken events such as page viewing, starting sessions and scrolling; engagement; acquisition; traffic channels; users´ geographic distribution; and content viewed.

User demographic characteristics were not available for webpage use due to restrictions in the data use agreement; however, it was available for Facebook and YouTube channels. When available, differences in COVID-19 InfoVaccines use according to gender and age were analyzed using a chi-squared (X²) test.

Results

COVID-19 InfoVaccines use

A total of 262,592 users accessed the COVID-19 InfoVaccines between February 11, 2021, and January 31, 2023. Around 46,000 users visited COVID-19 InfoVaccines more than once. The users undertook 3,258,588 events. A total of 118,510 (45.1%) and 46,644 (17.7%) users scrolled up to 35% and 75% of the COVID-19 InfoVaccines webpage, respectively. The average engagement rate was 71.61% with an average engagement time of 1 min. and 27 seconds.

The users were located in 231 countries and territories (Figure 2). Most were based in Ukraine (N = 38,404; 14.6%), Spain (N = 23,327; 8.9%), Argentina (N = 21,167; 8.1%), Venezuela (N = 11,478; 4.4%), Nicaragua (N = 10,625; 4.0%), Mexico (N = 10,473; 4.0%), Czechia (N = 9,932; 3.8%), the United States (N = 8,917; 3.4%), the United Kingdom (N = 7,985; 3.0%) and Latvia (N = 7,079; 2.7%). The most consulted languages on COVID-19 InfoVaccines were Spanish (N = 98,240; 37.3%), English (N = 61,482; 23.4%), Russian (N = 57,640; 22.0%) and Ukrainian (N = 15,940; 6.06%) (Figure 3).

Figure 2. Distribution of COVID-19 InfoVaccines users worldwide. The bubble size reflects the number of users, the largest the bubble is, the greatest the number of users. Most of the users were located in Ukraine, Spain, Argentina, Venezuela, Nicaragua, Mexico, Czechia, United States, United Kingdom and Latvia; 46 countries and territories had more than 1000 users, 16 provided between 986 and 505 users, while the rest of the 233 had less than 500 users.

Figure 3. Acquisition characteristics of COVID-19 InfoVaccines. a) distribution of the languages used by the visitors to consult COVID-19 InfoVaccines. b) distribution of the traffic channel by which the user was first acquired. c) distribution of the browsers used by COVID-19 InfoVaccines visitors. d) distribution of the devices used by COVID-19 InfoVaccines visitors.

User acquisition

Device and traffic channel

The majority of the users accessed COVID-19 InfoVaccines through mobile phones (N = 236,180; 90%); 9% (N = 23,331) accessed through desktop, while the rest used tablets. Following Android Webview (31%), Google Chrome (9%) was the most commonly used browser (Figure 3). The most frequent traffic channel by which the user was first acquired was direct search (25%) followed by paid social (18%), organic social (traffic from social media platforms) (18%), and organic search (unpaid listings on search engine results pages) (15%) (Figure 3) .

Organic Google search

A total of 43,877 organic Google search clicks drove the users to COVID-19 InfoVaccines. Users mainly inquired about COVID-19 vaccine doses, composition, and function. The most common search queries in English language were “janssen vaccine how many doses” (N = 1,441), “how many doses of johnson and johnson vaccine do you need” (N = 482), “what is considered fully vaccinated for COVID” (N = 434), “inactivated vaccine” (N = 396), “comirnaty bivalent” (N = 359), “protein-based vaccines” (N = 322), “sterilized vaccine” (N = 316), and “how do inactivated vaccines work” (N = 313).

Facebook

The organic Facebook page generated 1,097 followers, 12,128 lifetime post reach and 2,548,541 post interaction, an indicator of user engagement with the published content. To increase page visibility, 34 campaigns were launched on Facebook in English, Georgian, Russian Federation, Spanish, and Ukrainian. Eighteen of the 34 campaigns were undertaken exclusively in Ukraine by the WHO Country Office in Ukraine. Overall, the campaigns generated 38,849,849 impressions that reached 10,517,410 users, who clicked 3,037,098 times, generating a Click-Through-Rate (CTR) of 7.82%. CTR is a ratio of how often people who see an ad or free product listing end up clicking on it. Females were more likely than males to click on an advertisement on COVID-19 as it was reflected by click-to-reach (females: 33.12%; males: 20.41%; unknown: 29.59%; X² p-value <.0001) and CTR (females: 8.57%; males: 6.09%; unknown: 7.55%; X² p-value <.0001) ratios. Older Facebook users were also more interested in COVID-19 information than younger individuals (X² p-value <.0001). The click-to-reach ratio increased with age reaching 37.99% in users older than 65 years. Nonetheless, the CTR was slightly higher in individuals aged between 35 and 44 than in the older age categories, probably due to the higher activity of this age group on social media (Figure 4).

Figure 4. Statistics of COVID-19 InfoVaccines use through Facebook platforms. a) Distribution of clicks to reach ratio according to age. b) Distribution of clicks to reach ratio according to gender. c) Distribution of clickthrough rate according to age. d) Distribution of clickthrough rate according to gender.

Restricting the analysis to Facebook campaigns launched by WHO in Ukraine also revealed a higher CTR value for females (3.13%) than males (2.34%). In Ukraine, in individuals aged 25 or older, the CTR increased with age, with individuals older than 65 years having the highest CTR values (3.94%) compared to other age categories.

Stratifying the campaigns by trimestral period showed that the highest CTR values were obtained in the 2^nd trimester (May 16^th - August 14^th, 2021: CTR = 12.37%), 3^rd trimester: (August 15^th - November 13^th, 2021: CTR = 13.96%) and 4th trimester (November 14^th, 2021–February 12^th, 2022: CTR = 11.91%).

Youtube

COVID-19 InfoVaccines YouTube channel included 228 videos published in English with an overall number of views of 115,593. The average impressions CTR was 4.4%, with 60 videos having CTR greater than 5, falling in the range of average YouTube video CTR (2%–10%).^16,17 More than half of the audience were males (55.15%). Videos were more watched by older than younger adults (Figure 5). The most-viewed videos are summarized in Table 1. The audience was mainly in the United States (N = 20,393; 17.6%), Australia (N = 9,127; 7.9%), the Philippines (N = 6,678; 5.8%), Canada (N = 5,556; 4.8%) and the United Kingdom (N = 5,147; 4.5%) (Figure 5). About 43.0% of the audience did not use subtitles, while 42.0% used English subtitles. Eight hundred sixty-eight video shares and 702 video likes were registered.

Figure 5. Statistics of COVID-19 InfoVaccines adoption in YouTube. a) distribution of the number of views as a percentage of total views and the average view duration according to age. b) distribution of total watch time according to countries.

Table 1. Most-viewed videos on COVID-19 InfoVaccines YouTube channel.

Video title on youtube	Publishing data	Views	Average view %	CTR (%)
13_1 Is Novavax vaccine safe and effective?	Feb 20, 2022	20257	52.18	4.25
5_21 Could there be as yet unidentified side effects of the vaccines?	Dec 2, 2021	9442	39.43	6.36
5_7 Should a person who was previously infected with COVID-19 still be vaccinated?	Jan 11, 2022	5936	54.91	3.88
11_5 Why some countries haven’t resumed Vaxzevria vaccination campaigns despite WHO recommendation?	Jun 14, 2021	4344	40.89	3.86
5_17 Do mRNA vaccines cause myocarditis as an adverse reaction?	Oct 1, 2021	4050	41.99	2.74
12_2 How many doses of Janssen COVID-19 vaccine do I need to be protected?	Jan 11, 2022	3642	67.40	4.68
11_1 How does Vaxzevria, the Oxford-AstraZeneca COVID-19 vaccine work?	Apr 15, 2021	3454	58.86	3.7
5_22 Is there a limit to the number of vaccine doses a person can safely receive?	Feb 1, 2022	2845	61.41	4.06
1_9 How do protein based vaccines work?	Apr 15, 2021	2519	41.11	3.88
1_8 How do inactivated vaccines work?	Apr 15, 2021	2465	41.74	6.94
0_11 Are vaccines against COVID-19 equal or different?	May 3, 2021	2182	61.12	2.08
11_7 What are the early signals of potential blood clotting events following immunization that people	Jun 14, 2021	2022	63.92	3.05
5_16 Could a lump in the neck after receiving Pfizer’s or Moderna’s mRNA vaccine be linked to it?	Jul 17, 2021	1929	55.33	6.14
5_24 Can COVID-19 vaccines cause AIDS?	Apr 11, 2022	1560	52.14	12.52
3_1 How quickly does the vaccine work and how long does the protection last?	Jan 11, 2022	1501	76.10	5.54
11_4 Is Vaxzevria, the Oxford-AstraZeneca vaccine, safe?	Jan 11, 2022	1327	35.10	7.91
5_25 Do the COVID-19 vaccines weaken the immune system?	Apr 11, 2022	1312	50.89	13.2
11_4 Is Vaxzevria, the Oxford-AstraZeneca vaccine safe?	Sep 8, 2021	1264	31.32	7.07
12_1 How does the Janssen COVID-19 vaccine work?	Jul 17, 2021	1223	61.99	5.61
0_2 Vaccines protect against severe COVID-19 disease.	Apr 25, 2022	1169	58.56	7.73

The viewers mainly found the videos by browse features (30.5%), external search (27.4%), suggested videos (18.1%) and YouTube search (16.0%). About 65.1% (N = 73,016) viewers watched the videos on mobile phone, while 22.5% (N = 25,209) used computer, 7.2% (N = 8,805) used tablet and 5.1%(N = 5,730) used TV. The average view duration was the highest on TV (1 min., 14 sec.) and the lowest on mobile (1 min., 01 sec.).

Additionally, within the framework of promotional campaigns of COVID-19 vaccination in Ukraine the WHO Country Office in Ukraine launched a separate channel with Ukrainian dubbing. Covid-19infovaccinesUA YouTube channel includes 127 videos with an overall number of views of 1,359,144 (99% of which are from Ukraine). The average impressions CTR was 1.18%. More than half of the audience were males (56.8%). The audience aged 35 to 65+ accounted for 61% of viewers. The 10 most viewed videos are shown in Table 2. Only 19.4% of viewers used Ukrainian subtitles. The distribution by device showed that 53.7% (729,812) of viewers used TV, 25.6% (347,593) used mobile phone, 12,8% used computer (174,457) and 4.6% used tablet (62,412). The average view duration was the same for all devices.

Table 2. Most-viewed videos in “COVID-19 infovaccines Україна” YouTube channel.

Video title on youtube	Publishing data	Views	Average view %	CTR (%)
16_7 How can a person vaccinated in Ukraine obtain proof of COVID-19 vaccinations if he/she now resides outside of the country?	Sep 24, 2022	55410	75.5	0.8
16_5 Can a person receive routine vaccination for adults (e.g., Tdap vaccine against tetanus/diphtheria/pertussis) and COVID-19 vaccination during the same visit?	Sep 24, 2022	43214	87.5	0.4
6_5 Is there a maximum age for vaccination?	Nov 10, 2022	42330	96.4	0.8
4_8 Can COVID-19 and influenza (or other) vaccines be administered to a person during the same visit?	Nov 10, 2022	41230	84.7	1.1
5_22 Is there a limit to the number of vaccine doses a person can safely receive?	Nov 10, 2022	40015	95.8	-
16_3 If a COVID-19 ‘vaccination certificate’ obtained in Ukraine (or elsewhere) has expired, does it mean the protection against COVID-19 has worn off?	Sep 24, 2022	38927	89.4	1
2_24 If a vaccinated or unvaccinated person contracts COVID-19 should they be given antibiotics?	Dec 10, 2022	38314	91.3	-
16_3 What is the cost of COVID-19 vaccination abroad for people from Ukraine?	Sep 24, 2022	38223	98.3	2.5
3_20 Does vaccination prevent long COVID?	Dec 10, 2022	37997	97.6	2.6
16_6 If a person received one or more initial COVID-19 vaccine doses with a vaccine that is not currently available, should they complete the series with a different vaccine or wait until the original vaccine becomes available?	Sep 24, 2022	37773	84.3	0.8

Vimeo

Videos published in Vimeo COVID-19 InfoVaccines channel were seen 22,080 times and watched in full 7,300 (33.06%) times. The majority of these views took place in Ukraine (N = 5524; 25%), followed by Republic of Moldova (N = 1334; 6.0%), Latvia (N = 1015; 4.6%), Argentina (N = 949; 4.3%) and Georgia (N = 946; 4.28%). A total of 21582 (97.7%) views originated from the COVID-19 InfoVaccines webpage (covid19infovaccines.com). 17292 (78.3%) views took place on a mobile phone and 4323 (19.57%) were on a desktop. The most-viewed video on COVID-19 InfoVaccines Vimeo channel was “How does the Oxford-AstraZeneca COVID-19 vaccine work?” (N = 1,847; 1,261 views in English and 586 views in Russian).

Lifetime use of COVID-19 InfoVaccines

The lifetime user acquisition is presented in Figure 6. Before initiating the advertisement campaigns in April 2021, most of the new users accessed COVID-19 InfoVaccines through direct search. The traffic of new users through paid social channels varied in function of the periodicity of the advertisement campaigns (Figure 6); nonetheless, new users continued to adopt COVID-19 InfoVaccines through non-paid channels (direct, organic social, and organic search) (Figure 6).

Figure 6. Lifetime adoption of COVID-19 InfoVaccines represented in weekly units (W) and stratified by the traffic channel (direct, organic search, paid social, organic social and paid other). W1 refers to the first week of COVID-19 InfoVaccines launching, and successively until week 101 (W101).

The greatest adoption of COVID-19 InfoVaccines was observed between the weeks of June 20^th, 2021, and that of November 14^th, 2021 (total N = 104,041). Most of those users were in Ukraine (N = 15,859; 15.2%), Nicaragua (N = 9,368; 9.0%), Venezuela (N = 7,464; 7.2%), Mexico (N = 5,879; 5.7%) and Argentina (N = 5,302; 5.1%). The seven advertisement campaigns launched during that period in Czech, English, Russian, Spanish, and Ukrainian could have contributed to COVID-19 InfoVaccines adoption during that period. These campaigns generated a CTR value of 13.43%, which is higher than the average values for advertising campaigns in the health as well as education sectors for the year 2022.¹⁸

The organic Google search CTR was influenced by the marketing campaigns such as paid social ads. The CTR increased sharply on September 27^th, 2021 from 0% to 5.39% and reached a maximum of 22.90% on November 18^th, 2021, exceeding the average Google search CTR during the COVID-19 pandemic (5.90%).¹⁹ Subsequently, the CTR started to fluctuate according to advertisement intensity and the evolution of the COVID-19 pandemic (Figure 7).

Figure 7. Lifetime evolution of the clickthrough rate (CTR) of COVID-19 InfoVaccines.

COVID-19 InfoVaccines content

As of January 2023, the content of COVID-19 InfoVaccines was organized into 19 categories and videotaped after validation into almost four hours of video (Table 3). The content was regularly updated by removing videos with outdated information and updating others, and updating the downloadable PDF accordingly.

Table 3. COVID-19 InfoVaccines content (as of February 2023).

Number	Category	Videos – no. and duration
1	General questions	22 videos, 51 min
2	Vaccine and infection	7 videos, 11 min
3	Vaccine efficacy and duration of protection	15 videos, 24 min
4	Co-administration, dose-interval and interchangeability	3 videos, 4 min
5	Safety	19 videos, 29 min
6	Vaccine recommendations, precautions, or contradictions for special groups	30 videos, 40 min
7	Access, allocation and deployment	2 videos, 5 min
8	Regulatory approvals	3 videos, 5 min
9	Comirnaty® - Pfizer BioNtech vaccines	1 video, 3 min
10	mRNA 1273 – Moderna vaccine	1 video, 3 min
11	Vaxzevria – Oxford-Astrazeneca vaccine	7 videos, 14 min
12	Janssen COVID-19 Vaccine (Jonhson & Johnson)	4 videos, 6 min
13	Nuvaxovid/Covavax – Novavax vaccine	1 video, 3 min
14	Covaxin Vaccine	2 videos, 4 min
15	CanSino vaccine AD5-nCoV-S (Covidecia)	1 video, 2 min
16	COVID - 19 vaccination and the Ukraine crisis	13 videos, 17 min
17	Sinovac (CoronaVac) COVID - 19 vaccine	1 video, 3 min
18	Sinopharm BIBP COVID - 19 vaccine	1 video, 2 min
19	Valneva (VLA2001) vaccine	1 video, 2 min

Tables 4 and 5 respectively summarize the most visualized pages of COVID-19 InfoVaccines in English and Spanish, the most common languages used. Inquiries about a specific topic related to COVID-19 vaccination varied with time and location, probably due to changes in the epidemiological situation and the availability of new information about the pandemic and vaccines. For instance, in the 1st trimester (February 11^th - May 15^th, 2021) Spanish speakers were interested in “Should I do a test to check that I am protected after getting vaccinated?” while those who use the English language were curious about “How does the Oxford-AstraZeneca COVID-19 vaccine work?.” Certain inquiries such as “Why are some vaccinated people getting sick with COVID-19 and in some cases being hospitalized?” remained of high interest for more than one trimester. The number of views per page could be influenced by the public interest in a topic, the epidemiological situation as well as by the advertisement campaigns.

Table 4. Most-viewed content published in English in COVID-19 InfoVaccines, stratified by trimester period since the website launch in February 2021.

Period	Topic published in English	No. of views
1st trimester: February 11, 2021 - May 15, 2021	How does the Oxford-AstraZeneca COVID-19 vaccine work?	1,673
2nd trimester May 16, 2021 - August 14, 2021	How does the Oxford-AstraZeneca COVID-19 vaccine work?	1,213
	Can people still get COVID-19 after being vaccinated with one or both doses and can they transmit the virus to others?	974
	How quickly does the vaccine work and how long does the protection last?	843
3rd trimester: August 15, 2021 - November 13, 2021	How quickly does the vaccine work and how long does the protection last?	606
	How does the Janssen COVID-19 vaccine work?	464
	Could a lump in the neck after receiving Pfizer’s or Moderna’s mRNA vaccine against COVID-19 be linked to the vaccine?	436
4th trimester: November 14, 2021 - February 12, 2022	Could a lump in the neck after receiving Pfizer’s or Moderna’s mRNA vaccine against COVID-19 be linked to the vaccine?	3,493
	Could a lump in the neck after receiving Pfizer’s or Moderna’s mRNA vaccine against COVID-19 be linked to the vaccine?	1,515
5th trimester: February 13, 2022 - May 14, 2022	Why are some vaccinated people also hospitalized with COVID-19?	1,516
6th trimester: May 15, 2022 - August 13, 2022	Why are some vaccinated people also hospitalized with COVID-19?	3,881
	What are the early signals of potential blood clotting events following immunization that people should be aware of?	1,852
7th trimester: August 14, 2022 - November 12, 2022	What are the so-called COVID-19 bivalent booster vaccines?	4,377
	How many doses of Janssen COVID-19 vaccine do I need to be protected?	3,954
	Is there a minimum and maximum time interval between COVID-19 vaccine doses?	3,242
	Do the COVID-19 vaccines weaken the immune system?	2,622
	How do inactivated vaccines work?	2,525
	When should a person be considered fully vaccinated?	1,950
8th trimester: November 13, 2022 - January 31, 2023	How quickly does the vaccine work and how long does the protection last?	5,206
	Do the COVID-19 vaccines weaken the immune system?	5,147
	Is there a minimum and maximum time interval between COVID-19 vaccine doses?	4,892
	What are the so-called COVID-19 bivalent booster vaccines?	4,403
	How many doses of Janssen COVID-19 vaccine do I need to be protected?	3,442
	How do inactivated vaccines work?	2,312
	Why are some vaccinated people getting sick with COVID-19 and in some cases being hospitalized?	1,414
	To what extent can vaccines prevent people from getting infected with COVID-19?	1,399

Table 5. Most-viewed content published in Spanish in COVID-19 InfoVaccines, stratified by trimester period since the website launch in February 2021.

Period	Topic in Spanish [translation in English]	Views No.
1st trimester: February 11, 2021 - May 15, 2021	¿Debo hacerme un test para comprobar que estoy protegido después de vacunarme? [Should I do a test to check that I am protected after getting vaccinated?]	714
	¿Funciona la vacuna de AZD1222-AstraZeneca/Oxford en personas de edad avanzada? [Does the AZD1222-AstraZeneca/Oxford vaccine work in the elderly?]	498
2nd trimester May 16, 2021 - August 14, 2021	¿A partir de qué momento estoy protegido una vez que me he vacunado? [How quickly does the vaccine work?]	14,034
	¿Cuánto dura la protección de la vacuna? [How long does the protection last?]	11,657
	¿Cómo funcionan las vacunas de vectores como la de Astra-Zeneca? [How do vector vaccines work?]	11,416
	¿Cómo funcionan las vacunas de ARNm como la de Pfizer-BioNTech o Moderna? [How do mRNA vaccines, like Pfizer-BioNTech and Moderna COVID-19 vaccines, work?]	4,960
	¿Qué explicación tiene que una persona vacunada tenga una infección por COVID-19? [Can you be fully vaccinated against COVID-19 and get infected?]	4,312
	¿Son intercambiables las vacunas frente a COVID-19? [Can a person receive different vaccines for the first, second and/or booster doses?]	3,813
	¿Cómo es posible que se haya podido desarrollar una vacuna en tan poco tiempo? [How could vaccines be developed at record speed while still maintaining quality and safety?]	3,782
	¿Cuántas dosis de vacunas son necesarias? [How many doses of vaccines are necessary?]	3,513
	¿Qué eficacia tiene la vacuna AZD1222 de AstraZeneca/Oxford? [How effective is the AstraZeneca/Oxford AZD1222 vaccine?]	2,884
3rd trimester: August 15, 2021 - November 13, 2021	¿A partir de qué momento estoy protegido una vez que me he vacunado? [How quickly does the vaccine work?]	10,364
	¿Cuánto dura la protección de la vacuna? [How long does the protection last?]	9,713
	¿Cuántas dosis son necesarias de la vacuna Vaxzevria de AstraZeneca/Oxford? [How many doses of the AstraZeneca/Oxford Vaxzevria vaccine are needed?]	5,996
	¿Las nuevas variaciones del virus pueden interferir en la eficacia de la vacuna? [Are COVID-19 vaccines effective against new variants of concern of the SARS-CoV-2 virus?]	3,626
	¿Qué explicación tiene que una persona vacunada tenga una infección por COVID-19?	3,353
	¿Qué eficacia tiene la vacuna AZD1222 de AstraZeneca/Oxford? [How effective is the AstraZeneca/Oxford AZD1222 vaccine?]	3,249
	¿Cómo es posible que se haya podido desarrollar una vacuna en tan poco tiempo? [How could vaccines be developed at record speed while still maintaining quality and safety?]	2,908
	¿Cómo funcionan las vacunas de vectores como la de Astra-Zeneca? [How do vector vaccines work?]	2,569
	¿Deberíamos asustarnos por los alegatos de eventos graves después de las vacunas COVID-19? [Should we be scared by claims of serious events after COVID-19 vaccines?]	2,350
	Cuánto dura la protección de la vacuna de AZD1222 de AstraZeneca/Oxford? [How long does the AstraZeneca/Oxford AZD1222 vaccine last?]	2,345
4th trimester: November 14, 2021 - February 12, 2022	¿Por qué Omicron ha sido denominado una variante de preocupación? ¿Puede afectar a las personas ya vacunadas? [Why has Omicron been designated a variant of concern, and can it affect people who are already vaccinated?]	1,748
	¿Qué significa que una vacuna sea esterilizante? [Can COVID-19 vaccination lead to impotence?]	1,694
5th trimester: February 13, 2022 - May 14, 2022	¿Cuáles son los primeros indicios de los eventos posibles de coagulación sanguínea después de la inmunización que las personas deben tener en cuenta? [What are the early signals of potential blood clotting events following immunization that people should be aware of?]	1,753
	¿Por qué algunas personas vacunadas también están hospitalizadas con COVID-19? [Why are some vaccinated people getting sick with COVID-19 and in some cases being hospitalized?]	1,313
	¿Necesitamos actualmente las dosis adicionales de las vacunas frente a la COVID-19 para las personas vacunadas con la pauta completa? [Is a COVID-19 booster shot needed?]	1,159
6th trimester: May 15, 2022 - August 13, 2022	¿Cuáles son los primeros indicios de los eventos posibles de coagulación sanguínea después de la inmunización que las personas deben tener en cuenta? [What are the early signals of potential blood clotting events following immunization that people should be aware of?]	4,127
	¿Por qué algunas personas vacunadas también están hospitalizadas con COVID-19? [Why are some vaccinated people getting sick with COVID-19 and in some cases being hospitalized?]	3,926
	¿Existe algún número límite de dosis de vacunas que puede recibir una persona de modo seguro? [Is there a limit to the number of vaccine doses a person can safely receive?]	3,055
	¿Pueden las vacunas ARNm causar la miocarditis como el efecto adverso? [Do mRNA vaccines cause myocarditis as an adverse reaction?]	2,005
	¿Pueden ser vacunadas las personas inmunocomprometidas? [Can immunocompromised people be vaccinated?]	1,461
	Las vacunas protegen contra la enfermedad grave de COVID-19. [Vaccines protect against severe COVID-19 disease.]	1,387
	¿Cuánto rápido funcionan las vacunas frente a la COVID-19 y cuánto dura la protección? [How quickly does the vaccine work and how long does the protection last?]	1,335
7th trimester: August 14, 2022 - November 12, 2022	¿Existe algún número límite de dosis de vacunas que puede recibir una persona de modo seguro? [Is there a limit to the number of vaccine doses a person can safely receive?]	3,215
	¿Cuánto rápido funcionan las vacunas frente a la COVID-19 y cuánto dura la protección? [How quickly does the vaccine work and how long does the protection last?]	1,732
	¿Necesitamos actualmente las dosis adicionales de las vacunas frente a la COVID-19 para las personas vacunadas con la pauta completa? [Is a COVID-19 booster shot needed?]	1,406
	¿Cuáles son los primeros indicios de los eventos posibles de coagulación sanguínea después de la inmunización que las personas deben tener en cuenta? [What are the early signals of potential blood clotting events following immunization that people should be aware of?]	1,372
	¿Se puede estar vacunado completamente frente a la COVID-19 e infectarse? [Can you be fully vaccinated against COVID-19 and get infected?]	1,229
8th trimester: November 13, 2022 - January 31, 2023	¿Cuánto rápido funcionan las vacunas frente a la COVID-19 y cuánto dura la protección? [How quickly does the vaccine work and how long does the protection last?]	1,585
	¿Qué reacciones adversas están asociadas con la vacuna mRNA-1273 de Moderna? [What adverse reactions are associated with Moderna’s mRNA-1273 vaccine?]	1,574
	¿Se puede estar vacunado completamente frente a la COVID-19 e infectarse? [Can you be fully vaccinated against COVID-19 and get infected?]	1,040

The videos were categorized into themes on the main page “COVID-19 vaccines and vaccination explained.” The users were most interested in the following themes: general questions; vaccine efficacy and duration of protection; vaccine safety; vaccine co-administration, dose-interval and interchangeability; Janssen COVID-19 Vaccine (Johnson & Johnson); vaccine and infection; and Vaxzevria – Oxford-AstraZeneca vaccine.

The compiled document “COVID-19 vaccines and vaccination explained,” which includes scripts of all videos address common questions about COVID-19 vaccines was downloaded 3,879 times.

The series of experts’ opinions collected a total of 36,378 views. It encompassed the following eight topics between May 2 and November 1, 2021, that are listed by descending number of views: 1) Can you be fully vaccinated against COVID-19 and get infected? (9,731 views), 2) Will we need further COVID-19 vaccine doses? (8,072 views), 3) Are vaccines against COVID-19 equal or different? (7,497 views), 4) The paradox of vaccination success and opposition to vaccination (3,259 views), 5) Safety of vaccines: a question of understanding risks and statistics (2,932 views), 6) How can we deal with vaccine hesitancy? (2,403 views), 7) What important lesson should we have learned from this pandemic for future pandemics? (1,686 views), and 8) How does COVID-19 pandemic influence vaccine value perception? (798 views).

COVID-19 InfoVaccines includes a portal dedicated to WHO questions and answers (WHO Q&A), and that links a user´s specific inquiry on COVID-19 infection and vaccination to the specific portal on the WHO website. The website also addresses common questions related to COVID-19 and vaccination among vulnerable populations, such as people with cancer, AIDS, diabetes and stroke patients, as well as questions about health risks like antibiotic resistance. The “Featured” section provides specific portals linking to the international days of various underlying health conditions and the corresponding content in COVID-19 InfoVaccines. For example, the portal of “December 1st, World AIDS Day” brings the user to content related to COVID-19 and vaccination in AIDS and immunocompromised individuals such as “How quickly does the vaccine work and how long does the protection last?” and “Can COVID-19 vaccines cause AIDS?.” The portal of World Diabetes Day links to content like “Is COVID-19 vaccination safe and recommended for people who have diabetes?” and “Why are people with diabetes a priority group for COVID-19 vaccination?.”

Discussion

COVID-19 InfoVaccines initiative represents a vital intersection of public health advocacy, digital innovation, and global collaboration. The data-driven insights derived from website analytics may help to guide future endeavors in health communication, emphasizing the need for responsive, accessible, and reliable digital health resources.

COVID-19 InfoVaccines is a multimedia content website that was created as a digital initiative supported by the WHO Regional Office in Europe to serve as trustful information on COVID-19 and related vaccinations. It was initiated to help provide scientific-based information on COVID-19 vaccines presented by health professionals who have expertise in vaccines and infectious disease prevention. COVID-19 InfoVaccines was shared on the most commonly used social media applications Facebook and YouTube, which count over 2.9 billion and 2.5 billion active users according to 2022 statistics.²⁰ The website reached five continents, yet most of the users were in Ukraine, Spain, Argentina, Venezuela, Nicaragua, Mexico, Czechia, the United States, the United Kingdom, and Latvia; 46 countries and territories had more than 1000 users, 16 provided between 986 and 505 users, while the rest of the 233 had less than 500 users. Social media statistics revealed that COVID-19 InfoVaccines was well-received by the population based on CTR values.^18,19 COVD-19 InfoVaccines could have reached more users that cannot be measured, such as those who viewed the content by co-viewing, which has become a trend.²¹

Several factors underpin COVID-19 InfoVaccines’ adoption and diffusion: 1) initiation by a trustworthy source – WHO, together with the WHO Collaborating Center for Vaccine Safety; 2) variety and continuous updating of content; 3) availability in six languages; 4) accessibility, as it was designed to be able to be used on any device, especially mobile phones; and 5) advertisement campaigns, which helped attract a large fraction of the users.

COVID-19 InfoVaccines has been a useful tool in informing more than 200,000 individuals about COVID-19 vaccination. The tool, in combination with training courses undertaken by the WHO and the WHO Collaborating Center, helped to increase frontline doctors’ knowledge of vaccine safety and contraindications against vaccination and improved their confidence in recommending vaccination to their patients. The utility of the tool is ongoing as the general public and healthcare professionals need to be continuously informed on evolving evidence and recommendations related to COVID-19 vaccines including safety and effectiveness, protection duration, and new vaccine formulations.

The COVID-19 InfoVaccines website emerged in the context of misinformation and infodemics,²² illuminating a path for over quarter-million users seeking trustworthy information on COVID-19 vaccines. The detailed analytics of the website’s use reveal profound insights into the public’s engagement with digital health information sources, transcending geographic boundaries. Notably, the device and access trends – with a predominant use of mobile phones and discovery via organic Google search – underscore the public’s preference for readily accessible and searchable health information. The tendency toward mobile access also highlights the importance of optimizing health communication for mobile platforms to enhance reach and user engagement. Furthermore, the specific inquiries users made reveal a dynamic landscape of vaccine-related concerns that correlate with the evolving epidemiological situation, demonstrating the website’s role as a responsive and timely health resource.

Our findings that most users accessed the site via mobile devices and that older individuals engaged more with COVID-19 information on Facebook have crucial implications for tailoring communication strategies. The viewership patterns indicate that while the site was effective in reaching a broad audience, the video content had a significant impact, particularly among older users. This engagement with multimedia content highlights the importance of diverse content formats in health communication strategies.

The COVID-19 InfoVaccines website has functioned not only as an informational resource but also as a means for public health engagement, shaping the vaccine discourse in a rigorous scientific-based manner. The meticulous effort to present content in various languages and formats reflects a strong commitment to global health education and an understanding of the diverse needs of the website’s international audience. From these insights, the efforts of numerous individuals under this WHO-Europe initiative in maintaining and updating the COVID-19 InfoVaccines website seem well-justified. However, the discussion should also reflect on the resource allocation for such initiatives, with substantial human effort and funding behind the COVID-19 InfoVaccines project.

The pandemic has demonstrated the critical role of digital health tools in public health responses, and although the success of COVID-19 InfoVaccines reiterates the value of reliable online health resources, the magnitude of the impact of this initiative has to be carefully balanced against the important number of resources involved in the performance. The affirmative engagement metrics and widespread international use suggest that the website has been a valuable tool in combatting vaccine misinformation, and the website has likely contributed to informed decision-making among individuals considering COVID-19 vaccination. The success of such endeavors should be evaluated not only by the number of hits or views but also by the cascade effects this information may actually have when reaching the appropriate stakeholders and by their contribution to the overarching public health goal of increasing vaccine acceptance and uptake.

Conclusions

The endeavor to develop and maintain the COVID-19 InfoVaccines project, a collaborative effort between the WHO Collaborating Center on Vaccine Safety and the WHO Regional Office for Europe, has proven to be a substantial yet worthwhile undertaking. The analysis indicates that the website has effectively served its purpose of disseminating evidence-based information, as reflected by high engagement rates and widespread reach. The project’s ability to adapt content in real-time to the changing landscape of the pandemic has underscored the intrinsic value of such digital tools in public health communication.

The success of the COVID-19 InfoVaccines website lends credence to the potency of digital platforms in addressing public health challenges. However, it also opens a dialogue about the allocation of resources and the efficiency of digital campaigns. The substantial investment in creating and promoting accurate health information via the COVID-19 InfoVaccines website is a testament to the commitment to public health education and the fight against misinformation.

As the digital age ushers in a new paradigm for health communication, the experiences and lessons learned from COVID-19 InfoVaccines provide valuable insights. These should inform future strategies to enhance the reach and impact of online health resources, ensuring they remain a cornerstone of public health initiatives in the ongoing and future health crises.

Author contributions statement

Narmeen Mallah: investigation; literature review; data analysis and interpretation and writing original draft of the manuscript. Jacobo Pardo-Seco: Figures. Irene Rivero-Calle, Ouhao Zhu-Huang, María Fernández Prada, Catharina Reynen-de Kat, Oleg Benes, Liudmila Mosina, Siddhartha Sankar-Datta, Olga Aleksinskaya, Vusala Allahverdiyeva, Yevgenii Grechukha, Tamara Jobava, Mariia Savchyna, Pavla Kortusova, Ioana Novac: investigation; literature review; resources; and project administration. David Díaz: software; and resources. Federico Martinón-Torres: conceptualization; investigation; literature review; supervision; resources; funding; and project administration. All authors reviewed and revised the manuscript, approved it for publication and count responsible for its content.

Disclosure statement

Irene Rivero-Calle reports a relationship with GSK that includes speaking and lecture fees. Irene Rivero-Calle reports a relationship with Pfizer Inc that includes speaking and lecture fees. Irene Rivero-Calle reports a relationship with Sanofi Pasteur Inc that includes speaking and lecture fees. Irene Rivero-Calle reports a relationship with MSD Vaccins that includes speaking and lecture fees. Federico Martinon-Torres reports a relationship with GSK Vaccines SRL that includes consulting or advisory. Federico Martinon-Torres reports a relationship with Pfizer Inc that includes consulting or advisory. Federico Martinon-Torres reports a relationship with Sanofi Pasteur Inc that includes consulting or advisory. Federico Martinon-Torres reports a relationship with Janssen Pharmaceuticals Inc that includes consulting or advisory. Federico Martinon-Torres reports a relationship with MSD that includes consulting or advisory. Federico Martinon-Torres reports a relationship with Seqirus Pty Ltd that includes consulting or advisory. Federico Martinon-Torres has received support for the present work from the Instituto de Salud Carlos III (Proyecto de Investigación en Salud, Acción Estratégica en Salud): Fondo de Investigación Sanitaria (FIS; PI070069/PI1000540/PI1601569/PI1901090) del plan nacional de I+D+I and ‘fondos FEDER’ and Proyectos GaIN Rescata-Covid_IN845D 2020/23 (GAIN, Xunta de Galicia). Irene Rivero-Calle has acted as a sub-investigator in randomized controlled trials of Ablynx, Abbot, Seqirus, Sanofi Pasteur MSD, Sanofi Pasteur, Cubist, Wyeth, Merck, Pfizer, Roche, Regeneron, Jansen, MedImmune, Novavax, Novartis and GSK.

Additional information

Funding

This work was supported by Framework Partnership Agreement between the Consellería de Sanidad de la XUNTA de Galicia and GENVIP-IDIS-2021–2024 (SERGAS-IDIS March 2021; Spain); and consorcio Centro de Investigacion Biomédica en Red de Enfermedades Respiratorias (CB21/06/00103; F.M-T), DIAVIR (Instituto de Salud Carlos III (ISCIII)/DTS19/00049/Cofinanciado FEDER; Proyecto de Desarrollo Tecnologico ´ en Salud), Resvi-Omics (Instituto de Salud Carlos III (ISCIII)/PI19/01039/Cofinanciado FEDER), BI-BACVIR (PRIS-3; Agencia de Conocimiento en Salud (ACIS)—Servicio Gallego de Salud (SERGAS)—Xunta de Galicia; Spain), Programa Traslacional COVID-19 (ACIS—Servicio Gallego de Salud (SERGAS)—XUNTA de Galicia; Spain) and Axencia Galega de Innovacion ´ (GAIN; IN607B 2020/08—XUNTA de Galicia; Spain) [A.S]; and ReSVinext (Instituto de Salud Carlos III (ISCIII)/PI16/01569/Cofinanciado FEDER), Enterogen (Instituto de Salud Carlos III (ISCIII)/PI19/01090/Cofinanciado FEDER), OMI-COVI-VAC (PI22/00406/Cofinanciado FEDER, Grupos de Referencia Competitiva (IIN607A2021/05) and Axencia Galega de Innovación (GAIN; IN845D 2020/23—Xunta de Galicia; Spain) [F.M-T].