Catch the tweet to fight the flu: Using Twitter to promote flu shots on a college campus

Abstract

Objective: Over the 2018–2019 flu season we conducted a randomized controlled trial examining the efficacy of a Twitter campaign on vaccination rates. Concurrently we investigated potential interactions between digital social network structure and vaccination status. Participants: Undergratuates at a large midwestern public university were randomly assigned to an intervention (n = 353) or control (n = 349) group. Methods: Vaccination data were collected via monthly surveys. Participant Twitter data were collected through the public-facing Twitter API. Intervention impact was assessed with logistic regression. Standard network science tools examined vaccination coverage over online social networks. Results: The campaign had no effect on vaccination outcome. Receiving a flu shot the prior year had a positive impact on participant vaccination. Evidence of an interaction between digital social network structure and vaccination status was detected. Conclusions: Social media campaigns may not be sufficient for increasing vaccination rates. There may be potential for social media campaigns that leverage network structure.

Keywords:

• Seasonal influenza
• vaccination
• Twitter
• randomized controlled trial
• social network analysis

1. Introduction

Seasonal influenza is a major annual health burden that causes significant mortality and morbidity in the United States each year.¹ While the flu does not typically cause significant mortality in college-aged (18–24) individuals, this group can suffer high morbidity.^2–4 Contracting the flu negatively impacts a college student’s academic performance, social life, and ability to work. Outbreaks of seasonal influenza on a college campus also have the potential to rapidly spread given that many students inhabit densely populated shared living spaces, and have close proximity to others in classrooms and dining areas.^5–7 Undergraduates can serve as disease conduits to at-risk populations as the flu spreads from October to May,⁶ a period including winter break when many students come into contact with groups that suffer high mortality from the illness.

There has been great interest in understanding the vaccination behaviors of the US undergraduate population,^8–19 likely due to their reportedly low vaccination rate (found to be anywhere from 8–30%^15,20–22 of the student body). Cross-sectional survey techniques have illuminated social, psychological, and physical barriers to increased undergraduate vaccination.^{8,10,13,15,17,23,24} However, few published studies have probed the effectiveness of various interventions on collegiate vaccination rates,^{9,11,12,18,19,25} and of those even fewer have used participant vaccination outcome as the variable of interest.^9,11,18,19

Concurrently, there has been a surge in the presence of social networking services (SNS), with utilization increasing from 36% of US adults in 2009 to 72% in 2019 (70% to 90% utilization in college-aged Americans).²⁶ Researchers have leveraged this digital facet of life to fight influenza by developing flu surveillance methods using Twitter^27–31 or Google Trends^30,32–35 data, as well as demonstrating market segmentation possibilities through tweet sentiment analysis.^36–38 Only a handful of studies have examined SNS as platforms for public health intervention,^11,19 and in these studies SNS were often combined with offline interventions, making isolating the impact of the SNS difficult if not impossible. In the current literature there is little to no work directly examining the effectiveness of interventions utilizing SNS to raise vaccination rates, despite the presence of SNS in existing (CDC) campaigns.^39,40

The network aspect of SNS adds a wrinkle to this form of public health intervention not present in promotional interventions on traditional media sources like television and radio. The effect of offline social networks on flu vaccination choice has been previously demonstrated,^14,41–43 and digital social networks have been shown to be assortative according to vaccination sentiment.³⁶ Whether these digital networks are heterogeneous with respect to user vaccination status has not yet been explored. Assortative mixing of vaccination status on such networks has the potential to impact the effectiveness of a social media campaign.

In the 2018–2019 flu season we conducted a randomized controlled trial to investigate the efficacy of a social media campaign aimed at raising influenza vaccination uptake among undergraduates while probing the potential of digital social networks for future interventions.

2. Materials and methods

2.1. Study design

Our study was conducted from October 2018 to May 2019 at a large midwestern public university. Students were recruited over a one month period from August 2018 to September 2018. This design received approval from the institution’s Institutional Review Board (IRB) in the Spring of 2018.

2.1.1. Enrollment and participant demographics

The target population for this study consisted of all undergraduate students registered in the 2018 autumn semester who had a Twitter account and were over the age of 18. Of the target population, we enrolled n = 702 participants through an email solicitation sent to all 46,820 enrolled undergraduates in the autumn of 2018. Low recruitment rates have been reported for email solicitations for other studies.^44,45 Difficulty distinguishing legitimate content from spam is a challenge for email recruitment,⁴⁴ and may have played a role in the low response rate here. Students that expressed interest were sent a digital informed consent form.

Participants were randomized into an intervention group (n = 353) or a control group (n = 349). This split was stratified based on their self-reported gender and response to the question “Are you required to get a flu shot?” which we consider their self-reported flu requirements for the 2018–2019 flu season.

2.1.2. Intervention and control procedures

Intervention group members were asked to follow an intervention Twitter account that posted near daily tweets (1.24 tweets per day) promoting flu vaccination. In addition to direct tweet exposure, campaign engagement was incentivized with prize raffle entries. For each month of the study, an intervention group member could receive one raffle entry (up to seven over the study) by retweeting one of the promotional tweets, or by constructing their own tweet containing a hashtag that was unique to the campaign. In contrast, we requested that control group participants follow a control Twitter account that tweeted no content.

Indirect exposure is possible in this design. At the time of our study it was impossible to completely isolate content without also removing the main method of campaign engagement, retweeting. In order to emulate a realistic social media campaign we elected for a design that permitted indirect exposure and attempt to control for it in our analysis.

Monthly check-in surveys tracked flu vaccination status over the study period. Intervention group participants were incentivized to respond to these surveys with an additional prize raffle entry per response (a possible seven entries in addition to the seven possible entries as retweet incentives, for a total of fourteen maximum). Control group participants could receive two prize raffle entries per check-in survey response (up to fourteen maximum).

2.1.3. Intervention campaign content

Our campaign presented a mixture of comedic and informational content. All posts were created by the corresponding author. Any substantive information regarding seasonal influenza or the yearly flu shot was sourced from the CDC influenza website, https://www.cdc.gov/flu/. Comedic content leveraged tweet and meme formats that were popular among the intervention group. Popularity was anecdotally determined by browsing the Twitter feed of the intervention account. Informative posts were constructed by consulting the aforementioned CDC flu website and news sources found to contain relevant factual information. In an additional attempt to tailor the campaign to our audience we included content specific to their university. For example, a rendition of the school’s mascot kicking an influenza virus was utilized as the intervention account avatar.

Starting in December 2018 the intervention account also began tweeting an image unrelated to flu shots at the end of each month. This was done to measure actual exposure to the campaign across both groups. All images were found by scrolling to the bottom of Google images search results, so it was unlikely they would have independently appeared outside of the study account posts.

2.1.4. Data collection

All surveys were distributed via email and administered with Qualtrics software.

Each month of the observation period participants were sent check-in surveys in which they indicated whether they received a flu shot in the previous month. Starting in December each survey also contained a CAPTCHA-like⁴⁶ question that presented respondents with four images, and then asked them to select any they recalled seeing in their Twitter feed. One of the images was tweeted multiple times by the intervention account in the week prior to the check-in survey (see Sec. 2.1.3), the other three were of a similar theme.

Participant Twitter data was scraped on a weekly basis. Data was obtained using the Tweepy Python Twitter API Wrapper version 3.6.0 that was run using Python version 3.6.9. Collected data included the tweets, followers, and accounts they followed, from here on referred to as their friends, for each participant.

To augment self-reported flu vaccination data, the univeristy’s Student Health Center (SHC) provided flu vaccination records of all participants over the course of the study. Only a small percentage of the student body receives a vaccine at SHC each year, so this data cannot be used to directly analyze the impact of the campaign on the intervention group. However, we do calculate what percentage of the participants that reported getting vaccinated at SHC actually got vaccinated at SHC, which will provide some measure of desirability bias.⁴⁷

2.2. Data analysis

2.2.1. Missing data

Reliance on self-reported vaccination status led to missing outcome data for a non-negligible portion of the study population.

In addition to the “Since <first of the past month > have you received a flu vaccination?” question standard to all monthly check-in surveys, our final survey contained the question “Did you get a flu shot at any point since October 1, 2018?”. Combining the responses to that question with our standard flu shot inquiry allowed us to classify each participant as having gotten a flu shot over the course of the study, not having gotten a flu shot over the course of the study, or having missing data. Participants were classified as follows:

• Getting a flu shot if they answered “Yes” on any of the monthly surveys or they responded “Yes” to ever receiving a flu shot question in the final survey,

• Not getting a flu shot if they answered “No” to all of the monthly surveys, or if they answered “No” to the flu vaccination question added in the final survey while not indicating “Yes” in any of the monthly check-ins,

• Missing if they did not answer “Yes” on any of the monthly surveys, had some missing monthly survey responses and did not respond to the flu vaccination question added in the final survey.

Classification in this manner led to 156 (22.2%) participants with missing outcome data, of which 94 (26.6%) were in the intervention group and 62 (17.7%) were in the control. Unsure responses to demographic data survey questions were treated as missing data as well.

Missing values were not missing completely at random,⁴⁸ and thus we could not ignore them in any analysis. Multiple imputation,⁴⁹ performed in R⁵⁰ version 3.5.3 with the Multivariate Imputation by Chained Equations (mice) package⁵¹ was used to create 50 unique data sets with complete data. Statistical models discussed below were fit to each of the imputed data sets, and results were averaged for final analysis.

2.2.2. Analysis of intervention impact on vaccination outcome

If a participant indicated receiving a shot, they also identified a date in which they received that shot. Anyone that received a shot prior to the start of the intervention (October 1, 2018) was not included in the vaccination outcome analysis. This disqualified 75 of our participants, leaving 627 (315 in the intervention, 312 in the control) observations. The 156 participants missing vaccination outcome were not disqualified in this step.

Impact on vaccination rate was analyzed with a series of logistic regression models. We present four models, all of which have vaccination outcome as the dependent variable. Following Hsieh et al.⁵² we calculated the minimum percentage point increase we would be able to detect at 95% confidence and 80% power for each model. As a baseline we utilized a vaccination level of 46% for the control group, which was chosen based on the self-reported vaccination level of the cohort in 2017–2018.

Models 1 and 2 constitute an intent-to-treat analysis, where the only treatment-related independent variable included was study group. These models were fit using the data from all 627 participants that were not disqualified. Model 1 is a parsimonious model containing an intervention group indicator and indicators for receiving a flu shot last year and being required to get a shot. Model 2 then adds in demographic variables.

Models 3 and 4 assess the impact of campaign engagement and control for indirect campaign exposure. Data used to estimate these models were restricted to the 564 individuals (287 intervention and 277 control) for which we collected Twitter data. Twitter data collection was not possible for any participant that had a private account and did not accept a follow request from one of our study accounts. Of the 564, 23.6% were missing vaccination outcome (28.2% intervention and 18.8% control).

Model 3 is the same simple model as Model 1, restricted to this subset of participants. Model 4 adds in several Twitter activity variables including an indicator of whether or not a participant followed the intervention account, the total number of check-in pictures a participant reported seeing, an indicator of whether or not a participant tweeted our campaign hashtag, and an indicator as to whether or not the person posted a tweet containing the word “health” over the course of the study. The first three of these variables control for campaign exposure, while the last variable is meant to serve as a measure of the health mindfulness of the online personas of our participants.

2.2.3. Digital social network types

The interaction between digital social network structure and participant vaccination outcome was examined using three distinct network types.

The participant direct followership network is the subset of the entire Twitter follower network comprised of the study participants and the edges between them. In particular, in this network there is an edge from participant A to participant B if A follows B on Twitter. This network was constructed using the friends and follower data pulled at the start of the study.

Cofollowing networks connect two participants based on the Twitter accounts they both follow. Formally, we define them as a weighted directed network in which the weight of the edge from participant A to participant B is given by: (1) $$w_{AB}=\frac{n_{AB}}{n_A},$$(1) where n_A is the number of accounts followed by A, and n_AB is the number of accounts that both A and B follow.

Conversely, cofollower networks connect participants using mutual Twitter followership. In a cofollower network the edge from participant A to B is also given by (1), where in this setting n* refers to the number of accounts that follow * instead. Cofollower and cofollowing networks were constructed using the friend and follower data from the end of the study.

2.2.4. Vaccination Behavior over digital social networks

Over the direct followership network we calculated the assortativity coefficient⁵³ of vaccination outcome. Assortativity in this context measures how likely a participant is to follow another participant that has the same vaccination outcome as them in comparison to a random wiring of the nodes. A positive coefficient indicates that participants were more likely to make connections with those of the same vaccination status, while a negative coefficient indicates that participants were more likely to make connections with those of opposite vaccination status. To see how the observed coefficient compared to what was possible on the given network we peformed a permutation test, in which we randomly shuffled the vaccination states among the nodes and recorded the assortativity coefficient 10,000 times.

For the cofollower and cofollowing networks we examined the distribution of vaccinations across tighly-knit groups of participants, or so-called communities. Community labels for each participant were produced using a generalization^54,55 of the Louvain algorithm, a modularity maximation technique. Modularity is a measurement of community structure due to Newman and Girvan^56,57 that compares the inter- and intra-community edge weights compared to an equivalent random network for a given community assignment. A community assignment with high modularity has dense within community connections and sparse between community ties.

Generalizations of the Louvain algorithm for weighted directed networks have been applied to various Twitter networks.^58–60 We cross-referenced the resulting assignments with participant Twitter and demographic data to ensure that the detected groups were meaningful. Groups within the cofollowing network should reflect who the participants follow on Twitter, and may be indicative of group interests. Cofollower networks contain information on who follows our participants, and thus, given the modest number of followers of the cohort on Twitter, communities may be representative of offline friendships in the university community.

Multiple imputation was used to create 50 complete vaccination outcome data sets. We then examined the percent vaccinated in each of these sets across the Louvain-detected communities. An important step in determining if our communities exhibit differential coverage is comparing what is observed over the detected communities to what would be seen given an equivalent random community assignment.

In order to make this comparison we perform the same hypothesis test^61–63 using the Louvain detected communities, and random permutations of that labeling. Formally, the null hypothesis for this test assumes that p_i = p_j for all communities i and j, where p_k is vaccine coverage in community k. The alternative hypothesis was that there exists a pair of communities i and j such that pi ≠ pj. This test was performed for each of the 50 multiply imputed data sets and the resulting p-values were recorded. We refer to this collection as the set of nonrandom p-values. For every imputed data set, we randomly shuffled the participants between communities while maintaining the size of each community 1,000 times. The hypothesis test described above was then performed on each of the 50,000 random data sets, and we recorded the resulting p-values. We call this set of p-values the random p-values.

Random p-values were compared to nonrandom p-values with a one-sided Kolmogorov-Smirnov test.^64,65 The null hypothesis was that the cumulative distribution function (CDF) of the underlying distribution for the nonrandom p-values was not greater than the CDF of the underlying distribution for the random p-values. The alternative hypothesis was that the CDF of the underlying distribution for the nonrandom p-values was greater. We used the one-sided alternative because if there was differential vaccination coverage across the detected communities, the p-values from the tests for equal proportions across the detected communities should be stochastically lower than the p-values from the tests using random community assignments.

3. Results

3.1. Participant demographics

We present a demographic breakdown by study group in Table 1 and present aggregate medical history statistics relevant to flu vaccination in Table 2. The intervention and control groups are nearly identical with regard to race and age related variables. Both groups are also similar with respect to flu-relevant medical history.

Table 1. A Demographic breakdown comparing the intervention group to the control group.

		Intervention (n = 353)	Control (n = 349)
Gender		# (%)	# (%)
	Female	247 (70)	247 (70.8)
	Male	95 (26.9)	94 (26.9)
	Transgender	3 (0.8)	2 (0.6)
	None of the above	3 (0.8)	3 (0.9)
	Missing	5 (1.4)	3 (0.9)
Race		# (%)	# (%)
	White	299 (84.7)	291 (83.4)
	Asian	32 (9.1)	32 (9.2)
	Black	18 (5.1)	22 (6.3)
	Hispanic	10 (2.8)	7 (2)
	Native American	6 (1.7)	5 (1.4)
	Other	8 (2.3)	9 (2.6)
	Missing	7 (2)	3 (0.9)
Age
	Min	18	18
	25th perc.	18	18
	50th perc.	20	19
	75th perc.	21	21
	Max	63	83
	# Missing	2	3
School year		# (%)	# (%)
	1st	116 (32.9)	108 (30.9)
	2nd	72 (20.4)	70 (20.1)
	3rd	81 (22.9)	89 (25.5)
	4th	56 (15.9)	64 (18.3)
	5th +	20 (5.7)	13 (3.7)
	Missing	8 (2.3)	5 (1.4)

Categorical variables (gender, race, and school year), are presented as the total count in each value. Percentages are presented as a within group percentage. The sole continuous variable, age, is presented with the min, max and three quartiles. Note that the percentages for the race variable do not add to 100 because paticipants were able to identify as a member of more than one racial group. Randomization by self-reported gender and flu vaccination status led to both groups being quite similar with regard to these demographic variables.

Table 2. Medical history data by study group.

		Intervention (n = 353)	Control (n = 349)
Saw a doctor 2017–2018		# (%)	# (%)
	No	30 (8.5)	43 (12.3)
	Yes	311 (88.1)	299 (85.7)
	Unsure	8 (2.3)	5 (1.4)
	Missing	4 (1.1)	2 (0.6)
Got a flu shot in 2017–2018		# (%)	# (%)
	No	163 (46.2)	169 (48.4)
	Yes	163 (46.2)	157 (45.0)
	Unsure	23 (6.5)	21 (6.0)
	Missing	4 (1.1)	2 (0.6)
Flu shot required in 2018–2019		# (%)	# (%)
	No	212 (60.1)	212 (60.7)
	Yes	67 (19)	67 (19.2)
	Unsure	69 (19.5)	68 (19.5)
	Missing	5 (1.4)	2 (0.6)

Percentages are presented as a within group percentage. The two groups are quite similar in their responses to these questions.

While participants were not randomized according to their Twitter activity, an analysis using participant Twitter data prior to the start of the intervention on October 1, 2018 demonstrates that activity across the study groups is similar. We present a Twitter activity measurement table and daily post timing plot in Table 3 and Figure 1 respectively.

Figure 1. Total tweets by study group for each five-minute period of the day that were posted prior to October 1, 2018. Data reflects at most the 3,200 most recent tweets prior to October 1 per participant, and comes from 630 of the 702 participants with 320 from the intervention group and 310 from the control. The remaining 72 participants have missing tweet data. Time is presented in UTC. The control group can be seen on the left with points marked with blue circles and the intervention group can be seen on the right with points marked with red x’s. The timing of tweets between the two groups is similar, with the control group having sent out slightly more tweets than the intervention group prior to October 1, 2018.

Table 3. Summary statistics of participant twitter data prior to the start of the intervention.

		Intervention (n = 353)	Control (n = 349)
Self reported posts per day		# (%)	# (%)
	0	129 (36.5)	109 (31.2)
	1	154 (43.6)	164 (47.0)
	2	31 (8.8)	37 (10.6)
	3	10 (2.8)	10 (2.9)
	4	6 (1.7)	6 (1.7)
	5	5 (1.4)	5 (1.4)
	6+	13 (3.7)	16 (4.6)
	Missing	5 (1.4)	2 (0.6)
Actual posts per day
	Min	0	0
	25th perc.	0.15	0.17
	50th perc.	0.56	0.87
	75th perc.	2.05	2.33
	Max	35.80	37.65
	# Missing	4	10
Has a private account		# (%)	# (%)
	No	309 (87.5)	297 (85.1)
	Yes	40 (11.3)	42 (12.0)
	Missing	4 (1.1)	10 (2.9)
# of followers
	Min	1	0
	25th perc.	90	72
	50th perc.	239	254
	75th perc.	450	434
	Max	1849	2720
	# Missing	4	10
# of accounts followed
	Min	0	0
	25h perc.	129	124.5
	50th perc.	270	265
	75th perc.	427	441
	Max	2184	2459
	# Missing	4	10

All percentages are computed as within group percentages. Actual twitter data was computed based on participant data that was available prior to October 1, 2018. Despite randomization not taking twitter activity into account, the two groups are quite similar with respect to common twitter metrics.

3.2. Vaccination outcome

The 2018–2019 vaccination outcomes for both groups are presented in Table 4. Percent vaccinated is roughly 45% in both study groups, with the intervention group having a larger percentage of missing data (about 10 percentage points higher).

Table 4. Vaccination outcome in the 2018–2019 flu season by study group, percentages are based on the 353 in the intervention group and the 349 in the control group.

Study group	Vaccination outcome	Percentage
Control	Yes	46.7%
	No	35.5%
	Missing	17.8%
Intervention	Yes	44.5%
	No	28.9%
	Missing	26.6%

3.3. Intervention efficacy

3.3.1. Intent-to-treat

Following Hsieh et al.⁵² we found that the minimal detectable percentage point increase from a control rate of 46% to intervention at 80% power and 95% confidence for model 1 was 13 points. For model 2 the minimal detectable increase was 16 points.

Odds ratio point estimates and 95% confidence intervals for models 1 and 2 are presented in Table 5. Estimates from both models suggest that there was not a statistically significant effect from the intervention, and that subjects from the intervention group were just as likely as those from the control group to receive a vaccine. Such an estimate suggests that if the intervention does have a positive effect on vaccination uptake, it is well below the minimal detectable levels our sample size and models are able to detect.

Table 5. Logistic regression results for the intent to treat analysis.

Variable	Value	Model 1 odds ratio (95% CI)	Model 2 odds ratio (95% CI)
Study group	Control	1	1
	Intervention	1.03 (0.67, 1.59)	1.08 (0.69,1.68)
Flu shot in 2017–2018	No	1	1
	Yes	6.96 (3.97, 10.13)	${6.76}^{*}$ (4.14, 11.03)
Flu shot required in 2018–2019	No	1	1
	Yes	${2.69}^{*}$ (1.37, 5.28)	${2.93}^{*}$ (1.42, 6.04)
Age	1 year increase		1.03 (0.95, 1.11)
Gender	Female		1
	Male		1.01 (0.60, 1.69)
	Transgender		0.52 (0.05, 5.07)
	None of the above		1.22 (0.05, 32.80)
Race indicators
White	No		1
	Yes		1.93 (0.62, 6.02)
Black	No		1
	Yes		1.76 (0.46, 6.77)
Asian	No		1
	Yes		1.50 (0.47, 4.76)
Native American	No		1 (0.05, 2.43)
	Yes		0.35
Hispanic	No		1
	Yes		0.30 (0.05, 1.62)
Other	No		1
	Yes		1.16 (0.25, 5.33)
School year	1		1
	2		1.04 (0.55, 1.97)
	3		1.25 (0.68, 2.29)
	4		1.57 (0.78, 3.16)
	5+		0.73 (0.17, 3.04)

All estimates are presented as averaged over 50 imputations. The left column presents the fit for the model not including any demographic variables, the right adds in demographic variables. Performing a wald comparison test between the two models produces a p-value of 0.86. Models were fit with the 627 participants that did not receive a flu shot prior to October 1, 2018. * P < 0.01${}^{*}p<0.01$.

Across all covariates included in either model, the only significant effects came from being required to receive a flu shot or having received a flu shot in the 2017–2018 season. In this cohort we estimated that an individual that received their flu shot in 2017–2018 was over six times as likely to receive their flu shot in the 2018–2019 season. This effect was even larger than that attributed to a self-reported requirement to receive a flu shot.

Model 2 indicates that none of the demographic covariates had a significant effect on vaccination rate.

3.3.2. Campaign engagement

Following Hsieh et al.⁵² we found that the minimal detectable percentage point increase from a control rate of 46% to intervention at 80% power and 95% confidence for model 3 was 13.5 points. For model 2 the minimal detectable increase was 21 points.

Estimates for models 3 and 4 can be found in Table 6. Results are consistent with models 1 and 2 in finding that individuals assigned to the intervention group were just as likely as those in the control to receive a flu shot. The estimates for model 4 provide additional evidence that the campaign was ineffective, as those who did not follow or share intervention content were just as likely to receive a vaccination as those who did.

Table 6. Logistic regression results for the twitter engagement analysis.

Variable	Value	Model 3 odds ratio (95% CI)	Model 4 odds ratio (95% CI)
Study group	Control	1	1
	Intervention	0.96 (0.61, 1.49)	1.10 (0.54, 2.23)
Flu shot in 2017–2018	No	1	1
	Yes	6.34 (4.28, 11.31)	6.76 (4.28, 11.50)
Flu shot required in 2018–2019	No	1	1
	Yes	2.69 (1.32, 5.24)	2.93 (1.37, 5.49)
Followed intervention account	No		1
	Yes		0.87 (0.41, 1.87)
Total pics seen	1 Pic Increase		0.85 (0.70, 1.06)
Tweeted intervention #	No		1
	Yes		1.36 (0.69, 2.71)
Tweeted “health”	No		1
	Yes		1.13 (0.71, 1.80)

All estimates are presented as averaged over 50 imputations. The left column presents the fit for the model not including any twitter engagement variables, the right adds in twitter engagement variables. Performing a wald comparison test between the two models produces a p-value of 0.53. Models were fit with the 564 participants that did not receive a flu shot prior to October 1, 2018 for which we were able to get twitter data.* P < 0.01.

In Figure 2 we present the results of the CAPTCHA-like image identification question restricted to the 564 participants analyzed with models 3 and 4. At least a quarter of the intervention group reported seeing the image we tweeted out in all months except for February. In contrast, 8% or less of the control group identified the image we tweeted out in those same months. While a large portion of the intervention group may not have paid attention to the campaign, there is evidence that our posts were consistently viewed by a quarter of the intervention group. This allows us to rule out the possibility that the campaign was ineffective due to inattention.

Figure 2. Responses to the CAPTCHA-like image question in the December 2018–April 2019 monthly check-in surveys. The height of the blue bars correspond to the percentage of participants in either group that indicated seeing the image tweeted by the intervention account, red bars correspond to those that did not, and green bars correspond to participants that did not reply to the question. Percentages are with respect to the 277 total control group members and 287 intervention group members included in the fits of models 3 and 4.

Figure 3. The direct followership network from the beginning of the intervention period. Each node represents a participant and directed edges denote followership on Twitter. There are 260 participants in the network, remaining participants did not follow someone else in the study. Nodes are colored by their indicated flu shot coverage in the 2017–2018 season. The assortativity of this network by vaccination choice is examined in Figure 4.

Figure 4. The histogram of possible assortativity coefficients for 10,000 random arrangements of 2017–2018 vaccination status on the network pictured in Figure 3. Also denoted is the placement of the observed assortativity coefficient, 0.0793. The proportion of random coefficients larger than the observed coefficient was 0.0445 indicating that there is evidence of positive assortativity of 2017–2018 vaccination choice on this network.

Figure 5. Explatory plots for the nonrandom and random p-values for the May cofollowing network. On the left are the histograms from the two sets of p-values. On the right are the empirical cumulative distribution functions (CDFs). The Kolmogorov-Smirnov test for these data had D+= 0.26 with a corresponding p-value < 0.001.

Figure 6. Box and whisker plots for the multiply imputated vaccination percentages for the four Louvain-detected May cofollowing communities with more than ten nodes.

Odds ratio estimates for both models corroborate the finding that receiving a flu shot in 2017–2018, and being required to receive a shot had a significant effect on getting vaccinated in 2018–2019. The effect of prior vaccination was estimated to be slightly higher in this subset of the cohort, as those who received a flu shot in 2017–2018 were roughly seven times more likely to receive one in 2018–2019.

3.4. Vaccination choice over digital social networks

3.4.1. Assortativity of 2017–2018 choice on the direct followership network

We present the direct followership network with each participant colored by their self-reported 2017–2018 vaccination status. The assortativity coefficient associated with this network was calculated to be 0.0793. How this observed value compares to the distribution of 10,000 randomly drawn coefficients is displayed in Figure 4. We find that our observed value is in the upper 5% of this distribution, indicating that our network displayed significant assortativity with respect to 2017–2018 vaccination choice. In other words, participants were more likely to follow other participants of the same vaccination status in 2017–2018.

3.4.2. Community structure of cofollowing and cofollower networks

The Louvain community detection algorithm found the cofollowing network to have four communities with more than ten participants, while the cofollower network had nine such communities.

Since cofollowing networks connect individuals based on mutual following habits, we cross-referenced the Louvain algorithm community labels with the most followed Twitter accounts among the participants within each community. While there were accounts shared across multiple communities (for example, the account belonging to the university mascot), each community had accounts unique to them that suggest broad group interests. Community 1 primarily followed accounts related to the university community (e.g. student life, the university president and the school paper), community 2 followed sports-related accounts (e.g. professional and collegiate athletes), community 3 tended to follow accounts related to popular culture, (e.g. actors and musicians), and community 4 followed accounts associated with various political figures.

The communities detected on our cofollower network were examined with the demographic data of our participants. We present the breakdown of each community by various demographic variables in Tables 7 and 8. Investigation of these tables suggests that the Louvain algorithm was able to capture meaningful community structure over the cofollower network. For example, communities seemed to segment participants according to school year, racial identity and extracurricular group participation.

Table 7. Louvain-detected community by demographic variables for the cofollower network constructed using data pulled at the end of the study period.

		Gender		Race						School Year
Comm.	Size	Male	Female	White	Black	Asian	Hispanic	Native American	On Campus	1	2	3	4	5+
All	583	26.1%	71.4%	84.2%	6.2%	8.7%	2.1%	1.7%	46.8%	32.9%	20.8%	23.5%	16.5%	4.6%
1	150	32.7%	66.7%	90.0%	5.3%	6.7%	0.7%	1.3%	54.0%	42.0%	20.0%	20.0%	12.7%	5.3%
2	77	19.5%	80.5%	92.2%	1.3%	2.6%	3.9%	0.0%	16.9%	10.4%	3.9%	44.2%	31.2%	7.8%
3	63	27.0%	66.7%	87.3%	3.2%	6.3%	1.6%	4.8%	1.6%	25.4%	20.6%	12.7%	25.4%	31.7%
4	56	10.7%	89.3%	85.7%	5.4%	8.9%	3.6%	1.8%	53.6%	17.9%	37.5%	32.1%	8.9%	3.6%
5	39	30.8%	69.2%	87.2%	0.0%	10.3%	2.6%	2.6%	61.5%	30.8%	33.3%	15.4%	7.7%	10.3%
6	39	28.2%	66.7%	53.8%	35.9%	5.1%	0.0%	5.1%	35.9%	28.2%	25.6%	28.2%	10.3%	2.6%
7	34	8.8%	85.3%	91.2%	2.9%	5.9%	0.0%	0.0%	55.9%	41.2%	23.5%	11.8%	20.6%	2.9%
8	27	14.8%	85.2%	81.5%	0.0%	25.9%	0.0%	0.0%	63.0%	40.7%	22.2%	14.8%	22.2%	0.0%
9	16	43.8%	50.0%	93.8%	0.0%	12.5%	0.0%	6.2%	81.2%	87.5%	6.2%	6.2%	0.0%	0.0%

Table 8. Breakdown of louvain-detected communities by participation in extracurricular groups for the cofollower network from data pulled at the end of the study period.

		Extracurricular Group
Comm.	Size	Greek Life	Professional	Academic	Student Gov.	Cultural	Volunteer	Hobby	Religious	Athletic
All	583	11.7%	13.9%	29.0%	3.3%	7.0%	34.0%	24.5%	10.3%	23.7%
1	150	9.3%	14.7%	28.0%	2.0%	6.7%	34.7%	28.0%	13.3%	29.3%
2	77	26.0%	19.5%	40.3%	1.3%	2.6%	42.9%	15.6%	11.7%	22.1%
3	63	12.7%	9.5%	19.0%	0.0%	9.5%	25.4%	36.5%	6.3%	19.0%
4	56	17.9%	12.5%	28.6%	7.1%	8.9%	50.0%	30.4%	16.1%	30.4%
5	39	2.6%	20.5%	30.8%	2.6%	5.1%	41.0%	20.5%	2.6%	28.2%
6	39	2.6%	10.3%	15.4%	5.1%	17.9%	25.6%	7.7%	7.7%	5.1%
7	34	17.6%	8.8%	32.4%	5.9%	5.9%	35.3%	14.7%	2.9%	29.4%
8	27	7.4%	11.1%	37.0%	0.0%	7.4%	40.7%	11.1%	25.9%	33.3%
9	16	0.0%	12.5%	37.5%	12.5%	0.0%	6.2%	43.8%	0.0%	25.0%

These results suggest that our cofollowing and cofollower networks exhibit meaningful community structure. Cofollowing communities may be refelective of interest groups or market segments,⁶⁶ while cofollower communities may capture offline social circles.

3.4.3. Vaccination coverage of Louvain-detected communities

Given the overall vaccination coverage and the size of the cofollowing and cofollower communities it is unlikely for the random community rearrangements described in Sec. 4.4 to have violated the assumptions of the hypothesis test comparing community proportions.⁶⁷ We thus present the results of the Kolmogorov-Smirnov test in Table 9. We find statistically significant evidence that the p-values associated with the Louvain detected cofollowing communites were stochastically lower than those associated with equivalent random assignments. A graphical comparison of the two sets can be found in Figure 5. This indicates the vaccination distribution across our observed communities is not the result of random noise, but rather a true feature of the data. In particular, this suggests that some cofollowing communities had lower vaccination coverage than others. While the missing data make it difficult to perform direct comparison tests between the four communities, we do include boxplots of the percent vaccinated within each Louvain-detected community across the 50 multiply imputed data sets in Figure 6. These boxplots show cofollowing communities 1 and 4 as having lower vaccination levels than the remaining communities.

Table 9. Kolmogorov-Smirnov Test results for the cofollower and cofollowing networks.

Network	D^+ value	p-value
CoFollowing network	0.26304	0.0009957
CoFollower network	0.06598	0.6473

The null hypothesis for both tests was that the cumulative distribution function of the underlying distribution for the nonrandom p-values was not greater than the cumulative distribution function of the underlying distribution for the random p-values. This was tested against the alternative hypothesis that the nonrandom cumulative distribution function was greater.

The Kolmogorov-Smirnov test for the cofollower network did not provide evidence of differential vaccination coverage. As such, we do not examine the distributions of the nonrandom and random p-values nor do we provide boxplots as we did for the cofollowing data.

3.5. Cross-referencing with SHC vaccination records

We investigated desirability bias in participant self-reported vaccination decision by cross-referencing SHC vaccination records for the 2018–2019 season. Contingency tables comparing participants that reported receiving a shot at SHC to those that actually got one are presented in Table 10. We find that in both study groups roughly 70% of participants that indicated getting an SHC vaccine did not actually receive one. While this suggests a large desirability bias, we cannot confirm whether these students did not get a flu shot or just misclassified where they got their flu shot. If 70% was the true misreporting rate, the SHC data suggest that the rate is consistent across both study groups, so it should not have impacted the intent to treat analysis.

Table 10. Contingency tables for participants reporting receiving a flu shot at SHC vs actually receiving a vaccine from SHC.

		Intervention Group
		Received an SHC Flu Shot
		Yes	No	Total
Reported an SHC flu shot	Yes	18	49	67
	No	13	273	286
	Total	31	322	353
		Control Group
		Received an SHC Flu Shot
		Yes	No	Total
Reported an SHC flu shot	Yes	23	51	74
	No	8	267	275
	Total	31	318	349

Participants with missing vaccination data were coded as not informing us they received a flu shot from SHC. Tables include all 702 participants.

4. Discussion

4.1. Past behavior explains future results

Across the entirety of our analysis we found that being required to receive a flu shot and having received a flu shot the year before were the only variables to have an impact on vaccination uptake in 2018–2019.

Prior vaccination status has been shown to impact an individual’s decision to get a flu shot across a number of prior studies.^{8,15,43,68,69} As more independent studies reproduce this finding, it is likely that past behavior is predictive of future flu shot status. This highlights both the importance and potential benefits of recruiting the undergraduate population into this behavior. If public health entities are able to get college age individuals to start receiving vaccinations at their younger age, there is potential for it to transform into a lifelong habit.⁶⁹ Conversely, this could indicate that the key to raising collegiate vaccination rates lies in the pre-college/childhood years.

Our finding of misreporting of vaccination location using SHC data indicates that the association between prior and present vaccination status may not be as strong as our models suggest. Rather, these results could reflect that individuals who misreported their vaccination status in one year were also likely to misreport their vaccination status in the following year, but this is difficult to disentangle without actual vaccination records from the 2017–2018 flu season.

4.2. Campaign messaging and credibility

While we attempted to cater message content to our cohort, perhaps it was not persuasive enough to spur action. Effective messaging content is an active area of influenza vaccination promotion research.^9,12,25 Message effectiveness may have been improved by involving additional individuals and perspectives in crafting tweets. However, formative work and development may also be needed to receive feedback on content. Further, establishing behavioral health models that guide messaging content would improve interventions and the replicability of future studies.

Our campaign also used the same messages to appeal to the entire intervention group. However, the cofollowing network of our participants helped us identify four large subgroups of the cohort with unique interests. Catering the messaging to each specific community⁷⁰ could result in a more effective intervention and is a direction for future research. For example, messaging that appeals to cofollowing community 1 (which had lower than total cohort vaccination coverage) could focus on university-specific content.

The credibility of our account as a source of health care information is an unknown factor. While the information we shared on seasonal influenza and vaccination came directly from the CDC and reputable news outlets, the account itself was not a health care provider, a hub for medical information, nor a close friend to the participants. A number of studies have found that people are more likely to trust and act on medical information from sources they deem to be credible, namely their primary healthcare provider or close friends/family.^{42,43,69,71–73}

4.3. Effectiveness of reminder systems

In our intervention Twitter was utilized as a reminder system for the members of the intervention group. Reminder systems as the sole intervention have often not succeeded at significantly impacting the desired outcome over various promotional vaccination studies.^18,74,75 A study similar to ours¹⁸ found that flu shot reminders sent through university email systems were ineffective at raising vaccination rates compared to control across different message types. However, they did find that combining the email reminder system with a thirty dollar incentive rendered upon getting a flu shot did significantly increase vaccination rates.

Paying students to receive a flu shot is probably not an economically viable recommendation, but that study is just one instance of many that have shown that reminder systems can be part of successful vaccination campaigns.^11,18,19,76 Our results indicate that social media reminder campaigns may not be effective on their own, highlighting the need to pair them with additional interventions, especially given that they can be conducted for little financial cost and may be able to magnify the effects of other interventions by spreading the word.

For instance, many^{8,13,17,19,23,42,43,77} cross-sectional surveys of collegiate populations have shown that the physical act of going to get a flu shot is a major barrier to increased rates. An intervention that removed this barrier with in-dormitory and dining hall vaccination clinics utilized SNS in a successful manner by having peer health care workers advertise these vaccination clinics with a personalized social media campaign.¹¹

4.4. Vaccination choice over digital and physical social networks

Physical social networks have been shown to have an impact on vaccination choice.^14,41–43 People who were vaccinated were more likely to have friends and family members that received a shot, with an analogous result for the unvaccinated.^14,41 Anecdotal stories from close friends or family can weigh heavily on a person’s decision to receive a flu shot,^42,43 and a finding similar to ours has shown that the physical contact network of a high school exhibited positive network assortativity with regard to flu vaccination.⁷⁸

Conversely, other Twitter studies found that the networks underlying SNS can be highly assortative with regard to sentiments on flu shots^36,37 It has also been shown in mathematical models that if actual vaccination status was similarly polarized on physical social networks, the probability of large outbreaks would be greatly increased.^36,79

Our results bridge and build upon these two research areas by demonstrating evidence of assortativity and differential coverage of flu vaccination choice over the digital social networks of a cohort. To our knowledge, this study is the first work to produce such a finding. The breakdowns of the Louvain-detected cofollower communities in Tables 7 and 8 suggest that the digital social networks of our participants are representative of their physical social networks. It is thus possible that some highly connected segments of the undergraduate population could be more at risk for an influenza outbreak than others.

However, our study did not collect offline social network data, so we cannot directly examine relationships between offline and online vaccination coverages and thus future research is needed. One approach would be to adapt a standard technique for recruiting physical friendship networks in which randomly recruited participants are asked to identify friends for a second wave of recruitment.^14,80 Having the first round of participants invite both a set of physical friends and a set of digital friends would allow for digital social networks to be layered on top of physical social networks. The distribution of vaccination choice could be examined over both networks. If a correlation is seen to exist between online and offline vaccination coverage, online networks could be used as a more cost-effective proxy for offline network coverage.

4.5. Conclusion

Given the ubiquity of social networking services,²⁶ understanding their potential impact on public health is of the utmost importance. This is particularly true during the COVID-19 pandemic,⁸¹ which has seen widespread online misinformation on virus origins, ongoing vaccination campaigns, disease severity, and more.⁸² As such, it is crucial for public health entities to know what does and does not work when it comes to encouraging preventative health behaviors using SNS.

To what extent the results of this study are applicable to COVID-19 vaccination campaigns remains to be seen. The COVID-19 pandemic continues to unfold against a highly polarized political landscape,⁸³ and the accompanying divergence in perceptions of disease severity, vaccine efficacy and risk, and attitudes toward public health interventions such as stay-at-home orders, social distancing, and mask mandates may influence responsiveness to vaccination campaigns. The COVID-19 vaccines available to American undergraduates have been found to be significantly more effective^84,85 than influenza vaccines whose efficacy fluctuates from season to season.^86–88 Some universities have also decided to require a COVID-19 vaccination prior to returning to in-person activities in the Fall of 2021,⁸⁹ which would make promotion of this vaccine moot.

However, while our study may not be broadly applicable to all collegiate vaccination campaigns, our results suggest that campaigns encouraging best practices through social media messaging alone will not be sufficient. Effective campaigns will likely require additional interventions (e.g. in-dormitory vaccination clinics) or more sophisticated message delivery strategies (e.g. targeted marketing).

Conflict of interest disclosure

The authors report no conflicts of interest. The research presented in this article met ethical guidelines, including adherence to legal requirements of the United States and has been approved by the Ohio State University IRB. The funding source played no role in how the research was conducted.

Acknowledgments

The authors acknowledge the Ohio State University STEAM Factory “STEAM Powered Project” seed grant for providing funding for raffle incentives, and the Ohio State University Student Health Center for providing flu vaccination data.

Funding

No funding was used to support this research and/or the preparation of the manuscript.