An evaluation of the feasibility and usability of a proof of concept mobile app for adverse event reporting post influenza vaccination

ABSTRACT

The Canadian National Vaccine Safety network (CANVAS) gathers and analyzes safety data on individuals receiving the influenza vaccine during the early stages of annual influenza vaccination campaigns with data collected via participant surveys through the Internet. We sought to examine whether it was feasible to use a mobile application (app) to facilitate AEFI reporting for the CANVAS network. To explore this, we developed a novel smartphone app, recruited participants from a hospital influenza immunization clinic and by word of mouth and instructed them to download and utilize the app. The app reminded participants to complete the CANVAS AEFI surveillance surveys (“AEFI surveys”) on day 8 and 30, a survey capturing app usability metrics at day 30 (“usability survey”) and provided a mechanism to report AEFI events spontaneously throughout the whole study period. All survey results and spontaneous reports were recorded on a privacy compliant, cloud server. A software plug-in, Lookback, was used to record the on-screen experience of the app sessions. Of the 76 participants who consented to participate, 48(63%) successfully downloaded the app and created a profile. In total, 38 unique participants completed all of the required surveillance surveys; transmitting 1104 data points (survey question responses and spontaneous reports) from 83 completed surveys, including 21 usability surveys and one spontaneous report. In total, we received information on new or worsening health conditions after receiving the influenza vaccine from 11(28%) participants. Of the usability survey responses, 86% agreed or strongly agreed that they would prefer to use a mobile app based reporting system instead of a web-based system. The single spontaneous report received was from a participant who had also reported using the Day 8 survey. Of Lookback observable sessions, an accurate transmission proportion of 100% (n=290) was reported for data points. We demonstrated that a mobile app can be used for AEFI reporting, although download and survey completion proportions suggest potential barriers to adoption. Future studies should examine implementation of mobile reporting in a broader audience and impact on the quality of reporting of adverse events following immunization.

KEYWORDS:

• adverse event
• drug reaction reporting systems
• health technology
• immunization
• mobile apps

Introduction

Post-market surveillance to identify adverse events following immunization (AEFI) is a critical component of immunization programs globally.^1,2 It is particularly important for seasonal influenza vaccines, which are modified annually to include up- to-date circulating strains and, thus, each year need to be monitored for safety. In Canada, in addition to passive safety surveillance conducted by provinces and territories³ and active surveillance conducted by the Canadian Immunization Monitoring Program Active (IMPACT),^4,5 the Canadian National Vaccine Safety (CANVAS) network monitors for AEFI's related to the seasonal influenza vaccine. The CANVAS program relies on web-based survey responses from adults and parents of children 8 d following vaccine receipt regarding their post immunization experience. Additional telephone follow up is used for reports of severe adverse events.⁶

Mobile devices have enhanced the provision of health care both for individuals and providers. Smartphones offer an opportunity to enhance immunization practice and facilitate tailored communication between individuals and public health officials.^7-9 With increased mobile device and application (app) usage, the potential exists to capture, transmit, and monitor post immunization experience information in real-time using personal mobile devices. Thus mobile apps could potentially serve as a mechanism for AEFI reporting. Specifically, we saw an opportunity to use mobile apps to complement the efforts of the CANVAS network by providing individuals with the ability to both answer post vaccination surveys using their mobile devices as well as spontaneously report AEFI's as they occur instead of waiting until the time of survey distribution. However, the feasibility and usability of mobile reporting of AEFI's is unknown. We developed a standalone CANVAS app for iPhones and sought to examine whether it could facilitate AEFI reporting for the CANVAS network.

The objectives of this proof of concept study were to determine the feasibility and usability of mobile AEFI reporting for the CANVAS network. Specifically, we sought to: 1) determine if participant data from smartphone devices could be accurately transmitted into a secure cloud server accessible by study staff by determining the proportion of data elements accurately transmitted (i.e. accurate transmission proportion) and 2) evaluate participant usability of the app as measured by download rates, in-app AEFI survey response rates, and responses to a usability survey. We also report on the use of the app to report AEFI's.

Results

Participant recruitment and response rates

A total of 76 individuals consented to participate in this study, 59(78%) were recruited via hospital immunization clinics and 17(22%)via word of mouth (Appendix A, Fig. 1). There were 48(63%) participants who downloaded and logged into the app using an email address provided at enrollment (Fig. 1). Throughout the study period, 38 unique participants submitted 83 completed surveys. Of the AEFI surveys, there were 35 Day 8, 24 Day 30 and 3 Combined Day 8/30 Surveys. Twenty-one participants completed usability surveys, 15(71%) participants recruited via hospital clinics and 6(29%) via word of mouth (Appendix A, Fig. 1).

Figure 1. Overview of participant recruitment and response rates.

AEFI surveys

Day 8 AEFI surveys were completed by 38 participants (50% of total participants, 79% of those that downloaded the app). Day 30 AEFI surveys were completed by 27 participants (35% of total participants, 56% of those that downloaded the app). Three participants chose to complete the Day 8 and Day 30 AEFI surveys through a combined “Day 8 and Day 30” AEFI survey. Of the 38 who completed a Day 8 AEFI survey, participant age ranged from 20-29(34%), 30-39(29%), 40-49(26%) and 50-59(11%) (Table 1). Half (19) of participants were male and half (19) were female. All reported for themselves, rather than for a family member. Over 80%(32) had received their seasonal influenza vaccine for the past 2 influenza seasons: 31(82%) received it in a hospital immunization clinic, 3 at pharmacies and 4 at Public Health or Other types of clinics. It took participants an average of 12 d from time of vaccination to submit the day 8 survey (4 d from survey availability) and 32 d from vaccination to submit the day 30 survey.

Table 1. Demographics of study participants who submitted survey responses (n = 38).

What is your age?	n(%)
20-29	13(34)
30-39	11(29)
40-49	10(26)
50-59	4(11)
What is your gender?
Male	19(50)
Female	19(50)
Where did you receive your flu shot this year?
Hospital Immunization Campaign	31(82)
Pharmacy	3(7)
Public Health	1(3)
Other	3(8)
In the last 2 years, how many times have you received the flu shot?
0	2(5)
1	4(11)
2	32(84)

Usability surveys

Usability surveys were completed by 21 participants (28% of total participants, 44% of those that downloaded the app (Fig. 1). Of those who completed the usability survey (n=21), average age was 34.7(IQR 13) (Table 2).

Table 2. Demographics of study participants who completed the usability survey (n = 21).

Measure	n(%)
Age (yrs.)	34.7(IQR 13)
Male	11(52.4)
Female	10(47.6)
Years of Owning a Smartphone Device
Average	5.21
Education
Some post-secondary	1(4.8)
Community college / technical school diploma	3(14.3)
Undergraduate degree	9(42.9)
Graduate degree	5(23.8)
Other (please specify)	3(14.3)
Post doctorate	1(4.8)
Employment^1
Employed for wages	19(90.5)
Self-employed	3(14.3)
Not employed	0(0.0)
Technology Related Profession
Yes	7(33.3)
No	14(66.7)
Access to the Internet
> 5 times a day	21(100)
Mean Technology Readiness Index (TRI) Score	3.35

¹ Selections are not mutually exclusive and thus, may add to more than 100%.

Reporting of AEFIs

In total, we received an AEFI report on new or worsening health conditions after receiving the influenza vaccine from 11(28%) participants. We received 5 reports via the Day 8 AEFI survey, 4 in the Day 30 AEFI survey and 2 in the combined Day 8 and Day 30 survey. The single spontaneous report received was from a participant who had also reported using the Day 8 AEFI survey.

Of the Day 8 AEFI survey responses, 3 participants indicated they experienced a new health problem or exacerbation of an existing health problem in the first week following vaccination. All indicated this problem was “not severe enough to miss work/school or prevent/stop normal activities” and did not report it to a health care provider. Two additional participants answered, “Yes” to “Do you have any other concerns that you would like us to know about” and specified onset of new health conditions within 24 hours of vaccination.

Of the Day 30 AEFI survey responses, 4 participants reported developing a new health problem or exacerbation of an existing health problem in the last month following the flu shot. These four participants were distinct from the 5 who reported events in the Day 8 AEFI survey. Three provided details on the condition and 2 of the participants who reported conditions in the Day 30 AEFI survey said this was severe enough to “miss work/school or prevent/stop normal activities.” None of the participants saw their health care provider.

Of the participants who completed the combined Day 8 and Day 30 AEFI survey, one reported a condition in the first week that was severe enough see their health care provider (specified as “Clinic/Family Physician”) and the other reported a condition in the first week that was not severe enough to miss work/school or prevent/stop normal activities or see a health care provider.

Accurate transmission proportion

374 data elements from completed AEFI surveys and spontaneous reports were sent from the app and recorded in the server throughout the study period (Appendix A, Table 4). 321 data points were observed in AEFI surveillance and spontaneous reporting sessions captured using Lookback. Overall, 291 of participant entered data elements were captured by Lookback and were matched with the server, meaning they were successfully submitted to the server and observed as identical in Lookback. All of the data points from complete submitted surveys that were observed in properly recorded Lookback sessions were identical, resulting in an accurate transmission proportion of 100%.

The discrepancy between the number of participant entered data elements observed by Lookback (321) and the number present in the server (374) exist due to failure of the Lookback Software to record the sessions. On average, 25% of data was not observed in Lookback (Appendix A, Table 5).

Usability

Among the 21 participants who completed the usability survey, half (11) had reported events during the study period. The average age of participants who completed the usability survey was 34.7 y (IQR 13) and their mean technology readiness index score was 3.35 (Table 2). Our sample scored slightly lower than average in the dimension of “Optimism” (3.73 compared to 3.75 in public average) which describes the level to which an individual has a positive view of technology and a belief that it offers people increased control, flexibility and efficiency in their lives. Approximately half (52%) were male and the majority (89%) had some form of university or college education. Individuals, on average, owned their device for 5 y. 86% of participants agreed or strongly agreed, that they would “prefer to use a mobile app-based reporting system, such as the AEFI app, instead of a web-based system on a computer,” whereas only 14% disagreed, preferring the web based methods (Table 3). Participants agreed or strongly agreed that it was easy to navigate the app (95%), create a user profile (86%), locate the AEFI surveys in the app (67%), and complete the AEFI surveys (76%).

Table 3. Usability Survey Responses (n=21).

	Strongly disagree n(%)	Disagree n(%)	Neither agree nor disagree n(%)	Agree n(%)	Strongly agree n(%)
I would prefer to use a mobile app-based reporting system, such as the AEFI app, instead of a web-based system on a computer	0(0.0)	3(14.3)	0(0.0)	8(38.1)	10(47.6)
I found it easy to navigate through the app	1(4.8)	0(0.0)	0(0.0)	11(52.4)	9(42.9)
I found it easy to create a user profile	1(4.8)	0(0.0)	2(9.5)	8(38.1)	10(47.6)
I found it easy to locate the adverse event surveys within the app	1(4.8)	0(0.0)	6(28.6)	8(38.1)	6(28.6)
I found it easy to complete the adverse event surveys	1(4.8)	0(0.0)	4(19.0)	7(33.3)	9(42.9)
On one or more occasions I found it difficult to complete a task within the app	8(38.1)	8(38.1)	2(9.5)	3(14.3)	0(0.0)
The app was inviting to use	0(0.0)	0(0.0)	8(38.1)	8(38.1)	5(23.8)
I found it convenient to complete the adverse event survey on my mobile device	0(0.0)	0(0.0)	3(14.3)	7(33.3)	11(52.4)
The CANVAS app is an efficient and realistic method of reporting vaccine adverse events	1(4.8%)	0(0.0)	1(4.8)	9(42.9)	10(47.6)

Table 4. App usability feedback submitted by participants.

Feedback	Time of Report
'doesn't work on an ipad', ‘a link to calendar to set reminders for the survey dates would be great’	After creating profile for first time on phone (had already done so on iPad)
‘tell us what format you want the phone number in’	Immediately after creating profile
‘Wasn't immediately obvious where to enter my name in set-up screen. Tried to click on “First Name” rather than to the right; only found it after entering phone number and clicking on previous field button.’	Date recruited

Participants also agreed or strongly agreed that the app was inviting to use (62%), an efficient and realistic method of reporting vaccine adverse events (91%) and that they found it convenient to complete AEFI surveys on their mobile device (85%). Only 14% of participants agreed and no one strongly agreed that they found it difficult to complete a task within the app on one or more occasions (Table 3). Throughout the study period, we received 3 pieces of feedback from participants related to app usability (Table 4).

Discussion

The primary objective of this proof of concept study was to develop and test the functionality of a mobile app for the monitoring of safety outcomes after influenza immunization. Throughout the study period, we received 62 AEFI surveys, 1 spontaneous report, 3 pieces of usability feedback and 21 usability surveys. Participants reported a total of 11 events following immunization, one of which was categorized as serious. We were able to demonstrate that a mobile app could be used to successfully and accurately transmit data from vaccine recipients to a secure server as measured by a 100% accurate transmission proportion. These results need to be taken in the context of the proof of concept nature of the study and there may be limitations in the generalizability of the findings. Our findings support the potential of utilizing mobile applications to enhance or complement adverse event reporting following immunization.

However, a functioning app is not sufficient for this purpose if there are significant barriers to use and data from this study suggest this is an important consideration.¹⁰ Only 63% of recruited participants successfully downloaded the app and logged in. Of those who did, less than half (43%) completed the usability survey. While there was support for the use of mobile apps for AEFI reporting in this subset, it is difficult to draw conclusions based on our low response rate. Additionally, we only recruited individuals who owned and utilized iOS devices. Apple's smartphone market share is approximately 30-35% compared to Android, estimated at over 60% which may have introduced challenges in recruiting of participants.¹¹

Another potential impediment to post-recruitment adoption of the app may have been related to user motivation. Logistical barriers to downloading mobile apps are greater than barriers to web usage and may have discouraged participants from engaging right from the beginning. At the time of the study, downloading an app on an iPhone required first locating the app in the App Store and then authorizing the download through entry of a password. Shortly after, Apple released an update allowing users to disable password entry when downloading free applications from the app store which may reduce this inconvenience in the future. Including AEFI reporting features within an existing app on a user device, such as an app for vaccine tracking might reduce this factor.

Strategies to facilitate downloading and use of mobile applications center upon the Technology Acceptance Model concept of “perceived usefulness.”¹² Perceived usefulness can be thought of as the degree to which a person believes using a tool would enhance their ability to perform a task in home or work life. With this in mind, an application solely for AEFI reporting is possibly not useful enough to motivate people to download an app. However, if AEFI reporting tools were integrated into another app which serves a useful purpose for the user, they may be motivated to engage in the process. In either case, a scenario where the app would be downloaded by a health care provider may also improve usage.

Another possible factor resulting in low response rate is user adeptness or comfort with new technologies. Even of the 21 usability surveys completed, the mean technology readiness scores of participants was 3.35 compared to the public average of 3.02.¹³ Our sample scored higher in Innovativeness and lower than average in Discomfort, Optimism and Insecurity, potentially indicating our population was more comfortable trying new technologies than the general public, although it is difficult to make conclusions given our small sample size. Given that less than 30% of total recruits completed the usability survey, it is plausible that the participants who consented but did not download the app or complete the usability survey would have differed significantly from those for whom we did obtain scores.

Assuming the barriers to use can be overcome, significant advantages exist for post-market vaccine safety monitoring using mobile apps. Our observation that 11 participants (28%) used the app to report some sort of event, with one event meeting the criteria of “severe” suggests the potential of this modality to facilitate reporting. Apps provide a direct, accessible mechanism for individuals to spontaneously report AEFI's as they occur, which could potentially improve reporting time. The Canadian Adverse Events Following Immunization Surveillance System relies primarily on immunization or health care providers to report adverse events to local public health units.^14,15 Provincial/territorial and local health officials collect and analyze these reports which contribute to their post-market surveillance activities. Relying solely/mostly on health care providers to complete AEFI reports could serve as a barrier to AEFI surveillance.

Mobile reporting of AEFI using SMS and web-based reporting has previously been examined in several setting, including Australia and Cambodia, and was demonstrated to have high response rates and provide real time reporting.^16-20 Mobile AEFI reporting using apps could also leverage other functionality in smartphones. Assessment of local reactions or rash could be facilitated if individuals are able to photograph lesions and transmit them with their reports. Using smartphone cameras to scan the 2D bar codes on vaccine vials can permit the integration of lot number and global trade identification number with the AEFI report.^21,22 This would help to mitigate the challenge of improve the quality of the information received by public health officials and assist in identification of lot specific issues as well as assist individuals by automating the identification of the product, instead of requiring entry via drop down menu. This capability would also permit public health officials to send notifications about defective lots, with the app determining if the individual requires notification.

There are potential disadvantages to mobile AEFI reporting. Enabling individuals to more easily report AEFI's could result in a high number of reports of mild reactions that are not currently under surveillance, which would have resource implications as they may require follow-up by local public health agencies and introduce new challenges for signal detection. However, one study of consumer reporting of AEFI's demonstrated a higher rate of serious events being reported by vaccines than by health care providers, although more research is needed to explain why this occurs.^23,17 The impact of an app that facilitates mobile AEFI reporting on vaccine hesitancy also needs to be studied. Mobile AEFI reporting permitting direct reporting by individuals would resemble the United States Vaccine Adverse Event Reporting System (VAERS) in several ways and may reflect the advantages and disadvantages of this system.^24,25 Given that we have determined that the technology is functional and secure, a more comprehensive study is warranted to examine these questions.

While this was a proof of concept, there are some important strengths to this study. We believe, to the best of our knowledge, that this is the first evaluation of mobile monitoring of vaccine safety in North America. Globally, at least one other adverse event reporting app exists.²⁶ iDsurv was developed by the Indian Academy of Pediatrics as part of the Infectious Disease Surveillance for reporting serious AEFIs with the objectives “to develop an early warning system for pediatric infectious diseases in India, to generate data on burden of infectious diseases in India and to generate data on serious AEFI.”^27,28 Our use of Lookback was also unique as it permitted us to assess whether the data entered by the user matched the data on the server.

The primary limitations of this study relate to its sample size and the generalizability of the results given the nature of those who would participate in a technology oriented study. As this was a proof of concept study focusing on determining if the technology was functional it was not powered to examine usability as a primary outcome. A related limitation is that we only have usability data on those who used the app and completed the usability survey at day 30, which likely represented a biased sample. In the future it would be useful to have baseline surveys of participants to determine if those who completed the study differed from those who did not. A final limitation is that 25 % of completed AEFI survey data was not recorded in Lookback as a result of Lookback failure and could therefore not be compared with the server data. However, it is unlikely that the unrecorded data would differ systematically from the rest of the data with respect to accurate transmission proportions.

We have demonstrated that a mobile AEFI app is functional and may have the potential to complement existing post-market surveillance systems. Whether this technology is usable and acceptable to a large enough segment of the population to be of value still requires elucidation. While this study suggests a standalone app is technically feasible, we believe that integration into a general immunization app could encourage downloads and usage. In this scenario the app could be useful not only for health care workers influenza AEFI reporting as in this study but also for general adverse event reporting. Future studies should examine broader implementation of mobile AEFI reporting to examine its acceptability, and whether it has an impact on the quality and comprehensiveness of reporting of adverse events following immunization.

Methods

Population and setting

English speaking individuals over the age of 18 y who owned and used a compatible smartphone were eligible to participate in this study. Participants were required to use an iPhone running iOS 7 or higher. We formally recruited participants face-to-face for one day at a staff and family member's influenza immunization clinic run by Occupational Health at The Ottawa Hospital in November 2014 as well as through word of mouth using posters in the Occupational Health Offices at the Civic Campus. The Ottawa Hospital is one of the largest teaching hospitals in Canada with over 1100 beds across 3 Campuses employing over 1300 physicians. The clinic used for recruiting was held on a Saturday (11/16), at the Riverside, a day-hospital with no inpatient beds. The clinic was open to staff members and their immediate family members only. After being immunized, attendees were asked to wait for in a post-immunization waiting area in case of adverse reactions. Two study staff members approached potential participants during this waiting period and asked attendees if they were interested in participating. If they obliged, written informed consent was immediately collected. Participants had to provide study staff with their email address at the time of enrollment. Study staff then registered participants email address in the backend administration system permitting users to access the app after downloading it. When the email address was registered in the administration system, it generated an email to the participant with a link to download the app. Participants had to click on this link then were taken to the CANVAS app in the iOS app store where they could download the app, login and create a profile. If participants did not download the app and login using their email address provided, a reminder email notification with the link to the app was generated and sent to the address provided at enrollment.

In addition to the clinic, we recruited informally through word of mouth and posters in occupational health offices throughout November and December 2014. For participants recruited in this method, the same process of collecting consent and registering email addresses was followed.

The study was approved by the Ottawa Health Science Network Research Ethics Board and the University of British Columbia Research Ethics Board.

Intervention

Canadian national vaccine safety network

Since 2009, the CANVAS network gathers and analyzes safety data on thousands of individuals who receive the influenza vaccine to provide select safety information during the early stages of the annual influenza vaccination campaign.^29,30 Participants for CANVAS are recruited from acute care, public health and pharmacy influenza vaccination clinics in 5 provinces across Canada (British Columbia, Alberta, Ontario, Quebec, Nova Scotia) and complete a short web-based survey 8 d after immunization. Surveys identify if participants had severe events, defined as any new health problem or the exacerbation of an existing condition that is severe enough to cause work or school absenteeism/prevent daily activities or to require a medical consultation. Trained nurses follow-up selected severe reported events for validation. Longer-term follow up of 28 d and 6 months post-vaccination has occurred in previous years, but has not been included in recent years.

CANVAS surveys

The AEFI Surveillance surveys administered through the app were identical to surveys used by CANVAS either in 2014/2015 flu season or ones prior. Their content was not modified for the app.

Day 8 survey contains a maximum of 10 questions. The first 5 questions pertained to demographics and history of influenza vaccination, including age bracket, who they were completing this survey on behalf of and where they received the flu shot. They were then asked “In the first week (7 days) after your flu shot did you develop a new health problem or did an existing health problem get worse?.” If respondents answered “no” to this question, they were directed to the final survey questions. If they answered “yes” they were led to an additional 2 questions gathering details on their changing health condition.

The Day 30 survey contained a maximum of 8 questions, the first who they were completing the survey on behalf of and then moved directly onto “In the last month after your flu shot did you develop a new health problem or did an existing health problem get worse”?. Again, if respondents answered “no” to this question, they were directed to the final survey questions. If they answered “yes” they were led to an additional 2 questions gathering details on their changing health condition.

The Combined Day 8 and Day 30 survey had a maximum of 23 questions, covering the same demographical questions as the above surveys and asked specifics regarding health problems occurring both within the first week following vaccination and in the subsequent days leading up to Day 30.

Usability survey

Unlike the AEFI Surveillance surveys, the usability survey was developed specifically for use in this study. The survey was first pilot tested on 10 individuals to ensure the instructions and questions were clear. Data from the pilot surveys were not included in the final analysis.

The survey collected additional participant demographic information (e.g. age, education, employment status, experience with smartphones, smartphone device currently using, internet use), opinions regarding perceived ease of use and future intent to use the app, and concerns about the security and privacy of personal health information while communicating from mobile devices. The survey also measured key usability domains of the app and participant baseline technology readiness index (TRI) scores.^13,31 TRI quantifies 4 attitudes toward technology, 2 motivators (Optimism, Innovativeness) and 2 inhibitors (Discomfort and Insecurity) which gauge individual propensity to adopt a new technology to accomplish a goal in home or work life.

App usability was captured on a 5 point Likert scale with questions such as: “I would prefer to use a mobile app-based reporting system, such as the AEFI app, instead of a web-based system on a computer,” “I found it easy to navigate through the app,” “I found it easy to complete the adverse event surveys,” and “the CANVAS app is an efficient and realistic method of reporting vaccine adverse events.”

Mobile app and study procedure

The CANVAS app (Appendix B) was developed on the iOS platform for this study.³² The app includes a profile, where participants were asked to enter their name, phone number, influenza vaccine date and vaccine product received. A drop down menu was available for participants to select product received (Agriflu®, Fluad®, FluMist®, Fluviral®, FluZone®, Influvac®, Intanza®, Vaxigrip®, don't know). The app was pre-programmed with the capability of delivering all 4 surveys. We also included the opportunity for participants to report AEFI's spontaneously, at any time outside of the CANVAS surveys.

Participants were prompted to complete the surveys using automatically generated internal notifications at days 8 and 30 after immunization to prompt survey completion, similar to the email reminders sent to participants in the current validated CANVAS web based methods. Additional automated internal notifications at days 11 and 33 after immunization were sent to those participants who had not yet completed their survey. At day 30, a combined day 8 and 30 survey was provided to participants who did not complete the Day 8 survey.

At completion of the Day 30 AEFI survey, participants were prompted to complete the usability survey, to evaluate their user experience with the app. The app also provided a mechanism for participants to provide feedback to the research team on the app performance and usability throughout the study period and request technical assistance.

In either survey, those who reported no health changes after immunization were taken to the end of the survey and exited the app. Participants who reported a serious event (i.e., an event that prevent daily activities or resulted in a medical visit) were asked additional questions about the event.

All data entered into the app by the user was encrypted before being transmitted to the secure server. Study staff were provided a web portal with which to access the survey results and user data. An alert system was built in to notify staff if specific severe adverse events were reported through the app. Participants were notified in the app that they may be contacted by a nurse if they reported a severe adverse event.

Data collection

The proportion of participants who downloaded the app and completed the Day 8, 30 or combined surveys or used the spontaneous adverse event reporting option was recorded. The date and time participants submitted the surveys were also recorded. Responses to each survey question were also recorded in the server and defined as “data points.” This includes multiple choice question responses, drop down menus and free text fields.

Lookback, a software plug-in, was used to record a screen capture video of participant's app use.³³ Lookback was selected for use in this study because it permitted study staff to visually compare video of participants generating data points on the app to the data points residing in the secure server. Each app session recorded by Lookback where participants engaged with an AEFI surveillance survey or spontaneous report was extracted and manually reviewed by one member of the study staff and observed data points were entered in an excel spreadsheet then compared to the data points found on the server. Not all app sessions were successful recorded in Lookback as the software was still in its β testing phase.

Analysis

We defined accurate transmission proportion as the percentage of survey responses successfully delivered to the server that matched the survey responses sent from the app, as observed using Lookback. The total response rate, Day 8 survey, Day 30 survey and combined survey response rate were calculated by determining the number of individuals who submitted surveys divided by the total number of participants. We also calculated the mean time of Day 8 and Day 30 survey submission from time of immunization and from when the survey was made available to the participant. We calculated the mean value for Likert scaled responses from the usability survey. We recorded all reports of adverse events as well as comments on usability of the app.

Acknowledgments

Thank you to the Canadian Association for Immunization Research and Evaluations (CARIE) for their networking assistance. Finally, the authors gratefully acknowledge the expert assistance provided by the Vaccine Evaluation Center, public health and hospital collaborators.

Funding

Thank you to our funders, the Public Health Agency of Canada and the Canadian Institutes of Health Research.

Appendix A

 

Figure 1. Total Consenting Individuals Participating in Study.

Table 1. Demographics of total study participants who submitted survey responses.

Demographics	Total (n=38)	On-site (n= 25)	Off-site (n=13)
What is your age	n (%)	n (%)	n (%)
20–29	13 (34)	9 (36)	4 (31)
30–39	11 (29)	7 (28)	4 (31)
40–49	10 (26)	7 (28)	2 (15)
50–59	4 (11)	2 (8)	3 (23)
What is your gender	n (%)	n (%)	n (%)
Female	19 (50)	13 (52)	7 (60)
Male	19(50)	12 (48)	6 (40)
Where did you receive your flu shot this year?	n (%)	n (%)	n (%)
Hospital Immunization Campaign	31 (82)	24 (96)	7 (54)
Pharmacy	3 (7)	0 (0.0)	3 (23)
Public Health	1 (3)	1 (4)	0 (0)
Other	3 (8)	0 (0.0)	3 (23)
In the last two years, how many times have you received the flu shot?	n (%)	n (%)	n (%)
0	2 (5)	1 (4)	1 (7.5)
1	4 (11)	3 (12)	1 (7.5)
2	32 (84)	21 (84)	11 (85)

Table 2. Demographics of study participants who completed the usability survey.

Measure	Total n (%)	On-site (n= 15)	Off-site (n=6)
Age (yrs)	34.7 (IQR 13)	36.8 (IQR 14)	34.0 (IQR 10.5)
Male	11 (52.4)	6 (40)	3
Female	10 (47.6)	7 (60)	3
Years of Owning a smartphone device
Average	5.21	5.06	5.58
Education
Some Post-sec	1 (4.8)	1 (6.7)	0 (0.0)
Comm college	3 (14.8)	2 (13.3)	1 (16.6)
Undergrad deg	9 (42.9)	7 (46.6)	2 (33.3)
Grad deg	5 (23.8)	2 (13.3)	3 (50)
Other (Specify)	3 (14.3)	2 (13.3)	0 (0.0)
Post doctorate	1 (4.8)	1 (6.7)	0 (0.0)
Employment
Employed for wages	19 (90.5)	13 (86.6)	5 (83.3)
Self-Employed	3 (14.3)	1 (6.6)	1 (16.6)
Not employed	0 (0.0)	1 (student) (6.6)	0 (0.0)
Technology related profession
Yes	7 (33.3)	5 (33.3)	2 (33.3)
No	14 (66.7)	10 (66.6)	3 (66.6)
Access to the Internet
> 5 times/day	21 (100)	15 (100)	6 (100)
TRI score
Mean	3.35	2.68	3.54

Table 3. Usability Survey Responses Combined.

	Strongly Disagree	Disagree	Neither Agree nor Disagree	Agree	Strongly Agree
Total n (%)
On-site n (%)
Off-site n (%)
I would prefer to use a mobile app-based reporting system, such as the AEFI app, instead of a web-based system on a computer	0 (0.0)	3 (14.3)	0 (0.0)	8 (38.1)	10 (47.6)
	0 (0.0)	3 (20)	0 (0.0)	4 (27)	8 (53)
	0 (0.0)	0 (0.0)	0 (0.0)	4 (66)	2 (33)
I found it easy to navigate through the app	1 (4.8)	0 (0.0)	0 (0.0)	11 (52.4)	9 (42.8)
	1 (6.7)	0 (0.0)	0 (0.0)	8 (53.3)	6 (40)
	0 (0.0)	0 (0.0)	0 (0.0)	3 (50)	3 (50)
I found it easy to create a user profile	1 (4.8)	0 (0.0)	2 (9.5)	8 (38.1)	10 (47.6)
	1 (6.7)	0 (0.0)	1 (6.7)	6 (40)	7 (46.6)
	0 (0.0)	0 (0.0)	1 (16)	2 (33)	3 (50)
I found it easy to locate the adverse event surveys within the app	1 (4.8)	0 (0.0)	6 (28.5)	8 (38.1)	6 (28.6)
	1 (6.7)	0 (0.0)	5 (33.3)	4 (26.7)	5 (33.3)
	0 (0.0)	0 (0.0)	1 (16.7)	4 (66.6)	1 (16.7)
I found it easy to complete the adverse event survey s	1 (4.8)	0 (0.0)	4 (19.0)	7 (33.3)	9 (42.9)
	1 (6.7)	0 (0.0)	3 (20)	5 (33.3)	6 (40)
	0 (0.0)	0 (0.0)	1 (16.7)	2 (33.3)	3 (50)
On one or more occasions I found it difficult to complete a task within the app	8 (38.1)	8 (38.1)	2 (9.5)	3 (14.3)	0 (0.0)
	6 (40)	6 (40)	2 (13.3)	1 (6.7)	0 (0.0)
	2 (33.3)	2 (33.3)	0 (0.0)	2 (33.3)	0 (0.0)
The app was inviting to use	0 (0.0)	0 (0.0)	8 (38.1)	8 (38.1)	5 (23.8)
	0 (0.0)	0 (0.0)	6 (40)	6 (40)	3 (20)
	0 (0.0)	0 (0.0)	2 (33.3)	2 (33.3)	2 (33.3)
I found it convenient to complete the adverse event survey on my mobile device	0 (0.0)	0(0.0)	3 (14.3)	7 (33.3)	11 (52.4)
	0 (0.0)	0 (0.0)	2 (13.3)	5 (33.3)	8 (53.3)
	0 (0.0)	0 (0.0)	1 (16.7)	2 (33.3)	3 (50)
The CANVAS app is an efficient and realistic method of reporting vaccine adverse effects	1 (4.8)	0 (0.0)	1 (4.8)	9 (42.8)	10 (47.6)
	1 (6.7)	0 (0.0)	1 (6.7)	5 (33.3)	8 (53.3)
	0 (0.0)	0 (0.0)	0 (0.0)	4 (66.7)	2 (33.3)

Table 4. Source of Data Transmitted to Server.

Data elements transmitted (AEFI Surveys)	Data Points Observed in Lookback	Data Points Recorded in Server	Matched Data Points	Identical Data Points	Completed Survey Data Points Observed in Lookback	Percentage of data lost in Lookback
Day 8 Survey	231	253	218	218	193	23.71541502
Day 30 Survey	61	85	45	45	56	34.11765
Combined (Day 8 & 30) Survey	28	35	27	27	28	20
Total	321	374	291	291	278	25.73727

Table 5. Sessions examined in Lookback.

Data Source (n)	Number of Data Points (Server)
Completed Day 8 Surveys (35)	253
Completed Day 30 Surveys (24)	85
Completed Combined Surveys (3)	35
Spontaneous Reports (1)	1
Total	374