Public education about ShakeAlert^® earthquake early warning: evaluation of an animated video in English and Spanish

ABSTRACT

Publicly available technologies are only truly ‘public’ if people are aware of them and know how to use them. In the US, significant resources have been devoted to public education around the recently released ShakeAlert® earthquake early warning (EEW) system operated by the U.S. Geological Survey; however, those efforts have been inconsistently evaluated, leaving educators and communicators with questions about their effectiveness, particularly for engaging broad, diverse, and multilingual publics. This mixed methods, dual-language (English/Spanish) study addresses this gap by examining an animated video designed to increase awareness, understanding, and action related to ShakeAlert EEW. Results indicate that the video was enjoyable, easily understandable, and positively impacted viewers’ intentions to ‘Drop, Cover, and Hold On’ (DCHO) following a ShakeAlert-powered alert. Viewers also expressed a desire for more information about how to receive alerts and how to protect themselves in situations where DCHO is impractical. This study suggests that animated video, developed and disseminated in multiple languages, can be an effective tool for impacting public understanding and behavior related to geohazards. Additionally, the process and findings of this study suggest that outcomes may be improved by engaging multilingual and multicultural audiences earlier and throughout the video development process.

KEYWORDS:

• Earth science education
• visual media
• science, technology, and society
• earthquake early warning

Introduction

More than 50 million people living on the U.S. West Coast could potentially experience damaging earthquakes (FEMA, 2017). Though earthquakes cannot be predicted, earthquake early warning (EEW) systems can provide real-time notice of an earthquake that has begun, prompting people and systems to take immediate protective action. The EEW system in the United States, ShakeAlert^®, is operated and managed by the U.S. Geological Survey (USGS) and became active in the state of California in 2019, followed by Oregon and Washington in 2021 (e.g. McBride et al., 2022; Sumy et al., 2022). Currently, publicly available ShakeAlert-powered alerts can reach individual mobile devices in three ways: 1) the Integrated Public Alert and Warning System (IPAWS), which delivers Wireless Emergency Alerts (WEAs, the source of AMBER alerts); 2) built-in systems for Google Android mobile devices; and 3) select, downloadable apps, e.g. MyShake and others. Additionally, select transit systems, utilities, and other industries use data from the ShakeAlert system to enable site-specific, automated protective actions, such as slowing down a train or shutting off a gas valve.

Though ShakeAlert EEW is operational across the U.S. West Coast, the availability of the technology itself does not mean that the millions of geographically, culturally, and linguistically diverse people living in and visiting the region know what EEW is, are ready to receive alerts, can recognize alerts when received, and/or know how they should respond. For this reason, strategic, culturally responsive, and evidence-based education, communication, and evaluation must coincide with system development and rollout (Jenkins et al., 2022; Kelman & Glantz, 2014; Kohler et al., 2018; Sumy et al., 2022, 2022b).

In anticipation of the rollout of public alerting and in response to this need for strategic and inclusive public education, the USGS convened the ShakeAlert Joint Committee for Communication, Education, Outreach, and Technical Engagement (JCCEO&TE) in 2016 (McBride et al., 2022). Within this broad interdisciplinary committee of seismologists, social scientists, emergency managers, and educators, the Educational Resources Working Group (ERWG) is specifically tasked with developing, disseminating, and evaluating educational materials that increase public awareness and understanding of ShakeAlert (Sumy et al., 2022). The present study is focused on the effectiveness of a four-minute animated video about ShakeAlert, developed by ERWG and in partnership with the Incorporated Research Institutions for Seismology (IRIS, now merged with UNAVCO as Earthscope Consortium). The video, ‘ShakeAlert – Earthquake Early Warning System for the West Coast of the U.S. (2020),’ was made available in English and Spanish in September 2020. Throughout the article, we reference the animated video simply as the ‘ShakeAlert video’ (IRIS Earthquake Science (EarthScope Consortium), 2020a; IRIS Earthquake Science (EarthScope Consortium), 2020b).

For decades, significant investments have been made in establishing the technical aspects of the ShakeAlert system (e.g. Kohler et al., 2020); however, much less time, attention, and money has been devoted to establishing public awareness, attitudes, or behaviors related to that system. Existing studies indicate that in the U.S., overall awareness of ShakeAlert is low (Bostrom et al., 2022; Morgoch, 2022) and adults do not consistently respond to earthquake alerts and/or shaking as recommended (i.e. by practicing Drop, Cover, and Hold On) (Adams et al., 2022; Baldwin, 2022; Bostrom et al., 2022; McBride et al., 2022). Efforts to address these challenges through public education are ongoing (e.g. Sumy et al., 2022; 2022b). Such interventions have been inconsistently evaluated, however, leaving ShakeAlert educators to wonder about their impact on public awareness, understanding, attitudes, behaviors, or other outcomes vital for advancing public safety in the event of a damaging earthquake. This evaluation gap is reflective of the state of earthquake education in general, where hazards-based educational campaigns are too often based on hunch, anecdotal evidence, or entrenched habits, rather than data about what works and what does not (Marti et al., 2020; Nathe, 2000). Marti et al. (2020) performed a comprehensive review of published literature related to public earthquake education campaigns, finding just a handful of studies that assessed the impact of educational interventions on earthquake preparedness behaviors, interest, and information-seeking. Within this relatively small body of literature, no studies focused on educational efforts specifically related to earthquake early warning, which given its recent rollout, is understandable while also notable. Similarly, in the U.S., where an estimated 42.5 million individuals speak Spanish as their native language, we are aware of no studies assessing the effectiveness of earthquake education materials for Spanish-speaking or multilingual audiences – including materials developed by USGS (e.g. USGS, 2022), IRIS/EarthScope Consortium (e.g. Bravo et al., 2019, december), and other ShakeAlert partners (e.g. Benaron & Benaron, 2020).

In light of the general absence of evaluation in public earthquake education, this study has relevance for the broader field of educators and communicators who hope to meaningfully engage public audiences in multiple languages to increase preparedness for earthquakes and other hazards. This study evaluates the ShakeAlert video’s impact on three learning outcomes – understanding, enjoyment, and behavior – and examines the features of the video which may have contributed to the observed outcomes. In choosing these specific outcomes, we drew upon the empirically-based Protective Actions Decision Model (PADM), which focuses on the relationship between cognitive factors (including awareness and understanding) and behavior, as mediated by other social and environmental factors (Lindell & Perry, 2012). We also considered frameworks for informal science education, which focus on cognitive outcomes in parallel with non-cognitive outcomes such as enjoyment and interest (e.g. Allen et al., 2008; National Research Council, 2009; Wasserman et al., 2022). For the purposes of this study, we examine enjoyment, understanding, and intended behavior separately, while recognizing that in the context of natural hazards education, these constructs are related and potentially mutually reinforcing. For example, enjoyment may support engagement, which in turn supports deeper understanding (e.g. Benitez-Galbraith & Galbraith, 2021); knowledge may affect behavior (e.g. Lindell & Perry, 2012); and, engaging in a behavior may reinforce understanding (e.g. Vinnell et al., 2020).

The following questions guide our evaluation of the Spanish and English versions of the ShakeAlert video:

1. Understanding: How effectively does the video convey key messages related to ShakeAlert? What features of the video enable or hinder viewers’ understanding of key messages?

2. Attitude: To what extent do viewers find the video enjoyable? What features of the video relate to enjoyment?

3. Behavior: What impact does the video have on viewers’ intended behavior related to earthquakes and ShakeAlert EEW?

Video in science education and science communication

Educational videos have the potential to reach broad audiences and promote awareness and understanding of socially significant scientific topics such as EEW. Animated videos, in particular, have long been used in the formal science classroom, where they have been shown to improve student understanding of complex scientific concepts (e.g. Rundgren & Tibell, 2010; Willis et al., 2021; Yang et al., 2003). Animated media may be especially valuable for illustrating geoscience processes that learners may not have personally experienced (such as an earthquake), or cannot directly observe, due to the spatial or temporal scales on which they occur (e.g. Stern et al., 2020; Wysession & Baker, 2002). Additionally, compared to strictly textual, auditory, and/or visual presentation, animation – when it strategically combines these multiple modes of presentation – may reduce cognitive load and contribute to greater understanding and retention (Cook, 2006; Mayer & Moreno, 2002; Putortì et al., 2020).

While there is extensive evidence that animated media can lead to increased understanding of geoscience concepts, particularly in a formal education context, there are fewer studies examining the use of animated video outside the classroom or evaluating the impact of such videos on non-cognitive learning outcomes such as enjoyment or behavior. Velho et al. (2020), for example, examined a broad sample of science videos available on YouTube. They found that, of the multiple formats included in the sample (e.g. vlog, interview, documentary, animation, etc.), animated videos were the most popular. This suggests a relationship between the format of animation and the outcome of enjoyment. Additionally, various, short educational videos (though no animated videos, to our knowledge) have been linked to intended and/or actual behavior change, including engagement in conservation activities (e.g. Finkler et al., 2019), performance of healthcare behaviors (e.g. Garbers et al., 2015; Wiese et al., 2005), and practice of emergency response behaviors (e.g. Skurka et al., 2018). In one experimental study of earthquake protective behaviors, Sutton et al. (2019) found that participants who watched an educational video prior to being prompted to ‘Drop, Cover, and Hold On’ more consistently performed the correct protective action, compared to participants who did not watch the educational video. Together, these studies suggest that short-form, educational video (and animated video, in particular) is a promising, but under-researched avenue for promoting understanding, action, and even enjoyment related to the process of learning about science, technology, and safety.

As internet- and social media-based media platforms like YouTube and TikTok continue to grow in popularity, the potential for online video to reach broad, public audiences increases. Online video accounted for 80% of worldwide internet traffic in 2021 and usage continues to grow (SkyQuest Technology, 2022). Rosenthal (2018) notes that, ‘as of September 2016, ten of the more popular science-related YouTube channels collectively have close to 50 million subscribers and 6 billion video views’ (p. 5) and, for this reason, YouTube has become a significant informal science learning space. At the same time, for an online video to make an impact in this crowded market, it must appeal to learners in one or more ways, including satisfying their personal learning goals, sparking interest, or producing enjoyment (Moll & Nielsen, 2017; Rosenthal, 2018).

Shakealert video development and content

The ShakeAlert video was created by an interdisciplinary team of ERWG scientists, educators, and emergency managers, in partnership with IRIS (now the EarthScope Consortium) and with funding from USGS. The video took approximately six weeks to conceptualize, starting with a narration document. The document was then reviewed by the interdisciplinary team over a span of two to three weeks. Once the narration was settled on, the video took about two weeks to animate and another two to three weeks to gather input and review. Overall, the video took approximately six months to conceptualize, develop, and edit, and the final version was released in September 2020. The central goal of the video was to offer a single, short, clearly understandable media resource that addresses five key messages (Table 1) in a series of animated and visual sequences (Figure 1). The intended audience for the video is the broad public (recognizing the wide range of identities, abilities, and affiliations within that population), with a focus on people living in the West Coast states of California, Oregon, and Washington where ShakeAlert is currently available.

Figure 1. Scenes from ShakeAlert video.

Table 1. Key messages for ShakeAlert video.

	Key message (English)	Key message (Spanish)
1	ShakeAlert is an earthquake early warning system.	ShakeAlert es un sistema de alerta temprana de terremotos.
2	ShakeAlert does not predict earthquakes.	ShakeAlert no puede predecir los terremotos.
3	Depending on how close you are to an earthquake’s epicenter, you may get an alert before, during, OR after shaking.	Dependiendo de qué tan cerca estés del epicentro, es posible que recibas una alerta antes, durante o incluso después del terremoto.
4	ShakeAlert messages prompt individuals to take protective actions.	Los mensajes de ShakeAlert ayudan a que las personas se protejan.
5	If you feel an earthquake, you should Drop, Cover, and Hold On.	Si sientes un temblor o terremoto, debes agacharte, cubrirte y agarrarte*.

*For key message 5, the phrase ‘hold on’ was translated using the verb ‘sujetar’ in the video and ‘agarrar’ in the survey. This discrepancy was unintentional and, though it doesn’t change the overall meaning of the key message, it may have affected participants’ response to survey prompts related to key message 5.

The video opens by inviting the viewer to consider: What would you do if you had seconds to prepare for an earthquake that had already begun? (Figure 1a). The video then describes how the ShakeAlert System works, using a network of sensors (Figure 1b) to detect two main kinds of seismic waves: first, the fast-moving primary (P) waves, followed by the slower-moving, but typically more damaging, secondary (S) waves (Figure 1c). Seismic sensors then send data to processing centers, where advanced algorithms quickly calculate the location and magnitude of the earthquake and anticipate the intensity and extent of shaking. Alert delivery partners use this information to send alerts to mobile phones in impacted areas, prompting users to ‘Drop, Cover, and Hold On’ (Figure 1d). Because electronic data travel faster than seismic waves (particularly the slower S-wave), individuals outside of the late alert zone (Figure 1e) may receive seconds of warning before strong shaking begins. The video also explains that ShakeAlert-powered alerts may trigger automated actions, such as slowing down trains to prevent derailment. The video includes one segment of live-action footage from the 2018 Anchorage, Alaska earthquake (Figure 1f). In the clip, the initial jolt of the P-wave is clearly visible, followed by stronger S-wave shaking. The footage also models earthquake protective actions, with the teacher and students calmly dropping and covering beneath their desks as soon as shaking is felt.

Spanish video development process

Spanish is the second-most commonly spoken language in the U.S., with over 11 million people in California, Oregon, and Washington reporting they speak Spanish at home (Figure 2). Currently, ShakeAlert-powered alerts are available in Spanish via WEA, which delivers warnings to mobile phones in either Spanish or English, corresponding to the settings of the receiving phone. Though ShakeAlert-powered alerts are available in Spanish, simply making the technology compatible with a minority language is insufficient. Education efforts also must reach Spanish speakers and communicate in a way that meets their linguistic and cultural needs (e.g. Jenkins et al., 2022; Medina Luna et al., 2019).

Figure 2. Language spoken at home, age 5 and over, U.S. West Coast (CA, OR, WA), 2019. Figure developed based on data from Migration Policy Institute (2020).

The ShakeAlert video was translated into Spanish using a multi-step, multi-reviewer process. Three native Spanish speakers, including one geoscientist and two informal science educators, developed and reviewed a translation of the video; one of these team members provided the narration. While the video was thoughtfully translated, it was not initially developed with Spanish-speaking audiences in mind. The team that created the original, English language ShakeAlert video included individuals with extensive experience and expertise related to seismology and geoscience education, but no individuals who speak Spanish or identify as Latina/o/x. This fact informed our approach to the analysis and is a point we return to in the Discussion and Recommendations.

Methods

Study design

This study was conducted by a team with experience in informal science education, seismology, natural hazards risk communication, social sciences, and culturally responsive evaluation. One study team member was also part of the video development team. Given the nature of our study questions, which include both deductive inquiries (i.e. assessing the video’s effectiveness with regard to predetermined learning outcomes) and inductive inquiries (i.e. identifying features and qualities of the video that illustrate, explain, or otherwise relate to those outcomes), we designed a mixed-methods study using surveys and interviews conducted in both English and Spanish. Our mixed-methods design used a convergent parallel approach, in which quantitative data (from surveys) and qualitative data (from open-ended survey questions and in-depth interviews) were collected simultaneously, analyzed separately, and then considered together with regards to the study questions (Creswell, 2014). The integration of data was complementary, with quantitative data typically addressing the more deductive questions related to effectiveness and the qualitative data illuminating unforeseen issues and illustrating, contextualizing, and/or explaining the patterns seen in the quantitative data (Greene et al., 1989; Onwuegbuzie & Johnson, 2006).

All research instruments (including recruitment materials, survey, and interview script) were developed first in English and then translated into Spanish by a native Spanish speaker; two additional native Spanish speakers reviewed the translations to ensure accuracy and consistency. To encourage replication and future reuse, the team published the instruments in their entirety (Crayne et al., 2022). While data were collected in both English and Spanish, these data were drawn from non-equivalent samples; for this and other reasons, we refrain from comparing the English and Spanish versions of the ShakeAlert video to each other. For the purposes of this study, the two videos were treated as separate products, each of which was evaluated independently, relative to the research questions.

Both the English and Spanish surveys were hosted on the online survey platform Alchemer. Following an informed consent page, the first of four sections included closed-ended (Likert-type) and open-ended response items inviting respondents to report their existing knowledge and intended behaviors related to ShakeAlert. In the second section, participants were prompted to view the ShakeAlert video, which was embedded directly in the survey; a page timer prevented participants from advancing until four-minutes had elapsed, encouraging them to view the video in its entirety. In the third section, respondents reported their post-viewing knowledge and intended behaviors by responding to a repeat set of closed-ended response items from section one. Additionally, participants were asked to report how well the video communicated key messages about ShakeAlert and to what degree the video was understandable and enjoyable, again using both closed- and open-ended response items. Finally, in the fourth section, participants were asked to share demographic information, including age, gender, ZIP code, race/ethnicity, and educational attainment. The entire survey, including the time allowed to watch the video, took an average of 15 min to complete.

Study team members conducted interviews in English and Spanish with participants online via a video conferencing platform. Informed consent was performed verbally. To open, the interviewer first asked the participant whether they had ever experienced an earthquake and if so, to describe that experience. This question was included both as a ‘warm up’ and to set the tone of the interview by encouraging participants to supply personal answers based on their own experience, knowledge, and views, rather than prompting ‘correct’ answers, as in a quiz or a test. Following this introduction, participants were given a link to the video and asked to watch it on their local device. After watching, participants were invited, via a series of scripted questions, to share their impressions of the video, including what they learned and what about it was understandable and enjoyable (or not). In order to learn more about participants’ behavioral intentions, participants were asked, ‘If you were to get a ShakeAlert notification on your phone right now, what do you think you would do?’ Additionally, participants were invited to ask any questions they might have about ShakeAlert, which created an opportunity for a more open-ended conversation between the interviewer and interviewee. Finally, participants were asked to share demographic information, including age, gender, ZIP code, race/ethnicity, and educational attainment. Interviews lasted 20-30 min and respondents received a $20 Amazon gift card. The interview was audio recorded for later transcription and analysis.

Recruitment

Our team recruited adults (18+, survey and interviews) and youth (ages 13-17, survey only) living in California, Oregon, or Washington. In order to understand the role that prior awareness of ShakeAlert may play on viewers’ understanding and impressions of the video, recruitment materials did not mention ShakeAlert specifically, but rather invited people to participate in a study to ‘help improve educational materials about earthquakes.’ Participants were recruited via two main avenues: network recruitment and panel recruitment.

Network recruitment was achieved primarily through social media posts on Facebook and Twitter. Posts inviting individuals to participate in a survey or interview were initially posted by the Oregon Museum of Science and Industry (OMSI) and then shared by the USGS, IRIS, and other partner organizations associated with ShakeAlert CEO&TE. Flyers were also emailed to various educational and community-based organizations to share electronically with their members (Crayne et al., 2022). Because network recruitment strategies drew from a pool of people already associated with the organizations doing the recruitment, respondents recruited via these means were expected to have some prior awareness, knowledge, and interest relative to earthquakes generally and ShakeAlert EEW specifically. Network-based strategies failed to recruit sufficient numbers of survey respondents for the Spanish video (Table 2). Additionally, the network-recruited sample included an over-representation (relative to the general West Coast population) of respondents who lived in Oregon, had college/graduate degrees, and/or identified as White, non-Latina/o/x.

Table 2. Sample sizes for survey and interview, by video language and recruitment source.

	Survey		Interview
	English Video	Spanish Video	English Video	Spanish Video
Network recruitment	510	16	9	10
Panel recruitment	186	63	-	-
Total participants	696	79	9	10

In order to broaden the sample of participants to those outside of the previously described networks and to include more Spanish-speakers, people from Washington and California, and people reporting non-dominant racial and ethnic identities, supplemental respondents were recruited via a paid panel from Alchemer. Panel respondents were drawn from an existing pool of individuals who complete online surveys in exchange for small incentives (points, gift cards, etc.). Panel respondents included both English-speaking and Spanish-speaking individuals who were presented with English and Spanish versions of the survey, respectively. Some recruited participants were bilingual and provided responses in a mix of English and Spanish. For the purposes of analysis, we divided participants based on the language of the video they watched within the survey or interview. So, if the respondent watched the video in English, we retained this as an English response, even if the respondent identified as a Spanish-speaker and/or provided written responses in Spanish.

Participant demographics

After cleaning the dataset of records that were incomplete, duplicate, or suspected to have been supplied by bots (Simone, 2019), the final sample included 775 survey responses (696 for the English video; 79 for the Spanish video) and 19 interview responses (9 for the English video; 10 for the Spanish video), collected between August 2021 and March 2022 (Table 2).

Participant demographics were collected for the complete set of respondents participating in surveys and interviews (Table 3). Approximately half of survey participants had heard of ShakeAlert prior to watching the video. Both the interview and survey samples skewed female. In terms of race and ethnicity (for which participants were prompted to ‘select all that apply’), the majority of English video respondents identified as White/Caucasian and the majority of Spanish video respondents identified as Latina/o/x. In terms of age, education, and state of residence, patterns varied across the four categories of participation. Though the survey was open to youth, very few completed it.

Table 3. Profile of participants for English and Spanish video surveys and interviews.

	English video survey		English video interview		Spanish video survey		Spanish video interview
	Frequency	Percent*	Frequency	Percent*	Frequency	Percent*	Frequency	Percent*
Have heard of ShakeAlert (pre)
Yes	359	52%	N/A	-	36	46%	N/A	-
No	330	48%	N/A	-	43	54%	N/A	-
Total responding	689	-	-	-	79	-	-	-
Age
13–17	4	1%	0	0%	1	1%	0	0%
18–24	42	6%	1	11%	9	13%	0	0%
25–34	168	24%	1	11%	18	25%	7	70%
35–44	156	23%	2	22%	21	29%	3	30%
45–54	95	14%	3	33%	15	21%	0	0%
55–64	108	16%	2	22%	6	8%	0	0%
65–74	100	14%	0	0%	2	3%	0	0%
75+	20	3%	0	0%	0	0%	0	0%
Total responding	693	-	9	-	72	-	10	-
Gender
Female	362	55%	6	67%	41	57%	7	70%
Male	261	39%	2	22%	23	32%	3	30%
other/nonbinary	40	6%	1	11%	8	11%	0	0%
Total responding	663	-	9	-	72	-	10	-
Race and ethnicity**
American Indian, Alaska Native, or Indigenous	27	4%	2	22%	1	1%	0	0%
Black or African American	25	3%	0	0%	4	5%	0	0%
White/Caucasian	538	73%	7	78%	10	12%	0	0%
Asian American	32	4%	0	0%	1	1%	0	0%
Native Hawaiian or Pacific Islander	10	1%	0	0%	2	2%	0	0%
Latina/o/x or Hispanic	81	11%	3	33%	66	79%	10	100%
Other	15	2%	1	11%	0	0%	0	0%
Not sure	7	1%	0	0%	0	0%	0	0%
Total responses	735	-	9	-	84	-	10	-
Highest level of education
Middle school	10	1%	0	0%	2	3%	0	0%
Some high school	45	7%	0	0%	12	16%	0	0%
High school degree	142	21%	0	0%	21	28%	0	0%
Some college	259	38%	1	11%	11	14%	1	10%
College degree	62	9%	4	44%	16	21%	3	30%
Some graduate work	159	23%	0	0%	5	7%	0	0%
Graduate degree	2	0%	4	44%	7	9%	2	20%
Other	8	1%	0	0%	2	3%	1	10%
Total responding	687	-	9	-	76	-	10	-
State of residence
CA	229	34%	0	0%	51	77%	1	10%
OR	225	33%	7	78%	7	11%	8	80%
WA	116	17%	2	22%	1	2%	1	10%
Other	102	15%	0	0%	7	11%	0	0%
Total responding	672	-	9	-	66	-	10	-

*Demographic questions were optional. Percentages are reported in relation to number of respondents providing a response for that item, which may be fewer than the number of respondents included in the dataset as a whole.

**Race and ethnicity is ‘check all that apply’ so total responses will exceed sample size

Data analysis

Quantitative and qualitative data were analyzed first as separate sets, then considered together, in a complementary fashion, with regards to the study questions. Quantitative data were primarily analyzed using frequencies, which were then assessed relative to predetermined targets, or benchmarks for each item. This target-based approach, which is regularly used in evaluation and performance management, is utility-oriented, designed to reveal discrepancies between the intended and actual performance of a particular product or program (Patton, 2008; Theodos et al., 2021). Though there is no standard rule for establishing performance targets, achieving a ‘stretch’ target is more likely to indicate success or effectiveness (Patton, 2008; Theodos et al., 2021). With this in mind, for Likert-type items, we set a target of 80%, meaning that four-fifths of responses ‘somewhat agree’ or ‘strongly agree’ with each statement affirming the video’s understandability or enjoyability, for example. Another advantage of this approach is that it allowed our team to assess responses from specific demographic groups (for example, Latina/o/x participants) relative to the performance target, rather than assessing these responses relative to each other. The focus, then, becomes evaluating the product and its effectiveness for different kinds of learners, rather than evaluating the learners themselves. This approach is especially useful for studies such as this with a strong interest in engaging a range of public audiences. When comparing paired pre- and post-viewing responses, Wilcoxon signed-rank tests were used to assess the direction and magnitude of change; this statistical function is preferable to paired t-tests for analyzing pre–post change in non-parametric ordinal data.

As our analysis progressed, we performed statistical analyses on demographic subsamples of each dataset (English video viewers and Spanish video viewers). The demographic populations we examined are not mutually exclusive and include individuals with a high school education or less; individuals who had not heard of ShakeAlert prior to participating; individuals from Oregon and Washington (where, at the time of the survey, ShakeAlert EEW had only recently became available); Latina/o/x individuals; and individuals who identify with one or more non-White races. We found that, in general, responses from these key demographic populations track closely with the responses of the larger sample. For this reason, the figures we present below focus on data from the overall sample population. When the responses from a key demographic group do diverge from the larger sample or suggest an alternate perspective, we discuss those instances in the text.

We analyzed qualitative data (from open-ended questions in the survey and the interviews) using multi-reviewer inductive and deductive coding approaches (e.g. Braun & Clarke, 2006; Patton, 2015). While qualitative analysis is an inherently subjective process, steps were taken to support reliability and validity. First, two researchers coded each dataset, attaching descriptive codes to qualitative responses; these reviewers then met to share findings, reconcile differing interpretations, identify emergent ideas, and develop a set of thematic codes. A single researcher then reviewed each qualitative dataset again, applying the thematic codes. Frequencies were run to identify the relative prominence of thematic codes within each dataset. Spanish data were kept in the original language while being reviewed and coded by two native Spanish-speaking research team members. Once this process was complete, thematic codes emerging from the Spanish dataset, along with select, representative quotes, were translated for the purposes of presentation here.

Results

Understandability and key messages (study question 1)

Survey respondents were asked to assess the understandability of each video, including language and visuals used, on a five-point ordinal scale (Figure 3). Both the English and the Spanish videos exceeded the target (80% agree) for all three indicators of understandability.

Figure 3. Percentage of respondents indicating ‘somewhat agree’ or ‘strongly agree’ with each statement about understandability.

Participants were also asked to assess how well on a 5-point ordinal scale ranging from ‘very poorly’ to ‘very well’ the video conveyed key messages about ShakeAlert. Both videos met the target (i.e. 80% ‘well’ or ‘very well’) for four out of five key messages (Figure 4). For both videos, the key message that respondents felt was least well communicated was message 2, ‘ShakeAlert does not predict earthquakes.’ Seventy-eight percent of English video viewers and 74% of Spanish video viewers agreed that this message was well communicated – a clear majority, but the overall percentage did not meet the established target. Some qualitative responses cited a lack of clarity around this topic; for example, one survey respondent who had previously heard of ShakeAlert wrote, ‘I don't understand how it does NOT predict earthquakes, even though it says if the alert goes off an earthquake is imminent.’

Figure 4. Percentage of survey respondents indicating that each key message was communicated ‘well’ or ‘very well’.

In analyzing survey responses from specific demographic groups, we found that the patterns illustrated in Figures 3–4 persisted across groups, with over 80% of respondents indicating that the video was easy to understand and that messages 1, 3, 4, and 5 were the most well communicated. For all groups examined, the message that participants felt was least well-communicated was message 2. The only exception was responses from the Pacific Northwest (Oregon and Washington). Within this group, both English and Spanish video viewers rated all five messages as well communicated, including message 2.

Open-ended responses from the survey provide additional insight into which messages participants found most memorable and/or significant. Participants were asked, ‘If you were trying to teach a friend about ShakeAlert, what information from the video would you share?’ A summary of the responses to this question are shown in Table 4, and it is worth noting that in the survey, this open-ended response item preceded the ordinal response item (Figure 4) about key messages, so participants had not yet been cued as to what the intended key messages were meant to be.

Table 4. Memorable information about ShakeAlert, in response to survey item, ‘If you were trying to teach a friend about ShakeAlert, what information from the video would you share?’.

Theme Description	Count*		Illustrative responses		Related key messages
	English (out of 579 coded)	Spanish (out of 63 coded)	English	Spanish
Warning Indicates knowledge that ShakeAlert can provide warnings or alerts. May also (but not necessarily) make a distinction between warning and prediction. Warning	124	13	‘..that it's not a prediction of a quake but a few seconds warning that a quake has occurred in the area.’	‘Nos alerta cuando hay un sismo’ [It alerts us when there’s an earthquake]	1, 4
Late alert zone Indicates an understanding of the late alert zone	57	N/A	‘Areas closest to the epicenter might feel the shaking before the alert is relayed.’	N/A	3
Timing Indicates an understanding of how alerts may arrive before, during, or after shaking and/or indicates that advance warning is just seconds to minutes.	51	<3	‘That shake alert notifications might come a few seconds before or after shaking has begun.’	‘Que suena una alarma segundos antes de que un terremoto llegue.’ [That an alarm sounds seconds before an earthquake arrives.]	3
Protective actions Refers to taking protective actions, but does not specify drop, cover, and/or or hold on.	44	16	‘I'm going to share some of the things you can do to protect yourself when an earthquake strikes.’	‘Compartiria lo que tiene que hacer cuando haya un terremoto.’ [I’d share what you have to do when there’s an earthquake.]	4
DCHO Refers to Drop, Cover, and Hold On (including drop and cover, duck and cover, and other similar iterations of this theme).	168	<3	‘I would tell them to drop, cover and hold on for the earthquake.’	‘La parte de lo que hay que hacer cubrirse.’ [The part about having to take cover]	5
Respond to alert Refers to taking protective action as soon as alert is received, even if shaking is not (yet) felt.	73	<3	‘Take action right away, don't wait for shaking to start.’	‘ … que se tiene que agachar, cubrir y agarrarse, sin esperar a que el temblor suceda.’ [… that you have to drop, cover, and hold on without waiting for the shaking/earthquake to happen]	4, 5

*Note. Some responses received multiple codes; other responses were unclear or coded with themes that occurred infrequently and are not reported here. For this reason, frequencies of thematic codes will not sum to total responses coded.

In response to the English video, the most recurrent themes were ‘warning,’ ‘Drop Cover and Hold On (DCHO),’ and ‘respond to alert.’ These responses suggest an understanding of key messages 1, 4, and 5, in particular. In the smaller set of Spanish video responses, ‘warning’ was also a prominent theme, as was the generic theme of ‘protective actions’; very few Spanish video viewers cited specific actions related to DCHO (agacharse, cubrirse y sujetarse) in this response item. Alternate conceptions about ShakeAlert were also present in both datasets, though they occurred less frequently than responses indicating an accurate understanding of ShakeAlert EEW. Across survey and interview responses, some participants categorized ShakeAlert as an ‘app,’ which is technically inaccurate. (ShakeAlert is actually the system that detects and analyzes earthquakes; third-party apps use ShakeAlert data to generate alerts). Additionally, some participants suggested that ShakeAlert could warn of ‘pending’ or ‘imminent’ earthquakes; for example, one English video viewer wrote that ShakeAlert ‘Can sense an earthquake in advance.’ These responses are technically inaccurate and, though they may reflect a clear understanding of Key Message 1, they simultaneously reflect a lack of clarity around Key Message 2.

Despite participants’ overall assessment that the video was clear and easy to understand, survey respondents and interviewees identified two other areas where the video was confusing or lacking in information. The first theme related to protective actions in specific scenarios, particularly when dropping, covering, and holding on is not practical or possible. The second theme related to alert delivery, including how, and under what conditions, individuals can receive alerts on their mobile devices. Table 5 provides representative responses related to these two themes, each of which was coded more than 50 times in response to various prompts from the English video survey and interview; these themes were less prominent, but still present, in the smaller sample of Spanish video responses.

Table 5. Themes around which the video was unclear and/or lacking in information, drawn from both survey and interview responses (all items).

Theme Description	Illustrative responses
	English	Spanish
Specific protective actions Indicates concerns, questions, or a desire for more information related to protective actions in scenarios where DCHO is impractical or impossible to undertake	‘I guess it's confusing because every situation is different. What if I'm outside or inside an elevator. What do I do?’	‘[No entendí] la razón porque agacharme no la entendí y debajo de que puedo agacharme.’ [I didn’t understand the reason for ducking or what I should duck under].
	‘How does “drop, cover, hold on” apply if you're out in the open, as on the beach shown in the video?’	‘Si hubiera un fuerte temblor [y] estoy en el elevator … k hacer?’ [If there’s a strong earthquake and I’m in an elevator … what do I do?]
	‘It didn't clarify what anyone with disabilities should do. What if I can't drop to [the] floor?’	‘Las recomendaciones. Deberían ampliarlas según el lugar donde nos podamos encontrar en el momento que ocurra un temblor. Ejemplo: si vamos en el auto, si estamos dentro de un market, etc.’ [The recommendations. They should be expanded according to the places where we could find ourselves the moment an earthquake happens. Example: If we are in a car, if we are in a store, etc.]
	‘I am rarely in a place with a table. Is that important? The video makes it seem like it is. Some more info about what “cover” means exactly would help.’	‘ … que debes hacer si estas dormido que se debe hacer si te quedas en la cama o tratar de protegerte en el cuarto si no alcansas a llegar a donde puedas ponerte debajo de una mesa’ [… what should you do if you’re sleeping. What should you do if you stay in bed or try to protect yourself in the room if you can’t make it to where you can get under a table.]
Alert delivery Indicates concerns, questions, and/or a desire for more information related to the specifics of alert delivery.	‘Where is it [ShakeAlert] available? How do people sign up?’	‘no sé si me apunto al programa shakealert?’ [I don’t know if I sign up for the ShakeAlert program?]
	‘It's also unclear how I will receive this alert. Is it automatic (like an amber alert) or do I need to download an app?’	‘¿Viene siendo una aplicación que tengo que agregar a mi celular?’ [Is it an app that I have to add to my cell phone?]
	‘How does it know who to send the alerts to? If you are traveling, does it know?’	‘¿Es automático? ¿Debo inscribirme?’ [Is it automatic? Do I have to sign up?]
	‘If I turn off the Amber alert on my iPhone does that turn off shake alert?’	‘¿ … funciona sin recepción?’ [Does it work without (cell) reception?]

Enjoyability (study question 2)

Survey respondents were asked to assess the degree to which the video was enjoyable, using a five-point ordinal scale ranging from ‘strongly disagree’ to ‘strongly agree.’ The survey revealed that 81% of English video respondents and 85% of Spanish video respondents agreed or strongly agreed with the statement ‘the video was enjoyable to watch,’ meeting the target (80%) for both videos.

Table 6. Prominent themes in response to survey item, ‘What did you like most about the video?’.

Theme Description	Count*		Illustrative responses
	English (out of 652 coded)	Spanish (out of 66 coded)	English	Spanish
Qualities of video
Understandable Describes the video as easy to understand, clear, makes sense, etc.	155	7	‘Clear and understandable’	‘Lo explican de una forma muy fácil de entender’ [They explain it in a way that is very easy to understand]
Concise Describes the video as short, simple, concise, direct, straightforward, etc.	100	N/A	‘It was concise and not overly long’
Informative Describes the video as informative, informational, etc.	129	18	‘It was very informative’	‘Todo fue muy informativo’ [Everything was very informative]
Images Refers to quality of imagery, graphics, maps, animation, etc.	72	N/A	‘I liked the visuals used to explain the concept’
Content of video
Protective actions Refers to video content about ‘what to do,’ how to protect oneself, etc.	100	16	‘What l like most is the information on what to [do] when a earthquake happen[s] drop roll cover’	‘Muestra como podemos protegernos en casa de … temblores’ [It shows how we can protect ourselves at home from earthquakes]
How ShakeAlert works Refers to explanatory content describing how ShakeAlert works	92	5	‘The explanation of how the system responds to earthquakes was helpful.’	‘Aprendi com funsiona la alerta i que se debe hacer’ [I learned how the alert works and what to do]
Limitations of ShakeAlert Refers to video content explaining limitations of SA, including the late alert zone and/or the fact that it is not prediction	37	N/A	‘Explanation for the limitations of the alert were especially useful’
AK Video Refers to classroom footage from the Anchorage, AK earthquake	30	4	‘Video of classroom shows that a few seconds warning can make a difference’	‘ … la video del terremoto qué pasó en Alaska’ [… the video of the earthquake in Alaska]

*Note. Some responses received multiple codes; other responses were unclear or coded with themes that occurred infrequently and are not reported here. For this reason, frequencies of thematic codes will not sum to total responses coded.

Qualitative responses from the survey reveal which aspects of the video may have contributed to overall enjoyability. When asked, ‘What did you like most about the video?’ (Table 6), participants most often cited qualities and content that supported the video’s ability to inform and explain (as opposed to qualities and content that served to entertain, amuse, or serve other non-educational purposes).

Table 7. Explanation of behavioral intentions, in response to survey item, ‘If you would like to provide more detail, [about the previous closed-ended response items related to intended behavior] please do so here.’.

Theme Description	Count*		Illustrative responses
	English (out of 131 coded)	Spanish (out of 47 coded)	English	Spanish
DCHO Refers to Drop, Cover, and Hold On (including drop and cover, duck and cover, etc.)	28	6	‘The video made it clear to drop, cover and hold on.’	‘Al agacharme y sujetarse estoy más segura de un terremoto’ [By dropping and holding on I am safer in an earthquake].
Specific protective actions Indicates concerns, questions, or a desire for more information related to protective actions in scenarios where DCHO is impractical or impossible to undertake	22	N/A	‘I’m a diesel mechanic so at times it will be very difficult to climb off a machine to seek shelter’
Evacuation Indicates intent to evacuate or consider evacuating	10	<3	‘I don’t necessarily agree with drop, cover, and hold always being the right thing. If I’m in a rickety building, I might risk keeping moving to get outside of the building and far enough away to avoid debris.’	‘Si el terremoto me agarra en un edificio y es fuerte tengo oportunidad n corre[r] o cubrirme. K podria hscer,?’ [If an earthquake is strong and catches me in a building, I have the opportunity to run or cover. What could I do?]
Situational awareness Emphasizes importance of assessing situational factors first to determine appropriate response	17	N/A	Drop/Cover/Hold is a little situational. If I feel that the position I am in when I get the alert is inherently dangerous and I can quickly (seconds) get to a safer place before drop/cover/hold then I would do so.
Impulses Refers to behaviors or reactions done reflexively or impulsively	7	N/A	… my first instinct would be to run, I will try to remember to drop, cover and hold on.
Helping others Refers to helping others first or simultaneously with self protection	4	<3	I didn't strongly agree because … if my kids were in the house I'd instead move about the house to find them and group together first. Then we'd cover and hold on together …	‘Como padre de familia, mi interés es en tener a mis familiares seguros y a salvo primeramente.’ [As a father, my primary interest is to have my family members safe and sound.]

*Note. Some responses received multiple codes; other responses were unclear or coded with themes that occurred infrequently and are not reported here. For this reason, frequencies of thematic codes will not sum to total responses coded.

Qualitative responses to the subsequent survey prompt (‘Was there anything that you did not like about the video or found confusing or unclear?’) revealed specific aspects of the video that some participants disliked, including certain sound effects (too loud), graphics (too busy), the length (too long), and the amount of information (too much). Though these were the most common themes related to dislike, each of these themes occurred less frequently than the positive themes described in Table 6.

Intended behavior (study Question 3)

Survey participants were asked about their intended behavior related to ShakeAlert both before and after watching the video. For both the English and Spanish videos, the percentage of respondents agreeing with the statements ‘I intend to drop, cover, and hold on if I feel shaking’ and ‘I intend to drop, cover, and hold on if I get a ShakeAlert message’ increased from pre-viewing to post-viewing (Figure 5).

Figure 5. Percentage of survey respondents indicating ‘somewhat agree’ or ‘strongly agree’ with each statement about intended behavior before and after viewing the video.

The significance of these pre–post changes was evaluated using a Wilcoxon signed rank test. For all Wilcoxon signed rank tests, the effect size, r, was calculated by dividing the z value by the square root of N (Fritz et al., 2011). Designations of large, medium, and small effect sizes for r follow Cohen’s guidelines for social sciences (Fritz et al., 2011). Results indicate that intent to DCHO in response to shaking increased significantly from pre-viewing to post-viewing for both the English video (z = −10.91, p < .001) with a medium effect size of r = .41 and the Spanish video (z = −4.26, p < .001) with a large effect size, r = .50. Intent to DCHO in response to a ShakeAlert message (i.e. a ShakeAlert-powered alert) increased even more substantially from pre-viewing to post-viewing for the English video (z = −13.19, p < .001) with a large effect size, r = .50 and the Spanish video (z = −4.32, p < .001) with a large effect size, r = .51. In each of these cases, the negative z value reflects the fact that the initial value (degree to which participants agreed with the statement before viewing video) was lower than the ending value (degree to which participants agreed with the same statement after viewing the video).

The intent to DCHO increased for both English and Spanish video viewers as a whole and across all key demographic groups examined within those samples. At the same time, the percentages of respondents agreeing or disagreeing with the statements varied in relation to the target (Figure 5); for example, slightly fewer than 80% of Spanish video viewers indicated, post-viewing, that they would DCHO in response to an alert. Looking at key demographic groups, among English video viewers who identified as Latina/o/x (n = 81), the percentage of respondents indicating intent to DCHO in response to an alert increased from 59% to 76%; while this represents a significant increase, both values remain below the target of 80%. Similar results were seen for English video viewers with a high school degree or less.

Following the closed-ended items described above, an open-ended prompt invited survey participants to ‘provide more detail’ about their responses (Table 7). Here, some participants affirmed their intent to DCHO, citing prior earthquake experience, prior education about DCHO, and/or knowledge gained or reinforced by the video. For example, one English video viewer who had previously heard of ShakeAlert explained, ‘Before the video, I would’ve waited to feel shaking to confirm an actual earthquake prior to dropping. After the video, I know I need to drop immediately upon receiving the alert.’ Additionally, some participants explained why they might not perform DCHO and described additional/alternate behavioral responses. The theme of ‘specific protective actions’ emerged again, as participants reiterated concerns or confusion about their ability to perform DCHO in different contexts, such as while driving. Additionally, participants indicated that situational awareness, impulses, and the needs of others would affect their decision to perform DCHO, evacuate, or take other actions in response to shaking and/or an alert (also see Adams et al., 2022).

For the Spanish video, few survey respondents answered this particular open-ended item; however, interview data provide supplemental information about viewers’ behavioral intentions related to ShakeAlert. Interview participants were asked, ‘What do you think you would do if you received a ShakeAlert message?’ All 10 of the Spanish video interview participants responded that they would take protective actions, including dropping and/or covering. One participant touched on this theme, as well as the themes of ‘helping others’ and ‘impulses’ when she explained, ‘Lo trataría de compartir con amistades y familiares. Me lo tomaría en serio y hacer lo que aprendí en el video … Tratar de no entrar en pánico.’ [‘I would try to share it with my friends and family. I would take it seriously and do what I learned in the video … Try to not panic.’]

Discussion and recommendations

The present study contributes to the rapidly emerging education and social science literature concerning public understanding of earthquake early warning. Though this study is focused on the U.S. West Coast where ShakeAlert EEW is operational, EEW is truly an international concern. Other nations, including Mexico, Japan, and others, have had EEW systems in place for years (e.g. McBride et al., 2022) and similar technologies are being considered and developed worldwide, from Canada (Schlesinger et al., 2021) to New Zealand (Becker et al., 2020). In all of these settings, it is imperative that technical implementation of EEW coincide with culturally and linguistically appropriate public education and outreach campaigns. Further, integrating rigorous, demographically representative, and longitudinal evaluation research projects can help ensure that those who could benefit from early warning technology are aware of the system, are ready to receive and understand alerts, and are prepared to protect themselves in the event of an alert.

This study, which is the first to comprehensively evaluate the effectiveness of a dual-language, public educational video related to ShakeAlert in the U.S., addressed the following primary questions: 1) How effectively does the video convey key messages related to ShakeAlert? What features of the video enable or hinder viewers’ understanding of key messages? 2)To what extent do viewers find the video enjoyable? What features of the video relate to enjoyment?, and 3) What impact does the video have on viewers’ intended behavior related to earthquakes and ShakeAlert EEW?

Related to the first set of questions, we found that the majority of survey and interview participants indicated that the video was easy to understand and effective in conveying key messages about ShakeAlert EEW. Though ordinal items were subjective and not designed to assess viewers’ actual knowledge gains, responses suggest that technical language and ideas can be judged as understandable by learners when presented aurally and with the support of explanatory animations (e.g. Cook, 2006; Mayer & Moreno, 2002; Putortì et al., 2020). Additionally, more in-depth qualitative responses suggest that many participants clearly grasped the video’s key messages in both English and Spanish (Table 4). These results support the notion that animated video can be an effective tool for science education, not only in a formal education context (where it has been more often studied), but also in the public sphere where it may be used to support viewers’ understanding of science and technology that impacts their lives.

As the survey showed, the message viewers found least understandable was key message 2 (‘ShakeAlert does not predict earthquakes’). There are various plausible reasons for this result, some of which were also illuminated in the qualitative interviews. One possibility is that the video was unclear in the delivery of this message. Another related possibility is that the members of the public use the word ‘earthquake’ differently than seismologists. Scientifically speaking, the term ‘earthquake’ refers to a ‘sudden slip on a fault, and the resulting ground shaking’ (USGS, n.d., emphasis added); meanwhile, in common speech, people may use the term ‘earthquake’ to describe the experience of shaking without reference to the seismic slip event that caused it. For example, one survey respondent explained that ShakeAlert can ‘warn of an imminent earthquake.’ To a seismologist, this characterization is inaccurate because it suggests that ShakeAlert can anticipate a sudden slip on a fault. (It cannot.) However, for members of the public, who may be accustomed to using the term ‘earthquake’ in a non-technical way to refer simply to ground shaking, the quoted response could reflect an accurate understanding of ShakeAlert after all.

This particular communication challenge is exacerbated in Spanish, where three different words (terremoto, temblor, and sismo) are commonly used when referring to earthquakes. In the Spanish ShakeAlert video, the word ‘terremoto’ was chosen as a translation for ‘earthquake,’ while the word ‘temblor’ appears as a translation for ‘shaking.’ However, as Cube (2020) notes in an article for Univisión, most Spanish speakers use the terms interchangeably; when a distinction is made, the distinction is based on magnitude (with ‘temblor’ referring to a lower magnitude event and ‘terremoto’ referring to a higher magnitude event), not the separate concepts of earthquake slip and earthquake shaking. When describing their prior experience with seismic events, multiple Spanish-speaking interview participants in this study explained that they had never experienced a ‘terremoto’ but had experienced a smaller ‘temblor.’ This discrepancy between language use in the video and language use in practice confounds analysis; for example, when examining a Spanish video viewer’s comment that ShakeAlert ‘avisa o manda señales a nuestros aparatos antes de que ocurra el temblor’ – i.e. ‘it warns or sends signals to our devices before the [temblor] happens’ – it is impossible to know whether the respondent has a technically accurate conception about ShakeAlert relative to earthquake prediction, and therefore, whether the video successfully conveyed key message 2 for that viewer.

One possible strategy to improve the communication of this key message in the future is to focus on clarifying terms in both English and Spanish; this may require differentiating prediction from related processes (like detection, early warning, and forecasting) and differentiating fault slip from the ground shaking which follows. That said, ultimately, the goal of the video is not to develop public understanding of scientific terminology, but to help viewers understand what ShakeAlert EEW can and cannot do, and how to safely respond during an earthquake. With that in mind, another possible communication strategy is to bypass the concept of prediction entirely and instead focus on timing. If educators and communicators emphasize that ShakeAlert EEW can provide, at best, seconds to tens of seconds of advance warning about earthquake shaking (McGuire et al., 2021), then the technical issue of prediction becomes less relevant and the emphasis can shift to what people can do with those additional seconds of warning.

Related to this point, our research found that viewers want more information about: 1) how to protect themselves in an earthquake when DCHO is not practical or possible, and 2) how to receive alerts on their mobile devices. Regarding the first topic, other materials developed by IRIS, EarthScope Consortium, and ShakeAlert CEO&TE offer ample information about DCHO for individuals with limited mobility, in outdoor environments, and in other situations. Notable examples include the educational animation Earthquake! Steps to Take When It Strikes (IRIS, n.d.) and the ShakeAlert Messaging Toolkit (ShakeAlert, 2020), both of which are available in multiple languages, including English and Spanish. In light of these study results and the pressing need to provide culturally appropriate, actionable risk information to the public (West et al., 2021; Wood et al., 2012), future educational efforts should focus on making that information more widely available and integrating references and/or content into other ShakeAlert System communication products. Put simply, it might be useful in future videos to focus less on what scientists think people need to know, and more on what they should do under a range of conditions. Regarding the second question that was raised in the surveys and interviews concerning accessing alerts, public communication has been challenging because there are multiple ways for an individual to receive alerts on a mobile device (e.g. WEA, built-in apps, and downloadable apps). Further, new alert modalities are currently in development. To complicate matters, all these delivery mechanisms vary in their availability across device types and state lines. Explaining how to configure every type of device in every state where ShakeAlert EEW is available would require more time and specificity than a four-minute video could accommodate. However, there is an opportunity for future videos or other communication products to point viewers to websites or other materials that provide specific information for audiences who just want to know: How can I get an alert on my device? In addition, there may be an opportunity for more focused communication by audience type, such as videos that are specific to different institutional contexts (e.g. schools, healthcare settings, etc.), or population groups (e.g. teens, older adults, etc.).

Related to the second set of research questions, data suggest that the video was enjoyable for the majority of participants. Enjoyability matters in a free-choice learning environment like the internet, where learners can (and often do) leave an un-engaging virtual experience with the click of a button (Twenge, 2017). Indeed, in this study we observed a high incompletion rate among youth who began the survey, with two thirds of youth participants dropping out during the four-minute video; this result raises questions about the degree to which the video can engage this age group. That said, both youth and adults seek out and watch online video for any number of reasons, whether entertainment-oriented or information-oriented (e.g. Rosenthal, 2018). Qualitative data from this study indicate that participants who completed the survey most liked information-oriented features and content; that is, viewers appreciated the video’s ability to inform, explain, and educate above its ability to entertain or amuse. As one adult respondent noted, ‘I don’t think it’s the responsibility of producers of these types of video products to make them ‘enjoyable to watch’. Rather it is your responsibility to deliver the information responsibly, accurately and efficiently.’ With that in mind, however, we recognize that the sample included in this study was not representative of the population. As such, if the development team wishes to reach a wider portion of the population, they may need to consider creating different videos that are more tailored to diverse audiences.

Related to the third set of research questions, data suggest that the video had a positive impact on viewers’ intentions to drop, cover, and hold on if they feel shaking or receive a ShakeAlert-powered alert. Sociological theories, such as the theory of planned behavior, indicate that knowledge helps shape an individual’s behavioral intentions, which and in turn contribute to actual behavior (McBride et al., 2022). At the same time, factors like social context and perceived or actual control also play a significant role (Adams et al., 2022; Ajzen, 1991; Lindell & Perry, 2012; McBride et al., 2022). Qualitative data from this study, likewise, indicate that behavior (both intended and actual) is contingent on a variety of factors, which go beyond knowledge. Indeed, research using interviews and video footage to examine behavioral response during actual earthquakes indicates that, in response to an earthquake alert or shaking, many adults fail to perform DCHO as recommended, instead freezing, standing in a doorway, or running outside (Adams et al., 2022; Baldwin, 2022; Lambie et al., 2017). It is well understood in the science and risk communication fields that information, while necessary, does not alone lead to desired learning outcomes, including behavioral change (Wood et al., 2012). Because knowledge does not always translate to desired action, it is vital to couple educational products with drills, emergency plans, and other activities (Adams et al., 2022).

Regarding multilingual communication efforts, participant responses suggest that the Spanish video was understandable, enjoyable, and capable of influencing intended behavior. At the same time, in conducting this study, we experienced significant challenges recruiting Spanish-speaking participants. Spanish-language recruitment materials were promoted over the course of several weeks, through a variety of networks and social media channels associated with ShakeAlert CEO&TE; these efforts resulted in only 10 survey responses, which were only augmented through the panel recruitment strategies (Table 2). It was somewhat easier to recruit the 20 Spanish video interview participants through network-based dissemination and through offering paid compensation for participation. The difficulties that we experienced recruiting Spanish-speaking individuals through ShakeAlert CEO&TE’s primary channels of communication (i.e. social media, networks) suggest that more work needs to be done for the ShakeAlert System community – including the many organizations, like EarthScope Consortium, OMSI, USGS, and others affiliated with that community – to better understand and connect with Spanish-speaking audiences (Medina Luna et al., 2019). It is important to acknowledge such challenges and to develop culturally relevant strategies for future recruitment to ensure that the voices of historically underserved populations are included in future evaluation research efforts.

Ultimately, the majority of Spanish-speaking survey respondents were recruited through the paid panel with Alchemer. Here, too, the research process itself was revealing with regard to multilingual communication. The online survey for paid panel respondents was set up so that participants first chose whether to view the survey in English or Spanish, then, once inside the survey, they had to manually select whether to watch the video in English or Spanish. This resulted in a difficult-to-interpret dataset, with dozens of Spanish-speakers watching the English video and then responding in Spanish, or vice versa. As described in the Methods section, we divided the data based on the version of the video the participant watched, rather than the language of the survey instrument or the language participants used in open-ended responses. On the one hand, this outcome served as a methodological ‘lesson learned’ and illustrates some of the technical challenges associated with developing multilingual study designs. On the other hand, this outcome also reflects the fact that many Spanish speakers in the West Coast states of California, Oregon, and Washington are multilingual (Figure 2). Multilingual speakers frequently ‘code-switch,’ toggling between different languages for different purposes (e.g. Lipski, 2014). According to a Pew research report (2004), two situations where bilingual Spanish-English speakers are most likely to prefer English include news broadcasts and instructional manuals; for some Spanish-English speakers, this preference may apply to news and instructions related to ShakeAlert EEW. Ultimately, while it remains important to create and disseminate educational products in Spanish it is also important for science communicators to remember that language use is not binary; many individuals use multiple languages with various degrees of fluency, in various contexts, and for various purposes.

Because of the nature of this study, which relied on non-representative and non-equivalent samples for the English and Spanish video, we refrain from making direct comparisons between the two products or assess their efficacy relative to each other. That said, relative to the performance targets set, the English video performed more favorably across the indicators assessed. It is impossible to know whether this finding is a result of recruitment and sampling differences or differences in the quality and efficacy of the two videos for the two audiences. It is important to acknowledge these differences, because they may be rooted in the fact that the video was initially created in English and then translated to Spanish. There are other models that could be pursued in the future, such as language co-development, or even beginning by creating the Spanish-language version first (West et al., 2021). In addition, while a diversity of voices were included in the data collected, the demographics, as a whole, are not statistically representative of the West Coast population. In future evaluation efforts, it is important to pursue recruitment through multiple, science and non-science based networks while striving toward a representative sample of youth and adult respondents.

Conclusion

In summary, this study revealed that both the English and Spanish versions of the ShakeAlert video were enjoyable, effective in communicating key messages about ShakeAlert, and helped viewers appreciate the importance of performing protective actions in response to an earthquake alert. These findings support the use of animated video as a tool for engaging public audiences, particularly for improving knowledge and influencing future intended behavior. Future research could examine the relationship between animated video and other model-based outcomes for risk communication that have relevance for earthquake preparedness and response, such as awareness, trust in the system, and trust in the messenger (Mileti & Peek, 2000). Additionally, future research could examine the effect of video interventions on actual (versus intended) earthquake-related behaviors, such as participating in an earthquake drill, securing one’s space, or downloading an app that can receive ShakeAlert-powered alerts. Further, if a longitudinal evaluation panel study was established, West Coast respondents could be studied in the immediate aftermath of an event to learn about what they did during or immediately following shaking. The collection of such perishable data is rare, but important for advancing science and education (Adams et al., 2023).

A final and important recommendation emerging from this study is the need to engage and include focal audiences earlier in the video development process in order to understand the needs, concerns, and questions that different groups may have relative to the topic, the media product itself, and the mode(s) of dissemination (West et al., 2021). This is particularly important when one or more of the focal audiences – e.g. youth, persons with disabilities, Spanish-speakers – is minoritized, underrepresented, and/or culturally distinct from those responsible for video development. In their discussion of approaches for equitable and inclusive communication campaigns associated with ShakeAlert EEW, Jenkins et al. (2022) emphasize this point, encouraging communicators to ask themselves, ‘Have we spoken directly with the communities we hope to serve and cultivated a two-way flow of information?’ There are multiple strategies for cultivating this two-way dialogue from ideation through dissemination. One is to engage video developers who have identity and/or lived experience as members of the focal audience(s). Another strategy is to conduct focus groups or interviews – similar to the ones conducted as part of this evaluation study – during the front-end and formative stages of video development, rather than soliciting feedback only after the product is finalized. Other approaches, including collaborative risk communication models (West et al., 2021), co-development, and community-based participatory research, which have been used for video-based science communication (see, for example, Tremblay & Jayme, 2015 and Villar, 2021), take community engagement to another level by positioning key audience members as co-creators of the educational product from start to finish. These approaches may be perceived as more time consuming and resource-intensive than a top-down approach, but they also have the potential to produce mutual learning and more meaningfully engage public audiences. Moreover, such approaches could also address issues with reaching diverse and historically underrepresented populations at the time of sample recruitment.

This study is one of the first multi-method, multi-state evaluations of a ShakeAlert EEW educational tool. Our intent is for these study results to inform future ShakeAlert System education products and efforts, which are ongoing. Members of our team have presented the findings from this study to a variety of audiences, always emphasizing the importance of implementing rigorous evaluation protocols, building diverse teams at the outset of educational efforts, and applying lessons learned. The research presented here has the potential to inform future animated videos and educational materials. For science educators and communicators more broadly – who may be working with topics beyond the scope of ShakeAlert earthquake early warning – we hope these findings both encourage and advise with regard to the value of animated video as a tool for engaging with diverse and multilingual public audiences on topics at the intersection of science, technology, and society.

Ethics Statement

All components of this research involving human subjects were reviewed by the University of Colorado Boulder Office of Research Integrity and approved under protocol #21-0197.

Acknowledgements

We acknowledge the U.S. Geological Survey for supporting staff at OMSI and IRIS (now EarthScope) via Intergovernmental Personnel Act agreements. Staff and students at the Natural Hazards Center at the University of Colorado Boulder supported various aspects of the research process, which we gratefully acknowledge. We thank Michael Hubenthal, Jenda Johnson, and John Taber for their reviews of this manuscript. We also thank Raúl Preciado Mendez, Raquel Stewart, and Kim Deras for their contributions to Spanish-language data collection and translation. ShakeAlert is a registered trademark of the U.S. Geological Survey and is used with permission. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

IRIS (now EarthScope) is supported by the National Science Foundation’s Seismological Facilities for the Advancement of Geoscience (SAGE) Award #1851048. The Natural Hazards Center at the University of Colorado Boulder supported various aspects of the research process, including participant incentives for this study, through NSF Award #1635593.