Examining Factors Impacting the Effectiveness of Anti-Phishing Trainings

ABSTRACT

Approximately 65% of the organizations in the United States have fallen victim to a successful phishing attack. Many organizations offer anti-phishing training to their employees to defend against phishing attacks. The purpose of this study is to examine factors impacting the effectiveness of anti-phishing training and study the relationship between personality traits and phishing susceptibility. Participants filled out pre- and post-training surveys that included questions on identifying phishing and legitimate URLs and questions to determine DISC (Dominant, Influence, Steadiness, and Conscientiousness) personality traits. An analysis of the survey data shows that the participants’ average accuracy in detecting phishing URLs increased 8% (t = 2.144, p-value = 0.0374) and their confidence in their answer choices increased 6% (t = 2.032, p-value = 0.0464) from pre-training to post-training surveys. Before and after training, participants with the Influence personality trait had the lowest susceptibility while both Dominant and Steadiness personalities had the highest susceptibility before and after training respectively.

KEYWORDS:

• Phishing
• personality traits
• phishing susceptibility
• social engineering
• anti-phishing training
• cybersecurity education and training

1. Introduction

In today’s society, information and communication technology such as cellular phones, e-mails, Internet, laptops/workstations, phone applications, voice assistants (Siri, Alexa, etc.), etc. has become essential, particularly for businesses. This has enabled cyber criminals to easily exploit the security’s “weakest link” – the human element.^1–3 Such attacks are often accomplished through social engineering, which is the process of getting a person to complete a service or task that may cause the victim to unknowingly give away private and/or confidential information.⁴ As more and more cyber criminals rely on social engineering techniques, it is imperative that organizations utilize efficient training programs. Social engineering attacks are not new; however, as more technical experience is required for cyber criminals to launch attacks, many are falling back on social engineering.⁵

Phishing is the most common type of social engineering attack.⁶ Phishing involves an attacker – known as a phisher – posing as a trustworthy source in an attempt to have the victim release personal or private information.⁷ Phishers frequently pretend to be companies like eBay, PayPal, and various banking organizations in order to trick and deceive individuals into giving them information. These phishing deceptions can come in the form of e-mails or entire websites and have the ability to target large amounts of people en masse.

During the early days of phishing, phishing attacks rapidly grew in number due to the use of online do-it-yourself phishing kits and other similar factors.^8,9 In 2017, more than 7,700 organizations were targeted by business e-mail compromise scams. The targeted organizations were attacked, on average, around five times throughout the year.¹⁰ It is difficult to calculate just how much damage phishing attacks cost annually due to the number of unreported attacks; however, researchers have estimated that phishing attack costs could be anywhere from 61 USD million to 3 USD billion per year in the United States.¹¹ This does not include the costs of indirect damage to reputation, recovery measures, and other affected areas for companies.

Mitigating phishing attacks has long been a challenge in cybersecurity. There are three key aspects of phishing mitigation – detection of the attack, educating and training employees, and determining user susceptibility.¹² Detecting phishing attacks can be automatic or manual. Automatic methods are generally browser tools designed to block phishing websites. Many of the anti-phishing browser tools only block websites that appear on a blacklist and take no further consideration. A study reported that the highest accuracy produced by one of the tools was over 90%; however, the tool also falsely identified 42% of the legitimate websites as phishing websites.¹³ Because of such results, manual detection methods like the social engineering attack detection model created in 2015 by Mouton et al.¹⁴ were promoted. However, expecting users to detect phishing attacks requires them to be educated and trained on these malicious attacks.

Education and training programs are the most common method of informing and teaching employees about phishing attacks and phishing prevention. However, in a study by Douglas Twitchell,⁵ it appears that the majority of cybersecurity education and training programs do not discuss phishing attacks, let alone social engineering. Of the programs that discuss social engineering, only half of them mention prevention techniques. A university in the United States, henceforth referred to as “The University,” recently adopted a new security training program – KnowBe4 – in an attempt to offer education and training of employees, especially on topics of phishing. KnowBe4 is a web-based security training program that utilizes simulations, interactive security modules/activities, and newsletters, along with several other services. Jocelyn Sloan, MCS investigated phishing mitigation solutions for small and medium sized businesses and found that KnowBe4, as well as other software as a service products, can be affordable solutions.¹⁵ KnowBe4 also grants the ability to make personalized training modules to create perpetual assessments and perpetual security training/education sessions. Personalization comes in the form of being able to choose and select different types of training modules for different training sessions, which highlights one of the benefits of the KnowBe4 training program. In addition to the discussion of social engineering and phishing attacks, KnowBe4 also has the capability to provide short-term and long-term retention options. If the education and training programs are designed to provide participants’ training results immediately after receiving the information, it is short-term retention. However, KnowBe4 has the option to produce new training results for participants weeks or months after their initial training by supplying fake phishing attacks and other options. This shows how long participants retain the information learned during the training.

For this research, we focus on the anti-phishing training of KnowBe4. At the time of this study, The University had chosen to create their training using four modules. The first module is a combination of The University’s Information Security Policy and their Acceptable Use Policy. In this module, participants will be presented a recorded slide show detailing both policies. An example of the information covered in this section includes the different levels of confidential data and examples of each level for better understanding. Next up is the Common Threats module. The common threats module is a series of interactive videos for the participant to watch. This module discusses social engineering and phishing attacks and provides participants with valuable definitions regarding threats and security features. Participants must complete a quiz on the information and pass with at least an 80% to finish the module. The third module is the KnowBe4 Security Awareness Fundamentals Training. This module focuses on enhancing participants’ risk perception, awareness, and response. Participants are given information on phishing and other social engineering attacks, incident response, good security behaviors, and more. After exploring the information, participants must pass a quiz on the section with a score of at least 80% to complete the training. For all quizzes, if a participant does not score at least 80%, they must retake the quiz. The last module is Spot the Phish, where participants are given different scenarios and must spot the difference between the phishing scenarios and the legitimate scenarios. The scenarios may have different phishing or other social engineering features that participants must identify.

The purpose of this study is to evaluate the effectiveness of the anti-phishing training program and to examine user susceptibility to phishing attacks. The following two research questions are addressed in this study: (1) To what extent does the KnowBe4 training program impact participants’ knowledge, behavior, perceived risk, and performance associated with identifying phishing attacks? (2) To what extent do participants’ demographics and personality traits impact the effectiveness of the KnowB4 training program? This study tests the following three hypotheses: (1) The KnowBe4 training program improves participants’ self-reported phishing knowledge, behavior tendencies, and perceived risk; (2) Participants are able to identify phishing attacks at a higher accuracy after completing the KnowBe4 training; and (3) Participant’s demographics and personality traits will not have a significant impact on the effectiveness of the KnowBe4 training program.

The rest of the paper is organized as follows. Section 2 provides information on the demographics obtained and personality assessment utilized in the study. Section 3 focuses on related research associated with the topic. Section 4 discusses the methodology behind our research, including the survey. Section 5 discusses the results of the study. Section 6 discusses the limitations of this study and section 7 concludes the paper.

2. Background

2.1. Demographics and the DISC personality assessment

This research evaluates the effects of user demographics and personality traits on the effectiveness of a phishing security training program. The demographic information collected from each participant included age, gender, race/ethnicity, years of experience, education, department, and work status (full-time, part-time, student). In order to determine personality traits, participants were required to complete a personality test based on the DISC (Dominant, Influence, Steadiness, and Conscientiousness) personality assessment.¹⁶

Creation of the DISC Assessment is primarily credited to William Marston, an American physiological psychologist.¹⁷ According to Jones and Hartley,¹⁷ Carl Jung laid the original groundwork for how a person’s “psychological make-up ‘temperament,’ ‘style,’ or ‘type’” can impact choices they make – based on the work of Hippocrates. In creating the DISC Assessment, Marston was focused on improving human relationships. The assessment Marston created determines which of the four personality traits (i.e., Dominant, Influence, Steadiness, and Conscientiousness) a user most exemplifies.

People who display the Dominant personality trait as their primary personality are known for being outgoing and task-oriented. Additionally, Dominant individuals are also categorized as leaders, fearless, bold, and daring. People who display the Influence personality trait as their primary personality are known for being outgoing and people-oriented. This type of person is also found to be sociable, entertaining, and talkative. People who display the Steadiness personality trait as their primary personality are known for being reserved and people-oriented. Individuals with this primary trait also tend to be generous, loyal, patient, and calm. Finally, people who display the Conscientiousness personality trait as their primary personality are known for being reserved and task-oriented. Conscientious individuals are typically found to be calm, private, shy, and agreeable.

The DISC assessment was chosen because it “measures surface [personality] traits and is intended to explain how they lead to behavioral differences among individuals.”¹⁷ This information is important as we will be able to see how individuals with each personality trait behave after completing the KnowBe4 training. There have been several papers in which phishing and the Big Five personality traits – also known as OCEAN (Openness, Conscientiousness, Extroversion, Agreeableness, and Neuroticism) were investigated; however, this is the first work utilizing the DISC personality traits in this respect; therefore, this research could provide insights on how the four personality traits identified by this assessment can impact the effectiveness of organizational cybersecurity training programs.

2.2. Technology threat avoidance theory

In 2009, Liang and Xue proposed the Technology Threat Avoidance Theory (TTAT).¹⁸ The theory states that a person’s perceived threat susceptibility and perceived threat severity impact their overall perception of the threat. This means that a person’s awareness of their chances to fall victim to a threat and knowledge of the amount of damage the threat could cause to their devices/systems, impacts their ability to determine the true damage that can be caused by a threat. Likewise, a person’s perceived threat, along with safeguard effectiveness, cost, and ease of implementation impact their avoidance motivation. This means that a person’s ability to determine the magnitude of the threat, the effectiveness of available anti-threat tools, the cost (e.g., time, effort, and money) of those tools, and their belief in how well they can handle the threat play a role in how motivated they are to avoid the threat. Finally, a person’s avoidance motivation impacts their avoidance behavior. Meaning their motivation to avoid the threat directly impacts the behaviors they exude to avoid the threat. In summation, the TTAT states that a person’s perceived susceptibility and threat severity to cyber threats affects their awareness of the threats which, in turn, impacts their motivation and behavior to avoid the threats.^18,19 The TTAT advocates for recurring cybersecurity education and training programs as over time, people forget what is taught to them. Therefore, to keep users motivated to avoid cyber threats, it is imperative that education and training materials improve a person’s perceived susceptibility and threat severity.

Our first research hypothesis “The KnowBe4 training program improves participants’ self-reported phishing knowledge, behavior tendencies, and perceived risk” is based on the TTAT. The perceived threat susceptibility in TTAT relates to our perceived risk, while perceived threat severity relates to our phishing knowledge, and threat avoidance behaviors relates to our behavioral tendencies.

3. Related works

Several research papers discussed the effectiveness of anti-phishing education and training, as well as phishing related behaviors. Researchers have also conducted studies to discover what types of demographics and personality traits are most susceptible to phishing attacks.

3.1. Effectiveness of anti-phishing education and training programs

In order to improve the amount of information participants retain from education and training programs, researchers have been using games and interactive sessions/materials.^20,21 In KnowBe4, Spot the Phish is an example of an anti-phishing interactive training module. Alkhamis and Renaud²² created an interactive training video that helped participants improve their pretest questionnaire scores. Sheng et al.²³ created Anti-Phishing Phil, a game for educating people about phishing attacks. In their study, they discovered that participants in their gaming group experienced improved posttest scores and increased confidence in their answers. Kumaraguru et al.^24,25 observed a similar result when they used Anti-Phishing Phil and the PhishGuru cartoon. In a 2007 study, Kumaraguru et al.²¹ discovered that the PhishGuru cartoon also improved participants posttest scores. Three years later, Kumaraguru et al.²² found that between a traditional training methods group, a PhishGuru group, and an Anti-Phishing Phil group, only participants from the Anti-Phishing Phil group were able to receive similar scores on their immediate posttest and their one-week delayed posttest.²⁵

A study conducted at Columbia University (CU) employed an iteration-based ant-phishing training program.²⁶ The researchers sent out four types of phishing e-mails to students, staff, and faculty at CU. They collected e-mail addresses by using a crawler module on the CU directory. If a user fell for a phishing attack, the time the user clicked on a link, their role, and their department were recorded. Phished users were also displayed a message stating that the phishing e-mail was from an experiment conducted at CU and to be aware of suspicious e-mails in the future. Each user who fell for a phishing e-mail was sent another phishing e-mail in the next wave of phishing attacks. It took four iterations before no users fell victim to one of the phishing attacks. However, initially, 19.6% of the staff members targeted fell victim to the phishing attacks compared to 11.6% of the targeted students. The researchers stated that they believe “users can be trained using decoy technology to be cognizant of potential threats.”²⁶

An effective phishing education and training program should also be able to improve a person’s self-perceived risk and susceptibility. For instance, based on the protection motivation theory (PMT) discussed in McBride et al., “when individuals perceive that they are more susceptible to security threats … and when the threats are more severe, they are more likely to adopt a recommended response to the threat.”²⁷ This is further supported in a paper by Johnston and Warkentin in which they discuss how a person’s perceived threat susceptibility and severity impact their self-efficacy and response efficacy.²⁸

3.2. Phishing related behaviors

In 2013, Arachchilage and Love discussed a game design framework to help users avoid phishing attacks and increase the use of safeguard measures.²⁹ Using the TTAT model and relevant research, they created a questionnaire. To begin the study, participants were asked if they knew what phishing attacks were and if not, they were given a definition. The questionnaire contained topics on perceived susceptibility, perceived severity, perceived threat, and safeguard effectiveness, safeguard cost, self-efficacy, avoidance motivation, and avoidance behavior. Using pearson correlation analysis on the results of the questionnaire, they found that perceived threat is significantly determined by perceived susceptibility and severity and is influenced by the interaction between the two factors. Meanwhile, avoidance motivation is significantly determined by perceived threat, safeguard effectiveness, the interaction of these two factors, self-efficacy, and safeguard cost. Finally, avoidance behavior is significantly influenced by avoidance motivation. Using this information, Arachchilage and Love decided to incorporate elements from each topic into the game design framework. It is important to note that unlike other influences, the safeguard cost negatively impacted avoidance motivation. Meaning as the cost to safeguard increases, user motivation to avoid phishing attacks decreases. The researchers plan to further increase their knowledge by designing and evaluating a mobile game.

In 2021, Canham et al.³⁰ investigated phishing related behaviors at a southeastern university in the United States. The focus for this study was on determining employee cluster types for responses to phishing attacks based on positive and negative actions. Participants in this study were sent 20 training phishing e-mails from June 2018 to December 2019. Canham et al. utilized the KnowBe4 platform as the source of the phishing e-mail campaign and sent the messages to over 6,000 faculty and staff. Each employee’s behavior was categorized as either reported or failure. Employee behavior was marked as reported if they reported the phishing e-mail to the appropriate department. Otherwise, employee behavior was marked as failure if the employee clicked on any links in the e-mail, responded to the e-mail, or entered data after clicking the link in the e-mail. It was not stated whether employees that ignored or deleted the e-mails were placed in a group. They found that around 51% of the employees never fell for a phishing attack, while around 44% occasionally fell for an attack (1–3 times), and around 6% repeatedly fell for an attack (4+ times). Using k-means clustering, the researchers noted the development of four clusters – nicknamed Gaffes, Beacons, Spectators, and Gushers. The Gaffes group contained employees who typically clicked on links in the phishing e-mails at a high rate. The Beacons group contained employees who typically did not fall for the phishing e-mails and reported phishing e-mails. The Spectator group is the largest and contained employees who typically did nothing (neither clicked on links at a high rate nor reported phishing e-mails). Finally, the Gusher group contained employees who did not click on links as often as Gaffes but filled out data requested in phishing e-mails at a high rate. Out of 6,126 employees, 344 belonged to the Gaffes, 1,361 belonged to the Beacons, 4,050 belonged to the Spectators, and 371 belonged to the Gushers. Canham et al. noted that security training is “as expected” not perfect as even in the Beacon and Spectator clusters, around 3–4% of the employees fell victim.³⁰

3.3. Demographics and susceptibility

Sheng et al.³¹ conducted research to study how age and gender affect phishing susceptibility. Results of this study show that women are more likely than men to fall for a phishing attack and people in the age group 18–25 are more likely to fall for phishing than people over the age of 25. However, no strong correlation between education or race and phishing susceptibility were observed from this study. Jagatic et al.⁷ conducted a study that used social networks to target students at Indiana University with phishing attacks. The phishing attacks overall had a 72% success rate. It was found that 77% of the female participants fell for the phishing attacks compared to 68% of the male participants. Similarly, Alam et al.³² conducted a study that showed a relation between phishing susceptibility and the use of social media. Specifically, the researchers discovered that the amount and quality of information posted online by people in the age group of 18–25 years old could possibly increase their susceptibility to phishing and spear phishing attacks. This could explain why a large number of students fell for phishing attacks in the study conducted by Sheng et al.³¹ and the success rate experienced in the study by Jagatic et al.⁷

Mohebzada et al. conducted two phishing experiments on the faculty, staff, students, and alumni of the American University of Sharjah (AUS).³³ The first experiment impersonated the IT department and asked users to reset their university account’s password on a fake website. The e-mail was made to look like a legitimate e-mail from the IT department and the link went to a website that had a similar URL to the school’s official website. Out of 10,917 targets, with 10,217 students and alumni and 700 faculty and staff, 954 people fell victim. Of the victims, 485 (50.8%) of the victims were female, 469 (49.2%) were male, 918 (95.8%) were students and alumni, and 36 (3.8%) were faculty and staff. The second experiment ran immediately after the first. This time, the attackers posed as the university’s research office and sent a link out for the victims to complete a survey. The link actually sent the victims to a form on the researchers’ fake website and asked them to fill out personal information and optionally, their banking information. This time, the IT department sent out a warning just 2 hours after the phishing e-mail; however, 98 people still fell victim after the warning. There was a total of 224 people who fell victim to the second attack. Of the 224, 220 (98.2%) were students and alumni and four (1.8%) were faculty and staff. Of the 220 students who responded to the phishing attack, 86 (39.1%) were female and 134 (60.9%) were male. Based on these observations, the researchers noted that gender and age are not conclusive in determining susceptibility to phishing attacks. Anwar et al.³⁴ conducted a study to examine the effect of employment status on user behavior. They created a survey to measure participants’ self-reported cybersecurity behaviors and found that there were statistically significant negative correlations between employment status and perceived vulnerability, prior experience with cybersecurity practices, and computer skills. They concluded that organizations should place emphasis on training part-time employees as they are more susceptible than full-time employees.

3.4. Personality and susceptibility

In 2016, Cho et al.³⁵ conducted a study on the effect of the Big Five personality traits on phishing susceptibility. The results of this study showed that the Agreeableness and Neuroticism personality traits greatly impact how people perceive information in terms of trustworthiness. Consequently, individuals who score high in these two traits tend to be more susceptible to phishing attacks. In another study utilizing the Big Five personality traits, Halevi et al.³⁶ looked into the correlation between phishing, personality traits, and Facebook activity. The participants were asked to complete an online survey where the authors acquired their demographics/background information, online activity, and Facebook related information. Using the e-mail provided from the survey, the authors also sent out a fake phishing e-mail to all participants. Seventeen percent of the participants fell victim to the phishing attack. Fourteen percent of the male population fell victim along with 53% of the female population. In regard to the personality traits, for women, it was found that there was a very high positive correlation between Neuroticism and phishing. However, there were not any correlations found between men and any of the personality traits.

Several researchers have referenced principles of persuasion being used in effective phishing e-mails. Based on the work of Robert Caldini, the most common principles of persuasion are authority, social proof, scarcity, liking/similarity, commitment/consistency, and reciprocation.^37–39 However, in terms of phishing e-mails, researchers agree that authority is one of the most effective principles of persuasion.^37,39 As it relates to authority, Caldini states “Information coming from a recognized authority can provide us a valuable shortcut for deciding how to act in a situation.”⁴⁰ In addition, Jones’ and Hartley’s¹⁷ research discusses defining adjectives for the DISC traits whereby the adjectives listed for Steadiness include loyal, obedient, and obliging; while those listed for Conscientiousness include compliant, resigned, and well-disciplined. These adjectives can be associated with someone who respects and responds well to authoritative fFigures

4. Methodology

Based on previous work regarding user susceptibility and cyber security education and training, we developed several research questions and hypotheses. Our research questions aim to examine the effectiveness of training programs and how participants’ demographics may impact user susceptibility. Based on previous research, we know that training programs are capable of teaching participants about phishing^12,19. However, we also want to know how they impact a person’s ability to mitigate phishing attacks and determine risk. Taking this a step further, we also decided to investigate how a person’s demographics and personality traits impact the training program. Our hypotheses were formulated based on the objectives of the training program and previous studies on user susceptibility. Utilizing the TTAT and the KnowBe4 objectives – recognizing personal risk, demonstrating risk awareness, and taking actions to minimize risk – we developed our first hypothesis stating that the training will improve participants’ self-perceived knowledge, tendencies, and risk. Based on the assumption that security training programs that discuss phishing should decrease user susceptibility, we developed our second hypothesis stating that the training will improve participants’ ability to detect phishing attacks. Finally, based on the results of previous studies that provided conflicting demographic susceptibility results, we developed our last hypothesis stating that the demographics and personality traits will not have a significant impact on the effectiveness of the training.

The pre- and a post-training surveys were developed by adapting questions from relevant previously published studies. For example, questions pertaining to computer skills and perceived risk were adapted from work by Anwar et al.³⁴ By adapting questions from relevant studies, we were able to use vetted survey questions rather than implementing newly created questions that may not be suited for measuring objective related data. In regard to the personality trait portion of the survey, the questions were gathered from online DISC assessments.⁴¹ The study protocol received the approval of the IRB by way of an exempted status. The KnowBe4 training used for this research was purchased through a license by The University and was conducted by its IT department. Participants were recruited from The University’s employees who were registered to complete the training through KnowBe4. The link to the pre-training survey was sent out through an e-mail explaining the research and asking for participants. The link to the post-training survey was only sent out to participants who completed the pre-training survey. The training was supposed to be administered to the entire campus faculty and staff; however, due to the coronavirus outbreak, only a portion of the faculty and staff received training, which reduced the participant pool for the post-training survey.

In order to answer the research questions, participants were asked to fill out a survey before and after completing the training. Since participants were able to complete the training on their own terms, they were asked to fill out the post-training survey shortly after completing the training program. This is a practice used in several similar previous studies.^3,22,23 The pre-training survey consists of 75 questions split into eight sections while the post-training survey has 56 questions and includes six of the original eight sections. The sections are shown in Table 1 and the questions from the surveys can be found in Appendix A.

Table 1. Pre and post training survey sections.

Survey Section Topic	Pre-Training Survey	Post-Training Survey
Background/Demographics
Self-Perceived Computer Skills
Personality Traits Test
Phishing Knowledge
Behavioral Tendencies
Self-Perceived Risk
Phishing Identification Test
Scenarios

The demographics section included questions about gender, race/ethnicity, age, and education. The perceived computer skills section asked about the participant’s self-perceived computer skills. The personality traits section utilized a simple 14 question DISC personality assessment created using specific keywords to describe the traits. The questions for this assessment came from an online DISC assessment.³⁶ In the pre-training survey, the phishing knowledge section was used to gain understanding of the participants’ prior knowledge of phishing attacks. However, in the post-training survey, the phishing knowledge section was used to gain understanding of any new information learned by the participants. For the pre-training survey, the behavioral tendencies section depicts the current positive and negative behaviors of the participant; whereas, in the post-training survey, this information shows which tendencies the participant thinks are important or unimportant for preventing phishing attacks. This section included questions on topics such as opening e-mails from unknown senders and informing the IT department about suspicious e-mails. The perceived risks section assessed participants’ self-perceived risk by asking questions to determine how well the participants believe their work environment is protected/safe against phishing attacks. The phishing identification section consisted of a phishing test where participants were given several phishing and legitimate URLs and were asked to differentiate which URL belongs to which category and their confidence level (based on a Likert scale) in their answer. The participants were never given the answer to a question and were given the same URLs in the pre-training and post-training surveys in order to measure improvement. The scenarios section included five scenario-based short answer questions to determine susceptibility. The purpose of these five questions is to see the thought process behind the participants’ actions in the case of a phishing attack. Both the pre-training and post-training surveys were completed online via an online survey tool software – Qualtrics. A python program was used to extract the data from a csv file generated by Qualtrics and place it into a Pandas data frame. The data frames were traversed to determine any increases/decreases in the participants’ self-perceived categories.

5. Results and discussion

There were 119 responses to the pre-training survey and 33 responses to the post-training survey. Due to a technical error, nine of the responses for the pre-training survey had answers that were omitted when submitting, so those nine responses were not included in the data analysis. There was also one response from the post-training survey data that had an omitted answer; thus, it was not included in the data analysis. These ten responses were only used for examining the overall comparison between the pre- and post-training surveys. Due to the COVID-19 pandemic, the KnowBe4 training at The University was suspended after a month and a half. At the time this manuscript was written, training had not resumed which prevented the acquisition of more participants.

5.1. Pre and post training accuracy and confidence

The pre- and post-training accuracy in the phishing identification test and their confidence in their answers are presented in Table 2, while t-test results are presented in Table 3.

Table 2. Pre and post training average accuracy and confidence.

	Pre-Survey Information			Post-Survey Information			Difference
Category	# of Participants	Average Accuracy (% – std)	Average Confidence (% – std)	# of Participants	Average Accuracy (% – std)	Average Confidence (% – std)	Average Accuracy (%)	Average Confidence (%)
Including Incomplete Responses	119	59.524 – 16.852	66.653 – 18.221	33	67.677 – 19.956	72.828 – 14.586	8.153	6.175
Excluding Incomplete Responses	110	59.621 – 17.051	66.121 – 17.888	32	68.229 – 20.017	72.552 – 14.732	8.608	6.431

Table 3. Pre and post training accuracy and confidence T-Test.

	Pre and Post Accuracy T-Test		Pre and Post Confidence T-Test
Category	t-statistic	p-value	t-statistic	p-value
Including Incomplete Responses	2.1444	0.0374	2.0320	0.0464
Excluding Incomplete Responses	2.2104	0.0322	2.0658	0.0432

That data analysis showed that both average accuracy and confidence increased between the pre- and post-training surveys. The accuracy increased around 8% – from 59.524% to 67.677% – while confidence had an increase of around 6% – from 66.653% to 72.828%. This information was further analyzed through a two-sample unequal variance t-test with an alpha level of 0.05. The results of this analysis are a p-value of 0.0374 between the pre- and post-training average accuracies, and a p-value of 0.0464 between the pre- and post-training average confidence levels. A moderate positive correlation (Pearson’s r = 0.4996, p-value = 0.0036) between post-training accuracy and post-training confidence was also found. This confirms the hypothesis that the KnowBe4 training has a positive impact on participants’ performance in identifying phishing attacks.

5.2. Phishing knowledge, behavior, and risk

The data analysis also showed increases in phishing knowledge, behavior, and perceived risk from the pre- to post-training (shown in Table 4). The average for phishing knowledge score increased from 3.63 to 4.25 out of a total of 5.33. The average response for user behavior improved, whereby the average score increased from 2.86 to 3.28 out of a total of 5. The average score for self-perceived risk increased from 5.48 to 5.6 out of a total of 7. A two-sample unequal variance t-test results showed statistically significant differences in pre- to post-training phishing knowledge (t-stat = 4.86, p-value = 0.0000) and user behavior (t-stat = 2.09, p-value = 0.0419).

Table 4. Pre and post training self-perceived value averages.

Self-Reported Sections	Pre-Training Data	Post-Training Data	Max Value	t-statistic	p-value
Computer Skills	6.32	6.28	7	−0.2611	0.7953
Phishing Knowledge	3.63	4.25	5.33	4.8626	0.0000
User Behavior	2.86	3.28	5	2.0886	0.0419
Perceived Risk	5.48	5.6	7	0.715	0.4774

5.3. Personality traits and accuracy/confidence

For the rest of the analysis, a one-hot encoding on the Pandas data frame to change each subcategory into a binary value was used first. Then either PB Correlation or Biserial Correlation analyses between each binary subcategory value and the four components of the pre- and post-training data were performed. Table 5 displays the average accuracy with standard deviation and confidence with standard deviation per personality trait, while Table 6 contains the PB Correlation coefficients and p-values. Correlations are considered weak if r < 0.4, moderate if 0.4 ≤ r < 0.6, and strong if r ≥ 0.6. A box and whiskers plot of the pre- and post-confidence per personality trait are presented in Figure 1.

Figure 1. Pre and post training average confidence per personality trait.

Table 5. Pre and post training personality trait average accuracies and confidence.

	Pre-Survey Information			Post-Survey Information				Difference
Category	# of Participants	Average Accuracy (% – std)	Average Confidence (% – std)	# of Participants	Average Accuracy (% – std)		Average Confidence (% – std)	Average Accuracy (%)	Average Confidence (%)
Personality
Dominant	11	51.515 – 14.346	72.879 – 16.851	1	75 – 0	91.667 – 0		23.485	18.788
Influence	12	63.889 – 19.57	60.972 – 15.2	4	81.25 – 4.167	68.333 – 12.546		17.361	7.361
Steadiness	61	58.743 – 16.132	63.142 – 17.907	18	65.278 – 21.815	71.574 – 13.497		6.535	8.432
Conscientiousness	26	63.141 – 18.435	72.628 – 17.658	9	67.593 – 20.601	74.259 – 18.298		4.452	1.631

Table 6. Personality PB correlation coefficients and P-values.

	Pre-Survey PB Correlation				Post-Survey PB Correlation
Subcategory	Acc. Coeff.	Acc. p-value	Conf. Coeff.	Conf. p-value	Acc. Coeff.	Acc. p-value	Conf. Coeff.	Conf. p-value
Dominant	-0.1592	0.0967	0.1265	0.1879	0.0617	0.7372	0.2368	0.1920
Influence	0.0880	0.3607	-0.1012	0.2929	0.2498	0.1680	-0.1100	0.5491
Steadiness	-0.0577	0.5492	-0.1867	0.0509	-0.1699	0.3527	-0.0765	0.6774
Conscientiousness	0.1155	0.2301	0.2033	0.0332	-0.0202	0.9126	0.0737	0.6887

Due to the limited responses of the post-training survey data, only one participant belonged to the Dominant personality group; therefore, no inferences were made about this data. Four participants belonged to the Influence personality group, eighteen belonged to the Steadiness personality group, and nine belonged to the Conscientiousness personality group. Participants within the Influence (63.89%) and Conscientiousness (63.14%) personality groups had the highest accuracy scores, before training. The Influence group also had the highest accuracy score (81.25%) after training, while the Dominant group had the next highest accuracy score (75%). The Dominant and Conscientiousness groups ranked the highest in average confidence level in their answer choices before training: 72.88% and 72.63%, respectively. The Dominant group had the highest level of confidence in their choices after training with an average confidence level of 91.67%. The most susceptible personality group before training was the Dominant group with the lowest average accuracy of 51.52%. After training, the Steadiness group was the most susceptible with an average accuracy of 65.28%. The Conscientiousness group was not far behind with an average accuracy of 67.59%. In addition to this information, it was also noted that the Dominant group had the largest difference between pre- and post-training data followed by the Influence, Steadiness, and Conscientiousness groups. Significant correlations were found in participants’ confidence levels for both the Steadiness and Conscientiousness groups. For instance, there was a weak negative correlation between the Steadiness group and the participants’ pre-training confidence level (r_pb = −0.1867, p-value = 0.0509). This means participants in the Steadiness group had less confidence in their answer when identifying phishing URLs when compared to participants within the other personality groups. There was a weak positive correlation between Conscientiousness and participants’ pre-training confidence level as well (r_pb = 0.2033, p-value = 0.0332). This is the opposite of the participants in the Steadiness group, which implies participants in the Conscientious group have more confidence in identifying phishing URLs than participants in the other personality groups.

5.4. Gender and accuracy/confidence

About one third of the participants from each of the pre- and post-training groups were males. Table 7 shows the pre- and post-training survey information while Table 8 shows the PB Correlation coefficients and p-values. Only the results for the pre-training confidence scores were significant, such that the PB Correlation analysis shows a weak positive correlation for the males (r_pb = 0.1957, p-value = 0.0405) and a weak negative correlation for the females (r_pb = −0.1957, p-value = 0.0405). This means that before completing the training, males were more likely to be confident in their answer choices than females. A box and whiskers plot pertaining to the pre- and post-confidence for each gender can be found in Figure 2.

Figure 2. Pre and post training average confidence per gender.

Table 7. Pre and Post Training Gender Average Accuracies and Confidence.

	Pre-Survey Information			Post-Survey Information			Difference
Category	# of Participants	Average Accuracy (% – std)	Average Confidence (% – std)	# of Participants	Average Accuracy (% – std)	Average Confidence (% – std)	Average Accuracy (%)	Average Confidence (%)
Gender
Male	42	63.095 – 16.983	70.556 – 18.156	11	74.242 – 22.808	73.939 – 21.138	11.147	3.383
Female	68	57.475 – 16.861	63.382 – 17.289	21	65.079 – 18.185	71.825 – 10.553	7.604	8.443

Table 8. Gender PB correlation coefficients and P-values.

	Pre-Survey PB Correlation				Post-Survey PB Correlation
Subcategory	Acc. Coeff.	Acc. p-value	Conf. Coeff.	Conf. p-value	Acc. Coeff.	Acc. p-value	Conf. Coeff.	Conf. p-value
Male	0.1608	0.0932	0.1957	0.0405	0.2209	0.2244	0.0692	0.7065
Female	-0.1608	0.0932	-0.1957	0.0405	-0.2209	0.2244	-0.0692	0.7065

5.5. Race/ethnicity and accuracy/confidence

The next category with a significant correlation is the race/ethnicity category. There were a couple of subcategories that were not represented in the pre-training or post-training data (i.e., American Indian/Alaska Native and Native Hawaiian/Pacific Islander) and they were removed from the following race/ethnicity tables. For the other subcategories, the number of people – along with the average accuracy and confidence – can be seen in Table 9. As seen in Table 10, only Whites and Hispanics/Latinx experienced slight correlations; however, both subcategories have limited sample sizes, so no inferences can be made within these subcategories. The Whites had a weak positive correlation for pre-training accuracy (r_pb = 0.1977, p-value = 0.0385). This means that compared to other races before training, White participants were able to detect phishing URLs at a higher rate. Figure 3 displays the accuracy and confidence information for the Race/Ethnicity category.

Figure 3. Pre and post training average accuracy and confidence per race/ethnicity.

Table 9. Pre and post training race/ethnicity average accuracies and confidence.

	Pre-Survey Information			Post-Survey Information					Difference
Category	# of Participants	Average Accuracy (% – std)	Average Confidence (% – std)	# of Participants		Average Accuracy (% – std)		Average Confidence (% – std)	Average Accuracy (%)		Average Confidence (%)
Race/Ethnicity
White	24	65.972 – 17.535	67.917 – 18.14	6	79.167 – 13.693		71.111 – 15.975		13.195	3.194
Black	77	57.684 – 16.818	64.307 – 17.943	23	64.13 – 21.236		72.826 – 15.419		6.446	8.519
Hispanic/Latinx	2	58.333 – 11.785	90 – 14.142	2	75 – 11.785		79.167 – 1.179		16.667	−10.833
Asian	4	60.417 – 15.775	68.333 – 8.498	1	83.333 – 0		61.667 – 0		22.916	−6.666
Other	3	58.333 – 22.048	79.444 – 14.175	0	N/A		N/A		N/A	N/A

Table 10. Race/Ethnicity PB correlation coefficients and P-values.

	Pre-Survey PB Correlation				Post-Survey PB Correlation
Subcategory	Acc. Coeff.	Acc. p-value	Conf. Coeff.	Conf. p-value	Acc. Coeff.	Acc. p-value	Conf. Coeff.	Conf. p-value
White	0.1977	0.0385	0.0533	0.5805	0.2667	0.1401	-0.0477	0.7953
Black	-0.1743	0.0685	-0.1556	0.1045	-0.3326	0.0629	0.0302	0.8696
Hispanic/Latinx	-0.0103	0.9147	0.1825	0.0564	0.0887	0.6291	0.1178	0.5208
Asian	0.0091	0.9248	0.0241	0.8024	0.1377	0.4524	-0.1348	0.4619
Other	-0.0127	0.8952	0.1253	0.1922	N/A	N/A	N/A	N/A

5.6. Years working and accuracy/confidence

The years working at The University also resulted in significant accuracy and confidence correlations. The categorical information can be found in Table 11. This was also the first section using Biserial Correlation instead of PB Correlation as seen in Table 12. Here, participants who have been working at The University between 6–9 years had moderate negative correlations with both post-training accuracy (r_b = −0.4976, p-value = 0.0155) and confidence (r_b = −0.4559, p-value = 0.0285). Figure 4 shows the average accuracy and confidence for each subcategory group within the Years Working category.

Figure 4. Pre and post training average accuracy and confidence per years working.

Table 11. Pre and post training years working average accuracies and confidence.

	Pre-Survey Information			Post-Survey Information			Difference
Category	# of Participants	Average Accuracy (% – std)	Average Confidence (% – std)	# of Participants	Average Accuracy (% – std)	Average Confidence (% – std)	Average Accuracy (%)	Average Confidence (%)
Years Working (at The University)
< 1	22	59.848 – 17.75	64.47 – 16.922	10	74.167 – 15.933	74.333 – 13.613	14.319	9.863
1–2	14	64.286 – 16.482	71.786 – 13.72	5	70 – 20.917	76.667 – 19.614	5.714	4.881
3–5	21	65.079 – 17.204	66.667 – 18.143	3	58.333 – 28.868	68.333 – 5.774	−6.746	1.666
6–9	15	55.556 – 13.968	64.889 – 18.586	4	52.083 – 20.833	61.667 – 3.6	−3.473	−3.222
10+	38	56.36 – 17.484	65.175 – 19.746	10	70.833 – 20.127	74.333 – 17.448	14.473	9.158

Table 12. Years working biserial correlation coefficients and P-values.

	Pre-Survey Biserial Correlation				Post-Survey Biserial Correlation
Subcategory	Acc. Coeff.	Acc. p-value	Conf. Coeff.	Conf. p-value	Acc. Coeff.	Acc. p-value	Conf. Coeff.	Conf. p-value
< 1	0.0096	0.9360	−0.0663	0.5778	0.2660	0.2197	0.1084	0.6225
1–2	0.1678	0.1561	0.1942	0.0997	0.0586	0.7906	0.1851	0.3979
3–5	0.2256	0.0550	0.0215	0.8569	−0.2813	0.1932	−0.1630	0.4576
6–9	−0.1494	0.2073	−0.0432	0.7171	−0.4976	0.0155	−0.4559	0.0285
10+	−0.1801	0.1275	−0.0498	0.6760	0.1167	0.5962	0.1084	0.6225

5.7. Education level and accuracy/confidence

Tables 13 and 14 show the categorical information and Biserial correlation information for the Education category, respectively. The Bachelor’s degree group had a moderate negative correlation with post-training confidence (r_b = −0.4496, p-value = 0.0311) and the PhD/Master’s group had a moderate positive correlation with post-training confidence (r_b = 0.4246, p-value = 0.0431). This means, after training, employees with only a Bachelor’s degree had less confidence in identifying phishing and legitimate URLs than other groups while employees with a PhD/Master’s degree had more confidence than other groups. Figure 5 depicts the average pre- and post-training confidence per Education level of the participants.

Figure 5. Pre and post training average confidence per education level.

Table 13. Pre and post training education average accuracies and confidence.

	Pre-Survey Information			Post-Survey Information						Difference
Category	# of Participants	Average Accuracy (% – std)	Average Confidence (% – std)	# of Participants		Average Accuracy (% – std)		Average Confidence (% – std)		Average Accuracy (%)		Average Confidence (%)
Level of Education
High School	2	66.667 – 11.785	45 – 4.714	1	75 – 0		76.667 – 0		8.333		31.667
< 4 Years	19	56.14 – 19.413	66.404 – 18.047	6	65.278 – 31.069		77.5 – 17.725		9.14		11.096
Bachelors	33	60.859 – 15.378	66.061 – 19.315	10	69.167 – 21.89		65.167 – 13.731		8.308		−0.894
Professional	30	56.667 – 17.833	64.833 – 18.228	9	66.667 – 17.678		71.111 – 11.873		10		6.278
PhD/Master’s	26	63.462 – 16.679	69.103 – 15.806	6	73.611 – 8.193		81.389 – 15.071		10.149		12.286

Table 14. Education biserial correlation coefficients and P-values.

	Pre-Survey Biserial Correlation				Post-Survey Biserial Correlation
Subcategory	Acc. Coeff.	Acc. p-value	Conf. Coeff.	Conf. p-value	Acc. Coeff.	Acc. p-value	Conf. Coeff.	Conf. p-value
High School	0.1690	0.1530	−0.4830	0.0000	0.1526	0.4871	0.1260	0.5669
< 4 Years	−0.1386	0.2426	0.0107	0.9284	−0.2026	0.3540	0.2377	0.2746
Bachelors	0.0629	0.5973	−0.0029	0.9803	0.0420	0.8492	−0.4496	0.0311
Professional	−0.1429	0.2281	−0.0594	0.6181	−0.0661	0.7645	−0.0829	0.7072
PhD/Master’s	0.1735	0.1423	0.1284	0.2793	0.1903	0.3845	0.4246	0.0431

5.8. Previous training

The Previous Training category was formed by participants selecting whether or not they had participated in any previous security training. Participants who were unsure of previous training were grouped with participants without previous training. The categorical information is shown in Table 15 while Table 16 shows the PB correlation information. Figure 6 illustrates the accuracy and confidence information found in Table 15. There were significant correlations between previous training with pre-training accuracy and confidence level. There were weak positive correlations for participants with previous training for both pre-training accuracy (r_pb = 0.317, p-value = 0.0007) and confidence (r_b = 0.2185, p-value = 0.0218) while participants who were unsure of or had not had previous training had weak negative correlations with both pre-training accuracy (r_pb = −0.317, p-value = 0.0007) and confidence (r_b = −0.2185, p-value = 0.0218). This means that participants who experienced previous training were better and more confident in identifying phishing and legitimate URLs than participants who had not or were unsure of completing previous training.

Figure 6. Pre and post training average accuracy per previous training.

Table 15. Pre and post training previous training average accuracy and confidence.

	Pre-Survey Information			Post-Survey Information			Difference
Category	# of Participants	Average Accuracy (% – std)	Average Confidence (% – std)	# of Participants	Average Accuracy (% – std)	Average Confidence (% – std)	Average Accuracy (%)	Average Confidence (%)
Previous Training
Yes	42	66.468 – 17.117	71.071 – 16.091	16	69.792 – 21.704	76.042 – 17.18	3.324	4.971
No/Unsure	68	55.392 – 15.691	63.064 – 18.365	16	66.667 – 18.758	69.062 – 11.287	11.275	5.998

Table 16. Previous Training PB correlation coefficients and P-values.

	Pre-Survey PB Correlation				Post-Survey PB Correlation
Subcategory	Acc. Coeff.	Acc. p-value	Conf. Coeff.	Conf. p-value	Acc. Coeff.	Acc. p-value	Conf. Coeff.	Conf. p-value
Yes	0.3170	0.0007	0.2185	0.0218	0.0793	0.6661	0.2407	0.1846
No/Unsure	-0.3170	0.0007	-0.2185	0.0218	-0.0793	0.6661	-0.2407	0.1846

5.9. Age and employment status

The Age and Employment Status categories had no apparent influence on susceptibility (see Tables 17 and 18).

Table 17. Pre and post training age and employment status accuracies and confidence.

	Pre-Survey Information			Post-Survey Information			Difference
Category	# of Participants	Average Accuracy (% – std)	Average Confidence (% – std)	# of Participants	Average Accuracy (% – std)	Average Confidence (% – std)	Average Accuracy (%)	Average Confidence (%)
Age
26–36	24	62.847 – 15.731	63.611 – 17.732	7	70.238 – 12.599	75 – 8.872	7.391	11.389
37–59	69	57.85 – 18.682	66.498 – 17.85	21	68.254 – 22.301	72.619 – 15.308	10.404	6.121
60+	17	62.255 – 10.256	68.137 – 18.956	4	64.583 – 21.916	67.917 – 21.916	2.328	−0.22
Employment Status
Full-Time	104	59.936 – 17.305	66.33 – 17.983	30	67.5 – 20.453	72.278 – 14.671	7.564	5.948
Part-Time	5	53.333 – 12.638	65 – 18.028	2	79.167 – 5.893	76.667 – 21.213	25.834	11.667

Table 18. Age and employment status biserial correlation coefficients and P-values.

	Pre-Survey Biserial Correlation				Post-Survey Biserial Correlation
Subcategory	Acc. Coeff.	Acc. p-value	Conf. Coeff.	Conf. p-value	Acc. Coeff.	Acc. p-value	Conf. Coeff.	Conf. p-value
Age
26–36	0.1407	0.2352	−0.1044	0.3798	0.0756	0.7319	0.1251	0.5696
37–59	−0.1729	0.1436	0.0350	0.7687	0.0022	0.9919	0.0082	0.9703
60+	0.1008	0.3963	0.0736	0.5364	−0.1124	0.6099	−0.1941	0.3748
Employment Status
Full-Time	0.1586	0.1803	0.1004	0.3984	−0.2821	0.1919	−0.1442	0.5116
Part-Time	−0.1762	0.1360	−0.0300	0.8015	0.2821	0.1919	0.1442	0.5116

5.10. Discussion

Table 19 summarizes the findings in terms of the three hypotheses. Based on the results of the analyses, the first and third hypotheses can only be partially accepted, while the second hypothesis can be fully accepted. The first hypothesis stated that, following the logic of the TTAT and, because of the KnowBe4 training, participants would experience an improvement in phishing knowledge, user behavior, and perceived risk. After running a t-test on the data collected, only participants’ self-perceived phishing knowledge and behavior improved as a confirmed result of the training. However, improvement in participants’ self-perceived risk cannot be confirmed.

Table 19. Hypotheses results and explanations.

Number	Hypothesis	Result	Explanation
1	The TTAT principles and KnowBe4 training program improves participants’ self-reported phishing knowledge, behavior tendencies, and perceived risk.	Partially Accepted	We were able to confirm that the participants’ self-perceived phishing knowledge and behavioral tendencies improved as a result of the training. For the phishing knowledge this increased from 68% to 80%. For the behavioral tendencies, this increased from 57% to 66%. This also partially supports the TTAT principles as threat avoidance behavior and threat severity improved; however, perceived threat susceptibility did not.
2	Participants are able to identify phishing attacks at a higher accuracy after completing the KnowBe4 training.	Accepted	We were able to confirm using the phishing test in the surveys that participants accuracy and confidence in detecting phishing attacks increased after completing the training. Before training the average accuracy was around 60% while after the training it was 68%. Meanwhile, the average confidence was around 66% and after the training it was 73%.
3	Participant’s demographics and personality traits will not have a significant impact on the effectiveness of the KnowBe4 training program.	Partially Accepted	We were able to confirm from the subcategories with enough participants, that a majority of the subcategories did not perform significantly better or worse than other subcategories. However, we did notice that most of the categories had one or two subcategories that did have a significant difference.

The third hypothesis postulates that due to the various training methods, there will not be a difference between the subcategories within each demographic or the personality traits. This, again, is partially accepted and supported through the correlations between each subcategory and the accuracies and confidence levels. There were several subcategories that had a direct correlation with regard to producing higher accuracies and confidence levels and some that had a direct correlation with regard to producing lower accuracies and confidence levels. We can state that subcategories that produced significantly higher results are less susceptible to phishing attacks than their counterparts while subcategories with significantly lower results are more susceptible. Finally, based on the results, the second hypothesis, which states that because of the various training methods, the participants will be more adept at identifying phishing attempts, can be fully accepted based on the increase in accuracy and confidence level experienced after training. In addition to the t-test showing a significant increase in accuracy and confidence level, a direct correlation between post-training accuracy and confidence level was also discovered. This means that participants who were able to correctly identify phishing attacks were more confident in their answer choices.

Due to a lack of adequate participation in the post-training survey, inferences between many of the subcategories and their accuracy and confidence levels when detecting phishing URLs cannot be made. However, when looking into the Biserial and PB Correlation, there were some subcategories that were close to the 0.05 alpha level. There is a negative correlation between Blacks/African Americans with pre- and post-training accuracy. Meaning that Blacks/African Americans could be more susceptible than other races/ethnicities before and after the training. There is also a slight correlation between Hispanics/Latinx and pre-training confidence, indicating a higher chance in Hispanics/Latinx being confident in their answer choices more than other races/ethnicities. Meanwhile people working at The University 3–5 years had a positive correlation with pre-training accuracy, meaning that employees in this group were able to accurately identify phishing and legitimate URLs better than other groups. There were also other trends that could be significant with a larger sample size. After training, we can state that people with Bachelor’s degrees are less confident in identifying phishing URLs while people with PhDs/Master’s degrees are more confident in identifying phishing URLs. In addition, much like in Anwar’s 2016 paper, it appears that more security training is needed as a majority of the employees have not had or are unsure of having previous security training.³⁴ Also, interestingly, part-time employees are originally more susceptible to phishing attacks than full-time employees; however, this changed after both parties experienced the training. Similar to Anwar et al.’s⁴² study showing that females reported lower self-efficacy scores than males, we noted that females reported lower confidence than males before and after training. Our findings on personality traits do not support the conclusion by Cho that people expressing agreeableness are more susceptible to phishing attacks.³⁵ Agreeableness in the Big Five personality traits is comparable to people expressing the Conscientiousness DISC personality trait, which according to our research, is one of the least susceptible personality traits before training.

The results of our research suggests that women are more susceptible than men; however, based on the number of false negative responses, women are less susceptible after training. The percentage of false negative responses for women was 18.3% before training which decreased to 15.9% after training. Other subcategories saw improvements in true and false negative responses as well. The Steadiness group’s percentage of false negatives decreased from 20.3% before training to 13% after training and the percentage of true negatives increased from 47.2% to 52.8%. Two subcategories in the Years Working group improved their true and false positive responses as well. The 3–5 years group improved their true negatives from 44.4% to 50% and their false negatives from 16.7% to 0%. The 6–9 years group improved their false negatives from 29.2% to 12.5% while their true negatives did not change. Participants who only completed some college decreased their false negatives from 29.1% before training to 8.3% after training and increased their true negatives from 41.7% to 50%. Both participants who did not have previous training and were unsure of previous training improved their true and false negatives after the KnowBe4 training.

6. Limitations

There are several limitations in this work. The most obvious limitation is the difference in research participants between the pre- and post-training sessions. This difference caused issues with collecting information from each subcategory in each demographic section. Therefore, we were not able to come to a full conclusion about certain subgroups’ susceptibility to phishing attacks. For example, conclusions on race were not drawn for the Hispanic, Asian, and Other groups due to the small number of participants for those groups. This limitation was mostly caused by the impact of the COVID-19 global pandemic. Also, we did not consider how long it took participants to complete a training module or how many attempts it took for them to pass the quizzes as this information was not provided to us by the University. Another limitation is only testing the participants immediately after training. Participants are hyperaware of phishing attacks immediately after training which is not a normal phishing scenario.

7. Conclusion and future work

This research was designed to investigate the effects of specific user demographics and personality traits on anti-phishing training programs and user susceptibility as they relate to the identification of phishing URLs. Based on the data collected, significant results were found. For example, by running two-sample t-tests, we were able to determine that the anti-phishing training had a positive improvement on participants’ phishing knowledge, their behavioral tendencies, and their ability to identify phishing attacks. These t-tests also determined that people with different levels of education may be impacted differently by the training. We were also able to illustrate that the training does not improve a participants’ perceived risk or reduce their susceptibility based on the DISC personality traits. This research is important so that organizations may be able to target highly susceptible individuals for increased training to prevent any breaches in security. Based on the results of this research, we believe that organizations may benefit from increasing training for employees who have been working between 6–9 years, have a Bachelor’s degree as their highest level of education, or are unsure of having or have not had previous training. In the future, we would like to address the limitations and biases of this work. By interviewing participants who completed the anti-phishing training, we could inquire more on their thoughts and understanding of phishing attacks before and after training. We would also like to continue examining phishing susceptibility by expanding the research to a wider participant pool. With an increased number of participants, we would like to analyze multiple categories at once to determine phishing susceptibility. For example, we could analyze age and gender or race/ethnicity and education level. Another plan for future implementation is to work closely with The University in order to create a long-term retention study where we can continue to test participants over an extended period of time to see how much information they retain from the training and to gather data from a non-hyper aware group.

Appendix A: Participant Survey Questions

Q) What is your gender?

1. Male

2. Female

Q) What is your ethnicity?

1. White

2. Black or African American

3. American Indian or Alaska Native

4. Hispanic or Latino

5. Asian

6. Native Hawaiian or Pacific Islander

7. Other (please specify) _______________________________

Q) Please select your age group.

1. 18–25

2. 26-36

3. 37–59

4. 60+

Q) Please select your status.

1. Full-time

2. Part-time

3. Student

4. Other (please specify) _______________________________

Q) How many years have you worked here?

1. less than 1 year

2. 1–2 years

3. 3–5 years

4. 6–9 years

5. 10+ years

Q) What is your highest level of education?

1. High school graduate

2. Some College/Associate’s Degree

3. Bachelor’s Degree

4. Professional Degree

5. Doctorate/Master’s Degree

6. Other (please specify) _______________________________

Q) What department do you work in?

_______________________________________________

Q) Have you ever completed an information security training session before?

1. Yes

2. Unsure

3. No

Q) If yes, did the training session discuss phishing attacks?

1. Yes

2. Unsure

3. No

4. Not applicable

End of Block: Demographics/Background

Start of Block: Computer Skills

Q) How comfortable are you with using a computer?

1. Extremely comfortable

2. Moderately comfortable

3. Slightly comfortable

4. Neither comfortable nor uncomfortable

5. Slightly uncomfortable

6. Moderately uncomfortable

7. Extremely uncomfortable

Q) How comfortable are you with using your e-mail?

1. Extremely comfortable

2. Moderately comfortable

3. Slightly comfortable

4. Neither comfortable nor uncomfortable

5. Slightly uncomfortable

6. Moderately uncomfortable

7. Extremely uncomfortable

Q12 How would you rate your general computer knowledge/skills?

Q13 How would you rate your internet knowledge/skills?

End of Block: Computer Skills

Start of Block: Personality Questions

Q) Choose one statement that is most like you …

1. Talkative

2. Daring

3. Loyal

4. Reserved

Q) How do other people see you?

1. People see me as calm

2. People see me as being helpful

3. People usually like my company

4. People tend to look up to me

Q) You have to work on a team, what role do you take?

1. I always try to be the team leader

2. I ensure a strong team dynamic and like everyone to be involved and feel included

3. I just wait and I certainly don’t seek leadership

4. I’d rather work alone

Q) Choose one statement that is most like you …

1. Disciplined

2. Entertaining

3. Dominant

4. Patient

Q) Choose one statement that is most like you …

1. Charming

2. Decisive

3. Private

4. Cooperative

Q) Choose one statement that is most like you …

1. Fearless

2. Sociable

3. Generous

4. Analytical

Q) Choose one statement that is most like you …

1. Systematic

2. Bold

3. Even Tempered

4. Influential

Q) Choose one statement that is most like you …

1. I am very active, both at work and play

2. I always take notice of what other people say

3. I am always cheerful

4. I like to behave correctly

Q) There’s a conflict on your team, how do you react?

1. Let’s not upset anyone

2. Let’s hear more details

3. Let’s get to the point

4. Let’s talk about it

Q) Choose one statement that is most like you …

1. I enjoy chatting with people

2. I’m a good listener

3. I like to handle things with diplomacy

4. I enjoy competition

Q) Choose one statement that is most like you …

1. I am always willing to do new things – to take a risk

2. I am always willing to follow orders

3. I am always ready to help

4. I am a very social sort of person

Q) Choose one statement that is most like you …

1. I am very patient

2. I don’t treat life too seriously

3. I enjoy competition

4. I am an agreeable person

Q) Choose one statement that is most like you …

1. I don’t like arguments

2. I am a sensitive person

3. I control my emotions

4. I am an assertive person

Q) Choose one statement that is most like you …

1. Shy

2. Adventurous

3. Calm

4. Chatty

End of Block: Personality Questions

Start of Block: Phishing Knowledge

Q) How well do you know about general phishing attacks?

1. Extremely well

2. Very well

3. Moderately well

4. Slightly well

5. Not well at all

Q) When browsing the internet, I can identify a phishing website …

1. Strongly agree

2. Agree

3. Somewhat agree

4. Neither agree nor disagree

5. Somewhat disagree

6. Disagree

7. Strongly disagree

Q) When using my e-mail, I can identify a phishing e-mail …

1. Strongly agree

2. Agree

3. Somewhat agree

4. Neither agree nor disagree

5. Somewhat disagree

6. Disagree

7. Strongly disagree

Q) I know how to prevent a phishing attack …

1. Strongly agree

2. Agree

3. Somewhat agree

4. Neither agree nor disagree

5. Somewhat disagree

6. Disagree

7. Strongly disagree

Q) Have you ever received a phishing e-mail?

1. Yes

2. Unsure

3. No

Q) Have you ever fallen victim to a phishing attack?

1. Yes

2. Unsure

3. No

Q) Explain what phishing is to the best of your understanding …

________________________________________________

________________________________________________

________________________________________________

________________________________________________

End of Block: Phishing Knowledge

Start of Block: Behavioral

Q) How often do you open e-mails from unknown senders?

1. Always

2. Most of the time

3. Sometimes

4. Rarely

5. Never

Q) I carefully examine whether an e-mail I received has suspicious links or attachments, even when it comes from someone I know.

1. Always

2. Most of the time

3. Sometimes

4. Rarely

5. Never

Q) How often do you check to make sure a website is legitimate?

1. Always

2. Most of the time

3. Sometimes

4. Rarely

5. Never

When you receive a suspicious e-mail, how often do you …

Q) inform the rest of your department?

1. Always

2. Most of the time

3. Sometimes

4. Rarely

5. Never

Q) inform the IT department?

1. Always

2. Most of the time

3. Sometimes

4. Rarely

5. Never

End of Block: Behavioral

Start of Block: Perceived Risk

Q) Phishing is a risk in your department …

1. Strongly agree

2. Agree

3. Somewhat agree

4. Neither agree nor disagree

5. Somewhat disagree

6. Disagree

7. Strongly disagree

Q) Phishing attacks are a threat to the university …

1. Strongly agree

2. Agree

3. Somewhat agree

4. Neither agree nor disagree

5. Somewhat disagree

6. Disagree

7. Strongly disagree

Q) My e-mail’s security will stop phishing attacks from getting through …

1. Strongly agree

2. Agree

3. Somewhat agree

4. Neither agree nor disagree

5. Somewhat disagree

6. Disagree

7. Strongly disagree

Q) My chances of receiving a phishing e-mail are low …

1. Strongly agree

2. Agree

3. Somewhat agree

4. Neither agree nor disagree

5. Somewhat disagree

6. Disagree

7. Strongly disagree

Q) If a phishing attack were successful on my workstation/computer it would not affect my department or the university.

1. Strongly agree

2. Agree

3. Somewhat agree

4. Neither agree nor disagree

5. Somewhat disagree

6. Disagree

7. Strongly disagree

End of Block: Perceived Risk

Start of Block: Phishing Test

For the following section, please choose whether the website listed is a phishing website or a legitimate website and then mark your confidence level.

Q) https://web-ao.da-us.citibank.com/cgi-bin/

1. Phishing Website

2. Legitimate Website

Q) How confident do you feel in your answer?

1. Extremely confident

2. Very confident

3. Moderately confident

4. Slightly confident

5. Not confident at all

Q) http://account-verification.com/ebay/verify/

1. Phishing Website

2. Legitimate Website

Q) How confident do you feel in your answer?

1. Extremely confident

2. Very confident

3. Moderately confident

4. Slightly confident

5. Not confident at all

Q) https://www3.nationalgeographic.com/

1. Phishing Website

2. Legitimate Website

Q) How confident do you feel in your answer?

1. Extremely confident

2. Very confident

3. Moderately confident

4. Slightly confident

5. Not confident at all

Q) http://secure-signin.ebay.com.ttps.us/

1. Phishing Website

2. Legitimate Website

Q) How confident do you feel in your answer?

1. Extremely confident

2. Very confident

3. Moderately confident

4. Slightly confident

5. Not confident at all

Q) http://citibusinessonline.da-us.citibank-updates.com/

1. Phishing Website

2. Legitimate Website

Q) How confident do you feel in your answer?

1. Extremely confident

2. Very confident

3. Moderately confident

4. Slightly confident

5. Not confident at all

Q) www.amazon.com/help/confirmation

1. Phishing Website

2. Legitimate Website

Q) How confident do you feel in your answer?

1. Extremely confident

2. Very confident

3. Moderately confident

4. Slightly confident

5. Not confident at all

Q) http://www.paypal.com.ssl2.us/webscr.php?cmd=LogIn#

1. Phishing Website

2. Legitimate Website

Q) How confident do you feel in your answer?

1. Extremely confident

2. Very confident

3. Moderately confident

4. Slightly confident

5. Not confident at all

Q) http://www.msn-verify.com/

1. Phishing Website

2. Legitimate Website

Q) How confident do you feel in your answer?

1. Extremely confident

2. Very confident

3. Moderately confident

4. Slightly confident

5. Not confident at all

Q) http://pages.ebay.com/services/forum/feedback.html

1. Phishing Website

2. Legitimate Website

Q) How confident do you feel in your answer?

1. Extremely confident

2. Very confident

3. Moderately confident

4. Slightly confident

5. Not confident at all

Q) http://www.ebay.ca

1. Phishing Website

2. Legitimate Website

Q) How confident do you feel in your answer?

1. Extremely confident

2. Very confident

3. Moderately confident

4. Slightly confident

5. Not confident at all

Q) http://www.ebay-accept.com/login.php

1. Phishing Website

2. Legitimate Website

Q) How confident do you feel in your answer?

1. Extremely confident

2. Very confident

3. Moderately confident

4. Slightly confident

5. Not confident at all

Q) https://www.paypal.com/ssl2/us/webscr?cmd=_login-run

1. Phishing Website

2. Legitimate Website

Q) How confident do you feel in your answer?

1. Extremely confident

2. Very confident

3. Moderately confident

4. Slightly confident

5. Not confident at all

End of Block: Phishing Test

Start of Block: Personality Scenarios

Q) You receive an e-mail from your bank saying that someone may have access to your account. The e-mail contains a link so that you can sign into your online account, change your password, and view recent purchases to confirm if someone has hacked your account.

Is this an example of phishing?

1. No

2. Yes, please explain … ________________________________

Q) You receive the following e-mail.

Do you think this could be an example of phishing?

1. No

2. Yes, please explain … ________________________________

Q) You receive the following e-mail.

Do you think this could be an example of phishing?

1. No

2. Yes, please explain … _________________________

Q) You receive an e-mail from your boss, who is out of town. The e-mail asks for you to scan and send them a file that contains sensitive information that they forgot on their desk. You respond saying that you can’t send a file with such information. Your boss responds with the following. Please select which response would most likely make you send the file.

1. … ”You need to send the file. If you do not then I will write you up when I return.”

2. … ”I know you shouldn’t send sensitive information, but that file is VERY important for a conference call that I have 10 minutes.”

3. … ”It’s alright. I will ask someone else to send me the file.”

4. … ”I know you shouldn’t send sensitive information, that is why I need someone I can trust to make sure that the file is sent and you are the only one in the office that I can trust. “

Q) (optional) In regards to the previous question, is there anything you would have done before sending the file? (i.e. asked around the office, confirmed you were talking to your boss, etc.)

______________________________________________

______________________________________________

______________________________________________

______________________________________________