Using Design-Science Based Gamification to Improve Organizational Security Training and Compliance

ABSTRACT

We conducted a design-science research project to improve an organization’s compound problems of (1) unsuccessful employee phishing prevention and (2) poorly received internal security training. To do so, we created a gamified security training system focusing on two factors: (1) enhancing intrinsic motivation through gamification and (2) improving security learning and efficacy. Our key theoretical contribution is proposing a recontextualized kernel theory from the hedonic-motivation system adoption model that can be used to assess employee security constructs along with their intrinsic motivations and coping for learning and compliance. A six-month field study with 420 participants shows that fulfilling users’ motivations and coping needs through gamified security training can result in statistically significant positive behavioral changes. We also provide a novel empirical demonstration of the conceptual importance of “appropriate challenge” in this context. We vet our work using the principles of proof-of-concept and proof-of-value, and we conclude with a research agenda that leads toward final proof-in-use.

KEYWORDS:

• computer security
• gamification
• design science research
• hedonic motivation
• system adoption model
• immersion
• flow
• security compliance
• security education
• training
• awareness
• SETA

Supplemental Material

Supplemental data for this article can be accessed on the publisher’s website.

Notes

1. Generally, gamification is the application of game-like features to nongaming systems to help foster a useful outcome other than entertainment [32, 92]. The features include design elements, such as points, levels, leaderboards, and badges.

2. Namely, they statistically rejected the associated hypotheses “H2: Individuals who receive gamified training will exhibit greater knowledge acquisition than individuals who receive non-gamified training or no training.” See page 20 of their text for statistical details.

3. Ultimately, users’ behaviors should be influenced by the gamified tasks in which a flow experience — or “immersion” in the systems version [3, 22] — is the objective. This objective can be achieved either through intrinsic or extrinsic motivation, but intrinsic motivation tends to be stronger for an instrumental goal [61, 87]. Intrinsic motivation can be involved in the task itself, whereas extrinsic motivation results from external factors (e.g., financial rewards or career goals).

4. We aim for both application to practice but also to tackle the challenge of integrating our unique gamified security learning context into theory [50]. This is challenging because contextualization is about “linking observations to a set of relevant facts, events, or points of view that make possible research and theory that form part of a larger whole” [86, p. 1]. Following Johns [50], we carefully evaluated, designed, and implemented the implications of contextual appreciation for both theory building and practice to achieve the best possible match between theoretical relevance and practical implications.

5. Meaningful engagement in this context refers to the outcomes of the gamification design. That is, the gamified system should foster (1) enjoyment, (2) interaction/engagement, and (3) enhanced instrumental task outcomes [59].

6. Other studies have implemented several of the gamification design principles, but typically in fields like computer science. Such studies are especially important for advancing gamification-related design and algorithms. However, most either used student subjects, did not advance a “cohesive theoretical foundation,” or did not focus on achieving meaningful engagement, as suggested by Liu et al. [59].

7. The organization we worked with preferred to have a simple system implemented without too much interaction between employees to prevent distractions from their normal work. Thus, we did not apply pie/bar charts, activity stream, giving kudos, social networking, forming teams, providing cash incentives, personalized goals, or social support.

8. For example, a study found that playing the Super Mario Bros. game resulted in a significant gray matter increase, impacting spatial navigation, strategic planning, and working memory [56]. Another example is the use of video games by public safety and military organizations to recruit and train soldiers and to treat their psychological disorders by literally improving their coping and cognitive processes.

9. A couple of the more notable improvements we made included two major adjustments: (1) the number of times a participant could take a quiz was limited because some pilot participants had used automatic clicking tools (such as AutoClicker) as a workaround to earn additional points, and (2) a gamemaster role was implemented, as this role can be an important motivational factor for users.

10. We have no further survey data on the employees who opted to not participate. However, as an accepted surrogate test to assess nonresponse bias, we tested to ensure that there was no statistical difference between “early” and “late” respondents. We used time stamps of when they accepted joining the project. We grouped early and late respondents and compared their responses to the Likert-type scale questions using a MANOVA test. The results did not reveal any statistical significance (F = 1.976, p = 0.313).

11. The second step of model validation was to test for discriminant validity. Here, we first considered whether there was any discriminant overlap in the items in the factor analysis, and we consequently dropped two more items that yielded poor discriminant validity. We then examined overall discriminant validity by placing the square root of the reflective construct’s AVE on the diagonal line and the correlations between the constructs below it. The square root value of the AVE should be higher than all latent constructs, which was the case.

12. PEOU = perceived ease of use; PIU = perceived intrinsic usefulness; BI = behavioral intentions to follow security policies; OSC = organization security communication; TMSC = top management security commitment; OCM = organization computer monitoring.

13. As the design is unbalanced, we tested the equality of covariance matrices using Box’s M test. The result was not significant.

14. Ecological validity should not be confused with external validity. Ecological validity indicates the degree to which the findings of a research study can be generalized to real-life settings, often because they are collected or generated in real-life settings (e.g., actual employees trying to solve real work tasks). Although this form of validity — unlike internal and external validity — is not strictly required for a study to be valid, it is a particularly meaningful but often overlooked consideration for research areas that are highly intertwined with practice, such as security and privacy research [cf. 60].

15. To demonstrate these points empirically, we followed Chin et al. [18] The effect of adding our contextualized improvements to HMSAM (step 2 of model building) was calculated as follows [18]: ƒ² (Cohen’s effect size) = R²_{extended model} – R²_HMSAM) (.320)/(1 - R²_{extended model}) (.362). In this case, ƒ² = 0.884, which is a “huge” effect size (anything above 0.35 is considered “large”), is rarely seen in the organizational security literature. To test the statistical significance of this increase, we conducted a pseudo F-test as follows: ƒ² (Cohen’s effect size) * (n – k – 1), where n is the sample size and k is the number of independent variables. In our case, n = 384; and we conservatively set k to 11 for all of the constructs preceding BI. This resulted in F = 328.84, p < 0.001.

16. ƒ² (Cohen’s effect size) = R²_{covariate model} – R²_{extended model)} (.007)/(1 - R²_{extended model}) (.362). In this case, ƒ² = 0.019, which is a “trivial” effect size (“small” requires a size of 0.20 or greater).

17. The model summary statistics between Model 1 (linear) and Model 2 (curvilinear; quadratic) are listed in the following table:

Table

18. Using only the data in the e-mail treatment, the model summary statistics between Model 1 (linear) and Model 2 (curvilinear; quadratic) are listed in the following table:

Table

Additional information

Notes on contributors

Mario Silic

Mario Silic ([email protected]) is a post-doctoral researcher at the Institute of Information Management, University of St. Gallen, Switzerland. He holds a Ph.D. from that university. Dr. Silic’s research focuses on information security, open source software, human-computer interaction and mobile commerce. He has published in Journal of Management Information Systems; Security Journal; Information & Management; Computers & Security; and other journals.

Paul Benjamin Lowry

Paul Benjamin Lowry ([email protected]; corresponding author) is the Suzanne Parker Thornhill Chair Professor and Eminent Scholar in Business Information Technology at the Pamplin College of Business at Virginia Tech. He received his Ph.D. in Management Information Systems from the University of Arizona. His research interests include organizational and behavioral security and privacy; online deviance, online harassment, and computer ethics; human-computer interaction, social media, and gamification; and business analytics, decision sciences, innovation, and supply chains. Dr. Lowry has published over 130 journal articles in Journal of Management Information Systems (JMIS), MIS Quarterly, Information Systems Research, Journal of the AIS, and other journals. He is a member of the Editorial Board of JMIS, department editor at Decision Sciences Journal, and senior or associate editor of several other journals. He has also served multiple times as track co-chair at the International Conference on Information Systems, European Conference on Information Systems, and Pacific Asia Conference on Information Systems.