Toward Usability Testing of Motivational Affordances through Gamification

Abstract

Motivation is fundamental for user engagement in applications. In particular, in educational software systems, gamification is widely used to try to increase student performance and well-being, as it comprises motivational benefits. However, the effectiveness of gamification has sometimes been under question, since a variety of research experiments with the same gamification elements result in conflicting outcomes. Advancement in motivational psychology allows us to relate how perceptions are predictors of well-being and performance. With this purpose, relevant studies have been researched directing the user toward the desired outcomes, and a usability checklist has been developed to assess Motivational Information Systems. We then carried out an experiment from the perception questionnaires and the performance of the students to evaluate a particular gamification system in an educational context. The observed results from the field study have validated the applicability of the checklist, as the predictions were consistent with the literature.

Data availability statement

Data supporting the findings of this study are openly available in “Mendeley Data” at http://doi.org/10.17632/472ppv29jr.1

Disclosure statement

The authors declare that they have no known competing financial interests or personal relationships that could appear to influence the work reported in this document.

Notes

1 “Draco” is an educational gamification system to teach Compiler Design at the Universidad Politécnica de Madrid. Accessed December 31, 2021. https://dlsiis.fi.upm.es/draco/

2 In “Section 2.5.4,” both of two interconnections will be mentioned. However, assessment of competence expectancy as a predictor of Motive dispositions entails multiple measurements at different time-points (since it is a continuous circulation) and increases the cost and complexity of the measurement, making it impractical for widespread usage. Therefore, the usability checklist of motivational affordances will be based on the achievement-goal theory as a predictor of competence expectancy.

3 In certain experiments, the mastery goal orientation predicted intrinsic motivation, but not performance outcomes (Harackiewicz et al., 2008). However, the mastery goal orientation increases interest, engagement, and deep learning strategies (Senko, 2019), and these benefits eventually reflect themselves (directly or indirectly) on academic outcomes (Harackiewicz et al., 2008).

4 In this article, multiple regression analysis has been used since the studies in Section 3 are based on causal relationship among theories and they conduct path analysis. Path analysis (a form of Multiple regression) is the statistical technique that allow to analyse the relationship between single dependent variable and two or more independent variables when theoretical and empirical support exists (Pedhazur, 1997).

5 UMS provides effective results with a small number of survey items (Schönbrodt & Gerstenberg, 2012). The UMS questions are inherently composed of two main categories: “Statements” and “Goals.”

6 According to McClelland (Brunstein, 2019) (McClelland et al., 1989), implicit (unconscious) measurement methods predict “more primitive motivational system derived from affective experiences,” whereas explicit (conscious) methods are based on “more cognitively elaborated constructs.” Implicit methods are applied by professional psychologists and consequently would be a costly approach for a usability measurement method. However, Thrash and Elliot have proved that explicit and implicit measurements of motive dispositions are congruent for predicting achievement goal selection (Thrash & Elliot, 2002). Since our expectation from our method is to perform a regression analysis between motivated dispositions and achievement goal orientation, explicit measurements are feasible for a common HCI usability testing.

7 For consistency with the research results obtained, mastery-avoidance orientation has been excluded in this article. Some of the observed predictions from the literature review (Buechner et al., 2019; Duchesne & Ratelle, 2020; Elliot & Church, 1997; Elliot & Thrash, 2002, 2004; Lee et al., 2010; Levy-Tossman et al., 2007; Sekreter, 2016; Soylu, et al., 2017; Steinmayr & Spinath, 2009; Steinmayr et al., 2019) do not present the mastery-avoidance approach, which is a relatively recent goal orientation that Elliot incorporated into his hierarchical model.

8 IMI is derived from the elements Ryan and Deci created in their theory (Ryan & Deci, 2000) for intrinsic motivation measurements. IMI sources are permitted for academic use and are a preferred survey for most researchers. IMI consists of categories that measure three main components of the intrinsic motivation (perceptions of competence, relatedness, autonomy), and the perception of interest which is common approach to measure overall intrinsic motivation of an activity. Items can be adjusted according to the area of use, providing flexibility.

9 Likert Scale range of UMS is 1-to-5, AGQ-R is 1-to-6 and IMI is 1-to.7. In this paper, these ranges are not unified since guidelines each of the measurement module would not suggest customizing point range (Center for Self-Determination Theory, 2020; Elliot & McGregor, 2001; Schönbrodt & Gerstenberg, 2012).

10 Reliability of the Fear factors in UMS scale measured altogether. In this article, UMS-6 version has been used for the fear factors and its internal consistency (α = 0.79) is even higher compared to study of Schönbrodt and Gerstenberg (α = 0.78) (Schönbrodt & Gerstenberg, 2012).

11 Factorization to be significant, KMO should normally be at least 0.5 (Hair et al., 2014).

12 Bartlett's test for sphericity tests the hypothesis that the correlation matrix amongst the variables is an identity matrix, indicating that they share no common variance. Since the p-value is <0.05, that hypothesis is rejected (Hair et al., 2014).

13 Need for affiliation, hope of success, hope for power and all fear items accumulated together and constructed four factors as in the study of Schönbrodt and Gerstenberg. The total explained variance in this paper (64.87%) is even higher compared original result (55%) (Schönbrodt & Gerstenberg, 2012).

14 Mastery-approach, performance-approach and performance-avoidance items accumulated together and constructed three factors as in the study of Elliot and Church. The total explained variance in this paper (74.87%) is even higher compared original result (63.30%) (Elliot & Church, 1997).

15 Significant predictions obtained through multiple regression analysis in Section 5.3.

16 With the related coefficient value (positive or negative).

Additional information

Notes on contributors

Taygun B. Durmaz

Taygun B. Durmaz received his Master’s degree in software engineering from Universidad Politécnica de Madrid (UPM), Spain. He is currently pursuing his PhD in software systems and computing within the same organization. His main research interests involve Human-Computer Interaction, education, and technology road-mapping.

José L. Fuertes

José L. Fuertes received his PhD in Computer Science in 2003 from the Universidad Politécnica de Madrid (UPM). Fuertes’ major fields of research are software accessibility, software quality, and gamification. He is an Associate Professor (UPM) and has been Head of Department of Computer Languages and Systems and Software Engineering.

Ricardo Imbert

Ricardo Imbert received his PhD in computer science in 2005 from the Universidad Politécnica de Madrid (UPM), Spain. He is a researcher at the Madrid HCI Lab and is an Associate Professor at the UPM. His research interest deals with Human-Computer Interaction, Interactive Systems, User Experience, and Cognitive Architectures.