STUDENTS’ RECALLED DESIRABILITY OF USING GAME-BASED STUDENT RESPONSE SYSTEMS (GSRSS): a USER EXPERIENCE (UX) PERSPECTIVE

ABSTRACT

GSRSs are known to positively influence students’ learning behaviors and in-class participation. Kahoot! is one of the GSRSs widely adopted in the higher education (HE) sector and also widely studied in the HE marketing literature. Whilst varied theories have been applied to study the pedagogical value of Kahoot!, the user experience (UX) theory is seldom considered. Taking the first step to address this gap, the present study seeks to explore students’ UX desirability of using Kahoot!, the subsequent behavioral effects, and the moderating effect of students’ orientation to study. The empirical setting includes an online survey administered to 47 marketing students at a regional university in the UK. The findings suggest that: i) students recalled stronger positive desirability than negative desirability, ii) positive desirability has a significant positive effect on perceived usefulness and motivation to attend a class, while negative desirability does not; and iii) motivation to attend is lower for students with the external orientation to study. These insights contribute to the extant literature of GSRSs and UX, and also offer practical implications on how to use GSRSs to motivate students’ in-class learning and attendance.

Introduction

The ownership of digital devices like laptops, smartphones and tablets has become prevalent in today’s society. Planning, organizing and performing daily activities (e.g. meetings, bookings and shopping) on digital devices have become the social norms (Fuentes & Svingstedt, 2017). In the HE sector, students frequently rely upon their smartphones, along with other digital devices, to organize and support their learning, such as checking timetables, recording learning activities, and studying taught content (O’bannon & Thomas, 2014). Accordingly, many digital learning interfaces or tools have been developed to leverage this new trend, i.e. students as active smartphone users, and in turn to (better) facilitate students’ learning in the classroom (Tucker, 2006). Game-based student response systems (GSRSs) represent one type of the digital interfaces that is widely applied in the HE sector and also widely studied in the HE marketing literature (Ebner & Holzinger, 2007; Licorish, Owen, Daniel, & George, 2018). GSRSs broadly refer to interactive classroom systems that temporarily transform a classroom into a game show (Wang, 2015), and Kahoot! is posited as the most popular type of GSRSs (Wang & Tahir, 2020). The focus of the present study is thus on Kahoot! Particularly, it investigates students’ desirability of Kahoot!, focusing on the interplay of positive and negative emotions, and its effect on students’ learning behaviors.

Numerous positive and negative pedagogical values of Kahoot! have been studied (see Wang & Tahir, 2020). Positive pedagogical values include, but are not limited to, students’ engagement, motivation, better concentration, perceived learning improvement, and development of new knowledge (Stoyanova, Tuparova, & Samardzhiev, 2016; Wang, Zhu, & Sætre, 2016). Negative pedagogical values include students’ anxiety and distraction to learning due to unfamiliarity with the digital interface, inappropriate choice of games and time pressure to respond (Aljaloud et al. 2015; Klimova & Kacetl, 2018). Whilst students’ enjoyment has also been considered as a positive pedagogical value, the studies focusing on this research direction are few and far between. They share a common limitation that requires further investigation into this pedagogical value. That is, to what degree will the benefits of a positive emotion, like enjoyment, derived from using Kahoot! be undermined (or not) by the presence of a less positive emotion, like hopelessness? The use of Kahoot! is known to also evoke negative experiences (Wang & Tahir, 2020).

Humans are theorized to experience the existence of both positive and negative emotions within a consumption context. They intertwine to influence humans’ subsequent behaviors (Watson, Clark, & Teilegen, 1988). In our case, this would refer to students’ usage of Kahoot! in the classroom. When considering enjoyment alone, it is unsurprising to learn that the positive emotion enriches students’ learning behaviors, for example, motivation and classroom attendance (Kapp, 2012). What will happen to these positive outcomes, though, when a negative emotion like hopelessness co-exists with enjoyment? Will the positive emotion have the same degree of effect on students’ learning? The consideration of both positive and negative emotions is particularly emphasized in the user experience (UX) field, under the desirability factor (Partala & Kallinen, 2011). It refers to the positive and/or negative feelings that users derive from interacting with a digital interface (Morville, 2004), such as a GSRS in the classroom learning context (Licorish et al., 2018).

To address the mentioned research gap, the present study seeks to explore three related questions: 1) what types of desirability do students recall from their experiences with using GSRSs in the classroom? 2) what effects does the recalled desirability have on students’ subsequent learning behaviors? 3) are the effects of the recalled desirability on students’ learning behaviors moderated by their orientation to study? The exploration is grounded in the desirability factor that is commonly discussed in the UX field (Morville, 2004), and involves an online field survey administered to 47 marketing students, who are also Kahoot! users, enrolled in a British university.

Theoretical Background

Gamification, GSRSs and Kahoot!

Gamification is a method used to apply game elements to non-game contexts (Deterding, Dixon, Khaled, & Nacke, 2011). In educational settings, previous studies show that gamification helps learners to acquire the potential to develop critical thinking and cognitive abilities (Kapp, 2012; Papastergiou, 2009). Other pedagogical benefits include increased engagement and higher motivation of the students to learn and study (Chaiyo & Nokham, 2017; Kapp, 2012; Licorish et al., 2018). Gamification techniques integrated into GSRSs are especially effective to increase students’ attention in lecture-style teaching environments (Graham, 2015). One of the key features of GSRSs is the dynamics and engagement that gamification creates (Wang, 2015). Specifically, game elements such as scores and rankings (i.e. reward systems) are posited to motivate students to achieve intended learning outcomes (e.g. Deterding et al., 2011; Hofacker, De Ruyter, Lurie, Manchanda, & Donaldson, 2016). Kahoot! effectively incorporates some of these elements, thus becoming one of the most widely used GSRSs (Wang & Tahir, 2020).

Previous research on the use of Kahoot! in HE has reported mixed⎯and somewhat conflicted⎯ findings regarding the pedagogical values elicited by this tool (Wang & Tahir, 2020). One segment of studies (Licorish et al., 2018; Stoyanova et al., 2016; Wang et al., 2016) champions the positive pedagogical values of Kahoot! and reveals that it can enhance students’ engagement in-class, such as increased motivation, better concentration, perceived learning improvement, development of new knowledge. This segment also reports that Kahoot! positively affects classroom dynamics (i.e. student-to-student interactions). For instance, Plump and LaRosa (2017) reported that Kahoot! creates an energetic and fun learning environment and in turn activates the interaction, in the form of competition, amongst students. Another segment of studies (Aljaloud et al. 2015; Klimova & Kacetl, 2018) focuses on the negative pedagogical values of GSRSs and have identified the issues of, but not limited to, students’ anxiety and distraction to learning, perceived inappropriateness in the choice of games and time pressure to respond.

Kahoot!, as a digital interface, cannot operate by itself but requires a course of interactions from designated users (Cameron & Bizo, 2019). Users’ perceptions of the GSRS are expected to be shaped by the gamut of emotions experienced during the interaction process (Partala & Kallinen, 2011). Yet, the two fractions of studies mentioned earlier have not explored this fundamental user-oriented issue and thus left a gap in our current knowledge of GSRSs and its relationship with student learning. That is, what emotions can students (the designated user) recall to experience when using Kahoot! in the classroom, and what effects do the recalled emotions have on their subsequent behaviors? The next section discusses the relationship between users’ emotions and resultant perceptions via a UX theoretical lens.

UX and (Emotional) Desirability

UX relates to all aspects of the end-user interaction with a digital interface or system (Norman & Nielsen, n.d.). It is unclear whether UX is classified as a discipline, a theory or a mixture of both, and a detailed discussion on this distinction is beyond the scope of this paper. For the ease of discussion, we refer to UX as a research perspective. UX originates from the human-computer interaction field (Mahlke & Minge, 2008), and has inspired the development of many design⎯and diagnostic⎯frameworks. Amongst them, Morville’s honeycomb emerges as the most widely discussed framework (see Interaction Design Foundation, 2021). It contends that UX goes beyond usability but also includes other factors meaningful to inform the design, and diagnosis, process. They include usefulness, findability, credibility, accessibility, valuability and desirability. We are particularly interested in the desirability factor, which refers to the extent to which a digital interface is “wanted” by users (Interaction Design Foundation, n.d.). That is, users’ emotional responses to a digital interface. Desirability is posited to stem from users’ tastes and design aesthetics (e.g. visual layout and icons). It is suggested to be effective to keep users engaged with a digital interface from the beginning, entice them to interact with the interface and pay attention to the useful and usable features (Interaction Design Foundation, n.d.). Considering the mentioned traits of desirability, one would expect this UX factor to be popular amongst the studies investigating the pedagogical values of GSRSs. Surprisingly, though, the GSRSs literature has hardly considered the desirability factor and its utility to diagnose students’ emotional responses to GSRSs. As discussed in section 2.1, previous studies have consistently devoted attention to the positive and negative pedagogical values of GSRSs including Kahoot!. Tenable explanations may include 1) the lacking suggestions on what and/or how many emotions are relevant to measure the desirability factor, 2) desirability seem more subjective and less noticeable in comparison with other pedagogical values (e.g. learning motivation and classroom dynamics), and 3) desirability is about users “wanting” (i.e. freedom to choose) to use a digital interface but, in a classroom setting, this freedom may not be available to students especially when a GSRS is employed as an assessment tool (Wang & Tahir, 2020).

Research Model and Hypotheses

The present study seeks to explore students’ desirability of Kahoot! based on their recalled usage experiences in the classroom. The UX literature posits desirability as a key factor of promoting (or demoting) users’ interaction with a digital interface. The GSRSs literature hardly considers the relevance of desirability to diagnose students’ (anti-)usage of digital learning interfaces and reasons are unclear. To bridge the gap between the two mentioned literature streams, we opted to measure both positive and negative desirability that students recall from using Kahoot! in the classroom. We operationalized positive desirability with the enjoyment variable (Pekrun, Goetz, Frenzel, Barchfeld, & Perry, 2011) for four related reasons. That is, it is central to 1) a gaming experience, underpins the learning process delivered by a GSRS (Sweetser & Wyeth, 2005), 2) known to influence students’ perceived value like usefulness of GSRSs (Barzilai & Blau, 2014; Schell, 2014), 3) a key driver of educational outcomes like student engagement (Goetz, Hall, Frenzel, & Pekrun, 2006; Skinner & Belmont, 1993) and classroom dynamics (Chaiyo & Nokham, 2017; Plump & LaRosa, 2017), and 4) influential to activate students’ motivation to attend a class (Pekrun, Frenzel, Goetz, & Perry, 2007). Conversely, we operationalized the negative desirability with the hopelessness variable (Pekrun et al., 2011) for three related reasons. It is posited to occur when positive outcome is unattainable whereas negative outcome is inevitable (Pekrun et al., 2007), as an emotion central to students’ learning and achievement (Frenzel, Pekrun, & Goetz, 2007), and to be influential to nudge students’ responses to test-related tasks and academic achievement (Burić & Sorić, 2012). Building on this literature, we investigate the effects of recalled positive and negative desirability on two learning outcomes: perceived usefulness of the quiz and motivation to attend a class. Perceived usefulness refers to the degree to which a person believes that using a specific system would enhance their job performance (Davis, 1989). In this study, it refers to the students’ beliefs of Kahoot as a tool that enhances their learning performance. We also explore the effect of recalled desirability on motivation to attend a class, which represents students’ intentions to join the next class (Pekrun et al., 2011; Pintrich, 1991) The setting of the class (e.g. face-to-face or online) and its related effect on the attendance motivation is beyond the scope of the study. Based on the discussion earlier, we present:

H1: Positive desirability (enjoyment) of Kahoot! has a positive effect on (a) perceived usefulness and (b) motivation to attend a class.

H2: Negative desirability (hopelessness) of Kahoot! has a negative effect on (a) perceived usefulness and (b) motivation to attend a class.

As discussed in section 2.2, users’ desirability of a digital interface linked to personal tastes. In a classroom setting, the personal tastes around (dis)favoring the use of a GSRS can be measured by students’ orientation to study. It comprises both intrinsic and extrinsic dimensions (Pintrich, 1991). The intrinsic orientation to study is about students perceiving a learning task as the reward per se rather than a means to achieve other rewards. The extrinsic orientation to study, on the other hand, is about students perceiving a learning task as a means to achieve other rewards (e.g. social interaction) and the learning itself is supplementary (Wiersma, 1992). Intrinsic orientation has been reported to be positively correlated with student achievement (Gottfried, 1985). Some scholars (Kinman & Kinman, 2001) also note that students with intrinsic orientation to study are likely to perform better than their counterparts with extrinsic orientation to study. This notion is debatable in the context of enriching students’ learning with the use of GSRSs in the classroom.

In light of the positive pedagogical values delivered by Kahoot! (see section 2.1), we surmise that students with extrinsic orientation to study are inclined to recall greater desirability, either positively or negatively, toward using Kahoot! than their counterparts with intrinsic orientation to study. We thus present the following hypotheses:

H3a: Orientation to study significantly moderates the relationship between students’ positive desirability of using Kahoot! and motivation to attend a class, such that the relationship is stronger for students with extrinsic orientation to study.

H3b: Orientation to study significantly moderates the relationship between students’ negative desirability of using Kahoot! and motivation to attend a class, such that the relationship is stronger for students with extrinsic orientation to study.

As stated earlier, previous research shows that intrinsic orientation to study is linked to positive learning outcomes (Gottfried, 1985; Kinman & Kinman, 2001). Academic achievement is found to be higher for students with higher intrinsic orientation to study (Gottfried, 1985). For instance, in the elementary school context, intrinsic orientation is reported to be positively associated with achievement ratings (Lemos & Veríssimo, 2014). However, little is known about the positive effect of the orientation-to-study construct on student achievement in the HE context, where the learning settings are unique (e.g., less regimented timetable and voluntary attendance). In light of these arguments, this study thus proposes that students’ motivation to attend a class will differ by their extrinsic (versus intrinsic) orientation to study beyond the exposure to Kahoot!. Excluding the recalled desirability of Kahoot! from the conceptualization, we hypothesize that students with extrinsic orientation to study are likely to exhibit lower motivation to attend a class than their counterparts with intrinsic orientation to study. This hypothesis can serve as a “controlled condition” and further illuminate students’ recalled desirability of using Kahoot! in the classroom and its relationship with students’ orientation to study. Hence,

H4: Without considering Kahoot!, students with extrinsic motivation to study will show lower motivation to attend a class than their counterparts with intrinsic motivation to study.

Methodology

We adopted a deductive quantitative approach to test the hypotheses developed from the critical review of the literature of GSRSs and UX (Creswell & Creswell, 2017; Saunders, 2011). As depicted in Figure 1, we aimed to measure students’ recalled desirability of using Kahoot! in the classroom and the subsequent effects on their perceived usefulness of the GSRS and motivation to attend a class.

Figure 1. Research model and hypotheses.

Measures, Data Collection and Sampling

We sourced and adapted established scales to ensure the construct validity and reliability of our measurement, that is, positive and negative desirability from the scales of Nabi and Krcmar (2004) and Pekrun et al. (2011), the motivation to attend from the scale by Pekrun et al. (2011), perceived usefulness from the scale of Davis (1989), intrinsic and extrinsic orientations to study from the 1–7 scale by Pintrich (1991).

Table 1 shows the items included in the measurement scales.

Table 1. Measurement factors and statements.

Factors	Scale from	Sample of statements in the scale
Positive desirability (enjoyment)	Nabi and Krcmar (2004) , 5-point Likert scale	“I feel good,” “I enjoy it,” “I feel happy”
Negative desirability (hopelessness)	Pekrun et al. (2011) , 5-point Likert scale	“I feel like giving up,” “I am hopeless” “I feel resigned”
Orientation to study	Pintrich (1991) , 7-point Likert scale	“I prefer course material that really challenges me so I can learn new things” “The most satisfying thing for me is trying to understand the content as thoroughly as possible” “I prefer course material that arouses my curiosity, even if it is difficult to learn”
Perceived usefulness	Davis (1989) , 5-point Likert scale	“How useful was the quiz?” “Did the quiz help you to understand the subject?”
Motivation to attend	Pekrun et al. (2011) , 5-point Likert scale	“If I know I will play Kahoot in the next live session of this module … … I get excited about joining” … I am looking forward to learning a lot” … I am motivated to join”

We then transferred the measurement scales to an online questionnaire (“Online surveys”, 2020) and administered it to university students via e-mail invitation. We defined eligible participants as “any students that are enrolled in a marketing module, have prior usage with Kahoot! in the class and are able to recall the experience without great difficulty.” Eligible participants were self-identified and self-selected via an e-mail invitation (Keiding & Louis, 2018), which was sent out between April and May 2020. Three marketing lecturers who have frequently used Kahoot in their teaching activities supported the authors with the data collection. They were the colleagues of the authors working at the same regional British university. After gaining the favorable ethical opinion (ED182007) from the university, the lecturers sent e-mail invitations to students that had recently participated in a Kahoot! activity in class as part of the module delivery. Students were enrolled in the same marketing course at the university and thus the nature of the Kahoot! activity was the same (i.e. testing students’ understanding of the content explained in the class). Participation was voluntary. Given the exploratory nature of the present study, we set the acceptable sample size to be at 50, based on the three criteria suggested by Muthen and Muthen (2002). They are to determine an appropriate sample size and a power value of 0.80 which is a commonly accepted value for sufficient power. Despite the insightful findings that the sample size has yielded, we recognize its limitations and would address the issue as a direction for future research (see Section 5.3).

After having self-selected themselves for the survey, eligible participants were instructed to recall their prior usage experience with Kahoot! in the class and then complete the questionnaire based on the recalled experience, a technique that is prevalent in UX research and known as user recalled incidents (see Mentis & Gay, 2003).

Data Analysis

Our data analysis strategy involved a sequence of statistical tests, commencing with descriptive statistics, exploratory factor analysis and then concluding with multiple regression analysis (MRA). Serving as the main analysis, MRA was performed to test the developed hypotheses (Cohen, West, & Aiken, 2014), involving positive and negative desirability as the independent variables (IVs), perceived usefulness and motivation to attend as the dependent variables (DVs), and students’ orientation to study as the moderator (see Equations 1(1) $MA=\alpha +{\beta }_{0a}\cdot PD+{\beta }_{1a}\cdot ND+\sum {\beta }_{ia}\cdot C{V}_i+e$(1) –3(3) $MA=\alpha +{\beta }_{0c}\cdot PD+{\beta }_{1c}\cdot ND+{\beta }_{2c}\cdot OT+{\beta }_{3c}OT\cdot ND+{\beta }_{4c}OT\cdot PD+\sum {\beta }_{ic}\cdot C{V}_i+e$(3) ).

(1) $MA=\alpha +{\beta }_{0a}\cdot PD+{\beta }_{1a}\cdot ND+\sum {\beta }_{ia}\cdot C{V}_i+e$(1)

(2) $PU=\alpha +{\beta }_{0b}\cdot PD+{\beta }_{1b}\cdot ND+\sum {\beta }_{ib}\cdot C{V}_i+e$(2)

(3) $MA=\alpha +{\beta }_{0c}\cdot PD+{\beta }_{1c}\cdot ND+{\beta }_{2c}\cdot OT+{\beta }_{3c}OT\cdot ND+{\beta }_{4c}OT\cdot PD+\sum {\beta }_{ic}\cdot C{V}_i+e$(3)

where MA is motivation to attend, PU perceived usefulness, PD positive desirability, ND negative desirability, OT orientation to study, CV represents the control variables set, α and βi are the estimated coefficients, and e is the error term. Finally, some post-hoc analyses were performed.

Results

Response Rate and Socio-demographic Profile of Participants

We invited 188 students enrolled in a marketing course to partake in the project. 51 students responded and completed the online questionnaire. This generated a response rate of 27.12%, which other scholars (Nair & Adams, 2009) have described as an acceptable response rate for online survey. An attention check exercise suggested 4 questionnaires were not completed properly as the completion time was less than two minutes. Accordingly, we excluded these questionnaires and accepted 47 completed questionnaires as the final sample size, and 25.0% as the final response rate.

The majority of the participants were females, aged between 17 and 24 years, studying an undergraduate program full-time and were mainly domestic students. Interpreted differently, this finding suggests male, older and international students were less likely to partake in the project. This somewhat reflects the gender imbalance in the UK higher education (Hewitt 2020). That is, Hewitt (2020, para. 2) reported that the participation levels for females and males in the UK universities in 2017/18 were 56.6% and 44.1% respectively. This conveys the female-to-male ratio in the UK HE sector is roughly 1.3 to 1. Table 2 presents the socio demographic characteristics of the participants.

Table 2. Socio demographic characteristics.

Variable	n	%
Gender
Male	18	38.3
Female	28	59.6
Prefer not to say	1	2.1
Age categories
17–24	30	63.8
Over 24	17	36.2
Highest education
Post-graduate	12	25.5
Under-graduate	35	74.5
Enrollment mode
Full time	40	85.1
Part time	7	14.9
Student status
Domestic student	27	57.4
International student	20	42.6
Total	47	100.0%

Exploratory Factor Analysis (EFA) and Reliability Analysis

We performed EFA to examine the convergent and discriminant validity of the IVs, DVs and moderator under study (Child, 1990). They were positive and negative desirability (IVs), perceived usefulness and motivation to attend (DVs) and students’ orientation to study (moderator). Kaiser-Meyer Olking measure of sample adequacy established the convergent validity of the IVs and DVs where their scores exceeded the suggested benchmark of 0.7 (Hair, Anderson, Babin, & Black, 2010). For the moderator (i.e. orientation to study), however, we had to remove two items to achieve the required convergent validity due to their low individual measure of sampling adequacy values. The IVs, DVs and moderator also achieved satisfactory reliability where their Cronbach’s alpha (α) either exceeded or fulfilled the recommendation of 0.6 (Nunnally, 1994). Table 3 summarizes the EFA and reliability results.

Table 3. EFA and reliability analysis.

Dimensions	Number of items	Cronbach alpha	Aggregated mean score
Positive desirability (1–5)	5	0.873	4.206
Negative desirability (1–5)	5	0.881	1.750
Motivation to attend (1–5)	3	0.912	4.020
Perceived usefulness (1–5)	3	0.850	3.830
Intrinsic orientation (1–7)	3	0.683	5.228
Extrinsic orientation (1–7)	3	0.594	5.790

Effect of Desirability (IV)

We performed two regression analyses to investigate the effect of recalled desirability, inspired by the Morville’s (2004) honeycomb framework, on students’ perceived usefulness of Kahoot! and motivation to attend the class. As presented in Table 4 (Models 1 and 2), the variance inflation factor (VIF) values were less than 6 and thus indicated no existence of multicollinearity issues and that we could proceed with the analysis (Cohen et al., 2014). The adjusted R-square value indicated that the two IVs, notably, positive and negative desirability, accounted for around 40% variance explained in the DVs and the set of control variables. This finding suggests that both positive and negative desirability per se are not adequate to explain the DVs under study and other relevant factors exist. Considering the Morville’s (2004) honeycomb framework also suggests other factors underpinning a good UX, such as findability, accessibility and credibility, co-examining the effects of these factors with desirability on the DVs examined here will enrich our finding and thus serve as an insightful future research direction.

Table 4. Regression results.

	Model 1		Model 2		Model 3
DVs	Motivation to attend		Perceived usefulness		Motivation to attend
IVs	β	Std. coef.	β	Std. coef.	β	Std. coef.
Intercept	0.227		−0.112		0.355
Positive desirability (PD)	0.570	0.577***	0.534	0.547***	0.543	0.550***
Negative desirability (ND)	−0.13	−0.132	−0.068	−0.070	−0.144	−0.146
Orientation	-	-	-	-	−0.466	−0.234*
PD* Orientation	-	-	-	-	0.099	0.069
ND* Orientation	-	-	-	-	0.093	0.071
Gender	0.151	0.075	0.642	0.321**	0.129	0.064
Age	−0.494	−0.238*	−0.258	−0.126	−0.343	−0.166
International	−0.044	−0.022	−0.340	−0.171	−0.098	−0.049
R-squared	0.477		0.462		0.526
Adj. R-squared	0.412		0.395		0.424

*p < 0.1, **p < 0.05, *** p < 0.01.

In terms of predictive validity, positive recalled desirability (enjoyment) has a positive impact on both students’ perceived usefulness and motivation to attend (p < .01). Conversely, whilst negative recalled desirability (hopelessness) showed promising negative effects on the same DVs, they were not significant. Hence, we accepted both H1a and H1b but rejected H2a and H2b. We excluded the variables “Level” and “Student status” from the regression analysis as they were not correlated to any dependent or independent variables.

Effect of Orientation to Study

We tested the moderating effect of orientation to study on motivation to attend through an additional regression analysis. To do so, we treated this factor as a dichotomous variable taking the value 1 to signify extrinsic orientation was higher than intrinsic orientation to study, and the value 0 to represent the opposite. The use of dichotomous moderator variables is widely accepted to test moderator/s in a regression analysis (Aguinis & Pierce, 1998). The adjusted R-square indicated that the two IVs together with the orientation to study and the set of control variables accounted for 52.6% variance explained in the DVs. We have noted earlier that other non-desirability factors exist in the UX literature such as findability, credibility and accessibility (Morville 2004). Besides orientation to study, other personal and situational factors may also exist to moderate students’ recalled desirability of Kahoot! and their subsequent behavior (see future research section).

Regarding H3a and H3b, we were not able to accept the hypotheses as the moderating effects of orientation to study were not significant. That is, the relationship between students’ desirability of using Kahoot! and motivation to attend the class was not regulated by their orientation to study. Regarding H4, however, we established a negative effect of students’ orientation to study on their motivation to attend (p < .1), when the desirability factor was not considered. This finding suggests that students with the extrinsic orientation expressed a lower motivation to attend the class when Kahoot! was not applied in comparison to their peers with the intrinsic orientation. We could thus accept H4. Table 4 summarizes the regression analysis results.

Post-hoc Analyses

As a post-hoc analysis, we also investigated an additional DV, which measured students’ overall likelihood to attend the next learning session of the module (i.e. attendance intention), regardless of whether Kahoot! would be used. The measurement was adapted from Fishbein and Ajzen’s scale (1980), consisting of a single statement that participants rated from 0% to 100% – “How likely is it that you will join the next live session scheduled for this module?.” The regression analysis revealed that positive desirability has no significant effect on students’ likelihood to attend the next class, that is, there are no spillover effects on attendance intention if there is no certainty that the GSRS will be used in future sessions. When considered along with the results in Section 4.3, this additional finding further reveals that students are motivated to attend future sessions only when they know that they will use a GSRS.

To circumvent the limitation of the small sample size and yet to be able to gain further knowledge about the additional DV, we conducted further post-hoc analyses including paired comparisons across student groups based on four personal factors. They were orientation to study, gender, age categories and student status. These comparisons revealed several unique results. First, international students showed a higher level of positive desirability toward Kahoot! compared to domestic students. This result may be attributed to western versus eastern cultural differences in desiring the use of GSRSs in classroom learning (Otten, 2003; Watkins & Biggs, 1996). When language and teaching style may serve as barriers to classroom learning, international students seem to enjoy having a GSRS like Kahoot! to mitigate those barriers and support their learning (Holmes, 2005). Second, age played a significant role in driving students’ desirability and attendance intentions. That is, younger students⎯who were aged under 24 years old⎯expressed a higher level of negative desirability toward using Kahoot! than their older counterparts. However, older students⎯who were aged over 24 years old⎯showed a greater level of intrinsic orientation to study and thus higher motivation to attend a class. Third and final, female students expressed higher perceived usefulness toward using Kahoot! than their male counterparts and a possible reason include the greater social benefits that females perceive from gamification (Koivisto & Hamari, 2014). We found no significant gender difference pertaining to the desirability of using Kahoot!. Table 5 summarizes the results of the two-sided tests assuming equal variances.

Table 5. Paired comparison of mean scores across student groups.

	Orientation to study		Gender		Age		International
	Intrinsic (A)	Extrinsic (B)	Male (A)	Female (B)	Over 24 (A)	Under 24 (B)	Domest. (A)	Internat. (B)
Positive desirability								A
Negative desirability						A
Motivation to attend	B				B
Perceived usefulness				A
Attendance intention metric

For each significant pair, the key of the smaller category appears in the category with the larger mean. Significance level for upper case letters (A, B, C): .05a

Discussion, Implications and Limitations

The present study seeks to explore three related questions: 1) what types of desirability do students recall from their experiences when using GSRSs in the classroom? 2) what effects does the recalled desirability have on students’ subsequent learning behaviors? 3) are the effects of the recalled desirability on students’ learning behaviors moderated by their orientation to study? The exploration is guided by the desirability factor that is widely discussed in the UX field (Morville 2004).

Regarding research question 1, we identify two types of desirability that students recall to experience when using Kahoot! in the classroom. They are positive and negative desirability, and are measured by enjoyment (Nabi & Krcmar, 2004) and hopelessness (Pekrun et al., 2011) respectively. Our work lays the foundation for future research to explore the utility of UX factors, such as desirability, to diagnose students’ experience with a GSRS. Previous studies (Sharples, 2000) have mainly considered the gamification perspective and associated factors like motivation, concentration and class dynamics when diagnosing students’ usage of GSRSs including Kahoot!. No studies, to the best knowledge of the author/s, have considered the UX perspective and the desirability factor even though students’ interaction with GSRSs exemplifies a typical UX context and desirability as an instrumental driver of that context (Norman & Nielsen, n.d.). The desirability factor measures the extent to which a digital (learning) interface is “wanted” by its target users, notably, their positive versus negative emotional responses (Moody & Burtner, 2005). We learn that students are able to recall both positive and negative desirability toward using Kahoot!

Research question 2 is about the effects of the recalled desirability on students’ subsequent behaviors. Our study learns that students prefer positive over negative desirability. More specifically, it shows that positive desirability significantly increases both motivation to attend a class and perceived usefulness of the GSRS while the effects of negative desirability are not significant for both outcomes. A tenable explanation could be the more prominent impact of positive emotions (e.g. enjoyment) on subsequent behavior as portrayed by previous studies (Partovi & Razavi 2019) and supported by the control-value theory of achievement emotions (Pekrun et al., 2007). This finding reinforces the work of other scholars who have reported the significant relationship between GSRS usage and positive achievement emotions amongst students (e.g. Licorish et al., 2018; Pekrun, 1992, 2006; Plump & LaRosa, 2017). For instance, Wang (2015) reported that the use of Kahoot! could intensify students’ motivation, perceived learning and engagement in both one-off and semester-wide. Negative desirability does not negatively affect motivation to attend, which suggests that educators could design challenging GSRSs since negative deactivating emotions seem not to be elicited by these tools.

Research question 3 focuses on the moderating effect of students’ orientation to study on their recalled desirability for GSRSs and subsequent learning behaviors. We learn that the usage of GSRSs in the classroom does not vary across students with extrinsic orientation versus intrinsic motivation to study. This finding establishes the effectiveness of users’ desirability in boosting motivation to attend a class regardless of the differences in students’ orientation to study. However, we found a direct effect of orientation to study on motivation to attend, being those students with intrinsic motivation more willing to attend the next session. Possible explanations include the superiority of intrinsic motivation (versus extrinsic) to enhance cognitive and affective outcomes like curiosity and self-determination which could be linked to students’ motivation to attend (Kinman & Kinman, 2001).

Theoretical Implications

Our work contributes to the GSRSs literature on two main grounds. First, because of its inferential nature, our work presents a new research perspective (UX) and an additional methodology to diagnose the direct and indirect effects of students’ desirability related to GSRSs. Previous studies like Wang (2015) have focused mainly on the long- versus short-term effects of GSRSs on class dynamics while the interplay between independent variables and potential moderators has not been studied yet. Moreover, previous studies on GSRSs (e.g. Chaiyo & Nokham, 2017; Plump & LaRosa, 2017) have commonly relied on qualitative methods or descriptive quantitative work, this work enriches the GSRSs literature with solid quantitative evidence emerging from inferential analyses.

Second, work also extends the UX literature on theoretical and methodological grounds. We identify a boundary condition that influences students’ (recalled) desirability of using GSRSs in the classroom and the resultant effect on their learning behavior. The boundary condition is about students’ intrinsic versus extrinsic orientation to study (Pintrich, 1991). Theoretically, our work illustrates that, when the usage of GSRSs is withheld, students with a higher intrinsic orientation to study will be more motivated to attend a class than their counterparts with a higher extrinsic orientation to study. This distinction does not alter or clarify the relationship of the positive and negative desirability toward using a GSRS in the classroom. Methodologically, we introduce the utility of the user-recalled incidents technique (Mentis & Gay, 2003) to measure students’ usage experience with GSRSs. This UX research technique resonates with the critical incident technique (CIT) widely used in the marketing research literature (e.g. Greenwell, Lee, & Naeger, 2007), but is hardly considered by scholarly work around GSRSs and student learning.

Practical Implications

Our work provides educators and researchers with several practical implications, and one of them is the identification of a dual-dimensional measurement for the desirability factor of UX. It is parsimonious and easy to apply. The two dimensions are emotive in nature and labeled as positive and negative desirability. Whilst the UX literature has widely supported the utility of desirability, along with other factors, to design an engaging digital interface. No studies in the UX literature (e.g. Moody & Burtner, 2005), to the best knowledge of the author/s, have specified the emotional dimensions meaningful to measure desirability. They have merely presented conceptual definitions to clarify the distinction of desirability from other UX factors like usability (Barnum & Palmer, 2010).

Another practical implication is about the necessity for educators and researchers to pay attention to the UX desirability factor. Our work reveals that students’ positive desirability toward using a GSRS in the classroom can lead to greater perceived usefulness of the GSRS and greater motivation to attend a class. UX literature (“Balancing Desirability, Viability, and Feasibility: A Design Review Template”, 2022) suggests that positive desirability can be engineered by: understanding the target users’ explicit versus implicit desires for using a digital interface, designing or diagnosing the interface features that fulfil the desires; and applying or marketing the features to the usage (learning) process.

Third and final, we strongly recommend educators and researchers to embrace gamification, via the use of GSRSs, in the classroom. Consistent with other scholars’ work, (e.g. Four Challenges Facing Higher Education 2018; Carini, Kuh, & Klein, 2006), we learn that gamification can increase students’ in-class engagement and diagnose their in-class performance (Darling-Hammond, 2010). These pedagogical values are achieved by the digital capability of GSRSs to collect instant feedback and stimulate class dynamics (Göksün & Gürsoy, 2019).

Limitations and Further Research

The present study offers novel and UX-focused insights into the utility of GSRSs, especially Kahoot!, to enhance student learning in the classroom. The insights are however exploratory and retrospective in nature and thus should be interpreted with some limitations in mind. First, future research should be mindful of self-selection bias (Bethlehem, 2010) when recruiting eligible participants for a similar project. Our insights might have suffered some degree of self-selection bias, where the students that volunteered themselves to partake in the project might be favorable users of GSRSs and/or having stronger digital literacy. Accordingly, we might not have covered all users of GSRSs from all backgrounds, such as those with lower digital literacy (Prior, Mazanov, Meacheam, Heaslip, & Hanson, 2016) or those that are experiencing vulnerability issues (Prinsloo & Slade, 2016). Additionally, the demographic characteristics of the sample should be carefully considered when interpreting our findings. The majority of the participants were females from younger age groups (17–24 years). Students from different demographic backgrounds, say, older-aged females or males, may show different attitudes toward the usage of Kahoot! and thus yield different results reported in the present study.

Second, our insights are less generalizable to a wider student population as the data collection is based on a small convenience sample and on one British university only. Considering that there are around 2.75 million students (“Number of students in the UK 2021| Statista”, 2022) and around 164 HE institutions in the UK (“Number of universities in the UK | Statista”, 2022), there is bound to be heterogeneity in relation to students’ usage and institutions’ deployment strategies of GSRSs. More research is needed to develop a diagnostic framework of students’ desirability that is applicable to varied types of GSRSs, to varied student backgrounds and for varied HE institutions. This future direction can be achieved by expanding our research design to include: other GSRSs (e.g. Quizizz or Google forms) besides Kahoot! (Licorish et al., 2018); a larger student sample from the same university; and other student samples from other universities. Moreover, the research design could also be expanded to (re)investigate the effect of students’ orientation to study on their desirability for GSRSs and motivation to attend a class. Our work has found very limited effects, but other scholars (Kinman & Kinman, 2001; Wiersma, 1992) have noted that students’ orientation to study can have complex and multifaceted relationships with other learning behaviors and thus a more thorough and triangulated research approach should be considered.

Third, our research design focuses on two emotions (enjoyment and hopelessness). Further studies could extend our model by measuring the utility of other emotions to explain the UX desirability factor, such as surprise, enthusiasm and anxiety (D’Errico, Paciello, & Cerniglia, 2016).

Fourth and final, we have relied on the user recall incidents technique (Mentis & Gay, 2003) to measure students’ desirability of using GSRSs in the classroom. The technique focuses on retrospective experience and some inaccuracy may have occurred with regard to the reported desirability and behaviors (Schwarz, 2014). To address this issue, future research can consider adopting a real-time technique to “more authentically” measure desirability and subsequent behaviors associated with using GSRSs in the classroom. The real-time technique can also measure other moderating factors, such as game styles, game lengths and classroom sizes.

Disclosure Statement

No potential conflict of interest was reported by the author(s).