A learning analytics perspective on educational escape rooms

ABSTRACT

Learning analytics methods have proven useful in providing insights from the increasingly available digital data about students in a variety of learning environments, including serious games. However, such methods have not been applied to the specific context of educational escape rooms and therefore little is known about students' behavior while playing. The present work aims to fill the gap in the existing literature by showcasing the power of learning analytics methods to reveal and represent students' behavior when participating in a computer-supported educational escape room. Specifically, we make use of sequence mining methods to analyze the temporal and sequential aspects of the activities carried out by students during these novel educational games. We further use clustering to identify different player profiles according to the sequential unfolding of students' actions and analyze how these profiles relate to knowledge acquisition. Our results show that students' behavior differed significantly in their use of hints in the escape room and resulted in differences in their knowledge acquisition levels.

KEYWORDS:

• Educational escape rooms
• learning analytics
• serious games
• game-based learning
• computer science education
• sequence mining

1. Introduction

Escape rooms are time-constrained team-based immersive games in which players have to work collaboratively towards accomplishing a specific goal (often escaping from a room) (Nicholson, 2015). Despite being conceived as leisure activities, escape rooms have proven useful in bringing more benefits to the players than mere entertainment, e.g. by leveraging skills such as teamwork, leadership, creative thinking, and communication through play (Pan et al., 2017; Warmelink et al., 2017). Such benefits have made escape rooms appealing to educators who sought to harness the potentials of these novel educational activities to develop soft skills. Furthermore, several teachers have created escape rooms that require students to master field-specific knowledge and skills in order to solve the different puzzles that the activity encompasses. Such escape rooms are often referred to as educational escape rooms and have proven capable of enhancing students' learning through highly engaging experiences.

Recently, a growing attention to educational escape rooms has been kindled by a growing community of researchers and educators who aspire to build on the young generation's enthusiasm towards these popular games. Research on educational escape rooms has been so far limited to surveys examining students' perceptions, and less often, evaluation of knowledge acquisition (Veldkamp et al., 2020). However, little attention has been paid to students' behavior when participating in these novel educational activities. The use of learning analytics methods could help have an in-depth view of students' actions and performance when playing educational escape rooms.

Learning analytics has emerged during the past decade (Valtonen et al., 2022) to take advantage of the increasing availability of the data about learners that digital systems generate. More importantly, a central goal of learning analytics is to offer data-driven support based on the insights derived from the analysis of such data. Recent research in the game-based learning (GBL) literature has exploited learning analytics methods to gain insights from students' gameplay data (Alonso-Fernández, Calvo-Morata, et al., 2019). However, research on learning analytics applied to educational escape rooms is essentially non-existent. This could be explained –at least partially– by the fact that the research field of educational escape rooms is still in its infancy. Another possible explanation is that escape rooms are often physical activities. Collecting data from physical escape rooms could be performed through observation, which would be resource-intensive in escape rooms in which many teams participate at once. Another possibility is to collect players' data through dedicated hardware devices (e.g. sensors and cameras), which can be costly, as well as hard to install and maintain. Although earlier studies have reported basic quantitative analysis (e.g. the completion rate of educational escape rooms), previous research has been limited to basic descriptive statistics, without taking into account the temporal sequence and interplay of the different events that take place during these activities.

The present study aims to fill the gap in the existing literature by exploring the potential of learning analytics methods to reveal students' behavior when participating in a computer-based educational escape room. In particular, we take advantage of the novel developments in sequence mining methods to analyze the temporal and sequential patterns of the actions carried out by students during the activity. We further use clustering to identify different player profiles according to the sequential unfolding of students' actions and analyze their knowledge acquisition. To that end, our research questions are:

(1)	RQ1: How can learning analytics methods help represent and monitor students' actions in educational escape rooms?
(2)	RQ2: Are there distinct players profiles of students participating in educational escape rooms and, if yes, what are they and what are their defining criteria?
(3)	RQ3: What is the relation between students' player profiles and their knowledge acquisition during educational escape rooms?

The article is structured as follows. We first review the existing literature on educational escape rooms as well as on learning analytics (with an emphasis on sequence mining and GBL). We then describe the methods used in the article followed by the results obtained. The last section concludes with the discussion and conclusions of the paper and an outlook on future research.

2. Background

2.1. Educational escape rooms

The growing success behind educational escape rooms lies in their combination of key principles of game design and sound learning theories. One of the main learning theories behind educational escape rooms is active learning, which can be defined as “any instructional method that engages students in the learning process” (Prince, 2004). A fundamental aspect of active learning is that it requires students to do meaningful learning activities and think about what they are doing (Bonwell & Eison, 1991). Educational escape rooms offer an excellent environment for active learning, since students need to figure out how to solve the different puzzles in order to make progress in the activity. By facing the different puzzles, students test their knowledge and develop skills in an active way, especially problem-solving abilities (Huang et al., 2020). Related to active learning, collaborative learning is another central aspect of educational escape rooms. Collaborative learning can be defined as “any instructional method in which students work together in small groups toward a common goal” (Prince, 2004). Educational escape rooms provide a favorable atmosphere for collaborative learning, since students need to work together towards achieving a common goal. In addition, students learn how to use use their time, manage their resources and hone their individual skills in order to succeed in the activity.

Due to the novelty of educational escape rooms as learning activities, existing literature on this topic has initially focused on describing how these activities can be implemented in the classroom in a variety of fields, including healthcare, physics, chemistry, biology, mathematics, and computer science. See Table 1 for a complete list of references. Collecting students' opinions and perceptions of educational escape rooms through questionnaires has been the main method for evaluating the impact of these activities. Such line of research has established substantial evidence of students' high levels of engagement during educational escape rooms. Research has also sought to investigate whether participating in educational escape rooms contributes to students' learning. Most studies measured knowledge through self-reports, while few conducted knowledge tests before and after the activity. The majority of studies agree that students acquire new knowledge and/or skills by playing.

Table 1. Summary of the aspects under study in related work on educational escape rooms.

	Field	Level	Students' opinions	Knowledge assessment	Completion time	Completion rate	Hint statistics
Adams et al. (2018)	Nursing	University/Professionals	✓	Self-reported	✓	✓	✓
Gómez-Urquiza et al. (2019)	Nursing	University	✓	Self-reported	✓	✓
Brown et al. (2019)	Nursing	Baccalaureate	✓	Self-reported		✓
Kinio et al. (2019)	Medicine	University	✓	Self-reported	✓	✓
Guckian et al. (2020)	Medicine	University	✓
Boysen-Osborn et al. (2018)	Medicine	University	✓	Self-reported
Jambhekar et al. (2020)	Medicine	University	✓	Self-reported	✓	✓
Monaghan and Nicholson (2017)	Medicine	University				✓
Cain (2019)	Pharmacy	University	✓		✓	✓
Hermanns et al. (2018)	Pharmacy	University	✓	Self-reported		✓
Clauson et al. (2019)	Pharmacy	University	✓	Pre+Post-test		✓
Caldas et al. (2019)	Pharmacy	University	✓	Pre+Post-test		✓
Eukel et al. (2017)	Pharmacy	University	✓	Pre+Post-test
Cotner et al. (2018)	Pharmacy	University	✓	Pre+Post-test
Eukel et al. (2020)	Pharmacy	University				✓
Huang et al. (2020)	Science	Primary School	✓	Pre+Post-test
Vörös and Sárközi (2017)	Physics	High school	✓	Post-test
Dietrich (2018)	Chemistry	University	✓
Vergne et al. (2020)	Chemistry	University			✓	✓
Ang et al. (2020)	Chemistry	University	✓	Self-reported		✓
de la Flor et al. (2020)	Chemistry	University	✓			✓
Ferreiro-González et al. (2019)	Chemistry	University	✓	Self-reported	✓	✓
Monnot et al. (2020)	Chemistry	University	✓	Self-reported
Brady and Andersen (2019)	Biology	University	✓	Self-reported		✓
Mystakidis et al. (2019)	Biology	High School	✓	Pre+Post-test	✓	✓
Ho (2018)	Maths	University		Self-reported		✓
Jiménez et al. (2020)	Maths	High School	✓	Self-reported
Fuentes-Cabrera et al. (2020)	Maths	High School	✓	Self-reported
Borrego et al. (2017)	Computer Science	University
Gordillo et al. (2020)	Computer Science	University	✓	Pre+Post-test	✓	✓	✓
López-Pernas et al. (2019b)	Computer Science	University	✓			✓
López-Pernas et al. (2019a)	Computer Science	University	✓	Pre+Post-test	✓	✓	✓
López-Pernas, Gordillo, Barra, and Quemada (2021a)	Computer Science	University	✓	Pre+Post-test		✓	✓

Some studies have offered a quantitative view of students' performance in educational escape rooms by reporting the number or percentage of students that could successfully complete the activity, or the time taken to complete it. More fine-grained information on students' progress and use of their available resources (e.g. mean number of hints obtained or time to solve each puzzle) is often omitted in the existing literature with few exceptions (see Table 1). As such, little is known about students' interactions with the different elements of an educational escape room, how students advance, get stuck, ask for help, make use of hints or complete the activities. The timeline of such interactions and how they relate to knowledge acquisition is yet to be explored. Further inspection of students' behavior when participating in these novel educational activities can provide a more nuanced view of their player profile. Specifically, learning analytics methods could be useful in improving the game design and the overall learning process.

2.2. Learning analytics

Learning analytics has emerged over a decade ago aiming at using data generated by learners to understand and optimize teaching, learning, and the environments in which they occur. Significant progress has been achieved across several threads of research, e.g. predictive learning analytics, social network analytics, dispositional learning analytics, sequence and process mining, and multimodal learning analytics (Bergdahl et al., 2020). An emerging thread of research in learning analytics research has focused on examining students' online learning strategies, which can be defined as “any thoughts, behaviors, beliefs, or emotions that facilitate the acquisition, understanding, or later transfer of new knowledge and skills” (Weinstein et al., 2000). A common pattern within research on students' strategies is to explore the temporal nature of students' digital traces using learning analytics methods that make use of the wealth of the time information stored in the data, e.g. sequence mining (Jovanović et al., 2017; López-Pernas & Saqr, 2021; Matcha et al., 2020; Uzir, Gašević, Jovanović, et al., 2020). Sequence mining is particularly concerned with the analysis of time-ordered time-stamped learning actions, i.e. sequences of actions (Agrawal & Srikant, 1995). Sequence mining offers a wealth of methods for the representation, visualization, and analysis of temporal data, which made sequence mining an important tool for studying students' strategies (Romero & Ventura, 2020). Since students have diverse and heterogeneous strategies, clustering is often combined with sequence mining for the discovery of distinct strategies of learning. For instance, Jovanović et al. (2017) studied time-ordered click-stream data from a flipped classroom using sequence mining and clustering to investigate students' different strategies. and found five clusters of strategies: intensive, strategic, highly strategic, selective, and highly selective. Using similar techniques, Uzir, Gašević, Jovanović, et al. (2020) explored students' time management strategies, and were able to reveal the “meaningful and theoretically relevant” time management tactics and strategies. López-Pernas, Saqr, et al. (2021) analyzed students' behavior in a programming course and were able to infer students' strategies as they approached their programming assignments. The following subsection presents related work in which these methods –that take into account the sequence of students' actions– have been used to analyze data collected in game-based learning contexts.

2.3. Game learning analytics

Games have proven to be beneficial for learning in different domains (Boyle et al., 2016; Connolly et al., 2012). The information extracted from the application of analytical methods to data extracted from educational games can both reduce costs and complexity by simplifying game design and development (Alonso-Fernández, Calvo-Morata, et al., 2019). Previous research has made use of learning analytics methods to gain insights from students' gameplay data. For example, the work by Alonso-Fernández, Cano, et al. (2019) showcased how game learning analytics data can be effectively used for different purposes at different stages of the serious games' lifecycle, and specifically to validate the game design, simplify deployment of a game, and facilitate the assessment of learners with games. Similarly, Ruiperez-Valiente et al. (2020) used clustering to identify different engagement profiles in a multiplayer online game according to four dimensions: general activity, social, exploration, and quests. Using such granular data allowed them to identify four distinct engagement profiles: “integrally engaged,” “lone achiever,” “social explorer,” and “non-engaged.” Moreover, Kang et al. (2017) provided an analytical approach to understand students' sequential behavior patterns using in situ gameplay data. Using sequence mining allowed the authors to identify different sets of days as separate problem-solving stages, and to discover frequent patterns within each stage. Similarly, by using sequence and process mining metrics, Gomez et al. (2021) sought to understand the sequence of actions and common errors of students using a three-dimensional geometry game so they can better understand the learning process and conduct personalized interventions when appropriate. This study builds on the previous literature in learning analytics and game-based learning, by applying sequence mining methods to represent, visualize and analyze sequences of students' actions when they participate in an educational escape room. To the best of our knowledge, no study has reported on the use of learning analytics methods to analyze data from escape rooms before, and therefore, the present works offers a case study of interest to educators.

3. Methods

3.1. Context and description of the educational escape room

The present work analyzes an educational escape room carried out in an undergraduate frontend programming course at Universidad Politécnica de Madrid. The escape room was conducted as an optional activity to reinforce the key concepts covered in one of the main blocks of the course, including the basics of HTML, CSS, and JavaScript, and more complex technologies such as React, Redux and React Native. A complete description of an earlier edition of the educational escape room examined in this work can be found at (López-Pernas et al., 2019b).

The narrative of the educational escape room required students to deactivate a bomb by operating a web application with missing features. The educational escape room consisted of a combination of computer-based and physical puzzles arranged in a sequence, requiring students to solve them in a specific order to complete the missing features in the web application, by using the technologies studied in class, and thus achieve the final goal of the escape room (deactivate the bomb). An example of one of the puzzles can be seen in Figure 1. Students were given 1h 45m to accomplish this goal, which was the duration of the activity by design. Table 2 shows the escape room puzzles, the learning objectives they covered, and the game mechanics they included, which were based on the taxonomy presented in Nicholson (2015).

Figure 1. Screenshot of Puzzle 3 on the web application, which required students to program the code to display the different parts of the bomb on the web application and deactivate them in the right order by following one of the clues available to them.

Table 2. Summary of the puzzles of the educational escape room.

	Puzzle goal	Learning objective	Game mechanics
1	Downloading and starting the bomb-deactivating application	– Use GitHub for downloading a code repository	– Noticing something obvious in a room
		– Understand a “package.json” file	– Using something in an unusual way
		– Start a React application	– Fixing something that is broken
2	Accessing the advanced mode	– Invoke JavaScript functions	– Hearing
			– Researching using information sources
3	Displaying and deactivating the bomb parts	– Render lists in React using the map function	– Translating
		– Know the React lifecycle methods	– Searching for objects in images
			– Pattern identification
			– Noticing something obvious in a room
4	Triggering the Cooling Mechanism	– Understand the Flexbox layout mode	– Symbol substitution
		– Know the Redux architecture	– Shape manipulation
		– Review HTML, CSS and JavaScript basics	– Pattern identification
			– Assembly of a physical object
			– Riddle
5	Deactivating a bomb	– Perform an asynchronous request to a REST API	– Communication with an entity outside the room

Students were grouped in self-selected pairs which allowed them to take advantage of collaborative learning and enjoy the benefits of pair-programming (McDowell et al., 2002; Williams & Upchurch, 2001). The activity was carried out in two different shifts, supervised by two teachers, in which multiple teams participated at the same time, amounting to a total of 48 teams (96 students). If students got stuck when trying to solve the different puzzles, they could request a hint (drawn from a predefined set of hints elaborated by the teachers). Instead of handing out hints for free, students had to earn the right to get one by passing a small online quiz covering the theoretical content of the course, which complemented the practical programming skills that the escape room aimed to improve. Students had to answer at least four out of five questions right in order to receive a hint. Students could attempt to solve as many quizzes as they wished during the escape room. Nonetheless, this approach prevented students from continuously asking for help, since earning a hint requires an investment of time, a scarce resource during an escape room.

The Escapp platform (López-Pernas, Gordillo, Barra, & Quemada, 2021b) was used to facilitate the conduction of the educational escape room, handling student registration, team formation, control of the flow of the activity, management of resources, narrative events, gamification elements during the activity, hint management using the quiz approach earlier mentioned, etc. Most importantly, Escapp was used to track students' progress throughout the activity. The web application that students had to complete throughout the escape room made use of Escapp's API (Application Programming Interface) to monitor students' advance throughout the activity, verifying that they arrived to the correct solution of a puzzle before they could proceed to the next. The following subsection describes the data collected throughout the escape room by the Escapp platform.

3.2. Data collection

Data from each of the 48 participating teams' interactions with Escapp during the activity were downloaded from the Escapp platform. All of the participating students gave their consent for using their data for the purpose of the study. For each team, the logs for the following interactions were extracted (Table 3):

Table 3. Logs of students' interactions during the escape room extracted from Escapp.

Type	Description
Puzzle solving k	Students worked on a given puzzle ($k=1-5$) until they solved it
Hint obtained	Students attempted a quiz in order to obtain a hint and succeeded
Hint failed to obtain	Students attempted a quiz in order to obtain a hint and failed

In order to measure the learning gains of students participating in the educational escape room, a pre-test was conducted just before the start of the activity, and a post-test was conducted right afterward. Both tests contained the same ten multiple-choice questions covering the learning objectives addressed in the escape room. To correctly answer the questions in the tests, students had to have a clear understanding of the main concepts covered in the activity, know how to analyze programming code fragments and how to apply the acquired knowledge to solve specific programming problems. The right answers were not revealed to students until after completing the post-test. Moreover, students were not aware that they were going to take a post-test until the end of the activity. Students had ten minutes to solve each of the tests. They were awarded 1 point for each question they answered right and were subtracted 1/(N-1) points for each question they answered wrong, N being the number of options in each multiple-choice question. They were allowed to leave answers blank with no penalty. The maximum score achievable was 10 and the minimum score was 0.

3.3. Data preparation and analysis

The coded log records were anonymized, cleaned and prepared so that each team's time-stamped actions had a start time, end time, and duration. The prepared logs were used to construct a SPELL sequence (Gabadinho et al., 2011). SPELL sequences have been selected as they have a duration with a beginning and an end of each event. This matches the nature of the timed activities of the escape room, which had a linear structure, i.e. puzzles must be solved in order. An example of the sequence built for one of the teams can be seen in Figure 2, in which a team completed Puzzle 1 after 12 minutes, then works on Puzzle 2 from minute 12 to minute 18, then starts working on Puzzle 3, requests a hint at minute 38 and fails to obtain it, etc.

Figure 2. Example of a sequence built from the students' data.

The SPELL sequence data were analyzed using the TraMineR R! package (Gabadinho et al., 2011). The sequence was plotted using a distribution plot, in which the distribution of different events is plotted at each time point. An index plot was also created to represent the events as stacked horizontal bars of color coded events for each team (similar to Figure 2). The index plot is particularly useful in showing the timeline of students' activities and the actions they carried out during the educational escape room with their corresponding duration. Differential sequence mining was used to cluster groups of sequences into homogeneous sequences which represent similar patterns of behavior. The number of clusters was identified based on the resulting dendrogram similar to the methods described in Uzir, Gašević, Matcha, et al. (2020). The substitution matrix was calculated based on the Trate method, while the distances were based on the LCS (Longest Common Subsequence) algorithm which emphasizes the shared length of subsequences, suitable to our purpose of grouping students with similar behavior in the escape room. The clustering was performed using Agglomerative Hierarchical Clustering (AHC) using Ward's method to cluster similar player profiles.

Kruskal-Wallis non-parametric analysis of variance (ANOVA) test was performed (with Holm correction for multiple testing) to compare the identified clusters regarding their hint requests: the proportion of successful attempts to obtain a hint (i.e. when students passed the quiz) and failed ones (i.e. when students failed the quiz). Post hoc pairwise comparisons were performed through Dunn's test to compare the proportion of successful vs. failed attempts among groups. The epsilon-squared (${\epsilon }^2$) was used to measure the magnitude and significance of the effect sizes. According to Cohen's guidelines (Cohen, 1992), a value of ${\epsilon }^2$ under 0.02 represents a very small effect size, $0.02\le {\epsilon }^2<0.13$ represents a small one, $0.13\le {\epsilon }^2<0.26$ represents a medium effect size, and ${\epsilon }^2\ge 0.26$ represents a large effect size.

To determine the learning gains in each of the clusters, we used the Wilcoxon signed-rank test to compare the scores of the pre-test and the post-test. The rank-biserial correlation coefficient ($\hat{r}$) was used as the effect size measure. According to Cohen's guidelines (Cohen, 1977), $0.1\le \hat{r}<0.3$ represents a small effect size, $0.3\le \hat{r}<0.5$ represents a medium effect size, and $\hat{r}\ge 0.5$ represents a large effect size.

4. Results

All the 48 teams participating in the escape room managed to solve the first puzzle; 47 teams (97.92%) solved Puzzle 2 as well; 39 teams (81.25%) solved Puzzle 3; 25 (52%) solved Puzzle 4, and only 10 teams (20.83%) solved the fifth and last puzzle (completed the escape room). Overall, students made 476 attempts to obtain hints, of which 203 were successful with a per team average (M) of 4.51 hints ($\mathrm{SD}=2.46$), and 273 were failed (M=6.83, $\mathrm{SD}=6.71$).

The distribution of sequences (Figure 3-left) shows that half of the teams were able to complete Puzzle 1 as early as 25 min into the escape room. With the passage of time, all teams had solved the first puzzle after 69 min. The times to solve Puzzle 2 and Puzzle 4 were relatively short, whereas most teams struggled with Puzzle 3. A total of 25 teams reached the last puzzle. Only 10 teams managed to solve the fifth puzzle before the time ran out. Hint requests were spread out throughout the duration of the escape room although more frequent towards the middle of the activity.

Figure 3. Sequence distribution plot (left) and index plot (right) of teams' actions during the educational escape room.

The index plot (Figure 3-right) represents each team's activity as color coded stacked bars. The bottom of the plot shows the teams that were mostly struggling with Puzzle 3 while the top of the plot shows teams that finished before time (white) or blue (solving Puzzle 5). The plot also reveals that, even though some teams made numerous attempts to obtain hints, they still devoted more time to the puzzles, which was the main aim of the activity.

Differential sequence mining using clustering has the ability to reveal different behaviors of teams when approaching the puzzles in the escape room. The clustering revealed three distinct player profiles (Figure 4).

Figure 4. Sequence distribution plot (left) and index plot (right) of the actions of each cluster of teams during the educational escape room.

(1)	Accomplished (n=25): Teams in this cluster advanced smoothly –compared to the other two clusters– with relatively short time spent on each of the puzzles as well as a moderate number of hints. Although some teams in this group did get stuck at a certain puzzle (e.g. Puzzle 2 or 4), they seem to have compensated for the delay and got back on track eventually. Most of the teams (n=19) in this group advanced to Puzzle 5 (76% of all teams who advanced to this stage), and 9 of them managed to solve this last puzzle in time (90% of all teams who succeeded in the escape room belong to this group).
(2)	Relentless (n=15): Teams in this cluster had several failed attempts to request a hint (M=11.92, $\mathrm{SD}=9.64$), compared to the successful ones (M=4.33, $\mathrm{SD}=2.53$), which caused them to waste precious time by going back and forth from working on the puzzles to attempting to obtain a hint. They seem to have struggled mostly with Puzzles 1 and 3. None of the teams in this group managed to finish the escape room on time, and only 4 of them reached Puzzle 5.
(3)	Protracted (n=8): The last group of teams had a swift start but mostly became stuck at Puzzle 3. Only one team in this group managed to finish the activity after obtaining three hints to help them with this challenging puzzle. Three teams in this group did not take advantage of hints to help them advance, although the teams who made use of hints did not go beyond Puzzle 3. A single team from this group finished the escape room on time.

To compare the three clusters regarding their hint request strategies, we compared the proportion of successfully obtained hints (i.e. when students passed the quiz) and failed to get hints (i.e. when students failed the quiz). There was a statistically significant medium-sized difference among the three clusters (Figure 5). The pairwise comparison (Dunn's test) showed that the differences in the proportion of successful vs. failed attempts to obtain hints was statistically significant between the Accomplished and Protracted students, and between the Accomplished and Relentless students. In other words, the Accomplished were significantly and remarkably successful in getting hints, while the two other groups had lower success ratios and did not differ from one another in a statistically significant manner.

Figure 5. Violin plot comparing the success in obtaining hints among the three identified groups (for those teams who requested hints).

There has been no statistical significant difference among the three groups regarding the pre-test and post-test scores (see Appendix A). As shown in Figure 6, the Wilcoxon signed-rank test (non-parametric paired t-test) conducted showed that there was a statistically significant difference between the post-test and the pre-test scores with a large effect size in the Accomplished group ($\hat{r}=-0.80$, $p\le 0.001$) and the Relentless group ($\hat{r}=-0.69$, $p\le 0.001$), indicating that both groups have increased their knowledge of the course materials. The Protracted group did not show statistically significant differences in their scores ($p\ge 0.05$).

Figure 6. Violin plot comparing the scores between the pre-test and the post-test for the three identified groups.

5. Discussion

This study aimed at using learning analytics to represent and understand students' activities when playing educational escape rooms and how they relate to performance. The novelty of educational escape rooms with few computer-based instances so far has led to very few guiding examples in learning analytics research. Therefore, our first research question addressed how learning analytics can help represent and monitor students' activities in an educational escape room. We started by adapting the logs collected by the Escapp platform during students' participation, so that they could provide a more nuanced view of students' activities. Therefore, the starting time of each action, duration of work on every puzzle, and time of requesting hints were properly coded, and aligned. While sequence mining is an established method in learning analytics, the sequence representation methods (State Sequence Objects) (Gabadinho et al., 2011) make use of the time-ordered categorical representation of data, i.e. without consideration of the duration of the event (e.g. succession of clicks). Therefore, we had to resort to the SPELL format, which offers an accurate representation of a timed process such as solving puzzles in a sequence without compromising the duration of each activity. Such sequential and temporal representation format has been rarely implemented in the field of learning analytics and, therefore, our case study offers a guiding example for representing learning processes in which both order and duration matter. Two sequence mining visualization techniques were used. The first is the distribution plot, which gives a temporal outline of the time spent on each puzzle or hint. The distribution plot is useful in offering an overview of the time taken by the students to solve each puzzle. In our example, it was easily noticeable from the distribution plot that the third puzzle has been the most time-consuming and probably challenging for most students.

The second plot was the index plot which offered a more detailed view of each team's timeline of actions with color coded stacked bars whose width is proportional to the time taken at each step. The index plot offered an efficient method for monitoring individual students (e.g. Figure 2) so that they can track their own progress, or as a collective visualization (e.g. Figure 3-right) for students to compare their timeline with others and for teachers or administrators who need to monitor a group of students (Figure 3-left). The current implementation of sequence mining as a possible monitoring visualization is novel to the learning analytics community where sequence mining has always been suggested as an analytics method for understanding and classifying students' activities, e.g. Gomez et al. (2021), Kang et al. (2017), López-Pernas, Saqr, et al. (2021), Uzir, Gašević, Jovanović, et al. (2020), Uzir, Gašević, Matcha, et al. (2020), and Matcha et al. (2020).

Our second research question aimed at identifying and understanding students' profiles while playing educational escape rooms. The analysis revealed three distinct profiles. The Accomplished group used a moderate number of hints, were able to finish more puzzles and spent less time solving each puzzle. The Relentless group requested more hints, were less likely to succeed in obtaining them, and took longer to solve the puzzles (none could complete the five puzzles in time). The last cluster in our study was the Protracted group, who spent more time on each puzzle, solved fewer of them, and requested fewer hints than the other groups. Such profiles are different from commonly recognized clusters of students in online contexts, where three groups are commonly identified: an intense cluster who are usually the highest achievers, a moderately active or get-it-done cluster who do just enough to pass a course, and a disengaged cluster who do less than required (Matcha et al., 2020; Uzir, Gašević, Jovanović, et al., 2020; Uzir, Gašević, Matcha, et al., 2020). Two main differences can be observed here: the moderately active group is contrasted by a Relentless group in our study with repetitive attempts to obtain hints and pass the puzzles, while the disengaged group is contrasted by the Protracted group in our study, who are engaged throughout the whole activity but do not make use of the full potential of the available hints. Interestingly, the Relentless group is similar to the determined group described by López-Pernas, Saqr, et al. (2021) in a similar context, i.e. programming assignments. Such findings are indicative of the influence of context on students' profiles that might outweigh the influence of the learning environment, i.e. educational escape rooms vs. automated assessment tools.

Regarding the third research question, which aimed at exploring the relation between students' player profiles and their knowledge acquisition, we found that the Relentless group had the largest pre-post test positive difference of grades. Such findings may point at the idea that the hint approach used in this study (which required students to solve a small quiz in order to obtain a hint) may have a pedagogical value that could be further explored in future research. Both the Relentless and the Protracted groups would require a different type of support. The Relentless group may benefit from guidance of balancing hints with work on puzzles, whereas the Protracted group may need to be encouraged to resort to hints more often. Such personalized support could be provided manually by the instructors supervising the escape room or automatically by, e.g. adding artificial intelligence capabilities to the Escapp platform used in this study (López-Pernas, Gordillo, & Barra, 2021; Schöbel et al., 2021). By providing adequate support to all students, the escape room would be more likely to immerse students in what is known as a “state of flow” (Csikszentmihalyi, 2014), in which they are neither over-challenged nor under-challenged, and become “so involved in an activity that nothing else seems to matter.”

Although previous works examining educational escape rooms have analyzed, e.g. puzzle completion times and number of hints obtained (Adams et al., 2018; Gordillo et al., 2020; Kinio et al., 2019; López-Pernas et al., 2019a; López-Pernas, Gordillo, Barra, & Quemada, 2021a), no previous work, to the best knowledge of the authors, has identified and described students' player profiles and how they relate to their performance in educational escape rooms. Therefore, our findings constitute a unique contribution of interest for the game-based learning community and educators in general. Furthermore, our results show that students can increase their knowledge as a result of participating in educational escape rooms, which adds to the still limited evidence on the learning effectiveness of these novel educational activities.

The results of this study have some implications for educators. First of all, our study offers a data-driven approach to educational escape rooms that makes use of visualization and learning analytics methods. Educators may find some clues on how to record log data from computer-based escape rooms and guidance on the essential elements that enable a nuanced analysis. Another contribution is the implemented sequence technique, which enabled the visualization and monitoring of the timeline of the activities as well as further analysis with clustering techniques to understand different player profiles. The discovered player profiles and the in-depth analysis thereof highlighted the importance of hints, and the relevance of personalizing support according to these profiles.

This study is not without limitations. First of all, the sample of this study is comprised of 48 teams (of pairs) and was conducted in a programming course. Therefore, the generalization of the results remains to be tested across different contexts and educational environments. Future research could explore different types of escape room puzzles with varying difficulty levels. Of special interest would be using learning analytics methods to analyze students' player profiles in open-path educational escape rooms, in which students can solve the puzzles in any order. Since sequence mining methods are limited to sequential events (events cannot take place in parallel), process mining methods could be an alternative worth exploring. Furthermore, our results point to the potential of hints in offering a pedagogical value. Therefore, further research could explore different strategies for hints, e.g. gamification of hints with rewards or penalties. A dashboard for the visualization of students' activities, that shows them insights and recommendations would be interesting to examine the effect on their performance. Lastly, future research could shed light on students' motivation and affect as they are important factors that may influence students' engagement or performance in educational escape rooms. Furthermore, the study of affect in particular, could help understand the challenging nature of the setup, puzzles, or course materials. Triangulating such data with the digital data would help get a better understanding of the students' behavior.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research has been funded by Programa Propio de I+D+i 2021 from Universidad Politécnica de Madrid (UPM) through the call “II Convocatoria de ayudas al personal investigador en formación predoctoral contratados o becados OTT para realizar una estancia de investigación internacional para la obtención de la mencion internacional de doctorado.”

Notes on contributors

Sonsoles López-Pernas

Sonsoles López-Pernas completed her Ph.D. in telematics engineering at Universidad Politécnica de Madrid (UPM), where she also received her Bachelor and Master's degrees in telecommunications engineering. Since 2015, she has worked as a researcher with the Department of Telematics Engineering, UPM. She is currently an Assistant Professor in the Department of Computer Science in the same institution. Her research interests include learning analytics and technology-enhanced learning, with a focus on educational escape rooms.

Mohammed Saqr

Mohammed Saqr has a PhD in learning analytics from the Department of Computer and Systems Sciences, Stockholm University, Sweden. He works on learning analytics and big data in education, network science and scientometrics. His research in learning analytics focuses on social and temporal networks, machine learning, process- and sequence mining as well as temporal processes in general. He is also an active member of several scientific organizations and acts as an academic editor in leading academic publications.

Aldo Gordillo

Aldo Gordillo received the degree in telecommunications engineering and the Ph.D. degree in telematics engineering from the Universidad Politécnica de Madrid (UPM). He is currently an Assistant Professor with the Department of Computer Science, UPM. His research interests include the field of technology-enhanced learning, with a special focus on creation, evaluation, and dissemination of e-learning resources, computer science education, game-based learning, gamification, and e-learning systems.

Enrique Barra

Enrique Barra received the Ph.D. degree in telematics engineering with a minor in multimedia and technology enhanced learning from the Universidad Politécnica de Madrid (UPM). He has participated in many European projects, such as GLOBAL, FIWARE, and C@R. He is currently involved in several projects contributing to the generation and distribution of educational content in TEL environments. His research interests include videoconferencing, games in education, and social networks in education.

Appendix. Difference in pre-test and post-test scores for each cluster

Figure A1. Violin plot comparing the scores of the pre-test among the three identified groups.

Figure A2. Violin plot comparing the scores of the post-test among the three identified groups.