Translation, adaptation and validation of the German version of the Game User Experience Satisfaction Scale (GUESS-GA-18) for adolescents

ABSTRACT

The short version of the Game User Experience Satisfaction Scale (GUESS-18) uses nine factors to assess video game satisfaction. As video games play an important role for children and adolescents, the aim of this study was to translate and adapt the adult English version to an adolescent German version (GUESS-GA-18) and validate the new scale. Translation and adaptation were done in a participatory process with Austrian adolescents in two consecutive focus group sessions. In the validation study with 401 adolescents, fit indices demonstrated good model fit and a confirmatory factor analysis confirmed the nine-factor structure of the new scale. Internal consistency is acceptable (Cronbach’s alpha = .791). Concurrent validity shows that the GUESS-GA-18 measures whether children like the game. The German GUESS-GA-18 for adolescents is a short and comprehensive tool for iterative development of games and studies testing player experience with German-speaking adolescents.

KEYWORDS:

• Video games
• player experience
• usability
• feasibility
• participatory research
• questionnaire

Videogames, computer games and mobile games have become increasingly popular with adolescents in the last decades in both the U.S.A. and Europe (Bottel and Kirschner 2019; Tsitsika et al. 2014). In Germany, there were more than 34 million computer- and videogame players in 2020 and trends show that this number will rise rather than fall. Users become increasingly younger. Accelerated by the increase of mobile-based games (Marchand and Hennig-Thurau 2013), a large proportion of young adolescents now play games in German-speaking countries (Mittmann et al. 2021). To ensure good gaming experience, it is important to be able to measure acceptability, feasibility and usability of games – in game user research also called the player experience. Due to the wide range, anonymity and the possibility of capturing the experiences of many people at the same time in a cost-saving manner (Gangrade 1982), questionnaires are widely used as a valid and reliable tool to evaluate these qualities (e.g. Brockmyer et al. 2009; Inan Nur, Santoso, and Hadi Putra 2021), and a variety of questionnaires for testing player experience exist. Yet, to our knowledge, no comprehensive questionnaire to test player experience exists specifically for adolescents; and there is no validated German questionnaire covering all relevant factors of player experience.

Considering the rising trend of gaming in German-speaking countries and the increasing importance of games for adolescents, this study aimed to translate, adapt and validate a questionnaire for assessing player experience of German-speaking adolescents. For this, we chose the English version of the Game User Experience Satisfaction Scale (GUESS-18; Keebler et al. 2020), for two main reasons: First, the ‘best practice’-development of its long version, the GUESS, which led to a comprehensive tool that covers all relevant factors of player experience; and second, its shortness, which makes it ideal for use with adolescents.

1.1. Related work and the GUESS-18

Commonly used questionnaires to assess player experience of digital games (cited ≥1000 times) include the Immersion Questionnaire (Jennett et al. 2008), the Game Engagement Questionnaire (Brockmyer et al. 2009), the Gameplay Experience Questionnaire (Ermi 2005) and the Player Experience of Need Satisfaction Scale (Ryan, Rigby, and Przybylski 2006). An example for a German questionnaire is the Game Experience Questionnaire (Engl and Nacke 2013). The authors of the Game User Experience Satisfaction Scale (GUESS; Phan, Keebler, and Chaparro 2016) detected several limitations in the existing scales such as that they are only suitable for certain games or genres, important gaming aspects were not considered, items were too difficult to understand/interpret and a lack of psychometric validation. Consequently, they developed a new questionnaire following the ‘best practice’ for scale development and validation: After generating an item pool consisting of the content of 13 existing questionnaires, 15 lists of game heuristics and three user satisfaction questionnaires, an expert panel reviewed the items. Subsequently, a pilot study with gamers took place to make sure that the items were easy to understand. Finally, both an Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) were conducted to validate the scale (Phan, Keebler, and Chaparro 2016).

The GUESS covers nine factors of player experience: Usability/Playability, Narratives, Play Engrossment, Enjoyment, Creative Freedom, Audio Aesthetics, Personal Gratification, Social Connectivity, and Visual Aesthetics. The questionnaire consists of 55 items answered on a 7-point Likert scale. Due to the rather long format of the questionnaire, Keebler et al. (2020) developed and validated a short version of the GUESS, the GUESS-18. This short version still covers all nine factors, but with only 18 items (2 items per factor). Answers range from 1 (strongly disagree) to 7 (strongly agree) on a 7-point Likert scale with a maximum value of 63 and a minimum value of 9 for overall score of video game satisfaction. In a first study about psychometric features, the authors found excellent fit and construct validity for the nine factors. While Keebler et al. (2020) argue that a shorter version will mostly benefit iterative game development, we see another advantage in using this shorter version: Gathering children’s and adolescents’ opinions. As younger people lose motivation when the questionnaire is too long (Borgers, De Leeuw, and Hox 2000), a compressed format can help in keeping them focused and not lose concentration. While the original version of the GUESS has been adapted to Turkish by Berkman, Bostan, and Şenyer (2022), to our knowledge no German version exists. Table 1 gives an overview of some of the existing questionnaires for player experience.

Table 1. Existing questionnaires for player experience.

Name	Dimensions	Items number	Language
Immersion Questionnaire (Jennett et al. 2008)	Temporal dissociation, focused immersion, heightened enjoyment, control, curiosity, emotional involvement, transportation to a different place, attention, control, autonomy	33	English
Game Engagement Questionnaire (Brockmyer et al. 2009)	Absorption, flow, presence, immersion	19	English
Player Experience of Need Satisfaction Scale (Ryan, Rigby, and Przybylski 2006)	In-game-autonomy, in-game competence, intuitive controls, preference for future play, game enjoyment, presence		English
Player Experience Inventory (Abeele et al. 2020)	Ease of control, progress feedback, audiovisual appeal, goals and rules, challenge, mastery, curiosity, immersion, autonomy, meaning	52	English
Game Experience Questionnaire (Engl and Nacke 2013)	Flow, challenge, competence, tension, negative affect, positive affect, sensory and imaginative immersion	33	German
Game User Experience Satisfaction Scale (Phan, Keebler, and Chaparro 2016)	Usability/Playability, Narratives, Play Engrossment, Enjoyment, Creative Freedom, Audio Aesthetics, Personal Gratification, Social Connectivity, Visual Aesthetics	55	English
GUESS-18 (Keebler et al. 2020)	Usability/Playability, Narratives, Play Engrossment, Enjoyment, Creative Freedom, Audio Aesthetics, Personal Gratification, Social Connectivity, Visual Aesthetics	18	English

In summary, the goal of the current study was to create and validate a German version of the GUESS-18 with and for adolescents using a participatory approach (GUESS-German-Adolescents-18; GUESS-GA-18). Therefore, for this paper, we first translated the English version of the GUESS-18 to German and adapted it for adolescent users by using rigorous co-design methods. Then we validated the translated and adapted questionnaire in a sample of German-speaking adolescents.

2. Methods

2.1. Translation and adaptation

An overview of the methods used in this study can be found in Figure 1. The goal of the first part of the study was to translate the original English version of the GUESS-18 to German and then adapt the adult version to be suitable for adolescents, following standard procedure for cross-cultural adaptation of questionnaires (Beaton et al. 2000; Sousa and Rojjanasrirat 2011; Wild et al. 2005) and complying with recommendations for designing or adapting questionnaires for children (e.g. Aydın et al. 2016; Bell 2007). Consent to translate and adapt the GUESS-18 was obtained from the authors of the original questionnaire before the process was started. The process consisted of four steps: Forward translation by two translators, synthesis of the forward translation, expert committee and focus group sessions with early adolescents. For the full process, including changes made at different steps, see Table 2.

Figure 1. Process of translating, adapting and validating the GUESS-GA-18.

Table 2. Process of the translation and adaptation of the items of the GUESS-18 for the use with German-speaking adolescents.

Original item	Synthesis of translations (step 2)	Expert committee (step 3)	Focus groups with early adolescents (step 4)	English translation of final German version	Final German version (step 4)
Usability/Playability
1. I find the controls of the game to be straightforward.	Ich finde die Steuerung im Spiel einfach.	Ich finde, das Spiel ist einfach zu steuern.	„	I find the game easy to control.	Ich finde, das Spiel ist einfach zu steuern.
2. I find the game's interface to be easy to navigate.	Ich finde das Interface im Spiel (z.B. das Menü) einfach zu bedienen.	Ich finde die Menüs im Spiel einfach zu bedienen.	Ich finde das Menü im Spiel einfach zu bedienen.	I find the in-game interface easy to use.	Ich finde das Menü im Spiel einfach zu bedienen.
Narratives
3. I am captivated by the game's story from the beginning.	Die Geschichte des Spiels hat mich von Anfang an gefesselt.	„	Die Geschichte des Spiels hat mich von Anfang an begeistert.	The story of the game got me excited from the start.	Die Geschichte des Spiels hat mich von Anfang an begeistert.
4. I enjoy the fantasy or story provided by the game.	Ich mag die Handlung bzw. Geschichte des Spiels.	„	Ich mag die Geschichte des Spiels.	I like the story of the game.	Ich mag die Geschichte des Spiels.
Play engrossment
5. I feel detached from the outside world while playing the game.	Ich fühle mich von der Außenwelt losgelöst, während ich das Spiel spiele.	Ich fühle mich von der realen Welt (Außenwelt) losgelöst, während ich das Spiel spiele.	Manchmal verliere ich die Zeit aus den Augen, während ich das Spiel spiele.	Sometimes I lose track of time while playing the game.	Manchmal verliere ich die Zeit aus den Augen, während ich das Spiel spiele.
6. I do not care to check events that are happening in the real world during the game.	Ich interessiere mich nicht für Ereignisse, die in der realen Welt passieren, während ich spiele.	Ich vergesse, was in der realen Welt passiert, während ich spiele.	Ich vergesse, was in der Außenwelt passiert, während ich spiele.	I forget what is happening in the outside world while I play.	Ich vergesse, was in der Außenwelt passiert, während ich spiele.
Enjoyment
7. I think the game is fun.	Ich finde, das Spiel macht Spaß.	„	„	I think the game is fun.	Ich finde, das Spiel macht Spaß.
8. I feel bored while playing the game. (REVERSE CODE)	Ich bin gelangweilt, während ich das Spiel spiele.	„		I am bored while playing the game.	Ich bin gelangweilt, während ich das Spiel spiele.
Creative freedom
9. I feel the game allows me to be imaginative.	Ich finde, das Spiel erlaubt es mir, fantasievoll zu sein.	„	Ich finde, das Spiel erlaubt mir, etwas zu verändern.	I think the game allows me to make changes.	Ich finde, das Spiel erlaubt mir, etwas zu verändern.
10. I feel creative while playing the game.	Ich finde, ich kann kreativ sein, während ich das Spiel spiele.	„	„	I think I can be creative while playing the game.	Ich finde, ich kann kreativ sein, während ich das Spiel spiele.
Audio aesthetics
11. I enjoy the sound effects in the game.	Ich mag die Soundeffekte im Spiel.	„	„	I like the sound effects in the game.	Ich mag die Soundeffekte im Spiel.
12. I feel the game’s audio (e.g. sound effects, music) enhances my gaming experience.	Ich finde, der Ton (z.B. Soundeffekte und Musik) verbessert mein Spielerlebnis.	Ich finde, der Ton im Spiel (z.B. Soundeffekt und Musik) verbessert mein Spielerlebnis.	„	I think the sound in the game (e.g. sound effects and music) improves my gaming experience.	Ich finde, der Ton im Spiel (z.B. Soundeffekt und Musik) verbessert mein Spielerlebnis.
Personal gratification
13. I am very focused on my own performance while playing the game.	Ich bin sehr fokussiert auf meine eigene Leistung, während ich das Spiel spiele.	„	Ich bin sehr konzentriert auf meine eigene Leistung, während ich das Spiel spiele.	I am very focused on my own performance while playing the game.	Ich bin sehr konzentriert auf meine eigene Leistung, während ich das Spiel spiele.
14. I want to do as well as possible during the game.	Ich möchte in dem Spiel so gut wie möglich sein.	„	„	I want to do as well as possible during the game	Ich möchte in dem Spiel so gut wie möglich sein.
Social connectivity
15. I find the game supports social interaction (e.g. chat) between players.	Ich finde, das Spiel unterstützt soziale Interaktionen zwischen Spieler*innen.	Ich finde, das Spiel fördert soziale Interaktionen (z.B. reden, chatten) zwischen Spieler*innen.	Ich finde, das Spiel fördert die Kommunikation (z.B. reden, chatten) zwischen Spieler*innen.	I think the game promotes communication (e.g. talking, chatting) between players.	Ich finde, das Spiel fördert die Kommunikation (z.B. reden, chatten) zwischen Spieler*innen.
16. I like to play this game with other players.	Es gefällt mir, das Spiel mit anderen Spieler*innen zu spielen.	„	„	I like playing the game with other players.	Es gefällt mir, das Spiel mit anderen Spieler*innen zu spielen.
Visual aesthetics
17. I enjoy the game's graphics.	Ich mag die Grafik des Spiels.	„	„	I enjoy the game's graphics.	Ich mag die Grafik des Spiels.
18. I think the game is visually appealing.	Ich finde, das Spiel ist visuell ansprechend.	Ich finde die Grafik im Spiel cool/schön.	Ich finde, das Spiel sieht schön aus.	I think the game looks nice.	Ich finde, das Spiel sieht schön aus.

Note: Step – relating to the steps in the method section; „ – no changes to the prior version.

Though backward translation (i.e. translating the German translation back to English and comparing the original version with the back-translated one for similarity) by an independent third translator who does not know the original questionnaire (as a third step before discussing the version with an expert committee) is recommended in some guidelines for the translation of questionnaires (e.g. Beaton et al. 2000), there has also been some discussion about the shortcomings of such. The main argument against backward translation is that a literal translation sticking precisely to the wording of the original items might miss necessary context and comprehensibility components and therefore lead to a reduction in quality rather than an improvement (Behr 2017; Colina et al. 2017). Considering that we not only translated the questionnaire but also adapted it to adolescents’ needs, we decided against backward translation as we expected that some expressions had to be adapted to a more child-friendly language, thus rendering a step that tests for literal translation unnecessary.

2.1.1. Step 1 – forward translation by two translators

In this step, two independent translators created two German versions of the questionnaire. Two native German speakers (GM and IK) who are both fluent in English translated the original GUESS-18 items to German individually and independently.

2.1.2. Step 2 – synthesis with two translators and one moderator

In this step, the two versions of the German questionnaires were synthesised into one version. In a joint meeting, the two original translators and an additional moderator (VZ) compared the two German versions and synthesised it to one version. We followed the coordination process of Koller et al. (2012). This approach considers the original version, comprehensibility, culture, grammar and terminology and emphasises on reaching consensus instead of compromising.

2.1.3. Step 3 – expert committee

In this step, the synthesised version was discussed with experts of the relevant fields and changes were made considering their expert knowledge. The expert committee comprised the two original translators, the moderator, a software developer and gaming expert and a schoolteacher for German language in secondary school. Thus, the questionnaire was further discussed with a focus on content (gaming expert) and German language (teacher).

2.1.4. Step 4 – focus groups with early adolescents

Since the cognitive abilities of children and adolescents are still developing, it is important to adjust the difficulty and length of questionnaires for young people. When answering a question, respondents experience four phases: The question must be understood, relevant facts must be recalled, a judgment must be made and finally, an answer must be selected (Tourangeau 1984). The Optimising-Strategy is applied if the respondent undergoes every phase properly, whereas the alternative Satisficing-Strategy is used if at least one phase is either omitted or poorly performed. Whether the Satisficing-Strategy instead of the Optimising-Strategy is applied depends on the difficulty of the question, the respondent’s motivation and cognitive abilities (Jon 1991). Even though we did not develop a new scale but adapted an existing one for adolescent use, we kept the various guidelines and insights for designing questionnaires for children and adolescents in mind during the focus groups for adapting the GUESS-18 together with early adolescents. These can be found in Table 3.

Table 3. Guidelines for questionnaires for young people.

Guidelines	Reference
Short sentences with simple syntax	Bell (2007)
Direct and specific wording
Questions referring to the current moment or a specific reference period (e.g. ‘in the last 7 days’ rather than ‘recently’)
No ambiguous, hypothetical, negatively worded and suggestive questions
Full labelling for scales
Verbal labelling rather than numeric labelling
No more than three or four response options for younger than 11 years, a maximum of four to five options for 11 years or older
Only offer a ‘don't know’ or missing opinion should as an option if it can actually be a valid answer (e.g. as a result of lack of knowledge)
Cognitive pretesting with the target group	LaPietra, Urban, and Linver (2020)

Cognitive pretesting with the target group is recommended, which is an optimal method when conducting a participatory research approach. The term ‘Participatory Research’ describes a non-hierarchical approach to research (Abma et al. 2019), in which knowledge is produced collectively by and with the people whose lives are the focus of the research (Involve 2019). Through flexible, reflexive and bottom-up strategies, the co-researchers are encouraged to become active and make decisions at different levels of the research process (Unger 2013). This not only improves the quality of research and increases efficiency (Jagosh et al. 2012), but also empowers the people involved (Cahill 2007). In recent years, children and younger adolescents have also been actively involved in participatory research (Bevan Jones et al. 2020; Montreuil et al. 2021) and distance-based methods such as video and telephone-based methods, text-based focus groups or digital diaries were successfully used during the Covid-19 pandemic (Hall, Gaved, and Sargent 2021).

In line with participatory research and public involvement, in step 4, the German version of the questionnaire was presented to a group of early adolescents in two subsequent online focus group sessions. These sessions were approved by the Karl Landsteiner Commission for Scientific Integrity and Ethics (EK Nr: 1025/2020). Participants were recruited via existing networks of the research group. All participating early adolescents and a legal guardian signed informed consent. While the focus was on translation in the prior steps (1–3), in step 4 the focus was on the involvement of early adolescents in the adaptation of the questionnaire items. This was intentional to be able to let early adolescents themselves decide which words/expressions needed rephrasing to be comprehensible for them. Both online sessions were led by two trained focus group leaders and lasted 1.5 h. The first session included six early adolescents (4 male, 2 female; age_mean= 12). The goals/objectives of the focus group session were (1) the adaptation of the items and answer format and (2) to ensure the developmental validity by pretesting the questionnaire. This was achieved by combining the four steps of cognitive pretesting (1. Read the question, 2. Paraphrase the question, 3. Pick the best answer, 4. Explain the answer; Woolley, Bowen, and Bowen 2004); verbal probing (follow-up questions to fully understand participants responses, e.g. ‘Wie kann man denn mit anderen Kindern gemeinsam spielen? An was denkt ihr da?/How can you play with other kids? What are you thinking about?’; Willis 2004) as well as adaptive questions (e.g. ‘Wie würdet ihr die Frage formulieren? Was können wir ändern?/How would you phrase that question? What could we change?’).

For this purpose, one person at a time was asked to read the question aloud and to reproduce/paraphrase the meaning of the question in their own words. In the next step, the other early adolescents were also included and asked how they would interpret the item. For terms that might cause comprehension difficulties (e.g. ‘Interaktionen’), the adolescents were asked about its meaning and – if necessary – alternative child-friendly words. If there were major comprehension problems with items, an age-appropriate explanation was prepared, which could be referred to if necessary. After discussing and sometimes adapting the item, in a final round all early adolescents were asked again if anything needed to be changed to improve understandability. If early adolescents made different suggestions for adaptations, votings or surveys were used to find an agreement or make a final decision.

After the discussion of the first item, different answer formats were presented and consulted about their fit. We offered participating early adolescents different options for scales (4-point, 5-point). We also enquired participants’ response behaviour if the item asked something that might not apply to all games (e.g. what would they answer to the question ‘I enjoy the sound effects in the game’ if the game has no music/sound).

The new version of the items and the answer format were immediately presented to the participants to review their decision. In addition to that, audio of both focus group sessions were transcribed, analysed and reviewed by the two session leaders. Both reviewers wrote individual notes, including observations and comments on participants’ responses to each item. They also concluded summary statements for each item, including observations and issues. Subsequently, they met and presented their findings to each other. These findings and reflections also influenced the discussion of the second focus group.

The second focus group session took place one week later and was led by the same focus group leaders as the first session. Four participants of the first session participated in the second one as well as three new participants (5 male, 2 female; age_mean= 11.83). Including both early adolescents from the first session and naïve early adolescents was in line with participatory research and ensured that the new version of the questionnaire spoke to both the group of adolescents who adapted it as well as ‘new’ adolescents. Early adolescents were presented with the updated version of the questionnaire following the suggestions from the first session. They had the opportunity to discuss and vote on the design of the questionnaire (font, colour, emojis, etc.), fill out the questionnaire and discuss if they were content with the final version.

As a result of these focus groups, changes were made to nine out of the 18 items (see Table 2). Six of these changes were about replacing one word; two times, part of a sentence was changed; and once, the whole item was replaced with a more understandable and clearer item from the particular factor of the item pool of the GUESS-55 (Enjoyment2, see Table 2). Five out of six participating early adolescents preferred the 5-point Likert scale (‘stimme gar nicht zu’, ‘stimme eher nicht zu’, ‘teils/teils’, ‘stimme eher zu’, ‘stimme voll zu’/‘don’t agree at all’, ‘don’t really agree’, ‘undecided’, ‘mostly agree’, ‘totally agree’). Participants’ random response behaviour to items that do not apply to the game (e.g. no sound), which ranged from choosing the lowest to the middle to the highest option on the scale, led to the addition of a fallback option to the scale (‘gibt es in dem Spiel nicht/ not in the game’). All participants endorsed the final version. For better comparability, Table 2 also shows the English translation of the final German version.

2.2. Validation study

2.2.1. Participants and recruitment

For our validation study, we aimed for 300 participants between 10 and 18 following Clark and Watson (2016)’s suggestion for validation sample size. Ethics approval was granted by the Karl Landsteiner Commission for Scientific Integrity and Ethics (EK Nr: 1103/2021). The link to the online questionnaire was distributed online via social media, with a special (but not exclusive) focus on groups with a gaming background, a collaboration with a YouTube influencer, as well as shared with the network established during the research project (e.g. parent groups and school classes). Additionally, the GUESS-GA-18 was filled out by five classes during a series of workshops of the research group where participants played a serious game.

2.2.2. Procedure

Data was collected via an online questionnaire. Participants of the workshops were asked to fill out the questionnaire rating the serious game LINA (Mittmann et al. 2022) that they had played during the workshops. All other (online) participants were first asked for their own and their parents’ consent, then they were asked which game they had played most recently. Participants could either choose one out of seven popular games played by adolescents (Minecraft, Fortnite, Brawl Stars, Grant Theft Auto (GTA), Candy Crush, The Sims, FIFA, which were identified through Tenzer (2022) and through conversations with adolescents) or type in a different game. They were then asked to think of their chosen game when filling out the GUESS-GA-18. The items were presented in a random order. A single question (‘Wie gut gefällt dir das Spiel?/How much did you like the game?’) was used to assess concurrent validity. Subsequently, sociodemographic data was assessed, i.e. age, gender and German language level. If interested, participants could sign up for a voucher raffle at the end.

2.2.3. Analysis

Statistical analyses were conducted using IBM SPSS Statistics 27 and SPSS AMOS. The significance threshold was set at p = .05. Demographic information was described using range, means, standard deviations and for categorial data counts and percentages. Q-Q plots, skewness and kurtosis were used to test for normality. Maximum Likelihood (ML) Estimator was used as an estimation method in the factor analysis. Full information ML (FIML) was used as an estimation method for missing data.

Confirmatory Factor Analysis (CFA) was performed to test if the suggested nine-factor model from the original GUESS-55 and GUESS-18 matched the GUESS-GA-18. Each factor consists of two items (Usability/Playability1, Usability/Playability2, Narratives1, Narratives2, Play Engrossment1, Play Engrossment2, Enjoyment1, Enjoyment2, Creative Freedom1, Creative Freedom2, Audio Aesthetics1, Audio Aesthetics2, Personal Gratification1, Personal Gratification2, Social Connectivity1, Social Connectivity2, Visual Aesthetics1, Visual Aesthetics2). We used chi-square with associated degrees of freedom, Root Mean Square Error of Approximation (RMSEA) and Bollen’s Δ2 to evaluate the model. To further enhance construct validity, three alternative models, suggested by Phan, Keebler, and Chaparro (2016), were tested: eight-factor (Audio and Visual Aesthetics combined as one factor), seven-factor (Narratives and Creative Freedom additionally combined as one factor), and one-factor model. Factor intercorrelation was allowed for all the models.

Spearman-Rho between the mean score of the GUESS-GA-18 and the single question (‘Wie gut gefällt dir das Spiel?/How much did you like the game?’) was used to calculate concurrent validity. To assess the reliability of the overall scale, Cronbach’s Alpha was calculated. Special attention was paid to whether Cronbach’s Alpha increased with the exclusion of the recoded coded item ‘Ich bin gelangweilt, während ich das Spiel spiele/ I am bored while playing the game’. Spearman-Brown coefficient was used to assess the reliability of the subscales, as it is better suited for two-item scales than Cronbach’s alpha (Eisinga, Te Grotenhuis, and Pelzer 2013). Factor loadings, average variance extracted (AVE) and heterotrait-monotrait (HTMT) ratio of correlations were used to test convergent and discriminant validity of the nine-factor model.

3. Results

3.1. Participants

Thirteen cases were excluded from the analysis due to invalid answer patterns (always the lowest, middle or highest score), six cases were excluded due to too little playtime of the rated game (less than 30 min and less than once a week and for less than a month) and one case was excluded due to missing German language skills.

The total remaining sample consisted of N = 401 adolescents (30.2% female; 64.3% male; 4.7% don’t want to say; 0.7% other) between 10 and 18 years (age_mean= 14.58; SD = 2.83). For descriptive demographics, see Table 4.

Table 4. Demographics of the sample for the validation study.

		Sex (N)		Total
		Females	Males
Age (N)	10	10	15	25
	11	29	25	54
	12	12	21	33
	13	10	32	42
	14	5	24	29
	15	4	24	28
	16	2	28	30
	17	7	21	28
	18	34	67	101
Total		113	257	370

3.2. Descriptive analyses of the GUESS-GA-18

Fifty-one different games were rated by the participants (see Table 5). The response minimum of every item of the questionnaire was 1 and the maximum 5, which means that the whole spectrum was used. Means vary between 3.03 and 4.54 (see Table 6), which indicates that the games were rated rather positively. Most items intercorrelated. The Intercorrelation Matrix can be found in the Supplementary Material. The mean time for filling out the GUESS-GA-18 (excluding all other parts of the online questionnaire) was 104 s (M = 104.13; SD = 50.34; seven outliers were excluded due to too high or low z-scores (> 2.68 or < -2.68)).

Table 5. Distribution of games that were chosen by the participants of the validation study.

Game	n (Percentage)	nfemale	nmale
LINA^a	107 (26.7%)	45	58
Fortnite	68 (17.0%)	7	56
GTA	59 (14.7%)	4	53
The Sims	26 (6.5%)	24	2
Minecraft	23 (5.7%)	11	11
Brawl Stars	22 (5.5%)	2	18
FIFA	12 (3%)	2	9
Candy Crush	3 (0.7%)	2	0
Other	81 (20.2%)	25	51

^a Serious game played during workshops.

Table 6. Means and standard deviations of the rating for all items of the GUESS-GA-18.

	Usability/playability1	Usability/playability2	Narratives1	Narratives2	Play Engrossment1	Play Engrossment2	Enjoyment1	Enjoyment2 *	Creative Freedom1	Creative Freedom2	Audio Aesthetics1	Audio Aesthetics2	Personal Gratification1	Personal Gratification2	Social Connectivity1	Social Connectivity2	Visual Aesthetics1	Visual Aesthetics2
Mean	4.13	4.31	4.13	3.98	3.95	3.03	4.54	4.22	3.45	3.71	3.96	3.93	4.03	4.1	3.85	4.47	4.26	4.23
SD	0.96	0.87	1.02	1.15	1.18	1.36	0.77	1.03	1.24	1.18	1.07	1.13	1.06	1.05	1.16	0.89	0.83	0.84

Note: means can vary between 1 and 5; *item recoded.

3.3. Confirmatory factor analysis

3.3.1. Normal distribution and missing data

Q-Q plots revealed that not all items seemed normally distributed, yet skewness was only moderate as skewness and kurtosis of all items were under |2| and |7| respectively (Curran, West, and Finch 1996). FIML was used as an estimation method for missing data. The fallback option ‘gibt es in dem Spiel nicht/not in the game’ was treated as missing data. In total, 4.36% of the items were missing, varying from 0.2% to 19.5% between the items. Little’s MCAR test revealed that the data was not completely missing at random (χ² = 960.342, df = 710, p < .001), yet this is arguably not a problem for the current study. As our fallback option was treated as missing data, it makes sense that missing data was not random: Some games may have no sound or no storyline, but all games can be rated for their enjoyment. In line with that argument, the item with the most missing data was ‘Ich mag die Geschichte des Spiels/I like the story of the game’ and in fact only one of the proposed games in our online questionnaire is what could be called storyline-driven (GTA).

3.3.2. Evaluation of the nine-factor model

The path diagram of the factors can be found in Figure 2. To evaluate the model fit, several fit indices are available. Besides reporting chi-square and the degrees of freedom, most researchers specify two additional fit indices (McDonald and Ho 2002). There is no universally accepted number of fit indices, but it is recommended to report at least chi-square with the associated significance value and degrees of freedom, an incremental fit value (e.g. Tucker-Lewis index (TLI), comparative fit index (CFI) or Bollen's Δ2), and a residual-based measure (e.g. RMSEA or standardised root mean squared error (SMRM); Jackson, Gillaspy, and Purc-Stephenson 2009). Phan, Keebler, and Chaparro (2016) used chi-square including degrees of freedom, RMSEA, SRMR and Hoelter’s critical N (CN) for model evaluation of the GUESS-55, whereas the short version of the GUESS questionnaire used TLI, CFI and RMSEA in addition to chi-square and degrees of freedom. In this study, chi-square with associated degrees of freedom, RMSEA and Bollen's Δ2 were used to evaluate the model. The residual-based measure RMSEA was chosen because of two reasons: First, it is currently the most common measure for model evaluation and is used in almost all CFA-studies (Kenny 2015) and second, it allows a better comparability with the two GUESS studies (Keebler et al. 2020; Phan, Keebler, and Chaparro 2016). For the incremental fit value we used Bollen’s Δ2, as the other commonly used and recommended measures TLI and CFI should not be calculated if the RMSEA value is less than 0.158 (Kenny 2015), which was the case for our data. Table 7 shows the values resulting from the factor analysis, the cut-off-values, and for comparability, the fit values from the other two GUESS studies. The RMSEA value of .041 and Bollen’s Δ2 of .959 suggest a good model fit. Chi-square also shows a significant result (p < .001) and a ratio to degrees of freedom of ≤2, which is equally indicative of the model's acceptability. Thus, the nine-factor model represents an acceptable model for the GUESS-GA-18. It also has comparable indices with the GUESS-18 and the GUESS-55 (see Table 7). For better comparability, additionally to our indices, TLI, CFI and Hoelters’ CN are reported in Table 7.

Figure 2. Intercorrelated nine-factor model of the GUESS-GA-18.

Table 7. Model fit assessment with cut-offs of the GUESS-GA-18, The GUESS-18 and GUESS-55.

Model	GUESS-GA-18	GUESS-18	GUESS-55	Fit recommendations/cut-offs (Schreiber et al. 2006)
χ^2^a	179.077 p < .001	137.015 p < .001	4,428.63 p < .001	Relation to degrees of freedom ≤ 2 or 3
Degrees of freedom^a	108	100	1394
RMSEA^a	.041	.043	.053	<.06–.08
Bollen’s Δ2^a	.959	-	-	≥.95
TLI	.932	.961	-	≥.95
CFI	.957	.974	-	≥.95
Hoelters‘ CN (.05; .01)	298; 325	-	258; 265	>200 (adequate) < 75 (unacceptable)

^a Fit indices used to evaluate the model fit.

3.3.3. Model comparison

In order to find out whether the nine-factor model is the GUESS-GA-18 model that best approximates reality, it was compared with the one-factor, seven-factor and eight-factor models. For this purpose, in addition to the fit indices described above, the expected cross validation index (ECVI) was considered for model comparison. The smaller the value, the better the model fit (Schreiber et al. 2006). Chi-square, the RMSEA value and Bollen's Δ2 are also used for comparison. The same criteria/cut-off values as in Table 7 apply to these indices. The model comparison shows that the assumed nine-factor model shows better results for all values. This means that among the four models tested, the nine-factor model is best at inferring the latent factors from the observed manifest variables (see Table 8).

Table 8. Model comparison of different GUESS versions.

	1-factor	7-factor	8-factor	9-factor
χ^2	χ^2 (135) = 179.077, p < .001	χ^2 (114) = 390.471, p < .001	χ^2 (107) = 288.813, p < .001	χ^2 (108) = 179.077, p < .001
RMSEA	.108	.078	.065	.041
Bollen’s Δ2	.626	.838	.894	.959
ECVI	2.182	1.351	1.132	.853

3.4. Internal consistency

Overall internal consistency showed an acceptable Cronbach’s Alpha α = .791 for the GUESS-GA-18 (Mallery 2003). Despite the negative wording, Cronbach’s Alpha does not change considerably (α = .789) when omitting the item ‘Ich bin gelangweilt, während ich das Spiel spiele/I am bored while playing the game’. From this we can conclude that the participants had no problems with interpreting this item. Omitting any other item did not result in an increase of Cronbach’s Alpha. Spearman-Brown-coefficient ranged between poor (Play Engrossment, Enjoyment, Social Connectivity) to good (Narratives) reliability for the subscales (see Table 9).

Table 9. Internal consistency of the subscales of the GUESS-GA-18.

Subscale	Usability/playability	Narratives	Play Engrossment	Enjoyment	Creative freedom	Audio aesthetics	Personal gratification	Social connectivity	Visual aesthetics
Spearman-Brown-coefficieant	.665	.807	.575	.560	.610	.777	.645	.545	.722

3.5. Concurrent validity

Results revealed a significant correlation between the overall score of the GUESS-GA-18 (moderate correlation with r = .530, p < .001) (Dancey and Reidy 2007) as well as between all subscales and the single item ‘How did you like the game?’, which means that the overall GUESS-GA-18 is able to moderately identify if people like the game (see Table 10). Yet, most correlations of the item with the subscales are only weak to moderate.

Table 10. Correlations between the GUESS-GA-18 and the single question ‘How much did you like the game?’

	Overall scale	Usability/playability	Narratives	Play Engrossment	Enjoyment	Creative freedom	Audio aesthetics	Personal gratification	Social connectivity	Visual aesthetics
Spearman-Rho	.53**	.25**	.33**	.26**	.51**	.30**	.32**	.32**	.20**	.45**

Note: **. significance level 0.01 (2-sided).

3.6. Convergent and discriminant validity of the nine-factor model

We used AVE and standardised factor loadings to examine convergent validity. If AVE is above 0.5 and factor loadings are above 0.5, convergent validity can be concluded (Cheung and Wang 2017; Hair, Ringle, and Sarstedt 2011). AVE for the GUESS-GA-18 is 0.513 and all factor loadings of the GUESS-GA-18 are above 0.5, with 15 out of 18 factors loadings above .6 (see Figure 2). We tested discriminant validity with the HTMT ratio of correlations. To assess discriminant validity, HTMT values are compared to an absolute threshold, which can be either 0.85 or 0.9 (Henseler, Ringle, and Sarstedt 2015). If HTMT values are below this threshold, discriminant validity can be concluded. Results are shown in Table 11. All our HTMT values are below 0.85.

Table 11. HTMT values of the subscales of the GUESS-GA-18.

	Usability/playability	Narratives	Play Engrossment	Enjoyment	Creative freedom	Audio aesthetics	Personal gratification	Social connectivity	Visual aesthetics
Usability/playability
Narratives	0.17
Play Engrossment	0.23	0.37
Enjoyment	0.30	0.49	0.34
Creative freedom	0.36	0.39	0.47	0.48
Audio aesthetics	0.50	0.43	0.25	0.46	0.34
Personal gratification	0.41	0.17	0.43	0.50	0.42	0.29
Social connectivity	0.17	0.36	0.20	0.47	0.18	0.28	0.36
Visual aesthetics	0.39	0.52	0.35	0.74	0.56	0.55	0.42	0.32

4. Discussion

The aim of the current study was to translate the GUESS-18 to German and adapt it for use with adolescents by applying a participatory research approach. We further analysed psychometric properties and tested the factor structure in a sample of 401 German-speaking adolescents between 10 and 18. In accordance with findings from Phan, Keebler, and Chaparro (2016) and Keebler et al. (2020), our results support the nine-factor structure in the GUESS-GA-18.

All of our model fit indices (chi-square with associated degrees of freedom, RMSEA and Bollen's Δ2) suggest a good model fit and comparison with three other model structures revealed that the nine-factor-model has in fact the best model fit.

Correlations between items and even factors were to be expected as the questionnaire sums up the user satisfaction of various parts of a game. As we asked participants to rate a game they play a lot, it is highly likely that participants liked the game, consequently rating most of the items highly. For example, it makes sense that the factor Enjoyment is high when other factors are also high. In line with that, our results show that the serious game that participants were asked to play during a school workshop was rated lower than the games that they could choose themselves.

Our reliability results show an acceptable Cronbach’s Alpha for the overall scale, but also that some factors have poor internal consistency (Spearman-Brown-coefficient). This might be influenced by the fact that our factors consisted of only two items each (Eisinga, Te Grotenhuis, and Pelzer 2013). As the aim of the shortened version of the questionnaire is to be a tool to be used with adolescents, too many items might make adolescents lose motivation to fill out the questionnaire. It makes sense to have a minimum of items to ensure that adolescents undergo the Optimising – rather than the Satisficing-Strategy (see Methods/Focus groups with early adolescents) and thus ensure ecological validity.

Our data was not missing at random due to our fallback option. For example, missing data could be found for items concerning the narrative (factor Narratives) of the game. As most of the rated games were not narrative-driven, it makes sense that this factor had missing data. Narratives1 was changed during the adaptation process from ‘I enjoy the fantasy or story provided by the game’ to ‘I like the story of the game’, thus omitting the word ‘fantasy’. Arguably, ‘fantasy’ in the English language covers more than just story. For example, in FIFA, you have the fantasy of being a football player even though there is no continuous story line. There are three reasons for still omitting that word. First, it is not easily translated into German and the matter was thoroughly discussed during focus groups with early adolescents, which made clear that they do not acknowledge the difference between the two words. Second, the aspect of ‘fantasy’ and immersion is additionally partly covered by the factor Engrossment. And third, our data shows that GTA was not the only game getting ratings for Narratives, so adolescents still feel the ‘fantasy’ of the game.

Some items were changed in the course of adapting the questionnaire with adolescents and notably, one item was replaced with a different item of that factor from the original GUESS. Hence, the GUESS-GA-18 is not equivalent to the GUESS-18 anymore. Furthermore, the change of the 7-point answer format to a 5-point format might have decreased sensitivity of the measurement. Yet, literature backs the results of our participatory research approach in having less options for young people (Bell 2007) and rigorous validation shows the acceptable psychometric qualities of the new version of the questionnaire.

The increase in popularity of digital games has also led to a growing interest from scientists in that topic: Not only are researchers interested in gaming behaviour and possible effects of gaming (Blumberg and Fisch 2013), games are also increasingly used as a means for teaching various educational goals. The definition of so-called serious games is a game that surpasses mere entertainment purposes (Susi, Johannesson, and Backlund 2007). The main advantage of serious games is that they engage players with educational topics in a fun and entertaining way, thus motivating the player to engage and learn (Hamari, Koivisto, and Sarsa 2014; Subhash and Cudney 2018). Existing serious games often lack rigorous evaluation (Ke 2011). The main part of evaluating a serious game is testing the effectiveness of the game in teaching the intended educational content. Yet, as serious games rely on the players’ motivation to engage with the educational content, player experience is also a vital outcome when evaluating serious games (De Lima, de Lima Salgado, and Freire 2015). One game in our validation study was a serious game, and our results show that the questionnaire is a good tool for both general game user research and specifically for evaluating serious games.

4.1. Strengths, limitations and future work

The major strength of this study lies in the rigorous, theory-based and participatory approach and the inclusion of the target group in the adaptation of the questionnaire. Yet, the sample for the adaptation of the questionnaire consisted only of a limited number of early adolescents. Early adolescence is the bridge between childhood and adolescence. We argue that the questionnaire is also suitable for all adolescents as they will have no problems understanding the items.

For the validation study, we transparently report methods and results. The validation study included the whole age range of adolescence, and the sample size was even higher than recommended and therefore sufficient for our analysis. Yet, we faced considerable challenges recruiting 300 adolescents. First, this was due to the Covid-19 pandemic when schools were no option for recruitment due to ongoing lockdowns and different priorities of school personnel. Consequently, our strategies relied heavily on online recruitment, which led to two limitations: First, the majority of our sample was male. This was due to the fact that our recruitment strategies targeted mainly the gaming community (Facebook groups of the relevant games, YouTube Channels for gamers) and suggests that male players still dominate these gaming groups, even though we tried to communicate that we did not only search for gamers who played video games and computer games but also smartphone games (which have a less strong sex divide; Leonhardt and Overå 2021; Rodriguez-Barcenilla and Ortega-Mohedano 2022), as well as actively posted in groups that targeted female players. Second, 35% of our sample was 18 years old. This might have been due to the fact that social media groups on Facebook/YouTube attract older adolescents, probably because younger adolescents are not allowed to use social media apps in general and ‘newer’ (and hence more attractive to younger adolescents) social media apps such as TikTok or Instagram do not have the option to share a link for a questionnaire and could therefore not be used for recruitment. Studies with a video game context should have a clear strategy on recruitment, especially when targeting children and adolescents and a strategy how to recruit female participants. In line with that, we found that there were sex differences for the different games of our study, with more male adolescents playing games such as Fortnite and GTA, and more females playing The Sims and Candy Crush. Minecraft was the only one of our main games that was played almost equally by male and female participants.

In terms of statistical analysis, we only used a single question instead of a related questionnaire to assess construct validity, which means that this is only an initial value and should be interpreted with care. Even though the single question ‘How did you like the game?’ has a significant correlation with all subscales, correlations are weak for most of them, which underlines the argument that player experience is a multi-faceted construct that requires more than just one question to measure (Federoff 2002). Our question mostly related to the Enjoyment factor of the questionnaire. A recent study adapting the original GUESS to Turkish (Berkman, Bostan, and Şenyer 2022) found that, rather than a nine-factor structure, their questionnaire had a 10-factor structure, where Usability/Playability are divided into two different factors. We did not test for a 10-factor structure as our data collection for the adaptation and translation started before their article was published. As we believe that both items of the short version of the GUESS would belong in the Usability factor (see Berkman, Bostan, and Şenyer 2022 for definition), our questionnaire could not be tested for the 10-factor structure. Future research could test if the short version of the GUESS is missing the factor Playability entirely. Moreover, it might be advantageous to pre-test the questionnaire with older adolescents; test re-test reliability; examine sex differences further (e.g. sex differences for each game); and test validity with other measures.

The translated English items could be used to validate the English version of the GUESS-18 for adolescents. It might also be advantageous to have an adult German version of the GUESS, as games are not only played by adolescents. For this, translated items of the translation process before adapting them with adolescents might be used (see Table 2).

4.2. Practical implications

All items for the GUESS-GA-18 can be found in the last column of Table 2. Ideally, they should be presented in random order. Our results of the focus groups show that it is advantageous to add a fallback option to the questionnaire for items that cover factors irrelevant to the specific game. We termed this fallback option ‘gibt es in dem Spiel nicht/ not in the game’. In line with Keebler et al. (2020), item 8 needs to be reverse coded. For scoring, ratings of each subscale are averaged for a subscale score. Summing up subscales will create a total player experience score with a minimum of 9 and a maximum of 45. On average, it takes less than 2 min for adolescents to fill out the GUESS-GA-18.

5. Conclusion

In conclusion, the GUESS-GA-18 proves to be a satisfactory, valid and acceptable tool to test player experience in adolescents. It is short and easy-to-use, and our study shows support for good psychometric properties and original factor structure. Thus, the GUESS-GA-18 provides a new tool for testing general player experience with a target group of German-speaking adolescents. It might be used for both game design and evaluations of games, and it might be a useful tool for researchers developing serious games.

Acknowledgements

We thank Lina Kröncke for leading the adaptation workshops; the YouTube account Saliival for helping us recruit adolescents; Michael Weber, Uwe Graichen and Sascha Klee for their statistical consultation; and Mikki Phan for giving us permission to use the original GUESS-18. We acknowledge support by Open Access Publishing Fund of Karl Landsteiner University of Health Sciences, Krems, Austria.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Ludwig Boltzmann Gesellschaft and the Karl Landsteiner Universität für Gesundheitswissenschaften.