Commentary on Karhulahti et al. (2022): addressing ontological diversity in gaming disorder measurement from an item-based psychometric perspective

Abstract

The present commentary aims to extend the work conducted by Karhulahti et al. (2022), and more specifically to follow one of the research directions that they suggested but did not preregister, that is, to capitalize on network analysis (an item-based psychometric approach) to reinforce or – in contrast – to nuance the view that the four gaming disorder measurement tools that they scrutinized actually assess ontologically distinct constructs. Thanks to the open science approach endorsed by Karhulahti and colleagues, we were able to perform network analysis that encompassed all items from the four gaming disorder assessment tools used by the authors. Because of the very high density of connections among all available items, the analysis conducted suggests that these instruments are not reliably distinct and that their content strongly overlaps, therefore measuring substantially homogeneous constructs after all. Although not aligned with the main conclusions made by Karhulahti and colleagues, the current exploratory results make sense theoretically and require further elaboration of what is meant by ‘ontological diversity’ in the context of gaming disorder assessment and diagnosis.

Keywords:

• gaming disorder
• assessment
• ICD-11
• DSM-5
• addiction
• network analysis

Karhulahti et al. (2022) are to be complimented for their recent article aiming to address ontological diversity among gaming disorder (GD) measurements. Their research is timely as, despite GD having been recognized as a mental condition and listed in the ICD-11 (see Billieux et al., 2021; Reed et al., 2022), uncertainty still abounds regarding its optimal identification and diagnosis (King et al., 2020). In particular, available epidemiological studies have reported disparate prevalence rates (from 1% to more than 10%-15%; see Karhulahti et al., 2022). This ample variation is mainly because GD research has relied on heterogeneous assessment and sampling strategies, complicating comparisons among studies. The improvement and harmonization of assessment strategies is thus an absolute priority in this field (Carragher et al., 2022; Király et al., 2022).

Karhulahti and colleagues postulated that ontological diversity in GD measurement is likely to account for existing variation in reported prevalence rates, potential (lack of) overlap between measurement instruments, or differential relationships with health-related indicators (e.g., psychopathological symptoms, disability). They capitalized on the new Peer Community in Registered Reports platform to test their research questions through a well-designed and preregistered study conducted within a nationally representative sample. They also shared all their code, data, and other relevant materials via the following Open Science Framework (OSF) repository: https://osf.io/v4cqd/. Through this novel, rigorous, and open approach, Karhulahti and colleagues were able to show that four different GD measurement tools (i.e., GAS7, IGDT10, GDT, and THL1; see the original paper for more details), which the authors consider as having ‘developed from diverse ontological grounds’, produce significantly different prevalence rates and varying degrees of group overlap depending on the tools used¹. From these results, they recommend that researchers ‘clearly define their construct of interest’ in future GD-related empirical studies and meta-analyses. This conclusion aligns with the view that (problematic) technology usage scales too often measure poorly defined constructs (see Davidson et al., 2022, for a critical demonstration and discussion).

Here, we aim to extend the work conducted by Karhulahti and colleagues and more specifically to follow one of the research directions that they suggested (but did not preregister), that is, to ‘chart further ontological differences and similarities between constructs and/or instrument’ by capitalizing on an item-based network model. Thanks to the open science approach endorsed by Karhulahti and colleagues, we were able to perform network analysis that encompassed all items of the four GD assessment tools used by the authors (i.e., GAS7, IGDT10, GDT, and THL1).

Network analysis is a data-driven approach designed to investigate relationships (‘edges’) between variables (‘nodes’). From a psychopathological point of view, this approach no longer considers syndromes or disorders as latent constructs, but rather as networks of symptoms that dynamically interact (Borsboom, 2017). Crucially for our present aim, such analysis can also be used to detect potential communalities among the items incorporated in the network (community detection analysis; see Traag & Bruggeman, 2009).

For example, in a recent study, Baggio et al. (2022) capitalized on network analysis to show that different types of problematic online behaviors (i.e., online gaming, cyberchondria, problematic cybersex, problematic online shopping, problematic use of social networking sites, and problematic online gambling) constitute distinct entities – or constructs – with little overlap. According to Baggio et al. (2022), an implication of this finding is that ‘Internet addiction’ does not constitute a unitary umbrella construct encompassing different problematic online behaviors, and that researchers should instead focus on the specific activities that are performed online. The network depicted in Figure 1A was simulated to illustrate high modularity (i.e., strong within-community relationships and weak between-community relationships), thus implying that the (hypothesized) distinctness between entities is reliable, in line with the network results obtained by Baggio et al. (2022). In contrast, the network depicted in Figure 1B was simulated to illustrate low modularity (i.e., comparable within-community relationships and between-community relationships), thus implying that the (hypothesized) distinctness between entities is unreliable.

Figure 1. Simulated network analyses. The network depicted in Figure 1A was simulated to illustrate reliable distinctness between assumed entities (high modularity, i.e., strong within-community relationships and weak between-community relationships). The network depicted in Figure 1B was simulated to illustrate unreliable distinctness between assumed entities (low modularity, i.e., comparable within-community relationships and between-community relationships). Circles reflect items. Circle coloring reflects community membership. Green lines reflect positive pairwise associations between items, whereas red lines reflect negative pairwise associations between items. Line thickness reflects the magnitude of relationships between objects. The approach used to simulate these two networks and the related code are available from https://osf.io/p4x6t/. 

Against this background, conducting a network analysis on the data provided by Karhulahti and colleagues would be relevant in further demonstrating – and thus reinforcing – the view that the four GD measurement tools that they scrutinized actually assess distinct constructs. If this is the case, the network of GD items should be nearer to Figure 1A than to Figure 1B, with items from a single measurement tool (e.g., IGDT10) potentially clustering into a distinct community of items. Yet, and as displayed in Figure 2, the network analysis that we performed from the data provided by Karhulahti and colleagues instead suggests that the various GD measurement tools are not reliably distinct and that their content strongly overlaps. Because of the very high density of connections among all available GD items, the results of the community detection analysis conducted (see Traag & Bruggeman, 2009) failed to show reliable communities of items (modularity value <0.3; see Clauset et al., 2004), therefore not supporting the multiple distinct constructs account. The complete description of the analysis that we computed and the code to reproduce it are available from the following OSF link: https://osf.io/p4x6t/.

Figure 2. Network analyses. The network depicted in Figure 2 was estimated from the data provided by Karhulahti et al. (2022). The graphical behavior of the estimated network resembles the network depicted in Figure 1B, illustrating unreliable distinctness between assumed entities (low modularity, i.e., comparable within-community relationships and between-community relationships). Circles reflect items. Circle coloring reflects community membership. Green lines reflect positive pairwise associations between items, whereas red lines reflect negative pairwise associations between items. Line thickness reflects the magnitude of relationships between objects. The approach used to estimate this network and the related code are available from https://osf.io/p4x6t/. 

The present results question the view that instruments such as GAS7, IGDT10, GDT, and THL1 measure separate constructs. Although not aligned with the main conclusions made by Karhulahti et al. (2022), the current non-preregistered results make sense and require further elaboration of what is meant by ‘ontological diversity’.

To our view, exploring ontological diversity in GD measurement primarily necessitates taking into account the conceptual and etiological framework that has guided the development of these instruments. Crucially, research on excessive online behaviors has often been criticized for applying a confirmatory approach, which consisted in recycling existing criteria used to define GD, themselves being derived from the criteria used to define substance use disorders (Billieux et al., 2015; van Rooij et al., 2018). This is typically the case for measurement instruments such as the GAS7 or the IGDT10, which are clearly built on common grounds in terms of symptoms covered (e.g., tolerance, withdrawal). This confirmatory approach of assessment and diagnosis has proven problematic, as criteria used to define substance use disorder are not all valid for defining GD (Castro‐Calvo et al., 2021; Charlton & Danforth, 2007; Kardefelt-Winther et al., 2017). The GDT, although similarly anchored within a substance use disorder framework (GD is listed as a “disorder due to addictive behaviours” in ICD-11), follows a more conservative approach and only includes criteria that assess dysregulated or functionally impairing gaming patterns. The IGDT10, GAS7, and GDT thus share an important ontological ground: they all aim to measure an addictive disorder. Far for being non-problematic from a theoretical point of view, this reflects that this field has too often overlooked potential alternative etiological models to account for problematic video-gaming symptoms (Kardefelt-Winther, 2014; Starcevic & Aboujaoude, 2017).

Without a doubt, the heterogeneity in prevalence and group overlap identified by Karhulahti and colleagues is striking and further emphasizes that the assessment and the screening of GD need real improvements and harmonization (see also Carragher et al., 2022). Yet, we advance that the prevalence and overlap-related results reported by Karhulahti and colleagues reflect diversity in scoring methods or diagnostic systems rather than construct-related ontological diversity per se. For example, ICD-11 follows a monothetic approach (all criteria have to be present to endorse GD, which is not the case in the polythetical DSM-based approaches) and does not include overpathologizing criteria such as tolerance (see Castro-Calvo et al., 2021), thus producing a lower and more realistic prevalence rate. Similarly, the suggested scoring method for the GAS7 (criteria endorsed if answer is “sometimes”) is clearly laxer that that adopted for the IGDT10 (criteria endorsed if answer is “often”), thus explaining the huge difference in the prevalence rates observed. When it comes to group overlap, it is crucial to keep in mind that item formulation (or response modality) is likely to account for non-consistent answers between two instruments for the same participant. For example, the “loss of control” criterion refers to “time spent playing” in the IGDT10, but not in the GDT (see Table 1). This makes these two items hardly comparable, and supports the view that the GDT item has better content validity (it covers “loss of control” in a broader way) than the view that these two items measure different constructs.

Table 1. Item formulation differences.

Measurement instrument	Example of criterion	Item	Response modalities
IGDT-10 [based on DSM-5] (Király et al., 2017)	Loss of control (Item 4)	Have you ever unsuccessfully tried to reduce the time spent on gaming?	Never, Sometimes, Often (3 modalities)
GDT-4 [based on ICD-11] (Pontes et al., 2021)	Loss of control (Item 1)	I have had difficulties controlling my gaming activity.	Never, Rarely, Sometimes, Often, Very Often (5 modalities)

The current commentary aimed to complement the findings obtained by Karhulahti and colleagues in their well-designed study that combined the double advantage of fully endorsing open science principles and being conducted in a large and representative sample. From a meta perspective, our data-driven commentary also shows how the endorsement of open science practices promotes cumulative and complementary research efforts on a single high-quality data set.

Funding statement

Loïs Fournier is supported by the Swiss National Science Foundation (SNSF) under a Doc.CH Doctoral Fellowship [Grant ID: P000PS_211887]. The SNSF had no role in the study design, collection, analysis, or interpretation of the data, writing the manuscript, or the decision to submit the paper for publication.

Open Scholarship

This article has earned the Center for Open Science badges for Open Data and Open Materials through Open Practices Disclosure. The data and materials are openly accessible at https://osf.io/p4x6t/ and https://osf.io/p4x6t/. To obtain the author's disclosure form, please contact the Editor.

Disclosure statement

The authors have no conflict of interest to declare.

Notes

1 Other issues and results reported in Karhulahti et al. (2022), such as mischievous responding, are beyond the scope of the current commentary and thus not discussed.