Preliminary psychometric validation of a brief screening tool for athlete mental health among male elite athletes: the Athlete Psychological Strain Questionnaire

Abstract

There is growing interest in the area of elite athlete mental health, however brief, valid and reliable screening instruments designed to identify early indictors of athlete-specific distress and potential mental health symptoms are lacking. This study sought to develop a brief screening instrument for athlete populations – the Athlete Psychological Strain Questionnaire (APSQ) – and examine convergent, divergent and construct validity. A two-stage psychometric validation study was undertaken. Self-report data was collected from 1,007 currently competing Australian elite male athletes (M = 23.67, SD = 4.16). The sample was randomly partitioned into calibration (n = 497) and validation (n = 510) samples. Exploratory and confirmatory factor analysis, and tests of differential item functioning were conducted. Exploratory factor analysis, with parallel analysis, conducted on the calibration sample supported a three-factor solution, with subscales assessing Self-Regulation, Performance and External Coping accounting for 50.44% of total scale variance. Confirmatory factor analysis supported this three-factor model, yielding excellent model fit indices (CFI = 0.976, TLI = 0.966, RMSEA = 0.055, SRMR = 0.032), with the Bayesian Information Criterion supporting the superordinate structure. Differential item functioning analysis indicated item-equivalence relative to athletes’ level of education and ethnicity. As predicted, a multivariate effect indicated higher APSQ scores for currently injured athletes (p = .040) with a univariate effect on the Performance subscale. The APSQ may help identify early symptoms of athlete psychological strain facilitating timely management. Replication and validation studies in broader samples, including female athletes and comparison with other athlete-specific, coping and stress measures are needed.

Keywords:

• athletes
• psychological strain
• mental health
• performance
• coping
• assessment t

Introduction

While there is a large body of evidence regarding the assessment and management of physical injuries for elite athletes, there is comparatively less research into the mental health and psychological wellbeing of this population (Baron, Reardon, & Baron, 2013). Elite professional sportspeople are exposed to a wide array of stressors that make them potentially susceptible to psychological distress (Gorczynski, Coyle, & Gibson, 2017; Moesch et al., 2018), and there is a need for targeted research into how impaired wellbeing in elite athletes may manifest. Intense physical activity performed at the elite athlete level may compromise aspects of mental wellbeing (Peluso & Andrade, 2005), and when combined with other stressors, including injury, performance, selection or retirement/transition out of sport concerns, it is likely that elite athletes will experience fluctuating periods of increased vulnerability to mental ill-health or impeded wellbeing (Hughes & Leavey, 2012).

In order for professional sporting bodies to provide optimal support to their athletes and playing groups, and maintain performance excellence and cultures of achievement, attention must be given to athlete mental health (Taylor, Chapman, Cronin, Newton, & Gill, 2012). Interest in the domain of elite athlete mental health is reflected in recent consensus statements outlining mental health symptom identification and management in sports-specific and elite environments (Henriksen et al., 2019; Moesch et al., 2018; Reardon et al., 2019). There is also growing interest in the development of athlete-specific screening and identification processes for the early identification of mental health problems in this population (Donohue et al., 2019; Hussey, Donohue, Barchard, & Allen, 2019). While a number of athlete mental health-related measures currently exist, they are relatively lengthy. For example, the Recovery Stress Questionnaire (Kellmann & Kallus, 2001) (RESTQ-Sport) was developed as a 77-item scale, though recent research has validated a shortened 36-item version (Kallus & Kellmann, 2016). Similarly, the Profile of Mood States (POMS) was originally 65 items (McNair, 1971), and has been abbreviated to 40-item (Grove & Prapavessis, 1992) and 24-item versions (Terry, Lane, Lane, & Keohane, 1999). The POMS in particular uses an adjective-checklist, and there is conjecture in the field regarding the psychometric validity of this measurement approach (Sliter & Zickar, 2014). While the RESTQ-Sport and POMS have significantly furthered research in the field (Beedie, Terry, & Lane, 2000), their length has prohibited regular (i.e. weekly) monitoring or screening use in the sports environment (Taylor et al., 2012). Furthermore, much of the research undertaken to date on elite athlete mental health, including psychometric validation studies, has been with non-representative samples (Rice et al., 2016). Hence, selection bias and low study response rates (often <40%) limit the conclusions that can be drawn.

Salient to the development of mental health symptoms in athletes is the concept of psychological strain, characterised by a combination of perceived stress and difficulty coping. Psychological strain can be characterised on a continuum of emotional exhaustion and reaction to stressful experiences (Boswell, Olson-Buchanan, & LePine, 2004), and is likely to occur in situations where an individual believes they do not have sufficient resources to cope (King & Beehr, 2017). Where coping resources are extended beyond an athlete’s capacity, stress-related symptoms of psychological strain may emerge (Raedeke & Smith, 2004). Strain theory articulates three contributors to psychological strain, namely, failure to achieve positively valued goals, the removal of positively valanced stimuli, and presentation with negatively valanced stimuli – each of which produce a range of negative emotions (e.g. anger, frustration, hopelessness) that the individual can only manage by employing effective coping mechanisms (Agnew, 1992). Data from elite athletes experiencing distress highlights a tendency for some to conceal difficulties from coaching staff and teammates (Doherty, Hannigan, & Campbell, 2016). Additionally, elite athletes who engage in externalising coping strategies, such as problematic alcohol use (Gouttebarge et al., 2018; O’brien, Blackie, & Hunter, 2005; Schuring, Kerkhoffs, Gray, & Gouttebarge, 2017) may experience these coping responses as ineffective in the longer term, potentially enhancing psychological distress through cascading problems associated with heavy substance use.

Athletes are known to be a population that demonstrate stoicism and withhold disclosure of mental health problems (Breslin, Shannon, Haughey, Donnelly, & Leavey, 2017; Kerr, Register-Mihalik, Kroshus, Baugh, & Marshall, 2016), and it has been suggested that athletes may experience depression in ways beyond current diagnostic (e.g. DSM-5) criteria (Doherty et al., 2016). Assessment of somatic or externalising symptoms of depression may improve the detection of mental health problems in populations less likely to openly discuss emotional functioning (Brownhill, Harris, Harris, & Wilhelm, 2006). From a theoretical perspective, the “big build” model characterises escalating behaviours of avoidance, numbing, risk-taking, and anger or aggression as potential transdiagnostic markers of psychological distress (Brownhill, Wilhelm, Barclay, & Schmied, 2005). In the “big build” model, psychological distress is associated with difficulties in self-regulation including irritability and anger (Möller-Leimkühler & Yücel, 2010), which is especially true among younger adult populations (Fava et al., 2010; Massimiliano, Perret, Turecki, & Geoffroy, 2018), as well as problems with goal directed behaviour and motivation (Gillan, Kosinski, Whelan, Phelps, & Daw, 2016), and coping through use of substances (Lai, Cleary, Sitharthan, & Hunt, 2015).

Much of this line of research has focussed on male samples (e.g. Genuchi & Valdez, 2015; Martin, Neighbors, & Griffith, 2013), and relates to socialisation processes prohibiting expression of symptoms or behaviours associated with vulnerability or weakness (Seidler, Dawes, Rice, Oliffe, & Dhillon, 2016). Coupled with concerns related to external stressors experienced by many athletes, including worries related to performance (Moen, Myhre, Klöckner, Gausen, & Sandbakk, 2017), selection (Robertson, Bartlett, & Gastin, 2017) and transition beyond a competitive identity (Torregrosa, Ramis, Pallarés, Azócar, & Selva, 2015), it is possible that a broad conceptualisation of athlete strain indicators, consistent with the “big build” model, may assist in earlier identification of athletes experiencing distress through non-internalising symptoms. In line with strain theory, it is possible that some athletes experiencing significant and ongoing psychosocial pressures beyond their natural coping resources, and who are simultaneously striving for ideals of competitive excellence, may experience early distress symptoms as a combination of performance and self-regulation difficulties. Identification may be further compromised as sports practitioners report less comfort in discussing psychological health with athletes relative to physical health and conditioning (Rao & Hong, 2016).

In a broad sense, symptoms of psychological strain in athletes reflect difficulties of adaptation or adjustment to a change in circumstances, and are characterised by impairments in social and athletic functioning, and mood-related and impulse control problems (Glaesmer, Romppel, Brähler, Hinz, & Maercker, 2015). Stressors that elite athletes experience during their competitive years may be role-related (e.g. maladjustment to training pressures and maintaining an athletic career or position security) or personal in nature (e.g. non-sporting issues such as the work-life interface, financial problems, or relationship difficulties) (Jones & Tenenbaum, 2009; Mellalieu, Neil, Hanton, & Fletcher, 2009; Sarkar & Fletcher, 2014). Additionally, contemporary elite athletes may experience role-related and personal stressors becoming increasingly intertwined through traditional and social media pressures and obligations associated with sponsorship (Filo, Lock, & Karg, 2015; Sanderson, Snyder, Hull, & Gramlich, 2015).

Regardless of the cause, or perceived impact of a given stressor for an athlete, early identification and management of stress-related problems or mental health symptoms is an effective means to facilitate intervention in an earlier illness phase (Ferguson, Swann, Liddle, & Vella, 2018; Sebbens, Hassmén, Crisp, & Wensley, 2016). If instances of externalising responses are indeed early indicators of athlete distress, assessment of these domains may improve early detection and intervention for impacted athletes. Given existing screening tools that assess psychological distress (e.g. K-10; Kessler et al., 2002) or depression (e.g. PHQ-9; Kroenke, Spitzer, & Williams, 2001) have a strong focus on internalising symptoms, these measures may not be sufficiently sensitive to early manifestations of distress experienced by elite athletes, leading to less than optimal detection rates.

The purpose of the present study was to develop and validate a brief measure of sports-related psychological strain in a representative sample of elite-level Australian male athletes. Secondary aims were to establish convergent and divergent validity benchmarked against established measures of global psychological distress and psychological wellbeing, and examine outcome differences for athletes likely to be experiencing psychological strain. Consistent with psychological strain theory, it was hypothesised that athletes unable to compete due to injury would report significantly higher scores on the measure domains than those able to currently compete.

Methods

Participants

A total of 1,007 male elite, professional athletes from three national Australian sporting codes (Australian football, cricket, soccer) provided data, constituting 78.6% of the contracted athletes across these codes. All athletes were aged 18 years and over (M = 23.67, SD = 4.16 years). No exclusion criteria were applied.

Measures

Athlete Psychological Strain Questionnaire (APSQ)

The APSQ was conceptualised to be a brief, self-report rating scale specific to the athletic context. Candidate items for the APSQ were developed based on clinical and consulting experience in the field. Item development was further informed by our previous systematic review into elite athlete mental health indicating sources of stress and maladaptive avoidance coping patterns in this population (Rice et al., 2016), and other relevant studies highlighting that athletes experiencing psychological strain report relationship challenges within and outside sports settings (Doherty et al., 2016), problems with substance involvement (Donohue, Pitts, Gavrilova, Ayarza, & Kristina, 2013; Dunn & Thomas, 2012), performance concerns including life after elite competitive sport (Torregrosa, Boixadós, Valiente, & Cruz, 2004), and externalising behaviours such as anger or aggression and risk-taking (Doherty et al., 2016). Author Rice developed the initial item pool of 12-items. While larger candidate item pools are recommended when developing new scales (Carpenter, 2018), the present study was nested within a broader project into athlete wellbeing, and it was necessary to limit any time imposition for the participating athletes.¹ Authors Purcell and Parker reviewed the final wording of the item pool authors Rice, Purcell and Parker are all registered psychologists with significant experience in the fields of athlete and youth mental health. Candidate APSQ items were developed to assess for difficulties with team-based interactions, impaired impulse control and frustration tolerance, worries related to athletic performance and training stress, and transition to life beyond professional athletic pursuits. Items followed key recommendations for scale development, including brevity, reading level and tone (DeVellis, 2016; Worthington & Whittaker, 2006) and were designed to use the same response scale (i.e. 1 = “None of the time”; 5 = “All of the time”) and timeframe as the Kessler-10 distress scale.

Psychological distress

The Kessler-10 (K-10; Kessler et al., 2002) is among the most widely used measures of global psychological distress and was used in the current study to assess convergent validity with the APSQ. Scores are categorised to indicate respondents’ distress over the past 4 weeks. Responses to items (i.e. “About how often did you feel hopeless?”) are made on a 5-point scale (i.e. 1 = “None of the time”; 5 = “All of the time”). The K-10 was developed and validated using epidemiological samples, and is widely recommended as a useful screening tool or simple outcome measure for assessing treatment progress for common mental disorders such as anxiety and depression (Slade, Grove, & Burgess, 2011). The K-10 demonstrates robust psychometric properties (Sunderland, Mahoney, & Andrews, 2012) across varying populations (e.g. Sampasa-Kanyinga, Zamorski, & Colman, 2018) and cultures (Hajebi et al., 2018). In the current study the K-10 reported satisfactory reliability α = 0.86.

Psychological wellbeing

The Warwick-Edinburgh Mental Well-Being Scale (Stewart-Brown et al., 2009) (WEMWBS) is a 14-item self-report scale that assesses positive aspects of mental health (i.e. “I’ve been feeling good about myself”). The WEMWBS was used in the current study to assess convergent validity with the APSQ. Responses are made relative to the last 2-weeks, in the form of a 5-point scale (i.e. 1 = “None of the time”; 5 = “All of the time”). Population-level studies support the psychometric properties of the WEMWBS as a measure of mental wellbeing (Clarke et al., 2011; Lloyd & Devine, 2012), including cross cultural studies (Taggart et al., 2013; Trousselard et al., 2016). The WEMWBS is inversely related to measures of common psychopathology assessing symptoms of depression and anxiety (Smith, Alves, Knapstad, Haug, & Aarø, 2017) and is positively related to favourable psychosocial working conditions (Bartram, Yadegarfar, Sinclair, & Baldwin, 2011). In the current study the WEMWBS reported satisfactory reliability α = 0.95.

Procedure

Ethical approval for the study was granted by The University of Melbourne Human Research Ethics Committee (ID: 1442705). All participants were invited to complete an anonymous online survey, which took approximately 15 min to complete. Participants were approached by the researchers, with the support of the participating sporting codes, consistent with the approved research ethics procedures. The first page of the survey included informed consent information, including the voluntary nature of participation. This first page of the survey stated that consent was inferred based on provision of data. Participants were reminded of any missing items prior to progressing to the next page, resulting in no missing outcome data. The survey was completed in a group setting, with two researchers present to introduce and oversee the data collection. Participants were provided with individual access to a tablet or laptop computer to complete the survey. Data collection occurred locally, at each of the participant clubs/associations.

Data analysis

Descriptive statistics were used to characterise the sample. A two-stage process was undertaken to validate the factor structure of the APSQ using exploratory factor analysis (EFA) and parallel analysis in a randomly partitioned calibration sample, and confirmatory factor analysis (CFA) in a separate validation sample. Firstly, iterative exploratory factor analysis (principal axis factoring) was undertaken to determine the underlying factor structure of the APSQ (i.e. scale calibration) (DeVellis, 2016; Worthington & Whittaker, 2006). To enable this, the sample of N = 1007 was partitioned into two datasets according to random numbers allocated in SPSS. This resulted in a calibration sample (n = 497), and a validation sample (n = 510). In line with recent guidelines (Carpenter, 2018), oblique rotation (promax) was adopted as it is identified as providing more robust factor solutions (Thompson, 2004), and involves an oblique factor rotation where factors are permitted to correlate. Following this, the included APSQ items, as identified through the principal axis procedure, were subject to CFA in order to validate the reported factor structure within the separate calibration sample (Byrne, 2001). We adopted the typically accepted criteria to assess model fit CFI > 0.95, TLI > 0.95, RMSEA and SRMR < 0.50 (Byrne, 2001), in addition to the sample size adjusted Bayesian Information Criterion (BIC) to judge improved model fit for a second order model. Here, Δ BIC > 10 was the criteria to identify a “decisive” improved model fit (Kass & Raftery, 1995). To determine the stability of APSQ scores between the calibration and validation samples, ANOVA evaluated any sample group differences for the APSQ subscales and APSQ Total Score. Differential item functioning, using likelihood ratios, was undertaken to assess for potential differences in responding to APSQ items based on level of education (i.e. potential bias that may occur in differences for those with / without a university degree; Teresi & Fleishman, 2007) and region of birth (e.g. Australian born versus non-Australian born). Receiver operator characteristic (ROC) curve analysis was undertaken to determine the sensitivity and specificity of the APSQ at identifying high psychological distress as assessed by the K-10. The area under curve (AUC) value was used to examine how well the APSQ correctly identified athletes reporting high distress. The Youden Index was employed to identify a cut-off score for the APSQ Total Score that would balance sensitivity and specificity. This procedure maximises the vertical distance from the line of equality to the point [x, y] and determines the point at which sensitivity (i.e. the true positive rate; correctly identifying condition) and specificity (i.e. false positive rate; correctly identifying absence of condition) are considered equally important. Convergent and divergent validity was determined through examining patterns of statistically significant correlations with the K-10 (convergent validity evidenced by significant positive associations) and the WEMWBS (divergent validity evidenced by significant negative associations). Finally, construct validity was assessed using multivariate analysis of covariance (MANCOVA) to determine group differences on APSQ domains according to athlete injury status. The three-APSQ domains were dependent variables, and in-season status was entered as a covariate.

Results

Sample ages ranged from 18 to –39 years (M = 23.67, SD = 4.16, Median = 23). Most of the sample reported being partnered, and the majority reported moving from home for their respective sport. Respondents were most likely to be living in rental accommodation, although a small percentage owned their home outright. Relative to the general population, there was a relatively high proportion of the sample reporting an Aboriginal or Torres Strait Island cultural background (6.9%). Most of the sample were within the first five years of elite competition, and just over a quarter (28.6%) reported being injured or on an adapted training programme at the time of the survey. Most of the sample (88.3%) were not currently in their competitive season at time of data collection. Please see Table 1 in online supplemental materials for demographic details.

Exploratory factor analysis

Prior to conducting principal axis factoring, correlations were inspected, Bartlett’s test of sphericity was performed and the Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO) evaluated. The correlation matrix indicated most observed items were correlated (i.e. r > 0.35), Bartlett’s test of sphericity was significant (p < 0.001), and the KMO value was >0.6 (KMO = .921) indicating that factor analysis was appropriate for the present dataset (Field, 2009). Following recommendations, the exploratory analyses were iteratively re-run after item deletion to ensure the factor structure did not change with omission of items.

Parallel analysis was undertaken using the rawpar.sps procedure (O'Connor, 2000) undertaken in SPSS 22.0, using 5,000 parallel datasets, α = .01, principal axis factoring and permutations of the raw data set. Parallel analysis indicated a three-factor solution. Subsequently, principal axis factoring resulted in three-factors accounting for 50.16% of total variance. One item (“My sleep was worse than usual”) cross-loaded > 0.32, and was omitted. The subsequent analysis (KMO = .913, Bartlett’s test of sphericity p < .001) resulted in three-factors accounting for 51.08% of total variance, though there was another cross-loading item (“I covered up my difficulties”). The analysis was re-run (KMO = .890, Bartlett’s test of sphericity p < .001) resulting in three-factors accounting for 50.44% of total variance, with convergence in six iterations, suggesting good factor stability. Please see Table 2 in online supplemental materials for factor loadings. There were no cross-loading items > 0.32 as part of the final 10-item, three-factor solution. A final parallel analysis was undertaken using these 10-items, which further confirmed the three-factor solution.

The first factor comprised four items and was labelled Self-Regulation (calibration sample α = 0.79; validation sample α = 0.84, eigenvalue = 4.57), the second factor comprised four items and was termed Performance (calibration sample α = 0.75; validation sample α = 0.73, eigenvalue = 1.03), while the third factor comprised two items and was labelled External Coping (calibration sample α = 0.64; validation sample α = 0.72, eigenvalue = 0.87). Carpenter (2018) recommends that two-item factors should only be retained when the subscale reports internal consistency, is theoretically valid, and items are well correlated, each of which were the case in the present data (calibration sample r = 0.484, p < .001; validation sample r = 0.576, p < .001).

Confirmatory factor analysis

In order to validate the three-factor solution, data from the unanalysed validation dataset (n = 507) was subjected to CFA. Maximum likelihood estimation was undertaken yielding a significant p-value (χ² = 81.22(32), p < .001). Fit indices supported model fit; CFI = 0.976, TLI = 0.966, RMSEA = 0.055, SRMR = 0.032, sample size adjusted BIC = 9995.97 although the RMSEA was marginally above the suggested cut-off. A subsequent second-order factor model was evaluated, with the three APSQ factors loading on a higher-order “Athlete Psychological Strain” factor (see Figure 1). The second order factor model was also judged a good fit to the data, with improved SRMR and BIC values (χ² = 81.22(32), p < .001), CFI = 0.976, TLI = 0.966, RMSEA = 0.055, SRMR = 0.029, sample size adjusted BIC = 9965.49 (Δ SSA BIC = 30.48). Standardised CFA factor loadings and standard error values for the second order model are presented in Figure 1. Means were equivalent across the calibration and validation sub-samples for scale domains; APSQ Self-Regulation (calibration M = 5.79, SD = 2.32; validation M = 5.74, SD = 2.61; p = .784), APSQ Performance (calibration M = 6.51, SD = 2.80; validation M = 6.32, SD = 2.71; p =.281), APSQ External Coping (calibration M = 2.48, SD = 1.09; validation M = 2.52, SD = 1.16; p = .649), and APSQ Total Score (calibration M = 14.78, SD = 5.31; validation M = 14.58, SD = 5.62; p = .567).

Figure 1. Standardised regression weights (with standard error values) for the second-order APSQ factor structure.

Differential item functioning

Differential item functioning (DIF) was undertaken using the SPSS script DIFLRT to compute likelihood-ratio tests, including uniform and non-uniform DIF for differences according to education level (comparing those with and without degrees) and ethnicity (comparing those born in Australia and those born outside Australia). Uniform DIF exists when there is no interaction between ability level and group membership while non-uniform DIF exists where there is an interaction (i.e. differences in probabilities of item endorsement are not the same over education level) (Swaminathan & Rogers, 1990). All DIF tests were non-significant, indicating consistency of responses over the 10 APSQ items (i.e. no evident bias) according to level of participant education and ethnicity (Please see Table 3 in online supplemental materials for DIF values).

Sensitivity and specificity

ROC curve analysis was undertaken using the full sample (i.e. N = 1,007). The ROC curve state variable was selected as cases meeting the criteria of high distress on the K-10 (i.e. a K-10 score of ≥22), which included a total of 77 cases (i.e. the highest 7.6% of the distribution on K-10 scores). The area under curve (AUC) value for the APSQ was 0.955 [95% CI 0.938–0.972], SE = .009, p < .001. This AUC value indicates the APSQ is “excellent” at correctly identifying athletes experiencing high psychological distress. The Youden Index indicated the optimal cut-off score on the APSQ as 21, which reflected the uppermost 10.5% of the total sample. Based on use of the APSQ Total Score cut-off score of 21, the scale was able to accurately detect high psychological distress with a sensitivity of 93.5% and a specificity of 79.2%.

Convergent and divergent validity

The three domains of the APSQ demonstrated convergent validity with K-10 scores (see Table 1 for correlation coefficients). Robust moderate Pearson correlations (p’s < .001) were observed separately for the calibration and validation samples for the APSQ Self-Regulation, Performance and External Coping subscales. Divergent validity was observed with significant negative correlations (p’s < .001) between WEMWBS wellbeing scores and the three APSQ domains for the calibration and validation samples respectively.

Table 1. Zero-order correlations for study outcomes.

	APSQ – Self-regulation	APSQ – Performance	APSQ – External Coping	APSQ – Total
APSQ – Self-regulation	–	.61***	.55***	.87***
APSQ – Performance	.66***	–	.50***	.90***
APSQ – External Coping	.61***	.50***	–	.71***
APSQ – Total	.91***	.89***	.73***	–

Note: ***p < .001; Calibration sample (n = 497) values above diagonal; Validation sample (n = 507) values below diagonal.

Construct validity

A MANCOVA was used to determine differences for those injured or on an adapted programme versus those on standard training programmes. Injured athletes were hypothesised to be experiencing elevated role-related strain, and were thus expected to reported higher scores on the APSQ domains. As expected, there was a significant multivariate effect observed for current injury Λ = .992, F(3, 983) = 2.80, p = .040, η² = .008, with the in-season variable a significant multivariate covariate Λ = .989, F(3, 983) = 3.78, p = .010, η² = .011 (see Table 2 for means and SDs). At the univariate level, in season status was a significant covariate for all APSQ domains (p’s < .05). APSQ domain scores were higher for injured athletes on the Performance subscale (small effect size), with a non-significant trend for Self-Regulation subscale, and no difference for the External Coping subscale (see Table 2).

Table 2. Group comparisons on study outcome measures.

	Full sample (n = 1,003)	Injured / Adapted training (n = 288)	Uninjured (n = 715)	df	F	p	Effect size (partial η^2)
APSQ Domain	M (SD)	M (SD)	M (SD)
Self-Regulation	5.76 (2.47)	5.97 (2.76)	5.68 (2.34)	1,985	2.85	.092	.003
Performance	6.42 (2.75)	6.81 (3.00)	6.26 (2.63)	1,985	8.32	.004	.008
External Coping	2.50 (1.12)	2.58 (1.29)	2.46 (1.05)	1,985	2.07	.151	.002
APSQ Total	14.67 (5.47)	15.36 (6.10)	14.40 (5.18)	1,988	3.38	.007	.007
K-10 Total	14.16 (4.80)	14.08 (4.74)	14.19 (4.82)	1,1000	0.09	.760	.000
WEMWEBS	51.39 (8.91)	50.84 (9.10)	51.61 (8.83)	1,986	1.86	.173	.002

Note: Self-Regulation possible range 4–20; Performance possible range 4–20; External Coping possible range 2–10; APSQ Total possible range 10–50; K-10 Total possible range 10–50; WEMWEBS possible range 14–70.

Discussion

Findings from this representative sample of elite male athletes support the validity, reliability, factor structure, item functioning and sensitivity/specificity of the APSQ. As a brief, three-factor screening tool, the APSQ may be a useful component within a broader assessment battery examining athlete performance and wellbeing. Aspects of convergent and divergent validity indicated that higher APSQ scores were associated with lower wellbeing and greater psychological distress. The APSQ demonstrated both high sensitivity (i.e. hit rate) and high specificity (i.e. correct rejection rate) relative to those experiencing elevated distress as identified by the K-10. An independent sample, using CFA, is needed to confirm the dimensionality of the APSQ. Nonetheless, findings are noteworthy given that APSQ items tap into domains specific to athletes that are distinct from those assessed by the K-10, and that the K-10 is a well-regarded scale widely used in major population health studies (Kessler et al., 2002; Slade et al., 2011). An advantage of using the APSQ with athlete populations may be that APSQ items are perceived by athletes as less “intrusive” relative to those presented in general psychological distress measures (including the K-10), while simultaneously providing important information related to athlete mental health and adjustment.

The APSQ factors of Self-Regulation and External Coping are consistent with recent research suggesting that externalising symptoms may be early indicators of psychological strain (Genuchi & Valdez, 2015; Rice et al., 2015). In particular, Brownhill’s “big build” model identified externalising symptoms as part of a trajectory of distress (Brownhill et al., 2005), and population-based research (Martin et al., 2013) supports the notion that risk-taking, anger or aggression, and substance misuse may be associated with psychological strain or distress, especially in men (Cavanagh, Wilson, Kavanagh, & Caputi, 2017; Seidler et al., 2016), but also in women (Möller-Leimkühler & Yücel, 2010). Whereas the Self-Regulation factor assesses a combination of low motivation and interpersonal difficulties, the two-item External Coping factor assesses substance-related coping and risk-taking outside the sports context. These types of behaviours or coping responses are likely to impede athlete achievement, and increase the likelihood of athletes experiencing subsequent mental health or physical health problems (Conway, Swendsen, Husky, He, & Merikangas, 2016; Rao & Hong, 2016).

Though beyond the scope of the present study, use of the APSQ alongside athlete-specific wellbeing interventions (e.g. Donohue et al., 2015, 2018) may assist in determining the influence of scale domains on psychosocial and athletic functioning. If problems related to Self-Regulation and External Coping are associated with higher levels of psychological strain in athletes, it stands to reason that currently injured athletes would report higher scores on these factors, in addition to the athlete-specific Performance factor. In the current study, this was true at the multivariate level, and more specifically on the Performance factor. These effects were relatively small, and there were no group differences according to injured and uninjured athletes for the Self-Regulation and External Coping factors. That said, the APSQ may be more effective than athlete non-specific measures in identifying psychological strain experienced by injured athletes, as there was no group difference observed for injured athletes on either the K-10 or WEMWBS.

While the APSQ total score was sufficiently sensitive in identifying athletes reporting high psychological distress (as defined by the K-10), larger studies are needed to determine correlates of the APSQ domains, and their relationship to psychological strain. As highlighted above, CFA outcomes should be replicated in another sample and more work undertaken to validate the APSQ cut-off score. The results of this research support the potential utility of the APSQ, although prospective studies are needed to establish the longer-term predictive validity of the APSQ relative to broader aspects of performance and health. Doing so may see measures such as the APSQ become useful in the early identification of elite athlete distress, and thus, enable effective, early intervention for emerging mental ill-health. This may be especially relevant if the APSQ is utilised by sporting codes during periods that are associated with heightened athlete distress, such as injury and retirement (Gouttebarge, Backx, Aoki, & Kerkhoffs, 2015; Gouttebarge, Inklaar, & Frings-Dresen, 2014). Research into the utility of the APSQ is also warranted given that the peak years of elite athlete development and competition closely overlap with the peak ages for the onset of mental ill-health (12–25 years), and more specifically mental disorders (Rice et al., 2016).

There are a number of limitations that must be considered and addressed in future work related to the APSQ. The present validation samples were entirely male. The decision to use male-only samples in the present study was pragmatic, based on the data available to the researchers. There is therefore a pressing need for psychometric replication of the scale factor structure in elite female athletes. Studies are increasingly supporting an externalising distress phenotype in females (Innamorati et al., 2011; Möller-Leimkühler & Yücel, 2010) and previous work has highlighted that measurement models of externalising distress are consistent across male and female respondents (e.g. Rice et al., 2015; Rice, Fallon, Aucote, & Möller-Leimkühler, 2013). While it is possible that APSQ domains may be valid in female athletes, this is an empirical question in need of supporting data. We suggest that large combined samples be collected, enabling a broadening of the differential item functioning analysis, testing for possible gender bias (or true gender differences) in APSQ items. Given the cross-sectional nature of the present study, the test-retest stability of the APSQ could not be determined. Future longitudinal work should address this, and ideally examine trajectories over time. Such work should include additional outcome measures to examine associations of the APSQ factors with broader symptom domains such as disordered eating behaviours (Petrie, Greenleaf, Reel, & Carter, 2008), or sleep problems (Sargent, Lastella, Halson, & Roach, 2014). Additional work should focus on the convergent and divergent validity of the scale, including assessment at key phases of risk, such as periods of injury or transition out of competitive sport, as well as exploring predictive relationships between relevant physical aspects (i.e. acute injury) and neuropsychological outcomes such as concussion which are implicated in athlete mental health outcomes (Rice et al., 2018). Other psychosocial pressures (i.e. social media abuse) unique to this population should also be examined. This would ideally include baseline (i.e. pre-season) screening, and periodic re-assessment to enable early detection and monitoring of change over time. It is also noted that the majority of participants were not currently in their main competitive season at the time of data collection. Future research should explore the impact of season timing on athlete mental health. Finally, the External Coping factor comprised only two-items. While these two items were well correlated, future studies are required to verify the validity and reliability of this factor.

Conclusions

The field of athlete mental health is gaining momentum, with specific athlete-centric models of care being developed and implemented (Gulliver et al., 2012; Moesch et al., 2018; Reardon & Factor, 2010). As greater clinical and research attention follows, the need for brief, targeted, athlete-specific measures will grow. The APSQ is a progression in the assessment of psychological strain in elite athletes and conversely, the absence of wellbeing. Use of the scale may assist in early identification and provision of support, corresponding to improved mental health and wellbeing for this cohort.

Acknowledgments

The authors thank the participating sporting codes and competitors for taking part in the study. Simon Rice was supported by the Mary Elizabeth Watson Early Career Fellowship in Allied Health from Royal Melbourne Hospital. This study was, in part, made possible from funding from several sporting bodies, all of whom requested anonymity

Supplemental data

Supplemental data for this article can be accessed here: https://doi.org/10.1080/1612197X.2019.1611900.

Additional information

Funding

Simon Rice was supported by the Mary Elizabeth Watson Early Career Fellowship in Allied Health from Royal Melbourne Hospital [grant number MEW-2017-001]. This study was, in part, made possible from funding from several sporting bodies, all of whom requested anonymity.

Notes

1 Although these data were drawn from a larger study, to date, no other publications have occurred based on the data.