Assessing the Smallest Detectable Change of the Kessler Psychological Distress Scale Score in an Adult Population in Japan

Abstract

Background

Psychological distress is prevalent worldwide and can lead to the development of mental conditions such as major depression and anxiety disorders. It is essential to assess the severity of patient-reported outcomes to provide effective treatment for psychological distress. The Kessler Psychological Distress Scale (K6) is one of the most widely used psychological distress scales. However, the smallest detectable change (SDC) of the K6 score has not been elucidated. Therefore, the current study aimed to determine the SDC of the K6 score in a Japanese adult population.

Methods

Participants aged 20–59 years who are native Japanese speakers were recruited from the panel list of a web research firm. The K6 score was assessed at baseline (T1) and at 2-week follow-up (T2). SDCs were calculated at the individual (SDC_ind) and group (SDC_group) levels. Intraclass correlation coefficient agreement (ICC_agreement) was calculated to assess test–retest reliability and Cronbach’s alpha to evaluate internal consistency.

Results

A total of 3254 (1627 [50%] female) responded at T1 and T2. The mean (standard deviation) K6 scores were 5.71 (5.84) at T1 and 5.65 (5.83) at T2. The SDC_ind and SDC_group of the K6 score were 8.47 (35.31%) and 0.15 (0.63%), respectively. The ICC_agreement was 0.73, and the Cronbach’s alpha was 0.94.

Conclusion

Our study provided evidence on the reliability and interpretation of the K6 score. Calculating the SDC of the K6 score can help identify the significance of changes in psychological distress over time and can determine the efficacy of interventions for psychological distress.

Keywords:

• psychological distress
• reliability
• measurement error
• mental health

Introduction

Psychological distress is characterized by different nonspecific symptoms including lack of enthusiasm, sleep issues, depression, and hopelessness about the future.^1–3 It is prevalent worldwide, and previous studies showed that 15.1–24.7% of the general population experience moderate psychological distress.^4,5 Moreover, 3.1–4.2% present with severe psychological distress,^4,5 which is associated with various issues such as low quality of life,^6–8 cognitive decline,⁹ poor academic performance,^10–12 high medical costs,¹³ unemployment,¹⁴ and suicide.¹⁵ Importantly, psychological distress reflects not only the severity but also the chronicity of depressive and anxiety symptoms.^16,17 Further, it may lead to the development of mental disorders such as major depression and anxiety disorders.^18,19 Considering these issues, reducing psychological distress is an important public health priority. Therefore, several intervention strategies have been developed to address psychological distress, and their efficacy is examined.^20–23 Previous studies have provided substantial evidence that interventions such as cognitive behavioral therapy and the use of smartphone applications are effective in improving psychological distress.^22–25 In addition to these interventions, the application of patient-reported outcome (PRO) may be important in facilitating more effective and appropriate interventions.

The use of PROs, which has numerous advantages, in clinical practice has currently become a topic of interest.^26–28 Several PROs have been utilized in clinical trials.²⁹ Moreover, the systematic monitoring of patient symptoms using PROs has improved patient–clinician communication and symptom assessment.^27–29 In addition, PRO-based care has better outcomes than the usual care.^28,29 Due to these advantages, the use of PROs in clinical settings is important as more effective interventions can be provided for psychological distress. However, considering patient burden and the busy clinical situation, PROs must be simple for the respondent, scorer, or interpreter.³⁰ Therefore, there is an urgent need to develop PROs that can identify early-stage psychological distress and can assess changes in disease severity among patients receiving treatment using a reliable and simple method.

The Kessler Psychological Distress Scale (K6) is an effective PRO tool for screening not only the presence but also changes in the severity of psychological distress.^31–33 In fact, the K6 score is used as a standardized measure in cases in which a more detailed severity assessment is not possible.³¹ In addition, the K6 can be answered in approximately 2–3 min, and the result is easy to grade.^32,34 Therefore, it is simpler and less burdensome for respondents, scorers, or interpreters. Due to these advantages, the K6 is one of the most widely used psychological distress scales in both clinical and research settings.^4,5,33,35,36 Further, it has been translated in several countries, and its psychometric properties including reliability and validity have been evaluated.^{31,32,34,35,37–40}

However, the smallest detectable change (SDC) of the K6 score must be identified to support its utility in research and clinical practice. SDC is the value that can be considered an actual change independent of measurement error, and any change that is smaller than the SDC can be interpreted as a result of measurement error.^41–43 That is, a change in scores can represent an actual change only if it is greater than the SDC. In addition, the SDC and original values are displayed in the same units. Therefore, establishing the SDC could resolve important issues in determining whether changes in scores are clinically significant or are caused by measurement errors. This can then provide clinicians and epidemiologists with useful information for interpreting changes in scores.^41,44 Due to these advantages, SDC can be an essential component of the K6 in research and clinical use. However, the SDC of the K6 score is not fully elucidated. Therefore, the current study aimed to determine the SDC of the K6 score in an adult population in Japan.

Methods

Study Design

A prospective cohort study was conducted to examine the SDC, standard error of measurement (SEM), test–retest reliability, and internal consistency of the K6 score. Data were collected in December 2020 via an internet survey. The research was approved by the Ethical Review Board for Medical Research Involving Human Subjects of Gunma University (Approval no. HS2020-168) and was conducted in accordance with the Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) initiative.⁴³

Participants

The participants were recruited from the panel list (with over 3.5 million people) of a web research firm (Marketing Applications Inc.). A link to the website that provided information about the survey was sent to potential participants aged 20–59 years who are native Japanese speakers. The email included an option to provide an electronic informed consent. Those who provided consent by checking the “I agree” button participated in the survey. Participants who responded to the first internet survey (T1) completed the second internet survey (T2) after 2 weeks. The time interval between T1 and T2 was long enough to prevent recall bias.⁴⁵

Outcomes

K6

The K6 is a six-item self-report tool for assessing psychological distress over the past 30 days.^31,32 Moreover, it is a five-point Likert scale (4 = all the time, 3 = most of the time, 2 = sometimes, 1 = a little of the time, and 0 = none of the time) with a total score of 0–24. Higher scores indicate greater psychological distress. Previous studies have shown the reliability and validity of the K6.^{31,32,34,35,37–40,46} Further, it has an excellent internal consistency (Cronbach’s alpha = 0.85–0.90).^34,35 The Japanese version of the K6 was used in this study. The reliability and validity of the Japanese version of K6 have been verified in previous studies.^32,34,35,47

Sample Size

According to the COSMIN initiative, a sample size of ≥ 100 is sufficient for achieving statistical power when assessing test–retest reliability, SDC, and internal consistency.⁴³ Therefore, the current study included > 100 participants.

Statistical Analysis

All analyses were performed using R (version 4.0.2 for Windows; The R Project for Statistical Computing, Vienna, Austria).

Test–Retest Reliability

Test–retest reliability indicates the stability of the instrument and its ability to produce similar scores on repeated measurements.⁴³ The irr package was used to calculate intraclass correlation coefficients (ICC_agreement=σ²_p /[σ²_p + σ² _m + σ²_r]) based on a two-way random effects model.^43,48 σ² refers to the variance component, where p = systematic difference between the participants’ actual scores, m = error variance of the systematic difference between two measurements, and r = random error. The intraclass correlation coefficients were expressed as a value between 0 and 1, and values > 0.70 were acceptable.^43,48

Smallest Detectable Change

SDC is the smallest change of the K6 score that is considered an actual change (ie, a change greater than the measurement error).⁴³ The individual-level SDC (SDC_ind) was calculated using the following formula: 1.96 x √2 x SEM_agreement.^45,49 In addition, the group-level SDC (SDC_group) was calculated using the following formula: SDC_ind/√n.^50,51 SEM_agreement was calculated using the square root of the error component () of the ICC_agreement formula.⁴³ Error variances were evaluated with the linear mixed models using the lmerTest package.

Internal Consistency

Internal consistency expresses the degree of interrelatedness between items.⁴³ Cronbach’s alpha was used to assess the internal consistency of the K6 score. The Cronbach’s alpha ranges from 0 to 1. If the interrelationship between items is higher, the alpha value is greater. A Cronbach’s alpha of > 0.7 indicate a good consistency.⁴⁵ The psych package was used to evaluate Cronbach’s alpha.

Results

Characteristics of the Participants

Of 135,848 people invited to visit the survey website, 6632 completed the survey at T1. Among them, 3254 completed the survey at T2. The final participation rate was 2.40% (3254 of 135,848). Table 1 shows the characteristics of the participants. In total, 1627 (50%) were women. Further, 823 (25.29%) were aged 20–29 years; 811 (24.92%), 30–39 years; 812 (24.95%), 40–49 years; and 808 (24.83%), 50–59 years.

Table 1 Characteristics of the Participants (N = 3254)

Characteristics	n	%
Sex
Male	1627	50
Female	1627	50
Age (years)
20–29	823	25.29
30–39	811	24.92
40–49	812	24.95
50–59	808	24.83
Marital status
Not married	1626	49.97
Married	1628	50.03
Smoking habits
Nonsmoker	2383	73.23
Smoker	871	26.77
Drinking habits
Nondrinker	1739	53.44
Drinker	1515	46.56
Physical activity
Yes	2012	61.83
No	1242	38.17
Education
Without university degree	1942	59.68
With university degree	1312	40.32

SDC, SEM_agreement, and ICC_agreement of the K6 Scores

Table 2 shows the mean and SD, mean change, SEM_agreement, SDC, and ICC_agreement of the K6 scores. The mean (standard deviation) K6 scores were 5.71 (5.84) at T1 and 5.65 (5.83) at T2. The ICC_agreement and SEM_agreement of the K6 scores were 0.73 and 3.06, respectively. The SDC_ind of the K6 score was 8.47 (35.3%), and the SDC_group was 0.15 (0.6%).

Table 2 ICC, SEM, and SDC of the K6 Scores

	n	T1	T2	Changes
		Mean (SD)	Mean (SD)	Mean (SD)	SEMagreement	SDCind	SDCind (%)	SDCgroup	SDCgroup (%)	ICCagreement (95% CI)
Both sexes	3254	5.71 (5.84)	5.65 (5.83)	−0.07 (4.32)	3.06	8.47	35.31%	0.15	0.63%	0.73 (0.71–0.74)
20–29	823	7.09 (5.72)	7.04 (5.85)	−0.05 (4.81)	3.40	9.42	39.25%	0.33	1.38%	0.66 (0.61–0.69)
30–39	811	6.36 (6.11)	6.20 (6.10)	−0.16 (4.78)	3.38	9.37	39.04%	0.33	1.38%	0.69 (0.66–0.73)
40–49	812	5.28 (5.92)	5.24 (5.87)	−0.04 (3.95)	2.79	7.73	32.21%	0.27	1.13%	0.78 (0.75–0.80)
50–59	808	4.09 (5.10)	4.07 (5.02)	−0.02 (3.62)	2.56	7.10	29.58%	0.25	1.04%	0.74 (0.71–0.77)
Male	1627	5.78 (5.78)	5.66 (5.72)	−0.12 (4.39)	3.11	8.62	35.92%	0.21	0.88%	0.71 (0.68–0.73)
Female	1627	5.65 (5.89)	5.63 (5.94)	−0.02 (4.25)	3.01	8.34	34.75%	0.21	0.88%	0.74 (0.72–0.76)

Abbreviations: K6, Kessler Psychological Distress Scale; ICC, Intraclass correlation coefficient; SEM, Standard error of measurement; SDC, Smallest detectable change; T1, At baseline; T2, At 2-week follow-up; SD, Standard deviation; CI, Confidence interval.

Cronbach’s Alpha

As depicted in Table 3, the Cronbach’s alpha of the K6 score at T1 and T2 was 0.94, which indicated a good consistency.

Table 3 Internal Consistency of the K6 Scores

	n	T1	T2
		Cronbach’s Alpha Coefficient	Cronbach’s Alpha Coefficient
Both sexes	3254	0.94	0.94
20–29	823	0.93	0.93
30–39	811	0.95	0.95
40–49	812	0.95	0.94
50–59	808	0.94	0.94
Male	1627	0.94	0.94
Female	1627	0.94	0.94

Abbreviations: K6, Kessler Psychological Distress Scale; T1, At baseline; T2, At 2-week follow-up.

Discussion

Our study provided evidence on the reliability and interpretation of the K6 score. The K6 is one of the most widely used psychological distress scales in both clinical and research settings because it is simple.^4,5,33,35,36 That is, it can be answered in 2–3 min, and the result is easy to grade.^32,34 However, if the K6 cannot measure changes over time, it may not be useful in clinical practice. Therefore, it must identify changes in the patient’s condition even if with minimal changes in psychological distress. Calculating the SDC of the K6 score can help identify the significance of changes in psychological distress over time and can determine the efficacy of interventions for psychological distress.^41,44 In this research, the SDC of the K6 scores indicated that a change of at least 8.47 for SDC_ind and 0.15 for SDC_group is required to be 95% confident that the change in scores is caused by actual change rather than measurement error. Moreover, SDC_ind required a change of 35.3% of the full range of K6 scores, and SDC_group required a change of 0.6% of the full range of K6 scores. Since the K6 scores range from 0 to 24, SDC_ind may be perceived as a relatively significant measurement error. For PROs used in routine clinical practice, the measurement error must be extremely minimal.⁴³ This study found that the K6 score was more suitable at the population level (ie, clinical trials) than at the individual level. The large individual-level SDC values were common findings in self-reported questionnaires.⁵² However, several studies have reported small individual-level SDC values.^53–55 This result might be explained by the differences in the methods used to calculate SEM. Previous studies that reported large SDCs used the same method utilized in this analysis.^56–59 There are several methods used to calculate SEM. The inclusion of both systematic and random errors as part of the error variance can yield a higher SEM compared with methods that do not include both components.⁴³ Therefore, our SEM and SDC could have been higher. However, to distinguish between actual changes caused by treatment and measurement error, systematic error must also be considered as a part of measurement error.⁴³ Therefore, if the SEM is not calculated with consideration of account systematic errors, the SDC cannot be calculated accurately. Our SEM considered systematic errors and, thus, should reflect a more accurate SDC.

The test–retest reliability (ICC) was 0.73 (0.71–0.74). Since the acceptable ICC is > 0.7, the K6 had an excellent test–retest reliability.^43,48 Previous studies have assessed the test–retest reliability of the K6 score. The ICC of the Bangladesh version is 0.80, which is similar to our results.⁶⁰ The test–retest reliability of the K6 scores in other countries, including China and Iran, are also identified.^46,61,62 However, the Pearson or Spearman correlation coefficient was used as a measure of test–retest reliability in these countries. When evaluating test–retest reliability, it is necessary to consider not only random errors but also systematic errors.^43,45 However, the Pearson or Spearman correlation coefficient is not an extremely rigorous parameter for evaluating test–retest reliability because it does not account for systematic errors.^43,45 However, the ICC_agreement may indicate a more accurate test–retest reliability of the K6 score because it considers not only random errors but also systematic errors.

Internal consistency was evaluated using the standard Cronbach’s alpha method. In this study, the Cronbach’s alpha of the K6 score ranged from 0.94 to 0.95. The acceptable Cronbach’s alpha is 0.7 or higher.⁴⁵ Based on our results, the K6 score had an excellent internal consistency. This finding is similar to the previously reported Cronbach’s alpha (0.85–0.90) of the Japanese version of the K6 score.^34,35 The Cronbach’s alpha in several countries (Vietnamese [0.86], Arabic [0.81], Bangladeshi [0.87–0.88], Chinese [0.84], and Persian [0.87] versions) was similar.^{37,38,46,60,61} By contrast, Cronbach’s alpha increases with the number of items on the scale.⁴³ In fact, previous studies have shown that the Cronbach’s alpha of the K10 (Japanese version = 0.91, Persian version = 0.92) is higher than that of the K6 (Japanese version = 0.85, Persian version = 0.87).^34,46 Hence, the K6 assessed the same questions differently, and some items overlap.

This study had several limitations. First, it was challenging to calculate minimally important change (MIC) due to the study design. When interpreting changes in scores, particular attention should be paid to SDC and MIC. The MIC is the smallest change in score in the measured construct that the patient perceives as important.⁴³ To better interpret changes in PRO scores, both SDC and MIC must be identified. If the MIC is greater than the SDC, a change equivalent to the MIC can be considered as not only a measurement error but also a clinically important change.^43,52 However, if the MIC is smaller than the SDC, changes equivalent to the MIC are likely caused by measurement error.^43,52 Hence, it is difficult to distinguish the changed value from the measurement error. Therefore, higher-quality MIC studies must be conducted to better understand the significance of changes in K6 scores. Second, this study recruited participants from the panel members of a web research firm. This might have caused participant bias and other limitations. However, internet surveys can allow participants to answer questions easily due to anonymity.⁶³ Therefore, a more accurate response could have been obtained. Third, the study only included individuals whose native language was Japanese. Therefore, the generalizability of our results to other national populations is limited, and our findings should be replicated in different countries. Fourth, the response rate for this study was low (2.40%), which may have caused selection bias. Possible reasons for the low response rate include low motivation to participate in the survey and panelists may not have been aware of the invitation and did not respond to the baseline or second questionnaire. Therefore, the results of this study should be interpreted in light of the possible selection bias.

Conclusions

Our study provided evidence on the reliability and interpretation of the K6 score. Calculating the SDC of the K6 score can help identify the significance of changes in psychological distress over time and can determine the efficacy of interventions for psychological distress. Nevertheless, further studies must be conducted to assess MIC and SDC to better understand changes in K6 scores.

Ethics Approval and Consent to Participate

The research was approved by the Ethical Review Board for Medical Research Involving Human Subjects of Gunma University (Approval no. HS2020-168). All participants gave their informed permission. The data were solely utilized for research reasons, and all information collected from the participants was kept strictly private. All procedures were followed in conformity with all applicable rules and regulations based on the Helsinki Convention.

Author Statement

All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation, or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

Disclosure

None of the authors have any conflicts of interest to declare for this work.

Data Sharing Statement

Data cannot be shared publicly because of the restrictions of the ethics committee. Data are available upon a reasonable request to the corresponding author for researchers who meet the criteria for access to confidential data.

Additional information

Funding

This work was supported by Japan Society for the Promotion of Science (JSPS) KAKENHI Grant Number 19K19724 and 22K11111 (K.H). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.