Psychometric analysis of the Swedish panic disorder severity scale and its self-report version

Abstract

Background: Panic disorder, with or without agoraphobia (PDA or PD, respectively), is a major public health problem. After having established a PD diagnosis based on the DSM or the ICD systems, the Panic Disorder Severity Scale (PDSS) is the most widely used interview-based instrument for assessing disorder severity. There is also a self-report version of the instrument (PDSS-SR); both exist in a Swedish translation but their psychometric properties remain untested.

Methods: We studied 221 patients with PD/PDA recruited to a randomized controlled preference trial of cognitive-behavioral and brief panic-focused psychodynamic psychotherapy. In addition to PDSS and PDSS-SR the participants completed self-reports including the Clinical Outcome in Routine Evaluation – Outcome Measure, Montgomery Åsberg Depression Rating Scale, Sheehan Disability Scale, Bodily Sensations Questionnaire and the Mobility Inventory for Agoraphobia.

Results: PDSS and PDSS-SR possessed excellent psychometric properties (internal consistency, test–retest reliability) and convergent validity. A single factor structure for both versions was not confirmed. In terms of clinical utility, the PDSS had very high inter-rater reliability and correspondence with PD assessed via structured diagnostic interview. Both versions were sensitive to the effects of PD-focused treatment, although subjects scored systematically lower on the self-report version.

Conclusions: The study confirmed the reliability and validity of the Swedish versions of PDSS and PDSS-SR. Both versions were highly sensitive to the effects of two PD-focused treatments and can be used both in clinical and research settings. However, further investigation of the factor structures of both the PDSS and PDSS-SR is warranted.

Trial registration: ClinicalTrials.gov identifier: NCT01606592.

Keywords:

• Panic disorder
• assessment
• anxiety
• psychometric properties
• factor analysis

1. Introduction

Panic disorder, with or without agoraphobia (PDA or PD, respectively), is a commonly occurring public health problem around the world [1].¹ This is also true for Sweden where the present study was carried out. According to the Swedish Council on Technology Assessment in Health Care [2], the point prevalence of PD is 2–3%, with PD being one of the most common causes of illness for Swedes between the ages 15 and 44 years. As elsewhere, PD is associated with high rates of comorbidity and functional impairment [3,4] and tends to take a chronic course, despite the availability of evidence-based treatments [2]. To reduce the high burden of illness associated with PD, it is important that individuals with the disorder are identified, offered treatment, and the severity of their symptoms carefully monitored [1,2].

The Panic Disorder Severity Scale (PDSS) [5] is the most widely used, clinician-administered measure of PD severity. The 7-item scale assesses the frequency of panic attacks, distress during panic attacks, anticipatory anxiety, agoraphobic fear and avoidance, body-sensation fear and avoidance, and impairment in work and social functioning on 5-point scales (0–4). In 2002, a self-report version of the scale was developed (PDSS-SR) [6]. The items are identical but the time frame for the ratings (past month for the interview and past week for the self-report) was changed.

To date, nine studies have investigated the psychometric properties of the scales, five on the PDSS [5,7–10] and three studies on the PDSS-SR [6,11,12]. A single study examined both scales [13] but used a 5-item version of the PDSS-SR that did not include the items on occupational and social consequences. Overall, these studies find that the PDSS has excellent internal consistency, high levels of inter-rater reliability as a screening instrument for panic disorder, good construct validity, and is sensitive to the effects of panic-focused treatments [5,7–10]. The PDSS has been translated into several languages (i.e. French, Italian, Japanese, Turkish, and Spanish) and modified for use with children [14]. Likewise, the PDSS-SR has been found to possess excellent psychometric properties in the English-language original [6] as well as in Korean and Spanish versions [11,12].

There are, however, some discrepancies reported for the underlying factor structure of the scales. Shear et al. [5] found a two-factor structure for the PDSS in their original validation but later studies found a single factor study [9,10]. In the two-factor structure, items 1 (panic frequency) and 2 (level of distress during panic) formed the first factor and items 3 through 7 (assessing worry about panic, avoidance, and work and social impairment) the second [5]. The authors interpreted the initial two-factor structure as a consequence of the validation sample excluding individuals with moderate to severe agoraphobia [9]. However, subsequent studies found a similar two-factor structure [7,8]. In the later study, Lim et al. [7] analyzed PDA and PD patients separately as well as together and found the same two-factor structure in both groups. Santacana et al. [12] found the same two-factor solution in their validation of a Spanish-language version of the PDSS-SR. Further investigation of the factor structure of the PDSS and PDSS-SR is warranted. The Swedish versions of the PDSS and PDSS-SR have been used in several studies [15–18] and are now widely disseminated in primary health care in Sweden. However, their psychometric properties have not been previously reported upon. Nordic translations and psychometric analyses of instruments used internationally are important and there are several notable examples of this effort [19–23]. The aim of this study was to evaluate the psychometric properties (including the factor structure) of the PDSS and PDSS-SR in a large, treatment-seeking sample of adults with a DSM-IV diagnosis of PD or PDA who were recruited to a randomized controlled preference trial of cognitive behavioral therapy and brief, panic-focused psychodynamic therapy (Project POSE [24]).

2. Methods

2.1. Sample

The sample was composed of 221 adults with PD or PDA who were recruited to a clinical trial (Project POSE) from nine outpatient psychiatric clinics located across three regions in Southern Sweden. Inclusion criteria for the study were (1) age between 18 and 70 years at the time of recruitment; (2) a primary diagnosis of DSM-IV [25] PD or PDA; (3) willingness to stop on-going psychotherapy treatments and refrain from starting non-study treatments during the treatment phase of the trial; (4) medications, if used, to be held constant during the treatment phase of the trial and for a minimum of 4 weeks prior to intake assessment; and (5) the ability to complete the active treatment phase (not including follow-ups) within 16 weeks of trial assessment. Individuals were excluded from the trial if they had substance abuse/dependence (past 12 months), autism, psychosis, mania, active suicidal ideation, clinically significant medical conditions (e.g. brain damage, degenerative neurological condition), or involvement in ongoing litigation regarding their mental health. All participants gave informed consent prior to participation in the trial. Ethical approval was obtained from the Regional Ethical Review Board in Lund (Ref. no. DNR-2010/88). Characteristics of the sample are presented in Table 1 (the number of observations may vary depending upon missing values).

Table 1. Sample baseline characteristics.

	Total (n = 221)
Demographics
Age at entry, years, M, SD	35 (12.6)
Female, n, %	165 (74.7)
Education, highest level, n, %
Basic level education	23 (10.4)
High school	116 (52.5)
University education	82 (37.1)
Employment, n, %
Employed	124 (56.1)
Self-employed	10 (4.5)
Student	51 (23.1)
Pensioner	2 (0.9)
Unemployed	19 (8.6)
Long term sick leave	7 (3.2)
Other	8 (3.6)
Current Psychiatric Conditions, n, %
PD with agoraphobia	184 (83.3)
PD without agoraphobia	37 (16.7)
Any Axis I diagnosis besides PD/PDA	156 (7.6)
Any Axis II diagnosis (personality disorder)	52 (23.5)
No. Axis I diagnoses besides PD/PDA, M, SD	1.7 (1.7)
Clinical characteristics
Panic history, months, Md, IQR	72 (144)
Panic episode, months, Md, IQR	10 (29)
PDSS, M, SD	15.6 (4.1)
Previous psychotherapy, n, %	136 (61.5)
Psychotropic use, n, %	117 (53.4)

2.2. Measures

The Structured Clinical Interview for DSM-IV (SCID-I and SCID-II) [26,27] was used to establish a PD (or PDA) diagnosis and possible psychiatric comorbidities. The SCID-I and II have been found have good psychometric properties and to be valid measures of psychiatric (including personality) disorders [28].

The Clinical Outcome in Routine Evaluation – Outcome Measure (CORE – OM [29,30]) is a 34-item self-rating scale, assessing subjective well-being (four items), symptoms (12 items), functioning (12 items), and risk/harm behaviors (six items). Each item is rated on a five-point frequency scale for the past week (0 = Not at all; 4 = Most of the time). A mean score for all items is computed; higher scores indicate greater levels of distress/dysfunction. The Swedish-language versions of this scale have been validated and found to have good psychometric properties [30]. Cronbach’s alpha (internal consistency coefficient) for the present study for the total score was .89.

The self-rating version of the Montgomery Åsberg Depression Rating Scale (MADRAS-S [31]) is 9-item measuring the severity of depressive symptoms over the past 3 d. Each item is rated on a seven-point (0–6) severity scale; higher scores indicate higher levels of depression. A total score is calculated with scores above 11 indicating mild to severe depression [31]. The Swedish-language versions of this scale have been validated and found to have good psychometric properties [31]. Cronbach’s alpha in the present sample was .84.

The Sheehan Disability Scale (SDS [32]) is a 3-item self-report measure that assesses the extent of functional impairment in work, social life, and the family over the past week. Each item is rated on an 11-point scale (0 = Not at all; 10 = Extremely). A total score is computed with higher scores indicate higher levels of panic-related dysfunction. A Swedish translation exists and has been in use in research in Sweden [16] but to the best of our knowledge without formal analysis of its psychometric properties. Cronbach’s alpha in the present sample was .75.

The Bodily Sensations Questionnaire (BSQ [33]) is a 17-item self-report measure of catastrophic interpretations of bodily sensations. Respondents rate the degree to which each bodily sensation causes them fear on a 5-point scale (1 = Not frightened; 5 = Extremely frightened). The Swedish-language versions of this scale have been validated to some extent and found to have good psychometric properties [34]. Cronbach’s alpha in the present sample was .86.

The Mobility Inventory for Agoraphobia (MI [35]) is a self-report instrument for measuring the severity of agoraphobic avoidance. Respondents rate their level of avoidance (1 = Never avoid, 5 = Always avoid) in 24 places/situations when accompanied (Avoidance Accompanied) and when alone (Avoidance Alone). Participants are asked to skip items of no relevance to them. A mean is computed for the items on the Avoidance Alone and Avoidance Accompanied subscales (separately), and it is these scores that are used for research purposes. The Swedish-language versions of this scale have been validated to some extent and found to have good psychometric properties [34]. In the present sample Cronbach’s alpha for the Avoidance Alone scale was .94, and for Avoidance Accompained was .94.

The SDS, BSQ and the MI have been used in earlier tests of the concurrent validity of PDSS and PDSS-SR [10,12,13], while the CORE-OM and MADRAS-S have not.

2.3. Procedure

Two hundred and twenty-one patients were included in the trial. Twenty-one patients were initially randomized to a sparse-contact control condition for 3 months before being re-randomized to the randomized or self-selection condition. Four patients dropped out from the control condition. The remaining 217 patients were offered treatment and randomly allocated to the randomized condition or the self-selection condition in the following shares: 110:107. Randomization was done sequentially according to a randomization protocol on each site as patients were included in the study. Treatments were delivered and data were collected between the years 2012 and 2018. A detailed presentation of the procedures in the study can be found in the published trial protocol [24].

The PDSS and PDSS-SR were administered at intake, at termination, and 6, 12, and 24 months after termination of treatment (in the present study only intake and termination evaluations are used). PDSS-SR was also completed by the patients at each week during treatment. For both scales, we analyzed the mean across all items. The first two authors (T. N. and M. S.) administered assessments with all patients. They divided the interviews between themselves on a geographical basis. To assess the inter-rater reliability of PDSS three additional assessors where trained to rate videotaped PDSS interviews. A sample of 264 videotaped interviews was randomly selected for this external rating of PDSS.

2.4. Data analyses

Internal consistency was assessed using Cronbach's alpha. Test-retest reliability was examined using product-moment correlations. Aspects of validity were evaluated with product-moment correlations between the PDSS or PDSS-SR (total scores) and the different measures administered (MADRAS, CORE-OM, SDS, BSQ, and MIA). Cohen's [36] criteria were used to evaluate the size of the correlations: 0.50–1 defined as “large”; 0.30–0.49 as “medium”; and from 0.10–0.29 as “small”. To assess the inter-rater reliability of the PDSS we calculated the intra-class correlation (ICC) between the interviewers (T.N. and M.S) and the ratings by the three trained assessors. Cicchetti’s [37] criteria were used for interpretation of inter-rater agreement estimates: less than 0.40 defined as “poor”, 0.40 and 0.59 as “fair”, 0.60 and 0.74 as “good” and 0.75 and 1.00 as “excellent”. To assess sensitivity to change we tested the differences between intake and termination using paired samples t-tests. Cohen's d [36] criteria were used to measure the within-group effect size: below .2 defined as “small”, around .5 as “medium” and above .8 as “large”. The factor structure was analyzed using confirmatory factor analysis (CFA) and exploratory factor analysis (EFA), including common goodness-of-fit indices (χ²; CFI; RMSEA; SRMR). Models with significant p values (p<.05) were refuted. Data were analyzed with the SPSS software for Windows, version 24, and Mplus (version 7.1; [38]).

3. Results

3.1. Intake and termination statistics

At intake, the PDSS interview mean score was 15.61 (4.14) and at termination 9.23 (5.29). The average PDSS-SR at intake was 12.43 (4.51) and at termination 5.84 (5.26).

3.2. Internal consistency

At intake, the PDSS internal consistency (alpha) was .74 and at termination .86. Internal consistency would not improve by deletion of any item whether at intake or termination. At intake, the PDSS-SR internal consistency was .80 and at termination .93. Neither would improve by deletion of any item.

3.3. Retest reliability and sensitivity to change

As estimates of test–retest reliability the correlation for the PDSS between intake and termination in the waiting-list group (assuming no treatment effect) at week 12 was r=.91. As estimates of test–retest reliability the correlation for the PDSS-SR between intake and week 1 was r=.70. The inter-rater reliability of the PDSS scores was estimated on a subsample of 264 interviews, resulting in ICC(2, 1)=.98 for the total score. Sensitivity to change was estimated as the mean difference between scores on the PDSS at intake versus treatment termination, which was 6.27 (SD = 4.79) and highly significant, t(192)=18.198, p<.001. The within-group effect size (Cohen’s d) for the PDSS was 1.33. For the PDSS-SR, the mean difference between intake and treatment termination was 6.43 (SD = 5.19), which was also highly significant, t(188)=17.042, p<.001, and a within-group effect size d = 1.56.

3.4. Comparison between the PDSS and PDSS-SR

At intake and treatment termination, significant differences were observed between the total scores on the interview-based PDSS and self-report PDSS-SR (intake = 3.18, t(186)=13.39, p<.001; termination = and 3.44 t(186)=15.91, p<.001). All items contributed to these differences, but especially item 2 with 1.42 at intake and 1.18 at termination.

The correlation between the PDSS and PDSS-SR at intake was r = .73 and at treatment termination r=.83. After correcting for attenuation these correlations were estimated as r = .95 and r=.93, respectively. At the group level, the correlation between the mean change scores on the PDSS and the PDSS-SR was r=.67. After correcting for attenuation, the correlation was estimated as r=.85.

3.5. Validity

Table 2 presents the pairwise correlations (and their 95% confidence intervals) between the PDSS and PDSS-SR (separately) and the self-report measures of overall symptoms/distress (CORE-OM), depression (MADRAS-S), bodily symptoms (BSQ), agoraphobic avoidance (MIA), and panic-related disability (SDS). As can be seen, all correlations were highly significant and in the moderate to large range.

Table 2. Total score correlations with 95% confidence intervals.

Measure	PDSS	PDSS-SR
PDSS-SR	.728** (.655–.790)	–
CORE-OM	.487** (.379–.585)	.492** (.373–.588)
MADRAS-S	.492** (.387–.589)	.532** (.437–.623)
BSQ	.322** (.196–.447)	.325** (.201–.449)
MI	.572** (.483–.651)	.485** (.369–.586)
SDS	.644** (.557–.718)	.628** (.530–.712)

** p<.01 (2-tailed).

PDSS: Panic Disorder Severity Scale; PDSS-SR: Panic Disorder Severity Scale-Self-Report; CORE-OM: Clinical Outcome in Routine Evaluation – Outcome Measure; MADRAS-S: Self-rating version of the Montgomery Åsberg Depression Rating Scale; BSQ: Bodily Sensations Questionnaire; MI: Mobility Inventory; SDS: The Sheehan Disability Scale.

3.6. Factor structure

A confirmatory factor analysis (CFA) of the PDSS intake ratings refuted the one-factor solution on account of its significant chi square, χ2(14, N = 221)=25.94, p=.026; CFI=.953; RMSEA=.062; SRMR=.041. We then tested the two-factor solution reported in previous research, with items 1 and 2 in a separate factor. It was not confirmed either, χ2(13, N = 221)=22.94, p=.042; CFI=.961; RMSEA=.059; SRMR=.039. An exploratory factor analysis (EFA) was then performed in order to identify some alternative factor model. One factor had eigenvalue >1 and another an eigenvalue close to 1, χ2(8, N = 221)=7.97, p=.436; CFI = 1.000; RMSEA=.000; SRMR=.024), and they accounted for 53% of the total variance. The inter-factors correlation was r=.55. No simple structure was achieved; item 4 (agoraphobic fear avoidance) alone had a loading of 1.0 on the second factor, whereas the rest of the items had loadings ranging from .35 to .62 on the first factor.

For the PDSS-SR at intake, a CFA again refuted the one-factor solution, χ2(14, N = 220) = 112.62, p<.001; CFI=.779; RMSEA=.179; SRMR=.075. Nor was the two-factor model (items 1 and 2 in a separate factor) confirmed, χ2(13, N = 220)=24.51, p =.027; CFI=.974; RMSEA=.063; SRMR=.039. In an EFA two factors had eigenvalues >1, accounting for 62% of the total variance, χ2(8, N = 220)=9.68, p=.288; CFI=.996; RMSEA=.031; SRMR=.021, whereas a three-factor solution did not converge. Geomin rotation of the two factors did not achieve a simple structure; items 1 (frequency panic attacks) and 2 (level of distress during panic) loaded 1.00 and .57 (respectively) on one factor and the rest of the items had medium-level loadings (.45–.77) on the other. The correlation between the two factors was r=.40.

4. Discussion

The Swedish versions of the PDSS and the PDSS-SR were found reliable in terms of internal consistency and test–retest reliability, and were also highly sensitive to the effects of two PD-focused treatments. Although the self-ratings (PDSS-SR) were consistently lower than the interviewer’s ratings (PDSS), the ranking of respondents was very consistent between the interviewer’s and self-ratings at intake as well as at treatment termination. Both the PDSS and the PDSS-SR correlated in the moderate to the large range with general measures of distress, depression, and panic-specific difficulties (i.e. bodily sensations, agoraphobic avoidance, and panic-related functional impairments). This suggests that both measures possess high levels of convergent validity. We conclude that the PDSS-SR in its Swedish version is an efficient, useful, and convenient way to evaluate treatment of PD, not as a substitute for the PDSS interview but as a complement facilitating frequently repeated assessments.

Just as in the original validation study of the PDSS-SR [6], we found that self-ratings of panic symptoms tended to be lower than when assessed via a clinician. Houck et al. [6] point out that this difference may be due to the interviewer over-rating the severity of patient symptoms on the PDSS. However, our impressions from the intake-assessments in the clinical trial from which the current data were drawn, was that patients with chronic PD will tend to underestimate or deny the severity of their symptoms and this is reflected more in the self than interview ratings. Almost half of the difference between the PDSS and PDSS-SR was accounted for by responses to item 2, referring to the severity of the panic attacks. A good clinician may help elicit a more forthright rating of the severity of the patient’s panic attacks. Nevertheless, the difference between the two versions of the PDSS was relatively small. The PDSS-SR may prove most useful as a weekly measure of change in therapy with the interview-rated version (PDSS) more suitable for pre- and post-treatment assessments as part of a wider assessment of psychiatric symptoms and functioning.

An interesting finding was that neither the PDSS nor the PDSS-SR, when administered at intake, yielded a single factor structure in this sample. The two-factor structure of the PDSS, with one factor loading on the first two items and another loading on items 3–7, was first reported by Shear et al. [5] and later confirmed by Monkul et al. [8] and Lim, Yu and Kim [7] – but not by Shear et al. [9] or Yamamoto et al. [10], who both observed a unifactorial structure for the PDSS. Shear et al. [9] interpreted the two-factor structure of the PDSS from their earlier validation study as a result of the sample only including individuals with mild levels of agoraphobic symptoms. This interpretation does not fit well with our results given the severity of agoraphobic avoidance in our sample. The factor issue was obviously not settled by this study and requires additional research. In this study, an unexpected factor structure was found with item 4 loading on one factor and the rest of the items loading on another. Likewise, for the PDSS-SR, CFA did not confirm the strict two-factor structure, although an EFA exhibited a similar, but not simple, two-factor structure. Our results for PDSS-SR resemble those of Santacana et al. [12] but differ from the single factor structure observed by Lee et al. [11]. It has long been recognized that differences in sample selection procedures can significantly impact the observed factor structures, and may partly explain differences in factor models obtained across studies [39]. Further investigation of the factor structures of both the PDSS and PDSS-SR is warranted.

5. Limitations

The diagnostic homogeneity of the sample may have influenced some of our findings through restriction of range. The inclusion of a non-PD sample would have been advantageous as well as analyses of the influence of comorbid diagnoses. The patient sample in this study was rather highly educated, which may have been due to the setting of part of our study in a university region. As with other self-rating instruments the PDSS-SR may not work as well in a sample with compromised reading comprehension.

Disclosure statement

No member in the research team declares any conflict of interest.

Note

Additional information

Funding

Project POSE was generously supported by grants from the Swedish Research Council for Health, Working Life, and Welfare; REHSAM/Swedish Social Insurance Agency; L. J. Boëthius’ Foundation; and the Regions of Skåne and Halland. Project POSE have received grants from FORTE, REHSAM, and L.J. Boëthius foundation during the study. In addition to this PhD student Thomas Nilsson has received grants from Region Skåne and PhD student Martin Svensson has received grants from Region Halland. Also fundings were provided by Forskningsrådet om Hälsa, Arbetsliv och Välfärd; L.J. Boëthius´ stiftelse.

Notes

1 Unless otherwise specified, the acronym PD is used to refer to individuals with and without a history of agoraphobia.