Measuring the concurrent validity of the norwegian versions of the psychotic symptom rating scales (PSYRATS) and the positive scale from the positive and negative syndrome scale (PANSS)

Abstract

Purpose

The Positive and Negative Syndrome Scale (PANSS) is one of the most commonly used assessment tools for measuring psychotic symptoms. The Psychotic Symptom Rating Scales (PSYRATS) is another instrument created specifically to assess delusions and auditory hallucinations. However, research on the concurrent validity of PSYRATS with PANSS is limited. There are also inconsistent findings regarding the association between the PSYRATS scales and the PANSS positive scale. The present study aims to add to the understanding of the concurrent validity of these measures, while also incorporating a broader measure of psychiatric symptoms (the symptom scale from the Global Assessment of Functioning Scale – split version, GAF-S).

Materials and Methods

Spearman’s Rank Order Correlations (rho) were calculated for scores from the PANSS positive scale, PSYRATS and GAF-S in a sample of 148 participants with psychotic disorders at three time points.

Results

The findings indicate concurrent validity between PSYRATS and PANSS, while the PSYRATS scales were not consistently correlated with GAF-S.

Conclusions

PSYRATS may be a valid assessment tool for evaluating psychotic symptoms. The utility of PSYRATS in research and clinical practice should be investigated further.

Keywords:

• psychosis
• measurement
• symptoms
• PSYRATS
• PANSS
• GAF-S

1. Introduction

Several instruments have been developed to assess symptoms related to schizophrenia spectrum disorders (SSDs) (e.g. 1–3]). Such tools are important to evaluate treatment effects in clinical practice and research [4], especially considering the large personal and societal costs associated with SSDs [5]. One of the most commonly used measures for psychotic symptoms is the Positive and Negative Syndrome Scale (PANSS) [6] and the associated structured clinical interview (SCI-PANSS, translated to Norwegian by Bentsen, Notland and Munkvold) [7,8]. PANSS provides a total score, as well as scale scores indicating level of psychotic symptoms (positive scale), negative symptoms (negative scale) and general symptoms (general psychopathology scale) [6]. PANSS is considered the “gold standard” for assessing effectiveness in antipsychotic treatment trials, and several shorter versions have been developed [9–11].

While some have criticized PANSS for being too time-consuming [9], others have criticized the measure for not being detailed enough. One such critique is that the multidimensional nature of symptoms such as hallucinations and delusions are not fully represented [12]. The Psychotic Symptom Rating Scales (PSYRATS) [13] may contribute to more thorough assessment of psychotic symptoms. The instrument consists of two subscales measuring auditory hallucinations (the auditory hallucinations subscale, AHS) and delusions (the delusions subscale, DS).

Previous studies report significant correlations between the PSYRATS subscales the PANSS items assessing hallucinations and delusions (e.g. [12,14,15]). Their correlations with the PANSS positive scale are however seldom reported and findings are inconsistent [16–18]. The aim of the present study is to contribute to the literature by assessing the concurrent validity of the Norwegian version of PSYRATS and PANSS.

2. Methods

The presented data were obtained as part of the JUMP study in which 148 participants with psychotic disorders received a 10-month augmented vocational rehabilitation program. Participants were assessed at baseline, after 10 months and at a 2 year follow-up. Clinical assessments were made by trained and calibrated assessors who received supervision during the study. To ensure accurate and reliable symptom ratings, all assessors received PANSS training and underwent reliability checks before data collection began and during the study. Details about the study and the sample are reported elsewhere (e.g. [19,20]). The study was approved by the Regional Committee of Medical Research Ethics and the Norwegian Data Protection Authority. ClinicalTrials.gov Identifier: NCT01139502.

2.1. The Positive and Negative Syndrome Scale

Scoring of PANSS was based on the SCI-PANSS interview and was conducted at baseline, after 10 months and at 2 year follow-up to evaluate psychotic symptoms. PANSS has 30 items that are rated between 1 (absent) and 7 (extreme), with higher scores indicating higher symptom intensity. The positive scale includes seven items that measure delusions (P1), conceptual disorganization, hallucinatory behavior (P3), excitement, grandiosity, suspiciousness and hostility [6].

2.2. The Psychotic Symptom Rating Scales

PSYRATS was included in the test protocol 12 months after the study had already commenced to attain more thorough clinical evaluations of psychotic symptoms. The instrument contains a total of 17 items, distributed between the two subscales AHS and DS. AHS has 11 items that assess frequency, duration, location, loudness, and beliefs regarding origin of auditory hallucinations, as well as amount and degree of negative content, amount and intensity of distress, disruption to life, and beliefs about the controllability of auditory hallucinations. DS has six items that assess amount and duration of preoccupation of delusions, level of conviction, amount and intensity of distress, as well as disruption to life caused by these beliefs. All items are rated from 0 to 4, with higher scores indicating more symptoms/higher symptom related distress [13,21].

2.3. The Global Assessment of Functioning scale

In addition to PANSS and PSYRATS, the Global Assessment of Functioning (GAF) scale – split version [22] was used at all three time points. This is an instrument consisting of two scales that assess level of functioning (GAF-F) and psychiatric symptoms (GAF-S). Each scale is scored between 1 and 100 based on clinical descriptions, with higher scores indicating lower general symptom load (GAF-S) and higher functioning (GAF-F) [23]. GAF-F is outside the scope of the current paper, but GAF-S was included to provide a broader measure of clinician rated symptom load.

2.4. Statistical analyses

SPSS version 29.0 [24] was used for all statistical analyses. Spearman’s Rank Order Correlations (rho) were calculated to assess whether there were statistically significant associations between the PSYRATS subscales, the GAF-S, and the PANSS positive scale, and items P1 and P3 from PANSS. Analyses were conducted separately for the data collected at the different time points to prevent violating the assumption of independent observations. Correlation coefficients were interpreted according to the guidelines by Cohen [25], as reported in Pallant [26]. Hence, correlations are regarded as large for values .50 to 1.0, moderate between .30 and .49, and small for .10 to .29.

3. Results

There were moderate negative correlations between GAF-S and AHS at baseline and at the 2 year follow-up. There was a large negative correlation between GAF-S and DS after 10 months. There were large positive correlations between AHS and P3 at all assessment points. There was a large positive correlation between AHS and the PANSS positive scale after 10 months, but no significant correlations at the other time points. There were large positive correlations between DS and P1, as well as moderate to large positive correlations between DS and the PANSS positive scale, at all time points. Spearman’s rho and related significance values (p) are displayed in Table 1.

Table 1. Spearman’s Rank Order Correlations And related significance values for correlations between the Psychotic Symptom Rating Scales, the Global Assessment of Functioning Scale (symptoms) and the Positive and Negative Syndrome Scale.

	PSYRATS
	Baseline						After 10 months						After 2 years
	AHS			DS			AHS			DS			AHS			DS
	N	rho	p	N	rho	p	N	rho	p	N	rho	p	N	rho	p	N	rho	p
GAF-S	39	−.41**	.000	40	−.20	.222	24	−.17	.438	24	−.56**	.005	25	−.45*	.025	26	−.12	.564
PANSS
P1	38	.08	.635	39	.63**	.000	23	.45*	.029	23	.76**	.000	26	−.01	.977	27	.84**	.000
P3	38	.65**	.000	39	−.22	.185	23	.63**	.001	23	.37	.083	26	.72**	.000	27	.35	.071
Positive scale	38	.25	.133	39	.33*	.042	23	.50*	.015	23	.61**	.002	26	.19	.353	27	.56**	.002

Notes: PSYRATS: the psychotic symptom rating scales, AHS: the auditory hallucinations subscale from PSYRATS, DS: the delusions subscale from PSYRATS, GAF-S: the symptom scale from the global assessment of functioning scale (higher scores indicate lower symptom load), PANSS: the positive and negative syndrome scale, P1: the delusion item from PANSS, P3: the hallucinatory behavior item from PANSS, Positive scale: the PANSS positive scale score.

** : Correlation is significant at the 0.01 level (2-tailed).

*: Correlation is significant at the 0.05 level (2-tailed).

4. Discussion

The findings in this paper support the concurrent validity of the PSYRATS subscales and their related items in PANSS, as well as between the PSYRATS DS and the PANSS positive scale. There was however limited support for a relationship between AHS and the PANSS positive scale. The apparent discrepancy in the relationships between AHS and DS with the PANSS positive scale is corroborated by some previous studies [17,18]. Telles-Correia and colleagues [18] reported that the AHS accurately assess hallucinatory symptoms and suggested that it may provide additional information compared to the PANSS positive scale. This aligns with the proposal by Drake et al. [29] that PSYRATS may complement existing instruments with more detailed symptom assessment. Nevertheless, other studies have found significant correlations between the AHS and the PANSS positive scale [15,16]. The relationship between the two subscales therefore remains unclear.

There were some indications that higher scores on the PSYRATS subscales were related to higher levels of general symptoms. The lack of significance for half of the correlations between the subscales and GAF-S may be due to the influence of other symptoms associated with SSDs and/or comorbidity influencing the GAF-S score. There was no specific pattern regarding which PSYRATS subscale correlated significantly with GAF-S, and further research on the influence of specific symptoms on general symptom load may provide useful insight for clinical practice.

The use of data from several time points is a strength of the current study as it makes interpretation of the findings less vulnerable to coincidental correlations. Nevertheless, the lack of a clear association between AHS and the PANSS positive scale may also be related to the factor structure of the PANSS. Various studies support that other factor solutions may be more appropriate than the original positive, negative and general scales e.g. [27]. A meta-analysis conducted by Shafer & Dazzi [28] for instance suggested that some items from the positive scale better described other factors and that the item “Unusual thought content” (G5) should be included.

In conclusion, our findings support the concurrent validity between PSYRATS and PANSS. We recommend further investigation regarding the utility of PSYRATS in research and clinical practice, both regarding its potential usefulness as a supplement for assessment of psychotic symptoms and its validity compared with other scale versions or shortened versions of PANSS.

Acknowledgements

The authors would like to thank the participants, the clinical- and vocational staff for their time and effort devoted to the study.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This study was supported by The Norwegian Directorate of Health (08/9457), The Norwegian Labor and Welfare Administration, The South Eastern Norway Health Authority (9297) and The National Council for Mental Health/Health and Rehabilitation (2008/2/0310); Helse Sør-Øst RHF; Helsedirektoratet.

Notes on contributors

Olivia Schjøtt-Pedersen

Olivia Schjøtt-Pedersen is a clinical psychologist and researcher at the University of Oslo and Oslo University Hospital. She is also part of a network addressing employment as part of mental health treatment. Her research focus is mainly on severe mental illness and digital mental health interventions.

Helen Christine Bull

Helen Christine Bull is an occupational therapist and associate professor at Oslo Metropolitan University. Her research focuses on interventions in work and everyday life, vocational rehabilitation, and digital health solutions.

Erik Falkum

Erik Falkum is a senior physician and professor emeritus in psychiatry at the University of Oslo. He was the principal investigator of the JUMP study and has extensive experience with mental health research.

Torill Ueland

Torill Ueland is an associate professor of clinical neuropsychology at the University of Oslo and head of the cognitive research group at NORMENT, Oslo University Hospital. She was co-PI and responsible for the cognitive remediation intervention in the JUMP study. Her research focuses on neurocognition and psychosis.

Oda Skancke Gjerdalen

Oda Skancke Gjerdalen is a psychiatrist at Oslo University Hospital, Section for Early Psychosis Treatment, and a research fellow at the University of Oslo. She is currently working on a PhD thesis on Individual Placement and Support augmented with cognitive interventions.

Vegard Øksendal Haaland

Vegard Øksendal Haaland is the director at the clinic of mental health at Sørlandet Hospital and associate professor at the University of Oslo. His research focuses mainly on neuropsychology, cognitive neuroscience and clinical psychology.

Stig Evensen

Stig Evensen PhD, is head of the Norwegian Labour and Welfare Administration in Nesodden. His research focuses on vocational rehabilitation, severe mental illness and health economic evaluations.

June Ullevoldsæter Lystad

June Ullevoldsæter Lystad is an associate professor of clinical psychology and a senior scientist at the University of Oslo and Oslo University Hospital. Her research focuses on neurocognition and psychosocial interventions in psychosis such as Cognitive Behavioral Therapy, vocational rehabilitation, and cognitive rehabilitation.