Service User Perceived Criticism and Warmth (SU-PCaW) Questionnaire

ABSTRACT

Background

The outcomes for a wide range of physical and mental health conditions can be influenced by the level of criticism and warmth in the family environment known in the literature as Expressed Emotion (EE). This finding is also true for individuals with an At-Risk Mental State (ARMS) for psychosis with individual’s perceptions of EE predicting transition to psychosis. While the original concept of EE incorporates both positive and negative aspects of relationships, most measures of EE and research focus on the negative aspects of relationships. Positive factors are also important and associated with improvements in functioning and reduced symptoms. However, the absence of a comprehensive, self-report measure of positive aspects of EE has hindered research in this area.

Methods

This study aimed to investigate the reliability and validity of a new 10-item Service User Perceived Criticism and Warmth Questionnaire (SU-PCaW) based on the 2-item Perceived Criticism and 2-item Perceived Warmth questionnaires.

Results

The measure was quick and easy to administer and had high internal consistency and reliability.

Conclusion

This measure would aid services in screening families for therapy, be helpful in evaluating family intervention in groups such as the ARMS population and a useful measure for future research developments.

KEYWORDS:

• Psychosis
• expressed emotion
• warmth
• family
• measure
• psychometrics

Introduction

Expressed emotion (EE) can be conceptualised as one of the most significant factors in psychosocial research in psychosis (Brown et al., 1972). The concept of EE captures the family environment reflected by a relative’s attitude, behaviours, and communication. The construct of EE comprises of negative aspects: criticism, hostility, and emotional-over-involvement (EOI) and positive aspects: warmth and positive remarks (Leff & Vaughn, 1985). The dichotomous EE index as “high” or “low” characterises families, with high-EE reflecting higher levels of negative aspects. Subsequently, excluding positive aspects. High-EE carer attitudes have been associated with symptom relapse and poor clinical outcomes in psychosis (Hooley & Campbell, 2002), whilst increasing when symptoms of psychosis become more apparent (Hooley, 2007).

Despite a focus on the negative aspects of EE, Butler et al. (2019) systematic review discovered evidence for the protective effects of warmth on relapse up to 9 months follow-up. Furthermore, the review found stronger evidence for the association of EE warmth and outcomes compared with EE positive remarks, particularly in the early phase of psychosis. Lee et al. (2014) found warmth acts as a stronger predictor for relapse in individuals at First Episode Psychosis (FEP) than criticism. Similarly, in the ARMS population, positive EE aspects (i.e. warmth and positive remarks) of family relationships have been found to predict increased social functioning and reductions in symptoms at 6-month follow-up (Butler et al., 2019; Izon et al., 2018). ARMS individuals may experience sub-clinical psychotic-like symptoms, a brief psychotic episode or have a family history, as well as impaired social functioning (Yung et al., 2005). Thus, exploring this group is particularly important for predicting transition to psychosis. A warm and positive relationship between caregiver and child is critical for social development and a development of the sense of a positive self (Bowlby, 1969; Stern, 2009). Within the context of mental health, warmth and positive affect from key relatives can act as a buffer and minimise symptomology in psychosis patients (Medina-Pradas et al., 2013), as well as predict life satisfaction (Butler et al., 2019). Understanding which individuals would most benefit from treatments is important. Service users with family member or other carers who display high-EE and low levels of warmth would be obvious targets for FI. The development and usage of a self-report measure with service users could be useful to services; supplementing other validated measures (e.g. Family Burden Questionnaire or Family Perception of Mental Illness Questionnaires) and continuing to provide a more holistic assessment of the home, family environment. Furthermore, it enables clinicians to adapt to the specific needs and intervention for families, for example, the need of single-family therapy, group family therapy or family psychoeducation.

There are several instruments that can be used to assess the family environment in terms of EE. These measures vary in how they emphasise different concepts within EE, how they are administered and to whom (see, Table 1). The gold standard measure for EE is the Camberwell Family Interview (CFI, Leff & Vaughn, 1985). This semi-structured interview is conducted with relatives (typically a parent or a spouse) and has high predictive validity in terms of relapse in people with a diagnosis of schizophrenia (Butzlaff & Hooley, 1998). However, the CFI can be labour-intensive: requiring 40–80 hours of formal training, each interview taking 1–2 hours to administer and another 2–3 hours to code (Hooley & Parker, 2006). An alternative methodology includes measuring service users’ own perceptions of the relatives’ EE. This may be more important for the ARMS population where service users’ perceptions of EE were deemed important in increasing risk of transition to psychosis (Haidl et al., 2018; Izon et al., 2021).

Table 1. Measures to capture expressed emotion (EE) in the family environment.

					EE components
Measure	Completed by	Interview/ questionnaire	Identifies high/low EE		Criticism	Hostility	EOI	Positive Remarks	Warmth
CFI	FM	Interview		✓	✓	✓	✓	✓	✓
FMSS	FM	Interview		✓	✓	✓	✓	✓	✓
FQ	FM	Questionnaire		✓	✓		✓
FQi	SU	Questionnaire		✓	✓		✓
FAS	FM	Questionnaire		✓	✓	✓
PC	FM, SU	Questionnaire			✓
PW	FM, SU	Questionnaire							✓
SAS-III	FM	Questionnaire		✓					✓
FPS	FM	Questionnaire		✓
LEE	SU	Questionnaire		✓✓	✓		✓

Abbreviations: FM = Family Member/Carer; SU = service user EE = Expressed Emotion; EOI = Emotional-Over-Involvement; CFI = Camberwell Family Interview; FMSS = Five Minute Speech Sample; FAS = Family Attitude Scale; FQi = Family Questionnaire for service users; FQ = Family Questionnaire; PC = Perceived Criticism Questionnaire; PW = Perceived Warmth Questionnaire; SAS-III = Social Adjustment Scale; FPS = Family Perceptions Scale; LEE = Level of Expressed Emotion.

Whilst several self-report measures do exist which capture service users’ perspective (e.g. The Level of Expressed Emotion instrument, LEE, Cole & Kazarian, 1988; Perceived Criticism, PC; Hooley & Teasdale, 1989; Family Questionnaire for service users, FQi; Izon et al., 2021; Perceived Warmth, PW, Schlosser et al., 2010), no measure assesses users’ perspective of both negative and positive aspects of EE. The current study aimed to firstly develop a new scale based on the original PW and PC questionnaires, the Service User Perceived Criticism and Warmth Questionnaire (SU-PCaW). Secondly, validate service users’ perspectives of EE over time.

Methods

Phase 1: defining the Service User Perceived Criticism and Warmth Questionnaire (SU-PCaW)

The creation of the measure involved the following stages:

1. The structure of the measure was conceptualized based on clinical experience and a literature review. Initially starting with the Perceived Criticism (PC; Hooley & Teasdale, 1989) and Perceived Warmth (PW; Schlosser et al., 2010) Questionnaires. These well-validated measures of criticism and warmth, respectfully, were used as a base for a more holistic EE measure. They were considered inadequate, as they separately assessed a component of EE, and did not account for EOI acting as a protective factor when exhibited at moderate levels and within the context of warmth (O’Brien et al., 2006; Schlosser et al., 2010).

2. A pool of items was developed. These items were created through a literature review of EE and different EE measures in the ARMS population (Izon et al., 2018), and expert clinical experience within the team. This involved discussions in meetings around the relevance of items focusing on negative and positive aspects. A pool of 10 additional questions was generated.

3. To assess content validity, four clinical and research experts and two people with lived experience were contacted via e-mail, informed of the aims of the measure, and individually consulted with regarding the language, items and further items that may be needed. Following this consultation exercise, changes were made to the order and wording of items, along with changes to the instructions (SU-PCaW measure).

Phase 2: assessment of psychometric properties

Participants

The new measure was embedded within the Individual and Family Cognitive Behavioural Therapy (IF CBT) trial for people at risk of developing psychosis (Law et al., 2019). Participants were randomly allocated to either combined Individual Cognitive Behavioural Therapy and Family Intervention (IFCBT) or enhanced treatment as usual (ETAU). Participants were recruited across two mental health trusts in the Northwest of England between August 2016 and August 2018. Participants were eligible to take part in the trial if they met the following inclusion criteria: aged 16–35, help-seeking, meeting criteria on the Comprehensive Assessment of At-Risk Mental States of psychosis (CAARMS; Yung et al., 2005) and living with or in regular contact with family, carers or loved ones (we use the term “family/carer” throughout this article). Exclusion criteria for the trial were: receipt of an antipsychotic medication to treat symptoms of psychosis, insufficient fluency in English, significant risk to self or others, organic impairment, and moderate-severe learning disability. The study received NHS ethical approval (REC reference 16/NW/0278).

Phase 3

We modified the measure based on findings from phase 2 (see, Figure 1 for final version of the measure). This study aimed to develop a new tool to assess self-reported EE by participants, based on the 2-item Perceived Criticism and 2-item Perceived Warmth questionnaires and investigate the reliability and validity of the new 10-item Service User Perceived Criticism and Warmth Questionnaire (SU-PCaW). The study aimed to assess the usage of the measure over time, to assess sensitivity.

Figure 1. Service User Perceived Criticism and Warmth Questionnaire (SU-PCaW).

Analysis phase 3

Data collected and entered onto IBM SPSS Statistics (version 27). Normal distribution of the data was found, and missing data (1%) were pro-rated with the mean for the SU-PCaW sub-scale.

An exploratory factor analysis was conducted with principal component analysis. This was used to systematically examine which linear components exist within the data and how a particular variable might contribute to that component due to the small sample. A two-factor structure was based on the conception of two subscales with additional questions added to the design questionnaire. Rotation Varimax was chosen as underlying factors should not be related and from the correlation no variable was larger than 0.32. Although Stevens (2002) says a sample N < 200 is poor, the Kaiser-Meyer-Olkin Measure for individuals was .736, highly significant (p < .001), suggesting the sample size was adequate (Hutcheson & Sofroniou, 1999).

Expressed emotion (EE) measures

To comprehensively assess the concurrent validity of the SU-PCaW measure, the authors measured EE after phase 3, using two other validated measures.

Five minute speech sample (FMSS)

Family/carers were asked to talk for up to five minutes, expressing their thoughts and feelings about their loved one, what kind of person they are and how they get along with them. Monologues were audio-recorded without interruptions. All of the FMSS were rated by a trained rater (the first author) using the FMSS manual (Magaña et al., 1986). Over 15% of the FMSS were independently coded by a second trained researcher and different interpretations were discussed. Inter-rater reliability was 100% for coding overall high/low/borderline EE, with reliability at 86.9% (S.E = .07) for other EE-components when controlled for KAPPA.

Family questionnaire for relatives (FQ) and service users (FQi)

The original Family Questionnaire (Wiedemann et al., 2002) is a 20-item self-report measure, compelted by family mermbners, which includes 10 items assessing criticism and 10 assessing EOI. Wiedemann et al. found componenets of criticism, EOI and overall EE ratings correlated significantly with the gold standrad EE measure, the Camberwell Family Interview (CFI). The adapted FQ for service users (FQi) asks individuals questions to assess their perception of the nominated caregiver’s critical comments and EOI. Izon et al. (2021) found significant positive correlation between the family/carer and individual overall EE score on the FQ at baseline (r = .456, p < .001).

Results

Forty-five participants (53% male) took part in the study. Further sample characteristics can be found in Table 2.

Table 2. A summary of the sample characteristics.

Variables of Interest	Service User N (%) or M (SD) N = 45
Age Mean	22.04 (5.18)
Range (years)	16–35
Gender (male)	24 53.3%
Ethnicity (N, %)
White British/ Irish	39 (86.7%)
Mixed	4 (8.9%)
Asian	1 (2.2%)
Black/Black British	1 (2.2%)
Treatment ^δ (IFCBT)	23 (51.1%)

Abbreviations. N = number, M = mean, SD = standard deviation, IFCBT = Individual Cognitive Behavioural Therapy and Family Intervention.

^δParticipants were randomly allocated to either combined Individual Cognitive Behavioural Therapy and Family Intervention (IFCBT) or enhanced treatment as usual (ETAU).

Service User Perceived Criticism and Warmth Questionnaire (SU-PCaW)

For service users answering the 10 items of the questionnaire, two factors explained 58% variance. Table 3 shows the correlation for each component. Although the sample was small (n = 45) there was higher correlation with new items on the questionnaire than Hooley and Teasdale’s (1989) two item questionnaire, therefore demonstrating higher concurrent validity.

Table 3. Principal component analysis for service user perceived criticism and warmth questionnaire (SU-PCaW).

Item	Component
	PW	PC
3*	.805
4*	.658
5	.821
7	.706
9	.871
1*		.823
2*		.715
6		.638
8		.815
10		.409

*Questions 1 and 2 already included from Hooley and Teasdale (1989) and questions 3 and 4 from Tsai et al. (2015).

Abbreviations. PC = Perceived Criticism, PW = Perceived Warmth.

Reliability

Internal consistency

Cronbach’s alpha for the SU-PCaW: PW α = .840 and PC α = .728, which indicates a high level of internal consistency for our scale.

SU-PCaW and other EE measures

Pearson correlation

There were moderate correlations between the new 5-item criticism subscale and criticism from the FQ (r = .572, p < .001) and overall EE score on the FQ (r = .490, p = .001). These correlations were stronger than Hooley and Teasdale’s (1989) 2-item PC questionnaire with FQ-criticism (r = .480, p = .001) and overall EE score on the FQ (r = .364, p = .013).

Baseline data of the FMSS reported high levels of warmth, which aligns with the SU-PCaW questionnaire, where high levels of warmth were reported (see, Figure 2). The FMSS indicated 34.8% family members in the high-EE category. The findings fit with the literature that approximately one third of ARMS relatives had high-EE, like individuals with FEP (Meneghelli et al., 2011; O’Brien et al., 2006; Schlosser et al., 2010).

Figure 2. Service user and family/carer mean expressed emotion (EE) respective of their group allocation at all three-time points.

SU-PCaW over time

Changes in warmth and criticism on the SU-PCaW can be seen in Figure 2. The overall mean EE scores showed that participants receiving IFCBT had lower scores of warmth and higher scores of criticism compared to the ETAU group at baseline, 6-months and 12-months. Mean scores of criticism were also the highest for the IFCBT group at the 12-month time point, whereas the lowest for the ETAU group at 6-months. Overall, mean scores for warmth were higher compared to levels of criticism at all time points in both IFCBT and ETAU groups.

Discussion

Assessing long-term EE and changes over time is useful for our understanding of symptoms and outcomes for individuals at-risk of psychosis and their families. It may be that the prevention of high EE over time can be crucial in understanding and supporting the health of family and carers themselves (Hinojosa-Marqués et al., 2020). The authors developed a quick, easy, and valid questionnaire for individuals to assess their perceived warmth and criticism in the family environment, the 10-item SU-PCaW questionnaire. Subscales were internally consistent and correlated with other measures of EE.

The correlation between the questions in the SU-PCaW questionnaire was smallest in question 10 (see, Table 3) – “How bothered are you by the critical comments your mother makes about you?”. The authors reflect how a participant’s answer would provide further understanding to the participant-carer relationship whether they were “not at all” or “extremely” bothered, however one could argue that this assumes critical comments have been made. The authors would add further clarification, “if reported experiencing critical comments, how bothered are you by the critical comments your mother makes about you?” This may aid services with their recommendations about the type of intervention and participant’s likelihood engagement.

This study developed a self-report questionnaire for service users to assess familial criticism and warmth from service users’ perspectives. This questionnaire included both positive and negative aspects of EE, highlighting the importance of positive aspects a family environment and potentially normalising conversations around familial criticism. Although the SU-PCaW may seem more subjective than methods such as the CFI, it allows participants to self-define their own perceptions without the fear of judgment from an interviewer. Some individuals with an ARMS may experience high-EE environments at home, in addition to high warmth. This measure could aid an open dialogue and involvement of service users, families, and clinicians to further explore the home environment and look to improve outcomes for service users and their families. Understanding the perceptions of the environment from the individual allows healthcare providers to create a more enriched formulation and be more attuned when to involve families and provide more systemic support.

Limitations

This was a small, convenience sample of those taking part in the IFCBT trial. The sample primarily consisted of white British parents and high-risk individuals and therefore may not be representative of other relationships, cultures and limits the generalisability of the findings. Furthermore, aspects and definitions of warmth may have different cultural interpretations and meaning (Deater-Deckard et al., 2011), therefore perceived differently in the self-report measure. Furthermore, self-report measures may be influenced by reporting bias from participants (Rosenman et al., 2011). The need for a more qualitative understanding of warmth may help to explore the definition and implications, as even the concept of criticism on its own can be rather complex (Medina-Pradas et al., 2013).

O’Brien et al. (2006, p. 274) argued that “the evaluation of only one key relative for each patient allows significant contributions to the family atmosphere to remain unaccounted for”. However, Haidl et al. (2018) allowed individuals to nominate their most important key relative and found perceived irritability the key predictor of conversion for ARMS to FEP. These studies looked at the experiences of Western participants, where most caregivers were identified as first-degree relatives, typically the mother. Therefore, the results may be different in other cultures or socio-ethnic backgrounds.

Future research should provide more focus on warmth as a protection for ARMS individuals and their families. It could potentially enhance the understanding of warmth within different cultural and social contexts, particularly with how warmth is perceived within the family (Deater-Deckard et al., 2011). It may also help clinicians to address the understanding of behaviours linked to warmth and how we can be more open and sensitive to the use of SU-PCaW with families. Possible interventions and therapies could focus on positives, such as carer warmth and the patient’s engagement towards this. A more positive, warmth- focussed approach could help to build patient-therapist rapport, facilitate communication, and change within the family unit. The authors recognise the limitations of the small sample and use of the measure in a feasibility trial. It would be of interest in future research to investigate the relationship of these subscales with further symptoms associated with psychosis, including the understanding of EE in different cultures.

Conclusion

In conclusion, the questionnaire assesses individuals’ perceived warmth and criticism in the family environment, the 10-item Service User Perceived Criticism and Warmth Questionnaire (SU-PCaW). The authors developed a quick and easy questionnaire that had high internal consistency and reliability. Studies of this nature are essential to identify families that need intervention in the prevention of family functioning deficits, the type of intervention most suitable, but also recognise the strengths they bring in terms of warmth and support.

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Acknowledgments

The authors would like to acknowledge the valuable contribution made to this study by Saba Hussain. The authors would like to thank all the young adults and their families who participated in the research trial.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the NIHR research for Patient Benefit [Grant/Award Number: PB-PG-1014-35075].