Measuring process indicators and adverse events to assess the quality of care for inpatients with psychosis

Abstract

Background

Research into the quality of care in psychiatry is scarce. Data collection is falling behind that for other fields of medicine and therefore the opportunity to improve care is missed.

Aims

In this medical record study we aim to determine: (i) whether or not patients’ physical health indicators are assessed and pharmacological and behavioural treatment interventions applied; (ii) the incidence and nature of adverse events in psychotic inpatients.

Methods

Medical records of inpatients with psychosis admitted to psychiatric wards at Amsterdam UMC, location AMC, Department of psychiatry, were screened with a previously developed and tested two-step patient safety tool.

Results

Data of 299 admissions were included. Physical health indicators were not assessed in one-third of cases. Fifty-five percent of the patients were smokers but only 1% received an intervention. The family was actively involved in 43% of the cases. During 11,403 admission days, 235 adverse events had been recorded. The most frequent adverse event was adverse drug reactions (40%), which were mostly related to antipsychotic medication.

Conclusions

In conclusion, quality of care auditing is useful to prioritize areas that need improvement. Future research should focus on interventions to improve the quality of psychiatric care.

Keywords:

• Patient safety
• psychosis
• quality of care
• quality indicators

Introduction

The importance of quality and safety of health care is firmly established since 2000 by the reports “Crossing the quality Chasm: A new health system for the 21st century” and “To Err is Human” (Kohn, Corrigan, & Donaldson, 1999; Medicine, 2001). Both reports highlight six key elements of high-quality care: safe, effective, timely, efficient, equitable and patient-centred. Still, the impact of that message in psychiatry is very limited. The poor data infrastructure to measure quality of care in psychiatry might be one of the explanations for this backlog (Kilbourne et al., 2018; Pincus et al., 2007). Consequently, improvement opportunities are often missed (Kilbourne et al., 2018; Pincus, Spaeth-Rublee, & Watkins, 2011). The first important step to identify areas for improvement of care is an accurate measurement of quality of care (Kohn et al., 1999; Nath & Marcus, 2006). Particularly the dramatic mortality rates among patients with psychosis – mainly caused by cardiovascular disease – warrant greater attention (Olfson, Gerhard, Huang, Crystal, & Stroup, 2015). Patients with a psychotic disorder live up to 15 years shorter than the general population (Laursen, Munk-Olsen, & Vestergaard, 2012; Olfson et al., 2015).

Quality of care is a complex construct. The most frequently used framework to categorize the various measures is the Donabedian Framework, which describes quality indicators in the categories “structure”, “process” and “outcome” (Figure 1). The structure indicators reflect mostly organizational factors used to compare health care facilities on a system level (e.g. nursing-to-bed ratio). The process indicators, on the other hand, refer to what was done for or to the patient to improve his/her health and can be measured by the rate of physical assessments and evidence-based treatments such as antipsychotic medication or psychotherapy. Lastly, the outcome measures reflect the actual patient outcomes, for example, the level of improvement after treatment or unwanted outcomes such as premature death.

Figure 1. Quality indicators as assessed in current study (with the exception of structure indicators) based upon the Donabedian Framework.

Process indicators have been studied in a large sample of schizophrenia patients from the United Kingdom (Crawford et al., 2014). The researchers concluded that assessment of risk factors, such as weight and blood pressure, as well as treatment rates of these risk factors were far below standard (Crawford et al., 2014). Although process indicators provide a good picture of what was done for or to a patient, a broader scope is needed to oversee the actual outcome, for instance by combining process and outcome measures of inpatient care (McGrath & Tempier, 2003). An internationally recognized outcome indicator is the occurrence of adverse events (AEs) – AEs are defined as the negative unintended consequences of clinical care that led to injury, impairment, or other harm (Griffin & Resar, 2009; Kizer & Stegun, 2005). A previous safety study reported that severely mentally ill patients admitted to medical-surgical wards experienced a mean number of almost six AEs per hospitalization (Daumit et al., 2016). A first study into safety of hospitalized psychiatric patients reported that approximately one in five patients experienced a patient safety event (AEs or medical errors) (Marcus, Hermann, Frankel, & Cullen, 2017). To our knowledge, a study evaluating process and outcome indicators of psychiatric inpatient care for patients with psychosis is lacking.

We undertook a study to gain insight into the quality of care provided to psychotic patients admitted to psychiatric wards in Amsterdam UMC, location AMC, by combining different indicators to assess processes and outcomes in order to identify areas that need improvement. Specifically, we aimed at answering the following questions: (i) to what extent are physical health risk and status assessed and are pharmacological and behavioural treatment interventions applied (e.g. process indicators); (ii) what adverse events occur and what are their incidences; and (iii) what clinical characteristics are associated with adverse events (i.e. outcome indicators).

Methods

Setting and study sample

In this retrospective study, we included data of patients with a psychotic disorder discharged from three psychiatric wards of the academic hospital Amsterdam UMC, location AMC, Department of Psychiatry between January 2014 and October 2015; i.e. a high care unit, a medium care unit, and an early psychosis unit. Our catchment area covers an urban population in The Netherlands.

Patients were offered a multidisciplinary treatment program during admission (based on national guidelines (van Alphen et al., 2012)) that matches the illness phase, e.g. first episode or chronic. During treatment and depending on the progress, patients gained more autonomy and were stimulated to transfer from the high care unit to the medium care unit or early psychosis unit as quickly as possible. The treatment program consisted of a diagnostic phase and pharmacologically focused treatment by medical doctors and behavioural focused treatment by nurses, psychologists, movement therapists, occupational therapists, and social workers. The main goal of this program was to promote recovery in such a way that transfer to an ambulatory (home setting) was possible again. Alcohol drinking, cannabis use and the use of other drugs are prohibited during admission to our hospital. Smoking was allowed, however, inside three smoking areas in the building or in the garden of the wards.

Inclusion was restricted to patients diagnosed within the psychotic spectrum or bipolar I disorder with psychotic features, according to the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, 2000. This study has been submitted to the Medical Ethics Committee of Amsterdam UMC and was granted exemption of the Medical Research Involving Human Subjects Act (WMO, 1992). The Dutch Medical Research (Human Subjects) Act states that formal ethical approval is necessary when the study meets both criteria: (1) It concerns medical/scientific research; and (2) Participants are subject to procedures or are required to follow rules of behaviour (WMO, 1992). A medical record study does not fall under these last criteria and, therefore, the Medical Ethics Review Board of our institution decided that formal approval was not necessary (see Supplementary material). Besides, the data was used for quality improvement projects. Together, this implies that participants do not have to give consent to use their data anonymously in research and formal approval was not asked. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Anonymized data were used for analysis and reporting.

Definitions and procedure

Process indicators

Process indicators were selected based on a national audit program for schizophrenia in the United Kingdom (Crawford et al., 2014). These indicators are originally extracted from the NICE guidelines on schizophrenia (NICE, 2009) and therefore we selected only those which are also applicable according to the local multidisciplinary Schizophrenia Guidelines (van Alphen et al., 2012). A test or intervention was assumed not to have taken place if it was not documented (Crawford et al., 2014). The local guidelines dictate that each patient should be offered a basic somatic screening at intake. The percentage of patients not screened might thus reflect refusal or failure to offer screening. The following process indicators were determined: (i) Physical health risk factors: anti-diabetic medication, antihypertensive medication, lipid-lowering medication, cannabis misuse or abuse, alcohol misuse or abuse, tobacco use, polypharmacy, obesity reported as comorbidity; (ii) Physical health measures: glucose and lipids screening, blood pressure measurement, weight measurement; (iii) Treatment interventions during admission: evidence-based tobacco cessation intervention, consultation of a dietician, cognitive behavioural therapy initiated, social worker involved, occupational therapy, family involvement e.g. family attendance in psycho-education meetings, initiation of new antipsychotic medication. Family history of risk factors for cardiovascular disease was not consistently recorded and could therefore not be included. We collected the information from the relevant subsections of patients' files (e.g. glucose screening from lab results) and notes from all involved professionals such as nurses, psychologists, social workers, and medical doctors.

Adverse events

AEs were defined as the negative unintended consequences of clinical care that led to injury, impairment, or other harm (Griffin & Resar, 2009; Kizer & Stegun, 2005). Adverse drug reactions/event (ADRs) were defined as “a negative, unintended consequence of a medication that resulted in functional impairment or other significant harm” (Marcus et al., 2017). An ADR was assumed to be present if it matched one or more of three criteria: (A) an “always” list of reactions or symptoms, such as (benign) elevated liver enzymes; (B) medication stopped, held or additional medication was started due to an adverse reaction; and (C) impaired basic functioning (e.g. standing, walking, seeing, hearing, thinking, breathing) (Hermann, Marcus & Wiesel-Cullen). The definitions of all adverse events have been included as an appendix (Supplementary material) and described elsewhere (Marcus et al., 2017).

We determined the percentage of admissions with one or more AE, total number of AEs and the number of AEs per 1000 patient days. Furthermore, we determined the nature of AEs and characteristics associated with the occurrence of AE. Nine different types were distinguished (Supplementary material): fall, self-harm or other injury to self, sexual contact with other admitted patients, elopement, contraband on unit, patient assault (victim or perpetrator), medication error, adverse drug reaction and other non-drug patient safety events.

We identified AEs from narrative text in the patient files with the “Hospital Medical Record Data Collection Manual of Patient Safety in Inpatient Psychiatry” tool, based on the methodology of the Harvard Medical Practice Study (Marcus et al., 2017). This tool, tested and in-depth described in a previous study (Marcus et al., 2017), includes nine predefined types of AEs, ranging from assaults to ADRs (for an overview, see primary outcomes or Supplementary material) (Brennan et al., 1991). The patient files were reviewed in a two-stage method by three clinical reviewers: two supervised master students screened all files and the first author performed the final AE identification. All reviewers had been trained in audio-recorded training sessions presided by the creators of the tool (Hermann & Wiesel). To this aim, the three reviewers individually reviewed the same 20 training files and discussed the individual results in consensus meetings. Subsequently, ten extra training files were screened, after which the inter-rater reliability among the two screeners was determined. This was considered adequate (unweighted Cohen’s kappa was 0.69) (Hanskamp-Sebregts et al., 2016). The second stage, the actual identification of AEs from the patient files was conducted by a physician [JMV]; if necessary a psychiatrist [LdH] was consulted.

Statistical analysis

Characteristics of the study sample are presented as numbers and percentages for categorical data. Normally distributed continuous variables are presented as mean (standard deviation) or median (interquartile ranges) in case of non-normally distributed continuous variables. Process indicators are presented as rates (number of patients that had a risk factor, assessment or intervention divided by the total sample size). Frequencies and types of adverse events are listed as countable frequencies and percentages. The incidence density was calculated as the number of adverse events occurring per 100 admissions and as the number of adverse events per 1000 patient days. Characteristics of patients with and without AEs were compared using the chi-square test or Mann-Whitney test. A p-value of 0.05 (two-sided) was considered statistically significant. All analyses were performed in SPSS Statistics (IBM Corp, Armonk, NY, USA), version 24.

Results

Study sample

We included data of 299 admissions of 237 unique patients with a psychotic disorder. Fifty-eight percent of patients had been diagnosed with schizophrenia (Table 1). The total length of stay was11403 days; the median length of stay was 31 days (IQR 14-53).

Table 1. Study sample characteristics and associations with adverse events during admission (n = 299).

Variables	With adverse event n = 118 (39%)	No adverse event n = 181 (61%)	p Value
Gender, male/female, n (%)	79 (66.9)/39 (33.1)	121(66.9)/60 (33.1)	0.99
Age, median (IQR)	33 (23–39)	30 (23–42)	0.85
Diagnosis, n (%)			0.64
Schizophrenia	67 (56.8)	105 (58.0)
Psychosis NOS	23 (19.5)	36 (19.9)
Schizophreniform	6 (5.1)	12 (6.6)
Schizoaffective	15 (12.7)	18 (9.9)
Bipolar with psychotic features	7 (5.9)	7 (3.9)
Other psychotic disorders^a	0	3 (1.7)
Length of stay in days, median (IQR)	44 (27-66)	21 (10-43)	<0.001
Compulsory admission, n (%)	68 (57.6)	81 (44.8)	0.04
Seclusion during admission, n (%)	22 (18.6)	13 (7.2)	0.003

^a Other psychotic disorder can be specified as brief psychotic disorder and psychotic disorder due to medical condition. NOS: Not otherwise specified; GAF: global assessment of functioning; IQR: interquartile range. Significant findings are shown in bold.

Process indicators

Rates of process indicators are shown in Table 2. The most frequent physical health risk factors were nicotine use (55%) and cannabis abuse or misuse (28%). Physical health status had been assessed in 75% of the patients whose laboratory results had been documented and in 69% of the patients whose weight measurements had been documented. One percent of patients who smoked had received an evidence-based smoking-cessation intervention during admission. Occupational therapy was the most frequent treatment intervention (70%). The family was actively involved in 43% of the cases.

Table 2. Rates of process indicators in admissions (n = 299).

Physical health risk factors	Frequencies (%)
Tobacco use	55
Cannabis misuse or abuse	28
Alcohol misuse or abuse	8
Anti-diabetic medication	6
Polypharmacy^a	6
Obesity as comorbidity	6
Antihypertensive medication	5
Lipid lowering medication	2
Physical health measurements
Glucose and lipids screening	75
Blood pressure measurement	71
Weight
At least one measurement	69
(Range in kg)	(37–137)
Treatment interventions during all admissions
Occupational therapy	70
Initiation of new antipsychotic medication	63
Family involvement	43
Social worker involved	25
Dietary consultation	10
CBT initiated	4
Tobacco cessation in smokers	1

kg: kilograms; CBT: cognitive behavioural therapy.

^a Polypharmacy was defined by five or more medications.

Adverse events and associated characteristics

We found a total number of 235 AEs in 118 admissions, corresponding to 21 AEs per 1000 patient days (Table 3). The highest number of AEs recorded per admission was eight. One or more AEs had been documented for 39% of all admissions. The following admission characteristics were associated with AEs: length of stay (p < 0.001), compulsory admission (p = 0.037) and seclusion during admission (p = 0.003) (Table 1).

Table 3. Types and frequencies of adverse events (n = 235).

Type of adverse event	n (%)	n per 100 admissions	n per 1000 patient days
1. Adverse drug reactions of which:	93 (40)	31	8
Induced by SGA	50
Induced by FGA	41
Induced by other medication^a	2
2. Patient elopement	40 (17)	13	4
3. Assault	31 (13)	10	3
4. Medication error	27 (11)	9	2
5. Contraband on unit	26 (11)	9	2
6. Other patient safety events^b	9 (4)	3	<1
7. Sexual contact	5 (2)	2	<1
8. Self-harm or injury to self	4 (2)	1	<1
9. Patient fall	0	0	0
Total	235 (100)	79	21

SGA: second generation antipsychotic; FGA: first generation antipsychotic.

^a Other medication = mood stabilizer and somatic medication.

^b Patient readmission within three days or not allowed smoking at wards.

Most of the AEs were adverse drug reactions (40%), followed by elopement (17%) and assault (13%) (Table 2). The antipsychotic medication was switched in 47%, the dose was reduced in 18%, additional medication was started in 25% and treatment remained unchanged in 10% of the adverse drug reactions. One adverse drug reaction, classified as a post-olanzapine injection syndrome, resulted in severe harm (i.e. resuscitation with subsequently good outcome). Self-harm (n = 4) was the least frequent AE. No falls, suicides or suicide attempts had been documented.

Discussion

In this study, the quality of inpatient care for patients with psychosis was evaluated by combining process and outcome measures, expressed as adverse events. Strikingly, physical health assessment was missing in one-third of the patients and therewith potential treatment interventions could have been missed. Although most patients smoked, only 1% of smokers received an evidence-based smoking cessation intervention. With respect to treatment interventions, in only 43% of cases, the family was actively involved in the treatment. Adverse drug reactions were the most frequent type of adverse events in inpatients with psychosis and these had been mainly induced by antipsychotic medication.

Process indicators

Regarding the performance on process indicators, a national audit of schizophrenia patients in the UK showed, for example, that physical health indicators were measured in 51% of the patients having their BMI being assessed during the previous 12 months (Crawford et al., 2014), compared to 69% our patients. The UK sample consisted of both in- and outpatients, which might perhaps explain the discrepancy with our findings. The most plausible explanations for the unmeasured physical risk indicators in the current study are that patients refuse (parts of) the examination, that measurements are not performed or that measurements are performed but not documented. Unfortunately, based on the available data, we cannot distinguish between refused, not performed or not documented physical examinations. Nevertheless, we believe that hospitalization is a window of opportunity to perform somatic screening – especially in severely mentally ill patients, given their poor physical health, and their limited access to medical care (De Hert, van Winkel, Silic, Van Eyck, & Peuskens, 2010).

Tobacco use is an undisputed risk factor for early death. A previous study showed that adults with schizophrenia were almost 10 (SMR 9.9, 95%CI 9.6–10.2) times as likely to die from chronic obstructive pulmonary disease (COPD) than adults from the general population (Olfson et al., 2015). This number should be a strong argument against the old-fashioned view that treating tobacco addiction might not be one of the priorities during the treatment of acute psychosis. We argue that treating addiction to smoking is as important as recovery from psychosis and that evidence-based cessation interventions should be implemented into current care paths. There is sound evidence on achievement of smoking cessation in individuals with schizophrenia (Cather, Pachas, Cieslak, & Evins, 2017). Implementation of treatment programs in combination with smoke-free hospitals may eventually help to reverse the dramatic reduction in life span of patients with psychotic disorders (Tiihonen et al., 2009). Our setting is unlikely to be an exception when it comes to the lack of support of patients who aim to quit smoking in The Netherlands. Dutch psychiatric hospitals often lack a clear policy regarding smoking (Blankers, Buisman, ter Weijde, & van Laar, 2015). It would be valuable to monitor whether implementation of stop smoking therapies in patients in the form of nicotine replacement therapy, behavioural support or pharmacotherapy is successful. Besides, research showed that prohibiting smoking in psychiatric hospitals was associated with reduced rates of aggression, which might further motivate hospitals to change smoking guidelines (Robson et al., 2017).

Finally, there is substantial evidence that active family involvement improves the care for patients with psychosis, for example with respect to adherence to antipsychotic medication (Glick, Stekoll, & Hays, 2011). In the present study, family was actively involved in 43% of cases. Previous research found that the implementation of active family involvement often falls behind due to the paradigm shift in professionals from contact with to working with family (Eassom, Giacco, Dirik, & Priebe, 2014). Although the early psychosis unit in the current study organizes regular family meetings and offers psychoeducation and training to the family members of patients, the acute wards have not yet implemented this and this could be a plausible explanation for the relatively low numbers.

Adverse events

In this study, the occurrence of AEs (31 per 100 patient discharges) was slightly higher than in a recent study (28 per 100 patient discharges) in 40 acute psychiatric units from medical centres in the national Veteran Health Administration (VHA) system (Marcus et al., 2017). In the present study, the occurrence of AEs was associated with longer length of stay, seclusion during admission and compulsory admissions were related with. Longer length of stay and seclusion can also be a result of an AE, for example in the case of assault. Prevention of AEs, especially in compulsorily admitted patients, could, therefore, result in a shorter length of stay, and in turn, further reduce the occurrence of AEs (Vermeulen et al., 2018).

Adverse drug reactions occurred more than twice as often as any other AE. In many cases, these were caused by antipsychotics use, but reactions varied, such from weight gain, movement disorder to akathisia (Leucht et al., 2013). A previous study on AEs during psychiatric hospitalizations also found that adverse drug reaction was the most frequent AE (Marcus et al., 2017). A reason for the high frequency of adverse drug reactions might be the precarious balance between efficacy and tolerability (Leucht et al., 2013). In our opinion, adverse drug reactions caused by antipsychotics must be systematically evaluated in every patient-physician contact. The severity of an adverse reaction can be assessed by instruments such as the Barnes Rating Scale for Akathisia (Barnes, 1989), and if necessary, dosages should be adjusted.

The limited number of AE to somatic medications is somewhat surprising. We can only speculate why this should be so: somatic medication might have been little used or clinicians paid little attention to side-effects of somatic medication. The high frequency of elopements (17% of all AEs) is mostly explained by patients returning substantially later than agreed or not returning from temporary leave. As these events might endanger the safety of the patient and its environment, these were also scored as an AE. Temporary leave is included in the treatment program as a means of regaining autonomy but it is subject to the risk of harm. Lastly, interventions aimed to reduce aggressive behaviour that may lead to assault form an important quality improvement area. There are several interesting developments in the field, such as Safewards, which is a randomized controlled trial was found effective in reducing harmful situations (Bowers et al., 2015).

Research findings and implications

Despite the efforts of international committees to introduce the valid and reliable process and outcome measures (Addington et al., 2012; Hermann et al., 2002), improvement of the quality of care for psychotic patients so far is hindered by the lack of data (Pincus, 2010). A Cochrane review from 2012 showed that an audit of performance indicators and feedback of this data to care providers can improve quality of care (Ivers et al., 2012). The feedback to providers maybe even more effective when: baseline performance of providers is low, the source of feedback is a supervisor or a colleague, feedback is provided more than once both in verbal and written formats and when it includes explicit targets and an action plan (Ivers et al., 2012). Quality outcomes may well serve to improve care paths – with for example “Plan Do Study Act-cycles” (PDSA) (Etchells, Ho, & Shojania, 2015). A PDSA-cycle can guide quick improvement of a quality gap, also when working with small sample sizes (Etchells et al., 2015). Based on the results of an audit, patients and professionals should prioritize together which areas need the first focus for improvement. Our study shows that retrospective data collection from electronic medical records can be used to audit process and outcome measures without an additional burden to patients or clinicians; it provides easily interpretable results and does not intervene with the care process. On the other hand, this approach is labor-intensive as long as machine-learning strategies to identify events, such as a tool that identifies extrapyramidal side effects, are not yet used (Iqbal et al., 2015). Future qualitative research is needed to triangulate the views of health care professionals and patients regarding explanations and how to improve measurement of clinically important process and outcome indicators.

Limitations

The results of the current study should be interpreted in light of the following limitations. First, the retrospective design could have induced measurement error. Recall bias may have influenced the results since clinicians could have omitted to report AEs, risk assessment or treatment interventions. Consequently, frequencies might have been underestimated. Still, tools that use predefined outcome measures to identify AEs retrospectively as used in the current study yield at least ten times more AEs than a conservative method like voluntary reporting (Classen et al., 2011). Nevertheless, using standardized medical records that address clinically meaningful processes of care could improve data collection. Second, although the inter-rater-reliability was adequate, differences in assigning AEs or process indicators may have influenced the results. Third, we chose to measure process and outcome indicators (e.g. AEs) over the limited period of a patient’s admission. As recommended by the Organization for Economic and Community Development’s Health Care Quality Indicators Project (OECD-HCQI), inclusion of additional long-term outcome indicators such as mortality and re-admission after inpatient care is preferred (Hermann et al., 2006). Fourth, we assumed that a test or intervention had not taken place if it was not documented (Crawford et al., 2014). However, patients with psychosis might suffer from a lack of illness insight and therefore might have refused assessments. As mentioned earlier, we could not distinguish between refusal and missed assessments. Fifth, we included chronic and first episode patients but did not distinguish differences between these groups because data were lacking, whilst chronic patients are more likely to suffer from health issues. Sixth, social factors such as unemployment or isolation could influence patients’ outcomes but have not been measured in this study. It could be valuable to incorporate this in future studies into this topic. Lastly, generalizability is limited because this was a single centre study. Particular characteristics, such as being allowed to smoke in designated areas, hinder the comparison with smoke-free hospitals in other countries.

Conclusions

In conclusion, this study is novel in that it presents data on the quality of processes and outcomes derived from medical records of psychotic patients, which can provide a solid base to start improving care. Having applied this approach, we conclude that patients diagnosed with schizophrenia or other psychotic disorders are prone to adverse events and especially to adverse drug reactions. High-quality care is needed because patients with psychosis frequently have somatic comorbidities and a huge excess of mortality. The overarching goal of measuring quality is improvement of care – and collecting quality data is an important means to achieve this. Future studies should focus on interventions to improve the care and outcomes of patients with psychosis.

Acknowledgement

We are very grateful for the help from Ko Hagoort in editing this manuscript. This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.

Disclosure statement

No potential conflict of interest was reported by the authors.