Identifying uncertainties about the effects of treatments for schizophrenia

Sometimes we just don't know

How often when sitting down to discuss treatment possibilities with a patient do clinicians think “I'm not sure of the best treatment for you”, or “I don't know which is the best treatment for this condition”? It probably happens quite often. Far less common, we suspect, is the clinician who sits down with a patient in a consultation and freely admits “I don't know” and then tries to work out the best answer with the patient. In this editorial we describe the Database of Uncertainties about the Effects of Treatments (DUETs) for schizophrenia and a way in which patients and clinicians might jointly influence priorities for clinical trials.

In situations where we don't know or when we face uncertainty, a modern evidence-based clinician has a range of resources more or less ready to hand from which to try and find some answers. Some may turn to Pubmed or the National Electronic Library. Others may look up a systematic review within the Cochrane Library, where the many, high quality summaries of the available controlled evidence wait to allow us to make our best guess. But, even then, the conclusion of the review can be, in fact often is, that there is little evidence to inform practice.

Where the evidence doesn't yet exist and there is uncertainty, one option is to turn to expert consensus. Sometimes where established consensus exists and research is subsequently undertaken to test that consensus the results can be very surprising in terms of challenging clinical orthodoxy. A famous example is the MRC Crash trial (CRASH Group, 2004). This RCT of the impact of intravenous corticosteroids on death within 14 days in 10,008 adults with clinically significant head injury showed that thousands of people had died whilst receiving a treatment based on consensus opinion and practice. Whilst the results of clinical uncertainty in mental health are unlikely to be as dramatic as the MRC CRASH Trial, the potential for great harm being inflicted on many patients has always been with us.

There are many examples within psychiatry. Insulin shock therapy, for example killed or severely damaged many people but was only abandoned when chlorpromazine was shown to be more effective (Fink et al., 1958). Prior to this expert consensus had identified it as a treatment of clear value (Sargant & Slater, 1944). An online version of Sargant and Slater's classic article is available at http://www.priory.com/homol/insulin.htm. The measured tone of reasonableness, backed up by the reassuring authorial certainty and the appeal to evidence with which this article is written make for salutory reading.

More recently, Ballard and Waite (2006) reported in a Cochrane Review that people with Alzheimer's disease who take antipsychotic drugs to reduce aggression or treat psychosis have a significantly higher incidence than controls of serious adverse cerebrovascular events (including stroke) and death. The use of anti-psychotics in this way had been based on consensus and extrapolating beyond the evidence. Expert consensus guidelines continue to be developed in a wide range of areas where uncertainty exists but such guidelines are perhaps not the best way of handling uncertainties (Milner & Valenstein, 2002). There can also be wide variation between expert guidelines as has been the case until recently with rapid tranquilisation. Only now are we moving towards developing an evidence base (Alexander et al., 2004; Belgamwar & Fenton, 2004; Gibson et al., 2004).

A further difficulty arises from the choice of outcomes used in studies. For example, if someone with schizophrenia were to ask a clinician about what treatment might best help them get back to work, the clinician would probably not be able to answer with confidence that treatment A is best, as many research studies fail to ask about such outcomes. Instead most studies focus on changes in rating scales, some of which have not been evaluated to see if they do in fact measure what they claim.

Measuring outcomes that matter

Outcome measures used in clinical trials are developed by clinicians and generally reflect their view of what is a significant improvement in a participant's mental health. Further, these outcome measures are usually rated by clinicians or other medical or research informants. This system may not reflect what is important to users in terms of what they value in improvement or recovery and how these valued outcomes are to be assessed. There are a few outcome measures developed by users but these are rarely used in clinical trials or other forms of systematic research (Thornicroft et al., 2002). This situation is compounded by what trialists see as the necessity to drive a trial by a primary, easily measurable outcome measure. One reason for this is the premise that only single measures can be used for power calculations. Secondary outcome measures may be included in a trial but these are assessed separately from the primary one.

In trials of medication, symptom improvement is usually the primary outcome measure and a secondary one may be side-effects. However, because of the trial design, symptom improvement is given primacy and the two outcomes are measured separately. So, it is not possible to say how users make trade-offs between positive and negative effects. Users may be quite prepared to live with some symptoms to avoid the toxic effects of some psychiatric medications. This may explain the discrepancy between efficacy, effectiveness and how an intervention operates in practice. In the context of a trial, those who stop taking a medication may not be counted in the final analysis. Further, in the context of a trial there are pressures to stay in the study and aspects of the study that may be welcome, for example, regular follow-up appointments with the opportunity to talk to an interested person. In the context of everyday life, users may well give up a medication because of its side-effects and this possibility has not been picked up because the trial design has not examined the effects of context and has not made it possible to examine trade-offs. Users are then labelled ‘non-compliant’ with the implication that they are doing something irrational which may, in fact, be based on a quite rational assessment.

The recovery literature does not argue for the total suppression of symptoms. Rather, users who count themselves “recovered” may continue to have quite severe symptoms. The main point is that they have found their own ways to live with these. This may involve collective support such as hearing voices groups or the Bipolar Organization's self-management programme. Such systems include the possibility of professional intervention, including medication and psychological therapies, but they place the user in control of the process. Such a complex way of managing recovery would require a change in the design of trials and the formulation of outcome measures from the user's perspective developed with users themselves. Professionals may be quite suspicious of the idea that people with enduring mental health difficulties could manage their conditions in this way and so we are likely to see a continuation of standard trial design and clinician-derived and assessed outcomes. We, nevertheless, would argue that this should change.

An example of how user-derived outcomes can provide different results to received wisdom is given in a systematic review of electroconvulsive therapy (ECT) (Rose et al., 2004). The empirical workers on this review had received ECT themselves and there was also a Reference Group made up mainly of people who had also received ECT. The review included first-hand accounts of ECT, called “testimonies”, which were analysed qualitatively. It departed from standard systematic reviews in another way in that the “grey” literature was included. This model is called “Patient-Centred Systematic Reviews”. Two of the outcomes examined in this study were perceived benefit and memory loss. The professional view is that patients are satisfied with ECT and that permanent memory loss is not a problem. This was the position in the Royal College of Psychiatrists' fact sheet on ECT at the time of the study although this has since been withdrawn. With respect to both perceived benefit and memory loss, the patient-centred systematic review came to different conclusions to that of mainstream psychiatry. High reported rates of perceived benefit in clinical academic papers were argued to derive from a flawed methodology. It was found that, although these papers did not conclude that memory loss was a problem, at least a third of participants in their own studies said that this was a significant side-effect for them. By analysing the qualitative material, it became evident that one of the reasons that participants would not have ECT again (if they had the choice) was precisely because of the side-effect of memory loss. Qualitative analysis may not yet be quite acceptable in the field of psychiatry but by combining quantitative and qualitative analysis, the review was able to analyse these trade-offs from the user perspective.

The arguments made above apply to all interventions and not just medical ones. These include psychological treatments and particular types of service provision. Indeed, the arguments are sharper with respect to complex interventions as these are even more difficult to trial in the usual way. They may also be regarded by users as having their own negative effects and in this case it may be more likely that user research will identify these. They include effects on self-esteem if users do badly on psychological tasks or frustration with certain alternatives to hospital admission if these mean never seeing the same member of staff twice.

It may be argued that user-valued outcomes should be identified and the best way of measuring these should be decided in collaboration with users. This may involve extracting scales or items from existing measures or it may mean the development of completely new scales.

Research on the effects of treatments too often fails to address questions that matter to patients, and to the clinicians to whom they turn for help. This is particularly important when one of the guiding principles for change identified by the research for patient benefit working group was the engagement of stakeholders through consultation and negotiation rather than representation (Department of Health, 2004). One solution is to build stakeholder engagement into the research agenda.

Informing the research agenda

To date there has not been any way of collecting and making available to clinicians, patients and researchers systematic data on what it is we don't know to assist in setting the research agenda. The US National Institute for Mental Health (NIMH) explicitly states on it's website that in order to set research priorities, it went through a series of inclusive stages guided by “relevance traction and innovation”. This process involved soliciting input from various stakeholders – patients and their advocates, physicians/scientists and their professional societies, Congress, and the National Advisory Mental Health Council (NAMHC), which includes public members. They requested workgroups of the NIMH Council to examine their portfolios in basic science and clinical trials and recommend priority areas for future investment.

In the UK a strategic analysis of UK mental health research funding was published in November 2005 by the Mental Health Research Funders' Group (Mental Health Research Funders' Group, 2005). The review provided a much needed national overview “map” of mental health research activity in the UK by taking a snapshot of activity in March 2004. The review was based on the sound principle that in order to work out where you want to end up it can often be helpful to find out where you are starting from.

Interestingly the review also highlighted areas about which information that could not be reliably collected. The authors note that it was difficult to ascertain whether ongoing research involved users, was relevant to carers and black and minority ethnic groups. Such information would be invaluable for demonstrating the inclusive nature of mental health research. This marked an important first step towards providing a broad thematic overview of research for general strategic analysis and planning of future mental health research in England. The natural outworking of that report within the UK National Health Service is for NHS funded research to become a managed research system (Clark & Chilvers, 2005; Szmuckler, 2005). But there are critics. For example, Lelliott (2005) voiced concern that it marks a move towards centralization and the stifling of research through bureaucracy. To some extent this negative perspective ignores the extent to which pharmaceutical and other interest groups already have considerable influence over the research agenda. More pertinent is the observation that a number of significant funders were not included in the analysis such as the Wellcome Trust, international research charities, the European Union, the US National Institute of Health, and the pharmaceutical industry. In their own ways each of these bodies has a major influence on shaping the research agenda. And to a greater or lesser extent involve both clinicans and patients in their priority setting.

The optimistic view is that exercises like the UK Mental Health Research Funder's Group may help to make the mental health research community more rather than less inclusive though the meaningful involvement of users in research (Clark & Chilvers, 2005). The UK Department of Health (2006) sets out a declared intent to include all professionals who have a role in conducting and enabling health research as both leaders and collaborators. The aim is to engage patients increasingly in the identification, design, recruitment to, and dissemination of research projects.

Arguably the processes undergone by both NIMH and the UK Mental Health Research Funder's Group and NIMH could be enriched and facilitated by the existence of a database of uncertainties drawn systematically from patients and clinicians.

The Database of Uncertainty about the Effects of Treatments (DUETs)

The recent appearance of the Database of Uncertainty of the Effects of Treatments (DUETs) marks the beginning of an effort to collect the known unknowns, or, as has been suggested, to begin the National Therapeutic Ignorance Service. This database sets the definition of uncertainty as occurring when “no up-to-date systematic reviews exist, or up-to-date systematic reviews show that uncertainty continues”. DUETs gives priority to identifying and publishing unanswered questions about the effects of treatments which have been asked by patients and clinicians, while also noting therapeutic uncertainties identified through systematic reviews, clinical guidelines, and other formal mechanisms. The DUETs project has been set up by the James Lind Alliance (JLA) named after a pioneer of clinical trials, James Lind. Two hundred and fifty years ago, there were many conflicting ideas and unanswered questions about how to treat the deadly disease scurvy. James Lind – a Scottish naval surgeon – decided to confront this uncertainty by treating his patients within a clinical trial comparing six of the proposed remedies. His trial showed that oranges and lemons were dramatically better than the other supposed treatments.

The JLA has been established to help identify and confront uncertainties about the effects of treatments considered important by patients and clinicians (Chalmers, 2004). The JLA will promote two principles: first, that addressing uncertainties about the effects of treatments should become accepted as a much more routine part of clinical practice; and second, that patients and clinicians should work together to agree which, among those uncertainties, matter most and thus deserve priority attention.

Specifically, the JLA will facilitate the identification of research priorities shared by patients and clinicians, hence its strapline – “tackling treatment uncertainties together”. This approach to identifying research priorities remains very rare. Most funding bodies consult professionals when they decide which areas of research to support, and sometimes patients and/or the public are involved in the design of particular projects. But few actively seek to establish which areas both professionals and patients agree require further investigation.

As part of the DUETs project a database of uncertainties about the effects of treatments for schizophrenia is being set up. It is being developed jointly by the James Lind Alliance and Mental Health Research Network Wales in a project founded by Welsh Office of R&D and led from Swansea University. The work is being taken forward in close collaboration with MIND Cymru and Hafal, meaning “equal” in Welsh, a charity dedicated to empowering people with severe mental illness and their families. Many more information sources will be needed to develop further the schizophrenia DUETs and more details can be found at: http://www.chiral.swan.ac.uk/ourresearch.html and http://www.lindalliance.org/.

The aim of collecting questions of uncertainty is to allow a research prioritization process to be undertaken, where those uncertainties which are shared by patients and clinicians are identified and prioritized. The process of prioritization is being developed with the James Lind Alliance. This may offer one solution to building meaningful stakeholder engagement into the complex process of setting the mental health research agenda.

We also need systems such as the DUETs exercise to help us work out what it is we don't know that matters. To this end the schizophrenia DUETs project is now underway. We wish to harvest as many questions about treatment uncertainties as we can. This is an international project and will use all sources of data wherever they are found. If you would like to get involved in the schizophrenia DUETs please contact [email protected]. For more general information about the DUETs project visit http://www.duets.nhs.uk/. Information in DUETs will help those responsible for promoting and supporting research to address important gaps in knowledge about the effects of treatments to take account of the information needs of patients and clinicians.