What is evidence? bridging the gap between trials and treatment implementation

How to bridge the gap between treatment and implementation into services has been discussed for many years and described in detail by Rand Europe (Wooding et al., 2014). For instance, why is family therapy for those with a diagnosis of schizophrenia only rarely used in services despite showing a high level of evidence (Fadden & Heelis, 2011; James et al., 2006)? Why is there a postcode (zipcode) lottery on prescribing certain medications or providing the number of psychological treatment sessions to have the hope of a reasonable benefit (MIND, 2019)? We could say it is resources, the people to provide the treatment, the background culture of a health service or just the biases of clinicians. There are many health service researchers who have spent countless years trying to understand these barriers which are often person-, service- or culture specific (Arnaez et al., 2020; Godier-McBard et al., 2022; Silva et al., 2022). Rather than reprising those findings this editorial will discuss a couple of ways that clinical academics might make it easier to move beneficial treatments into real world by changing how they evaluate them.

The best evidence

Most regulators demand evidence from randomised controlled trials so they are sure that they have the highest quality evidence of effectiveness or efficacy of interventions. Personally, I am not swallowing a pill unless it has received this sort of scrutiny. It is used to make decisions about what health professionals can prescribe or that health services will purchase. These trials are designed to be fair studies with an equal chance of finding a benefit or not, and that means that those running or participating should have clinical equipoise (no specific bias), which is very easy to say, but much harder in practice. Allocation to the new treatment or service should be random and carried out so that the process cannot be interfered with and no-one assessing the benefits should know which arm the person is allocated to. These are the general design “rules”, but they are not always adhered to, and so despite providing the highest status evidence, they can still be subject to bias because they have used high- or low-quality methods.

What affects regulator decisions?

Regulators take notice of the comparison treatment and generally prefer placebo treatment that accounts for the non-specific effects of treatment so that the key ingredient of the novel treatment is assessed. But placebos are almost impossible in psychological treatments or service interventions because they can provide a benefit (Bernstein & Brown, 2017; Szabo, 2013; Zhao et al., 2015). Some have also argued that placebos are unethical in the treatment of mental health difficulties (especially when standard care is beneficial), and that the placebo effect is complex (McQueen et al., 2013). Comparisons with treatment-as-usual can provide evidence of whether the new treatment provides a similar benefit but is cheaper. Less favoured is a wait list control as individuals will know that they will receive treatment later and so this may affect whether they are likely to improve naturally. However, in mental health services a waitlist control often means that the person receives absolutely no treatment for a considerable time, and it is problematic when there are issues of safety.

These methodological issues affect the interpretation of the strength of the results for ease of implementation, but the key to moving a novel treatment quickly from a trial to a service is the choice of the primary outcome. This is the outcome that captures improvement. It is often set by the research funder or the regulator or, in my experience, by the research team. This outcome is usually an assessment used in similar studies and is a valid and reliable measure that provides confidence that any noticeable improvements are real and not just an error of measurement. The current choices of primary outcome are not universally the ones chosen by either those with lived experience or those who provide mental health services. The valued outcome is affected by the type of mental health service, the diagnosis or set of symptoms, and the predominant culture. Any outcome needs to be acceptable for the group or groups it is intended for if the study results are going to overcome any staff or user resistance.

We already know that some trial designs may or may not be feasible or acceptable as most funders will expect that information in any application. But we (and the funders) rarely consider whether the choice of primary outcome is appropriate. In the next few paragraphs I want to examine what treatment acceptability might mean from the perspectives of those will receive it and those who will usually provide it.

Maximising treatment acceptability

There are many reasons why treatments that have shown benefit fail to be adopted by services or are declined by those using them. One downside is any side effect, particularly any adverse or serious adverse events. Drug treatments have always measured these effects and more recently tests of psychological treatment have also begun to log them. These are usually those effects associated with the treatment or the trial design that have harmful consequences and this monitoring could also pick up issues that will prevent a treatment being implemented. Those with lived experience are often more concerned with an unwanted effect that is not serious or life limiting, but that has a profound effect on day-to-day life and overall recovery. These specific concerns can also change over time and with stage of recovery. For instance, a slight shake of the hand might not be important until you want to use a smartphone regularly. What is needed is a measure that can account for this detail and is tied to the experience of people who use the novel treatment. Our team developed such a measure for antipsychotic medication in a mixed methods study. The final measure not only provides the opportunity to rate side effects by their frequency and the distress experienced, but service users can choose three of the most worrying ones in discussions about type and dose of medication with their doctor. This sort of measure would make a useful addition to defining the adverse effects of a treatment from the point of view of the participant rather than the academic researcher (Wykes et al., 2017). This example was for a drug study, but it is not beyond the wit of academics to develop similar measures for psychological treatment and service evaluations.

Who decides how to measure benefit?

It is hard to pinpoint what will be an accepted measure of outcome by regulators, funders or the academic community that assesses the final publication. One option is to ask the community that will be affected what treatment benefits might persuade them to adopt it into services. In the past many studies concentrated on improving global measures of symptoms for efficacy, although this might be important for clinicians this is not universally accepted by people with lived experience. Surveys and studies of acceptable outcomes have been identified in this journal and highlight these differences. For instance, Crawford and colleagues (Crawford et al., 2011) using nominal group methods to investigate the views of service users on twenty-four widely used outcome measures. The consensus was that many did not capture their experiences. Patient-rated measures were assessed as more relevant and appropriate than staff-rated measures, and concerns were raised about some widely used measures (e.g. Global Assessment of Functioning, European Quality of Life scale). This study shows that many global measures do not capture service user priorities and that we need to look much harder for those that will be acceptable to both service users and staff.

There is one example of a process of obtaining an acceptable measure in a recently published study of how cognitive remediation should be implemented (Wykes et al., 2023). This study adopted a measure that was acceptable to both clinicians and those who would receive the treatment. The study tested cognitive remediation (CIRCuiTS^TM) in first episode psychosis services in the UK NHS. The treatment improves thinking skills to improve functioning and one option would be to adopt some of the global measures that fared less well in the Crawford study. But the participants were mostly young people and they had very varied recovery aims so a measure of general function was unlikely to capture their disparate goals. The research team consulted clinical staff and those with lived experience about a trial outcome that would persuade them that CIRCuiTS^TM was worth investing in. Both groups said that it was whether the treatment helped people attain their personal goals. The research team then chose a measure with face validity and sound psychometric properties that is sensitive to change for individuals, perhaps more so than global measures. The measure is the Goal Attainment Scale (Kiresuk et al., 2014; Logan et al., 2022). It has the benefit of capturing heterogeneous personal goals and aspirations of those in early intervention services as some want to return to education, others were aiming for employment or even increasing their social activities. This method of consultation is likely to make the trial outcomes (whatever they are) more acceptable to those providing services. Now I admit that this is a project that I led and so I know that this outcome was not appealing to many of the reviewers of our final academic paper. Some suggested that it did not properly “measure psychosocial function”, others wanted the more usual measures and others did not accept it as it was a patient reported measure, despite many accepted outcomes being patient-reported.

Other methods of investigating acceptable outcomes include Discrete Choice Experiments (DCE) and Multi-Criterion Decision Modelling (MCDM). The DCE is a method borrowed from health economics that allows us to discover the preferences that service users and clinicians have for factors or outcomes by asking them to trade them off against each other. This method provides weights for the specific factors that can then be used to choose an outcome as the study is being designed. MCDMs are slightly different and less complex and can allow a retrospective view of the outcomes collected in a study. The various groups (service users, health care professionals, service providers and service payers) can all rate the importance of various outcomes, and these specific or overall weights can be used to understand the trial results in a different way that does not rely solely on the choice of a primary outcome. Both these methods are gaining traction and are recommended for further investigation by triallists (Khan et al., 2023; McCrone et al., 2021; Sotoudeh-Anvari, 2022).

In summary, the choice of primary outcomes for testing the effectiveness of treatments must consider the views of those who will receive the treatment and those who provide it. Often outcomes are specific and then a global measure might be used, but just as often, the aspirations of service users and the services they use are disparate and we need to be able to capture that variability. We can show services and service users that there is a gap in our treatment offers and we can show that there are treatments that are effective in resolving some problems, but if we do not have the right evidence that chimes with their preferences then we will still be waiting the 19 or so years found by the Rand Report for effective treatments to be implemented into services.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

Til Wykes acknowledges the support of NIHR Maudsley Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London. The views expressed are those of the author and not necessarily those of the NIHR or the Department of Health and Social Care.