Paradoxical outcomes in psychotherapy: Theoretical perspectives, research agenda and practice implications

ABSTRACT

Client self-report symptom measures are widely used in counselling and psychotherapy research and practice to track client progress and evaluate outcome. A growing body of evidence indicates differences between self-report assessments and information collected through other sources. This paper considers a range of theoretical and empirical perspectives on this issue, including the concept of illusory mental health. Particular emphasis is given to the relevance of single-case research as a means of identifying different patterns of paradoxical outcome. Implications for practice and research are discussed.

Zusammenfassung: In der Beratungs- und Psychotherapieforschung und -praxis werden häufig Maßnahmen zur Selbstberichterstattung des Klienten eingesetzt, um den Fortschritt des Klienten in der Therapie zu verfolgen und das Ergebnis zu bewerten. Eine wachsende Zahl von Beweisen weist auf Unterschiede zwischen Selbstberichtsbewertungen und Informationen hin, die aus anderen Quellen gesammelt wurden. Dieser Artikel betrachtet eine Reihe von theoretischen und empirischen Perspektiven zu diesem Thema, einschließlich des Konzepts der illusorischen psychischen Gesundheit. Besonderes Augenmerk wird auf die Relevanz der Einzelfallforschung als Mittel zur Identifizierung verschiedener Muster paradoxer Ergebnisse gelegt. Implikationen für die Praxis und die Forschung werden diskutiert.

RESUMEN

La cantidad de síntomas del informe del propio cliente, se utilizan ampliamente en la investigación y la práctica de asesoramiento y psicoterapia para realizar un seguimiento del progreso del cliente en la terapia y evaluar los resultados. Un gran número de pruebas, indica diferencias entre las evaluaciones de diagnósticos y la información recopilada a través de otras fuentes. Este documento enumera una serie de perspectivas teóricas y empíricas sobre este tema, incluido el concepto de salud mental ilusoria. Se hace especial hincapié en la pertinencia de la investigación de un solo caso como medio de identificar diferentes patrones de resultado paradójico. Se discuten las implicaciones en la práctica y la investigación.

Abstract

I self-report di valutazione dei sintomi sono ampiamente utilizzati nel conselling, nella pratica clinica e nella ricerca in psicoterapia per monitorare i progressi del cliente nella terapia e valutarne l’esito. Un numero crescente di evidenze indica differenze tra le valutazioni ottenute tramite I self-report e le informazioni raccolte attraverso altre fonti. Questo articolo prende in considerazione una serie di prospettive teoriche ed empiriche su questo problema, incluso il concetto di salute mentale illusoria. Particolare enfasi è data alla rilevanza della ricerca su “caso singolo” come mezzo per identificare diversi pattern di esito paradossale. Vengono discusse le implicazioni per la pratica e la ricerca.

Les mesures d’auto-évaluation du symptôme remplies par les clients sont largement utilisées dans la recherche et la pratique en psychothérapie et permettent à la fois de suivre les progrès thérapeutiques du client et d’évaluer le résultat. Un nombre croissant de preuves indique des différences entre les auto-evaluations et les informations recueillies par d’autres biais. Cet article examine un éventail de perspectives théoriques et empiriques sur cette question, y compris le concept de santé mentale illusoire. Un accent particulier est mis sur la pertinence de la recherche sur les cas cliniques uniques comme moyen d’identifier différents modèles de résultats paradoxaux. Les implications pour la pratique et la recherche sont discutées.

Περίληψη

Οι μετρήσεις αυτό-αναφοράς συμπτωμάτων των πελατών χρησιμοποιούνται ευρέως στην έρευνα και την πρακτική της συμβουλευτικής και της ψυχοθεραπείας, έτσι ώστε να γίνεται δυνατή η παρακολούθηση της προόδου των πελατών και η αξιολόγηση της έκβαση της θεραπείας. Ολοένα και αυξανόμενα ερευνητικά δεδομένα υποδεικνύουν διαφορές ανάμεσα σε μετρήσεις αυτό-αναφοράς και δεδομένα που συλλέγονται από άλλες πηγές. Η συγκεκριμένη μελέτη εξετάζει μια σειρά από θεωρητικές και εμπειρικές προοπτικές πάνω σε αυτό το θέμα, συμπεριλαμβανομένης της έννοιας της «ψευδαίσθησης» της ψυχικής υγείας. Ιδιαίτερη έμφαση δίνεται στη σημασία της έρευνας μεμονωμένων περιπτώσεων ως μέσο προσδιορισμού διαφορετικών μοτίβων παράδοξων αποτελεσμάτων. Συζητούνται οι ερευνητικές και πρακτικές εφαρμογές.

KEYWORDS:

• Case study research
• illusory mental health
• measurement
• paradoxical outcome
• psychoanalytic theory.

SCHLÜSSELWÖRTER::

• Fallstudienforschung
• illusorische psychische Gesundheit
• Messung
• paradoxes Ergebnis
• psychoanalytische Theorie.

PALABRAS CLAVE::

• Investigación en el estudio del caso
• salud mental ilusoria
• medición
• resultado paradójico
• teoría psicoanalítica.

PAROLE CHIAVE::

• ricerca a “caso singolo”
• salute mentale illusoria
• misurazione
• esito paradossale
• teoria psicoanalitica.

MOTS-CLËS::

• recherche sur études de cas
• santé mentale illusoire
• mesure
• résultat paradoxal
• théorie psychanalytique.

Λέξεις κλειδιά::

• Έρευνα μελέτης περίπτωσης
• ψευδαίσθηση ψυχικής υγείας
• μέτρηση
• παράδοξα αποτελέσματα
• ψυχαναλυτική θεωρία.

Introduction

The measurement of outcome represents a key issue in counselling and psychotherapy research and practice. Outcome studies on the relative effectiveness of different models of therapy have had a significant impact on policy-making within health-care systems, on the nature of therapist training, and on the direction of the profession as a whole. Many therapists use routine outcome monitoring as a source of feedback that informs the work that takes place with clients. Both empirically supported interventions and feedback-informed therapy are grounded in an assumption that the data produced by existing outcome measures are valid and can be trusted to provide a reliable guide for action.

The majority of therapy outcome studies and client feedback tools involve the use of client self-report symptom measures. These instruments provide a cost-effective, straightforward means of assessing the severity of a client’s difficulties. However, it seems clear that, at least in some cases, there are disparities between information generated by client self-report symptom measures, and data on the same clients obtained from sources such as interviews with clients, projective tests, and clinical observation. Lack of correspondence between different outcome indicators can be described as the issue of ‘paradoxical outcome’.

It is possible to identify several types of paradoxical outcome. Contrasting outcome results may be generated by different quantitative measures, or data from different points of observation (e.g. client, therapist, co-worker, etc.). Good outcomes on a quantitative self-report measure may be accompanied by a picture of poor outcome based on qualitative sources, such as an interview with the client, or therapist notes. Conversely, qualitative or clinical information may suggest a good outcome, while self-report data may indicate that therapy was unhelpful.

An important pattern of paradoxical outcome is associated with cases in which a client may be troubled, but nevertheless record a symptom score that is in the normal range. In this scenario, sometimes described as ‘illusory mental health’ (Shedler et al., 1993) changes in symptom scores over the course of therapy are difficult to interpret: does an increase in score (i.e. apparently more severe symptoms) indicate that the client is deteriorating, or that they are improving in the sense of being more willing to acknowledge their actual level of distress?

At the present time, relatively little research has been conducted into the prevalence of different types of paradoxical outcome, or how to make sense of this phenomenon. It is possible that paradoxical outcome is a rare phenomenon, with minimal implications for research and clinical practice. Alternatively, evidence that paradoxical outcome occurs in a significant proportion of cases would require substantial re-thinking around such questions as the overall effectiveness of therapy, the relative efficacy of different types of therapy, the use of symptom measures in intake decision-making, and the use of self-report measures in routine feedback monitoring.

The aim of the present paper is to stimulate awareness of the issue of paradoxical outcome, in order to encourage greater attention to this topic, and to preview the contribution made to this topic by articles in this Special Issue of the European Journal of Psychotherapy and Counselling. The paper introduces a range of theoretical and methodological perspectives that may be relevant to the task of making sense of paradoxical outcome. Within the space available, it has not been possible to undertake a detailed critical evaluation of each perspective. Instead, the intention has been to outline the nature and scope of different approaches to this issue, to inform future research and clinical practice.

Evidence for the existence of paradoxical outcome

Many years ago, one of the pioneers of psychotherapy outcome research, Hans Strupp, proposed that an adequate appreciation of the outcomes of psychotherapy required taking account of evaluation criteria from three quite different sources: society as a whole, the individual receiving therapy, and the expert opinion of the therapist or another mental health professional who has assessed the client or patient (Strupp & Hadley, 1977). In subsequent discussion of relevant sources of outcome evaluation, Lambert et al. (1992) went further, in suggesting that the external social perspective needed to incorporate the views of both significant others such as family and friends, and institutional sources such as the client’s employer or social worker. The implication of these evidence-source taxonomies was that there did not exist one single ‘true’ estimate of whether therapy had been effective for a particular client. Instead, external observers, health professionals, and the clients themselves were each operating from different value systems, and would inevitably adopt different positions regarding whether a particular course of therapy had been helpful. Strupp and Hadley (1977) fleshed out this assumption by describing eight distinct outcome profiles that might be expected if all sources of evidence were available. Some outcome studies carried out at that time sought to incorporate these principles. For example, in a study of counselling for young people with emotional problems, Hagborg and Konigsberg (1991) analysed outcome data collected from clients, therapists and teachers. What they found was that each set of raters had a significantly different estimate of outcome, consistent with their own interests and the information available to them. The findings of this study are notable because, in this instance, there were considerable opportunities for exchange of views between clients, teachers and therapists, which might be expected to have produced convergence between their respective outcome assessments. A more recent example of this type of research can be found in a study of therapy for depressed adolescent clients, conducted by Krause et al. (2020) in which outcome criteria identified through interviews with clients, their parents, and therapists indicated that each stakeholder perspective was associated with different evaluative schemata, which were in turn different from the criteria used in research studies that had investigated this type of therapy.

More recently, a review of outcome research reported that client self-report and therapist ratings are associated with different outcome estimates (Cuijpers et al., 2010). Evidence of paradoxical outcome has also been reported in multi-method single case studies in which data have been analysed from both self-report scales completed by clients and qualitative interviews (Desmet, 2018; Elliott, 2009). Qualitative studies that involved interviews with clients selected on the basis of good or poor outcomes, as indicated by change in self-report measures, have found themes in client personal accounts of satisfaction with therapy do not necessarily match outcome status (McElvaney & Timulak, 2013; De Smet et al., 2020; Stänicke & Killingmo, 2013).

The concept of illusory mental health

A pattern of paradoxical outcome that has generated a substantial research and theoretical literature is the phenomenon of ‘the illusion of mental health’ (Shedler et al., 1993). In a series of studies, university staff and students completed self-report symptom measures, assessment of autonomic arousal, and a narrative-based projective technique (the Early Memory Test). It was found that around 10–20% of participants recorded a low (i.e. healthy) score on symptom measures accompanied by indications of mental health problems in both the projective and physiological data. Shedler et al. (1993) suggested that these paradoxical findings could be understood as the expression of ‘illusory mental health’ arising from defence against awareness of threatening memories and emotions. Subsequent analysis of the Early Memory narratives produced by participants who exhibited a pattern of illusory mental health identified recurring themes of negativity and neediness, other people portrayed as sources of pain, punishment, frustration, fear and hurt, with caregivers being depicted as abandoning, unprotective and abusive (Shedler et al., 2003).

Shedler et al. (1993) concluded that their findings implied that “mental health scales assess different things in different people“ (p. 1138), with some individuals responding to items in a way that provided an accurate representation of their psychological problems, while others responded in a manner that conveyed an illusory impression of emotional well-being. They further proposed that such defensiveness may have health costs due to underlying psychological issues being presented as somatic symptoms. Links between illusory mental health and use of health-care services were observed by Cousineau and Shedler (2006) and illusory mental health has been identified in some patients receiving treatment for chronic fatigue syndrome (Bram et al., 2018, 2019).

The illusory mental health studies reviewed above relate to investigations carried out with non-clinical samples, or patients seeking assistance for physical health problems. Shedler (2006) believed that the level of denial associated with illusory mental health meant that such cases would be unlikely to be encountered in mental health and psychotherapy settings: ‘individuals with illusory mental health regard themselves as psychologically healthy and appear healthy to observers. They do not have psychiatric diagnoses’ (p. 229; emphasis in original).

It is possible that individuals who seek psychotherapy may have sufficient appreciation of their psychological problems, with the result that their responses to self-report symptom measures would be consistent with other sources of information. Nevertheless, there also exist data from large-scale therapy outcome studies that are consistent with an illusory mental health perspective. Naturalistic outcome studies of therapy, in which self-report data are available for all clients seeking help, typically find that at least 10% of clients record intake scores in the normal range (Østergård et al., 2019; Stiles et al., 2008). Ziem and Hoyer (2019) found that therapist ratings of client symptom severity at intake were higher than ratings made by the clients themselves.

At the present time, the ‘illusory mental health’ model offers a framework for making sense of cases of paradoxical outcome characterised by discrepancies between data from self-report symptom measures and other sources of information. Further work needs to be done regarding the prevalence of this pattern in psychotherapy practice, whether it emerges in all types of self-report measure (e.g. measures in which the client rates self-identified goals), its consistency over time and in different therapy settings and cultural contexts, association with type of presenting problem, and the conditions under which contrasting outcome sources may move into alignment over the course of therapy. Further research is also necessary to determine the implications of this perspective for therapeutic interventions with individuals exhibiting chronic health problems that may include a psychological dimension.

Theoretical perspectives

In order to guide future research and practice around the topic of paradoxical outcomes, it is necessary to consider additional ways in which this phenomenon can be conceptualised, in addition to the illusion of mental health model. The following sections provide overviews of some potentially relevant theoretical perspectives, and their implications for research and practice.

A psychoanalytic perspective on paradoxical outcome

Psychoanalytic theory provides a rich resource for making sense of paradoxical outcome. For example, Otto Kernberg (1984) argued that a psychoanalytic perspective requires assessing a patient along three dimensions of personality structure: capacity for reality testing, level of maturity in defences, and level of identity cohesion. As a result, any attempt to assess therapy outcome merely in terms of symptoms is likely to miss important aspects of the life difficulties for which the patient is seeking help. For example, a patient who reports few symptoms of anxiety or depression on a standard self-report measure may at the same time have a personality organization in which fear of abandonment may represent a major theme. The fact that the person records only moderate levels of symptoms at intake may mean that they have developed strong defences against awareness of their underlying fear of abandonment. If these fears come to the surface during therapy, they may become more symptomatic, and be defined as a poor outcome. On the other hand, if they receive a therapy that helps them to manage their symptoms (a good outcome as defined by self-report measures), they may leave therapy with a sense of dissatisfaction that deeper issues had not been addressed.

A further relevant psychoanalytic perspective can be found in the concept of epistemic trust (Fonagy et al., 2015, 2017). This construct builds on attachment theory, in suggesting that we are born dependent on others for learning life-saving knowledge about how to not only get food, comfort and shelter, but as sources of essential knowledge about social community, reproduction and power. Fonagy argues that attachment to significant others (trust) allows access to ‘episteme’ (knowledge) on these highly important areas of life. Conversely, individuals who have a history of insecure and unstable attachment find it hard to trust, and are hypervigilant in relation to real or imagined sources of threat. As a consequence, their responses to any request to supply information about their problems, or their progress in therapy, may be heavily influenced by the extent to which they trust the person who is collecting the information. Evidence that client responses to self-report symptom measures are shaped by the relational context within which such measures are administered, is available in a client interview study carried out by Alves et al. (2016) and an intensive case study published by Truijens et al. (2019a).

The concept of defence mechanism is central to a psychoanalytic understanding of paradoxical outcome. In psychoanalysis – and most other forms of psychodynamic therapy – humankind is understood as being vulnerable and in need of different ways of defending oneself, not only from outer threats but also from inner needs, impulses, phantasies and desires that in one way or the other are in conflict with ideals, norms and morality.

The illusion of mental health may be understood as a consequence of patients having different reasons for defending against acknowledging that they might be experiencing emotional or relational issues. For example, Bollas (1987) describes clinical encounters with patients who strive to be as ‘normal’ as possible to fit in with societal expectations. Such a patient will not report symptoms on a checklist because their whole way of life has been organised around being in control of their emotions, and being seen as rational and not having any psychological problems.

The work of Shedler et al. (1993) on the illusion of mental health is grounded in a defence mechanism perspective. A psychoanalytic perspective that extends this notion can be found in the work of Lecours and Bouchard (1997), who suggest that unrealistically low symptom self-ratings occur not as a result of warding off or dissociation, but as a result of insufficient mentalization. Their model presents the capacity of mentalization as being a process where expression of drive-affects – which are bodily needs and sensations – follow a developmental line from rudimentary representations to symbolic ones, which may be expressed through different channels such as bodily, motoric, imaginary or verbal. The illusion of mental health may according to this model be explained as a result of a patient’s lack of capacity to mentalise his/her issues, resulting in a mainly somatic presentation of problems.

A further psychoanalytic perspective on paradoxical outcome, initially developed in the context of studies of recovery from schizophrenia, is the distinction between ‘sealing over’ and ‘integrating’ troubling psychotic experiences (McGlashan et al., 1976, 1975). In these studies, patients who sealed-over had a tendency to avoid talking about difficult episodes, and did their best to play the part of being a normal person. By contrast, patients who integrated could be observed to work on making sense of such episodes. A similar process was noted by Jansen et al. (2016) in a study of individuals who had experienced childhood trauma. In similar fashion to illusory mental health, an individual who coped with emotionally difficult memories by sealing over might appear, on a symptom measure to be ‘well’.

Self-multiplicity perspective

The idea that the self is experienced and/or performed in terms of multiple interacting parts or sub-selves represents a highly influential tradition within personality theory and psychotherapy practice (Rowan & Cooper, 1998). Examples of personality constructs that reflect such assumptions include ego states (Berne, 1972), self-positions (Hermans, 2006), and voices (Honos-Webb & Stiles, 1998). The existence of paradoxical outcome is entirely predictable by any of these models. For example, at intake a client might respond to a symptom questionnaire on the basis of rational, controlled ‘adult’ ego state functioning or dominant ‘voice’. Once a sufficiently secure therapeutic relationship has been established, the same individual might shift to responding to the same measure from a more emotionally vulnerable ‘child’ ego state position or vulnerable voice (recording a higher symptom score, even though therapy has been helpful). Self-multiplicity models posit that clients will tend to arrive in therapy with ‘warring’ sub-selves (expressed in fluctuating, contradictory or extreme self-reports), and that effective therapy will result in a more integrated sense of self in which different parts or positions are able to engage in dialogue (thus yielding more stable self-reports).

Person-situation interaction perspective

An important line of personality theory and research has focused on the idea that everyday behaviour is not solely determined by underlying dispositions (e.g. personality traits), but is also significantly shaped by situational influences (Dreier, 2011; Furr & Funder, 2021). A situational perspective draws attention to the relevance of the contexts within which therapy outcome data are collected, and how the meaning of such contexts may differ across individuals. For example, an individual who is sensitive to external judgment may find it easier to acknowledge personal difficulties when independently completing a self-report measure, but drawn toward presenting a more socially acceptable (i.e. healthy) image of self when being interviewed by a researcher. Symptom measures used in therapy research and practice have generally been developed in emotionally and interpersonally neutral standardised situations of no great personal consequence, in which individuals have been asked to complete questionnaires that will generate data for test validation purposes. As a result, situational influences on responses have not been particularly salient. By contrast, when deployed in actual therapy contexts, a complex set of contextual influences may be evoked.

Existential-phenomenological perspective

Existential-phenomenological concepts developed by the radical psychiatrist RD Laing provide a potentially valuable perspective from which to begin to make sense of paradoxical outcomes and illusory mental health. His programme of research incorporated clinical (Laing, 1960, 1961), social psychological (Laing et al., 1966) and poetic (Laing, 1970) forms of inquiry. Using a methodological approach based in the existential philosophy of Sartre and Heidegger, the aim of Laing’s investigations was to explore why it was hard for individuals within contemporary society to possess an authentic or secure sense of being in the world. What he observed was that ‘ontological insecurity’ arose from spirals of mystification in which the person first pretends to be something they are not, then pretends to be what they ‘really’ are (‘elusion’). Alongside this process, other people may confirm or disconfirm various aspects of the person’s experience, and collude (i.e. play along with) their pretence. Laing (1961) describes ‘illusion’ (e.g. illusory mental health) as self-deception. By contrast, collusion is more complex, involving two or more people who not only engage in self-deception and other-deception, but also are unable to admit, or be aware, that this is what they are doing. A similar perspective can be found in person-centred therapy, an approach that incorporates an existential-phenomenological perspective. For example, individuals who are more ‘self-actualised’ are more likely to respond to inquiries about their well-being from an authentic sense of self grounded in an internal locus of evaluation (Rogers, 1961). By contrast, persons who are troubled may respond inconsistently, on the basis of a fragile sense of self that makes it hard to remain in contact with painful internal memories (Warner, 2000).

Socio-political perspectives

An alternative perspective on paradoxical outcome involves moving outside of a psychological paradigm, and adopting a socio-political perspective. Sociological and philosophical critics of psychotherapy, such as Ole Jacob Madsen (2014), have argued that it makes sense to view mental health problems as a manifestation of underlying social issues, rather than as forms of personal or individual psychological dysfunction. For example, exposure to exploitative employment practices can result in individuals being encouraged to enter therapy because they feel anxious, depressed and stressed. Some of these individuals may have an awareness of the social and political meaning of their distress. When they are invited to complete a symptom measure, or are interviewed, at the start or end of therapy, it is possible that they may interpret the meaning of items and questions in quite a different manner to those clients who implicitly accept the validity of psychological ideas. This may involve challenging the basic notion of collecting psychological data. Foucault (1967) argued that mental health care historically began as strategies used by governing states in the western world to discipline and control their citizens. According to Foucault, the development of psychiatry, the building of asylums, and the later emergence of psychotherapy, all had the effect of introducing a strong differentiation between ‘normality’ and ‘madness’, in which the latter was increasingly subjected to confinement and stigmatisation. From a Foucauldian perspective, the use of self-report symptom and outcome measures can be viewed as part of a broader trend in the direction of using metrics to enable both external and self-surveillance (Ajana, 2018; Mau, 2019).

Methodological perspectives

In some situations, apparently paradoxical outcome findings are associated with the adoption of different methods of data collection. The following sections provide overviews of some potentially relevant methodological perspectives in terms of their implications for research and practice.

Psychometric perspective: The psychology of test-taking

The medium through which outcome data are obtained about clients makes a difference to the type of information that is produced. McAdams (1995, p. 380) has suggested that self-report measures are grounded in ‘the psychology of the stranger’, an understanding of how the person might be understood, in terms of broad patterns of behaviour, to someone who does not really know them. By contrast, closer contact with the individual gives access to the narratives of their life, which provide ‘a more detailed and nuanced description of a flesh-and-blood, in-the-world person, striving to do things over time, situated in place and role, expressing herself or himself in and through strategies, tactics, plans, and goals’ (McAdams, 1995, p. 366). It is therefore not surprising that, in at least some instances, the picture of a person obtained from a self-report measure might be quite different from that which emerges from narrative sources such as interviews.

Studies in which individuals are invited to describe their experience of completing symptom measures used in therapy research have found that participants describe items as irrelevant or hard to follow, or interpret items in ways that are different from the meanings intended by the test designer (see, for example, Galasiński & Kozłowska, 2013; Paz et al., 2020; Truijens et al., 2019b). There are many factors that influence the ways that respondents approach the task of answering questionnaire items (McClimans, 2010; McLeod, 2001; Truijens, 2017; Truijens et al., 2019a). For example, a person may understand specific words, such as ‘always’ or ‘sometimes’ in an idiosyncratic or culturally determined manner, they may want to reply ‘it depends’ rather than provide a general answer, or they may be influenced by what they believe the purpose of the measure to be or their feelings about the individual administering it.

Other research into the psychology of test-taking has investigated general sources of bias that exist at a group level. In relation to paradoxical outcomes, the most relevant findings from this field of inquiry are the Dunning-Kruger Effect, and the phenomenon of response shift. The Dunning-Kruger Effect (Dunning, 2011, 2019; Kruger & Dunning, 1999) refers to the tendency of those who are less competent in respect of a skill or attribute to over-estimate their ability, and those who are most competent to under-estimate it. Although there are different interpretations of this effect (see, for example, Kim et al., 2016), it has been replicated in many studies. What seems to happen is that a person whose knowledge or skill around a particular topic (e.g. financial management, or cooking) is limited, is not competent to make accurate judgements around where they compare to others in respect of that skill. As a result, they believe that they exhibit a sufficient degree of competence. By contrast, those who are highly competent in a skill tend to assume that others similarly find that skill easy to perform, and as a result underestimate their level of ability. In principle, the Dunning-Kruger Effect is applicable to situations in which troubled individuals complete self-report symptom measures that include items relating to areas such as the capacity to form satisfying interpersonal relationships or the ability to regulate emotions. People who struggle with such skills are likely to answer self-report measures in a manner that makes them look ‘healthier’ than they actually are. Individuals who are highly competent in such areas as interpersonal skills and emotion self-regulation are of less immediate interest to therapists. However, in relation to interpreting findings of therapy outcome studies, the Dunning-Kruger theory would predict that those who have maximum benefit from therapy, in terms of a transformative personal shift, would be likely to downplay that achievement when completing an end-of-therapy or follow-up measure.

The Dunning-Kruger Effect focuses mainly on the accuracy of self-judgment at one point in time. By contrast, the response-shift phenomenon considers what happens when self-report measures are used to estimate change, for example, by comparing a person’s score at the end of an intervention with their score recorded pre-intervention. In a classic paper, Golembiewski et al. (1976) argued that pre-post change could be interpreted in three ways:

• genuine change in the attribute being measured. For example, a client has a lower score on a depression scale because they have genuinely become less depressed;

• recalibration. A client records a lower depression score at the end of therapy but this reflects the fact that they have acquired a more differentiated understanding – they have learned that the range of possible levels of depression is much greater than they had ever imagined, with the result that they could now see that what they had previously regarded as severe depression was in fact only a mild episode;

• reconceptualization. A client develops a different perspective on their depression, for instance, by coming to see it as an emotion-focused experience rather than as a problem that is manifested through negative thoughts. As a result, their lower score on a cognitively oriented measure reflects a stance that these symptoms are no longer relevant.

The second and third of these options represent what George Howard, a leading figure in the therapy research community at that time, characterised as a ‘response shift’: a change in the underlying basis on which questionnaire responses are understood and formulated (Howard & Dailey, 1979). The topic of response shift has been widely debated in health research, because it calls into question the meaning of outcome studies (Sajobi et al., 2018). Two main approaches to investigating response shift have been developed. One method involves statistical analysis of the factor structure of item responses pre- and post-intervention, to detect whether participants may have moved to a more coherent (recalibration) or structurally different (reconceptualisation) pattern. An alternative method is to ask participants in a pre/post study to retrospectively rate how they recall feeling before the intervention was initiated (this is sometimes called a ‘thentest’). Relatively little research has been carried out into response shift in relation to therapy outcome studies (Bulteau et al., 2019). Using statistical methods, response shift analyses of therapy outcome data, carried out by Carlier et al. (2019), Fokkema et al., (2013) and Wu (2016) have reported findings that suggest that both recalibration and reconceptualization occur. There do not appear to be any ‘thentest’ studies of psychotherapy outcome, but studies on interview skills training (Howard & Dailey, 1979) and impact of receiving a hearing aid (Arthur et al., 2016) reported marked recalibration shifts indicative of significant under-reporting of problems (i.e. overestimate of competence/well-being) at the pre-intervention assessment point.

The Dunning-Kruger Effect and the response-shift phenomenon are relevant to the issue of paradoxical outcome in therapy, in reflecting general sources of bias that may occur when self-report measures are used. However, these concepts refer to group-level effects. In psychotherapy theory and practice, it is necessary to be able to understand the conditions under which ratings made by particular sub-groups of clients (e.g. those who exhibit ‘illusory mental health’), or individual clients, provide a reliable and valid estimate of their difficulties. Findings from research into the experience of completing measures, summarised above, suggest that there are likely to be substantial differences between individuals.

Everyday or common-sense criteria used by clients to evaluate therapy outcome

Therapy outcome measures, such as self-report scales used in research and in routine outcome monitoring and feedback systems, are based on an assumption that the aim of therapy is to reduce symptoms such as depression and anxiety. By contrast, qualitative outcome research, in which clients are interviewed at the end of therapy about what has changed, or are interviewed before therapy about their hopes and expectations for change, have found that clients use a much wider set of criteria for evaluating outcome. Although some clients use symptom/disorder criteria to evaluate their therapy, they are also likely to use criteria based in readily available common-sense ways of understanding emotional distress (McLeod et al., 2021), such as negative identity repair (‘I got my life back on track’) and enhanced agency (‘I learned some really helpful skills that I can use whenever I feel scared’). The common-sense perspective used by clients to evaluate their therapy allows for the possibility that benefit and harm/disappointment can co-exist (rather than being distributed along a linear scale with well-being at one end and dysfunction at the other). It also tends to include the idea that transformational change (i.e. something more fundamental than symptom reduction) is possible. These aspects of the client’s view of therapy mean that there will inevitably be tensions between outcome estimates based on symptom ratings, and those that reflect the client’s everyday common-sense knowledge.

Projective techniques as sources of information about outcome

The issue of how to make sense of paradoxical outcome data in psychotherapy has almost entirely focused on tensions between outcome evidence generated by either self-report measures, interviews or therapist clinical observations as sources of information. Although each of these methods is different in important respects, they are nevertheless similar in directly asking for the opinion of the client (or therapist) on what has changed and whether therapy was helpful. By contrast, projective techniques offer an alternative methodological approach, in which indirect evidence of client difficulties or change is collected by inviting the person to complete a task that does not overtly ask their opinion about either their emotional state or their view of therapy helpfulness. An example of a projective technique, used in the illusory mental health research conducted by Shedler et al. (1993), is the Early Memory Technique (Fowler et al., 2000; Mayman & Faris, 1960). In this procedure, the client is invited to tell, or write down, the earliest memories from their life. These stories can then be analysed in respect of relationship patterns and emotional themes. The activity of telling a story represents a performance episode – the individual expresses or performs how they are in the world, rather than being asked to articulate their idea or concept of how they are. Projective techniques require a particular type of performance, that draws on the imaginative or symbolic capacities of the person. An example of how projective techniques can contribute to a constructive response to the existence of paradoxical outcome can be found in the practice of ‘collaborative assessment’ (Fischer, 2000). This involves a client entering therapy working with a consultant to develop an understanding of their life difficulties and goals for therapy, and collaboratively writing a report that is passed on to the therapist. What seems to be particularly helpful within this procedure is that client and consultant have access to multiple perspectives, that each tell a different story, such as self-report questionnaires, interviews, projective techniques, and ability measures. The process of integrating evidence from these diverse sources not only provides a reliable and clinically useful assessment of issues to be explored in therapy, but is also highly meaningful for clients in relation to the development of insight and self-understanding (Aschieri et al., 2016).

Time perspective: Concurrent vs retrospective data collection

An important aspect of information about psychotherapy outcome concerns the time at which it was collected, and the time-period covered by the data. For example, when a client completes a questionnaire measure, or is asked a question in an interview, they may be invited to report on how they feel now, how they have felt or behaved within a recent time-period (e.g. the last week, or last month), or how they have felt or behaved in general (i.e. a non-specific time period, which participants presumably interpret as referring to an episode or stage in their life). Data may be collected during therapy, and after therapy has been completed. Data can be collected at different points (e.g. at the start and finish of therapy, or weekly) or only at the end (i.e. retrospective evaluation). Each of these choices may produce different outcome estimates (Flückiger et al., 2019; Sandell & Wilczek, 2016).

Articles in this special issue

It is essential that the issues associated with paradoxical outcome and illusory mental health are investigated using a wide range of methodologies that enable different aspects of these phenomena to be examined. Case study research has a particularly valuable part to play within this programme of research, because of its capacity to analyse complex interacting processes that unfold over time, in a specific context. The papers in this Special Issue of the European Journal of Psychotherapy Counselling illustrate the type of contribution that case-based inquiry can make to the development of practice-relevant knowledge around paradoxical outcomes.

In the first of these papers, Krivzov et al. (2021) offer an introduction to the strengths and limitations of the existing case study literature around paradoxical outcome and illusory mental health. These authors are members of a team at the University of Ghent, who have established an on-line Single Case Archive that gives access to all psychotherapy case study reports published in journals. Their paper focuses on cases that were categorised as examples of psychotherapy failure. The topic of failure is highly relevant to an understanding of paradoxical outcome because the illusory mental health model predicts that a significant proportion of apparent failure cases represent therapies in which the client recorded an illusory low symptom score pre-therapy, and then could only get worse (in terms of the measure).

De Smet et al. (2020) analysed four cases of paradoxical outcome selected from data collected in large-scale outcome studies in Ghent and Stockholm. These were cases in which the client was categorised as a good outcome in terms of pre- and post-therapy measures, but expressed considerable dissatisfaction with therapy in an interview conducted after therapy had been completed. The narratives of these clients demonstrate ways in which their responses to questionnaires, and their personal perspective as conveyed to an interviewer, represent different ways of making sense of the experience of therapy.

Each of the next two papers, by Thoresen et al. (2020), and Ward and McLeod (2021), present individual case studies. The Thoresen et al. (2020) case refers to a client receiving therapy for an eating disorder, whose quantitative outcome scores contradicted her own report about how therapy had been helpful. The Ward and McLeod (2021) case provides an analysis of 12 sessions in the middle of therapy, during which the client made a life-changing personal breakthrough, accompanied by a pattern of significant deterioration in symptom scores. Both of these cases include material from therapy sessions that provide an insight into the real-life therapy process that lies behind responses to symptom measures. A common feature of these cases was the way in which the use of a therapist intervention that evoked the symbolic and imaginative capacity of the client, had the effect of making it possible for them to make a fundamental shift in their way of understanding and communicating their key problems. Specifically, key moments of client change were associated with imaginative exploration of early memories that had enabled developmental change in the capacity of ‘object trust’ (Stänicke & Killingmo, 2013).

The final paper, by Benelli et al. makes two important contributions to the evidence base around paradoxical outcome and illusory mental health. First, these researchers conducted a conceptual mapping exercise to identify how an illusory mental health client profile can be translated into categories available within three widely used diagnostic systems. This piece of work has the potential to allow psychotherapists and other mental health practitioners to more readily identify illusory mental cases within their client populations. Second, Benelli et al. used these diagnostic categories to analyse the occurrence of illusory mental health within a case series of nine published mixed-method case studies of psychotherapy for depression. What they found was that six out of the nine cases could be classified as including elements of illusory mental health. If this result is found in other studies, it suggests that the basis on which outcome is assessed, in routine practice as well as research, will need to be re-visited.

Conclusion

We are in agreement with Desmet (2018), who has argued that, in relation to the question of evaluating therapy outcomes, paradox and uncertainty represent unavoidable aspects of psychotherapy research and practice. Similar dilemmas are found in all domains of health, social care and education. For example, writing on the topic of breathing difficulties, an issue of central importance for many fields of health care, Carel (2018) describes an irreconcilable ‘rift between objective measurement and subjective experience’ (p. 332). In similar fashion to our medical colleagues whose everyday clinical duties require them to reconcile these standpoints, psychotherapy practitioners and researchers need to be willing to embrace uncertainty, and acknowledge the value of different modes of knowing (Elliott, 2008). The emergence of a paradox, to the extent that its contours and fault-lines begin to be apparent, may be viewed as a sign of the maturity of a domain of inquiry and practice. The existence of paradoxical outcome should not be regarded as indicating an underlying weakness in psychotherapy research and practice, but an opportunity to develop new ways of understanding and new practices.

We hope that this Special Issue will encourage researchers to conduct further investigation into the nature of paradoxical outcome and illusory mental health, and practitioners to write about their professional experience in relation to these topics. We believe that further work is required in three main areas: conceptualisation, research methodology, and practice. In relation to conceptualisation, different theoretical and methodological perspectives provide contrasting epistemologies of outcome. For instance, self-report symptom measures are built on an assumption that knowledge about outcome is obtained through observation (mainly by the person themselves, but sometimes by others) of disorder-specific behaviour. By contrast, psychoanalytic and existential perspectives represent contrasting versions of a position that knowledge about outcome is only possible through interaction with a person and learning about how they regulate their emotions on a moment-to-moment basis within a relationship. A perspective that is informed by psychometric science takes the position that it is necessary to combine information from different sources, because each source will inevitably be coloured by its own distinctive pattern of bias. From a socio-political viewpoint, it is essential to take account of the meaning of the social context within which outcome information is collected. For example, a therapy client may respond in a way that resists hegemonic power, through either hiding or exaggerating their problems depending on how they believe that such personal information will be used. At the present time, each of these outcome epistemologies largely exist in separation from each other, with an absence of conceptual bridges between them. We suggest that the task of building conceptual bridges will require analysis of outcomes in single cases in which data has been collected that is sufficiently rich to allow an integrative theory or model to emerge. An example of such an approach can be found in Stänicke and Killingmo (2013), in which analysis of paradoxical findings arising from different sources of outcome data leads to the emergence of a new concept – object trust. We anticipate that further work of this kind has the potential to generate a more differentiated conceptual language for talking about outcome. It would be valuable to include stakeholders in this conversation: clients, their significant others, service managers and policy-makers, just as much as therapists and researchers. A crucial aspect of this process includes being willing to think more critically and reflexively about how we talk about people who are in distress, and where researchers and clinicians position themselves in relation to them.

The existence of paradoxical outcome has implications for therapy training and practice. Little is known about how therapists take account of different types of information around the progress of therapy (Daniel & McLeod, 2006). Greater knowledge about the strengths and limitations of different data collection strategies, and the possible meaning of contradictory outcome indicators, could enable therapists to be more competent in this area of their work. For example, when using routine outcome monitoring and feedback tools, the existence of a discrepancy between qualitative and quantitative estimates of the mental health of a patient might represent a valuable opportunity or invitation to open up a conversation with the client around what these different sources of information might mean to the client. In some therapy clinics, client intake scores on brief symptom measures are used as screening devices, with clients with low scores being denied access to treatment and clients with high scores being seen immediately. In the light of what we know about paradoxical outcome, these practices are not scientifically justifiable, and may be unethical. Similar issues arise when symptom measures are used to inform decisions about the termination of therapy.

Disclosure statement

No potential conflict of interest was reported by the authors.