A borderline focused Reflective Functioning measure – Interrater reliability of the Mentalization Breakdown Interview

Abstract

Objective

Mentalizing difficulties can be considered the core psychopathology of borderline personality disorder (BPD). Typical failures of mentalizing are targets in therapy for BPD. They are related to severe distress, relational problems, self-destructive behaviors, violence, or substance misuse. A major obstacle in BPD treatment research is the lack of suitable and easily administrated methods to assess mentalizing ability during treatment. The Mentalization Breakdown Interview (MBI) is a new method for capturing episodic mentalizing difficulties occurring in close relationships. Interviews are videotaped and scored in accordance with the Reflective Functioning Scale (MBI-RF). In this way the patients’ ability to retrospectively reflect over such episodes are evaluated. This study investigates the interrater reliability of MBI-RF.

Methods

The study includes videotapes of MBIs from 32 patients with BPD in an outpatient clinic specialized on mentalization-based treatment (MBT). The MBIs were performed by MBT therapists. Three certified raters scored MBI-RF.

Results

The interrater reliability was good for MBI-RF.

Conclusions

The MBI is promising as a BPD-focused method for the assessment of Reflective Functioning.

Keywords:

• Mentalization
• mentalization breakdown
• RF assessment
• borderline personality disorder

Introduction

Mentalizing is defined as the imaginative mental activity that enables us to perceive and interpret human behavior in terms of intentional mental states like beliefs, thoughts, and feelings [1]. The investigation of a link between mentalizing as a developmentally acquired capacity and psychopathology has paved the way for a new way of understanding and treating mental illness [2]. In this model, psychopathology is seen as a result of disordered mentalizing ability, and studies suggest that impairments in mentalizing result from disruptions in early attachment relationships, most likely in interaction with genetic vulnerability [3–5].

In borderline personality disorder (BPD), impaired mentalizing capacity is assumed to underlie core problems such as poor affect regulation, impulse control problems, and incoherent internal representations of self and others [6,7]. The mentalizing impairments in patients with BPD are characterized by significant fluctuations, primarily in attachment contexts, sometimes evolving into major mentalizing breakdowns. Such mentalizing breakdowns can lead to severe relational problems, self-destructive behavior, violence, or substance misuse and appear as obvious targets in therapy for this condition. For instance, Mentalization-Based Therapy (MBT) is an evidence-based manualized psychodynamic psychotherapy for patients with BPD which focuses specifically on the patients’ mentalizing difficulties [7].

Accordingly, it is assumed that improvement in mentalizing capacity is associated with positive treatment outcome in BPD patients. Despite some encouraging studies the research is still limited on this point [8–10]. We have insufficient knowledge regarding to what degree patients’ mentalizing capacity improves during therapy, if such improvements mediate treatment outcome, and whether the potentials for increased mentalizing and clinical improvement vary depending on patients’ initial levels of mentalizing abilities. One major obstacle in this field of research has been the lack of suitable methods for the assessment of patients’ mentalizing capacity during treatment [11,12]. The present study addresses this challenge through the investigation of the reliability of a new tool for assessing mentalizing capacity specially adapted to central BPD mentalizing difficulties, the Mentalization Breakdown Interview (MBI).

Assessment of mentalizing

Mentalizing capacity is usually operationalized as Reflective Functioning (RF) and quantified with the RF Scale, ranging from − 1 (i.e. rejection of mentalization, or unintegrated, bizarre or inappropriate mentalization) to + 9 (full or exceptional mentalization). The gold standard for RF assessment is the application of the RF Scale on transcripts of the Adult Attachment Interview (AAI) [13]. The RF Manual [14] provides a comprehensive guide for the coding process which is only to be performed by certified coders. However, the RF Scale applied on the AAI (AAI-RF) is a time-consuming and costly method which limits its use in research. This has led to the development of other RF assessment methods in recent years, mainly self-report measures which are easy to administer and potentially time saving [15–18]. Yet, mentalizing comprises complex phenomena and qualities, and the paradox in the use of self-report measures is that the individuals need to rely on their subjectively perceived capacity for mentalizing. This bias in meta-perspective is well known [15,16].

In order to increase applicability of the interview format for RF assessment, some studies have applied the RF Scale to AAI interviews which have been modified in length and focus, e.g. the Brief Reflective Functioning Interview (BRFI) [19,20]. The RF Scale has also been applied to other semi-structured interviews, like the Object Relations Inventory (ORI) [21].

Another approach is symptom-specific RF interviews, which explore the patient’s capacity to reflect on the psychological underpinnings of their current specific condition or symptoms. While assessing a subject’s capacity to reflect on his or her psychological processes as they connect to a particular symptom, the symptom-specific approach nonetheless can maintain the complexity of the original RF concept. Brief and easily administrated semi-structured symptom-specific RF interviews have been developed for OCD, panic disorder, PTSD, and depression [22,23].

To our knowledge, there are no existing RF assessment methods which specifically focus on BPD patients’ ability to reflect on mentalizing breakdowns in current close relationships. The MBI was developed to fill this gap and supplement existing RF assessment methods particularly adapted to this patient group, and thus also contribute to further research on the mechanisms of change in the treatment of borderline personality pathology.

The mentalization breakdown interview

The interview was developed to assess the patient’s ability to retrospectively reflect on current episodic mentalizing breakdowns and includes enquiry about the frequency and severity of such episodes. It is a semi-structured and easily administered 30-minute interview that can be conducted by a clinician with knowledge of the theory of mentalizing and MBT.

The development of the MBI was inspired by the studies of symptom-specific RF interviews [23], based on an interview guide originally outlined in an MBT manual, and further developed for research purposes [24]. It encompasses a systematic exploration of episodes of impaired mentalizing occurring in an interpersonal context within the past six months. The interview starts with a short explanation of the terms mentalizing and mentalizing breakdown. Thereafter the interviewee is asked if he/she has experienced such episodes and whether these are typical to their life. The interviewee then selects a relevant and preferably severe and recent episode in a close relationship for further exploration. The course of events, interpersonal context, triggers and consequences of the episode are clarified. The interview then proceeds to an in-depth exploration of this episode with the purpose of uncovering the patient’s mentalizing ability in his/her effort to make sense of the event. The interviewer is instructed to take a not-knowing stance and not to mentalize on behalf of the patient.

Like the AAI the MBI comprises both so-called demand- and permit questions. Two of the questions in the MBI are demand questions, that is, questions which demand a demonstration of mentalizing activity: ‘Why do you think you reacted like this in that situation?’ and ‘Have your thoughts about this changed over time, in that case how?’ Two additional questions are so-called permit questions, that is, they permit the interviewee to demonstrate their mentalizing ability, but without an explicit demand to do so. They tap into the context and the relational setting of the episode; ‘Have you noticed if you tend to react like this in certain situations?’ and ‘Have you noticed anything in your relationships with other people which makes you react like this?’ Any positive responses to these questions are followed up by demand questions; ‘Why do you think this happens?’

In the last part of the MBI there are ten questions concerning the frequency and severity of such mentalization breakdown episodes, thus providing additional quantitative data on the mentalizing difficulties. The method is developed for both research and clinical use. Clinically, the interview is to be used by the clinician to elaborate an individual case formulation in collaboration with the patient, based on the mentalizing difficulties uncovered in the interview. For research purposes, videotapes of the interview provide material for systematic rating of RF (MBI-RF).

Aim of the study

The aim of this study was to investigate the inter-rater agreement of MBI-RF scores, pertaining to recent episodes of patients’ mentalizing breakdowns.

Materials and methods

Study sample

The study was performed at Oslo University Hospital in an outpatient clinic specialized in treating poorly functioning patients with BPD, where the treatment offered was MBT [25]. The study involved regular patients and clinicians who applied the MBI as part of their assessment practice.

The study sample includes 32 MBI interviews conducted during the period autumn 2019 to autumn 2020, either in the initial assessment or at one of the routine six-month evaluations during treatment. This spread in time aimed to promote a broader range of RF scores for the reliability test. However, to prevent bias the raters were blind for the time points of the interviews. Only one MBI per patient was included.

Before starting MBT, all patients had gone through a thorough diagnostic evaluation according to the DSM-5 [26] using respectively the Structured Clinical Interview for DSM-5 Personality Disorders for PDs (SCID-5-PD) [27] and the Mini International Neuropsychiatric Interview (MINI) [28] for symptom disorders. Patients with schizophrenia, schizoaffective disorder, bipolar disorder type I, cognitive impairment, autism spectrum disorder, and alcohol or substance dependency were not included in the treatment program. The included patients had moderate to severe BPD and were in the age of 19-30 years (M = 23.3, SD = 3.0). The sample comprised five men (16%) and 27 women (84%). We don’t have access to information of co-occurring personality disorder (PD) diagnoses for these selected patients. However, the average number of PDs for patients starting treatment during the period 2018 to 2020 (n = 88) was 1.44. The most common co-occurring PDs were avoidant (24%), paranoid (7%) and obsessive-compulsive (7%).

MBI training

The clinicians attended a series of group-based training sessions aided by video-recorded MBIs. Video-based MBI group supervision sessions were continued regularly throughout the study period to maintain interview competence. The training included specific feedback on the administration of the interview, not the least to ensure the suitable level of activity on behalf of the interviewer in the exploration of the mentalization breakdown episode. A suitable level would be when the interviewer is stimulating the interviewee’s own mentalizing exploration without adding anything to it. This may be an unattainable ideal, but the training nevertheless sought to minimize the influence of the interviewer in this process, stressing the importance of asking simple follow-up questions and not mentalizing on behalf of the patient.

Raters

Video recorded interviews were rated by three of the authors (MSJ, TW & DAU), all certified as reliable RF coders at the Anna Freud Centre in London. MSJ & DAU are also experienced clinicians working in the MBT outpatient clinic.

Rating procedures

Thirteen clinicians performed the interviews. The gender distribution was respectively four men and nine women. Their age ranged from mid-twenties to late sixties. Two of the interviewers were graduate students in psychology, while the rest were employed at the outpatient clinic. Their professional background was respectively: occupational therapist (1), social worker (1), psychiatric nurse (1), psychologist (2), specialist in clinical psychology (4) and psychiatrist (2). The majority of the interviewers had extensive training and education in MBT, all interviewers were familiar with the theory of mentalizing and MBT and all received weekly, video-based MBT supervision. The number of interviews per interviewer was in the range 1-5 (Median 2). Median administration time for the MBI was 30 min, with a range of 15 to 52 min.

For initial calibration, three interviews were first rated separately, and then compared and discussed in fellowship in order to reach consensus on discrepant scores. By these procedures a mutual agreement was established to optimize adherence to the RF Manual. The following 32 interviews were scored successively. All raters scored all the interviews. The raters met and watched the videos together, but did not reveal scores before all ratings were completed. After each rater’s score was disclosed and registered for the reliability test, a consensus score was also agreed upon, to be used in further research.

Statistics

All three raters rated the same set of targets, the raters are considered a random sample of possible raters, and the issue was whether the raters ranged the targets equally. Then interrater reliability of MBI-RF was estimated by the Intra-class Correlations Coefficient for Two-way random effects, average measure (ICC, 2.k). Agreement was analyzed for all three raters as a group, as well as for each pair of raters. According to Koo and Li ICC values less than 0.5 are regarded as poor, between 0.5 and 0.75 as moderate, between 0.75 and 0.9 as good, and greater than 0.90 as excellent agreement [29].

Results

The interrater reliability (ICC, 2.k) of the MBI-RF scores for all three raters as a group was 0.77 (95% C.I.: 0.59–0.88), which indicates good reliability [29]. When analyzed in pairs the reliability was in the moderate to good range: respectively 0.61 (95% C.I.: 0.20–0.81), 0.65 (95% C.I.: 0.28–0.83) and 0.81 (95% C.I.: 0.61–0.91). The reliability was good for one of the pairs (0.81 for MSJ & DAU), while it was moderate for the other two pairs (MSJ & TW and DAU & TW).

To investigate possible associations between interview time and magnitude of disagreement, difference scores for each rater on each interview were computed (rater score-mean score) and correlated (Spearman’s rho) with interview length. The magnitude of the correlation coefficients ranged from 0.13 to 0.02, none of which was significant below the 05-level. Moreover, to investigate possible interviewer-bias a One-Way ANOVA was conducted with interviewer as a factor, after excluding four interviewers who had conducted only one interview. The results revealed no support for any interviewer-bias with respect to rater disagreement.

The RF scores for the three raters in all 32 cases are displayed in Figure 1. The RF scores were in the range 1–6 for two of the raters, and 1-5 for the third rater. Mean RF was 3.0, 2.9, and 3.1 respectively, while mean RF consensus score was somewhat lower, 2.8.

Figure 1. RF scores for the three raters in the 32 cases. Note: R: rater; C: case, interview time in parentheses (minutes).

The MBI interview guide underlines that the episode under investigation preferentially should be in a close relationship, and the raters agreed that this was the case in 81–83% of the interviews.

Discussion

The aim of this study was to investigate the interrater reliability of RF ratings based on the MBI. The results indicate that information from the MBI can give rise to reliable scorings of RF pertaining recent episodes of breakdown in mentalizing in close relationships in patients with BPD. In line with our intention the interview had a quite short administration time. The short administration time translates directly to faster rating of videotaped interviews, compared to transcribed AAIs.

Although the rater agreement of MBI-RF was good, the differences in RF scorings point to some important issues and possible pitfalls in both the administration and the scoring of the interview. We will discuss some potential sources of disagreement in the RF ratings.

Even if a short administration time of the interview was intended and achieved in this study, the short duration of the MBI interview can also be a possible source of discrepancy among the RF raters as short interviews could potentially provide less information about the interviewee’s mentalizing ability. A situation where the interview becomes prolonged can indicate a more elaborate mentalizing process. On the other hand more extended responses could also be a sign of the interviewee being in the non-mentalizing state called pretend mode. This mode is characterized by the use of mental state language, often in prolonged responses, but decoupled from reality or genuine experience [30]. Pretend mode can have a convincing, or even seductive, quality, and it can be difficult to distinguish from genuine mentalizing. However, in this material we found no significant association between interrater differences in RF scorings and the duration of the interviews.

Regarding the pretend mode in itself, careful probing and clarifying questions from the interviewer might facilitate its uncovering. This can be crucial in the differentiation of this mode from genuine mentalizing in the scoring procedure. In the current study, some of the disagreements in RF scores seem to have been caused by this lack of differentiation.

Other disagreements in RF scores might stem from different interpretations of self-serving qualities in some of the interviews. Responses which are indicative of this distortion are described as egocentric, self-aggrandizing or self-justifying. According to Fonagy et al. [14], these self-serving biases are designed to enhance the individual’s self-esteem, at the expense of a plausible understanding of oneself or others. When such maladaptive mentalizing process is not revealed, the interview can be assigned a too high RF. Some of the rated MBIs in this study contained examples of a self-serving tendency, like: ‘I feel I give too much of myself, and other people don’t see it. They don’t appreciate it.’ In this study self-serving tendencies were overlooked in some instances, which led to significant differences in RF scores among the raters.

Proper vigilance and adherence to the RF Manual is a pivotal measure to optimize the scoring procedure, and to prevent overlooking pitfalls such as the pretend mode and self-serving propensities. This of course applies to the use of the RF Scale in general, but is even more crucial when the RF Scale is used on shorter narratives like the MBI. Regular training to ensure a calibrated use of the RF Scale among the raters is highly recommendable, and in our group this was done initially by rating and discussing three interviews not included in the study. Moreover, each of the ratings in the study were followed by discussions, focusing on possible differences in the understanding and application of the RF Scale. The experiences from this study suggest that the corrective effect of these procedures was best for the self-serving issues, while the pretend mode was more difficult to spot systematically. The latter may be obscured by the use of a more or less sophisticated mental state language, and thus be given an erroneously high RF score, while a self-serving response can be more apparent to the rater. The mean RF consensus (2.8) was somewhat lower than each of the raters mean RF (3.0, 2.9, and 3.1), indicating a ‘cooling effect’ of the discussions following the scorings. Thus, to ensure sufficient reliability in RF scorings of the MBI, we recommend rating in groups with two or more raters. If using only one rater extra care should be taken to include regular calibration of the ratings with another RF expert [31].

Strengths and limitations

The present study involved patients with moderate to severe BPD treated in an outpatient clinic, and the interviews were performed by clinicians familiar with BPD and MBT. This contributes to the ecological validity of the MBI as a BPD-focused RF assessment method.

Moreover, in line with the intentions the interview had a quite short administration time and the rating procedure was considerably shorter than for the AAI rating procedure. This can be a significant gain in psychotherapy research, avoiding the bottlenecking effect of the labor-intensive AAI-RF rating procedure.

There are three elements of importance when interpreting the reliability estimates. The first is the number of objects of measurement, in this case 32 patients. The second is the number of independent raters, in this case three, and the third is the range of scores. The number of patients in the current study is sufficient as long as they are seen as representative of the target group, of which they are. The number of raters, however, is relatively low, making the estimates more vulnerable by even small disagreements of scores. When it comes to the range of scores, these are between one and six on an eleven-point scale. With a greater spread of the ratings the probability of even higher interrater reliability would have increased. However, the RF-levels in the present sample are representative for low-functioning BPD-patients [32,33]. Thus, achieving good interrater reliability in this range of ratings is of high relevance. Strictly, the reliability estimates cannot be generalized to the upper bond of the scale. However, the selection of the current study sample somewhat excludes scores in this upper range. All in all, the reliability estimates are in an acceptable range. Increasing the number of raters would most likely stabilize the estimated coefficients.

Women were grossly overrepresented in this study (84%), but this reflects the gender distribution in our specialized BPD-clinic as well as most clinical BPD samples [25,34].

As noted by Taubner and colleagues workgroups may develop idiosyncratic coding routines that differ from that of other workgroups and the RF Manual [35]. In the present study, two of the raters have a long history of rating RF together (MSJ & TW), whereas the third (DAU) is new in the rating group. Thus, the strong agreement obtained in ratings of MSJ & DAU supports the interrater reliability of the MBI and its generalizability to other RF rating-groups.

Even though the instructions in the MBI specifically ask for episodes in close relationships nearly 20% of the interviews in our present study did not meet this criterion. Interestingly, in most of these instances the interviewee was not capable of finding a more recent episode in a close relationship. This can possibly reflect the relational instability which is typical for the studied population, with few or no close relationships at the time for the interview. These interviews were also rated for RF, albeit the RF Manual states that the rated episodes should be in an attachment context [14].

The RF Scale is validated for use on transcripts of the AAI, whereas in the present study RF ratings were based on video recordings of the MBI. The use of video recordings has some potential advantages. The most obvious advantage is the possibility to watch the interaction between the interviewee and the interviewer as it unfolds and the possibility to pick up non-verbal cues relevant for the RF assessment.

Finally, this study included a sample of patients diagnosed with BPD. The co-occurrence of several PD diagnoses often seen in poorly functioning PD patients is regarded as a major challenge to the validity of the PD categories and one of the reasons for the development of dimensionally based PD classifications in ICD-11 [36] and the alternative model in DSM-5 [26]. These dimensional models are based on impaired capacities of personality functioning in general, that is, self- and relational functioning, combined with assessment of specific personality traits. Both systems have however retained BPD as a borderline pattern specifier (ICD-11) or a distinct category (AMPD). There is theoretical and empirical support for BPD as an index of severe personality more generally [37–40], even if the relative importance of a general PD factor representing borderline pathology and need for additional specific factors to account for variance in personality pathology is still disputed [41]. Unfortunately we have limited knowledge of mentalizing problems in other PDs than BPD, and there is a possibility that different forms or profiles of personality pathology are associated with distinctive mentalizing difficulties [5,38,42]. MBI was developed to assess the ability to reflect on mentalizing breakdowns in episodes of strong emotions in close relationships typically seen in patients with BPD. However, seeing borderline personality pathology as representing severe personality dysfunction more generally, the MBI could turn out to be a useful method for a broader range of severe personality pathology.

Conclusions

The MBI is a new method to assess mentalizing capacity in patients with BPD, focusing on patients’ ability to reflect on mentalizing breakdown episodes in retrospect. The interrater reliability for the MBI-RF scores was good. The MBI as a short easily-administered interview may be a feasible method in psychotherapy research aiming to provide more insight into the mechanisms of change in the treatment of borderline personality pathology. While in need for replication, a next step is also to investigate the clinical validity of the MBI as an RF assessment method in patients with BPD.

Ethical approval

All participating patients gave their written consent to use anonymous, clinical data for research purposes. The interviews were videotaped and stored in accordance with procedures approved by the data protection officer at Oslo University Hospital. Since the data extracted from the videos were saved anonymously, formal approval from the Norwegian State Data Inspectorate and Regional Committee for Medical Research and Ethics was not required.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The dataset used and analyzed during the current study can be made available on special request.

Additional information

Funding

The research was funded by Oslo University Hospital. Otherwise, no funds, grants, or other support was received.

Notes on contributors

Dag Anders Ulvestad

Dag Anders Ulvestad (MD) is senior consultant at Section for Personality Psychiatry and Specialized Treatments, Oslo University Hospital. He is certified MBT therapist and supervisor in psychodynamic psychotherapy. He is currently a PhD student at the University of Oslo on a project concerning assessment of mentalizing within psychotherapy research.

Merete Selsbakk Johansen

Merete Selsbakk Johansen, (MD, PhD) is a psychiatrist in private practice. She is specialized in psychoanalysis and member of the Norwegian Psychoanalytic Society and the International Psychoanalytical Association (IPA). She is an active researcher, and specialized in assessment of reflective functioning, certified and trained by Anna Freud Centre, London.

Elfrida Hartveit Kvarstein

Elfrida Hartveit Kvarstein, (MD, PhD) is head senior consultant at Section for Personality Psychiatry and Specialized Treatments, Oslo University Hospital and associate professor at the University of Oslo (UiO). She is leader of the Research group for Personality Psychiatry, UiO and specializes in clinical research on personality disorder.

Geir Pedersen

Geir Pedersen, (MA, PhD) is leader of the Norwegian Network for Personality Disorders at Section for Personality Psychiatry and Specialized Treatments, Oslo University Hospital. He is an active researcher, and specializes in clinical research on personality disorder, assessment methods, clinical quality systems and psychometrics.

Theresa Wilberg

Theresa Wilberg is a professor of psychiatry at University of Oslo and senior researcher at Oslo University Hospital. She has published several scientific articles and book chapters within the field of personality psychiatry, and has co-authored a textbook on personality psychiatry and a book on avoidant personality problems.