2,176
Views
10
CrossRef citations to date
0
Altmetric
Research Article

Validity of A Web-based Measure of Borderline Personality Disorder: A Preliminary Study

ORCID Icon, ORCID Icon, , ORCID Icon, & ORCID Icon

ABSTRACT

Purpose

Early identification of borderline personality disorder (BPD) is important as timely interventions are beneficial and cost-effective. Unrecognized BPD not only results in suffering for individuals and their families, but also leads to considerable social costs. Although web-based measures have the potential to facilitate screening assessment of BPD for research and clinical purposes, little is known about whether the results of web-based measures of BPD are valid. This preliminary study aims to examine the validity of a web-based measure of BPD.

Method

We analyzed data from five independent samples (N = 828 in total).

Results

The web-based BPD measure had a consistent relationship with relevant variables, including trauma exposure, mental well-being, depression, post-traumatic stress, dissociation and psychotic features across samples. It was also strongly correlated with another BPD measure. The web-based BPD measure could discriminate between participants with and without BPD and the discrimination performance was excellent (area under the curve =.853).

Discussion

The initial findings suggest that the web-based BPD measure used in the present study is valid and may be helpful for research and screening purposes, although it should be followed up with a more comprehensive assessment in clinical settings. Implications are discussed. Given the limitations in this study, further studies are needed.

Borderline personality disorder (BPD) is a mental health condition characterized by “a pervasive pattern of instability of interpersonal relationships, self-image, and affects, and marked impulsivity” (American Psychiatric Association, Citation2013, p. 663). According to DSM-5, there are nine specific diagnostic criteria for BPD (e.g., impulsivity, chronic emptiness, identity disturbance, unstable and intense interpersonal relationships); to be positive for BPD, one should meet at least five criteria (American Psychiatric Association, Citation2013). BPD is often considered a severe disorder as it is also associated with chronic impairments in psychosocial and occupational functioning (Alvarez-Tomás et al., Citation2017; Gunderson et al., Citation2011), suicidal behaviors (Black et al., Citation2004) and high health care costs (Bode et al., Citation2017; Meuldijk et al., Citation2017). The lifetime prevalence of BPD is about 1.7% in the general population, 6% in primary care settings, and 15-28% in psychiatric settings (Gunderson et al., Citation2018).

Individuals with BPD often require psychosocial interventions to tackle their interpersonal and intrapersonal problems (e.g., interpersonal dysfunction, fear of abandonment, anger) (Goodman & Siever, Citation2012), and therefore social workers and other psychosocial service providers could play an important role in the management of BPD. As the management of BPD typically needs a systems approach, social work’s tradition of multi-level practice may be particularly important when working with individuals with BPD (Smith, Citation2008).

Early identification of BPD is important because the empirical literature suggests that specific interventions for BPD are beneficial and cost-effective (e.g., Meuldijk et al., Citation2017; Soler et al., Citation2009). However, for a variety of reasons (e.g., insufficient diagnostic rigor, stigma, high comorbidity), BPD is often under-assessed and underdiagnosed in clinical settings (Morgan & Zimmerman, Citation2015). Failing to identify BPD has been said to be a public health issue given its severe personal and economic consequences for both the individual and society (Porr, Citation2017). Standardized assessments are helpful and can assist in the assessment of BPD.

Forty years ago, Gunderson and Kolb (Citation1978) first demonstrated that BPD could be discriminated from other diagnostic groups using standardized assessments. Since then, a number of BPD measures have been developed and validated. Given the rapid technology development in recent years, online methods can now largely facilitate assessment and intervention in health care and social service settings (Barak & Grohol, Citation2011; Chan et al., Citation2017; Chan & Holosko, Citation2016; Nguyen et al., Citation2015). Online assessment has been increasingly common for both research and clinical purposes to improve screening, monitoring and evaluating service users or research participants. There are many advantages of using online methods to assess mental health conditions, such as reduced costs and higher accessibility. Online assessment is also very important when face-to-face activities are not very possible (e.g., during pandemics). Yet, whether or not web-based measures are valid remains a concern in the field. Therefore, some efforts have been made to examine the validity of web-based measures of different mental health conditions, such as perceived stress and depression (Herrero & Meneses, Citation2006), anxiety (Donker et al., Citation2011), post-traumatic stress (Fung, Chan et al., Citation2019) and dissociation (Fung, Choi et al., Citation2018). However, little is known about whether web-based BPD measures can assess BPD with acceptable validity. A recent study that examined the validity of a web-based self-report questionnaire for DSM-IV disorders (TeleScreen) found that it has moderate to good validity for BPD in a Dutch Context (N = 89) (Chakhssi et al., Citation2019). To our knowledge, no other study has been conducted to examine the validity of web-based measures of BPD.

Against this background, the present study aims to examine the validity of a web-based BPD measure, that is the BPD section of the Self-Report Dissociative Disorders Interview Schedule (SR-DDIS), in five independent samples. Informed by the literature that suggests that BPD is associated with trauma (Distel et al., Citation2011), poor social support (Beeney et al., Citation2018; Zielinski & Veilleux, Citation2014), aggression (especially reactive aggression) (Raine, Citation1993; Ross & Babcock, Citation2009) and mental dysfunction (e.g., depression, post-traumatic stress, psychotic features, dissociation) (Grant et al., Citation2008; Kuijpers et al., Citation2011; Schroeder et al., Citation2013; Trull, Citation1995; Zanarini et al., Citation2000), we made and tested the following hypotheses:

  1. The web-based BPD measure would have a consistent, positive relationship with trauma exposure across samples.

  2. The web-based BPD measure would have a consistent, negative relationship with social support across samples.

  3. The web-based BPD measure would have a consistent, negative relationship with mental well-being across samples.

  4. The web-based BPD measure would have a consistent, positive relationship with other mental health problems, including depression, post-traumatic stress, psychotic features and dissociation.

  5. The web-based BPD measure would have a stronger relationship with reactive aggression than with proactive aggression.

  6. The web-based BPD measure would be strongly correlated with another BPD measure assumed to be measuring the same construct.

  7. Participants with BPD would be more likely to endorse all nine BPD symptoms and to score significantly higher on the web-based BPD measure than participants without BPD.

  8. The web-based BPD measure could detect clinically diagnosed BPD with an acceptable discrimination performance (area under the curve [AUC] > .7).

Method

The five independent samples

Sample 1 derived from a study on trauma and mental health in a convenience online sample of Chinese females with disabilities (81.9% with physical disabilities) (N = 94). This study was approved by the Executive Committee/Board of the Association of Women with Disabilities Hong Kong and has been reported elsewhere (Sample 4 in that paper) (Fung, Liu et al., Citation2019). Participants were recruited through online channels to complete an online survey that included measures of trauma, family support, mental well-being and mental health problems.

Sample 2 derived from a study on childhood adversities and mental health in a convenience online sample of Hong Kong adults (N = 418) (part of the data has been reported elsewhere)(Fung, Chung, & Ross, Citation2020). This study was approved by the institutional review board of the City University of Hong Kong (CityU). Hong Kong adults were recruited through online channels to complete an online survey that included measures of childhood adversities, family well-being and mental health problems.

Sample 3 derived from a study on dissociation and aggression in a sample of college students (N = 190). This study was also approved by the institutional review board of the CityU has been reported elsewhere (Fung, Ling et al., Citation2019). College students of CityU were recruited through either flyers or internal e-mail invitations to complete an online survey that included measures of dissociation, BPD and aggression.

Sample 4 derived from a study on patients with self-reported depression (N = 68). This study was approved by The Hong Kong Polytechnic University (PolyU) Human Subjects Ethics Sub-committee and has been described and reported elsewhere (Fung & Chan, Citation2019). Hong Kong patients who self-reported to have been clinically diagnosed with depression were recruited through online channels to complete an online survey that included measures of dissociation, BPD and related symptoms.

Sample 5 derived from a validation study in a sample of Taiwan psychiatric outpatients (N = 58). This study was also approved by the Human Subjects Ethics Sub-committee of PolyU (part of the data have been reported elsewhere) (Fung, Chan et al., Citation2019). In this study, adult psychiatric patients with and without BPD from a Taiwan psychiatric clinic were invited to complete different online forms that included the same set of measures (measures of trauma, BPD and post-traumatic stress). Their BPD status was confirmed with clinical diagnostic interviews, which were conducted by an experienced psychiatrist according to DSM-5 rules. There were no significant differences in gender and trauma exposure (assessed with the Life Events Checklist for DSM-5) between participants with and without BPD. Participants with BPD (n = 21) were significantly younger (M = 29.2; SD = 7.25 vs M = 35.2; SD = 12.5)(t = 2.008, p = .049), rated their own mental health significantly lower (M = 2.14; SD = 1.06 vs M = 3.35; SD = 1.18)(t = 3.874, p < .001), and scored significantly higher on the Post-traumatic Stress Disorder Checklist for DSM-5 (M = 59.7; SD = 14.1 vs M = 41.4; SD = 18.7)(t = 3.907, p < .001), compared with participants without BPD (n = 37).

Informed consent from each participant was obtained in the online survey in each study. No personally identifiable information (e.g., phone number, name, date of birth) was recorded. Demographic background of each sample is summarized in . Sample 3 was recruited in a university setting (flyers or internal e-mail), Sample 5 was recruited in a clinical setting (flyers in the clinic), and the other three samples were recruited completely online (e.g., social media platforms, web-based groups) (therefore, comparisons of the web-based BPD measure results were only made among Sample 3 and Sample 5). All surveys were conducted in Chinese. All surveys included the same web-based BPD measure, that is the BPD section of the Self-Report Dissociative Disorders Interview Schedule (SR-DDIS-BPD).

Table 1. Demographic background of each sample.

Measures

Borderline Personality Disorder (BPD) was assessed with the web-based SR-DDIS-BPD in all samples. The DDIS is a reliable and valid diagnostic instrument for a number of DSM-5 diagnoses (including dissociative disorders and BPD) and can be used as a self-report measure (Ross & Browning, Citation2017; Ross et al., Citation1989). The BPD section, in which the items were taken verbatim from DSM-5, includes nine items that correspond to the nine diagnostic criteria for BPD in DSM-5. An example of the items is: “intense anger or lack of control of anger, e.g., frequent displays of temper, constant anger, recurrent physical fights”. Participants can answer “yes”, “no” or “not sure” for each item. The total score is the number of “yes”. According to DSM-5 rules, one must endorse at least five items to meet the DSM-5 BPD criteria. In the Chinese version, the ninth item of the BPD section provides additional examples of dissociative symptoms because dissociation is a rarely-mentioned difficult-to-understand term in the local context. The Chinese version of the SR-DDIS has been validated and can discriminate between participants with and without a dissociative disorder (Fung, Choi et al., Citation2018). The DDIS-BPD/SR-DDIS-BPD has been used in a number of studies to assess BPD (Fung, Ho et al., Citation2018; Ross, Citation2007; Ross et al., Citation2002, Citation2014; Xiao et al., Citation2006). However, the performance of the DDIS-BPD/SR-DDIS-BPD as a measure for assessing BPD has not been evaluated. In addition, the 20-item Taiwan version of the Borderline Personality Inventory (BPI-T20) was also administered to Sample 5. The BPI-T20 is a reliable and valid measure and can detect BPD with a sensitivity of 89.8% and a specificity of 82.8% using 11/12 as the cutoff point (Lee et al., Citation2009).

Trauma exposure was assessed with the Chinese versions of the Life Events Checklist (LEC) (Chu, Citation2004; Gray et al., Citation2004), LEC for DSM-5 (LEC-5) (Fung, Chan et al., Citation2019; Weathers et al., Citation2013) and the 10-item Adverse Childhood Experiences (ACE) Questionnaire (Bruskas & Tessin, Citation2013; Fung, Ross et al., Citation2019) in Sample 1, Sample 5 and Sample 2, respectively.

Social support was assessed with the Chinese version of the Multidimensional Scale of Perceived Social Support-Family Subscale (MSPSS-FS) (Chou, Citation2000; Zimet et al., Citation1990) in Sample 1 and Sample 4. In addition, a 4-item Chinese measure, the Overall Family Well-being Scale (OFWS) (Hong Kong Family Welfare Society, Citation2018), was used to assess family well-being in Sample 2.

Mental well-being was assessed with the Chinese version of the Subjective Happiness Scale (Lyubomirsky & Lepper, Citation1999; Nan et al., Citation2014) in Sample 1. In addition, a single-item measure of self-rated mental health (SRMH) (Ahmad et al., Citation2014) was used to assess self-rated mental health in Sample 1, Sample 2 and Sample 5 (on a scale ranging from 1 = poor, to 5 = excellent).

Depression was assessed with the Chinese version of the Patient Health Questionnaire-9 (PHQ-9) (Kroenke & Spitzer, Citation2002; Yeung et al., Citation2008) in Sample 1 and Sample 2.

Post-traumatic stress was assessed with the Chinese versions of the Post-traumatic Stress Disorder Checklist (PCL) (Weathers et al., Citation1993; Wu et al., Citation2008) and PCL for DSM-5 (PCL-5) (Blevins et al., Citation2015; Fung, Chan et al., Citation2019) in Sample 2 and Sample 5, respectively. In addition, the Chinese version of the Primary Care PTSD Screen (PC-PTSD) (Cameron & Gusman, Citation2003) was used in Sample 1, although its psychometric properties have not been evaluated.

Dissociation was assessed with the Chinese versions of the Dissociative Experiences Scale-Taxon (DES-T) (Waller & Ross, Citation1997) and the 5-item Somatoform Dissociation Questionnaire (SDQ-5) (Nijenhuis et al., Citation1997), both of which have been validated in the Chinese context (Fung, Choi et al., Citation2018). The DES-T was used in Sample 3 and Sample 4, while the SDQ-5 was used in Sample 2, Sample 3 and Sample 4.

Psychotic features were assessed with the 4-item positive symptom dimension (frequency scale) of the Chinese version of the Community Assessment Psychic Experiences (CAPE-P) (Konings et al., Citation2006; Mark & Toulopoulou, Citation2017) in Sample 1. In addition, psychotic features were also assessed with the Schneiderian first-rank symptoms section (e.g., “voices arguing in your head”, “hearing your thoughts out loud”) of the SR-DDIS in Sample 3 and Sample 4.

Aggression was assessed with the Chinese version of the Reactive–Proactive Aggression Questionnaire (RPQ) (Kwan, Citation2011; Raine et al., Citation2006) in Sample 3.

shows the measures that were used in each sample.

Table 2. Pearson correlations of the web-based BPD measure with other psychosocial and mental health variables across samples.

Data analysis

Statistical analysis was conducted using SPSS 22.0. Pearson’s correlation was used to investigate the relationship between SR-DDIS-BPD scores and other measures across samples. Differences in other major variables between participants with and without BPD were also examined using one-way ANOVA and chi-square tests. We also calculated the sensitivity and specificity of the SR-DDIS-BPD for detecting DSM-5 BPD in Sample 5 (receiver operating characteristic curve analysis).

Results

As shown in , the web-based BPD measure (i.e., the SR-DDIS-BPD scores) had a consistent, positive correlation with trauma exposure, mental health problems (including depression, post-traumatic stress, dissociation and psychotic features) and a consistent, negative correlation with mental well-being across samples. The SR-DDIS-BPD scores had a negative correlation with social support in two out of three samples. The SR-DDIS-BPD scores also had a stronger correlation with reactive aggression than with proactive aggression.

As shown in , the SR-DDIS-BPD can also discriminate between patients with and without BPD based on clinical diagnostic interviews, and between patients with BPD and healthy college students recruited in a university setting (i.e., Sample 3). Compared with the other two groups, patients with BPD were more likely to endorse all nine BPD symptoms and they also scored significantly higher on the SR-DDIS-BPD (total score).

Table 3. Differences in the web-based BPD measure between participants with and without BPD.

In Sample 5 in which the BPD status was confirmed with clinical diagnostic interviews, the area under the curve (AUC) of the SR-DDIS-BPD was .853, which is considered excellent (Hosmer & Lemeshow, Citation2000). The sensitivity and specificity of the SR-DDIS-BPD were reported in . According to DSM-5 rules, one must endorse at least five items to meet the diagnostic criteria for BPD. In this sample, using 5 as the cutoff score, the SR-DDIS-BPD can detect BPD with a sensitivity of 95.2% and a specificity of 64.9%. A cutoff score of 6 seems to yield an optimal balance between sensitivity and specificity. Moreover, the two web-based BPD measures (i.e., the SR-DDIS-BPD and the BPI-T20) were strongly correlated with each other (r = .792, p < .001).

Table 4. Sensitivity and specificity of web-based BPD measure in detecting BPD in Sample 5.

Discussion

This preliminary study examined the validity of a web-based BPD measure, that is the SR-DDIS-BPD. Data from five independent online surveys were analyzed. The web-based BPD measure had a consistent relationship with relevant variables, including trauma exposure, mental well-being, depression, post-traumatic stress, dissociation and psychotic features. Two different web-based BPD measures (i.e., the SR-DDIS-BPD and the BPI-T20) were strongly correlated with each other. The SR-DDIS-BPD could discriminate between participants with and without BPD and the discrimination performance was excellent. The initial findings supported our hypotheses, except for hypothesis 2 (the SR-DDIS-BPD was negatively correlated with social support only in two out of three samples). As mentioned, we expected that there is a negative correlation between the SR-DDIS-BPD scores and social support but have no specific explanation or hypothesis for why this was not true in one sample; further investigation is needed. This preliminary study supports the validity of the web-based BPD measure.

This study suffers from some limitations. For example, the test-retest reliability of the web-based BPD measure and its sensitivity to change as an evaluation measure were not examined; most participants were female; the sample size for evaluating the discrimination performance was not large enough; the participants were not matched in any way; in addition, patients with BPD in Sample 5 exhibited high levels of post-traumatic stress, and therefore they may not represent BPD patients with low levels of post-traumatic stress. Moreover, only one single psychiatrist was involved in the diagnostic process in Sample 5, and no structured diagnostic interviews were used. Therefore, The findings should be interpreted with caution and future studies are needed to further examine the validity of web-based BPD measures (e.g., testing other BPD measures such as the McLean Screening Instrument for Borderline Personality Disorder, employing a more representative sample, using structured diagnostic interviews for BPD) in various language and cultural contexts. Despite the limitations, our initial findings are encouraging and have some implications that require further discussion.

Our findings are consistent with the recent study which suggests that BPD can be assessed online with good validity (Chakhssi et al., Citation2019). A valid web-based BPD measure, just like other web-based mental health measures, would be very useful for both research and clinical purposes. For example, it would largely facilitate research on BPD and other issues related to BPD as online survey can include the SR-DDIS-BPD to assess BPD symptoms together with other self-report measures. A web-based BPD measure with excellent discrimination performance would have the potential to encourage more low-cost epidemiological studies of BPD that use online methods to collect data. It would be interesting if future studies could compare the results of paper-based vs. web-based BPD measures.

In addition, it may also be helpful for screening for BPD for clinical purposes, although further evaluation is needed and it cannot replace diagnostic interviews. For instance, social workers and other psychosocial service providers may use the web-based BPD measure as initial screening in health care and social service settings. Clients or potential clients may be asked to complete the measure using a tablet, mobile device or computer in the waiting room or at home. Mental health service providers may also set up a website that provides a valid web-based BPD measure for people to self-assess their BPD symptoms for the purposes of early identification and outreach for those in need in the community.

Web-based measures would be particularly helpful for BPD because BPD is often under-identified and because people with BPD are often stigmatized in the health care system and in the general population. Unrecognized BPD has been said to be a considerable public health issue (Porr, Citation2017). As some practitioners may find it challenging to assess BPD (Biskin & Paris, Citation2012), a valid web-based measure with excellent sensitivity may help prevent missing cases of BPD in non-specialized settings – once a client screens positive for BPD in the online assessment, he/she can be referred for follow-up assessment and differential diagnosis. This could facilitate early identification of BPD so as to ensure that more people with BPD could receive timely BPD-specific treatments. It is important to note that, as long as the correct assessment is made, there are empirically-supported interventions, including online interventions (Rizvi et al., Citation2016) and psychotherapies that can be delivered by trained social workers.

This preliminary study shows that, just like other mental health problems (e.g., depression, anxiety and dissociation), BPD can be assessed online – the web-based BPD measure that we evaluated in the present study are valid. While the web-based measure cannot replace clinical assessment and structured interviews, as Nguyen et al. (Citation2015) said, “for many consumers who are unable or unwilling to access traditional services, Internet-based programs could offer a ‘good enough’ alternative for identifying mental health disorders.” The web-based BPD measure may be helpful for social workers and other psychosocial service providers to assess BPD symptoms. It also has the potential to facilitate screening for BPD in research settings and in non-specialized health care and social service settings. Nevertheless, further studies are needed.

Acknowledgment

The preparation of this manuscript was partially supported by fundings from the Hong Kong Polytechnic University and the Hong Kong Jockey Club Charities Trust.

References