Using the Delphi Method for Determining Key Performance Elements for Delivery of Optimal Suicide-Specific Interventions in Emergency Departments

Abstract

Objective

Evidence-based suicide prevention interventions directed to those seeking psychiatric crisis services for suicidality in the emergency department (ED) can reduce death by suicide and related suffering. Best practice guidelines for the care of suicidal patients in the ED exist but are not accompanied by fidelity tools for use in determining whether the interventions were applied, particularly when more than one intervention is delivered concurrently. We sought to develop a universal, treatment-agnostic Suicide Care Fidelity Checklist comprised of Key Performance Elements (KPE) across the recommended suicide-specific ED interventions.

Method

A comprehensive review of published care standards was first conducted to determine suicide-specific ED best practice treatment domains and KPEs. Subject matter experts (SMEs) were identified for each domain. Using the Delphi Consensus method, SMEs iteratively revised and refined the KPEs within their domain until achieving KPE item consensus.

Results

A total of three iterations was required to obtain consensus in five of six domains: comprehensive suicide assessment, lethal means counseling, suicide crisis planning, behavioral skills training, and psychoeducation about suicidality. Consensus was not fully attained for the domain involving engagement with people with lived experience.

Conclusions

We successfully identified six intervention domains and 74 KPEs across domains (60 deemed essential, and 14 deemed optional), with full consensus reached for 70 KPEs. While replication of the initial findings is required, the Suicide Care Fidelity Checklist can be used as a fidelity checklist to verify delivery of suicide-specific ED interventions.

HIGHLIGHTS

• Applied Delphi Consensus method with suicide-specific subject matter experts.

• Generated a treatment-agnostic, universal set of suicide prevention KPEs for EDs.

• Expert-derived KPEs help real-world settings to assess suicide care fidelity.

Keywords:

• Delphi method fidelity
• emergency department
• suicide

With 44,834 suicides in 2020, suicide is currently the 11th leading cause of death among all ages (Centers for Disease Control and Prevention (CDC), 2016) and the second leading cause of death among those 10–34 years old (Centers for Disease Control and Prevention (CDC), 2014) in the U.S. and a significant, yet preventable, public health problem. Approximately 1.4 million adults made a suicide attempt in 2019; worldwide, nearly 1.5 million people make a suicide attempt annually (National Institute of Mental Health (NIMH), n.d.). In addition to those who die by suicide or make an attempt, another 12 million American adults had serious suicidal thoughts in 2019 (Ahmedani et al., 2014; Centers for Disease Control and Prevention (CDC), n.d.; National Institute of Mental Health (NIMH), n.d.; Substance Abuse and Mental Health Services Administration, n.d.). With the exception of 2019 and 2020 (Centers for Disease Control and Prevention (CDC), 2021; Kochanek, Xu, & Arias, 2020), suicide rates have steadily increased on an annual basis since 2005, from 11 per 100,000 to 14.8 per 100,000 (Centers for Disease Control and Prevention (CDC), 2016, Insel, 2017), despite the expansion and evaluation of numerous suicide-specific interventions and related tools (Brodsky, Spruch-Feiner, & Stanley, 2018; McCabe, Garside, Backhouse, & Xanthopoulou, 2018; Stanley & Brown, 2012).

Healthcare systems in general and emergency departments (ED) in particular are among the most important settings in which to focus suicide prevention efforts as those who seek ED services following a suicide attempt are at elevated risk for making another attempt and eventually dying by suicide (Asarnow, Babeva, & Horstmann, 2017). For example, as many as 25% of patients seeking ED services following a suicide attempt are likely to make another attempt, and between 5 and 10% are likely to eventually die by suicide (Asarnow et al., 2017). Of those who do die by suicide, a significant number will have visited the ED for suicidality the year before their death (Gairin, House, & Owens, 2003; Larkin & Beautrais, 2010; Wilson & Klein, 2000). Additionally, the risk of another suicide attempt or death is highest within the first 30 days following discharge from an ED or inpatient psychiatric hospitalization, with up to 70% of patients never attending their first outpatient appointment after leaving the ED following a suicide attempt (Knesper, 2010). Directing evidence-based suicide prevention interventions to those who seek psychiatric crisis services for suicidality in the ED holds the promise of reducing suicide rates—by as much as 20% (National Action Alliance for Suicide Prevention: Research Prioritization Task Force, 2015).

In light of their heightened suicide risk and the unique opportunity to intervene, expert-derived care standards for suicidal ED patients have been recommended (Suicide Prevention Resource Center, 2015), including: comprehensive suicide risk assessment, brief patient education, safety planning, lethal means counseling, rapid referral to aid in the care transition, caring contacts, and crisis center information. Other recommended suicide-specific care standards (American Psychiatric Association, 2010, 2015; National Action Alliance for Suicide Prevention: Transforming Health Systems Initiative Work Group, 2018; Stone et al., 2017; The Joint Commission, 2016, 2017, 2019) include the active involvement of people with lived experience (PLE) to share hope and wisdom and serve as credible messengers (Dimeff & Goering, 2020; Dimeff & Jerome, 2019, 2020; Narino, 2019; Rittenbach et al., 2019; Simon et al., 2016; Suicide Attempt Survivors Task Force & National Action Alliance for Suicide Prevention, 2014; Whiteside, 2017; Whiteside et al., 2014; Zero Suicide In Health and Behavioral Healthcare, n.d.), as well as behavioral skills training to reduce intense emotional distress and regulate emotions (Linehan, 1993; Lynch, Trost, Salsman, & Linehan, 2007) while in the ED and after returning home during the window of risk. While guidance for suicide-specific ED interventions is well-described, determining whether a particular intervention was actually delivered is less clear. Key performance elements (KPEs) and adherence measurements for these interventions either do not exist, are designed for implementation of research protocols and unwieldly, are buried in guidance documents, and/or are intended for use when delivering a specific evidence-based protocol (e.g. Safety Planning (Stanley & Brown, 2012), Crisis Response Planning (Bryan, 2010; Bryan et al., 2017), or Stabilization Planning (Jobes, 2012)) but are ill-suited for use in many real-world settings that combine a range of different evidence-based procedures.

The present study was undertaken as part of an ongoing research effort involving the development and evaluation of Jaspr Health (Dimeff et al., 2020, 2021), an evidence-based suicide prevention digital platform designed for use by acutely suicidal patients seeking psychiatric crisis care in the ED. Grounded in David Jobes’ Collaborative Assessment and Management of Suicidality (CAMS), Jaspr Health guides patients in completing a comprehensive suicide risk assessment and lethal means counseling, builds a crisis stabilization plan, and teaches behavioral skills to reduce imminent distress; video-recorded messages by PLEs offer wisdom and hope for getting through suicide crises. Information gathered by Jaspr Health is summarized for the care team to aid in discharge disposition planning. A companion app, Jaspr-at-Home, provides ongoing support post-discharge. In our view, Jaspr Health’s potency is the substance of the evidence-based interventions contained within it rather than the digital form in which the delivery takes place—in other words, the extent to which it delivers suicide-specific EBPs to fidelity. Like a medication given in either pill or liquid form, the active ingredient remains the medication itself, not the medium through which it is delivered.

To evaluate Jaspr Health’s fidelity across multiple evidence-based suicide interventions, we first had to develop a universal, treatment-agnostic Suicide Care Fidelity Checklist comprised of KPEs using the Delphi Consensus method (Diamond et al., 2014) that we could then use to evaluate Jaspr Health’s fidelity across multiple evidence-based suicide interventions. Originally developed to predict cold war enemy attack probabilities (San-Jose & Retolaza, 2016), the technique has been used widely to seek expert opinion using a structured, iterative approach (Diamond et al., 2014). While not a requirement of Delphi, consensus opinion is most often an expected goal, where experts eventually arrive at a shared opinion. Key features of this method include anonymity between participants and controlled feedback provided in a structured manner by the research team (Diamond et al., 2014). Participants then adjust their initial ratings based on feedback from others in a number of subsequent iterations (Black et al., 1999; Duffield, 1993; Hsu & Sandford, 2007; Keeney, Hasson, & McKenna, 2006).

METHOD

A comprehensive review of published care standards from the Action Alliance for Suicide Prevention (National Action Alliance for Suicide Prevention: Transforming Health Systems Initiative Work Group, 2018), American Association of Emergency Psychiatry (American Association of Emergency Psychiatry, 2017), American College of Emergency Physicians (ACEP Emergency Medicine Practice Committee, 2014; American College of Emergency Physicians Clinical Policies Subcommittee (Writing Committee) on the Adult Psychiatric Patient, 2017), American Psychiatric Association (American Psychiatric Association, 2010, 2015), Centers for Disease Control and Prevention (Stone et al., 2017), Emergency Nurses Association (Emergency Nurses Association, 2012), Centers for Medicare and Medicaid Services (Stone et al., 2017), the Department of Veteran Affairs and Department of Defense (The Assessment & Management of Risk for Suicide Working Group, 2013; 2019), Harvard T. H. Chan School of Public Health (Harvard Injury Control Research Center, n.d.), the Joint Commission, Massachusetts Board of Registration in Medicine (Commonwealth of Massachusetts Board of Registration in Medicine, Quality and Patient Safety Division, 2014), Suicide Prevention Resource Center (Suicide Prevention Resource Center, 2015), and Substance Abuse and Mental Health Services Administration (Substance Abuse and Mental Health Services Administration (SAMHSA), 2020) was conducted. Major domains were identified and KPEs relevant for ED care were extracted. Inclusion criteria for domains included care standards for acutely suicidal patients that could be provided in the ED (versus after a patient was discharged from the ED). These criteria thus precluded inclusion of universal screening of suicidality and caring contacts. Extracted domains included: comprehensive suicide risk assessment, lethal means safety counseling, crisis stability planning, psychoeducation, behavioral skills training, and engagement with PLEs. An initial list of KPEs for each domain was generated based on this initial review and served as a starting point for subsequent expert review.

Subject matter experts (SMEs) for each domain were then identified to affirm and/or refine the initial list of KPEs. SME inclusion criteria included: treatment developers and/or their protégés for a specific suicide intervention; recognized researchers who publish in particular domain; internationally-recognized suicide prevention experts; and nationally-recognized PLEs. A total of 29 experts across the initial six identified domains were invited to participate in the study. Five (17%) declined participation due to time constraints but recommended their protégés to participate on their behalf. Ultimately, 24 (83%) agreed to participate: four SMEs for the comprehensive suicide risk assessment domain (David Jobes, PhD, Kate Comtois, PhD, Peter Gutierrez, PhD, and Amy Brausch, PhD); four for the lethal means counseling domain (Marian Betz, MD, Mike Anestis, PhD, Amy Barnshorst, MD, and Shannon Frattaroli, PhD); four for the suicide crisis plan domain (Craig Bryan, PsyD, Christa Labouliere, PhD, Kelly Green, PhD, and Lauren Weinstock, PhD); five for the behavioral skills training domain (Joan Asarnow, PhD, Lucas Zullo, PhD, Jocelyn Meza, PhD, Nick Salsman, PhD, and Andrada Neacsiu, PhD); four for the psychoeducation domain (DeQuincy Lezine, PhD, Edwin Boudreaux, PhD, Jeffrey Sung, MD, and Julie Goldstein Grumet, PhD); and four for the PLE domain (Kelechi Ubozoh, Dese’Rae Stage, Jessica Stohlmann-Rainey, and DeQuincy Lezine, PhD). SMEs were informed at the start that our intention was to apply consensus process to define a universal set of behaviorally-specific KPEs for their domain. To guard against potential bias and other factors that may compromise a truly independent evaluation, SMEs were blind to other domains and domain SMEs with the exception of one rater who was assigned to the two domains (lived experience and psychoeducation.

Delphi Consensus Method and Task Process of Development

Consistent with the Delphi method, we proceeded in a structured, step-wise process defined by anonymity between participants. We operationally defined consensus apriori as 100% agreement by SMEs about KPEs in their respective domain. In total, each SME was asked to complete a total of three tasks. The first task involved reviewing our original list of KPEs for their respective domain for purposes of generating an exhaustive expert-verified and created list. They first added KPEs to the list that were not previously identified. They then sorted the elements into two categories: essential (i.e. must be included in fidelity list for their domain) or non-essential/optional (i.e. could be removed from the list). After receiving their ratings, we then compiled a second document that summarized all domain KPEs generated and verified in Task 1 as well as SME ratings for each domain. This second list did not interpret any of their findings, but instead simply displayed the items deemed essential or unessential by each rater. A separate column was generated for each rater to easily see their own original ratings for ease in comparing their responses to others. This summary document for each domain was provided to domain SMEs upon conclusion of Task 1.

Task 2 sought to verify KPEs based on feedback from other domain experts. Raters reviewed the compiled summary document from Task 1 then rated each KPE as “Essential,” “Optional,” or “Remove” (i.e. those that did not belong in the domain). To aid with the consensus process and illuminate their reasoning behind a particular rating, SMEs provided a rationale for items where there were differences in views from Task 1. A third summary document was then created for each domain that again compared their answers to others domain SMEs. This document also included item-specific comments to help other raters better understand their peers’ reasoning. In contrast to the second document, this third summary document reduced redundancy across items by consolidating like-items. All such edits were notated for later review and approval by SMEs.

The final task involved SMEs reevaluating those items where consensus had not been achieved, as well as items edited in the earlier step to reduce redundancy. Each SME viewed the de-identified ratings of other domain experts, as well as comments provided by other experts during the prior task. They were again asked to provide a rating of “Essential,” “Optional,” or “Remove” for the remaining items. Unlike the previous tasks, SMEs were also asked to rate the veracity of their view using a three-point scale (1= low strength, e.g. they did not feel strongly about their rating and could easily go along with the opinion of other domain experts if others felt more strongly about the particular item; 2= medium strength, e.g. they were open to hearing the views of their peers and reevaluating their rating; and = high strength, e.g. they were unlikely to change with their ratings).

The Task 3 ratings were again reviewed by the team and open items were identified. Researchers then emailed those whose views remained outside the others’ ratings for purposes of better understanding their position and determining next steps. To preserve the blind, effort was made to resolve the disparity between item differences by having the research team communicate directly between SMEs to refine the wording of specific items until consensus was achieved.

RESULTS

Participants

A total of 23 of the original 24 SMEs (95.8%) participated in all tasks. One SME completed the first task but was unable to complete the remaining tasks due to time constraints and/or other personal reasons.

Comprehensive Suicide Assessment

The four domain experts reached consensus on 16 items (Table 1). Fourteen items were rated in the final third round as essential, and two items were rated as optional. Examples of essential tasks included: assesses reasons for living and reasons for dying, including “drivers” of suicidality; determines level of risk; assesses and discusses events and factors that led to recent suicide attempt; assesses access to lethal means and willingness to reduce access, including steps to take to accomplish that; and assesses patient’s risk and protective factors.

TABLE 1. Suicide care fidelity checklist.

Domain	Item
Comprehensive suicide assessment	Essential items
	Assesses reasons for living and reasons for dying, including "drivers" of suicidality
	Determines level of risk
	Assesses and discusses events and factors that led to recent suicide attempt
	Assesses access to lethal means and willingness to reduce access, including steps to take to accomplish that
	Assesses patient’s risk and protective factors
	Assesses immediate danger
	Assesses treatment needs
	Includes decision support tool that assesses thoughts of carrying out a plan
	Includes decision support tool that assesses suicide intent
	Includes decision support tool that assesses past suicide attempts
	Includes decision support tool that assesses irritability, agitation, aggression
	Includes decision support tool that orients patient to process
	Includes decision support tool that tells them what happens next
	Includes decision support tool that has documentation
	Optional items
	Determines if substance use disorder
	Determines if significant MH condition and whether MH treatment is indicated
Lethal means counseling	Essential items
	Orients to steps patient and/or family/support system can take to make home safer
	Asks patient about access to lethal means (general)
	Asks specifically about access to firearms and their current location (specific)
	Asks about reasons for firearm ownership
	Identifies strategies to limit access
	Makes plan for reducing access as specific and personalized as possible
	Discusses what “safe storage” means – home protection vs. suicide prevention
	Avoids prescribing specific storage practice that is “required” – chooses instead to validate perspective of firearm owner and help them move along spectrum of safety
	Documents that the counseling discussion occurred and plan for reducing lethal means
	Orients to importance of lethality of methods
	Optional items
	Asks about types of firearms in the home
	Makes sure that firearm owners know how to use specific storage practices (e.g. how to install cable locks on different types of firearms)
	Follows up to assure appropriate care (preferably a warm handoff while still in the ED)
	Follows up to assure lethal means are removed/secured
	Orients to topic importance: rapid escalation
Suicide crisis planning	Essential items
	Includes warning signs
	Includes internal coping strategies (ways to distract oneself in a crisis)
	Family members/friends who can provide distraction and/or support
	Professionals and agencies who can help
	Can be developed digitally or on paper
	Orients patient to task (including providing rationale for each step and explaining how to use the plan in a crisis)
	Orients patient to task including description of suicide risk curve
	Discusses and plans where the person will keep the safety plan
	Identifies barriers to using safety plan and solutions to barriers at each step of the safety plan
	Discusses how to orient other people listed on the plan to the plan/discusses sharing the plan with identified support persons
	Task performed in a collaborative fashion
	Conducts a narrative interview of a recent suicidal crisis to elucidate warning signs/illustrate that a suicidal crisis passes
	Optional items
	Discusses how to orient outpatient provider to plan
	Tells patient that while plan is helpful, outpatient treatment can help them to better understand and address reasons for wanting to kill self
Psychoeducation about suicide	Essential items
	Provides information to patient about what to expect in the ED when suicidal
	Teaches behavioral skills to effectively manage distress while in the ED
	Provides strategies and skills to manage imminent suicide crises
	Teaches strategies for coping effectively post-discharge
	Reviews risk factors and warning signs for patient and family
	Discusses the nature of suicidal thought variability, transience, and the likely role of triggers and precipitants
	Discusses crisis resources including Lifeline and Text Line
	Teaches and/or provides information to loved ones about diagnosis and care
	Helps patient create a game plan for ensuring their safety during future suicide crises and understand its utility in a crisis
	Discusses rationale for means restriction during crisis
	Optional items
	Describes treatment options across continuum of care
	Teaches skills to talk about thoughts of suicide with care professionals
Behavioral skills training	Essential items
	Orients patient to rationale/purpose of the skill
	Instructs patient in skill to be learned
	Instructions are easy to follow and understand
	Provides example(s) of skill
	Links skill/concept to patient's needs and goals (relevance)
	Illustrates how to use and apply the skill/concept
	Practices skill with patient in session (skills strengthening)
	Elicits agreement to apply the skill (commitment)
	Verifies understanding on how to apply the information
	Troubleshoots barriers to use
	Optional items
	Discloses/describes use of skill in own life
	Increases the directedness of coaching in the skill to be used, depending on emotional intensity at the time of the intervention
	Engages in the same skill practice as the patient at the first demonstration of the skill
	Identifies and build on patient’s current skills
People with lived experience	Essential items
	Provides personal examples of coping with shame
	Provides examples of ways to get through ED visit (including information on what to expect throughout the process)
	Includes honest stories about the post-attempt experience, including the fact that the survivor/suicidal person may feel anger that they lived, or that they might not immediately feel like they want to live or that their life is worth living, and that this is okay
	Conveys belief in the individual's ability to survive this moment, and that "hope and recovery" sometimes feel hollow
	Conveys a sense of shared experience
	Provides personal perspective on the ED experience
	Validates the challenges of suicidal experiences
	Provides examples of things to consider when going home
	Provides examples of ways to maintain wellness
	Provides encouragement for life after suicide experience
	Provides examples of how to get through difficult moments
	Contextualizes and normalizes thoughts and feelings around suicide to mitigate shame and promote help seeking behavior
	Conveys expectation that suicidal thoughts may return and planning will be helpful
	Optional items
	Uses a strengths-based perspective

Lethal Means Counseling

Consensus by the four experts was achieved on 15 items within this domain (Table 1). Specifically, they rated 10 items as essential and five as optional. Examples of essential items include: orients to steps patient and/or family/support system can take to make home safer; orients to importance of lethality of methods; asks patient about access to lethal means (general); asks specifically about access to firearms and their current location (specific); and asks about reasons for firearm ownership, identifies strategies to limit access.

Suicide Crisis Planning

In this domain, the four SMEs reached consensus on 14 tasks (Table 1). Twelve tasks were rated as essential and two tasks were rated as optional. Examples of essential items include: orients patient to task (including providing rationale for each step and explaining how to use the plan in a crisis); orients patient to task including description of suicide risk curve; includes warning signs; conducts a narrative interview of a recent suicidal crisis to elucidate warning signs/illustrate that a suicidal crisis passes; includes internal coping strategies (ways to distract oneself in a crisis); and family members/friends who can provide distraction and/or support. No consensus was reached on the item “Ways to make environment safe (i.e. lethal means reduction).” Specifically, three SMEs rated this item as essential, and one SME rated this item as optional.

Behavioral Skills Training

A total of five experts participated in the ratings of this domain. A total of 14 items were identified by domain experts, 10 of which were considered essential (Table 1). Examples of essential items include: orients patient to rationale/purpose of the skill; instructs patient in skill to be learned; instructions are easy to follow and understand; provides example(s) of skill; and links skill/concept to patient's needs and goals (relevance).

Psychoeducation about Suicide

Four experts participated in the initial task while three completed all three tasks to reach a final consensus. A total of 12 items comprise the final list of domain tasks, 10 of which were considered essential (Table 1). Examples of essential items include: provides information to patient about what to expect in the ED when suicidal; teaches behavioral skills to effectively manage distress while in the ED; provides strategies and skills to manage imminent suicide crises; teaches strategies for coping effectively post-discharge; and reviews risk factors and warning signs for patient and family.

Experience with People with Lived Experience

Four experts completed the first two rating tasks, and three completed the last task. Unlike the other domains that emerge from decades of research and peer-reviewed publications, this domain is a relatively new area for inclusion in working with people who are suicidal seeking help in EDs. After completing the first task, it became clear that any attempt to seek consensus would impose an arbitrary, artificial process on this important emerging area. As a result, we did not attempt to amend areas of disagreement in order to reach consensus. Consensus was however achieved on 14 items and not achieved on three items (Table 1). Essential items include: provides personal examples of coping with shame; provides examples of ways to get through ED visit (including information on what to expect throughout the process); includes honest stories about the post-attempt experience, including the fact that the survivor/suicidal person may feel anger that they lived, or that they might not immediately feel like they want to live or that their life is worth living, and that this is okay; and conveys belief in the individual’s ability to survive this moment, and that "hope and recovery" sometimes feel hollow. The domain experts did not reach consensus on the following items: provides a sense of hope that recovery is possible; content conveyed using recovery centered/person-first language; and replaces recovery centered/person-first language with a statement about treating people with dignity (they are not the same and should not be combined). Importantly, disagreements stem from differences in preferred language, and not fundamental differences of opinion about the importance of treating suicidal individuals in the ED with respect, dignity, and attempting (when warranted) to instill hope.

DISCUSSION

In the absence of existing intervention-agnostic fidelity tools for the recommended suicide prevention care domains, we sought to create one. The specific impetus for this study was to develop a tool we could use to empirically evaluate fidelity of Jaspr Health, a suicide prevention digital technology for acutely suicidal people. Like the EDs it is designed to support, Jaspr Health delivers multiple expert-recommended EBPs within a single session. However the more general aim was to develop a tool that could serve the broader purpose of a fidelity tool with allegiance to science rather than to a specific treatment intervention. We were inspired by the SPRC ED Consensus Guidelines, derived through agreement by suicide scientific experts, and ultimately applied the Delphi Consensus method, where domain experts worked together (albeit blinded) to develop behaviorally-specific KPEs across EBPs required for fidelity of expert-recommended suicide care across six domains.

Through this effort, we succeeded in the expert-generated Suicide Care Fidelity Checklist consisting of 60 essential KPEs across six domains of recommended care for suicidal people, as well as 14 “nice to have” non-essential items. For five of the six domains, we were successfully able to achieve nearly full (98.6%) consensus after three iterations, well beyond the usual 85% agreement or better (Diamond et al., 2014) threshold typically used in the Delphi method. For the PLE domain however, while its domain experts fundamentally agreed on basic principles, they were unable to agree on precise language to describe the principles. Given the historic absence of PLE views within a pervasive medical model and mindset, we understood the importance of not forcing consensus—but to simply acknowledge the differences. That said, even within this group, consensus emerged on 14 of the 17 KPEs (82%).

The potential utility of this work far exceeds its original intended purpose. Concretely, it augments the SPRC guidelines and other suicide care recommendations by defining behaviorally-specific core elements for each recommended domain. Healthcare systems and their EDs wishing to adopt one, some, or all of its recommendations now have expert-derived clear treatment-agnostic guideposts for domain-specific key elements that must be included in order to claim fulfillment of a particular domain-specific intervention. The Suicide Care Fidelity Checklist can also aid healthcare systems in their efforts to train their ED care teams in a specific domain by focusing the training on building staffs’ competencies in performing the KPEs for relevant domain(s). It can also be used for quality assurance purposes—to ensure that providers continue to meet the fidelity mark after training is complete. When incorporated into the electronic health record, it can provide real-time guidance for providers and robust documentation that evidence-based care was delivered—a key strategy in protecting systems and their providers against malpractice lawsuits.

The successful application of the Delphi Consensus method in behavioral health research and development of an expert-derived, treatment agnostic fidelity tool may serve as models for other similar endeavors. A treatment-agnostic fidelity checklist has far greater real-world potential for use than intervention-specific fidelity measures developed for clinical research as most systems prioritize their workflow and system needs (e.g. blend multiple interventions into a single hybrid that can be easily implemented without significant demands on personnel time). A treatment-agnostic fidelity instrument like the Suicide Care Fidelity Checklist has the benefit of being flexibly used by systems of care, where only those items for a specific domain of interest are used. The Delphi Consensus process itself is one that facilitates engagement among those who develop and evaluate treatments that share largely common, essential elements. In our experience, the anonymity afforded by the Delphi Consensus method and focus on behaviorally-specific KPEs helped to facilitate this process. Finally, as more and more behavioral treatments are translated into digital formats, this approach provides a model of determining the extent to which the digital technology actually delivers an evidence-based intervention to fidelity. Methods for completing manipulation checks are a key consideration of internal validity.

A further significant strength of this project is its inclusion of KPEs for people with lived experience. While many advocacy groups, the National Institute of Mental Health, and Zero Suicide Institute all affirm the importance of survivors with lived experience to serve as beacons of hope and wisdom for others who are acutely suicidal, we know of no effort to date that seeks to behaviorally define the core elements required by PLEs when intervening in the ED with a person who is acutely suicidal. While inclusion of PLEs remains aspirational for many healthcare systems and EDs serving acutely suicidal patients, we hope the KPEs associated with this domain provide a roadmap for further innovation and programming in this area.

A number of limitations should be mentioned. First, it is possible that a different group of experts may have arrived at a different set of KPEs for their domains, or potentially different domains altogether. While we sought to objectively select SMEs, we acknowledge that many of the suicide experts engaged in this study—while meeting our inclusion criteria—are our colleagues with whom we have stood shoulder-to-shoulder in our efforts over decades to fight suicide. It is possible that this resulted in an unintentional bias in our selection. This potential bias may have also resulted in their own understandable desire to “help out” by arriving at consensus before the ideas had fully ripened. Second, while the relatively small number of experts per domain is not unusual when using the Delphi method, it is possible that a larger group may have generated more items or generated more varied critiques of the existing items that might have resulted in more review rounds and ultimately different conclusions. Finally, this is an initial effort to organize KPEs across multiple domains. Just as clinical interventions require replication before fully accepted as efficacious, this project should be replicated at a different time and by different investigators to determine what differences, if any, emerge through their inquiry.

ED care for people who are suicidal is wrought with challenges, yet it remains a promising location for effective suicide-specific interventions and care that can result in lives saved. The present study built the Suicide Care Fidelity Checklist, an expert-derived treatment-agnostic fidelity checklist to aid in determining whether suicide-specific interventions are delivered to fidelity to people in need. While our original intention was circumscribed to assessing Jaspr Health’s own fidelity, this simple tool holds the promise of improving the quality of evidence-based care provided to suicidal patients in the ED and ultimately saving lives.

ACKNOWLEDGEMENTS

The sponsor had no role in review or approval of this manuscript. The authors wish to express their gratitude to Adam Haim, PhD, Chief, Treatment and Preventive Intervention Research Branch at the National Institute of Mental Health (NIMH) and Program Officer, for his continued guidance of our work, as well as Jane Pearson, PhD, Chair, Suicide Research Consortium at the National Institute of Mental Health, for her early encouragement to include people with lived experience in our research efforts. Her gentle “nudge” in this direction transformed our approach to our work, including this effort. We are indebted to the many suicide experts who participated as domain experts for this study. They include: Michael Anestis, Joan Asarnow, Amy Barnhorst, Marian Betz, Edwin Boudreaux, Amy Brausch, Craig Bryan, Kathrine Comtois, Shannon Frattaroli, Julie Goldstein Grumet, Kelly Green, Peter Gutierrez, Christa Labouliere, DeQuincy Lezine, Jocelyn Meza, Andrada Neacsiu, Nick Salsman, Dese’Rae Stage, Jessica Stohlmann-Rainey, Jeffrey Sung, Kelechi Ubozoh, Lauren Weinstock, and Lucas Zullo. Finally, we would like to thank Kelly Koerner, PhD, co-founder of Jaspr Health, for her thoughtful edits to this final manuscript.

DISCLOSURE STATEMENT

Linda Dimeff co-owns the Evidence-Based Practice Institute, Inc., a for-profit company and recipient of the NIMH grant award that funds this effort. She receives royalties for her books from Guilford Press. She is co-owner of Portland DBT Institute, Inc., a for-profit outpatient specialty clinic that provides training and consultation services. David Jobes is the treatment developer of CAMS and receives royalties for his books from American Psychological Association Press and Guilford Press. He is a founder and co-owner of CAMS-care, LLC, a for-profit training and consultation organization.

Additional information

Funding

This research was supported by the National Institutes of Health (NIH)/National Institute of Mental Health [R44MH108222].

Notes on contributors

Linda A. Dimeff

Linda A. Dimeff, PhD, Evidence Based Practice Institute, Inc., Seattle, WA, USA.

David A. Jobes

David A. Jobes, PhD, Tia Tyndal, BA, and Irene Zhang, MA, The Catholic University of America, Washington, DC, USA.

Tia Tyndal

David A. Jobes, PhD, Tia Tyndal, BA, and Irene Zhang, MA, The Catholic University of America, Washington, DC, USA.

Irene Zhang

David A. Jobes, PhD, Tia Tyndal, BA, and Irene Zhang, MA, The Catholic University of America, Washington, DC, USA.

Susan Stefan

Susan Stefan, JD, Rutland, Rutland, MA, USA.

Nadia Kako

Nadia Kako, BS, Hannah Lawrence, BA, and Maria Ilac, BS, Evidence Based Practice Institute, Inc., Seattle, WA, USA.

Hannah Lawrence

Nadia Kako, BS, Hannah Lawrence, BA, and Maria Ilac, BS, Evidence Based Practice Institute, Inc., Seattle, WA, USA.

Maria Ilac

Nadia Kako, BS, Hannah Lawrence, BA, and Maria Ilac, BS, Evidence Based Practice Institute, Inc., Seattle, WA, USA.