Quantifying the economic impact of a digital self-care behavioral health platform on Missouri Medicaid expenditures

Abstract

Aim: The primary objective of this study is to estimate the economic benefits relative to return on investment (ROI) of a state-wide initiative to integrate digital behavioral health (BH) self-care into the community BH system.

Methods: The observational study reviewed claims data of 799 people who registered with the digital tool (myStrength) and 715 matched control study participants. The control group was formed via coarsened exact match with blocking variables, including presence on the claims file, volume of health care utilization, participation in a medical health home, BH diagnosis and volume of psychotherapy claims. The primary study analysis of cost differences for the BH self-care tool group versus the control group were calculated by cost setting and the ROI of the BH self-care tool was estimated. Cost settings assessed include inpatient, emergency services, general and psychiatric outpatient, and outpatient psychotherapy.

Results: An incremental annual difference in difference reduction of $382 per user was observed over the 11 month study period in the self-care tool group. Sensitivity analysis indicated an ROI of between 142% and 695%.

Conclusions: Augmenting BH offerings to include digital BH self-care tools appears to generate positive ROI via reduced total cost of care.

Keywords:

• Depression
• digital behavioral health
• claims
• health care cost
• population health
• return on investment

JEL classification codes:

• I10
• I19

Introduction

While about 20% of Americans (44.7 million in 2016) live with a behavioral health (BH) condition, only about 43% of these will receive BH care in a given year¹. Stigma, low perceived need, lack of access, cost and dearth of qualified providers all contribute to this care gap^2–4.

At the same time, BH conditions, including depression, anxiety, insomnia and substance use disorders, are major drivers of cost in the US health care system, doubling or tripling total cost of care⁵. Increased utilization of emergency departments and inpatient days are significant contributors to increased costs⁶. At the same time, these are often avoidable medical consumption^7–9 which can be effectively reduced through the expansion of BH services⁹.

Digital health solutions have emerged as a potential solution to the dual challenge of increasing access to care while also reducing the BH cost burden. Over the course of the past decade, literature has accumulated which points to the clinical efficacy of digital solutions across a wide range of BH concerns including depression, anxiety, insomnia, stress management, and problematic substance use^10–18. Digital self-care solutions can be highly cost effective at potentially only a small fraction of the cost of live care provision. Introduction of digital self-care tools thus may create secondary cost savings via reduced need for and/or duration of in-person BH care¹⁹. While initial studies suggest that digital tools may be able to generate cost savings, early work in this area has also called for increased research focus on cost of care impacts of widespread introductions of digital health tools^20–23.

The current work focuses on a cost–efficacy analysis of the use of a digital BH tool across the state of Missouri (MO). The state of Missouri faces trends aligned with the above national statistics. Nearly 859,000 people in the state of Missouri experienced some form of mental illness in 2014²⁴. Fifty-five percent of those received no treatment. Likewise, approximately 419,000 people in Missouri struggled with a substance use disorder (SUD) in 2016, and 84% did not receive treatment²⁴. Annual cost burden in direct medical costs of untreated SUD and BH in MO was estimated at over $250 million in 2016²⁵.

In 2016 a unique partnership between the Missouri Coalition for Community BH care, the Missouri Department of Mental Health and the Missouri Foundation brought myStrength, a leading digital BH self-care tool, to 25 community mental health clinics (CMHCs) across the state. It was introduced both to each agency’s client base as well as to each agency’s broader community via the center’s outreach efforts.

As part of this effort, a formal evaluation of the program’s financial impact was conducted. While the tool’s clinical impact has been documented in other work^13,26, cost impact and return on investment (ROI) of a broad implementation was not previously available. The evaluator therefore was tasked with making an actuarially sound and also highly conservative assessment of the medical cost impact of this statewide initiative. More specifically, the evaluation assessed: (1) to what extent introducing the digital tool helped to drive cost reductions in avoidable medical costs beyond those being achieved by other available services; (2) to what extent was its introduction reducing direct cost of care for BH services; and (3) to what degree was there indication of a positive return on investment (ROI) from the statewide roll-out. This paper presents the approach and results of this third-party evaluation.

Methods

Program

The myStrength program is a web and mobile-based BH platform, delivering evidence-based resources to consumers. Interactive applications address challenges including depression, anxiety, insomnia, chronic pain management, stress and SUD. It integrates empirically proven psychotherapy models – such as cognitive behavioral therapy, motivational interviewing, acceptance and commitment therapy, positive psychology and mindfulness – along with mood tracking, sharing of community and personal inspirations, and a searchable library of BH and wellness/well-being resources. The user experience is highly personalized, based on individual interests. It can be used as a stand-alone tool or in conjunction with in-person, professional BH care. To foster engagement users are sent weekly BH-focused emails encouraging them to log in and use the tool.

Study participants

The digital BH self-care tool was made available via 25 CMHCs across the state of Missouri. The mission of the CMHCs is to provide BH services to Missouri residents suffering from severe and persistent mental illness, substance use and addiction disorders, poverty, homelessness and trauma. Approximately 60% of Missouri-based CMHCs serve rural populations. The continuum of services provided includes screening and assessment, case management, medication management, individual and group outpatient therapy, coordination for inpatient treatment, community psychiatric and substance abuse treatment rehabilitation, 24 hour emergency services, community support, consultation, education and prevention, and administrative coordination. In addition, all the CMHCs also provide outreach and education into the communities in their local catchment areas.

Agency staff were trained to make referrals to the BH self-care tool both as a supplement to clinical care as well as for more general community outreach. Agencies used the tool as an extension to in-session treatment, wait-list management, group session content, post-discharge support and community outreach. There were no limits placed on who could set up an account beyond the need to have an agency-wide access code which was made widely available on marketing materials.

As part of the registration process, new users were asked to voluntarily provide their first name, last name and date of birth for use in research linking them to their Medicaid claims. The first users to register with this process enrolled in November 2016. Users who were a part of this analysis joined the program no later than July 2017. Thirty-five percent of new registrants during this period chose to share this identifying data (n = 1869). In terms of age and gender, the volunteer sample is representative of all newly registered users affiliated with Missouri CMHCs during the study period: 67% are female, and approximately 60% are 31 years of age or older.

Data source and institutional review board oversight

A full-service, private institutional review board (IRB) with expertise in BH protocols approved the study design and research objectives (Solutions IRB #2017/12/13). Solutions IRB is registered with the Office for Human Research Protections. All data storage was compliant with Health Insurance Portability and Accountability Act guidelines.

Via a partnership with the Missouri BH coalition the third-party evaluator received a de-identified file with claims data for 869 users successfully matched to claims data. In addition, claims data on a random sample of 4029 adults enrolled in Missouri Medicaid who were not part of the identified set of users was also delivered to the third-party evaluator for use in this evaluation.

Evaluation approach

The evaluation presented here faced several significant challenges which drove the selection of methodological framework. First, the project’s funding cycle dictated that the evaluation needed to be completed within 15 months of the program’s launch, creating a very tight time-frame to assess economic impact. Due to the nature of the program, users were introduced to the BH self-care tool throughout the evaluation period. Second, the state of Missouri had launched a successful health homes initiative with demonstrated cost savings in both BH and avoidable medical expenses, which was being expanded into the same community BH clinics (CBHCs) at the same time as the digital BH program was introduced. Finally, given the real-world nature of the program structure, there was considerable variability in the extent to which other BH services were accessed alongside the BH self-care tool and to what degree people were exposed it prior to setting up an account. For example, in many clinics, the BH self-care tool is used in direct care sessions via a provider’s account. In those settings, patients were encouraged to set up their own accounts only after they have become familiar with the platform.

In light of these considerations, instead of conceptualizing the study as a classic experimental or quasi-experimental pre–post design, the evaluation framed the launch of the BH self-care tool as akin to the introduction of a coupon campaign into a retail sales environment. Rather than trying to determine impact from the day a coupon entered a person’s hand, the classical experimental perspective, the impact of a coupon campaign is best modeled by looking at differential sales (sales lift) between an exposed and naïve population across the entire period of the campaign. By analogy, this work compares difference in difference costs between people with evidence of exposure to the “campaign” (BH self-care tool registration) to those who remained BH self-care tool naïve (no account created) regardless of the date on which they registered. This shift afforded an approach which would be robust across the rolling window of registration dates. In addition, this approach ensured equivalent claims windows for both conditions.

In addition, the real-world considerations of this observational study clearly required a matching strategy that would be highly effective at balancing conditions and reducing potential bias. King and Nielsen²⁷ argue that a propensity score approach, the general go-to tool in observational studies, is highly problematic as it systematically reduces power, increases model dependence and also introduces bias. Instead, they advocate the use of coarsened exact matching to ensure tight and unbiased control over multiple factors.

Therefore, a coarsened exact match using micro-strata was implemented. Pre-analysis of the data indicated that there were five available covariates which were highly influential on total cost of care. Five coarsened blocking variables were used to form a set of mutually exclusive micro-strata which fully covered the experimental population. The five blocking variables are as follows:

1. Presence on the claims files and Medicaid eligibility in the given period: Presence on the claims file across each of these two pre-periods and two post-launch periods between people in the control and experimental groups to ensure similar eligibility.

2. Disease management and health care home participation: The micro-stratification scheme nested participation in neither, either or both of these cost control initiatives which were being introduced in MO concurrent to the BH self-care tool initiative.

3. Psychiatric diagnosis: The stratification schema created deciles by volume of claims for core BH (depression and anxiety), severe BH (bipolar and schizophrenia) and/or SUD over the entire study period. These were then reduced to four groups – high, medium, low and no claims – to form strata.

4. Claims setting: The stratification scheme created deciles of claims by setting over the entire study period. Settings included inpatient, ambulance/emergency, ambulatory procedures and outpatient health clinic/psychiatric facility. These were then reduced to four groups – high, medium, low and no claims – to form strata.

5. Psychotherapy: Volume deciles of claims over the entire study period with a Current Procedural Terminology (CPT) code indicating a psychotherapy encounter. These were then reduced to four groups – high, medium, low and no claims – to form strata.

Why do claims volume metrics figure so prominently in the stratification scheme? Returning to the analogous situation of a coupon campaign, controlling for the regularity with which a shopper visits a store is critical when looking for sales lift impact between a coupon exposed intervention group and a naïve control group. By analogy, controlling for the amount of interaction with the health care system was important for this study, whose mandate was to establish a highly conservative estimate of the impact of launching the “campaign”. Because the propensity to interact with the health care system varies greatly among people and this variability clearly impacts claims costs, several of the stratification levers are based on volume of claims over the entire study period.

The 869 users were placed into the micro-strata produced by the blocking schema as were the 4029 available controls. Within each stratum, random pruning reduced the controls to be equivalent to the n of the treatment group. In 25 of 182 strata there were not enough controls available to fully fill the strata. Likewise, 70 users were pruned from the data set as no controls matched to their micro-strata. Figure 1 presents a consort diagram for this data pre-processing. Table 1 presents core demographic data on the BH self-care tool versus control groups.

Figure 1. Consort flow for subject selection.

Table 1. Demographic break-down of self-care tool group versus control group.

	Self-care tool	Control
	Percentage/mean	Percentage/mean
Overall
Gender (female)	72%	58%**
Age	36	38*
Marital status
Married	7%	6%
Education
More than HS	13%	12%
Living arrangement
Independent	39%	37%
Depression ICD-10 diagnosis	56%	54%

*At p < .05 level.

**At p < .01 level.

Abbreviation. HS, High School.

Because there were significant differences in gender and age between control and BH self-care tool groups, the stratification schema was re-run with these added as blocking variables. Doing this resulted in 241 additional subjects being dropped due to non-matching strata. Review of historical myStrength data showed no evidence of differential impact of the program by age or gender. Therefore, a decision was made to retain the original blocking scheme in favor of the larger sample size.

Claims were broken out into four mutually exclusive cost centers: inpatient, emergency department (ED) and ambulance (emergency services), outpatient psychiatric facility/health clinic (outpatient) and psychotherapy. The psychotherapy category was first pooled to encompass all claims billed under CPT codes for outpatient therapy. Thus, the ED, outpatient and inpatient codes include no claims for psychotherapy – even if therapy was received in one of these settings. Difference in difference comparisons on all members with at least one claim in both the pre- and the post-period for any given cost area were used to evaluate deltas in total cost of care between conditions.

Results

Before assessing difference in difference costs between the intervention and control conditions, a check on the stability of the sampling frame was made by running four independent control group pools and looking at variability across each estimate. Total claims cost showed no significant or even borderline significant (p < .1) differences across all four control group pools by treatment setting and/or in any pairwise comparisons.

Results, presented in Table 2, indicate that core highly avoidable medical care consumption, reflected in the inpatient and emergency settings, evidenced the largest difference in difference cost reductions. While the p values for both of these differences do not meet traditional levels of statistical significance, given the skewed nature of claims data in general and this data in specific, it is still possible that there is a genuine trend here.

Table 2. Difference in difference health care spend by setting.

	BH Self-Care Tool Group (n = 799)				Control Group (n = 715)					p value
	n*	Pre-Period Costs	Post-Period Costs	Cost Difference	n*	Pre-Period Costs	Post-Period Costs	Cost Difference	Difference in Difference
Inpatient	62	$263	$116	−$148	86	$296	$264	−$32	−$115	.17
Emergency Services	274	$805	$482	−$323	255	$857	$694	−$163	−$160	.08
Psychiatric Facility/Health Clinic	123	$179	$97	−$82	103	$147	$136	−$11	−$71	<.01
Psychotherapy	71	$98	$43	−$55	70	$118	$98	−$20	−$35	.05

*n for each row includes only members who had at least one claim in both pre- and post-periods in this particular care setting.

Costs associated with visits to psychiatric facilities and/or to outpatient clinics evidence significantly more difference in difference cost reduction in the BH self-care tool group than in the control group. At the same time, the absolute magnitude of these differences is smaller than those for high cost avoidable claims.

While not included in Table 2, difference in difference spending was also calculated for outpatient procedures, a setting in which cost was not believed to be impacted by BH. Outpatient procedures included any planned procedure done in a hospital setting where there is no hospital admission associated with the procedure. The difference in difference cost on these procedures between groups was $7 on a total pre–post cost difference of about $600 in both conditions (p = .98).

In keeping with the mandate of this evaluation to establish conservative estimates of ROI, a series of ROI calculations were undertaken to determine both the likely range of ROI from the study period as well as some guidance on what future ROI might look like. ROI is calculated as the expected per user difference in difference savings times the number of enrolled users that year, divided by the annual cost of the statewide BH self-care tool contract. Note that because the study period did not align with the contract year cost period, we have elected to utilize the number of actual enrollments over the first year of the program contract, as opposed to that of the claims analysis window.

ROI estimate were made for: (1) all observed savings in keeping with the generally accepted actuarial standard of p < .25 indicating probable cost differences; (2) a more conservative standard at p < .1; and (3) the normative standard in experimental research of p < .05. In addition, three scenarios were computed to give a sense of where ROI lands under different levels of program engagement (e.g. with 3000 vs. 5000 vs. 8000 users) (Table 3). These were all computed with the most conservative estimate of cost savings (p < .05).

Table 3. Program ROI scenarios.

Scenario	ROI
Table 2 observed results Actual year 1 registrations (n = 4078)	695%
Table 2 results where p < .05 Actual year 1 registrations (n = 4078)	193%
Table 2 results where p < .1 Actual year 1 registrations (n = 4078)	484%
Table 2 results where p < .05 Reduced registrations (n = 3000)	142%
Table 2 results where p < .05 Increased registrations (n = 5000)	237%
Table 2 results where p < .05 Stretch target registrations (n = 8000)	379%

Discussion

Results of this study indicate that from an ROI perspective the statewide introduction of the BH self-care tool does appear to have generated more savings than expenses – netting the state an ROI of somewhere between 1.9 and 6.9 times savings in reduced medical spend over the cost to bring the program to the state. The most convincing evidence of cost savings is in reduced cost for BH care in psychotherapy and psychiatric facility and/or health clinic settings. Results indicate that people provided with a digital self-care tool have lower total cost for in-person, BH and routine outpatient care. Informal review of claims volume underlying the cost data hint that this finding may be being driven by reduction in volume of claims from the people who used the most psychotherapy in the pre-study period. In other words, providing “frequent flyers” with a digital BH self-support tool may be helping them to make more prudent use of BH and other outpatient resources – perhaps because they feel more empowered to manage on their own.

In terms of high cost medical expenses, while not quite achieving traditional levels of statistical significance as assessed with t-tests based on a normal distribution, the data does evidence a trend towards reduced expenses in avoidable high cost expenses, namely emergency department visits and inpatient stays. Review of claims volume by consumption levels suggests a slightly different driver of cost trends here. For emergency department usage, volume did not decrease more for users than controls between pre- and post-periods, but costs did. This would suggest that, when emergency services were used, they were used more efficiently. Perhaps self-help tools help constrain such visits to a more focused set of needs and thus drive down costs of these visits. With regards to inpatient stays, volume changed the most for people in the middle brackets in terms of claims volume. While most certainly speculative, this could indicate that the mechanism via which BH drives inpatient costs is via reduced inpatient days for people who have more subjectivity in their stay length and who feel more able to go home earlier when armed with a bit of additional support.

These findings must be viewed in light of several notable limitations in the work. First, demographic differences between the users of the BH self-care tool and the control sample might be biasing results. While we had no reason to believe that the program was working differently based on age and gender, we did do a re-run of the difference in difference model using the samples generated from a coarsened exact match which included gender and age. Results trended in the same ways as the results presented here, though the reduced sample size impacted statistical significance levels. While we would have liked to have had access to a deeper pool of potential control members which might have allowed us to match on demographics as well as to achieve full matching in all strata, unfortunately, the stipulations of the partnership which allowed us access to the claims file precluded such access.

Interesting as these hints from claims volume review may be, they also point to a major limitation of this work. Because of tight restrictions on the type and window of data available, we felt it was necessary to block on claims volume – thus making it impossible to also look at claims volume as a dependent variable. Future work would be welcomed that dove deeply into the dynamics of for whom and under what circumstances digital BH tools do reduce health care usage.

An additional source of bias might come from users in the BH self-care tool group self-selecting to try the program resulting in a population with elevated initiative and interest in self-improvement. At the same time, the program cost–efficacy frame of this work focuses more on understanding whether the cost of bringing the program was less than the cost of not bringing the program, net to the state. In that regard, even if the program only provided an alternative service to a more motivated population, it would still be reducing overall net cost of care.

This work may err on the overly conservative side with regard to cost savings estimation. First, our decision to block on claims volume across the entire study period, while on the one hand essential for ensuring that control and experimental conditions are well balanced on proclivity to utilize health care, has likely also reduced the observable program cost effect. In addition, this work looks only at direct health care impacts. Given the dramatic demonstrated cost of BH on workplace productivity²⁸ as well as disability and unemployment²⁹, it is possible that indirect cost savings are also being generated by the initiative evaluated here.

Past research gives voice to the concern that evaluation of digital health tools is predominantly undertaken on crowd-sourced or other convenience samples, as opposed to in real-world implementations and actual clinical practices^30,31. The current work grapples with the complexities of a real-world evaluation – complete with real-world constraints on timing and availability of data and a myriad of varied and uncontrolled use cases. Drawing on a methodological frame common in the complex world of consumer spending patterns, we are able to demonstrate that even under very conservative scenarios the statewide initiative to supplement traditional BH services with a broadly available digital BH tool appears to have generated a positive ROI.

In addition, our results begin to move the literature on digital self-care interventions for BH beyond “do they work” and into “do they deliver economic value in real-world settings”. A recent international work group of digital behavior change interventions concluded that two of the most pressing research needs in this area are: 1) work which takes projected uptake and reach into account in economic evaluations; and 2) research which isolates health care cost impact from other program impacts (like health outcomes, productivity, etc.)³². The current evaluation provides a range of ROI scenarios under different uptake conditions while also retaining focus on health care cost impact.

Future research would be served well by continued focus on understanding how actual consumers in actual real-world settings are impacted by the explosion of digital health tools entering the marketplace. Likewise, increased scrutiny of claims by digital health providers to deliver population health and/or cost benefits, would likely result in more economic impact studies such as this one. This would serve to both better inform the marketplace and enrich our understanding of the potential of digital BH tools.

Transparency

Declaration of funding

This research was funded by myStrength Inc. and the Missouri Health Foundation.

Declaration of financial/other relationships

SA has disclosed that he is the owner of Propensity4 and was paid as a consultant to provide methods and analysis for the study. A.H. has disclosed that she is a paid consultant for, and owns stock in, myStrength and serves as its Chief Clinical Officer.

Peer reviewers on this manuscript have received an honorarium from JME for their review work, but have no relevant financial or other relationships to disclose.

Acknowledgements

Krista Schladweiler coordinated work and data exchange between partners and helped with writing early drafts of the paper. Alissa Link and Britni Meyers helped collect citations and made finals edits to the paper. Ed Jones and Steven Schwartz both provided high level strategy consultation.