A pragmatic methodology for the evaluation of digital care management in the context of multimorbidity

Figures & data

Table 1. Challenges in evaluating digital care management.

Choice of appropriate and robust outcome metrics	Cost as a primary outcome metric is widely-used in the healthcare industry. This presents two issues. First, trends in costs are not uniform across domains i.e. a reduction in inpatient utilization is generally desirable as is often an increase in outpatient utilization. Second, the distributions of costs and other utilization metrics are highly skewed, enabling potential undue influence of outliers. This in turn requires the choice of robust methodology and metrics.
Identification of a causal pathway	The causal pathway is likely to be dense and not known ex-ante. Patients have multiple comorbidities which have to be managed at the same time. As such, successful supportive interventions are heterogeneous with multiple clinical programs and service interventions delivered simultaneously as is typical in primary care.
Limitations of RCTs	RCTs require resources and timelines that are often not realistic for payors and providers evaluating digital health in anything more than a one off manner. RCTs often face challenges in generalizability.
Identification of a comparative control population	When implementing an observational study in lieu of an RCT, the identification of an appropriate control population is a major challenge as it requires the identification of a set of characteristics upon which to match control individuals with individuals who received the intervention. In part related to the lack of understanding of a causal pathway described in the second challenge above, the choice of matching characteristics presents a major obstacle.
Small sample sizes in early evaluations	Digital health platforms are typically initially implemented at a small-scale. Thus, there is significant class imbalance between the much larger control group and the much smaller intervention group.

Table 2. General framework for evaluating digital care management for multimorbid populations.

Step	Description
Define intervention group	Define the inclusion/exclusion criteria for individuals to be included in the analysis as a member of the intervention group. For the purposes of digital health, certain features such as “level of engagement” may be considered, as it may not be appropriate to consider someone as receiving the intervention if they did not engage to a certain level. Defining such an engagement level can be likened to taking a per protocol approach, while including any member who initiated the intervention, regardless of their engagement level, is similar to an intent to treat model^38.
Define control group	Define the inclusion/exclusion criteria for individuals to be considered for membership in the control group of the analysis prior to any matching. This should be all-inclusive and define the entire possible population for controls. For the purposes of digital health, a control population could be a group of individuals who were eligible for the intervention but did not respond to outreach.
Define study time frame	Choose the pre- and post-intervention time frames to be included in the assessment. This choice should take into consideration the duration over which an intervention is expected to cause an effect and should include enough pre-intervention time for a robust comparison and sufficient on/post intervention horizon to estimate impact.
Propensity match	This step consists of choosing variables to match controls to intervention members, as well as the matching methodology that is most appropriate for the given class imbalance. Our methodology includes variable selection with subject matter experts and matching with bootstrapping to randomly sample both populations when there is a large class imbalance. Simulations also help to account for effects of unobservable variables and create a framework that is robust to outliers.
Perform statistical outcome assessment	Choose the appropriate statistical tests to measure outcomes in the post-intervention period and compare these between the control and intervention populations to report a final outcome of the study.

Figure 1. Illustrative example of the randomized propensity matching technique.

Figure 2. Distribution of multi-morbidity of chronic conditions across the population. Of the Intervention population, 82% of individuals had two or more chronic conditions and 84% of the Control population had two or more chronic conditions. Note that this is the distribution prior to matching. Table 3 shows distributions over specific chronic conditions for three sample matching iterations. Mental health diagnosis includes depression and anxiety disorders.

Figure 3. Relationship between number of comorbid chronic conditions and the number of inpatient visits in the year prior to intervention (A), number of ER visits (B), number of outpatient non-preventive visits (C), and number of preventive outpatient visits (D) in the combined Intervention and Control populations, prior to matching.

Table 3. Population demographics and utilization patterns at baseline and after three random matching iterations.

	Baseline		Random match 1		Random match 2		Random match 3
	Intervention population	Control population	Intervention population	Control population	Intervention population	Control population	Intervention population	Control population
n	1,778	14,596	597	597	566	566	607	607
Average age	54.8	56.6	55.5	54.5	54.9	53.9	54.6	53.2
% with malignancy	22%	20%	20%	22%	22%	22%	21%	22%
% with diabetes	30%	32%	30%	29%	30%	30%	30%	32%
% with CKD	7%	14%	5%	6%	7%	6%	8%	8%
% with hypertension	55%	61%	54%	52%	56%	54%	56%	53%
% with liver disease	10%	9%	10%	9%	9%	9%	9%	9%
% with COPD	11%	17%	12%	10%	11%	11%	12%	11%
% with asthma	16%	14%	15%	17%	15%	16%	16%	15%
% with mental health diagnosis	48%	49%	46%	44%	46%	44%	44%	44%
Average PM number of inpatient visits in the year preceding intervention	0.27	0.38	0.25	0.30	0.23	0.27	0.28	0.28
Average PM number of monthly ER visits in the year preceding intervention	1.0	1.3	1.0	1.0	1.0	1.0	1.0	1.0
Average PM number of monthly preventive visits in the year preceding intervention	0.55	0.43	0.54	0.52	0.54	0.52	0.49	0.53
Average PM number of monthly outpatient office visits in the year preceding intervention	9.1	8.1	8.7	9.0	8.9	8.5	9.3	8.9

Figure 4. Left: distribution of match rate across all 100 randomized bootstrap iterations in the intervention population. Right: distribution of match rate across all 100 randomized iterations in the control population.

Figure 5. Distributions of outcomes in each group over 100 randomized iterations for (A) ER Visits per 1,000 members, (B) preventive medicine visits per 1,000 members, (C) non-preventive outpatient visits per 1,000 members, (D) inpatient visits per 1,000 members, over the 3 months following the initiation of the digital health platform intervention or the beginning of eligibility for the platform in the control population.

Table 4. Utilization outcomes.

	Intervention	Control	Statistical significance	Group delta (Intervention - control)
ER visits per 1,000 members	119	130	p < .0001	−11 Visits per 1,000 Members
Preventive medicine visits per 1,000 members	93	72	p < .0001	21 Visits per 1,000 Members
Non-preventive outpatient visits per 1,000 members	1,671	1,484	p < .0001	187 Visits per 1,000 Members
Inpatient visits per 1,000 members	54	63	p < .001	−9 Visits per 1,000 Members

Figure 6. Left: Inpatient utilization results of our propensity with bootstrapping methodology using three different matching criteria across three dates and sample sizes. Center: results using standard 1:1 propensity matching. Right: results using 1:5 propensity matching. (n.s. = not significant group difference at p < .05).

Ranganathan P, Pramesh C, Aggarwal R. Common pitfalls in statistical analysis: intention-to-treat versus per-protocol analysis. Perspect Clin Res. 2016;7(3):144.

 PubMedGoogle Scholar

Data availability statement

The data that support the findings of this study are available from a Wellframe health plan customer, but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of the health plan in question.