The value of health interventions: evaluating asthma case management using matching

Abstract

Despite large public investments in asthma interventions, there are few rigorous assessments of these programmes and little understanding of what comprises an effective intervention. There is a lack of appropriate data, little technical support is provided, and the programs themselves have little incentive to conduct these analyses. In this study, we apply optimal full matching using propensity scores to estimate the impact of an asthma intervention programme across a range of health outcomes. Our participation model is derived using the Deletion, Substitution and Addition (DSA) algorithm, a method used in epidemiology for model selection. We find that the asthma programme in question has no significant effect on participants that distinguishes them from matched nonparticipants, but it is not clear whether this is due to the effectiveness of the programme, heterogeneity of effects or barriers outside the programme's control. Our findings do show how current programmes could be modified to increase their effectiveness and better inform future research.

Keywords:

• propensity score
• treatment effects
• public health programs
• asthma
• children's health

JEL Classification::

• D61
• H43
• H51
• I12
• I18

Notes

¹ Guidelines for the effective treatment of asthma first were disseminated in 1991 by the National Asthma Eduction and Prevention Programme (NAEPP) and updated in 2007 (Urbano, 2008).

² See CDC web page for list of case studies: http://www.cdc.gov/asthma/interventions/default.htm.

³ In 2003, the CDC spending on asthma programmes ($36.9 million) was greater than for autism ($10.8 million) but less than from diabetes ($63 million) or lead prevention programmes ($42.0 million). The projects funded in 2003 included seven tracking programmes, 49 interventions and 39 community partnerships (For more information see http://www.cdc.gov/asthma/aag07.htm#more).

⁴ The Inner-City Asthma Intervention was located in 23 sites. The programmes all had asthma educators who were based in health organizations that treated low-income inner-city children. CAACP was also based in low-income urban areas, but each intervention was locally developed and varied in the design and components. CAACP was implemented in seven cities.

⁵ The number of days with asthma symptoms over a 2-week period averaged over six observations during the year.

⁶ The 95% confidence intervals are (34.59 to 38.72) versus (18.22 to 19.04) per 10 000 residents (Stockman et al., 2003).

⁷ Between fiscal years 2005 and 2006, the budget at the CDC was decreased by $500 million, and there was a $900 million cut to the funding for the Health Resources and Services Administration (CDC Coalition, 2006).

⁸ A cornerstone in evaluation methodology across disciplines is the seminal work by Rosenbaum and Rubin (1983, 1985).

⁹ The key trade-off in selection of the optimal size of the matched sets is that of reducing bias at the cost of losing efficiency (Ming and Rosenbaum, 2000).

¹⁰ There are several options for using the score to create comparison groups; see D’Agostino (1998) for a summary of three approaches. While pair matching has a long history in the literature, full matching was introduced relatively later (Rosenbaum, 1991).

¹¹ The advantage of using the Mahalanobis distance is that it takes into account the covariance among the variables in calculation of distances; however, binary elements that are rare can overly influence matching based on Mahalanobis distances (Rosenbaum and Rubin, 1983). Inclusion of the propensity score in the vector of covariates can reduce bias in these situations essentially by balancing some deviations between individual covariates with closer matches on the composite of those covariates as reflected in the propensity score (Rubin and Thomas, 2000). Given that we know outcomes such as hospitalizations are rare, we include the propensity score in the vector of pre-treatment variables.

¹² This penalty serves the same function as a caliper (Haviland et al., 2007) where the standard caliper is a quarter of a SD of the differences in estimated propensity scores (Rosenbaum and Rubin, 1985).

¹³ The asthma programme, data collection tools and all protocols for protection of personal data were approved by the Committee for the Protection of Human Subjects, UC, Berkeley.

¹⁴ Information on doses per canister was compiled using a Medline search on each prescription inhaler. Documentation available from authors by request.

¹⁵ To create medication intensity scale, we ran a stepwise logistic regression of number of prescriptions on an indicator for a visit to the emergency department for asthma. Cut-points in the number of prescriptions were defined by breaks in the odds ratio for ED visit (see Table A2 in the Appendix for regression results). This approach to defining medication intensity has been found to be predictive of future utilization.

¹⁶ A complete list of the procedure and billing codes used to create each variable is available from the authors upon request.

¹⁷ Having an emergency room visit or urgent outpatient visit increased the probability of participation most likely because it served to highlight the negative outcomes associated with unmanaged asthma. The positive relationship between having controller medication and an allergy diagnosis and participation could be interpreted in two ways. Either those children with more severe asthma or comorbidities are more likely to participate, or that individuals with higher propensity to seek out health services are those who are more likely to participate. The number of primary care providers is probably capturing many factors that could affect continuity of care.

¹⁸ Using the optmatch package in R.

¹⁹ The matched sets’ quartile case: control ratios (0.25, 0.5, 0.75, 1.0) are 1:13, 1:4, 1:1, 8:1; one matched set contained eight treated participants and one control. Hence, the total number of matched sets (247) is less than the total of matched treated participants (266).

²⁰ Note that conditional logistic regression for matched case-control groups is equivalent to fixed-effects logit for panel data; because of the health nature of the study, we will use the term most commonly used in biostatistics.

²¹ We decided not to consider pulmonary testing because without a detailed chart review there is no way to accurately categorize this type of encounter as being due to an exacerbation or use as a preventative measure.

²² For example, a child may need to have an inhaler at home and school. If prior to OKA the child did not have an inhaler at one of those places, or had an expired inhaler, then adding an inhaler after OKA is actually a risk-reducing behaviour.

²³ Sullivan et al. (2002) derive cost estimates using the mean Medicaid reimbursement rates for each service from the Medicaid Statistical Information System of the Health Care Financing Administration. The estimates from Piecoro et al. (2001) are similarly from Medicaid reimbursement rates but are limited to those in the state of Kentucky in 1996. All costs are inflated using the CPI inflation calculator available at http://cost.jsc.nasa.gov/inflateCPI.html. We compared these estimates from literature to a sample of reimbursements to Alameda Alliance to confirm that they are of appropriate order and magnitude for our setting.

²⁴ Prior to the intervention, the SD of expenses was $877, the minimum and twenty-fifth percentile were zero, the seventy-fifth percentile was $229 and the maximum was $4689. After the intervention the SD of expenses was $744, the minimum and twenty-fifth percentile were zero, the seventy-fifth percentile was $153 and the maximum was $7409.

²⁵ The four possible values are strongly and logically associated with changes in direct health care costs discussed in the previous section. Recall that the mean decrease in costs for all OKA participants was $133. Participants who worsened had a mean increase of $166, while those who improved had a mean decrease of $377. Those who remained stable over both periods had a mean increase of $2, and those who remained in the negative state had a decrease of $65. Because these numbers are not relative to the control group, they do not reflect the impact of the programme itself. However, they show the benefit to be gained by moving children from the negative to the stable state.

²⁶ This limitation is evident in previous studies as well (Evans et al., 1999; Harish et al., 2001; Morgan et al., 2004; Krieger et al., 2005).

²⁷ Estimates are primarily dependent on whether the indirect cost of premature death was included in the calculation of indirect cost.

²⁸ Schuck and Zeckhauser differentiate between those individuals who derive little benefit either relative to the resources consumed or relative to others in the programme (bad best) from those individuals who actually impose externalities on other participants (bad apples). We are primarily interested in avoiding bad bets because most of the asthma programmes are focused on small groups or individuals.

²⁹ In our case, we found that the most severe (based on utilization and reported symptoms previous 3 months) were not more likely to improve, although markers of exacerbations such as ER use were more associated with the probability of participation.