Abstract
While machine learning approaches can enhance prediction ability, little is known about their ability to predict 30-day readmission after hospitalization for Chronic Obstructive Pulmonary Disease (COPD). We identified patients aged ≥40 years with unplanned hospitalization due to COPD in the Diagnosis Procedure Combination database, an administrative claims database in Japan, from 2011 through 2016 (index hospitalizations). COPD was defined by ICD-10-CM diagnostic codes, according to Centers for Medicare and Medicaid Services (CMS) readmission measures. The primary outcome was any readmission within 30 days after index hospitalization. In the training set (randomly-selected 70% of sample), patient characteristics and inpatient care data were used as predictors to derive a conventional logistic regression model and two machine learning models (lasso regression and deep neural network). In the test set (remaining 30% of sample), the prediction performances of the machine learning models were examined by comparison with the reference model based on CMS readmission measures. Among 44,929 index hospitalizations for COPD, 3413 (7%) were readmitted within 30 days after discharge. The reference model had the lowest discrimination ability (C-statistic: 0.57 [95% confidence interval (CI) 0.56–0.59]). The two machine learning models had moderate, significantly higher discrimination ability (C-statistic: lasso regression, 0.61 [95% CI 0.59–0.61], p = 0.004; deep neural network, 0.61 [95% CI 0.59–0.63], p = 0.007). Tube feeding duration, blood transfusion, thoracentesis use, and male sex were important predictors. In this study using nationwide administrative data in Japan, machine learning models improved the prediction of 30-day readmission after COPD hospitalization compared with a conventional model.
Introduction
COPD (COPD) is an important public health problem [Citation1, Citation2]. There are approximately 700,000 hospitalizations for COPD per year in the United States [Citation3], with one-fifth of patients requiring readmission within 30 days [Citation4]. As readmissions are costly and considered preventable, an increasing body of literature has emphasized the importance of 30-day readmission risk prediction tools [Citation5, Citation6]. To date, studies have identified predictors and developed models for predicting readmission in patients hospitalized for COPD [Citation7–9]. These models have incorporated predictors in terms of basic demographics such as age or sex and in-hospital procedures [Citation9–11].
Recently, machine learning approaches have attracted attention due to their superior ability over conventional approaches to predict patient outcomes [Citation12–16]. The advantages of machine learning approaches include their ability to process complex non-linear relationships between predictors and outcomes, and yield more stable predictions. Despite the advantages of machine learning approaches, only a few studies have examined their ability to predict 30-day readmission within 30 days of discharge from hospitalization due to COPD. While a prior study reported moderate prediction ability of machine learning models [Citation17], the lack of important variables (baseline severity of the respiratory condition, medication use during hospitalization) might limit their prediction ability. In this context, we utilized machine learning approaches with highly-detailed nationwide data to build clinically-useful prediction models with better performance. In addition, prediction models using routinely-collected, administrative data will be implementable to many hospitals that use electronic health records.
To address the knowledge gap in the literature, we aimed to examine the performance of machine learning approaches to predict 30-day readmission after hospitalization for COPD using data from the Japanese Diagnosis Procedure Combination (DPC) database, a routinely-collected nationwide administrative database containing details of inpatient care.
Methods
Study design and setting
We conducted a prognostic study using the DPC database from 2011 through 2016. The details of the DPC database were described previously [Citation18, Citation19]. Briefly, the database includes discharge abstracts and administrative claims data from >1000 acute-care hospitals and covers approximately 90% of all tertiary-care emergency hospitals in Japan. The database contains the following information for each patient: patient characteristics; primary and subcategorized secondary diagnoses, preexisting comorbidities at admission, and complications during hospitalization recorded using International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) codes and text data entered in Japanese; severity of disease conditions (e.g. Hugh-Jones score); consciousness level at admission and discharge; activities of daily living (ADL) score at admission and discharge; surgical and nonsurgical procedures and procedure dates; dates and doses of drugs; and discharge status. A previous validation study demonstrated good sensitivity and excellent specificity of the recorded diagnoses in the database [Citation19]. The present study was approved by the Institutional Review Board of the University of Tokyo with an informed consent waiver.
Study population
We identified all unplanned hospitalizations (index hospitalizations) for patients aged ≥40 years with a primary hospital discharge diagnosis of COPD using ICD-10-CM diagnostic codes, or a primary diagnosis of respiratory failure and secondary diagnosis of COPD (Supplemental Table 1), according to the Centers for Medicare and Medicaid Services (CMS) publicly-reported readmission measures [Citation10]. Patients who were transferred to another acute-care facility or died during the index hospitalization were excluded [Citation10]. Only the first readmission within 30 days of discharge was considered readmission, with additional readmissions within the 30-day period not counted as readmissions or index hospitalizations [Citation10]. Subsequent hospitalizations occurring beyond 30 days from discharge were considered index hospitalizations if they met the inclusion criteria [Citation10].
Predictors
The selection of predictors for the machine learning models was based on a priori knowledge. Although machine learning approaches enable the use of more than 1000 predictors, we focused on clinically intuitive and potentially relevant factors as the study aimed to determine the utility of machine learning approaches over conventional approach. Specifically, the predictors included patient age, sex, body mass index, smoking index [Citation20], ADL score at admission and discharge (Barthel index) [Citation21], severity of respiratory condition (Hugh-Jones score) [Citation22], number of comorbidities (Charlson comorbidity index) [Citation23], mode of arrival, number of admissions in the past 12 months prior to index hospitalization, duration from last admission to index hospitalization, in-hospital procedures, surgical interventions, and medications. The machine learning models included 420 variables.
Outcome measure
The primary outcome measure was readmission for any reason within 30 days of discharge from unplanned hospitalization for COPD (i.e. index hospitalization). We used this definition to maintain consistency with the CMS-defined 30-day readmission measures [Citation10].
Statistical analysis
In the training set (randomly-selected 70% of sample), we developed a reference model and two machine learning models to predict the probability of 30-day readmission. For the reference model, we fitted a logistic regression model, including age, comorbidities, and invasive mechanical ventilation use, according to the CMS readmission measures [Citation10]. Subsequently, we used these predictors to construct two machine learning prediction models: logistic regression with lasso regularization (lasso regression) and deep neural network [Citation24]. Lasso regularization extends a standard regression model by enabling the selection of important predictors (feature selection), and the resulting model is more interpretable and clinically useful, compared with the standard logistic regression model using many predictors. For the lasso regression, we used minimal lambda to penalize large beta coefficients. The minimal lambda was calculated by 10-fold cross-validation using the glmnet package. The deep neural network is a class of machine learning algorithms consisting of multiple layers of non-linear processing units to learn the values of the parameters that result in the best prediction of outcome. In the deep neural network, we constructed a five-layer feedforward model with an adaptive moment estimation optimiser [Citation25] using keras implemented in R [Citation24]. For the deep neural network, we developed the final model by randomly and manually tuning the hyperparameters, such as numbers of layers and hidden units, learning rate, learning rate decay, dropout rate, batch size, and epochs, using the keras package [Citation24]. To minimize potential overfitting in the two machine learning models, we used lasso regularization, and dropout, Ridge regularization, and batch normalization during the model development. In the logistic regression with lasso penalization (regularization), the penalty function shrank the large coefficients toward zero, thereby minimizing potential overfitting. In the deep neural network, to minimize potential overfitting, we used dropout that randomly removed portions of the units in the network, Ridge regularization that shrank the large coefficients, and batch normalization that normalized the means and variances of layer inputs. We used missing indicators for the treatment of missing values.
In the test set (remaining 30% of sample), we measured the prediction performance of each model by computing the 1) C-statistic (area under receiver-operating-characteristic [ROC] curve) and 2) prospective prediction results (sensitivity and specificity). In addition, to gain stable predictions, we measured the prediction performance of each model with 5-fold cross-validation.
To gain insights into the contribution of each predictor to the machine learning models, we also determined important predictors based on regularized beta-coefficients in the lasso regression model with minimal lambda. Values of p < 0.05 were considered statistically significant. All analyses were performed with R version 3.4.1 (R Foundation, Vienna, Austria).
Results
Characteristics of the cohort
From 2011 through 2016, we identified 44,929 index hospitalizations for COPD. For the index hospitalizations, 81% were men, the median age was 77 years, and the median body mass index was 21 kg/m2 (). Approximately one-third of the subjects were frequent smokers (smoking index ≥400) and had no disability. Half of the subjects had severe dyspnea (Hugh-Jones class 4 or 5). The majority of the subjects (>80%) had no prior hospitalizations within one year before the index hospitalization, no altered mental status at admission, and were discharged home.
Table 1. Characteristics of 44,929 hospitalizations for COPD recorded in a nationwide administrative database in Japan.
Prediction of 30-day readmission
Overall, 3413 (7%) subjects were readmitted within 30 days after discharge. The discrimination abilities of the different models, as represented by ROC curves, are shown in . The reference model had the lowest discrimination ability (C-statistic: 0.57 [95% confidence interval (CI) 0.56–0.59]) (). The two machine learning models had moderate, statistically higher discrimination abilities compared with the reference model (C-statistic: lasso regression, 0.61 [95% CI 0.59–0.61], p = 0.004; deep neural network, 0.61 [95% CI 0.59–0.63], p = 0.007). In addition, compared with the reference model, the machine learning models had lower sensitivity (e.g. 0.75 [95% CI 0.72–0.77] for reference vs. 0.67 [95% CI 0.65–0.70] for deep neural network), but higher specificity (e.g. 0.37 [95% CI 0.36–0.38] for reference vs. 0.51 [95% CI 0.50–0.52] for lasso regression) to predict 30-day readmission. The positive predictive values of all models were low (≤0.10 in all models), and the negative predictive values were high (0.95 in all models). Similar prediction abilities were obtained in the analysis using 5-fold cross-validation, with modest improvement of the predictive value in the lasso regression (C-statistic 0.64 [95% CI 0.60–0.68]) (Supplemental Table 2).
Figure 1. Prediction performances of the reference and machine learning models for readmission within 30 days after hospitalization for COPD. The ROC curves for predicting overall 30-day readmissions after hospitalization for COPD are shown. The reference model had the lowest discrimination ability (C-statistic: 0.57). The two machine learning models had moderate discriminative ability (C-statistic: 0.61 for both).
Table 2. Prediction abilities of the reference and machine learning models in subjects hospitalized for COPD.
Important variables for prediction
demonstrates the ten most important variables for predicting 30-day readmission in the lasso regression according to the regularized beta-coefficients with minimal lambda. The most important variable for prediction of 30-day readmission was the duration of tube feeding, followed by the use of blood transfusion, use of thoracentesis, and male sex.
Figure 2. Variable importance using lasso regression to predict 30-day readmission after hospitalization for COPD. The most important variable for prediction of 30-day readmission after hospitalization for COPD was duration of tube feeding, followed by use of blood transfusion, use of thoracentesis, and male sex.
Discussion
In this analysis of 44,929 COPD hospitalizations in a nationwide administrative database, machine learning models (lasso regression and deep neural network) improved the overall discrimination ability to predict 30-day readmission compared with a reference model based on CMS readmission measures. The important predictors were the duration of tube feeding, use of blood transfusion, use of thoracentesis, and male sex.
The utilization of machine learning approaches to predict patient prognosis is an appealing theme in current clinical research. For short-term outcomes, machine learning approaches achieved high discrimination ability in previous studies on in-hospital mortality in patients with cardiovascular diseases [Citation26], early recognition of sepsis [Citation27], and emergency department disposition in patients with COPD [Citation15]. While the superiority of machine learning approaches for the development of prediction models over conventional approaches shows promise, studies have failed to demonstrate their superior ability in the prediction of 30-day readmission for several conditions [Citation28, Citation29]. For example, in patients with heart failure (another CMS-targeted condition), use of machine learning approaches did not improve prediction ability for 30-day readmission compared with conventional prediction models for 56,477 patients in a Medicare population (C-statistic: 0.62 for tree-augmented naive Bayesian network vs. 0.62 for logistic regression) [Citation28]. However, in patients with COPD, a recent study on 67,771 COPD hospitalizations demonstrated a slightly improved prediction ability for machine learning approaches, with a best C-statistic of around 0.65 compared with a C-statistic of 0.60 using knowledge-driven features [Citation17]. Consistent with these previous findings, our study showed the superiority of machine learning models compared with a conventional CMS model.
There are several potential explanations for the improved prediction ability of machine learning approaches. First, our models incorporated patient baseline data, such as Charlson index and ADL, and inpatient care data, such as medication use during hospitalization. As shown in , these factors were important predictors. Second, machine learning approaches allow us to deal with non-linear interactions between predictors, which cannot be addressed by traditional modeling approaches such as logistic regression models. Third, machine learning approaches can include numerous predictors (>1000) in a model, while conventional approaches have limited numbers of potential predictors due to the zero-cell problem and multicollinearity.
Despite the superior prediction ability of machine learning approaches over conventional approaches with highly detailed data, their prediction ability of machine learning approaches remains imperfect. The potential explanations include heterogeneity in the 30-day readmission outcomes [Citation17]. Quality of care during the index hospitalization is the beginning part of the readmission process, and its contribution diminishes after discharge. Indeed, the effects of inpatient management on the readmission risk diminish rapidly after discharge, reaching a nadir at post-discharge day 7 [Citation30]. Alternatively, the lack of social factors may limit the prediction ability regardless of the methodological approach. For example, positive social support provided by family members was shown to be associated with improved quality of life [Citation31] and better pulmonary rehabilitation adherence [Citation32], resulting in reduced readmissions.
One important strength of the present study is the use of highly detailed, nationwide administrative data. Effective use of administrative databases to predict patient outcomes remains an important health care topic [Citation33]. The advantages of administrative data are well-documented and involve standardized data with tagged codes, such as ICD-10-CM codes [Citation33]. Furthermore, individual hospitals can routinely collect the required information. Although machine learning approaches can enable the treatment of more than 1000 predictors, including real-time data such as vital signs, there remain some concerns, including difficulty in collecting other real-time data, and additional costs for implementation of systems to collect information. With the use of highly detailed administrative data, machine learning approaches possess scalability. For example, machine learning approaches have the potential to improve their performance by integrating natural language processing [Citation34]. Therefore, further investigations on the effective use of administrative databases are warranted to develop risk prediction tools for the most high-risk patients and improve overall clinical outcomes.
Limitations
Our study has several potential limitations. First, hospital readmission risk and mortality risk are competing clinical risks. Patients who die after discharge cannot be readmitted, which means that the risk of readmission and risk of death after discharge are unclear. However, there could be some common causes for both risks, including condition exacerbation. Second, we were unable to identify patients who were readmitted to different hospitals after the index hospitalization in the DPC database. Third, as with other studies using administrative data, misclassification of hospitalizations is possible. Nevertheless, we identified hospitalizations for COPD according to the CMS definitions [Citation10], which have high specificity and positive predictive value (both ≥90%) [Citation35]. Moreover, a previous validation study showed good sensitivity and excellent specificity for diagnoses in the DPC database [Citation19]. Fourth, the DPC database does not contain several important clinical information such as social factors, thus precluding us from evaluating the contributions of social factors and post-discharge care. Lastly, as our study consisted of Japanese patients, there may be limited generalisability to other population. Nevertheless, the prediction ability was consistent with other findings in a different population [Citation17], and machine learning approaches themselves should not vary across health care settings.
Conclusions
In this analysis of 44,929 COPD hospitalizations in a nationwide administrative database in Japan, we found that the application of machine learning models improved the discriminative ability to predict 30-day readmission compared with a CMS-defined model. The important predictors were the duration of tube feeding, use of blood transfusion, use of thoracentesis, and male sex. Although the developed models were imperfect for clinical use, our findings facilitate further investigations to predict patient outcomes by combining the use of administrative data and machine learning approaches.
| Abbreviations | ||
| ADL | = | activities of daily living |
| COPD | = | chronic obstructive pulmonary disease |
| CMS | = | Centers for Medicare and Medicaid Services |
| DPC | = | Diagnosis Procedure Combination |
| ICD-10-CM | = | International Classification of Diseases, Tenth Revision, Clinical Modification |
| ROC | = | receiver-operating-characteristic |
Acknowledgement
We are grateful to Dr. Sarah Kyuragi Luthe for proofreading.
Disclosure statement
The authors have no conflicts of interest to declare.
References
- Centers for Disease Control and Prevention. Chronic obstructive pulmonary disease among adults–United States, 2011. MMWR Morb Mortal Wkly Rep. 2012;61(46):938–943.
- Kochanek KD, Xu J, Murphy SL, et al. Deaths: final data for 2009. Natl Vital Stat Rep. 2011;60(3):1–116.
- HCUPnet: Agency for Healthcare Research and Quality. US Department of Health & Human Services; [cited 2019 October 29]. Available from: http://hcupnet.ahrq.gov/HCUPnet.jsp.
- Goto T, Faridi MK, Gibo K, et al. Trends in 30-day readmission rates after COPD hospitalization, 2006-2012. Respir Med. 2017;130:92–97. doi:10.1016/j.rmed.2017.07.058.
- Leppin AL, Gionfriddo MR, Kessler M, et al. Preventing 30-day hospital readmissions: a systematic review and meta-analysis of randomized trials. JAMA Intern Med. 2014;174(7):1095–1107. doi:10.1001/jamainternmed.2014.1608.
- Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306(15):1688–1698. doi:10.1001/jama.2011.1515.
- Lau CS, Siracuse BL, Chamberlain RS. Readmission after COPD Exacerbation Scale: determining 30-day readmission risk for COPD patients. COPD. 2017;12:1891–1902. doi:10.2147/COPD.S136768.
- Shah T, Press VG, Huisingh-Scheetz M, et al. COPD readmissions: addressing COPD in the era of value-based health care. Chest. 2016;150(4):916–926. doi:10.1016/j.chest.2016.05.002.
- Sharif R, Parekh TM, Pierson KS, et al. Predictors of early readmission among patients 40 to 64 years of age hospitalized for chronic obstructive pulmonary disease. Annals ATS. 2014;11(5):685–694. doi:10.1513/AnnalsATS.201310-358OC.
- Yale New Haven Health Services Corporation/Center for Outcomes Research & Evaluation. 2014. Measures Updates and Specifications Report Hospital-Level 30-Day Risk-Standardized Readmission Measures 2014 [cited 2019 October 29]. Available from: http://altarum.org/sites/default/files/uploaded-publication-files/Rdmsn_Msr_Updts_HWR_0714_0.pdf.
- Amalakuhan B, Kiljanek L, Parvathaneni A, et al. A prediction model for COPD readmissions: catching up, catching our breath, and improving a national problem. J Community Hosp Intern Med Perspect. 2012;2(1).
- Berlyand Y, Raja AS, Dorner SC, et al. How artificial intelligence could transform emergency department operations. Am J Emerg Med. 2018;36(8):1515–1517. doi:10.1016/j.ajem.2018.01.017.
- Chong SL, Liu N, Barbier S, et al. Predictive modeling in pediatric traumatic brain injury using machine learning. BMC Med Res Methodol. 2015;15(1):22. doi:10.1186/s12874-015-0015-0.
- Wellner B, Grand J, Canzone E, et al. Predicting unplanned transfers to the intensive care unit: a machine learning approach leveraging diverse clinical elements. JMIR Med Inform. 2017;5(4):e45. doi:10.2196/medinform.8680.
- Goto T, Camargo CA, Jr., Faridi MK, et al. Machine learning approaches for predicting disposition of asthma and COPD exacerbations in the ED. Am J Emerg Med. 2018;36(9):1650–1654. doi:10.1016/j.ajem.2018.06.062.
- Goto T, Camargo CA, Faridi MK, et al. Machine learning–based prediction of clinical outcomes for children during emergency department triage. JAMA Netw Open. 2019;2(1):e186937. doi:10.1001/jamanetworkopen.2018.6937.
- Min X, Yu B, Wang F. Predictive modeling of the hospital readmission risk from patients’ claims data using machine learning: a case study on COPD. Sci Rep. 2019;9(1):2362. doi:10.1038/s41598-019-39071-y.
- Jo T, Michihata N, Yamana H, et al. Reduction in exacerbation of COPD in patients of advanced age using the Japanese Kampo medicine Dai-kenchu-to: a retrospective cohort study. COPD. 2018;14:129–139. doi:10.2147/COPD.S181916.
- Yamana H, Horiguchi H, Fushimi K, et al. Comparison of procedure-based and diagnosis-based identifications of severe sepsis and disseminated intravascular coagulation in administrative data. J Epidemiol. 2016;26(10):530–537. doi:10.2188/jea.JE20150286.
- Nagata N, Niikura R, Shimbo T, et al. Alcohol and smoking affect risk of uncomplicated colonic diverticulosis in Japan. PLoS One. 2013;8(12):e81137. doi:10.1371/journal.pone.0081137.
- Sato S, Demura S, Kobayashi H, et al. The relationship and its change with aging between ADL and daily life satisfaction characteristics in independent Japanese elderly living at home. J Physiol Anthropol. 2002;21(4):195–204. doi:10.2114/jpa.21.195.
- Hugh-Jones P, Lambert AA. simple standard exercise test and its use for measuring exertion dyspnoea. Br Med J. 1952;1(4749):65–71. doi:10.1136/bmj.1.4749.65.
- Quan H, Li B, Couris CM, et al. Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am J Epidemiol. 2011;173(6):676–682. doi:10.1093/aje/kwq433.
- R interface to Keras. 2017. [cited 2019 October 28]. Available from: https://github.com/rstudio/keras/.
- Kingma DP, Ba J. 2014. Adam: a method for stochastic optimization. arXiv preprint, arXiv:14126980.
- Kwon JM, Kim KH, Jeon KH, et al. Deep learning for predicting in-hospital mortality among heart disease patients based on echocardiography. Echocardiography. 2019;36(2):213–218. doi:10.1111/echo.14220.
- Churpek MM, Snyder A, Sokol S, et al. Investigating the impact of different suspicion of infection criteria on the accuracy of quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores. Crit Care Med. 2017;45(11):1805–1812. doi:10.1097/CCM.0000000000002648.
- Frizzell JD, Liang L, Schulte PJ, et al. Prediction of 30-day all-cause readmissions in patients hospitalized for heart failure: comparison of machine learning and other statistical approaches. JAMA Cardiol. 2017;2(2):204–209. doi:10.1001/jamacardio.2016.3956.
- Golas SB, Shibahara T, Agboola S, et al. A machine learning model to predict the risk of 30-day readmissions in patients with heart failure: a retrospective analysis of electronic medical records data. BMC Med Inform Decis Mak. 2018;18(1):44. doi:10.1186/s12911-018-0620-z.
- Chin DL, Bang H, Manickam RN, et al. Rethinking thirty-day hospital readmissions: shorter intervals might be better indicators of quality of care. Health Aff (Millwood). 2016;35(10):1867–1875. doi:10.1377/hlthaff.2016.0205.
- Bennett SJ, Perkins SM, Lane KA, et al. Social support and health-related quality of life in chronic heart failure patients. Qual Life Res. 2001;10(8):671–682.
- Fischer MJ, Scharloo M, Abbink JJ, et al. Drop-out and attendance in pulmonary rehabilitation: the role of clinical and psychosocial variables. Respir Med. 2009;103(10):1564–1571. doi:10.1016/j.rmed.2008.11.020.
- Gavrielov-Yusim N, Friger M. Use of administrative medical databases in population-based research. J Epidemiol Community Health. 2014;68(3):283–287. doi:10.1136/jech-2013-202744.
- Wu JT, Dernoncourt F, Gehrmann S, et al. Behind the scenes: a medical natural language processing project. Int J Med Inform. 2018;112:68–73. doi:10.1016/j.ijmedinf.2017.12.003.
- Quan H, Li B, Saunders LD, et al. Assessing validity of ICD-9-CM and ICD-10 administrative data in recording clinical conditions in a unique dually coded database. Health Serv Res. 2008;43(4):1424–1441. doi:10.1111/j.1475-6773.2007.00822.x.