Estimating Dynamic Treatment Regimes in Mobile Health Using V-Learning

References

• Ali, A. A., Hossain, S. M., Hovsepian, K., Rahman, M. M., Plarre, K., and Kumar, S. (2012), “mPuff: Automated Detection of Cigarette Smoking Puffs From Respiration Measurements,” in Proceedings of the 11th International Conference on Information Processing in Sensor Networks, ACM, pp. 269–280.
   Google Scholar
• Bergenstal, R. M., Garg, S., Weinzimer, S. A., Buckingham, B. A., Bode, B. W., Tamborlane, W. V., and Kaufman, F. R. (2016), “Safety of a Hybrid Closed-Loop Insulin Delivery System in Patients With Type 1 Diabetes,” Journal of the American Medical Association, 316, 1407–1408. DOI: 10.1001/jama.2016.11708.
   PubMed Web of Science ®Google Scholar
• Bexelius, C., Löf, M., Sandin, S., Lagerros, Y. T., Forsum, E., and Litton, J.-E. (2010), “Measures of Physical Activity Using Cell Phones: Validation Using Criterion Methods,” Journal of Medical Internet Research, 12, e2. DOI: 10.2196/jmir.1298.
   PubMed Web of Science ®Google Scholar
• Chakraborty, B., and Moodie, E. E. (2013), Statistical Methods for Dynamic Treatment Regimes, New York: Springer.
   Google Scholar
• Cobry, E., McFann, K., Messer, L., Gage, V., VanderWel, B., Horton, L., and Chase, H. P. (2010), “Timing of Meal Insulin Boluses to Achieve Optimal Postprandial Glycemic Control in Patients With Type 1 Diabetes,” Diabetes Technology & Therapeutics, 12, 173–177. DOI: 10.1089/dia.2009.0112.
   PubMed Web of Science ®Google Scholar
• Dai, Y.-H. (2002), “Convergence Properties of the BFGS Algorithm,” SIAM Journal on Optimization, 13, 693–701. DOI: 10.1137/S1052623401383455.
   Web of Science ®Google Scholar
• Doya, K. (2000), “Reinforcement Learning in Continuous Time and Space,” Neural Computation, 12, 219–245. DOI: 10.1162/089976600300015961.
   PubMed Web of Science ®Google Scholar
• Ertefaie, A. (2014), “Constructing Dynamic Treatment Regimes in Infinite-Horizon Settings,” arXiv no. 1406.0764.
   Google Scholar
• Free, C., Phillips, G., Watson, L., Galli, L., Felix, L., Edwards, P., Patel, V., and Haines, A. (2013), “The Effectiveness of Mobile-Health Technologies to Improve Health Care Service Delivery Processes: A Systematic Review and Meta-Analysis,” PLoS Med, 10, e1001363. DOI: 10.1371/journal.pmed.1001363.
   PubMed Web of Science ®Google Scholar
• Haller, M. J., Stalvey, M. S., and Silverstein, J. H. (2004), “Predictors of Control of Diabetes: Monitoring May Be the Key,” The Journal of Pediatrics, 144, 660–661. DOI: 10.1016/j.jpeds.2003.12.042.
   PubMed Web of Science ®Google Scholar
• Hastie, T., Tibshirani, R., and Friedman, J. H. (2009), The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.), New York: Springer.
   Google Scholar
• Hernan, M. A., and Robins, J. M. (2010), Causal Inference, Boca Raton, FL: CRC Press.
   Google Scholar
• Klasnja, P., Hekler, E. B., Shiffman, S., Boruvka, A., Almirall, D., Tewari, A., and Murphy, S. A. (2015), “Microrandomized Trials: An Experimental Design for Developing Just-in-Time Adaptive Interventions,” Health Psychology, 34, 1220. DOI: 10.1037/hea0000305.
   Web of Science ®Google Scholar
• Kober, J., and Peters, J. (2012), “Reinforcement Learning in Robotics: A Survey,” in Reinforcement Learning, eds. M. Wiering and M. van Otterlo, Berlin, Heidelberg: Springer, pp. 579–610.
   Google Scholar
• Kosorok, M. R. (2008), Introduction to Empirical Processes and Semiparametric Inference, New York: Springer.
   Google Scholar
• Kosorok, M. R., and Moodie, E. E. (2015), Adaptive Treatment Strategies in Practice: Planning Trials and Analyzing Data for Personalized Medicine (Vol. 21), Philadelphia, PA: SIAM.
   Google Scholar
• Kowalski, A. (2015), “Pathway to Artificial Pancreas Systems Revisited: Moving Downstream,” Diabetes Care, 38, 1036–1043. DOI: 10.2337/dc15-0364.
   PubMed Web of Science ®Google Scholar
• Kumar, S., Nilsen, W. J., Abernethy, A., Atienza, A., Patrick, K., Pavel, M., Riley, W. T., Shar, A., Spring, B., Spruijt-Metz, D., and Hedeker, D. (2013), “Mobile Health Technology Evaluation: The mHealth Evidence Workshop,” American Journal of Preventive Medicine, 45, 228–236. DOI: 10.1016/j.amepre.2013.03.017.
   PubMed Web of Science ®Google Scholar
• Laber, E. B., Linn, K. A., and Stefanski, L. A. (2014), “Interactive Model Building for Q-Learning,” Biometrika, 101, 831. DOI: 10.1093/biomet/asu043.
   PubMed Web of Science ®Google Scholar
• Lai, T. L., and Robbins, H. (1985), “Asymptotically Efficient Adaptive Allocation Rules,” Advances in Applied Mathematics, 6, 4–22. DOI: 10.1016/0196-8858(85)90002-8.
   Web of Science ®Google Scholar
• Levine, B.-S., Anderson, B. J., Butler, D. A., Antisdel, J. E., Brackett, J., and Laffel, L. M. (2001), “Predictors of Glycemic Control and Short-Term Adverse Outcomes in Youth With Type 1 Diabetes,” The Journal of Pediatrics, 139, 197–203. DOI: 10.1067/mpd.2001.116283.
   PubMed Web of Science ®Google Scholar
• Liao, P., Klasnja, P., Tewari, A., and Murphy, S. A. (2016), “Sample Size Calculations for Micro-randomized Trials in mHealth,” Statistics in Medicine, 35, 1944–1971. DOI: 10.1002/sim.6847.
   PubMed Web of Science ®Google Scholar
• Linn, K. A., Laber, E. B., and Stefanski, L. A. (2017), “Interactive Q-Learning for Quantiles,” Journal of the American Statistical Association, 112, 638–649. DOI: 10.1080/01621459.2016.1155993.
   PubMed Web of Science ®Google Scholar
• Long, N., Gianola, D., Rosa, G. J., Weigel, K. A., Kranis, A., and Gonzalez-Recio, O. (2010), “Radial Basis Function Regression Methods for Predicting Quantitative Traits Using SNP Markers,” Genetics Research, 92, 209–225. DOI: 10.1017/S0016672310000157.
   PubMed Web of Science ®Google Scholar
• Ly, T. T., Breton, M. D., Keith-Hynes, P., De Salvo, D., Clinton, P., Benassi, K., Mize, B., Chernavvsky, D., Place, J., Wilson, D. M., and Kovatchev, B. P. (2014), “Overnight Glucose Control With an Automated, Unified Safety System in Children and Adolescents With Type 1 Diabetes at Diabetes Camp,” Diabetes Care, 37, 2310–2316. DOI: 10.2337/dc14-0147.
   PubMed Web of Science ®Google Scholar
• Ly, T. T., Roy, A., Grosman, B., Shin, J., Campbell, A., Monirabbasi, S., Liang, B., von Eyben, R., Shanmugham, S., Clinton, P., and Buckingham, B. A. (2015), “Day and Night Closed-Loop Control Using the Integrated Medtronic Hybrid Closed-Loop System in Type 1 Diabetes at Diabetes Damp,” Diabetes Care, 38, 1205–1211. DOI: 10.2337/dc14-3073.
   PubMed Web of Science ®Google Scholar
• Maahs, D. M., Mayer-Davis, E., Bishop, F. K., Wang, L., Mangan, M., and McMurray, R. G. (2012), “Outpatient Assessment of Determinants of Glucose Excursions in Adolescents With Type 1 Diabetes: Proof of Concept,” Diabetes Technology & Therapeutics, 14, 658–664. DOI: 10.1089/dia.2012.0053.
   PubMed Web of Science ®Google Scholar
• Maei, H. R., Szepesvári, C., Bhatnagar, S., and Sutton, R. S. (2010), “Toward Off-Policy Learning Control With Function Approximation,” in Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 719–726.
   Google Scholar
• Moodie, E. E., Richardson, T. S., and Stephens, D. A. (2007), “Demystifying Optimal Dynamic Treatment Regimes,” Biometrics, 63, 447–455. DOI: 10.1111/j.1541-0420.2006.00686.x.
   PubMed Web of Science ®Google Scholar
• Murphy, S. A. (2003), “Optimal Dynamic Treatment Regimes,” Journal of the Royal Statistical Society, Series B, 65, 331–355. DOI: 10.1111/1467-9868.00389.
   Google Scholar
• Murphy, S. A. (2005), “A Generalization Error for Q-Learning,” Journal of Machine Learning Research, 6, 1073–1097.
   PubMed Web of Science ®Google Scholar
• Murphy, S., Deng, Y., Laber, E., Maei, H., Sutton, R., and Witkiewitz, K. (2016), “A Batch, Off-Policy, Actor-Critic Algorithm for Optimizing the Average Reward,” arXiv no. 1607.05047.
   Google Scholar
• Murphy, S. A., van der Laan, M. J., and Robins, J. M. (2001), “Marginal Mean Models for Dynamic Regimes,” Journal of the American Statistical Association, 96, 1410–1423. DOI: 10.1198/016214501753382327.
   PubMed Web of Science ®Google Scholar
• Puterman, M. L. (2014), Markov Decision Processes: Discrete Stochastic Dynamic Programming, New York: Wiley.
   Google Scholar
• Quinn, C. C., Shardell, M. D., Terrin, M. L., Barr, E. A., Ballew, S. H., and Gruber-Baldini, A. L. (2011), “Cluster-Randomized Trial of a Mobile Phone Personalized Behavioral Intervention for Blood Glucose Control,” Diabetes Care, 34, 1934–1942. DOI: 10.2337/dc11-0366.
   PubMed Web of Science ®Google Scholar
• R Core Team (2016), R: A Language and Environment for Statistical Computing, Vienna, Austria: R Foundation for Statistical Computing.
   Google Scholar
• Robins, J. M. (2004), “Optimal Structural Nested Models for Optimal Sequential Decisions,” in Proceedings of the Second Seattle Symposium on Biostatistics, New York: Springer, pp. 189–326.
   Google Scholar
• Rubin, D. (1978), “Bayesian Inference for Causal Effects: The Role of Randomization,” The Annals of Statistics, 6, 34–58. DOI: 10.1214/aos/1176344064.
   Web of Science ®Google Scholar
• Schulte, P. J., Tsiatis, A. A., Laber, E. B., and Davidian, M. (2014), “Q- and A-Learning Methods for Estimating Optimal Dynamic Treatment Regimes,” Statistical Science, 29, 640–661. DOI: 10.1214/13-STS450.
   PubMed Web of Science ®Google Scholar
• Steinhubl, S. R., Muse, E. D., and Topol, E. J. (2013), “Can Mobile Health Technologies Transform Health Care?,” Journal of the American Medical Association, 310, 2395–2396. DOI: 10.1001/jama.2013.281078.
   PubMed Web of Science ®Google Scholar
• Sutton, R., and Barto, A. (1998), Reinforcement Learning: An Introduction, Cambridge, MA: The MIT Press.
   Google Scholar
• Tang, Y., and Kosorok, M. R. (2012), “Developing Adaptive Personalized Therapy for Cystic Fibrosis Using Reinforcement Learning,” The University of North Carolina at Chapel Hill Department of Biostatistics Technical Report Series, Working Paper 30.
   Google Scholar
• Weinzimer, S. A., Steil, G. M., Swan, K. L., Dziura, J., Kurtz, N., and Tamborlane, W. V. (2008), “Fully Automated Closed-Loop Insulin Delivery Versus Semiautomated Hybrid Control in Pediatric Patients With Type 1 Diabetes Using an Artificial Pancreas,” Diabetes Care, 31, 934–939. DOI: 10.2337/dc07-1967.
   PubMed Web of Science ®Google Scholar
• Wolever, T., and Mullan, Y. (2011), “Sugars and Fat Have Different Effects on Postprandial Glucose Responses in Normal and Type 1 Diabetic Subjects,” Nutrition, Metabolism and Cardiovascular Diseases, 21, 719–725. DOI: 10.1016/j.numecd.2010.12.005.
   PubMed Web of Science ®Google Scholar
• Zhang, B., Tsiatis, A. A., Laber, E. B., and Davidian, M. (2012), “A Robust Method for Estimating Optimal Treatment Regimes,” Biometrics, 68, 1010–1018. DOI: 10.1111/j.1541-0420.2012.01763.x.
   PubMed Web of Science ®Google Scholar
• Zhang, B., Tsiatis, A. A., Laber, E. B., and Davidian, M. (2013), “Robust Estimation of Optimal Dynamic Treatment Regimes for Sequential Treatment Decisions,” Biometrika, 100, 681–694.
   Web of Science ®Google Scholar
• Zhao, Y., Kosorok, M. R., and Zeng, D. (2009), “Reinforcement Learning Design for Cancer Clinical Trials,” Statistics in Medicine, 28, 3294–3315. DOI: 10.1002/sim.3720.
   PubMed Web of Science ®Google Scholar
• Ziegler, R., Heidtmann, B., Hilgard, D., Hofer, S., Rosenbauer, J., and Holl, R. (2011), “Frequency of SMBG Correlates With HbA1c and Acute Complications in Children and Adolescents With Type 1 Diabetes,” Pediatric Diabetes, 12, 11–17. DOI: 10.1111/j.1399-5448.2010.00650.x.
   PubMed Web of Science ®Google Scholar