The use of reinforcement learning algorithms to meet the challenges of an artificial pancreas

Abstract

Blood glucose control, for example, in diabetes mellitus or severe illness, requires strict adherence to a protocol of food, insulin administration and exercise personalized to each patient. An artificial pancreas for automated treatment could boost quality of glucose control and patients’ independence. The components required for an artificial pancreas are: i) continuous glucose monitoring (CGM), ii) smart controllers and iii) insulin pumps delivering the optimal amount of insulin. In recent years, medical devices for CGM and insulin administration have undergone rapid progression and are now commercially available. Yet, clinically available devices still require regular patients’ or caregivers’ attention as they operate in open-loop control with frequent user intervention. Dosage-calculating algorithms are currently being studied in intensive care patients , for short overnight control to supplement conventional insulin delivery , and for short periods where patients rest and follow a prescribed food regime . Fully automated algorithms that can respond to the varying activity levels seen in outpatients, with unpredictable and unreported food intake, and which provide the necessary personalized control for individuals is currently beyond the state-of-the-art. Here, we review and discuss reinforcement learning algorithms, controlling insulin in a closed-loop to provide individual insulin dosing regimens that are reactive to the immediate needs of the patient.

Keywords::

• artificial pancreas
• continuous glucose monitoring
• diabetes mellitus
• machine learning
• personalized medicine
• reinforcement learning

Acknowledgements

MK Bothe wrote the manuscript. L Dickens, K Reichel, A Tellmann, B Ellger, M Westphal and AA Faisal reviewed and edited the manuscript.

Financial & competing interests disclosure

M Westphal, A Tellmann, K Reichel and M Bothe are employees of Fresenius Kabi Deutschland GmbH, Germany. Fresenius Kabi Deutschland GmbH distributes pumps for insulin delivery. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.

Key issues

• • Maintaining normoglycemia is crucial in patients with diabetes mellitus or severe illness and is usually achieved by administration of insulin.

• • An artificial pancreas system for closed-loop insulin delivery consists of a continuous glucose monitoring device, an algorithm calculating the correct amount of insulin and a pump delivering insulin.

• • Current challenges for calculation of the correct dose include technical issues most notably with regard to the time delay between changes in the glucose concentration and the maximum effect of insulin.

• • Use of newly identified substances for glucose regulation, such as glucagon or amylin, increases both the required flexibility and the complexity of the approach.

• • Have reliable and safe coverage for all feasible metabolic states and to respond appropriately in novel situations.

• • The individualized treatment regimes and the complex glucose regulating parameters elevate the need for smart algorithms that have reliable and safe coverage for all feasible metabolic states and respond appropriately in novel situations.

• • Algorithms used in the past were initially based on model predictive control or proportional integral derivative control.

• • Machine learning algorithms and especially reinforcement learning algorithms provide the advantages to learn the individual glucose pattern of a diabetic patient in spite of a time delay and to handle complex and external information to provide adaptive drug delivery after a learning procedure.

• • For machine learning approaches, care must be taken to acquire appropriate data for the learning phase whereby the data should be representative, sufficient and optimally noise-reduced.

• • To maximize the effectiveness of data-driven approaches, cross-validation and regularization techniques should be used and an extensive testing phase has to be performed.

• • Glucose regulation is more than just beta-cell dynamics, therefore we need smarter algorithms that learn to take into account the bigger picture.