In-Silico Blood Glucose Forecasting Study with The Epsilon Group
To measure accuracy of the Diabits algorithm for predicting future blood glucose levels in diabetic patients, we performed validation testing in 4 studies, 2 of them using in-silico virtual patient data, and 2 with live patients.
The first validation study was an in-silico simulation designed and run by The Epsilon Group, the developer of the UVA-Padova T1DM Simulator. This simulator is accepted by the FDA as a substitute for preclinical trials of certain diabetes treatments. It has been used by 32 academic research groups and has been cited in over 63 peer-reviewed publications. The simulator contains 300 distinct virtual subjects: 100 adult, 100 adolescent and 100 pediatric. These subjects are described by 36 physiological parameters which are derived from experimental studies of healthy human physiology including glucose, insulin, and carbohydrate metabolism. Each virtual subject has a unique combination of values of the 36 different parameters. Therefore, each virtual subject represents a unique physiological profile related to glucose control.
Virtual subjects were studied in four patient cohorts representing four distinct populations: adults with mixed hypoglycemia awareness, adults with impaired hypoglycemia awareness, pediatrics with mixed hypoglycemia awareness and pediatrics with impaired hypoglycemia awareness. Cohorts with mixed hypoglycemia awareness were defined as 80% of subjects recognizing hypoglycemia symptoms at blood glucose levels of 55-70 mg/dL and 20% recognizing symptoms at 40-55 mg/dL. In the impaired hypoglycemia awareness cohorts, all the virtual patients recognized hypoglycemia symptoms at 40-55 mg/dL.
This test included the generation for 300 subjects: 100 adults, 100 adolescents and 100 children, using the T1DM FDA-accepted populations protocols. The simulated population was assigned a randomized meal and insulin schedule. The virtual subjects would consume food when a certain amount of time had passed since their last meal. A meal would become more likely as more time had passed since the last meal. The size of meals was randomized within a range, scaled based on the patient's weight. The number of carbs in each meal was capped at a value also determined by their weight. Insulin basal rates for patients were chosen as the optimal values for the patient, while the bolus dose for the patient was calculated for each meal. Additional corrective bolus insulin doses and meals were applied in cases of out-of-range blood glucose levels.
For each patient, the Insulin-on-board (IOB) and Carbs-on-Board (COB) information was recorded. Based on COB and IOB, two different estimation models were trained for each patient. The Production model, which is based on CGM data only, and the ICE model, which is a more advanced estimation model that takes into account CGM data, IOB, COB, and exercise information. The results of the simulation showed that, in general, the ICE model will be the most accurate. Therefore, if a Diabits user inputs this additional data, the more accurate ICE model will be automatically selected by the system when making estimates.
Measuring accuracy presents a significant challenge, given that we estimate future blood glucose assuming no change in the state of the user between measurement and estimation. The Parkes error grid analysis was designed to help better assess the clinical accuracy of reference values against a measured value of blood glucose. It is our primary accuracy metric used in all four validation testing studies. The Parkes error grid specifies five risk levels, labeled as the A through E regions. We classify A region as accurate, and B regions as clinically acceptable, meaning little to no effect on clinical outcome and will not negatively impact a user’s treatment decisions.
During this in-silico test 15,552,000 data points were evaluated to confirm accuracy, meaning this many CGM data points were compared to the Diabits estimation model’s values. The accuracy of estimates increased with age, and these accuracies hold regardless of the number of data points. Below are the Parks A and A+B accuracy, Root Mean Square Error (RMSE), and Mean Absolute Relative Difference (MARD) values for 15, 30, 45, and 60 minute intervals as based on the Production Model. Since the ICE Model is automatically selected for Diabits users who input additional data on top of CGM data, these accuracy values will be higher in practice.
15 minute accuracy - For the Parkes Error Grid Analysis region A, there was a 96.35% ± 0.02% accuracy and a region A+B accuracy of 99.90% ± 0.01%. The RMSE was 12.51 ± 0.02, with a
MARD of 0.0646 ± 0.0004.
30 minute accuracy - For the Parkes Error Grid Analysis region A, there was a 86.24% ± 0.05% accuracy and a region A+B accuracy of 99.20% ± 0.02%. The RMSE was 2.48 ± 0.03, with a
MARD of 0.1216 ± 0.0007.
45 minute accuracy - For the Parkes Error Grid Analysis region A, there was a 75.95% ± 0.11% accuracy and a region A+B accuracy of 97.83% ± 0.06%. The RMSE was 30.37 ± 0.04, with a
MARD of 0.1689 ± 0.001.
60 minute accuracy - For the Parkes Error Grid Analysis region A, there was a 67.23%±0.15% accuracy and a region A+B accuracy of 96.22% ± 0.09%. The RMSE was 36.46±0.04, with a
MARD of 0.207±0.0012.
The results provided by this in-silico simulation gave Diabits the validation we needed in order to start trials with live patients. You can find more information on our live trials here (link to BCCH study here).