Bio Conscious Newsroom
Bio Conscious Technologies released the results of Diabits accuracy testing using the FDA-approved Padova T1D Simulator
April 6th, 2020 7:07 AM PST
Bio Conscious Technologies Inc. published the results of the latest in-silico testing of their AI algorithm’s reliability using the FDA approved Padova simulator.
Bio Conscious Technologies (BCT) is the developer of Diabits, an app for people with diabetes who use continuous glucose monitors (CGMs) to manage their daily blood sugar fluctuations. Diabits is powered by a proprietary machine-learning algorithm designed by BCT to learn and calculate the patient’s future blood sugar values 60 minutes into the future using only CGM data.
In the past BCT, has shown that Diabits users can increase their time-in-range and maintain healthy blood sugar levels for longer compared to patients using only CGMs. Using Diabits, patients improve their time in range, HbA1c levels, and decrease blood sugar variability.
Predicting future blood sugar values is a significant challenge. A lot can happen between the time a prediction is made, and 60 minutes later when it can be validated. For example, a patient may have a meal, exercise, take insulin or attend a stressful meeting.
Using the Padova T1D simulator, BCT is able to generate different sets of food, insulin, and exercise data, impacting blood glucose outcomes for each in-silico patient. Using this data, BCT analyzed how the Diabits predictive model performed in each case.
3,110,400 total data points were evaluated during this study. Each blood sugar value was compared to the corresponding predicted point and assessed for accuracy.
Using the Padova T1D simulator for accuracy, BCT evaluated what blood sugar levels were predicted and what would happen 60 minutes later if no events occurred. Padova also allowed BCT to generate alternate outcomes and compare these results to the base-case prediction.
Accuracy was evaluated for the results in this study using the Parkes Error Grid, an iteration of the Clarke Error Grid that was developed during a 2000 study by Joan L Parkes and collaborators. This grid system allows researchers to quickly compare a large dataset of predicted or measured points to a reference value. In this case, the reference value is the simulated data point generated at 60 minutes after a prediction was made.
Diabits was previously tested in a blind pilot study on live subjects in 2017 at BC Children’s Hospital in Vancouver, the results of this study further demonstrate Diabits predictive capabilities.
The Parkes Error Grid specifies five risk levels, labeled A through E. The A region is classified as accurate, and the B region is clinically acceptable, meaning values estimated in this region will have little to no effect on clinical outcome and will not negatively impact a patient’s treatment decisions. As such, the results described in this analysis will be the proportion of predictions which fell in the A and B regions of Parkes, for any given assessment.
Using the FDA approved Padova simulator provided by the Epsilon Group, the study included 30 virtual patients and measured the predictive accuracy of the Diabits algorithm. Each virtual patient is entirely unique and represents a possible profile of a real Diabits user’s glucose metabolism.
The subjects were studied in four cohorts representing four distinct populations: adults with mixed hypoglycemia awareness, adults with impaired hypoglycemia awareness, pediatric patients with mixed hypoglycemia awareness and pediatric patients with impaired hypoglycemia awareness. Patients also had simulated behaviors related to diabetes management such as a randomized meal and insulin schedule based on the patient’s age, weight, and the amount of insulin necessary based on carbs consumed. For each patient, a total of 360 days of blood glucose behavior was simulated and insulin-on-board and carbs-on-board information were recorded.
Two different blood sugar prediction models were trained and tested for each patient. The first model, Production, is the algorithm that is used in Diabits today. The second model, ICE, is a more advanced variation of the Production model which puts more substantial weight on insulin, food, and exercise information. The model is named ICE because of I for insulin, C for carbs in food, and E for exercise. The results of the simulation showed that in general while the Production model is highly accurate, the ICE model will slightly outperform the Production model in most cases.
As a result of this study, the ICE model was implemented into Diabits. Today, if a user inputs food or insulin information, the ICE model will be automatically selected and used for that user’s predictions. If a user does not input this additional information, the Production model is automatically selected.
Table 1: Detailed results for the assessment of the production and ICE models
Because of the nature of simulated data, estimates made during in-silico testing will always be more idealized than estimates made on live subjects. To account for this, measurement error was also simulated. Without the simulated error, 100% of values were in regions A and B of the Parkes Error Grid for the 60 minute prediction of both the ICE and Production models. Taking into account the simulated error, 97.72% of values remained in the A and B regions for the 60 minute prediction on the Production model, and 99.52% for the ICE model.
“Achieving this kind of accuracy doesn’t happen overnight,” said Amir Hayeri, CEO of BCT. “We have been conducting validation testing internally and externally with other research organizations since 2015. We’ve worked hard to perfect our machine learning methodology and validate it. We’ve performed various tests, two of which used state-of-the-art blood sugar simulation to produce large data sets in-silico. This is the smaller one, we will be announcing a larger one at the ADA 2020.”
Table 2: Detailed results for the assessment of the production and ICE models with simulated measurement error in the data