Background: There is much interest in the use of prognostic and diagnostic prediction models in all areas of
clinical medicine. The use of machine learning to improve prognostic and diagnostic accuracy in this area has been
increasing at the expense of classic statistical models. Previous studies have compared performance between ...
Background: There is much interest in the use of prognostic and diagnostic prediction models in all areas of
clinical medicine. The use of machine learning to improve prognostic and diagnostic accuracy in this area has been
increasing at the expense of classic statistical models. Previous studies have compared performance between these
two approaches but their findings are inconsistent and many have limitations. We aimed to compare the
discrimination and calibration of seven models built using logistic regression and optimised machine learning
algorithms in a clinical setting, where the number of potential predictors is often limited, and externally validate the
models.
Methods: We trained models using logistic regression and six commonly used machine learning algorithms to
predict if a patient diagnosed with diabetes has type 1 diabetes (versus type 2 diabetes). We used seven predictor
variables (age, BMI, GADA islet-autoantibodies, sex, total cholesterol, HDL cholesterol and triglyceride) using a UK
cohort of adult participants (aged 18–50 years) with clinically diagnosed diabetes recruited from primary and
secondary care (n = 960, 14% with type 1 diabetes). Discrimination performance (ROC AUC), calibration and
decision curve analysis of each approach was compared in a separate external validation dataset (n = 504, 21% with
type 1 diabetes).
Results: Average performance obtained in internal validation was similar in all models (ROC AUC ≥ 0.94). In
external validation, there were very modest reductions in discrimination with AUC ROC remaining ≥ 0.93 for all
methods. Logistic regression had the numerically highest value in external validation (ROC AUC 0.95). Logistic
regression had good performance in terms of calibration and decision curve analysis. Neural network and gradient
boosting machine had the best calibration performance. Both logistic regression and support vector machine had
good decision curve analysis for clinical useful threshold probabilities.
Conclusion: Logistic regression performed as well as optimised machine algorithms to classify patients with type 1
and type 2 diabetes. This study highlights the utility of comparing traditional regression modelling to machine
learning, particularly when using a small number of well understood, strong predictor variables.