Calculate confidence intervals on accuracy metrics

Question

I've trained a logistic regression model on sklearn and then used it on some sample data which outputted accuracy, sensitivity and specificity values. Is there a way I can use the number of samples tested to estimate some form of confidence intervals for these metrics? i.e. how confident am I that the accuracy is the calculate 87%, and can I calculate a range like 84%-90%? Many thanks

Answer 1

Let me start by saying that accuracy is not an amazing metric. Not only can it be gamed very easily by guessing the most prominent class, but it also is not a proper scoring rule. Meaning that choosing the model with the largest accuracy does not mean choosing the model which best models the data. Additionally, sensitivity/specificity are poor measures of performance when we intend to use the model prospectively since they get the order of conditioning wrong (sens/spec condition on the outcome. We can't condition on the outcome prospectively, that's the whole point of making a model). With that aside, there may be ways to none the less obtain a confidence interval for accuracy.

Accuracy is just a binomial outcome (number correct over number predicted), so you could apply any number of binomial confidence intervals. Ideally, you would use the test set to do this, but there are other ways to doing this which benefit from larger samples.

In particular, you could bootstrap the optimism corrected bootstrap as shown here. That example uses the {rms} library, but I've created a blog post showing how to do the optimism corrected bootstrap with python and sklearn here.