When do we require to calculate the confidence Interval?

Question

I am using various machine learning algorithms for last 7 years. To validate the model in classification algorithm we use precision, recall, f1 score.

For regression methods we use R^2,RMSE kind of metrics. But I never used or anyone asked me to show confidence interval.

Recently I published research paper where I do comparative study of 10 ML regression algorithms.

We have validated the model performance comparing Variance, RMSE and R^2 for each algo.

But reviewer asked me to use Confidence Interval. Does it really required for ML algorithm?

In sklearn do we have any method which calculates CI?

Answer 1

It depends and traditional practices differ between fields (e.g. statistics vs. computer science vs. mechanistic biological modeling vs. bioinformatics etc.). I think when comparing different algorithms this really makes sense.

Why? Well, let's say I'm trying out a few algorithms on one dataset. Let's assume the dataset is so well representative of tasks I'd want these algorithms to typically do that being better on this dataset (or perhaps rather data like that dataset of which we assume our dataset to be a representative random sample) really is a meaningful question. Let's also assume the algorithms reliably produce the same answer (i.e. there's no extra uncertainty due to random seeds when subsampling observations or features for an iteration, random initialization etc. - if those things also play a role, then taking the randomness in the resulting performance into account would also be needed and would make things a bit more complicated).

Now we ask ourselves:

• To what extent is a difference I see between two algorithms truly due to algorithm A doing generally better on this kind of data than algorithm B?
• Or could this have been just random chance even though in truth algorithms A is no better or even worse than B (e.g. we randomly picked two items from the population of items that favor algorithm A over B and only one item that favors algorithm B over A)?

Confidence intervals would be the difference in performance would answer that (of course, one can also do things like a null hypothesis test). One easy way to get them for metrics, for which there is no closed-form CI, would be to bootstrap the records in the test set (i.e. for a test set of size N randomly draw N items with replacement from these N and then calculate the difference in the metric of interest between the algorithms - in fact you can sample with replacement the calculated differences on each record).

In particularly large machine learning datasets this may not really be necessary when you see a large difference, e.g. if you have millions of records in the test set, then a difference in, say, accuracy of 75% vs. 70% will of course be substantially beyond what could happen by chance. For small differences or smaller test sets, it makes sense though to look at confidence intervals.