intuition behind classification model confidence intervals

Question

What do confidence intervals mean in classification problems?

I recently did a study with glmnet in R, and got this confusion matrix :

Confusion Matrix and Statistics

          Reference
Prediction  0  1
         0 29  6
         1  9 12

               Accuracy : 0.7321         
                 95% CI : (0.597, 0.8417)
    No Information Rate : 0.6786         
    P-Value [Acc > NIR] : 0.2400         

                  Kappa : 0.4118         

 Mcnemar's Test P-Value : 0.6056         

            Sensitivity : 0.7632         
            Specificity : 0.6667         
         Pos Pred Value : 0.8286         
         Neg Pred Value : 0.5714         
             Prevalence : 0.6786         
         Detection Rate : 0.5179         
   Detection Prevalence : 0.6250         
      Balanced Accuracy : 0.7149         

       'Positive' Class : 0 

I understand in linear regression, that if the model predicts 10, with a confidence interval of +-2, then with 95% confidence the true value should be 10 +- 2.

But with binary classification, the solution is 0 or 1, or a probability of a data point being 0/1.

What does the confidence interval mean in this case? Does it mean, a prediction by the model is between 59.7 and 84.1% accurate 95% of the time?

Answer 1

As you are looking for an intuition of what they mean (for a punctual answer about the way they are calculated when there exists no parametric way look at bootstrapping for CI and permutations for pvalues).. the CI referred to a certain measure means that, if you take a lot of iid samples drawn from the same population of the data, and you calculate that statistic for each sample, then your statistic would fall between the upper and lower limit of the CI for x% of the samples (where x% is the level chosen). This holds in general sense, regardless the statistic you are considering and whether this calculation admits a parametric solution. So here, in case it is referred to the accuracy (as it actually seems), it means that if you bootstrap and repeat that experiment with bootstrapped data for x times (I.e you draw the new sample and you estimate the model and evaluate the accuracy each time), then around x*95% times that value will fall into the interval (if you have any doubts you can look at bootstrapping).

So even more intuitively and informally, this is an estimated range where the true value of the accuracy in the population should fall with a certain confidence level.

For further help on this, see this reference