Confidence Interval - Binary classification [duplicate]

Question

How do we calculate a confidence interval for a result in binary classifiers ?

CI for regression problems makes sense since we have a variable estimated output that I can calculate its estimated mean and then get the SE around it.

For classification problems, We only have metrics like Fpr/Tpr/AuC, precision/accuracy & class probabilities. Besides, class distribution is not usually approximated to a known distribution.

I am implementing a RandomForest classifier via Python for a biased binary classification problem.

Answer 1

You may apply bootstrap to calculate confidence intervals.

Under this method, you draw a random sample of input data, train the model and calculate the error (be it accuracy, precision, Matthews coefficient, etc.). You repeat this procedure N times, and from the output distribution error you may then easily extract the confidence intervals.

You may find complete information on bootstrapping here: https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf

Answer 2

I'm not sure for which property you need the confidence interval, but here we go:

• In case you need confidence intervals for the validation results (i.e. classifier has accuracy of p ± Δp), for proportions you can calculate binomial confidence intervals.

• In case you ask about confidence intervals for the predictions:

  • For an ensemble voting class labels (random forest), you could construct similar intervals for the proportion of trees that voted for the class which is predicted.
  • You could also do something like bolstered error estimation: perturb your input data and measure the distribution of predictions to which the perturbed input is mapped. You'd need to have a good idea of the noise structure on your input, though.