1
$\begingroup$

This question already has an answer here:

How do we calculate a confidence interval for a result in binary classifiers ?

CI for regression problems makes sense since we have a variable estimated output that I can calculate its estimated mean and then get the SE around it.

For classification problems, We only have metrics like Fpr/Tpr/AuC, precision/accuracy & class probabilities. Besides, class distribution is not usually approximated to a known distribution.

I am implementing a RandomForest classifier via Python for a biased binary classification problem.

$\endgroup$

marked as duplicate by Sycorax, mdewey, mkt, kjetil b halvorsen, Peter Flom Sep 15 '18 at 12:02

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1
$\begingroup$

I'm not sure for which property you need the confidence interval, but here we go:

  • In case you need confidence intervals for the validation results (i.e. classifier has accuracy of p ± Δp), for proportions you can calculate binomial confidence intervals.

  • In case you ask about confidence intervals for the predictions:

    • For an ensemble voting class labels (random forest), you could construct similar intervals for the proportion of trees that voted for the class which is predicted.
    • You could also do something like bolstered error estimation: perturb your input data and measure the distribution of predictions to which the perturbed input is mapped. You'd need to have a good idea of the noise structure on your input, though.
$\endgroup$
0
$\begingroup$

You may apply bootstrap to calculate confidence intervals.

Under this method, you draw a random sample of input data, train the model and calculate the error (be it accuracy, precision, Matthews coefficient, etc.). You repeat this procedure N times, and from the output distribution error you may then easily extract the confidence intervals.

You may find complete information on bootstrapping here: https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading24.pdf

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.