How to estimate confidence level for SVM or Random Forest?

Question

I have two classes (say 1 and 0), and want to build a classifier. It is possible to use artificial neural networks (ANN) or any "real" classifying method such as SVM or Random Forest. In case of ANN, one can easily estimates confidence level of classification. For example, if we have binary task (with outputs as 0 or 1), and ANN results for some sample is 0.92, one can suppose that ANN "sure" in classification to 1 class. Alternatively, if ANN outputs 0.52, it is considered as unsteady classification to 1 flass.

But if we use Random Forest or SVM how it is possible to confidence level of classification?

Answer 1

For random forests you can look at the vote counts instead of just the winning class. Ie did 92% or 52% of the trees in the ensemble vote for class 1. How you do this will depend on the implementation.

Answer 2

Confidence intervals for SVMs are often obtained via bootstrapping or by taking a Bayesian approach.

For classification, bootstrapping may be a good approach to take. This slide deck illustrates some concepts of how to estimate confidence intervals to a test error of a certain classifier based on continuous functionals.

Another approach given here:

Estimating the Confidence Interval for Prediction Errors of Support Vector Machine Classifiers Bo Jiang, Xuegong Zhang, Tianxi Cai; 9(Mar):521--540, 2008.

is to use perturbation-resampling (similar to bootstrapping) to obtain the desired confidence intervals. This paper is a bit more interpretable than the slide deck. To summarize, first, it proposes that the classifier be estimated using cross-validation, and that as an output one will have $\hat{D}$ the estimated prediction error (Equation (5)). They go on to point out that it is desirable to know something about the distribution of this estimated prediction error (for instance if we know something about its distribution we can try to obtain confidence intervals for it). In order to do so for a general class of distributions (a problem confronted in the slide deck as well), they detail their resampling approach in Section 3.2. This is consolidated in an algorithm that they provide in this section.

You may also have luck attempting to learn something from the LS-SVM toolbox posted by this work group. They implement confidence intervals for Least Squares SVMs for regression using a Baysian confidence interval. This is not for classification, but it may be possible that you can adapt it to your problem.