7
$\begingroup$

I have two classes (say 1 and 0), and want to build a classifier. It is possible to use artificial neural networks (ANN) or any "real" classifying method such as SVM or Random Forest. In case of ANN, one can easily estimates confidence level of classification. For example, if we have binary task (with outputs as 0 or 1), and ANN results for some sample is 0.92, one can suppose that ANN "sure" in classification to 1 class. Alternatively, if ANN outputs 0.52, it is considered as unsteady classification to 1 flass.

But if we use Random Forest or SVM how it is possible to confidence level of classification?

$\endgroup$
  • 2
    $\begingroup$ It sounds like you mean "confidence level" in a sense that might be different from common statistical terms (like, e.g., a 95% confidence interval for a proportion or mean estimated from a random sample). What distinguish 0.92 from, say, 0.75 -- if these are predicted probabilities to belong to a particular class, given a cut off of 0.5? $\endgroup$ – chl Apr 23 '14 at 12:02
  • 4
    $\begingroup$ For SVM you can use the distance to the separating hyperplane as a measure of confidence. The further, the more confident. Points within the margin are dodgy classifications. $\endgroup$ – Marc Claesen Apr 23 '14 at 17:50
6
$\begingroup$

For random forests you can look at the vote counts instead of just the winning class. Ie did 92% or 52% of the trees in the ensemble vote for class 1. How you do this will depend on the implementation.

$\endgroup$
  • $\begingroup$ This doesn't measure a confidence interval $\endgroup$ – user46925 Jul 30 '16 at 12:28
  • $\begingroup$ No it doesn't but it is analogous to the example the op asked about. $\endgroup$ – Ryan Bressler Jul 31 '16 at 22:49
9
$\begingroup$

Confidence intervals for SVMs are often obtained via bootstrapping or by taking a Bayesian approach.

For classification, bootstrapping may be a good approach to take. This slide deck illustrates some concepts of how to estimate confidence intervals to a test error of a certain classifi er based on continuous functionals.

Another approach given here:

Estimating the Confidence Interval for Prediction Errors of Support Vector Machine Classifiers Bo Jiang, Xuegong Zhang, Tianxi Cai; 9(Mar):521--540, 2008.

is to use perturbation-resampling (similar to bootstrapping) to obtain the desired confidence intervals. This paper is a bit more interpretable than the slide deck. To summarize, first, it proposes that the classifier be estimated using cross-validation, and that as an output one will have $\hat{D}$ the estimated prediction error (Equation (5)). They go on to point out that it is desirable to know something about the distribution of this estimated prediction error (for instance if we know something about its distribution we can try to obtain confidence intervals for it). In order to do so for a general class of distributions (a problem confronted in the slide deck as well), they detail their resampling approach in Section 3.2. This is consolidated in an algorithm that they provide in this section.

You may also have luck attempting to learn something from the LS-SVM toolbox posted by this work group. They implement confidence intervals for Least Squares SVMs for regression using a Baysian confidence interval. This is not for classification, but it may be possible that you can adapt it to your problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.