How to estimate confidence level for SVM or Random Forest? I have two classes (say 1 and 0), and want to build a classifier. It is possible to use artificial neural networks (ANN) or any "real" classifying method such as SVM or Random Forest. 
In case of ANN, one can easily estimates confidence level of classification. For example, if we have binary task (with outputs as 0 or 1), and ANN results for some sample is 0.92, one can suppose that ANN "sure" in classification to 1 class. Alternatively, if ANN outputs 0.52, it is considered as unsteady classification to 1 flass. 
But if we use  Random Forest or SVM how it is possible to confidence level of classification?
 A: One general strategy for any classifier is to use some form of cross-validation to map the prediction output score to a probability, using the following scheme:


*

*Leave out part of the dataset.

*Train classifier.

*Make prediction on left-out data.

*For each prediction score, calculate what is the probability of making an error given that score or a better score. This can also be done by fitting a function that maps prediction score to probabilities. A natural choice would be mapping to a logistic function or some other form of sigmoidal function that can compress any range of scores to [0,1] as required from a probability.


For SVMs, this technique is called Platt scaling or Platt probabilities and is even included in some SVM implementations.
A: Well, I don't know about SVM. But with Random Forest you got at least two options.
Random Forest is composed from trees and in every tree at terminal node (leaf) you can calculate so called leaf statistics,that is - number of covered instances / number of misclassified instances. Or better number of correctly classified instances / number of covered instances.  That could be your confidence for one tree in Random Forest. For the final confidence for whole forest you can calculate average of this score from all trees that classified particular instance correctly. I think this way it is calculated in Weka and Rapidminer. The second option is easier, simply take ratio of number of trees that classified instance correctly / total number of trees as confidence. I think there are other ways, original (Breiman's) algorithm has so called "out of bag error" - OOB and I think little more error estimates of individual trees that you probably can use in confidence calculation.
Sorry, I would post this as a comment, since I did't say a word about SVM, but I don't have enough credits to post comments and I thought that it may be helpful to you.
A: In SVM, the Euclidean distance of the test datapoint from the separating hyperplane can be a measurement of confidence score of the classification. The higher is the distance, the higher the confidence level.
