does probability estimate for SVM have to be cross-validated? I want to obtain the probability estimates for an SVM to calculate its perplexity - what is the correct way to do this?
1) do k-fold cross-validation and average the probability estimates?
2) train on e.g. 80% of the data and get probability estimate from remaining 20% in a single instance?
3) train on all the data and get the probability estimate by testing on the same data?
4) some other way?
 A: SVMs don't output probabilities natively, but probability calibration methods can be used to convert the output to class probabilities. Various methods exist, including Platt scaling (particularly suitable for SVMs) and isotonic regression.
A probability calibration model must be fit to the data. For example, Platt scaling uses logistic regression to estimate class probabilities, given the output of the SVM (or other classifier) as input. Fitting a probability calibration model on the same training set used to fit the classifier would introduce bias. Instead, an independent set of data should be used. This can be the same validation set used to fit classifier hyperparameters (e.g. SVM kernel parameters). If probabilities are used to measure classifier performance, this should be done using a further, independent test set to avoid bias (as always).
Therefore, three independent sets of data are needed: 1) Training set (to fit the classifier), 2) Validation set (to select classifier hyperparameters and fit probability calibration model), 3) Test set (to estimate performance). If you have enough data you can use simple holdout (i.e. assign a fixed portion of the data to each of these roles). Otherwise, use nested cross validation.
References:
Niculescu-Mizil and Caruana (2005). Predicting Good Probabilities With Supervised Learning.
