I remember a paper from 1999 (13 years ago!) called Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods (1999) by John Platt that outlined a method for getting probabilistic outputs out of an SVM. From the abstract:

Instead, we train an SVM, then train the parameters of an additional sigmoid function to map the SVM outputs into probabilities.

Whilst this provides a solution of sorts, it is slightly unsatisfactory as it means performing two separate (and seemingly somewhat unrelated) optimisation problems.

Is there a more modern approach to this problem (without resorting to Gaussian Process classification for example)?


I don't know whether there are recent approaches to the problem, but I think I know why John Platt solved the problem in this kind of unsatisfactory way. Many machine learning algorithms can be written as regularizer plus loss function. For example, ridge regression would be $\lambda ||w||^2 + \sum_i (y_i - w^\top x_i)^2$. The minimizer of this is equivalent to the MAP of a Gaussian prior on $w$ and a Gaussian likelihood (just take an exp around the whole expression and put a minus in front). The SVM objective function looks similar. The squared norm regularizer stays the same, but the loss function is replaced by the hinge loss. The problem is now that the exp of the hinge loss does not correspond to a proper likelihood. Maybe there are approaches to change it, in order to make it into one, but then the question is whether it would still be called SVM.

Edit: One thing one could always do is bagging. One could train $n$ SVMs on different parts of the dataset and then simply count the fraction of positive/negative voting SVMs at testing stage. However, this is not specific to SVMs, of course.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.