SVM - can I use the decision function for calculating AUC?

Question

An SVM returns a real-valued prediction for each of the input data samples, which corresponds to its distance from the separating hyperplane.

Platt's scaling is often used to output a "probability" value instead.

Is there something wrong with using the SVM decision function values to calculate area under the ROC curve, instead of bothering with fitting probability estimates?

I find that there tends to be a higher concentration of correctly classified examples, further from the hyperplane - so in practice, using the decision function appears to have value. But are there dangers to this of which I am not aware?

Answer 1

Using the decision values of an SVM is perfectly valid, in fact this is the standard approach.

Platt scaling to obtain probabilities is essentially nothing more than running your SVM decision values $f(\mathbf{z})$ through the logistic function $l(x)$ to squash them into the $[0, 1]$ range:

$$ \begin{align} f(\mathbf{z})&:\ \mathbb{R}^d \mapsto \mathbb{R} \\ l(x) &= \frac{1}{1 + \exp(-\beta x)}, \ \beta > 0 \\ l(f(\mathbf{z})) = (l \circ f)(\mathbf{z})&:\ \mathbb{R}^d \mapsto [0, 1] \end{align}$$

Note that you don't need probabilities at all to compute ROC curves, all you need is a ranking of the test set. As a direct result, any strictly monotonically increasing transformation (which preserves ranks) will have no effect on resulting ROC curves. You can find more details on ROC curves in one of my papers.

Answer 2

Yes - you should be able to use Area Under the Receiver Operating Curve (AUC) with SVM classification. Many classification algorithms return (or can return) probability values instead of discrete classifications, and the results are evaluated with AUC. Probabilities can then be converted to class values based on a threshold value (e.g., Prob > 0.5 = Presence; Prob <=0.5 = Absence), and this can be done based on AUC output (e.g., finding the point where Sensitivity=Specificity). There are lots of examples in the Species Distribution Modeling literature, among other fields.

If you're using R a couple of packages that might be helpful are Presence Absence and ROCR.

And yes - I it is common that as you go further from the decision boundaries, you'll have more accurate classification - that's because along that boundary it might difficult to differentiate classes (e.g., if you have a gradient from white to black, it will be tough to classify a gray color near the middle of the spectrum as one or the other). Given this, there are lots of ways to set the threshold value, and some are given in the aforementioned Presence Absence package. Also, there's a table I find useful in this book, which summarizes some threshold options (sorry - don't have the page or chapter handy). A search for "Threshold probabilities classification" and similar seems to turn up some informative results.

Hope that helps!