16
$\begingroup$

My understanding of SVM is that it's very similar to a logistic regression (LR), i.e. a weighted sum of features is passed to the sigmoid function to get a probability of belonging to a class, but instead of the cross-entropy (logistic) loss function, training is performed using the hinge loss. The benefit of using the hinge loss is that one can do various numerical tricks to make kernelisation more efficient. A drawback, however, is that the resulting model has less information than a corresponding LR model could have. So, for example, without kernelisation (using a linear kernel) the SVM decision boundary would still be at the same location where LR would output a probability of 0.5, BUT one cannot tell how quickly the probability of belonging to a class decays away from the decision boundary.

My two questions are:

  1. Is my interpretation above correct?
  2. How does using the hinge loss make it invalid to interpret SVM results as probabilities?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
13
$\begingroup$

A SVM does not feed anything into a sigmoid function. It fits a separating hyperplane to the data that tries to put all data points from your training set that are of one class on one side, and all points of the other class on the other. Consequently, it assigns class based on which side your feature vector is on. More formally, if we denote the feature vector as $\mathbf{x}$ and the hyperplane coefficients as $\mathbf{\beta}$ and $\beta_0$ the intercept, then the class assignment is $y = sign(\beta \cdot \mathbf{x} + \beta_0)$. Solving an SVM amounts to finding $\beta, \beta_0$ which minimize the hinge loss with the greatest possible margin. Therefore, because an SVM only cares about which side of the hyperplane you are on, you cannot transform its class assignments into probabilities.

In the case of a linear SVM (no kernel), the decision boundary boundary will be similar to that of a logistic regression model, but may vary depending on the regularization strength you used to fit the SVM. Because the SVM and LR solve different optimization problems, you are not guaranteed to have identical solutions for the decision boundary.

There are many resources out there about the SVM which will help clarify things: here is one example, and another one.

Improve this answer
$\endgroup$
2
  • $\begingroup$ higgs boson This is very helpful, thank you! Just a few follow-up questions: (1) could you give any intuitive example when an SVM decision boundary would NOT be similar to a LR?, (2) is one of linear SVM and LR generally better than the other, or are there types of problems for which either is preferable? $\endgroup$
    – GingerBadger
    Commented Sep 12, 2017 at 11:33
  • 3
    $\begingroup$ Alex: in general, linear SVMs and LR generally perform comparably in practice. If you want a probabilistic output, then use LR. If you only care about class assignments, you can use either. If you wanted an example where their decision boundaries would be very different, you could imagine a linearly separable data set with a handful of points of the wrong class far away from the decision boundary. The outliers would pull the logistic regression boundary towards themselves, but if you had an SVM with a sufficiently large regularization term, it would effectively ignore the outliers. $\endgroup$
    – the higgs broson
    Commented Sep 12, 2017 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.