37
$\begingroup$

Can someone please give me some intuition as to when to choose either SVM or LR? I want to understand the intuition behind what is the difference between the optimization criteria of learning the hyperplane of the two, where the respective aims are as follows:

  • SVM: Try to maximize the margin between the closest support vectors
  • LR: Maximize the posterior class probability

Let's consider the linear feature space for both SVM and LR.

Some differences I know of already:

  1. SVM is deterministic (but we can use Platts model for probability score) while LR is probabilistic.
  2. For the kernel space, SVM is faster (stores just support vectors)
$\endgroup$
  • 2
    $\begingroup$ This statement is wrong: "LR: Maximize the posterior class probability". Logistic regression maximises the likelihood, not some posterior density. Bayesian logistic regression is a different story, but you need to be specific about it if that's what you're referring to. $\endgroup$ – Digio Feb 14 '18 at 8:25
30
$\begingroup$

Linear SVMs and logistic regression generally perform comparably in practice. Use SVM with a nonlinear kernel if you have reason to believe your data won't be linearly separable (or you need to be more robust to outliers than LR will normally tolerate). Otherwise, just try logistic regression first and see how you do with that simpler model. If logistic regression fails you, try an SVM with a non-linear kernel like a RBF.

EDIT:

Ok, let's talk about where the objective functions come from.

The logistic regression comes from generalized linear regression. A good discussion of the logistic regression objective function in this context can be found here: https://stats.stackexchange.com/a/29326/8451

The Support Vector Machines algorithm is much more geometrically motivated. Instead of assuming a probabilistic model, we're trying to find a particular optimal separating hyperplane, where we define "optimality" in the context of the support vectors. We don't have anything resembling the statistical model we use in logistic regression here, even though the linear case will give us similar results: really this just means that logistic regression does a pretty good job of producing "wide margin" classifiers, since that's all SVM is trying to do (specifically, SVM is trying to "maximize" the margin between the classes).

I'll try to come back to this later and get a bit deeper into the weeds, I'm just sort of in the middle of something :p

$\endgroup$
  • 1
    $\begingroup$ But that still doesn't answer my question on what's the intuitive difference in the objective functions of SVM v/s LR, which are as follows: (a) SVM: Try to maximize the margin between the closest support vectors (b) LR: Maximize the posterior class probability $\endgroup$ – user41799 Apr 27 '14 at 2:46
  • $\begingroup$ I mean, that's a completely different question. Are you asking about when to use the models, or what motivates the form of their objective functions? $\endgroup$ – David Marx Apr 27 '14 at 3:01
  • 1
    $\begingroup$ I am more interested in what motivates the form of their objective functions $\endgroup$ – user41799 Apr 27 '14 at 3:14
  • 4
    $\begingroup$ I'll try to come back to this later and get a bit deeper into the weeds, I'm just sort of in the middle of something Four years later... $\endgroup$ – user1717828 Apr 8 '18 at 17:19
23
$\begingroup$

Logistic Regression Vs SVM

Image signifies the difference between SVM and Logistic Regression and where to use which method

this picture comes from the coursera course : "machine learning" by Andrew NG. It can be found in week 7 at the end of: "Support vector machines - using an SVM"

$\endgroup$
  • $\begingroup$ By "features", do you mean the number of unique attributes or the total number of unique values belonging to those attributes? $\endgroup$ – Ahmedov Jun 19 '17 at 10:29
  • $\begingroup$ eg : in price price prediction of rubber, one feature is petrol price one is weather etc ..... $\endgroup$ – JSONParser Jun 19 '17 at 11:18
  • $\begingroup$ Actually, the image does not say anything about their differences... $\endgroup$ – Jan Kukacka Oct 16 '18 at 8:56
  • $\begingroup$ difference may be wrong word comparison can be better $\endgroup$ – JSONParser Oct 16 '18 at 9:02
1
$\begingroup$
  • LR gives calibrated probabilities that can be interpreted as confidence in a decision.
  • LR gives us an unconstrained, smooth objective.
  • LR can be (straightforwardly) used within Bayesian models.
  • SVMs don’t penalize examples for which the correct decision is made with sufficient confidence. This may be good for generalization.
  • SVMs have a nice dual form, giving sparse solutions when using the kernel trick (better scalability)

Check out Support Vector Machines vs Logistic Regression, University of Toronto CSC2515 by Kevin Swersky.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.