1
$\begingroup$

I have a very basic question, but it's not really clear to me. When would one choose logistic regression over SVM? Maximum margin property seems more justifiable then whatever log. reg does. Is it because scores or interoperability?

$\endgroup$
1
  • 5
    $\begingroup$ "Whatever log(istic) reg(ression) does.", do you know, or are you just being flippant? $\endgroup$ Jan 23, 2016 at 22:26

2 Answers 2

11
$\begingroup$

No technique is superior in every case (cf., the No Free Lunch Theorem). If your problem is amenable to logistic regression, why not use a method which tends to provide well-calibrated scores? SVMs don't, and in most of my work, well-calibrated scores are very useful.

Here's an interesting quote from Murphy (Machine Learning: A Probabilistic Perspective) that doesn't say that you would not want to use an SVM, but it does give you a perspective that alternatives (not necessarily logistic regression) are something worth considering:

Note that SVMs are very unnatural from a probabilistic point of view. First, they encode sparsity in the loss function rather than the prior. Second, they encode kernels by using an algorithmic trick, rather than being an explicit part of the model. Finally, SVMs do not result in probabilistic outputs, which causes various difficulties, especially in the multi-class classification setting.

(Personally, I was initially taken by SVMs, but when applying them to real-world problems have found them to be fiddly and slow.)

$\endgroup$
2
  • 1
    $\begingroup$ Fidgety? Please elaborate. $\endgroup$
    – rolando2
    Jan 24, 2016 at 2:55
  • 1
    $\begingroup$ @rolando2: Remember, this is a "Personally, I..." statement... SVMs have a variety of settings that I find over-sensitive and don't feel like I can adjust in a principled way. (And I'm not a fan of grid searches over parameter combinations.) They're not straightforward to tune, in my opinion, so perhaps "fiddly" is a better word. $\endgroup$
    – Wayne
    Jan 24, 2016 at 21:36
1
$\begingroup$

SVM is a machine learning method, while logistic regression is a statistical one. There is a nice paper by Leo Breiman, where he compares what he calls "Data Modelling" (and what I call "statistics") to "Algorithmic Modelling" (i.e. machine learning).

Briefly, if you are interested in discovering why and how some variables affect other variables, and also want to know how credible the answers are, and if you can make some rather strong assumptions about your data (see my explanation on logistic function), then statistics is the way to go. On the other hand, if you just want to have someone tell you for your input data which output (response) to expect (perhaps because you have so many variables that you cannot even hope to make sense out of them), machine learning is the way to go.

For example, we know that men are, on average, taller than women, and that height is approximately normally distributed. If you were to guess the sex of an unknown person based only on his/her height, logistic regression is a reasonable thing to do.

If, on the other hand, you want to do face recognition, you'd probably want a machine learning algorithm, like SVM. We have no statistical theory about the distribution of pixels in face images and even if we had, you probably don't care which pixels influence classification in which way.

$\endgroup$
2
  • 1
    $\begingroup$ While I agree with you on the different applications of algorithms as per statistics vs machine learning, I disagree that logistic regression is not a machine learning algorithm. People think of logistic regression as statistics since it’s been around since before machine learning was in vogue. If logistic regression were to just be discovered now no doubt we would refer to it as a machine learning algorithm. $\endgroup$
    – astel
    Oct 5, 2020 at 19:50
  • 1
    $\begingroup$ I agree that the exact boundary is somewhat subjective. Some people consider even linear regression to be a machine learning algorithm. Seen that way, any statistical algorithm can be thought of as a machine learning algorithm. But I don't see how this helps clarity. $\endgroup$
    – Igor F.
    Oct 6, 2020 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.