Why use logistic regression instead of SVM?

Question

I have a very basic question, but it's not really clear to me. When would one choose logistic regression over SVM? Maximum margin property seems more justifiable then whatever log. reg does. Is it because scores or interoperability?

Answer 1

No technique is superior in every case (cf., the No Free Lunch Theorem). If your problem is amenable to logistic regression, why not use a method which tends to provide well-calibrated scores? SVMs don't, and in most of my work, well-calibrated scores are very useful.

Here's an interesting quote from Murphy (Machine Learning: A Probabilistic Perspective) that doesn't say that you would not want to use an SVM, but it does give you a perspective that alternatives (not necessarily logistic regression) are something worth considering:

Note that SVMs are very unnatural from a probabilistic point of view. First, they encode sparsity in the loss function rather than the prior. Second, they encode kernels by using an algorithmic trick, rather than being an explicit part of the model. Finally, SVMs do not result in probabilistic outputs, which causes various difficulties, especially in the multi-class classification setting.

(Personally, I was initially taken by SVMs, but when applying them to real-world problems have found them to be fiddly and slow.)

Answer 2

SVM is a machine learning method, while logistic regression is a statistical one. There is a nice paper by Leo Breiman, where he compares what he calls "Data Modelling" (and what I call "statistics") to "Algorithmic Modelling" (i.e. machine learning).

Briefly, if you are interested in discovering why and how some variables affect other variables, and also want to know how credible the answers are, and if you can make some rather strong assumptions about your data (see my explanation on logistic function), then statistics is the way to go. On the other hand, if you just want to have someone tell you for your input data which output (response) to expect (perhaps because you have so many variables that you cannot even hope to make sense out of them), machine learning is the way to go.

For example, we know that men are, on average, taller than women, and that height is approximately normally distributed. If you were to guess the sex of an unknown person based only on his/her height, logistic regression is a reasonable thing to do.

If, on the other hand, you want to do face recognition, you'd probably want a machine learning algorithm, like SVM. We have no statistical theory about the distribution of pixels in face images and even if we had, you probably don't care which pixels influence classification in which way.