Support Vector Machine (SVM) and logistic regression (LR) have been discussed widely in machine learning community, I know that both of them achieve pretty good performance. But, I am not sure how in general SVM compared to logistic regression? Why sometime SVM can perform better than LR? And sometime not? What factors determine these.

  • $\begingroup$ This question needs to be more clear, as there are many ways to do SVM and LR. Which two ways precisely are you referring too, or is it in general? I doubt there are identifiable and provable factors that can be submitted to determine when to use one over the other for classification performance. The No Free Lunch Theorem says no one model is the best model for all datasets or problems (in a nutshell). Just try different methods and report the results is all we can and should do. $\endgroup$ – Jane Wayne Nov 16 '14 at 21:18

LR can only be compared to linear SVM - both work in the data space. I dont know exactly when logistic regression will perform better than linear SVM, but you should not that they approach the problem from a different angle - logistic regression will look at all the data in both classes while a linear SVM will only take into consideration the data in the border between both classes.

A RBF or polynomial SVM will (in my limited experience) perform better than a LR in most cases. There was a very recent paper on the JMLR http://jmlr.org/papers/v15/delgado14a.html that reached that same conclusion. They also probably tested LR and linear SVM on a large set of datasets, so you can compare if linear SVM has a similar performance to LR, for practical purposes.

This answer assumes that you are talking about a classification problem, despite the fact that you included a tag on regression.

  • $\begingroup$ You can do kernel logistic regression (e.g.), though it's not that common. Presumably also your first word was meant to be LR. $\endgroup$ – Dougal Nov 12 '14 at 23:53
  • 2
    $\begingroup$ Without reading the jmlr paper I suspect it only compared with naive linear LR. This has been obsolete since 1984, since regression splines started to be used. In most situations, LR done using modern tools is highly competitive, especially if penalization is used when the number of parameters is not supported by the number of events in the sample. $\endgroup$ – Frank Harrell Nov 13 '14 at 13:54
  • 1
    $\begingroup$ I only quickly glanced at the paper and although penalization was mentioned, I did not find any mention of non-linearity for LR. It is still a mystery to me why we are comparing non-linear methods with forcefully linear ones and come to the conclusion that non-linear ones perform better in many circumstances. Regarding LR and SVM they are closely related related: stats.stackexchange.com/questions/118215/… $\endgroup$ – Thomas Speidel Nov 13 '14 at 15:50
  • $\begingroup$ The paper tested a "standard" LR (implemented in the glm function in R), penalized LR (implemented in the glmnet package) and some other variations, all linear (pg 3152). penalized LR performs well, in the top 1/4, but below linear SVM (pg 3155). The other LRs perform in bottom 1/4. I have not yet read the methodological details - so I dont know yet what is the metric of ranking. $\endgroup$ – Jacques Wainer Nov 14 '14 at 0:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.