2
$\begingroup$

Support Vector Machine (SVM) and logistic regression (LR) have been discussed widely in machine learning community, I know that both of them achieve pretty good performance. But, I am not sure how in general SVM compared to logistic regression? Why sometime SVM can perform better than LR? And sometime not? What factors determine these.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ This question needs to be more clear, as there are many ways to do SVM and LR. Which two ways precisely are you referring too, or is it in general? I doubt there are identifiable and provable factors that can be submitted to determine when to use one over the other for classification performance. The No Free Lunch Theorem says no one model is the best model for all datasets or problems (in a nutshell). Just try different methods and report the results is all we can and should do. $\endgroup$
    – Jane Wayne
    Nov 16, 2014 at 21:18

1 Answer 1

Reset to default
4
$\begingroup$

LR can only be compared to linear SVM - both work in the data space. I dont know exactly when logistic regression will perform better than linear SVM, but you should not that they approach the problem from a different angle - logistic regression will look at all the data in both classes while a linear SVM will only take into consideration the data in the border between both classes.

A RBF or polynomial SVM will (in my limited experience) perform better than a LR in most cases. There was a very recent paper on the JMLR http://jmlr.org/papers/v15/delgado14a.html that reached that same conclusion. They also probably tested LR and linear SVM on a large set of datasets, so you can compare if linear SVM has a similar performance to LR, for practical purposes.

This answer assumes that you are talking about a classification problem, despite the fact that you included a tag on regression.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ You can do kernel logistic regression (e.g.), though it's not that common. Presumably also your first word was meant to be LR. $\endgroup$
    – Danica
    Nov 12, 2014 at 23:53
  • 2
    $\begingroup$ Without reading the jmlr paper I suspect it only compared with naive linear LR. This has been obsolete since 1984, since regression splines started to be used. In most situations, LR done using modern tools is highly competitive, especially if penalization is used when the number of parameters is not supported by the number of events in the sample. $\endgroup$
    – Frank Harrell
    Nov 13, 2014 at 13:54
  • 1
    $\begingroup$ I only quickly glanced at the paper and although penalization was mentioned, I did not find any mention of non-linearity for LR. It is still a mystery to me why we are comparing non-linear methods with forcefully linear ones and come to the conclusion that non-linear ones perform better in many circumstances. Regarding LR and SVM they are closely related related: stats.stackexchange.com/questions/118215/… $\endgroup$
    – Thomas Speidel
    Nov 13, 2014 at 15:50
  • $\begingroup$ The paper tested a "standard" LR (implemented in the glm function in R), penalized LR (implemented in the glmnet package) and some other variations, all linear (pg 3152). penalized LR performs well, in the top 1/4, but below linear SVM (pg 3155). The other LRs perform in bottom 1/4. I have not yet read the methodological details - so I dont know yet what is the metric of ranking. $\endgroup$
    – Jacques Wainer
    Nov 14, 2014 at 0:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.