18
$\begingroup$

I know that logistic regression finds a hyperplane that separates the training samples. I also know that Support vector machines finds the hyperplane with the maximum margin.

My question: is the difference then between logistic regression (LR) and support vector machines (SVM) is that LR finds any hyperplane that separates the training samples while SVM finds the hyperplane with the maximum margin? Or am I wrong?

Note: recall that in LR when $\theta \cdot x = 0$ then the logistic function gives $0.5$. If we assume $0.5$ as a classification threshold, then $\theta \cdot x = 0$ is a hyperplane or a decision boundary.

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
12
$\begingroup$

You are right if you are talking about hard SVM and the two classes are linearly separable. LR finds any solution that separates the two classes. Hard SVM finds "the" solution among all possible ones that has the maximum margin.

In case of soft SVM and the classes not being linearly separable, you are still right with a slight modification. The error cannot become zero. LR finds a hyperplane that corresponds to the minimization of some error. Soft SVM tries to minimize the error (another error) and at the same time trades off that error with the margin via a regularization parameter.

One difference between the two: SVM is a hard classifier but LR is a probabilistic one. SVM is sparse. It chooses the support vectors (from the training samples) that has the most discriminatory power between the two classes. Since it does not keep other training points beyond that at the test time, we do not have any idea about about the distribution of any of the two classes.

I have explained how LR solution (using IRLS) breaks in case of linearly separability of the two classes and why it stops being a probabilistic classifier in such a case: https://stats.stackexchange.com/a/133292/66491

Improve this answer
$\endgroup$
4
  • 3
    $\begingroup$ What kind of least squares does the logistic regression optimize? LR uses cross-entropy as a loss. $\endgroup$
    – Artem Sobolev
    Commented Mar 14, 2015 at 17:17
  • 2
    $\begingroup$ just because logistic regression uses IRLS that doesn't mean it's least squares - the reweighting in IRLS is a function of the present estimate of the parameters, making the actual function being optimized quite different from least squares. $\endgroup$
    – Glen_b
    Commented Mar 15, 2015 at 5:02
  • 1
    $\begingroup$ summarization, SVM is an improved variant of LR, because it finds the hyperplane with the maximum marging while LR just finds a (sort of speak random?) hyperplane. Do you agree with this summarization? $\endgroup$
    – LandonZeKepitelOfGreytBritn
    Commented Aug 13, 2017 at 16:54
  • $\begingroup$ @LandonZeKepitelOfGreytBritn Your summarization is what the question suggested. It is almost correct. I explained the subtilties. In case when the two classes are not linearly separable, it does not find the hyperplane with maximum margin. Margin is one the two terms in the cost function. The cost function favors large margin and it trades it off against the number of points that the plane gets wrong. The weight of the tradeoff, is the regularization parameter. $\endgroup$
    – Leila
    Commented May 5, 2022 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.