0
$\begingroup$

Hard-margin SVM doesn't seem to work on non-linearly separable data. It seems to only work if your data is linearly separable. What happens if you try to use hard-margin SVM? Does the algorithm blow-up?

I have the same question for logistic regression, but it's not clear to me what happens when the data isn't linearly separable.

$\endgroup$

1 Answer 1

3
$\begingroup$

logistic regression would blow up if the data are linearly separable, if they are not you would just learn a probability of the outcome given the input variable, the decision boundary is just a byproduct of this. Intuitively, since we are looking at the function that assigns a probability of an outcome, if the data are linearly separable, the probability wil be either 0 or 1 and there are infinitely many solutions for this problem. Also, logistic regression is based on likelihood, and likelihood, which will go to infinity for probabilities of 0 and 1 exactly (I think)

SVM is a constrained optimization problem and with a hard margin and nonlinearly separable data, the constrains cannot be met so the solution does not exist, what happens depends on the optimizer you will use, it might crash, tell you the solution does not exist not converge and go forever, or not converge and stop after a fixed number of iterations

$\endgroup$
3
  • $\begingroup$ Wait, why does logistic regression blow up if data IS linearly separable? $\endgroup$ Oct 5, 2020 at 18:30
  • $\begingroup$ I edited my answer. There are also much better explanations of the separability problem of logistic regression on this site. $\endgroup$
    – rep_ho
    Oct 5, 2020 at 18:43
  • 1
    $\begingroup$ I agree with rep_ho's answer, but I offer my explanation here regarding logistic regression: stats.stackexchange.com/a/462504/169343 $\endgroup$
    – Igor F.
    Oct 5, 2020 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.