Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Hard-margin SVM doesn't seem to work on non-linearly separable data. It seems to only work if your data is linearly separable. What happens if you try to use hard-margin SVM? Does the algorithm blow-up?
I have the same question for logistic regression, but it's not clear to me what happens when the data isn't linearly separable.
logistic regression would blow up if the data are linearly separable, if they are not you would just learn a probability of the outcome given the input variable, the decision boundary is just a byproduct of this. Intuitively, since we are looking at the function that assigns a probability of an outcome, if the data are linearly separable, the probability wil be either 0 or 1 and there are infinitely many solutions for this problem. Also, logistic regression is based on likelihood, and likelihood, which will go to infinity for probabilities of 0 and 1 exactly (I think)
SVM is a constrained optimization problem and with a hard margin and nonlinearly separable data, the constrains cannot be met so the solution does not exist, what happens depends on the optimizer you will use, it might crash, tell you the solution does not exist not converge and go forever, or not converge and stop after a fixed number of iterations
