# Practical exercise to solve SVM and logistic regression

I have a simple task regarding SVM and logistic regression, even though I know the theory behind SVM and logistic regression I still have a problem to solve them.

Exercise. We want to learn a hard-margin linear SVM from the following points, where green points have positive label and red points have negative labels.Find the linear support vector machine? What will you get by training a logistic-regression classifier on the given dataset?

Solution: In order to solve SVM let's find $w$ and $b$. There are two hyper-planes with the largest margin, the first one goes through points $(1,-1)$ and $(1,1)$, the second hyper-plane goes through $(-1,0)$ and parallel to the first one. I hope it's correct?

We get three equaltions

$(-1, 0) \cdot w + b = -1$

$(1, -1) \cdot w + b = 1$

$(1, 1) \cdot w + b = 1$

This transforms to

$-w_1 + b = -1$

$w_1 -w_2+ b = 1$

$w_1 + w_2+ b = 1$

I cannot find exact solution, however by substitution we can check that $b=0$ and $w^T=(1,0)$ does the work. I am not sure if this is correct.

I am highly confused by the second question: What will you get by training a logistic-regression classifier on the given dataset?

I would start by defining margins according to sigmoid function.

$w_1 + b < 0$; $-2w_1 + 2w_2 +b <0$; $-2w_1 - 2w_2 +b <0$

$w_1 -w_2 + b > 0$;$w_1 +w_2 + b > 0$;$2w_1 -2w_2 + b > 0$;$2w_1 + b > 0$;$2w_1 +2w_2 + b > 0$

I am not sure how to proceed from here.

Here I have two questions, whether the used approach to solve SVM is correct? How to answer the second question regarding logistic regression?

• What happens to logistic regression when there exists a line that perfectly separates the two classes? – Sycorax Mar 6 '15 at 19:33
• @user777, I found the answer "lack of convergence" but I can not understand why. – com Mar 6 '15 at 19:45
• The unique solution to the three equations in the first question is $(w_1,w_2,b)=(1,0,0)$, found using any method of solving systems of simultaneous linear equations you like. Since you're shaky on this, and it is so fundamental for understanding hyperplanes, separation, linear regression, and much more, I warmly recommend you review methods for solving linear systems before you proceed further with your studies. – whuber Mar 6 '15 at 19:52
• @user777, could you please elaborate a little bit, I assume you mean "the lack of convergence" – com Mar 7 '15 at 6:29
• @fog I think that following whuber's advice would be a brilliant way to proceed. – Sycorax Mar 8 '15 at 2:22