# svm functional margin and geometric margin

I know that the functional margin has the following formula

and I have read that given a training set we define the function margin of (w.b) with respect to S to be the minimum of this functional margins:

but why it says that one should find the maximum geometrical margin? should it not be also to find the maximum of the functional margin from the beginning? For what I know that geometrical margin is only the functional margin normalized by ||w|| to consider the distances between the points to the decision boundary.

Another question (because it is greatly related and I consider that is not necessary to open another thread), why is it better to find a wide margin instead that a narrow one?

I think that the proper way to write the functional margin is

$$\gamma_i = y_i(w^Tx_i + b),$$

while the geometric margin is simply

$$\hat{\gamma}_i = \frac{\gamma_i}{||w||}.$$

[...] the functional margin would give you a number but without a reference you can't tell if the point is actually far away or close to the decision plane. The geometric margin is telling you not only if the point is properly classified or not, but the magnitude of that distance in term of units of |w|.

Regarding the second question, see what happens to the Perceptron algorithm. It tries to build a hyperplane between linearly separable data the same as SVM, but it could be any hyperplane. So depending on the training data you used you could have very different hyperplanes, ergo, very different predictions in presence of new data.

SVM tries to avoid that by finding the optimal hyperplane, that's why the margin has to be the widest possible, to reduce the chance of misclassification in presence of new data.

Regarding following question

but why it says that one should find the maximum geometrical margin? should it not be also to find the maximum of the functional margin from the beginning?

You are very right our goal is to maximize both geometrical margin or functional margin but the confusion is that following formula is for set of points, ideally we want to maximize the margin width (sum of functional margin of decision boundary (separation boundary) to the closest points (support vectors) belonging to different classes (consider 2 classes only to simplify the explanation here).

In this formula, we use minimum to find out the support vectors (closest points to the decision boundary) and then once we find these support vectors (by using this minimizing optimization problem). We will try to maximize the functional margin (that is we choose the best decision boundary to maximize the functional margin).

Please confirm whether this explanation clarify some of your doubts or just made it worse (obviously sorry for the latter part)

but why it says that one should find the maximum geometrical margin?

Because the geometric margin is invariant to the scaling of the vector orthogonal to the hyperplane. Please see the answer here.

should it not be also to find the maximum of the functional margin from the beginning?

Since scaling the parameters w and b results nothing meaningful and the parameters are scaled in the same way as the functional margin, then if we can arbitrarily make the ||w|| to be 1(resulting in maximizing the geometric margin) we can also rescale the parameters to make them subject to the functional margin being 1(then to minimize ||w||).

Then the solution is to transform maximizing the geomatric margin to minimizing the magnitude of the vector orthogonal to the hyperplane $$min_{\gamma, w, b} \frac{1}{2}||w||^2$$ subject to $$y^{(i)}(w^Tx^{(i)}+b)\ge 1$$ which is optimizable. It is different from finding the maximum of $$\frac{the\_functional\_margin}{||w||}$$ with the constraint of $$y^{(i)}(w^Tx^{(i)}+b)\ge the\_functional\_margin$$ which is normally impossible to be optimized.

For what I know that geometrical margin is only the functional margin normalized by ||w|| to consider the distances between the points to the decision boundary?

Let's say $$\gamma^{(i)}$$ is a geometric margin. w/||w|| is a unit-length vector orthogonal to the hyperplane. A represents $$x^{(i)}$$, then point B is given by $$x^{(i)}-\gamma^{(i)} * w/||w||$$. Since B lies on the decision boundary, and $$w^Tx+b=0$$, then,

$$w^T(x^{(i)})-\gamma^{(i)}\frac{w}{||w||}+b=0$$

Solving for $$\gamma^{(i)}$$ obtain:

$$\gamma^{(i)}=\frac{w^Tx^{(i)}+b}{||w||}=(\frac{w}{||w||})^Tx^{(i)}+\frac{b}{||w||}$$

As you can see the geometric margin is only the functional margin normalized by ||w|| to consider the distances between the points to the decision boundary.

Another question ... why is it better to find a wide margin instead that a narrow one?

The wider the margin the more confident we say the hyperplane is tuned well. The more distant a point is from the hyperplane, as you can imagine the more condident or possible that the point is divided into the right group, otherwise the hyperplane just moves a little and the point is divided into the other side of it and hence is grouped wrongly.