svm functional margin and geometric margin

Question

I know that the functional margin has the following formula

and I have read that given a training set we define the function margin of (w.b) with respect to S to be the minimum of this functional margins:

but why it says that one should find the maximum geometrical margin? should it not be also to find the maximum of the functional margin from the beginning? For what I know that geometrical margin is only the functional margin normalized by ||w|| to consider the distances between the points to the decision boundary.

Another question (because it is greatly related and I consider that is not necessary to open another thread), why is it better to find a wide margin instead that a narrow one?

Answer 1

I think that the proper way to write the functional margin is

$$ \hat{\gamma}_i = y_i(w^Tx_i + b), $$

while the geometric margin is simply

$$ \gamma_i = \frac{\hat{\gamma}_i}{||w||}. $$

You can find the answer to your first question in here:

[...] the functional margin would give you a number but without a reference you can't tell if the point is actually far away or close to the decision plane. The geometric margin is telling you not only if the point is properly classified or not, but the magnitude of that distance in term of units of |w|.

Regarding the second question, see what happens to the Perceptron algorithm. It tries to build a hyperplane between linearly separable data the same as SVM, but it could be any hyperplane. So depending on the training data you used you could have very different hyperplanes, ergo, very different predictions in presence of new data.

SVM tries to avoid that by finding the optimal hyperplane, that's why the margin has to be the widest possible, to reduce the chance of misclassification in presence of new data.

Answer 2

but why it says that one should find the maximum geometrical margin?

Because the geometric margin is invariant to the scaling of the vector orthogonal to the hyperplane. Please see the answer here.

should it not be also to find the maximum of the functional margin from the beginning?

Since scaling the parameters w and b results nothing meaningful and the parameters are scaled in the same way as the functional margin, then if we can arbitrarily make the ||w|| to be 1(resulting in maximizing the geometric margin) we can also rescale the parameters to make them subject to the functional margin being 1(then to minimize ||w||).

Then the solution is to transform maximizing the geomatric margin to minimizing the magnitude of the vector orthogonal to the hyperplane $min_{\gamma, w, b} \frac{1}{2}||w||^2$ subject to $y^{(i)}(w^Tx^{(i)}+b)\ge 1$ which is optimizable. It is different from finding the maximum of $\frac{the\_functional\_margin}{||w||}$ with the constraint of $y^{(i)}(w^Tx^{(i)}+b)\ge the\_functional\_margin$ which is normally impossible to be optimized.

For what I know that geometrical margin is only the functional margin normalized by ||w|| to consider the distances between the points to the decision boundary?

Let's say $\gamma^{(i)}$ is a geometric margin. w/||w|| is a unit-length vector orthogonal to the hyperplane. A represents $x^{(i)}$, then point B is given by $x^{(i)}-\gamma^{(i)} * w/||w||$. Since B lies on the decision boundary, and $w^Tx+b=0$, then,

$$w^T(x^{(i)})-\gamma^{(i)}\frac{w}{||w||}+b=0$$

Solving for $\gamma^{(i)}$ obtain:

$$\gamma^{(i)}=\frac{w^Tx^{(i)}+b}{||w||}=(\frac{w}{||w||})^Tx^{(i)}+\frac{b}{||w||}$$

As you can see the geometric margin is only the functional margin normalized by ||w|| to consider the distances between the points to the decision boundary.

Another question ... why is it better to find a wide margin instead that a narrow one?

The wider the margin the more confident we say the hyperplane is tuned well. The more distant a point is from the hyperplane, as you can imagine the more condident or possible that the point is divided into the right group, otherwise the hyperplane just moves a little and the point is divided into the other side of it and hence is grouped wrongly.

reference: http://www.stanford.edu/class/cs229/notes/cs229-notes3.pdf

Answer 3

Regarding following question

but why it says that one should find the maximum geometrical margin? should it not be also to find the maximum of the functional margin from the beginning?

You are very right our goal is to maximize both geometrical margin or functional margin but the confusion is that following formula is for set of points, ideally we want to maximize the margin width (sum of functional margin of decision boundary (separation boundary) to the closest points (support vectors) belonging to different classes (consider 2 classes only to simplify the explanation here).

In this formula, we use minimum to find out the support vectors (closest points to the decision boundary) and then once we find these support vectors (by using this minimizing optimization problem). We will try to maximize the functional margin (that is we choose the best decision boundary to maximize the functional margin).

Please confirm whether this explanation clarify some of your doubts or just made it worse (obviously sorry for the latter part)

Answer 4

Not going into unnecessary complications about this concept, but in the most simple terms here is how one can think of and relate functional and geometric margin.

Think of functional margin -- represented as 𝛾̂, as a measure of correctness of a classification for a data unit. For a data unit x with parameters w and b and given class y = 1, the functional margin is 1 only when y and (wx + b) are both of the same sign - which is to say are correctly classified.

But we do not just rely on if we are correct or not in this classification. We need to know how correct we are, or what is the degree of confidence that we have in this classification. For this we need a different measure, and this is called geometric margin -- represented as 𝛾, and it can be expressed as below:

𝛾 = 𝛾̂ / ||𝑤||

So, geometric margin 𝛾 is a scaled version of functional margin 𝛾̂. If ||w|| == 1, then the geometric margin is same as functional margin - which is to say we are as confident in the correctness of this classification as we are correct in classifying a data unit to a particular class.

This scaling by ||w|| gives us the measure of confidence in our correctness. And we always try to maximise this confidence in our correctness.

Functional margin is like a binary valued or boolean valued variable: if we have correctly classified a particular data unit or not. So, this cannot be maximised. However, geometric margin for the same data unit gives a magnitude to our confidence, and tells us how correct we are.So, this we can maximise.

And we aim for larger margin through the means of geometric margin because the wider the margin the more is the confidence in our classification.

As an analogy, say a wider road (larger margin => higher geometric margin) gives higher confidence to drive must faster as it lessens the chance of hitting any pedestrian or trees (our data units in the training set), but on the narrower road (smaller margin => smaller geometric margin), one has to be a lot more cautious to not hit (lesser confidence) any pedestrian or trees. So, we always desire wider roads (larger margin), and that's why we aim to maximise it by maximising our geometric margin.