Why does the supporting vector satisfy $y_i(\mathbf{w}^T\mathbf{x_i}+b) = 1$ instead of $> 1$ or $= 2$

Question

The SVM is about solving the constrained optimization such that

$$\min_{\mathbf{w}} \dfrac{1}{2} \mathbf{w}^T\mathbf{w}$$ subject to $$y_i(\mathbf{w}^T\mathbf{x_i}+b)\geq{1}, i=1, 2, ...,n$$

Suppose that $(\mathbf{w}, b)$ is the optimal hyperplane that we get by solving the SVM problem, then it must be true that the supporting vectors satisfy the following relationship:

$$y_i(\mathbf{w}^T\mathbf{x_i}+b) = 1$$

My question is why this must be the case?

Our constraint says that our hyperplane is the optimal one as long as

$$y_i(\mathbf{w}^T\mathbf{x_i}+b)\geq{1}, i=1, 2, ...,n$$

is satisfied.

So why couldn't the supporting vectors satisfy $$y_i(\mathbf{w}^T\mathbf{x_i}+b) > 1?$$ or $$y_i(\mathbf{w}^T\mathbf{x_i}+b) = 2?$$

or $$y_i(\mathbf{w}^T\mathbf{x_i}+b) = 100?$$

Answer 1

I think the confusion comes from the definition of support vectors : a point $i$ is defined as a support vector precisely if $y_i(w^Tx_i+b) = 1$. The inequality $y_i(w^Tx_i+b) \geq 1$ must hold for all data points, but the support vectors are the data points for which this constraint is attained the least.

Support vectors satisfy $y_i(w^Tx_i+b) = 1$... because that's how they are defined !

For more information (and especially to have a nice graphical explanation of the role of support vectors), you may want to have a look at the excellent lecture notes written by Andrew Ng on SVM, which are freely available and are a must-read.

Answer 2

The support vectors are just the points that are used to define your hyperplane. These are basically the points of the two classes that are the closest together, and that are used to define the margins. From a visual perspective, your support vectors are the points that sit on your margins.
Indeed, if you add training data to an SVM, any point that does not change the support vectors (i.e. any new point for which $y_i(w^Tx_i+b)>1$) does not entail any computation or additional training. New training is needed only if you add a training point for which $y_i(w^Tx_i+b)<1$, and in that case the margins will need to be recomputed and this new point will become most likely a support vector, so $y_i(w^Tx_i+b)=1$ (here $w$ and $b$ are changed)

I really suggest listening to this MIT lecture, I really found it eye-opening at the time: MIT SVM lecture