Why is the bias term in SVM estimated separately, instead of an extra dimension in the feature vector?

The optimal hyperplane in SVM is defined as:

$$\mathbf w \cdot \mathbf x+b=0,$$

where $b$ represents threshold. If we have some mapping $\mathbf \phi$ which maps input space to some space $Z$, we can define SVM in the space $Z$, where the optimal hiperplane will be:

$$\mathbf w \cdot \mathbf \phi(\mathbf x)+b=0.$$

However, we can always define mapping $\phi$ so that $\phi_0(\mathbf x)=1$, $\forall \mathbf x$, and then the optimal hiperplane will be defined as $$\mathbf w \cdot \mathbf \phi(\mathbf x)=0.$$

Questions:

1. Why many papers use $\mathbf w \cdot \mathbf \phi(\mathbf x)+b=0$ when they already have mapping $\phi$ and estimate parameters $\mathbf w$ and theshold $b$ separatelly?

2. Is there some problem to define SVM as $$\min_{\mathbf w} ||\mathbf w ||^2$$ $$s.t. \ y_n \mathbf w \cdot \mathbf \phi(\mathbf x_n) \geq 1, \forall n$$ and estimate only parameter vector $\mathbf w$, assuming that we define $\phi_0(\mathbf x)=1, \forall\mathbf x$?

3. If definition of SVM from question 2. is possible, we will have $\mathbf w = \sum_{n} y_n\alpha_n \phi(\mathbf x_n)$ and threshold will be simply $b=w_0$, which we will not treat separately. So we will never use formula like $b=t_n-\mathbf w\cdot \phi(\mathbf x_n)$ to estimate $b$ from some support vector $x_n$. Right?

• Dec 9, 2015 at 21:33

Why bias is important?

The bias term $$b$$ is, indeed, a special parameter in SVM. Without it, the classifier will always go through the origin. So, SVM does not give you the separating hyperplane with the maximum margin if it does not happen to pass through the origin, unless you have a bias term.

Below is a visualization of the bias issue. An SVM trained with (without) a bias term is shown on the left (right). Even though both SVMs are trained on the same data, however, they look very different.

Why should the bias be treated separately?

As user logistic pointed out, the bias term $$b$$ should be treated separately because of regularization. SVM maximizes the margin size, which is $$\frac{1}{||w||^2}$$ (or $$\frac{2}{||w||^2}$$ depending on how you define it).

Maximizing the margin is the same as minimizing $$||w||^2$$. This is also called the regularization term and can be interpreted as a measure of the complexity of the classifier. However, you do not want to regularize the bias term because, the bias shifts the classification scores up or down by the same amount for all data points. In particular, the bias does not change the shape of the classifier or its margin size. Therefore, ...

the bias term in SVM should NOT be regularized.

In practice, however, it is easier to just push the bias into the feature vector instead of having to deal with as a special case.

Note: when pushing the bias to the feature function, it is best to fix that dimension of the feature vector to a large number, e.g. $$\phi_0(x) = 10$$, so as to minimize the side-effects of regularization of the bias.

• What program did you use to generate the plots, out of curiosity? Dec 9, 2015 at 18:26
• @d0rmLife: this is just a cartoon that I made using MS PowerPoint!
– Sobi
Dec 9, 2015 at 19:38
• Dec 9, 2015 at 21:33

Sometimes, people will just omit the intercept in SVM, but i think the reason maybe we can penalizing intercept in order to omit it. i.e.,

we can modify the data $\mathbf{\hat{x}} = (\mathbf{1}, \mathbf{x})$, and $\mathbf{\hat{w}} = (w_{0}, \mathbf{w}^{T})^{T}$ so that omit the intercept $$\mathbf{x} ~ \mathbf{w} + b = \mathbf{\hat{x}} ~ \mathbf{\hat{w}}$$ As you said, similar technique can be used in kernel version.

However, if we put the intercept in the weights, the objective function will slightly different with original one. That's why we call "penalize".

• I aggree that we will have different objective functions. Case when we do not include intercept $b$ in parameters leads to the optimization problem $\min_{\mathbf w,b} ||\mathbf w||^2$ subject to constraint, while otherwise we have problem $\min_{\mathbf w,b} ||\mathbf w||^2 + b^2$. But, I don't understand why panalizing intercept more or less is important for the model. Dec 1, 2015 at 13:32
• What comes to my mind, is that the main reason we have intersect is maybe because in dual problem, intercept allows us to have constraint $\sum \alpha_n t_n=0$ which is important to apply SMO algorithm, and if we do not have intercept we will have only constants $\alpha_n\geq 0$ and the dual optimization would be harder in that case. Dec 1, 2015 at 13:33
• @Petar One thing I known is that it becomes powerful when we consider about the Dual form of this model. This technique will eliminate the linear constrain. Dec 1, 2015 at 13:34
• @Petar I don't think the dual optimization will be harder, since we have easier domain. Dec 1, 2015 at 13:36
• @Petar For specific algorithm, it may be harder. However, mathematically, i think box domain maybe better : ) Dec 1, 2015 at 13:39

In additional to the reasons mentioned above, the distance of a point $$x$$ to a hyperplane defined by slope $$\theta$$ and intercept $$b$$ is $$\frac{|\theta^T x + b|}{||\theta||}$$ This is how the concept of margin in SVM is movitated. If you change the $$\theta$$ to include the intercept term $$b$$, the norm of the $$\theta$$ will be affected by the size of the intercept, which will cause the SVM to optimize towards a small intercept, which does not make sense in many cases.

• Even thought the distance of a point to a hyperplane is correct and the explanation looks interesting, I do not see correlation between this formula and training SVMs. Can you explain better how this formula is using during training or provide some additional link. Aug 2, 2019 at 8:12
• @Dejan The idea behind a SVM is to find the hyperplane that maximizes the minimum margin of a dataset. The margin is the "distance" ($\frac{\theta^T x + b}{||\theta||}$, without taking absolute value, which indicates the confidence the classifier has towards its hypothesis) of that point to the hyperplane times its label, which is in $\{-1, 1\}$. The product is $\frac{y(\theta^T x + b)}{||\theta||}$, which is positive if the classifier output matches the label and negative otherwise. In practice, we simply scale our model so that the minimum margin of the data set is $\frac{1}{||\theta||}$. Aug 2, 2019 at 9:13
• @Dejan you can find more details in Andrew Ng's Notes: cs229.stanford.edu/notes/cs229-notes3.pdf Aug 2, 2019 at 9:14