11
$\begingroup$

The optimal hyperplane in SVM is defined as:

$$\mathbf w \cdot \mathbf x+b=0,$$

where $b$ represents threshold. If we have some mapping $\mathbf \phi$ which maps input space to some space $Z$, we can define SVM in the space $Z$, where the optimal hiperplane will be:

$$\mathbf w \cdot \mathbf \phi(\mathbf x)+b=0.$$

However, we can always define mapping $\phi$ so that $\phi_0(\mathbf x)=1$, $\forall \mathbf x$, and then the optimal hiperplane will be defined as $$\mathbf w \cdot \mathbf \phi(\mathbf x)=0.$$

Questions:

  1. Why many papers use $\mathbf w \cdot \mathbf \phi(\mathbf x)+b=0$ when they already have mapping $\phi$ and estimate parameters $\mathbf w$ and theshold $b$ separatelly?

  2. Is there some problem to define SVM as $$\min_{\mathbf w} ||\mathbf w ||^2$$ $$s.t. \ y_n \mathbf w \cdot \mathbf \phi(\mathbf x_n) \geq 1, \forall n$$ and estimate only parameter vector $\mathbf w$, assuming that we define $\phi_0(\mathbf x)=1, \forall\mathbf x$?

  3. If definition of SVM from question 2. is possible, we will have $\mathbf w = \sum_{n} y_n\alpha_n \phi(\mathbf x_n)$ and threshold will be simply $b=w_0$, which we will not treat separately. So we will never use formula like $b=t_n-\mathbf w\cdot \phi(\mathbf x_n)$ to estimate $b$ from some support vector $x_n$. Right?

$\endgroup$
10
$\begingroup$

Why bias is important?

The bias term $b$ is, indeed, a special parameter in SVM. Without it, the classifier will always go through the origin. So, SVM does not give you the separating hyperplane with the maximum margin if it does not happen to pass through the origin, unless you have a bias term.

Below is a visualization of the bias issue. An SVM trained with (without) a bias term is shown on the left (right). Even though both SVMs are trained on the same data, however, they look very different.

enter image description here

Why should the bias be treated separately?

As Ben DAI pointed out, the bias term $b$ should be treated separately because of regularization. SVM maximizes the margin size, which is $\frac{1}{||w||^2}$ (or $\frac{2}{||w||^2}$ depending on how you define it).

Maximizing the margin is the same as minimizing $||w||^2$. This is also called the regularization term and can be interpreted as a measure of the complexity of the classifier. However, you do not want to regularize the bias term because, the bias shifts the classification scores up or down by the same amount for all data points. In particular, the bias does not change the shape of the classifier or its margin size. Therefore, ...

the bias term in SVM should NOT be regularized.

In practice, however, it is easier to just push the bias into the feature vector instead of having to deal with as a special case.

Note: when pushing the bias to the feature function, it is best to fix that dimension of the feature vector to a large number, e.g. $\phi_0(x) = 10$, so as to minimize the side-effects of regularization of the bias.

$\endgroup$
1
$\begingroup$

Sometimes, people will just omit the intercept in SVM, but i think the reason maybe we can penalizing intercept in order to omit it. i.e.,

we can modify the data $\mathbf{\hat{x}} = (\mathbf{1}, \mathbf{x})$, and $\mathbf{\hat{w}} = (w_{0}, \mathbf{w}^{T})^{T}$ so that omit the intercept $$\mathbf{x} ~ \mathbf{w} + b = \mathbf{\hat{x}} ~ \mathbf{\hat{w}} $$ As you said, similar technique can be used in kernel version.

However, if we put the intercept in the weights, the objective function will slightly different with original one. That's why we call "penalize".

$\endgroup$
  • $\begingroup$ I aggree that we will have different objective functions. Case when we do not include intercept $b$ in parameters leads to the optimization problem $\min_{\mathbf w,b} ||\mathbf w||^2$ subject to constraint, while otherwise we have problem $\min_{\mathbf w,b} ||\mathbf w||^2 + b^2$. But, I don't understand why panalizing intercept more or less is important for the model. $\endgroup$ – Dejan Dec 1 '15 at 13:32
  • $\begingroup$ What comes to my mind, is that the main reason we have intersect is maybe because in dual problem, intercept allows us to have constraint $\sum \alpha_n t_n=0$ which is important to apply SMO algorithm, and if we do not have intercept we will have only constants $\alpha_n\geq 0$ and the dual optimization would be harder in that case. $\endgroup$ – Dejan Dec 1 '15 at 13:33
  • $\begingroup$ @Petar One thing I known is that it becomes powerful when we consider about the Dual form of this model. This technique will eliminate the linear constrain. $\endgroup$ – Ben Dai Dec 1 '15 at 13:34
  • $\begingroup$ @Petar I don't think the dual optimization will be harder, since we have easier domain. $\endgroup$ – Ben Dai Dec 1 '15 at 13:36
  • $\begingroup$ @Petar For specific algorithm, it may be harder. However, mathematically, i think box domain maybe better : ) $\endgroup$ – Ben Dai Dec 1 '15 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.