# Linear Constraint in SVM optimization

$\newcommand{\loss}{\operatorname{loss}}$Recently I am surveying different SVM optimization algorithms. I came across a strange scenario:

When we formulate the SVM primal problem like the following, $$\min_w \frac{1}{2}w^Tw + C\sum_{i=1}^{m}\loss(w, x_i, y_i),$$ $$\text{such that }\quad y_i(w^Tx_i + b) \ge 1 - c_i,$$ because of the linear constraint, we will have the following constraint in the dual formulation: $$\alpha^Ty = 0,$$ and this means that we need to optimize at least two variables at a time.

But some papers will formulate the SVM problem in the unconstrained version, simply: $$\min_w \frac{1}{2}w^Tw + C\sum_{i=1}^{m}\loss(w, x_i, y_i).$$ Then because we no longer have the linear constraint, we can apply method such as coordinate descent, which updates only one variable at a time.

I am confused, what is the difference between these two formulations of SVM?

Note that generally, the loss$(w,x_i,y_i)$ term you have written would actually be the $c_i$ value from the constraint, in both versions.
The dual-formulation constraint $\alpha^T y = 0$ arises from the bias term $b$ which offsets the solution plane from the origin. The most common alternative to using the bias term is extending all the inputs $x_i$ by adding on a dimension with constant value $1$. A less commonly used approach is to use a slightly modified kernel function formulation to replicate $b$ (i.e., so that calculating $<w,x>_{k'}$ with the modified kernel $k'$ is similar to evalutating $<w,x>_k + b$ with the original kernel).
• When you remove the bias term, the SVM hyperplane will always pass through the origin in kernel space. Adding an additional dimension gives another degree of freedom (roughly equivalent to the bias term, although note that in the biased SVM formulation the $b$ term is not included in the normalization, yet, for unbiased SVM, this additional degree of freedom may show up in the normalization - for example, with a linear kernel, $w$ grows by 1-dimension and the value for $w^Tw$ will increase as a result). – MotiNK Nov 17 '16 at 12:42
• $b$ and $w$ are generally optimized together. In the dual formulation one solves for $\alpha$ and one can then calculate $b$ and $w$ from these values. – MotiNK Nov 17 '16 at 12:45