Understanding the tradeoff with regularization in SVMs

In the linear SVM model, one may have the following equation to describe how to achieve a maximal margin while still classifying the data into 2 groups: $$L(w, \epsilon) = w\cdot w + \lambda\sum\limits_{i=1}^R \epsilon_i$$ In the above, $w$ is the weight vector and the $\epsilon_i$ are the error distances for a mis-classification of $x_i$. There are $R$ mis-classifications total.

My textbook tells me that if $\lambda$ is small we prefer a large Margin over a few erros and vice versa. I do not understand this reasoning at all. I do not see how varying $\lambda$ affects the minimization of $L$.

Could someone please explain to me how varying $\lambda$, changes what is being minimized in $L$?

Minimizing the total cost $L$ can be split up into two distinct parts, namely minimizing the squared norm of the separating hyperplane $\|\mathbf{w}\|^2$ and the misclassification penalties $\lambda \sum_{i=1}^T \xi_i$. Note that minimizing $\|\mathbf{w}\|^2$ is equivalent to maximizing the margin, which is $1/\|\mathbf{w}\|$.

If $\lambda$ is large, minimizing the latter term becomes increasingly important to minimize the overall cost $L$ and vice versa. If this is still unclear, imagine the extreme cases $\lambda = 0$ and $\lim_{\lambda\rightarrow\infty}$. In the former, we only care about minimizing $\|\mathbf{w}\|^2$ whereas in the latter we only care about minimizing $\sum_{i=1}^R \xi_i$.

Training an SVM really involves finding the coefficients $\alpha$ and $b$ in the model. The model is a separating hyperplane in feature space, e.g., $$f(\mathbf{z}) = \sum_{i=1}^{n_{SV}} y_i \alpha_i \mathbf{x}_i^T \mathbf{z} + b,$$ where $\mathbf{y}$ is the vector of labels and $\mathbf{x}_i$ are support vectors. For notational simplicity I assume we work with the linear kernel.

A certain solution, e.g., a vector of $\alpha$ values and $b$, induces a certain margin (inversely related to $\|\mathbf{w}\|^2$) and a certain number of misclassifications / points within the margin (quantified by $\sum_{i=1}^N \xi_i$). The goal now is to find a sweet spot between having a larger margin on one hand (e.g. minimize $\|\mathbf{w}\|^2$, simple model) and having fewer training misclassifications (e.g., minimize $\sum_{i=1}^N \xi_i$, complex model). We need to make a tradeoff. That is what $\lambda$ is for.

Consider the following contrived example (this is a simplified version of how it really works): suppose that we have two possible solutions, with the following properties:

1. $(\alpha^{(1)}, b^{(1)})$ which yields $\|\mathbf{w}^{(1)}\|^2$ = 1 and $\sum_{i=1}^N \xi_i^{(1)}$ = 2.
2. $(\alpha^{(2)}, b^{(2)})$ which yields $\|\mathbf{w}^{(2)}\|^2$ = 2 and $\sum_{i=1}^N \xi_i^{(2)}$ = 1.

For $\lambda < 1$, solution 1 is preferable since it yields lower cost ($L^{(1)} < 3$ vs $L^{(2)} > 3$). On the other hand, if $\lambda > 1$, solution 2 is preferable ($L^{(1)} = 1 + 2\lambda$ vs $L^{(2)} = 2 + \lambda$).

• Writing it this way makes it seem like you can only minimize either $||w||^2$ or $\lambda\sum\limits_{i=1}^T \epsilon_i$. Is this the case? – CodeKingPlusPlus Apr 24 '14 at 0:34
• @CodeKingPlusPlus no, since $\lambda$ can range from $0$ to $\infty$ you can have any balance you want. In the extreme cases you are only minimizing a single term. – Marc Claesen Apr 24 '14 at 6:52
• Can you explain what you mean by 'balance,' I think that is the key idea I am missing. – CodeKingPlusPlus Apr 25 '14 at 1:50
• @CodeKingPlusPlus I've added an example to clarify things. – Marc Claesen Apr 25 '14 at 7:32