Understanding the tradeoff with regularization in SVMs

Question

In the linear SVM model, one may have the following equation to describe how to achieve a maximal margin while still classifying the data into 2 groups: \begin{equation} L(w, \epsilon) = w\cdot w + \lambda\sum\limits_{i=1}^R \epsilon_i \end{equation} In the above, $w$ is the weight vector and the $\epsilon_i$ are the error distances for a mis-classification of $x_i$. There are $R$ mis-classifications total.

My textbook tells me that if $\lambda$ is small we prefer a large Margin over a few erros and vice versa. I do not understand this reasoning at all. I do not see how varying $\lambda$ affects the minimization of $L$.

Could someone please explain to me how varying $\lambda$, changes what is being minimized in $L$?

Answer 1

Minimizing the total cost $L$ can be split up into two distinct parts, namely minimizing the squared norm of the separating hyperplane $\|\mathbf{w}\|^2$ and the misclassification penalties $\lambda \sum_{i=1}^T \xi_i$. Note that minimizing $\|\mathbf{w}\|^2$ is equivalent to maximizing the margin, which is $1/\|\mathbf{w}\|$.

If $\lambda$ is large, minimizing the latter term becomes increasingly important to minimize the overall cost $L$ and vice versa. If this is still unclear, imagine the extreme cases $\lambda = 0$ and $\lim_{\lambda\rightarrow\infty}$. In the former, we only care about minimizing $\|\mathbf{w}\|^2$ whereas in the latter we only care about minimizing $\sum_{i=1}^R \xi_i$.

---

Training an SVM really involves finding the coefficients $\alpha$ and $b$ in the model. The model is a separating hyperplane in feature space, e.g., $$f(\mathbf{z}) = \sum_{i=1}^{n_{SV}} y_i \alpha_i \mathbf{x}_i^T \mathbf{z} + b,$$ where $\mathbf{y}$ is the vector of labels and $\mathbf{x}_i$ are support vectors. For notational simplicity I assume we work with the linear kernel.

A certain solution, e.g., a vector of $\alpha$ values and $b$, induces a certain margin (inversely related to $\|\mathbf{w}\|^2$) and a certain number of misclassifications / points within the margin (quantified by $\sum_{i=1}^N \xi_i$). The goal now is to find a sweet spot between having a larger margin on one hand (e.g. minimize $\|\mathbf{w}\|^2$, simple model) and having fewer training misclassifications (e.g., minimize $\sum_{i=1}^N \xi_i$, complex model). We need to make a tradeoff. That is what $\lambda$ is for.

Consider the following contrived example (this is a simplified version of how it really works): suppose that we have two possible solutions, with the following properties:

1. $(\alpha^{(1)}, b^{(1)})$ which yields $\|\mathbf{w}^{(1)}\|^2$ = 1 and $\sum_{i=1}^N \xi_i^{(1)}$ = 2.
2. $(\alpha^{(2)}, b^{(2)})$ which yields $\|\mathbf{w}^{(2)}\|^2$ = 2 and $\sum_{i=1}^N \xi_i^{(2)}$ = 1.

For $\lambda < 1$, solution 1 is preferable since it yields lower cost ($L^{(1)} < 3$ vs $L^{(2)} > 3$). On the other hand, if $\lambda > 1$, solution 2 is preferable ($L^{(1)} = 1 + 2\lambda$ vs $L^{(2)} = 2 + \lambda$).