In the linear SVM model, one may have the following equation to describe how to achieve a maximal margin while still classifying the data into 2 groups: \begin{equation} L(w, \epsilon) = w\cdot w + \lambda\sum\limits_{i=1}^R \epsilon_i \end{equation} In the above, $w$ is the weight vector and the $\epsilon_i$ are the error distances for a mis-classification of $x_i$. There are $R$ mis-classifications total.

My textbook tells me that if $\lambda$ is small we prefer a large Margin over a few erros and vice versa. I do not understand this reasoning at all. I do not see how varying $\lambda$ affects the minimization of $L$.

Could someone please explain to me how varying $\lambda$, changes what is being minimized in $L$?


Minimizing the total cost $L$ can be split up into two distinct parts, namely minimizing the squared norm of the separating hyperplane $\|\mathbf{w}\|^2$ and the misclassification penalties $\lambda \sum_{i=1}^T \xi_i$. Note that minimizing $\|\mathbf{w}\|^2$ is equivalent to maximizing the margin, which is $1/\|\mathbf{w}\|$.

If $\lambda$ is large, minimizing the latter term becomes increasingly important to minimize the overall cost $L$ and vice versa. If this is still unclear, imagine the extreme cases $\lambda = 0$ and $\lim_{\lambda\rightarrow\infty}$. In the former, we only care about minimizing $\|\mathbf{w}\|^2$ whereas in the latter we only care about minimizing $\sum_{i=1}^R \xi_i$.

Training an SVM really involves finding the coefficients $\alpha$ and $b$ in the model. The model is a separating hyperplane in feature space, e.g., $$f(\mathbf{z}) = \sum_{i=1}^{n_{SV}} y_i \alpha_i \mathbf{x}_i^T \mathbf{z} + b,$$ where $\mathbf{y}$ is the vector of labels and $\mathbf{x}_i$ are support vectors. For notational simplicity I assume we work with the linear kernel.

A certain solution, e.g., a vector of $\alpha$ values and $b$, induces a certain margin (inversely related to $\|\mathbf{w}\|^2$) and a certain number of misclassifications / points within the margin (quantified by $\sum_{i=1}^N \xi_i$). The goal now is to find a sweet spot between having a larger margin on one hand (e.g. minimize $\|\mathbf{w}\|^2$, simple model) and having fewer training misclassifications (e.g., minimize $\sum_{i=1}^N \xi_i$, complex model). We need to make a tradeoff. That is what $\lambda$ is for.

Consider the following contrived example (this is a simplified version of how it really works): suppose that we have two possible solutions, with the following properties:

  1. $(\alpha^{(1)}, b^{(1)})$ which yields $\|\mathbf{w}^{(1)}\|^2$ = 1 and $\sum_{i=1}^N \xi_i^{(1)}$ = 2.
  2. $(\alpha^{(2)}, b^{(2)})$ which yields $\|\mathbf{w}^{(2)}\|^2$ = 2 and $\sum_{i=1}^N \xi_i^{(2)}$ = 1.

For $\lambda < 1$, solution 1 is preferable since it yields lower cost ($L^{(1)} < 3$ vs $L^{(2)} > 3$). On the other hand, if $\lambda > 1$, solution 2 is preferable ($L^{(1)} = 1 + 2\lambda$ vs $L^{(2)} = 2 + \lambda$).

  • $\begingroup$ Writing it this way makes it seem like you can only minimize either $||w||^2$ or $\lambda\sum\limits_{i=1}^T \epsilon_i$. Is this the case? $\endgroup$ – CodeKingPlusPlus Apr 24 '14 at 0:34
  • $\begingroup$ @CodeKingPlusPlus no, since $\lambda$ can range from $0$ to $\infty$ you can have any balance you want. In the extreme cases you are only minimizing a single term. $\endgroup$ – Marc Claesen Apr 24 '14 at 6:52
  • $\begingroup$ Can you explain what you mean by 'balance,' I think that is the key idea I am missing. $\endgroup$ – CodeKingPlusPlus Apr 25 '14 at 1:50
  • $\begingroup$ @CodeKingPlusPlus I've added an example to clarify things. $\endgroup$ – Marc Claesen Apr 25 '14 at 7:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.