1
$\begingroup$

According to Wikipedia, the goal of the soft-margin SVM is to minize the hinge loss function:

$$\left[\frac{1}{n} \sum_{i=1}^{n} \max \left(0,1-y_{i}\left(\vec{w} \cdot \vec{x}_{i}-b\right)\right)\right]+\lambda\|\vec{w}\|^{2}$$

Could you tell me more why we add $\lambda$? What is its effect on the minimization?

$\endgroup$
0
$\begingroup$

The goal of $\lambda$ in that equation is to serve as a regularization term (helping to avoid overfitting) which determines the relative importance of minimizing $\Vert w \Vert^2$ w.r.t. minimizing $\frac{1}{n}\sum_{i=1}^n\max(0, 1-y_i(w\cdot x_i - b))$.

By minimizing $\frac{1}{n}\sum_{i=1}^n\max(0, 1-y_i(w\cdot x_i - b))$ we are looking forward to correctly separate the data and with a functional margin $\geq 1$, otherwise the cost function will increase. But minimizing only this term may lead us to undesired results.

This is because in order to separate the samples correctly, the SVM may overfit the dataset. This usually leads to higher values of $\Vert w \Vert^2$ due to the increasing complexity needed to fit the whole dataset correctly.

To prevent this, we add a regularization term $\rightarrow \lambda\Vert w \Vert^2$. By doing this, we are not only penalising the fact that the functional margin is $<1$, but also high values of $\Vert w \Vert^2$.

However, we should not minimize $\Vert w \Vert^2$ indefinitely, because by doing this we are reducing the capacity of the SVM to fit the data $\rightarrow$ we may end up with the opposite problem than before i.e. underfitting the dataset.

So, to sum up, a good balance between minimizing $\frac{1}{n}\sum_{i=1}^n\max(0, 1-y_i(w\cdot x_i - b))$ and minimizing $\Vert w \Vert^2$ needs to be met and this why $\lambda$ is used.

$\endgroup$
2
  • $\begingroup$ Thanks for the explanation! So is it similar to the Ridge penalization? $\endgroup$ – user1315621 Oct 24 '20 at 16:48
  • $\begingroup$ I'm not familiar with Ridge penalization but looking at this post from Statlec it seems that the objective of both regularization terms is the same. Both penalize high values of $\Vert w \Vert^2$ ($\Vert b \Vert^2$ in the linked web) to avoid overfitting and $\lambda$ is the parameter which balances the relative importance of the terms in the cost function. $\endgroup$ – Javier TG Oct 24 '20 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.