# Minimization of the loss function in soft-margin SVM

According to Wikipedia, the goal of the soft-margin SVM is to minize the hinge loss function:

$$\left[\frac{1}{n} \sum_{i=1}^{n} \max \left(0,1-y_{i}\left(\vec{w} \cdot \vec{x}_{i}-b\right)\right)\right]+\lambda\|\vec{w}\|^{2}$$

Could you tell me more why we add $$\lambda$$? What is its effect on the minimization?

The goal of $$\lambda$$ in that equation is to serve as a regularization term (helping to avoid overfitting) which determines the relative importance of minimizing $$\Vert w \Vert^2$$ w.r.t. minimizing $$\frac{1}{n}\sum_{i=1}^n\max(0, 1-y_i(w\cdot x_i - b))$$.

By minimizing $$\frac{1}{n}\sum_{i=1}^n\max(0, 1-y_i(w\cdot x_i - b))$$ we are looking forward to correctly separate the data and with a functional margin $$\geq 1$$, otherwise the cost function will increase. But minimizing only this term may lead us to undesired results.

This is because in order to separate the samples correctly, the SVM may overfit the dataset. This usually leads to higher values of $$\Vert w \Vert^2$$ due to the increasing complexity needed to fit the whole dataset correctly.

To prevent this, we add a regularization term $$\rightarrow \lambda\Vert w \Vert^2$$. By doing this, we are not only penalising the fact that the functional margin is $$<1$$, but also high values of $$\Vert w \Vert^2$$.

However, we should not minimize $$\Vert w \Vert^2$$ indefinitely, because by doing this we are reducing the capacity of the SVM to fit the data $$\rightarrow$$ we may end up with the opposite problem than before i.e. underfitting the dataset.

So, to sum up, a good balance between minimizing $$\frac{1}{n}\sum_{i=1}^n\max(0, 1-y_i(w\cdot x_i - b))$$ and minimizing $$\Vert w \Vert^2$$ needs to be met and this why $$\lambda$$ is used.

• Thanks for the explanation! So is it similar to the Ridge penalization? Oct 24, 2020 at 16:48
• I'm not familiar with Ridge penalization but looking at this post from Statlec it seems that the objective of both regularization terms is the same. Both penalize high values of $\Vert w \Vert^2$ ($\Vert b \Vert^2$ in the linked web) to avoid overfitting and $\lambda$ is the parameter which balances the relative importance of the terms in the cost function. Oct 24, 2020 at 17:16
• When it comes to soft margin SVM, people often mention the introduction of a slack variable $ξ$ for each $x$. In the above loss function, can I interpret $1-y_i(w\cdot x_i - b)$ as $ξ$? May 21 at 13:48