Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
According to Wikipedia, the goal of the soft-margin SVM is to minize the hinge loss function:
$$\left[\frac{1}{n} \sum_{i=1}^{n} \max \left(0,1-y_{i}\left(\vec{w} \cdot \vec{x}_{i}-b\right)\right)\right]+\lambda\|\vec{w}\|^{2}$$
Could you tell me more why we add $\lambda$? What is its effect on the minimization?
The goal of $\lambda$ in that equation is to serve as a regularization term (helping to avoid overfitting) which determines the relative importance of minimizing $\Vert w \Vert^2$ w.r.t. minimizing $\frac{1}{n}\sum_{i=1}^n\max(0, 1-y_i(w\cdot x_i - b))$.
By minimizing $\frac{1}{n}\sum_{i=1}^n\max(0, 1-y_i(w\cdot x_i - b))$ we are looking forward to correctly separate the data and with a functional margin $\geq 1$, otherwise the cost function will increase. But minimizing only this term may lead us to undesired results.
This is because in order to separate the samples correctly, the SVM may overfit the dataset. This usually leads to higher values of $\Vert w \Vert^2$ due to the increasing complexity needed to fit the whole dataset correctly.
To prevent this, we add a regularization term $\rightarrow \lambda\Vert w \Vert^2$. By doing this, we are not only penalising the fact that the functional margin is $<1$, but also high values of $\Vert w \Vert^2$.
However, we should not minimize $\Vert w \Vert^2$ indefinitely, because by doing this we are reducing the capacity of the SVM to fit the data $\rightarrow$ we may end up with the opposite problem than before i.e. underfitting the dataset.
So, to sum up, a good balance between minimizing $\frac{1}{n}\sum_{i=1}^n\max(0, 1-y_i(w\cdot x_i - b))$ and minimizing $\Vert w \Vert^2$ needs to be met and this why $\lambda$ is used.
