How Lasso regression helps feature selection of model by making the coefficient zero?
I could see few below with below diagram. Can anyone explain in simple terms how to correlate below diagram with i.) how lasso shrinks the coefficient to zero, ii.) how ridge dose not shrink the coefficient to zero?
Consider you have weights, i.e. coefficients, in a weight-vector: $$w = \left[\matrix{w_1& \dots &w_p}\right]$$
Consider both the $\ell_1$ and $\ell_2^2$ norms of this vector:
$$ \cases{ \ell_1(w)=\|w\|_1=\sum_i|w_i|\\ \ell_2^2(w)=\|w\|_2^2=\sum_i w_i^2} $$
When minimizing $\sum_i|w_i|$, minimizing any of the $|w_i|$ by a $\delta$ helps equally to the overall minimization. Thus, you take the same bit from each when minimizing. So, some may reach $0$ before others: these are "pruned", and stop contributing to the problem.
When minimizing $\sum_i w_i^2$, however, diminishing the same little bit each $|w_i|$ does not help equally! Due to the square, larger $|w_i|$ contributes more to the loss than smaller $|w_i|$. So, the optimization prioritizes removing more from the large elements. To be precise, the amount of magnitude in play here, to be "shrinked", is proportional to the magnitude of the weights: larger weights lose more, smaller ones lose less. When you keep going down this gradient path, weights become progressively smaller. The only way to shrink them to zero is to make them all zero at the same time, so that no weight prevails over other due to their magnitude being larger.
Consider two scenarios, where the value of certain element in my weight matrix (matrix of parameters) undergo the following changes:
scenario 1: 1000 ---> 999 [the value reduces from 1000 to 999]
scenario 2: 1.1 ---> 0 [the value reduces from 1.1 to 0]
Now you could inspect for the L1 (Lasso) and L2 (Ridge) regularization, which change in weight value led to a greater reduction in the loss function.
Assume that the common loss term attributed to the data has a constant value and the only value affecting the total loss is the regularization term.
For L1 regularization, since the regularization term is $|w_{i}|$, the change in error for:
Since scenario 2 leads to a higher reduction in total Loss, the Lasso regression tries to execute scenario 2
For L2 regularization, since the regularization term is $|w_{i}|^{2}$, the change in error for:
Since scenario 1 leads to a higher reduction in total Loss, the Ridge regression tries to execute scenario 1.
Now I hope you understand why Lasso drives parameters to zero. This why its called the sparsity constraint :)
