I need to have job interview soon, one of the questions may be L1. vs L2 regularization. Yann LeCun explained best to my knowledge the difference between L1 and L2 regularization.
L1 or Lasso: Weights are shrunk every iteration towards zero by a constant equals to ηα. It will eliminate weights (he said inputs in error) that are not very useful.
L2 or Ridge: In the absence of any gradients from C the weights exponentially decay to zero. It tried to tell the system: Minimize my cost function but to it with the weight vector that is as short as possible.
As you can see both the regularization types will minimize/eliminate the weights that are not very useful to zero. So what's the difference really? If I just say L1 will do it based on L1 norm, and L2 based on L2 norm this is probable true but that will not be the scientific mindset.
