When we use L1-regularization in neural networks, it is pretty intuitive how the regularization will influence the learned weights. Namely, weights will not become needlessly large and unimportant weights will go to 0, resulting in a sparser matrix.
I wonder if there is a similar intuitive intuition for L2 regularization. This could also be in terms of the linear maps that are described by the weights (maybe less extreme stretching of dimensions?).
When using SVMs, we minimize the weights in the L2-norm to get the hyperplane classifier that maximizes the geometric margin. Is there a relation between L2-norm and margin in neural networks?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Welcome to Cross Validated! How would you interpret L2 regularization in a simple regression model like a linear regression? $\endgroup$– DaveAug 31, 2022 at 17:07
-
$\begingroup$ Thank you for your reply! As I understand it now, L2 regularization will result in a smaller Frobenius norm and thus in smaller eigenvalues of the map (which answers the first part of my question, thanks!). I think that I still don't understand the connection to generalization though. Is there something you can point me to regarding the second part? $\endgroup$– user2611844Aug 31, 2022 at 17:22
Add a comment
|