Why limiting the weights to not grow to big numbers help to avoid overfitting? I understand that in order to avoid overfitting we need to reduce the complexity of the network. Or, in other words we can reduce the degree of polynomial. L1-norm does exactly this - reduces the degree of polynomial by making the solution sparse.
In L2-norm case the weights don't become zero but small. Thus, the degree of polynomial does not decrease, meaning the network does not become less complex, meaning it did not solve the overfitting. So, I don't understand how limiting the weights to not grow to big numbers help to avoid overfitting.
 A: Reducing the weights limits the range of possibilities the model can achieve.
Imagine a simple linear model like this:
$$Y = b_0 + b_1x_1$$
For example's sake say $Y$ is the height of the person and $x_1$ is the weight of the person.
If we don't limit the range of $b_1$ then the model can fit any value to it. Based on what data we use to estimate the parameters, the model might decide that with each additional cm of height the weight of the person increases by 10kg.
By adding a penalty to high values of weights we prevent the model from choosing extremely high estimates. The model is more constrained - it becomes regularized. This limits overfitting because the range of possible values the model can have now is limited.
In turn when you have a lot of features such a constraint will prevent the model from basing all of it's results on a single feature. In the optimization step where the possible solutions will be between choosing one extremely reliable feature and several less reliable ones - the model will use several features. Just because putting big weight on one of them will become too costly. This again might reduce overfitting, because the model that uses several features often can be more robust.
Bottom line: by adding regularisation the space of possible models you can achieve is reduced. And simpler models are less variable, hence less overfitting (but potentially more biased).
The same reasoning should transfer to neural networks.
