3
$\begingroup$

The effect of applying the L2-norm regularization in neural networks is that it penalizes large weights in the model. How does this prevent overfitting?

My assumption is that large weights means that the neurons are heavily reliant on the output from other neurons. For example, in a multilayer perceptron, if one of my neuron has a large weight, it means that it relies heavily on a neuron in the previous layer. Using the L2-norm and forcing the weights to be smaller makes this neuron use the output from the previous layer in a more balanced fashion. Is this reasoning correct?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

This can be viewed in two ways. First, you can see L2 regularization as a prior on the weights to be distributed near 0. Since you have prior information on the possible distribution of weights, this rules out many improbable solutions. If your prior was correct, this will theoretically increase the chance that you get a reasonable answer -- one which will generalize and not overfit. This is sort of like Occam's razor -- in Occam's razor, we prefer short solutions to long solutions. Here, we prefer small weight solutions to large weight solutions.

The second way to see it is that a neural network is a function approximator which is extremely flexible and has a huge model capacity. By flexible I mean that it is relatively easy from an optimization standpoint to tweak the weights a little and arrive at your desired function. L2 regularization helps by lowering model capacity (too much of which causes overfitting) while leaving the network flexible enough to perform well even with the added regularization.

Improve this answer
$\endgroup$
3
  • $\begingroup$ +1 I agree with your answer: L2 penalties limit capacity and act as a prior that favors small weights. But, capacity would be equally limited by many other restrictions (e.g. centering an identical prior over some value other than zero). So, a natural question is: why should small weights be preferred? For example, in the case of sigmoidal activation functions, one reason might be that small weights tend to keep downstream units out of the saturated regime. Would be curious to hear your thoughts. $\endgroup$
    – user20160
    Commented Oct 2, 2017 at 0:21
  • $\begingroup$ Yes, for sigmoid activation, that is a good reason. For relu as well, I suppose being near zero helps take advantage of the nonlinearities. I think what you are asking is an issue which also appears when using occam's razor: why prefer shorter explanations instead of explanations whose sha256 hash is closer to some particular bitstring? The convention of preferring smaller weights or shorter explanations is equally arbitrary. I think it's mostly for convenience. $\endgroup$
    – shimao
    Commented Oct 2, 2017 at 0:31
  • 1
    $\begingroup$ True...I recall some papers saying that Occam's razor is somewhat arbitrary, in that it depends on how we choose to measure complexity. It will work well when our measure assigns low complexity to good models, and not well otherwise. E.g. from an MDL perspective, short codes correspond to large values of the prior, and we do well if we choose a prior concentrated around the 'true' values. Interesting point about relu. $\endgroup$
    – user20160
    Commented Oct 2, 2017 at 1:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.