5
$\begingroup$

I see this expression: "regularization works because it keeps weights small" almost everywhere on the internet. So I'm going to make a semi-confident assertion here which is more of a question to test my understanding.

This is a misleading statement. Hand-wavingly making weights smaller won't achieve anything. If you take all the weights of a model that's overfitting and downsize them, the resulting function will be just as overly-complex but scaled down.

The real trick of regularization is that you are forcing the training to make choices about which weights it wants to keep large, and which weights it should get rid of by pushing them to zero. So regularization just imposes a kind of weights economy or rationing of weights to the model. The model has to choose which weights will give the most bang for their buck, and get rid of weights which add a small amount of value but don't contribute to a general fit.

So I would rephrase the general expression to: "Regularization works because it keeps a subset of the weights small".

EDIT

I think this answer confirms my thought process.

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ “Keeping the weights small” isn’t such a bad way to describe it. The goal is variance reduction, so if your regularized weights can only be between -2 and 2, there should be less variability than if they can be between-20 and 20 (or really anywhere for unregularized). $\endgroup$
    – Dave
    Commented Mar 29, 2020 at 17:51
  • 1
    $\begingroup$ @Dave Regularization minimizes some target error, hopefully a target not chosen naively, but the usual case is to choose naively or empirically. The connection between error propagation minimization and weights is abstract, so, no, I wouldn't say that. $\endgroup$
    – Carl
    Commented Mar 30, 2020 at 4:00
  • $\begingroup$ @Dave I thought the variance being reduced is in what the model would resolve to when trained under varying training sets. So if the function that the model produces by learning with different training sets varies a lot then that's the variance people talk about - and it's not a good thing. But if the model function has strong variations in parameter space (a totally different concept) and those are indeed a good representation of the function it's trying to model, then that's fine. $\endgroup$
    – Alexander Soare
    Commented Mar 30, 2020 at 5:40

1 Answer 1

Reset to default
5
$\begingroup$

I don't think it is misleading, it is just a bit short. As regularization is (often) on some norm of the parameter (in your terminology weights) vector, like $\| \beta \|^2$ (ridge), it keeps the overall size of the parameters (weights) down. But it does not do so in a blind way, it does so while minimally destroying the other part (negative log likelihood, sum of squares, ...) of the criterion (loss) function. That is consistent with the explanation in your linked post, as restricting the model space.

Since the penalization term can be interpreted as a Bayesian prior in the model space, we are reducing the model space, not by cutting some parts of it off (as we would do by omitting some predictor, say), but by introducing a measure on the space, thereby down-weighting parts of it.

It turns out that we are downweighting those parts corresponding to large parameter vectors.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Good answer. I still think it might be misleading but that becomes more of an argument about English semantics than anything. $\endgroup$
    – Alexander Soare
    Commented Oct 25, 2021 at 19:03
  • $\begingroup$ Sometimes we see the total sum of weights increasing temporarily with increased lambda, what type of interaction in the model space do you think is happening then?\ $\endgroup$
    – Oliver
    Commented Sep 3, 2022 at 6:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.