# Weight decay VS “model-capacity-reduction” regularization

In artificial neural networks, is weight decay regularization the same sort of regularization that would reducing the capacity of the model be?

I've learned that applying weight decay shrinks the weights towards smaller values, and this tends to avoid overfitting. Okey, I see that. In the figure below it is clear that the overfitted model has regions of large curvature, and this could be a consequence of having high values of the weights.

I've learned that reducing the model's capacity, i.e. reducing the number of layers or the number of units per layer, also avoids overfitting (depends also on the complexity of the task), because the more flexible the model the more it will tend to overfit. This can also be seen in the figure below, where the model that overfits chose that particular solution but could have chosen many others that would still fit the data, but not the underlying "data generation process" (expression that I don't get to fully grasp).

So again, my question is, are weight decay and model-capacity-reduction achieving the same sort of regularization? Or they achieve something different? And if so, how?

(Figure taken from www.deeplearningbook.org by Ian Goodfellow et al. )

Weight decay is actually a way of reducing model capacity. Model capacity measures how well a variety of functions can be fit. A way to reduce model capacity is to constrain the hypothesis space--that is, the set of all possible functions the learning algorithm can produce. In neural nets, this is the set of functions corresponding to every possible choice of parameters. Clearly, removing units and/or layers reduces this set. I'll explain why weight decay does too.

Weight decay is commonly expressed as penalty on the $\ell_2$ norm of the weights. Say $L(w)$ represents the error on the training set for weights $w$. The learning problem is:

$$\min_w \ L(w) + \lambda \|w\|^2$$

where the hyperparameter $\lambda$ controls the strength of weight decay. This problem has an equivalent, dual formulation, expressed in terms of a constraint rather than a penalty:

$$\min_w \ L(w) \quad \text{s.t. } \|w\| < c$$

This means that for every choice of $\lambda$, there's a corresponding choice of $c$ such that the two problems have the same solutions. The constraint formulation makes it clear that weight decay is constraining the solution to lie in a hypersphere of radius $c$. Increasing the strength of weight decay (i.e. increasing $\lambda$) corresponds to shrinking this hypersphere (i.e. decreasing $c$).

Because weight decay discards all sets of weights that lie outside the hypersphere, the corresponding functions are no longer part of the hypothesis space. So, weight decay reduces model capacity.

• Thanks for the comment @user20160, I had never seen this dual formulation with the hypersphere. But one thing remains unclear for me: Does weight decay reduce the model's capacity in exacly the same way as reducing the number of layers for instance? Why having weight decay then? and not just reduce the number of parameters (reducing layers or units per layer)... It can't only be for a more handy way of reducing model's capacity, or? – sdiabr May 31 '18 at 12:10
• A follow-up question on this: is the model the formulation with the specific weights or is it the multi-layer architecture before the training? Because once the training is done, there are specific weights to the formulation and so there's no meaning of talking about capacity of this, I think? – SPRajagopal Jul 3 at 10:19

In the limit as decay goes to infinity, weight decay is the same as setting the corresponding parameter to zero. In that sense, weight decay is a generalization of reducing model complexity by removing elements.

• Thanks for the answer @Neil G. I see what you say from user20160's answer "Increasing the strength of weight decay (i.e. increasing λ) corresponds to shrinking this hypersphere (i.e. decreasing c)." , but could you elaborate a bit more from where do you draw your first statement?, I don't see it from the "traditional" way of deriving weight decay (for example, page 231 of the book I mentioned) – sdiabr May 31 '18 at 12:16
• @sdiabr If the weight decay is infinite, the weights are zero since the loss penalty is infinite for all nonzero weights. – Neil G May 31 '18 at 19:31