# How to decouple weight decay strength and model size?

Consider the neural networks' loss function with the cross entropy term and the $$L^2$$ weight decay term, which are usually written as:

$$E = \frac{1}{N_{samples}} \sum_{i=1}^{N_{samples}} \text{cross_entropy}\left(x_i, y_i\right) + \lambda \sum_{j=1}^{N_{parameters}}\left(w_j\right)^2$$

The weight decay term can be written as either "sum square" or "mean square". They are equivalent by a scaling of $$\lambda$$ when the number of parameters is fixed, as discussed here and here.

However, the problem appears when the number of parameters increases, and we have to re-tune the weight decay strength $$\lambda$$. Let's consider the two options:

1. The "sum square" of parameters can become huge; thus, it can completely dominate the cross entropy loss, which is relatively unchanged in magnitude regardless of model size. This means the model is overly regularized and we need to decrease $$\lambda$$ to reduce bias. The good side of this option is that, the derivative of the weight decay term is $$\lambda w_j$$, meaning we reduce each parameter by a fixed amount $$\lambda$$ at each gradient update regardless of the model size. Thus, this option seems "bad" when considering the relative loss values, but seems "correct" when considering the gradient; how to unite this discrepancy?

2. For "mean square" weight decay, the weight decay term is relatively unchanged in magnitude regardless of model size; thus, the relative magnitude between the cross entropy loss and the weight decay loss is unchanged. So, $$\lambda$$ can stay the same (or set to slightly larger value to account for overfitting risk in larger model). However, the bad side of this option is that, the derivative is $$\frac{\lambda}{N_{parameters}} w_j$$, which become very small when the model size increases. Thus, this option seems "good" when considering the relative loss values, but seems "bad" (incorrect?) when considering the gradient; how to unite this discrepancy?

I cannot make my mind which option is better. Is it reasonable to use "mean square" weight decay to have a stable $$\lambda$$ regardless of the model size, or did I miss something?

• Note that different to other related questions, this question concerns the heuristics for the stability of the optimal $\lambda$ when model size changes. – THN Apr 26 at 5:39

The loss function should be either: $$E = \sum_i^{N_{samples}} \text{cross_entropy}(x_i,y_i) + \lambda \sum_j^{N_{params}} w_j^2$$ Or the averaged version: $$\bar E = \frac{1}{N_{samples}} E \\ = \frac{1}{N_{samples}} \sum_i^{N_{samples}} \text{cross_entropy}(x_i,y_i) + \frac{1}{N_{samples}} \lambda \sum_j^{N_{params}} w_j^2$$
• Please note the two distinct $N_{samples}$ and $N_{parameters}$ in my equation. Your suggestion seems reasonable, but I am concerning the averaging over $N_{parameters}$ of the weight decay term. – THN Apr 26 at 8:05
• The problem is cross entropy loss stays relatively the same when model size increases, thus the increase in penalty for the "sum" version can be too much. In practice I have experienced strong underfitting when I used the same $\lambda$ for larger model size. The "mean" version seemed to be more stable. However, I agree that the "sum" version has more elegant maths and nice meaning. Maybe I will use it and retune $\lambda$ from scratch every time. – THN Apr 26 at 13:13
• I think your suggestion on averaging the weight decay term over number of samples is a special case of the suggestion in this paper by Bengio (arxiv.org/abs/1206.5533). It is to account for the number of gradient updates in one epoch of (mini-batch) SGD. However, when batch size > 1, we should divide by (N sample / batch size). This can be omitted and tuned in $\lambda$ directly if all mini batches have the same size. – THN Apr 26 at 13:20