Why not perform weight decay on layernorm/embedding? I am learning the code of minGPT. In the function, the author excluded layernorm and embedding layer from experiencing weight decay and I want to know the reasons. Besides, what about batchnorm?
 A: After some searching, I find some explanations:
karpathy's explanation and a disscusion in the pytorch forum.

@morganmcg1 the purpose of L2 regularization is to "spread out" the weights in dot products, ensuring that more "independent measurements" (dimensions of the input) get used more equally, instead of any one feature dominating the computation. This only makes sense for matrix multiply layers, which the embeddings and the layernorm parameters are not. L2 regularization is covered eg in CS231n.

A: The mechanism of weight decay seems to be not clearly understood in the research field. For example, a research paper [1] reported that "the regularization effect was concentrated in the BN layer.

As evidence, we found that almost all of the regularization effect of weight decay was due to applying it to layers with BN (for which weight decay is meaningless).

The reason why such an implementation is widely used in the first place might be that Google's public BERT implementation[2] and any other pioneer's works did so. However, since this trick is not described in the paper[3], I don't think there is clear evidence for it.
Therefore, I believe think there is no evidence beyond "that's what the pioneers did."

Note that, as discussed in the forum[4], the reason for excluding weight decay from updating Layer norm and bias might be based on the paper[5], where the author states "when Batch/Layer/Weight Normalization and weight decay are used together, weight decay regularization is not expected to be effective". However, this paper only points out that "setting |w|=1 avoids variation in the effective learning rate" and does not state whether weight decay should be disabled in the normalization layer.
Reference

*

*[1] https://arxiv.org/abs/1810.12281

*[2] https://github.com/google-research/bert/blob/master/optimization.py#L65

*[3] https://arxiv.org/abs/1908.08962

*[4] https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm

*[5] https://arxiv.org/abs/1706.05350
