Should we average weight decay loss in neural network?

Question

In a typical neural network, which way is the common way to add regularization?

Assuming regression task, regression error loss is Mean-squared-error

Then we can have two choice of regularization on weights:

1. $\lambda$ * $\sum ||W||^2$
2. $\lambda$ * $\textbf{average} ||W||^2$

I have seen most people use the first option, just being curious to ask.

Answer 1

Using the average implicitly rescales $\lambda$. This means that choosing the average or the sum isn't really consequential, because whatever the optimal $\lambda$ is on the mean scale has an equivalent choice of $\lambda$ on the sum scale, and vice versa. $$ \begin{align} \lambda \sum_i w_i^2 &= \lambda\sum_iw_i^2 \\ &= {n\lambda} \left[\frac{1}{n}\sum_iw_i^2 \right]\\ \end{align} $$